运行pyspark时出现conda env问题

khbbv19g  于 2021-07-14  发布在  Spark
关注(0)|答案(0)|浏览(457)

我正在尝试使用pyspark传输python环境，具体如下：
https://community.cloudera.com/t5/community-articles/running-pyspark-with-conda-env/ta-p/247551httpshttp://henning.kropponline.de/2014/07/18/virtualenv-hadoop-streaming/https://community.cloudera.com/t5/community-articles/using-virtualenv-with-pyspark/ta-p/245932
上次尝试时，我发出了以下命令：

  1. conda create -y -n py35 python=3.5 numpy pandas scikit-learn
  2. conda activate py35
  3. cd /opt/cloudera/parcels/Anaconda/envs/
  4. zip -r py35.zip py35
  6. # Runing the job:
  8. PYSPARK_DRIVER_PYTHON=/opt/cloudera/parcels/Anaconda/envs/py35/bin/python \
  9. PYSPARK_PYTHON=/opt/cloudera/parcels/Anaconda/envs/py35/bin/python \
  10. pyspark2 \
  11. --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=/opt/cloudera/parcels/Anaconda/envs/py35/bin/python \
  12. --master yarn \
  13. --deploy-mode client \
  14. --archives /opt/cloudera/parcels/Anaconda/envs/py35.zip#py35 test.py

测试.py：

  2. # !./py35/bin/python
  4. # -*- coding: utf-8 -*-
  6. from pyspark import SparkConf
  7. from pyspark import SparkContext
  8. conf = SparkConf()
  9. conf.setAppName('spark-yarn')
  10. sc = SparkContext(conf=conf)
  12. def some_function(x):
  13.     # Packages are imported and available from your bundled environment.
  14.     import sklearn
  15.     import pandas
  16.     import numpy as np
  18.     # Use the libraries to do work
  19.     return np.sin(x)**2 + 2
  21. rdd = (sc.parallelize(range(1000))
  22.          .map(some_function)
  23.          .take(10))
  25. print(rdd)

意外结果：异常：worker中的python版本2.7与driver 3.7不同，pyspark无法使用不同的次要版本运行。请检查环境变量pyspark\u python和pyspark\u driver\u python是否正确设置。

  1. 21/04/12 18:23:17 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, server, executor 1): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  2.   File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/pyspark/worker.py", line 267, in main
  3.     ("%d.%d" % sys.version_info[:2], version))
  4. Exception: Python in worker has different version 2.7 than that in driver 3.7, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.
  6.         at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:452)
  7.         at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:588)
  8.         at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:571)
  9.         at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:406)
  10.         at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
  11.         at scala.collection.Iterator$class.foreach(Iterator.scala:891)
  12.         at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
  13.         at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
  14.         at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
  15.         at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
  16.         at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
  17.         at org.apache.spark.InterruptibleIterator.to(InterruptibleIterator.scala:28)
  18.         at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
  19.         at org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28)
  20.         at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
  21.         at org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28)
  22.         at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
  23.         at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
  24.         at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
  25.         at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
  26.         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  27.         at org.apache.spark.scheduler.Task.run(Task.scala:121)
  28.         at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
  29.         at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1405)
  30.         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
  31.         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  32.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  33.         at java.lang.Thread.run(Thread.java:748)
  35. 21/04/12 18:23:17 ERROR scheduler.TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job
  36. Traceback (most recent call last):
  37.   File "<stdin>", line 1, in <module>
  38.   File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/pyspark/rdd.py", line 1055, in count
  39.     return self.mapPartitions(lambda i: [sum(1 for _ in i)]).sum()
  40.   File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/pyspark/rdd.py", line 1046, in sum
  41.     return self.mapPartitions(lambda x: [sum(x)]).fold(0, operator.add)
  42.   File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/pyspark/rdd.py", line 917, in fold
  43.     vals = self.mapPartitions(func).collect()
  44.   File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/pyspark/rdd.py", line 816, in collect
  45.     sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
  46.   File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
  47.   File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/pyspark/sql/utils.py", line 63, in deco
  48.     return f(*a,**kw)
  49.   File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
  50. py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
  51. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, server, executor 1): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  52.   File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/pyspark/worker.py", line 267, in main
  53.     ("%d.%d" % sys.version_info[:2], version))
  54. Exception: Python in worker has different version 2.7 than that in driver 3.7, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.
  56.         at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:452)
  57.         at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:588)
  58.         at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:571)
  59.         at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:406)
  60.         at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
  61.         at scala.collection.Iterator$class.foreach(Iterator.scala:891)
  62.         at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
  63.         at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
  64.         at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
  65.         at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
  66.         at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
  67.         at org.apache.spark.InterruptibleIterator.to(InterruptibleIterator.scala:28)
  68.         at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
  69.         at org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28)
  70.         at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
  71.         at org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28)
  72.         at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
  73.         at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
  74.         at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
  75.         at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
  76.         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  77.         at org.apache.spark.scheduler.Task.run(Task.scala:121)
  78.         at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
  79.         at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1405)
  80.         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
  81.         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  82.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  83.         at java.lang.Thread.run(Thread.java:748)
  85. Driver stacktrace:
  86.         at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1889)
  87.         at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1877)
  88.         at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1876)
  89.         at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
  90.         at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
  91.         at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1876)
  92.         at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
  93.         at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
  94.         at scala.Option.foreach(Option.scala:257)
  95.         at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)
  96.         at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2110)
  97.         at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)
  98.         at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)
  99.         at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
  100.         at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)
  101.         at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
  102.         at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
  103.         at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
  104.         at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126)
  105.         at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:945)
  106.         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  107.         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  108.         at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
  109.         at org.apache.spark.rdd.RDD.collect(RDD.scala:944)
  110.         at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:166)
  111.         at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
  112.         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  113.         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  114.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  115.         at java.lang.reflect.Method.invoke(Method.java:498)
  116.         at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
  117.         at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
  118.         at py4j.Gateway.invoke(Gateway.java:282)
  119.         at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
  120.         at py4j.commands.CallCommand.execute(CallCommand.java:79)
  121.         at py4j.GatewayConnection.run(GatewayConnection.java:238)
  122.         at java.lang.Thread.run(Thread.java:748)
  123. Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  124.   File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/pyspark/worker.py", line 267, in main
  125.     ("%d.%d" % sys.version_info[:2], version))
  126. Exception: Python in worker has different version 2.7 than that in driver 3.7, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.
  128.         at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:452)
  129.         at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:588)
  130.         at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:571)
  131.         at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:406)
  132.         at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
  133.         at scala.collection.Iterator$class.foreach(Iterator.scala:891)
  134.         at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
  135.         at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
  136.         at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
  137.         at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
  138.         at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
  139.         at org.apache.spark.InterruptibleIterator.to(InterruptibleIterator.scala:28)
  140.         at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
  141.         at org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28)
  142.         at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
  143.         at org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28)
  144.         at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
  145.         at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
  146.         at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
  147.         at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
  148.         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  149.         at org.apache.spark.scheduler.Task.run(Task.scala:121)
  150.         at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
  151.         at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1405)
  152.         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
  153.         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  154.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  155.         ... 1 more

我运行的是spark 2.4.0，基本python是2.7.16
合同通用条款7.3.0/ol6
cdh5.13.3-是的，它很旧。。
如有任何意见，我们将不胜感激。谢谢。

yarnapache-sparkpysparkcloudera-cdhconda

暂无答案！

目前还没有任何答案，快来回答吧！

我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com