pyspark 无法将spark应用程序提交到docker compose

wgeznvg7  于 2024-01-06  发布在  Spark
关注(0)|答案(1)|浏览(191)

我尝试在docker-compose上提交一个小的spark应用程序到spark集群。但是，它总是失败。下面是我在docker-compose上的spark集群（我已经有一些Kafka代理忽略它）：

  1. version: '3'
  2. services:
  3.   spark-master:
  4.     image: bitnami/spark:3.5
  5.     container_name: spark-master
  6.     hostname: spark-master
  7.     environment:
  8.       - SPARK_MODE=master
  9.       - SPARK_MASTER_HOSTNAME=spark-master
  10.       - SPARK_MASTER_PORT=7077
  11.       - PYSPARK_PYTHON=/opt/bitnami/python/bin/python3
  12.     ports:
  13.       - "4040:4040"
  14.       - "6066:6066"
  15.       - "7077:7077"
  16.       - "8080:8080"
  17.     deploy:
  18.       resources:
  19.         limits:
  20.           cpus: "1" # Adjust as needed
  21.           memory: 1g  # Adjust as needed
  22.     # command: bin/spark-class org.apache.spark.deploy.master.Master
  23.     networks:
  24.       - spark-network
  26.   spark-worker-1:
  27.     image: bitnami/spark:3.5
  28.     container_name: spark-worker-1
  29.     environment:
  30.       - SPARK_MODE=worker
  31.       - SPARK_MASTER_URL=spark://spark-master:7077
  32.       - SPARK_WORKER_MEMORY = 3g
  33.       - SPARK_WORKER_CORES = 2
  34.       - PYSPARK_PYTHON=/opt/bitnami/python/bin/python3
  35.     deploy:
  36.       resources:
  37.         limits:
  38.           cpus: "2" # Adjust as needed
  39.           memory: 4G  # Adjust as needed
  40.     depends_on:
  41.       - spark-master
  42.     networks:
  43.       - spark-network
  45.   spark-worker-2:
  46.     image: bitnami/spark:3.5
  47.     container_name: spark-worker-2
  48.     environment:
  49.       - SPARK_MODE=worker
  50.       - SPARK_MASTER_URL=spark://spark-master:7077
  51.       - SPARK_WORKER_MEMORY = 3g
  52.       - SPARK_WORKER_CORES = 2
  53.       - PYSPARK_PYTHON=/opt/bitnami/python/bin/python3
  54.     deploy:
  55.       resources:
  56.         limits:
  57.           cpus: "2" # Adjust as needed
  58.           memory: 4G  # Adjust as needed
  59.     depends_on:
  60.       - spark-master
  61.     networks:
  62.       - spark-network
  64.   zookeeper:
  65.     image: bitnami/zookeeper:3.9
  66.     container_name: zookeeper-server
  67.     restart: always
  68.     ports:
  69.       - "2181:2181"
  70.     environment:
  71.       - ALLOW_ANONYMOUS_LOGIN=yes
  73.   kafka1:
  74.     image: bitnami/kafka:3.5
  75.     container_name: broker-1
  76.     hostname: broker-1
  77.     ports:
  78.       - "9093:9093"
  79.     environment:
  80.       - KAFKA_BROKER_ID=1
  81.       - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
  82.       - KAFKA_CFG_LISTENERS=PLAINTEXT://:9093,INTERNAL://:9092
  83.       - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9093,INTERNAL://:9092
  84.       - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,INTERNAL:PLAINTEXT
  85.       - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=INTERNAL
  86.     depends_on:
  87.       - zookeeper 
  89.   kafka2:
  90.     image: bitnami/kafka:3.5
  91.     container_name: broker-2
  92.     hostname: broker-2
  93.     ports:
  94.       - "9094:9094"
  95.     environment:
  96.       - KAFKA_BROKER_ID=2
  97.       - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
  98.       - KAFKA_CFG_LISTENERS=PLAINTEXT://:9094,INTERNAL://:9092
  99.       - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9094,INTERNAL://:9092
  100.       - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,INTERNAL:PLAINTEXT
  101.       - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=INTERNAL
  102.     depends_on:
  103.       - zookeeper
  105.   kafka3:
  106.     image: bitnami/kafka:3.5
  107.     container_name: broker-3
  108.     hostname: broker-3
  109.     ports:
  110.       - "9095:9095"
  111.     environment:
  112.       - KAFKA_BROKER_ID=3
  113.       - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
  114.       - KAFKA_CFG_LISTENERS=PLAINTEXT://:9095,INTERNAL://:9092
  115.       - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9095,INTERNAL://:9092
  116.       - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,INTERNAL:PLAINTEXT
  117.       - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=INTERNAL
  118.     depends_on:
  119.       - zookeeper  
  120.   
  121. networks:
  122.   spark-network:
  123.     driver: bridge

字符串
简单的Spark App：

  1. from pyspark.sql import SparkSession
  2. from pyspark.sql.functions import *
  4. # Create a SparkSession
  5. spark = SparkSession.builder \
  6.    .appName("My App") \
  7.    .master("spark://172.28.240.1:7077") \
  8.    .getOrCreate()
  10. rdd = spark.sparkContext.parallelize(range(1, 100))
  12. print("THE SUM IS HERE: ", rdd.sum())
  13. # Stop the SparkSession
  14. spark.stop()

型
当我提交时，它会引发错误：

  1. 23/12/08 01:44:54 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0) (172.20.0.2 executor 0): java.io.IOException: Cannot run program "C:\Program Files\Python310\python.exe": error=2, No such file or directory
  2.         at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)
  3.         at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)
  4.         at org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:239)
  5.         at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:139)
  6.         at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:107)
  7.         at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)
  8.         at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:174)
  9.         at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:67)
  10.         at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
  11.         at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
  12.         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
  13.         at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
  14.         at org.apache.spark.scheduler.Task.run(Task.scala:141)
  15.         at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
  16.         at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
  17.         at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
  18.         at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
  19.         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
  20.         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  21.         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  22.         at java.base/java.lang.Thread.run(Thread.java:840)
  23. Caused by: java.io.IOException: error=2, No such file or directory
  24.         at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
  25.         at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:314)
  26.         at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:244)
  27.         at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)
  28.         ... 20 more
  30. 23/12/08 01:44:54 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job
  31. Traceback (most recent call last):
  32.   File "D:\IT4931-bigdata-Movie-processing\test_submit.py", line 12, in <module>
  33.     print("THE SUM IS HERE: ", rdd.sum())
  34.   File "C:\Users\ABC\AppData\Roaming\Python\Python310\site-packages\pyspark\rdd.py", line 2291, in sum
  35.     return self.mapPartitions(lambda x: [sum(x)]).fold(  # type: ignore[return-value]
  36.   File "C:\Users\ABC\AppData\Roaming\Python\Python310\site-packages\pyspark\rdd.py", line 2044, in fold
  37.     vals = self.mapPartitions(func).collect()
  38.   File "C:\Users\ABC\AppData\Roaming\Python\Python310\site-packages\pyspark\rdd.py", line 1833, in collect
  39.     sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
  40.   File "C:\Users\ABC\AppData\Roaming\Python\Python310\site-packages\py4j\java_gateway.py", line 1322, in __call__
  41.     return_value = get_return_value(
  42.   File "C:\Users\ABC\AppData\Roaming\Python\Python310\site-packages\pyspark\errors\exceptions\captured.py", line 179, in deco
  43.     return f(*a, **kw)
  44.   File "C:\Users\ABC\AppData\Roaming\Python\Python310\site-packages\py4j\protocol.py", line 326, in get_return_value
  45.     raise Py4JJavaError(
  46. py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
  47. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 5) (172.20.0.2 executor 0): java.io.IOException: Cannot run program "C:\Program Files\Python310\python.exe": error=2, No such file or directory
  48.         at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)
  49.         at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)
  50.         at org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:239)
  51.         at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:139)
  52.         at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:107)
  53.         at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)
  54.         at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:174)
  55.         at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:67)
  56.         at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
  57.         at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
  58.         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
  59.         at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
  60.         at org.apache.spark.scheduler.Task.run(Task.scala:141)
  61.         at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
  62.         at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
  63.         at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
  64.         at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
  65.         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
  66.         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  67.         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  68.         at java.base/java.lang.Thread.run(Thread.java:840)
  69. Caused by: java.io.IOException: error=2, No such file or directory
  70.         at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
  71.         at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:314)
  72.         at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:244)
  73.         at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)
  74.         ... 20 more
  76. Driver stacktrace:
  77.         at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2844)
  78.         at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2780)
  79.         at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2779)
  80.         at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
  81.         at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
  82.         at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
  83.         at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2779)
  84.         at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1242)
  85.         at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1242)
  86.         at scala.Option.foreach(Option.scala:407)
  87.         at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1242)
  88.         at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3048)
  89.         at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2982)
  90.         at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2971)
  91.         at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
  92.         at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:984)
  93.         at org.apache.spark.SparkContext.runJob(SparkContext.scala:2398)
  94.         at org.apache.spark.SparkContext.runJob(SparkContext.scala:2419)
  95.         at org.apache.spark.SparkContext.runJob(SparkContext.scala:2438)
  96.         at org.apache.spark.SparkContext.runJob(SparkContext.scala:2463)
  97.         at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1046)
  98.         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  99.         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  100.         at org.apache.spark.rdd.RDD.withScope(RDD.scala:407)
  101.         at org.apache.spark.rdd.RDD.collect(RDD.scala:1045)
  102.         at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:195)
  103.         at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
  104.         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  105.         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  106.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  107.         at java.lang.reflect.Method.invoke(Method.java:498)
  108.         at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
  109.         at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
  110.         at py4j.Gateway.invoke(Gateway.java:282)
  111.         at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
  112.         at py4j.commands.CallCommand.execute(CallCommand.java:79)
  113.         at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
  114.         at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
  115.         at java.lang.Thread.run(Thread.java:748)
  116. Caused by: java.io.IOException: Cannot run program "C:\Program Files\Python310\python.exe": error=2, No such file or directory
  117.         at java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)
  118.         at java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)
  119.         at org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:239)
  120.         at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:139)
  121.         at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:107)
  122.         at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:124)
  123.         at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:174)
  124.         at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:67)
  125.         at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
  126.         at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
  127.         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
  128.         at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
  129.         at org.apache.spark.scheduler.Task.run(Task.scala:141)
  130.         at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
  131.         at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
  132.         at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
  133.         at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
  134.         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
  135.         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  136.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  137.         at java.lang.Thread.run(Thread.java:840)
  138. Caused by: java.io.IOException: error=2, No such file or directory
  139.         at java.lang.ProcessImpl.forkAndExec(Native Method)
  140.         at java.lang.ProcessImpl.<init>(ProcessImpl.java:314)
  141.         at java.lang.ProcessImpl.start(ProcessImpl.java:244)
  142.         at java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)
  143.         ... 20 more

型
我只是一个初学者在Spark。我已经试图修复它一整天，仍然有错误。有人可以帮助我，请？谢谢。

pyspark

1条答案

按热度按时间
dgtucam1

dgtucam11#

看起来这个问题与Spark无法找到Python可执行文件（C:\Program Files\Python310\python.exe）有关。这可能是由于该路径特定于Windows环境，但您的Spark集群运行在Dockerized Linux环境中。
1.更新Spark应用程序中的Python路径，并使用127.0.0.1代替spark-master：

  1. spark = SparkSession.builder \
  2.    .appName("My App") \
  3.    .master("spark://127.0.0.1:7077") \
  4.    .config("spark.executorEnv.PYSPARK_PYTHON", "/opt/bitnami/python/bin/python3") \
  5.    .getOrCreate()

字符串
1.使用spark-submit提交Spark申请：

  1. spark-submit --master spark://127.0.0.1:7077 spark_app.py

型
1.使用http：//localhost：8080上提供的Web UI监控Spark应用程序

展开查看全部
赞(0）分享  回复(0）举报  2024-01-06
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com