SparkMesos星团为执行器中的Spark类设置了错误的路径

wtzytmuj  于 2021-05-27  发布在  Spark
关注(0)|答案(1)|浏览(460)

我有一个flaskapi,使用pyspark启动spark并将作业发送到mesos集群。
执行器失败是因为它参与了flaskapi中spark类所在的路由。
日志:

  1. I0903 13:16:14.295672 4754 exec.cpp:164] Version: 1.9.0
  2. I0903 13:16:14.299304 4760 exec.cpp:237] Executor registered on agent 3ac2aa4b-ff5b-4340-a56e-efe96b3b07a2-S2
  3. I0903 13:16:14.301093 4758 executor.cpp:190] Received SUBSCRIBED event
  4. I0903 13:16:14.301807 4758 executor.cpp:194] Subscribed executor on XXXXX
  5. I0903 13:16:14.301923 4758 executor.cpp:190] Received LAUNCH event
  6. I0903 13:16:14.302549 4758 executor.cpp:722] Starting task 3
  7. I0903 13:16:14.310398 4758 executor.cpp:738] Forked command at 4764
  8. sh: /home/user/source/environment/lib/python3.7/site-packages/pyspark/./bin/spark-class: No such file or directory
  9. I0903 13:16:14.397004 4760 executor.cpp:1039] Command exited with status 127 (pid: 4764)
  10. I0903 13:16:15.398034 4763 process.cpp:935] Stopped the socket accept loop

粗体路径是spark类实际位于flask服务器(作业请求者)中的部分
sh:/home/user/source/environment/lib/python3.7/site packages/pyspark//bin/spark class:没有这样的文件或目录
在这里:

  1. -rwxr-xr-x 1 root root 3.2K Sep 1 11:07 /home/user/source/environment/lib/python3.7/site-packages/pyspark/bin/spark-class

可能我缺少一个环境变量,或者我可以将路径作为参数发送吗?

s2j5cfk0

s2j5cfk01#

通过添加指向spark二进制文件的此属性,在executor中添加spark所在的路径来解决:

  1. spark.mesos.executor.home

相关问题