emr步骤中的spark应用程序级日志

dsf9zpds  于 2021-05-27  发布在  Spark
关注(0)|答案(2)|浏览(331)

我正在emr步骤中运行spark应用程序,但由于某些错误,作业失败,我想查看该错误。我已经检查了stderr,但是它没有给出任何关于错误的详细信息。是这么说的

  1. Exception in thread "main" org.apache.spark.SparkException: Application application_1593934145491_0002 finished with failed status
  2. at org.apache.spark.deploy.yarn.Client.run(Client.scala:1149)
  3. at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1526)
  4. at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:853)
  5. at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
  6. at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
  7. at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
  8. at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:928)
  9. at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:937)
  10. at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  11. 20/07/05 07:50:37 INFO ShutdownHookManager: Shutdown hook called

有人能帮我吗?我想查看应用程序级日志。

uujelgoq

uujelgoq1#

启用调试模式并在客户机上运行脚本后,我能够在steps/step\u id/stdout.gz中看到spark应用程序级日志

zf2sa74q

zf2sa74q2#

它应该总是在/container下,但是如果找不到它,请尝试ssh主节点并运行spark submit

相关问题