我正在emr步骤中运行spark应用程序,但由于某些错误,作业失败,我想查看该错误。我已经检查了stderr,但是它没有给出任何关于错误的详细信息。是这么说的
Exception in thread "main" org.apache.spark.SparkException: Application application_1593934145491_0002 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1149)
at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1526)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:853)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:928)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:937)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
20/07/05 07:50:37 INFO ShutdownHookManager: Shutdown hook called
有人能帮我吗?我想查看应用程序级日志。
2条答案
按热度按时间uujelgoq1#
启用调试模式并在客户机上运行脚本后,我能够在steps/step\u id/stdout.gz中看到spark应用程序级日志
zf2sa74q2#
它应该总是在/container下,但是如果找不到它,请尝试ssh主节点并运行spark submit