使用ApacheLivy创建spark会话时出错

ljo96ir5  于 2021-05-31  发布在  Hadoop
关注(0)|答案(0)|浏览(450)

我正在尝试使用apachelivy创建一个spark会话,但是我无法在spark集群上创建会话,也没有得到任何有用的日志。
我有没有办法找出这个错误的真正原因。
任何帮助都将不胜感激。
我在livy ui上看到的日志是:

Session 7 unexpectedly reached final status 'dead'. See logs:
stdout: 

stderr: 
20/05/03 14:44:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/05/03 14:44:55 INFO client.RMProxy: Connecting to ResourceManager at streamflux-master/172.17.0.2:8032
20/05/03 14:44:55 INFO yarn.Client: Requesting a new application from cluster with 1 NodeManagers
20/05/03 14:44:55 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (10240 MB per container)
20/05/03 14:44:55 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
20/05/03 14:44:55 INFO yarn.Client: Setting up container launch context for our AM
20/05/03 14:44:55 INFO yarn.Client: Setting up the launch environment for our AM container
20/05/03 14:44:55 INFO yarn.Client: Preparing resources for our AM container
20/05/03 14:44:55 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
20/05/03 14:44:57 INFO yarn.Client: Uploading resource file:/tmp/spark-f25d699f-f00d-400d-bd47-9a42adf1fd01/__spark_libs__8419058456894841499.zip -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/__spark_libs__8419058456894841499.zip
20/05/03 14:44:58 INFO yarn.Client: Uploading resource file:/home/streamflux/usr/local/livy/rsc-jars/livy-rsc-0.6.0-incubating.jar -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/livy-rsc-0.6.0-incubating.jar
20/05/03 14:44:58 INFO yarn.Client: Uploading resource file:/home/streamflux/usr/local/livy/rsc-jars/netty-all-4.0.37.Final.jar -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/netty-all-4.0.37.Final.jar
20/05/03 14:44:58 INFO yarn.Client: Uploading resource file:/home/streamflux/usr/local/livy/rsc-jars/livy-thriftserver-session-0.6.0-incubating.jar -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/livy-thriftserver-session-0.6.0-incubating.jar
20/05/03 14:44:58 INFO yarn.Client: Uploading resource file:/home/streamflux/usr/local/livy/rsc-jars/livy-api-0.6.0-incubating.jar -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/livy-api-0.6.0-incubating.jar
20/05/03 14:44:58 INFO yarn.Client: Uploading resource file:/home/streamflux/usr/local/livy/repl_2.11-jars/livy-core_2.11-0.6.0-incubating.jar -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/livy-core_2.11-0.6.0-incubating.jar
20/05/03 14:44:58 INFO yarn.Client: Uploading resource file:/home/streamflux/usr/local/livy/repl_2.11-jars/commons-codec-1.9.jar -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/commons-codec-1.9.jar
20/05/03 14:44:59 INFO yarn.Client: Uploading resource file:/home/streamflux/usr/local/livy/repl_2.11-jars/livy-repl_2.11-0.6.0-incubating.jar -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/livy-repl_2.11-0.6.0-incubating.jar
20/05/03 14:44:59 INFO yarn.Client: Uploading resource file:/home/streamflux/usr/local/spark/R/lib/sparkr.zip#sparkr -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/sparkr.zip
20/05/03 14:44:59 INFO yarn.Client: Uploading resource file:/home/streamflux/usr/local/spark/python/lib/pyspark.zip -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/pyspark.zip
20/05/03 14:44:59 INFO yarn.Client: Uploading resource file:/home/streamflux/usr/local/spark/python/lib/py4j-0.10.7-src.zip -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/py4j-0.10.7-src.zip
20/05/03 14:44:59 WARN yarn.Client: Same name resource file:///home/streamflux/usr/local/spark/python/lib/pyspark.zip added multiple times to distributed cache
20/05/03 14:44:59 WARN yarn.Client: Same name resource file:///home/streamflux/usr/local/spark/python/lib/py4j-0.10.7-src.zip added multiple times to distributed cache
20/05/03 14:44:59 INFO yarn.Client: Uploading resource file:/tmp/spark-f25d699f-f00d-400d-bd47-9a42adf1fd01/__spark_conf__6956912608326152031.zip -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/__spark_conf__.zip
20/05/03 14:45:00 INFO spark.SecurityManager: Changing view acls to: streamflux
20/05/03 14:45:00 INFO spark.SecurityManager: Changing modify acls to: streamflux
20/05/03 14:45:00 INFO spark.SecurityManager: Changing view acls groups to: 
20/05/03 14:45:00 INFO spark.SecurityManager: Changing modify acls groups to: 
20/05/03 14:45:00 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(streamflux); groups with view permissions: Set(); users  with modify permissions: Set(streamflux); groups with modify permissions: Set()
20/05/03 14:45:01 INFO yarn.Client: Submitting application application_1588509039643_0015 to ResourceManager
20/05/03 14:45:01 INFO impl.YarnClientImpl: Submitted application application_1588509039643_0015
20/05/03 14:45:01 INFO yarn.Client: Application report for application_1588509039643_0015 (state: ACCEPTED)
20/05/03 14:45:01 INFO yarn.Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: default
     start time: 1588517101092
     final status: UNDEFINED
     tracking URL: http://streamflux-master:8088/proxy/application_1588509039643_0015/
     user: streamflux
20/05/03 14:45:01 INFO util.ShutdownHookManager: Shutdown hook called
20/05/03 14:45:01 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-f25d699f-f00d-400d-bd47-9a42adf1fd01
20/05/03 14:45:01 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-028c83f0-9c34-4f2c-9466-a1fbcd03ac6b

YARN Diagnostics: Application killed by user.

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题