spark应用程序停留在可接受状态

qfe3c7zg  于 2021-06-03  发布在  Hadoop
关注(0)|答案(3)|浏览(377)

我在一台ubuntu14.04服务器上安装了一个新的cloudera5.4示例,并希望运行一个spark应用程序。
这是命令:

sudo -uhdfs spark-submit --class org.apache.spark.examples.SparkPi --deploy-mode cluster --master yarn /opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/jars/spark-examples-1.3.0-cdh5.4.5-hadoop2.6.0-cdh5.4.5.jar

这是输出:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/jars/avro-tools-1.7.6-cdh5.4.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/08/29 12:07:56 INFO RMProxy: Connecting to ResourceManager at chd2.moneyball.guru/104.131.78.0:8032
15/08/29 12:07:56 INFO Client: Requesting a new application from cluster with 1 NodeManagers
15/08/29 12:07:56 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (1750 MB per container)
15/08/29 12:07:56 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
15/08/29 12:07:56 INFO Client: Setting up container launch context for our AM
15/08/29 12:07:56 INFO Client: Preparing resources for our AM container
15/08/29 12:07:57 INFO Client: Uploading resource file:/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/jars/spark-examples-1.3.0-cdh5.4.5-hadoop2.6.0-cdh5.4.5.jar -> hdfs://chd2.moneyball.guru:8020/user/hdfs/.sparkStaging/application_1440861466017_0007/spark-examples-1.3.0-cdh5.4.5-hadoop2.6.0-cdh5.4.5.jar
15/08/29 12:07:57 INFO Client: Setting up the launch environment for our AM container
15/08/29 12:07:57 INFO SecurityManager: Changing view acls to: hdfs
15/08/29 12:07:57 INFO SecurityManager: Changing modify acls to: hdfs
15/08/29 12:07:57 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hdfs); users with modify permissions: Set(hdfs)
15/08/29 12:07:57 INFO Client: Submitting application 7 to ResourceManager
15/08/29 12:07:57 INFO YarnClientImpl: Submitted application application_1440861466017_0007
15/08/29 12:07:58 INFO Client: Application report for application_1440861466017_0007 (state: ACCEPTED)
15/08/29 12:07:58 INFO Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: root.hdfs
     start time: 1440864477580
     final status: UNDEFINED
     tracking URL: http://chd2.moneyball.guru:8088/proxy/application_1440861466017_0007/
     user: hdfs
15/08/29 12:07:59 INFO Client: Application report for application_1440861466017_0007 (state: ACCEPTED)
15/08/29 12:08:00 INFO Client: Application report for application_1440861466017_0007 (state: ACCEPTED)
15/08/29 12:08:01 INFO Client: Application report for application_1440861466017_0007 (state: ACCEPTED)
15/08/29 12:08:02 INFO Client: Application report for application_1440861466017_0007 (state: ACCEPTED)
15/08/29 12:08:03 INFO Client: Application report for application_1440861466017_0007 (state: ACCEPTED)
15/08/29 12:08:04 INFO Client: Application report for application_1440861466017_0007 (state: ACCEPTED)
15/08/29 12:08:05 INFO Client: Application report for application_1440861466017_0007 (state: ACCEPTED)
15/08/29 12:08:06 INFO Client: Application report for application_1440861466017_0007 (state: ACCEPTED)
15/08/29 12:08:07 INFO Client: Application report for application_1440861466017_0007 (state: ACCEPTED
.....

它将显示循环中的最后一行。你能帮忙吗?如果你还需要什么,请告诉我。

mklgxw1f

mklgxw1f1#

我增加了 yarn.nodemanager.resource.memory-mb . 现在一切都好了

r6l8ljro

r6l8ljro2#

当Yarn的插槽被其他作业占用并且集群处于其容量时,就会发生这种情况。作业被卡在接受状态,等待轮到它运行。你能从yarn资源管理器ui检查一下集群上是否有其他任何东西正在运行,这可能会减慢这个应用程序的速度吗?可以通过转到访问rm uihttp://104.131.78.0:8088,假设您的rm地址仍然是104.131.78.0,如日志中所示。您应该能够看到1)是否有任何其他应用程序正在集群上运行,以及2)导航到正在集群上运行的spark uihttp://applicationmasteraddress:4040进行进一步分析。

xzabzqsa

xzabzqsa3#

我在Spark1.5.2上遇到了一个类似的问题,通过使用scala解决了这个问题 object 控制我的情绪 main 函数,而不是scala class

相关问题