提交Yarn簇上的spark作业

epggiuax  于 2021-06-02  发布在  Hadoop
关注(0)|答案(3)|浏览(384)

我现在挣扎了两天多，有以下问题。
我用scala写了一个基本的“helloworld”脚本：

object Hello extends App{
  println("WELCOME TO A FIRST TEST WITH SCALA COMPILED WITH SBT counting fr. 1:15 with sleep 1")
  val data = 1 to 15

  for( a <- data ){
    println( "Value of a: " + a )
    Thread sleep 1000
  }

然后我用sbt进行编译以获得jar编译版本。
然后，我用hdp2.2.4.2将集群上的所有内容（这是运行在虚拟linux机器上的horthonworks沙盒）都传输了。
实际上，我可以使用以下命令在集群上使用一个yarn客户机运行作业：

spark-submit --verbose --master yarn-client --class Hello SCALA/hello.jar

但是，在尝试使用以下命令提交Yarn集群上的同一helloworld作业时

spark-submit --verbose --master yarn-cluster--class Hello SCALA/hello.jar

作业首先正常运行（输出是预期的输出，它退出0），但随后作业停止并执行以下操作：

15/06/05 15:52:09 INFO Client: Application report for application_1433491352951_0010 (state: FAILED)

15/06/05 15:52:09 INFO Client:
         client token: N/A
         diagnostics: Application application_1433491352951_0010 failed 2 times due to AM Container for appattempt_1433491352951_0010_000002 exited with  exitCode: 0
For more detailed output, check application tracking page:http://sandbox.hortonworks.com:8088/proxy/application_1433491352951_0010/Then, click on links to logs of each attempt.
Diagnostics: Failing this attempt. Failing the application.
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1433519471297
         final status: FAILED
         tracking URL: http://sandbox.hortonworks.com:8088/cluster/app/application_1433491352951_0010
         user: root
Error: application failed with exception
org.apache.spark.SparkException: Application finished with failed status
        at org.apache.spark.deploy.yarn.ClientBase$class.run(ClientBase.scala:522)
        at org.apache.spark.deploy.yarn.Client.run(Client.scala:35)
        at org.apache.spark.deploy.yarn.Client$.main(Client.scala:139)
        at org.apache.spark.deploy.yarn.Client.main(Client.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:367)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:77)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

然后，我决定使用以下命令行检查日志：

yarn logs -applicationId application_1433491352951_00010

我得到：

15/06/05 15:56:33 INFO impl.TimelineClientImpl: Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/
15/06/05 15:56:33 INFO client.RMProxy: Connecting to ResourceManager at sandbox.hortonworks.com/192.168.182.129:8050
15/06/05 15:56:35 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
15/06/05 15:56:35 INFO compress.CodecPool: Got brand-new decompressor [.deflate]

Container: container_e08_1433491352951_0010_01_000001 on sandbox.hortonworks.com_45454
========================================================================================
LogType:stderr
Log Upload Time:Fri Jun 05 15:52:10 +0000 2015
LogLength:2050
Log Contents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/hadoop/yarn/local/usercache/root/filecache/28/spark-assembly-1.2.1.2.2.4.2-2-hadoop2.6.0.2.2.4.2-2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.4.2-2/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop/yarn/local/usercache/root/filecache/29/hello.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/06/05 15:51:18 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
15/06/05 15:51:20 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1433491352951_0010_000001
15/06/05 15:51:21 INFO spark.SecurityManager: Changing view acls to: yarn,root
15/06/05 15:51:21 INFO spark.SecurityManager: Changing modify acls to: yarn,root
15/06/05 15:51:21 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, root); users with modify permissions: Set(yarn, root)
15/06/05 15:51:21 INFO yarn.ApplicationMaster: Starting the user JAR in a separate Thread
15/06/05 15:51:21 INFO yarn.ApplicationMaster: Waiting for spark context initialization
15/06/05 15:51:21 INFO yarn.ApplicationMaster: Waiting for spark context initialization ... 0
15/06/05 15:51:31 INFO yarn.ApplicationMaster: Waiting for spark context initialization ... 1
15/06/05 15:51:36 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0
15/06/05 15:51:41 ERROR yarn.ApplicationMaster: SparkContext did not initialize after waiting for 100000 ms. Please check earlier log output for errors. Failing the application.
15/06/05 15:51:41 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED
15/06/05 15:51:41 INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1433491352951_0010

LogType:stdout
Log Upload Time:Fri Jun 05 15:52:10 +0000 2015
LogLength:300
Log Contents:
WELCOME TO A FIRST TEST WITH SCALA COMPILED WITH SBT counting fr. 1:15 with sleep 1
Value of a: 1
Value of a: 2
Value of a: 3
Value of a: 4
Value of a: 5
Value of a: 6
Value of a: 7
Value of a: 8
Value of a: 9
Value of a: 10
Value of a: 11
Value of a: 12
Value of a: 13
Value of a: 14
Value of a: 15

Container: container_e08_1433491352951_0010_02_000001 on sandbox.hortonworks.com_45454
========================================================================================
LogType:stderr
Log Upload Time:Fri Jun 05 15:52:10 +0000 2015
LogLength:2050
Log Contents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/hadoop/yarn/local/usercache/root/filecache/28/spark-assembly-1.2.1.2.2.4.2-2-hadoop2.6.0.2.2.4.2-2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.4.2-2/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop/yarn/local/usercache/root/filecache/29/hello.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/06/05 15:51:45 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
15/06/05 15:51:47 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1433491352951_0010_000002
15/06/05 15:51:48 INFO spark.SecurityManager: Changing view acls to: yarn,root
15/06/05 15:51:48 INFO spark.SecurityManager: Changing modify acls to: yarn,root
15/06/05 15:51:48 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, root); users with modify permissions: Set(yarn, root)
15/06/05 15:51:48 INFO yarn.ApplicationMaster: Starting the user JAR in a separate Thread
15/06/05 15:51:48 INFO yarn.ApplicationMaster: Waiting for spark context initialization
15/06/05 15:51:48 INFO yarn.ApplicationMaster: Waiting for spark context initialization ... 0
15/06/05 15:51:58 INFO yarn.ApplicationMaster: Waiting for spark context initialization ... 1
15/06/05 15:52:03 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0
15/06/05 15:52:08 ERROR yarn.ApplicationMaster: SparkContext did not initialize after waiting for 100000 ms. Please check earlier log output for errors. Failing the application.
15/06/05 15:52:08 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED
15/06/05 15:52:08 INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1433491352951_0010

LogType:stdout
Log Upload Time:Fri Jun 05 15:52:10 +0000 2015
LogLength:300
Log Contents:
WELCOME TO A FIRST TEST WITH SCALA COMPILED WITH SBT counting fr. 1:15 with sleep 1
Value of a: 1
Value of a: 2
Value of a: 3
Value of a: 4
Value of a: 5
Value of a: 6
Value of a: 7
Value of a: 8
Value of a: 9
Value of a: 10
Value of a: 11
Value of a: 12
Value of a: 13
Value of a: 14
Value of a: 15

我接受了有人建议的helloworld项目，重新编译，然后再试一次。现在我遇到了另一个问题：当我使用以下命令提交任务时：

spark-submit --verbose --master yarn-cluster SCALA/hello.jar

我得到了如下评论：

15/06/08 16:42:35 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

我真的不明白，因为当程序应该从沙盒运行在hadoop集群上时，服务器似乎没有响应。

hadoopscalaapache-sparkJobs

3条答案

按热度按时间
vmpqdwk3

vmpqdwk31#

我在你的代码中看不到任何spark上下文初始化。因此，通过将jar作为spark submit提交，您只是在设置类路径。你的代码没有使用spark。但是，如果您想创建一个上下文并在集群模式下运行它，那么不要在代码中设置任何主节点。会有用的。例如： val config = new SparkConf().setAppName("HelloWorld") val context = new SparkContext(config); 然后使用 spark-submit --verbose --master yarn-cluster--class Hello SCALA/hello.jar

赞(0）分享  回复(0）举报  2021-06-03
rsaldnfx

rsaldnfx2#

在我的例子中，我使用了：

val config = new SparkConf()
config.setMaster("local[*]")

提交作业时使用：

spark-submit --master yarn-cluster ..

一旦我搬走 config.setMaster 根据我的代码，问题得到了解决。

赞(0）分享  回复(0）举报  2021-06-03
dsf9zpds

dsf9zpds3#

您必须将应用程序的主控形状设置为“Yarn簇”，如下所示：

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

class Hello extends App {
    val config = new SparkConf().setAppName("HelloWorld")
    config.setMaster("yarn-cluster");
    val context = new SparkContext(config);

    println("Application executed!")
}
赞(0）分享  回复(0）举报  2021-06-03
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com