我已经为namenode和resourcemanager设置了一个带有ha的3节点hadoop集群。我还在一台namenode机器上安装了spark作业服务器。
我已经测试了运行job服务器的测试示例,比如wordcount示例和longpi job,它运行得非常完美,没有任何问题。我还可以从远程主机发出curl命令,通过spark作业服务器读取结果。
但是,当我将“spark-examples-1.6.0-hadoop2.6.0.jar”上传到spark job server/jars并尝试运行sparkpi job时失败了,
[hduser@ptfhadoop02v lib]$ curl -d "" 'ptfhadoop01v:8090/jobs?appName=SparkPi&classPath=org.apache.spark.examples.SparkPi'
{
"status": "ERROR",
"result": {
"message": "Ask timed out on [Actor[akka://JobServer/user/context-supervisor/ece2be39-org.apache.spark.examples.SparkPi#-630965857]] after [10000 ms]",
"errorClass": "akka.pattern.AskTimeoutException",
"stack":["akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)", "akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)", "scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694)", "scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:691)", "akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)", "akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)", "akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)", "akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)", "java.lang.Thread.run(Thread.java:745)"]
}
我还尝试手动将sparkpi.scala作业放置在/usr/local/hadoop/spark jobserver/job server tests/src/spark.jobserver下,并使用sbt构建包,但它抛出了相同的错误。
版本信息
[hduser@ptfhadoop01v spark.jobserver]$ sbt sbtVersion
[info] Set current project to spark-jobserver (in build file:/usr/local/hadoop/spark-jobserver/job-server-tests/src/spark.jobserver/)
[info] 0.13.11
Spark Version - spark-1.6.0
Scala Version - 2.10.4
关于如何消除这个错误并从spark-examples jar文件中获取输出有什么建议吗
1条答案
按热度按时间li9yvcax1#
我通过修改代码来扩展sparkjob,从而成功地让它工作了,感谢您的澄清