spark提交给kubernetes上的Yarn资源经理

dzhpxtsq  于 2021-05-29  发布在  Spark
关注(0)|答案(0)|浏览(258)

我在kubernetes上创建了Yarn,HDF,如下面的豆荚:

  1. hd2-hdfs-datanode-0 1/1 Running 0 40m
  2. hd2-hdfs-datanode-1 1/1 Running 0 39m
  3. hd2-hdfs-datanode-2 0/1 Running 14 39m
  4. hd2-hdfs-httpfs-6b948765d6-nd6j7 1/1 Running 0 40m
  5. hd2-hdfs-namenode-0 2/2 Running 2 40m
  6. hd2-hdfs-yarn-nm-0 1/1 Running 1 40m
  7. hd2-hdfs-yarn-nm-1 1/1 Running 1 39m
  8. hd2-hdfs-yarn-rm-0 1/1 Running 0 40m

服务:

  1. hd2-hdfs ClusterIP 10.233.38.246 <none> 8020/TCP,50070/TCP 4m45s
  2. hd2-hdfs-datanode ClusterIP None <none> 50075/TCP 4m45s
  3. hd2-hdfs-httpfs ClusterIP 10.233.46.69 <none> 14000/TCP 4m45s
  4. hd2-hdfs-namenode ClusterIP None <none> 8020/TCP,50070/TCP 4m45s
  5. hd2-hdfs-namenode-exporter ClusterIP 10.233.7.161 <none> 5556/TCP 4m45s
  6. hd2-hdfs-yarn-nm ClusterIP None <none> 8088/TCP,8082/TCP,8042/TCP 4m45s
  7. hd2-hdfs-yarn-rm ClusterIP 10.233.17.3 <none> 8088/TCP,8032/TCP 4m45s
  8. hd2-hdfs-yarn-ui ClusterIP 10.233.37.237 <none> 8088/TCP 4m45s

我正试图将spark部署到kubernetes系统中。Spark应该在Yarn上运行(在kubernetes上),集群模式。因此,我将bitnami spark helm chart部署到这个k8s集群中,并尝试使用以下命令运行spark示例

  1. spark-submit --master yarn --deploy-mode cluster --class org.apache.spark.examples.SparkPi examples/jars/spark-examples_2.11-2.4.6.jar 5

但它不能到达Yarn资源管理器(我想),因为我看到下面的日志:

  1. 20/06/15 06:49:39 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  2. 20/06/15 06:49:40 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  3. 20/06/15 06:49:41 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  4. 20/06/15 06:49:42 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  5. 20/06/15 06:49:43 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  6. 20/06/15 06:49:44 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  7. 20/06/15 06:49:45 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  8. 20/06/15 06:49:46 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  9. 20/06/15 06:49:47 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  10. 20/06/15 06:49:48 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  11. ^C20/06/15 06:50:07 INFO ShutdownHookManager: Shutdown hook called

我想这是因为资源管理器在不同的pod中找不到它。有人试过这个吗??

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题