我在kubernetes上创建了Yarn,HDF,如下面的豆荚:
hd2-hdfs-datanode-0 1/1 Running 0 40m
hd2-hdfs-datanode-1 1/1 Running 0 39m
hd2-hdfs-datanode-2 0/1 Running 14 39m
hd2-hdfs-httpfs-6b948765d6-nd6j7 1/1 Running 0 40m
hd2-hdfs-namenode-0 2/2 Running 2 40m
hd2-hdfs-yarn-nm-0 1/1 Running 1 40m
hd2-hdfs-yarn-nm-1 1/1 Running 1 39m
hd2-hdfs-yarn-rm-0 1/1 Running 0 40m
服务:
hd2-hdfs ClusterIP 10.233.38.246 <none> 8020/TCP,50070/TCP 4m45s
hd2-hdfs-datanode ClusterIP None <none> 50075/TCP 4m45s
hd2-hdfs-httpfs ClusterIP 10.233.46.69 <none> 14000/TCP 4m45s
hd2-hdfs-namenode ClusterIP None <none> 8020/TCP,50070/TCP 4m45s
hd2-hdfs-namenode-exporter ClusterIP 10.233.7.161 <none> 5556/TCP 4m45s
hd2-hdfs-yarn-nm ClusterIP None <none> 8088/TCP,8082/TCP,8042/TCP 4m45s
hd2-hdfs-yarn-rm ClusterIP 10.233.17.3 <none> 8088/TCP,8032/TCP 4m45s
hd2-hdfs-yarn-ui ClusterIP 10.233.37.237 <none> 8088/TCP 4m45s
我正试图将spark部署到kubernetes系统中。Spark应该在Yarn上运行(在kubernetes上),集群模式。因此,我将bitnami spark helm chart部署到这个k8s集群中,并尝试使用以下命令运行spark示例
spark-submit --master yarn --deploy-mode cluster --class org.apache.spark.examples.SparkPi examples/jars/spark-examples_2.11-2.4.6.jar 5
但它不能到达Yarn资源管理器(我想),因为我看到下面的日志:
20/06/15 06:49:39 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
20/06/15 06:49:40 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
20/06/15 06:49:41 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
20/06/15 06:49:42 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
20/06/15 06:49:43 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
20/06/15 06:49:44 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
20/06/15 06:49:45 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
20/06/15 06:49:46 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
20/06/15 06:49:47 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
20/06/15 06:49:48 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
^C20/06/15 06:50:07 INFO ShutdownHookManager: Shutdown hook called
我想这是因为资源管理器在不同的pod中找不到它。有人试过这个吗??
暂无答案!
目前还没有任何答案,快来回答吧!