spark:在web ui上获取应用程序的详细信息

qxgroojn  于 2021-05-29  发布在  Spark
关注(0)|答案(0)|浏览(396)

我已经在kubernetes上使用bitnami/spark-helm图表部署了一个大师和一个工人的spark。

CHART               APP VERSION
spark-1.2.22        2.4.5

当我试图通过web界面访问当前运行的spark提交的应用程序的一些细节时,链接被破坏了。
我是说,他们指向的网址是:
应用程序ui详细信息:http://10.0.2.15:4040/
标准输出:http://10.130.2.112:8081/logpage?appid=app-20200604143941-0000&executoid=11&logtype=stdout
正如你所见,这些网址的主机是固定的IP地址,而且,他们是内部IP地址。。。
我还查看了spark worker pods日志:

20/06/04 15:04:08 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(spark); groups with view permissions: Set(); users  with modify permissions: Set(spark); groups with modify permissions: Set()
20/06/04 15:04:08 INFO ExecutorRunner: Launch command: "/opt/bitnami/java/bin/java" "-cp" "/opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/*" "-Xmx1024M" "-Dspark.driver.port=46253" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@10.0.2.15:46253" "--executor-id" "12" "--hostname" "10.130.2.112" "--cores" "1" "--app-id" "app-20200604143941-0000" "--worker-url" "spark://Worker@10.130.2.112:46839"

如您所见,这里有如下参数: --hostname" "10.130.2.112 即上述固定ip。
重现行为的步骤:
helm install spark bitnami/spark --set worker.replicaCount=1 --set service.type=NodePort MASTER=spark://<host>:<port> ./run-example SparkPi 我想看看网上的应用程序日志
helm和kubernetes版本:
的输出 helm version :

version.BuildInfo{Version:"v3.1.2", GitCommit:"d878d4d45863e42fd5cff6743294a11d28a9abce", GitTreeState:"clean", GoVersion:"go1.13.8"}

的输出 kubectl version :

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"clean", BuildDate:"2020-04-08T17:38:50Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0+d4cacc0", GitCommit:"d4cacc0", GitTreeState:"clean", BuildDate:"2020-01-20T23:21:20Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题