通过traefik在远程spark集群上运行pyspark

tmb3ates  于 2021-05-18  发布在  Spark
关注(0)|答案(0)|浏览(311)

我使用此示例连接到远程主机:

conf = SparkConf()
conf.setMaster('spark://ip:80')
conf.setAppName('spark-yarn')
sc = SparkContext(conf=conf)

def mod(x):
    import numpy as np
    return (x, np.mod(x, 2))

rdd = sc.parallelize(range(1000)).map(mod).take(10)
``` `ip:8080/dashboard` 导致 `traefik`  Jmeter 板。
这些是火星上的数据 `traefic`  Jmeter 板,在哪里 `web` 入口点有端口80。
![](https://i.stack.imgur.com/GZTN3.png)
设置主url的正确方法是什么?我认为我设置它的方式是错误的,因为我收到了这个错误:

Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.NullPointerException
at org.apache.spark.SparkContext.(SparkContext.scala:613)
at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题