我尝试在kubernetes上部署spark，并遵循这个指南https://dev.to/akoshel/spark-on-k8s-in-jupyterhub-1da2，我成功地从本地系统使用spark-submit运行了示例pi，它运行得很好，我能够看到pod处于完成状态。

/opt/spark/bin/spark-submit \
  --master k8s://https://127.0.0.1:62013 \
  --deploy-mode cluster \
  --driver-memory 1g \
  --conf spark.kubernetes.memoryOverheadFactor=0.5 \
  --name sparkpi-test1 \
  --class org.apache.spark.examples.SparkPi \
  --conf spark.kubernetes.container.image=spark:latest \
  --conf spark.kubernetes.driver.pod.name=spark-test1-pi \
  --conf spark.kubernetes.namespace=spark \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  --verbose \
  local:///opt/spark/examples/jars/spark-examples_2.12-3.3.2.jar 1000

然后我开始部署jupyterhub后配置所有的服务帐户，角色和角色绑定，但我面临的问题是，每当我试图在我的jupyterhub运行spark config时，它显示错误，主要是无法连接到pod，我使用minikube测试部署，如果有人能帮助我，那就太好了。
从pyspark import SparkConf，SparkContext conf =（SparkConf（）. setMaster（"k8s：//www.example.com"）#您的主地址名称。set（"spark. Kubernetes。集装箱image"，"pyspark：latest"）#Spark镜像名称。set（"spark.司机port"，"2222"）#需要匹配svc. set（"spark.司机blockManager。端口”、“7777”）。set（" www.example.com "，" driver-service. jupyterhub服务集群local "）#需要匹配svc。set（" spark.司机bindAddress”、“www.example.com”）。set（"spark. kubernetes。命名空间”、“spark”）。set（" spark. Kubernetes。认证。司机。serviceAccountName "，" spark"）。set（" spark. Kubernetes。认证。serviceAccountName"，" spark"）。set（" spark.遗嘱执行人示例”、“2”）。set（" spark. Kubernetes。集装箱图像。pullPolicy”、“IfNotPresent”）。set（"https://127.0.0.1:52750") # Your master address name .set("spark.kubernetes.container.image", "pyspark:latest") # Spark image name .set("spark.driver.port", "2222") # Needs to match svc .set("spark.driver.blockManager.port", "7777") .set("spark.driver.host ", "driver-service.jupyterhub.svc.cluster.local") # Needs to match svc .set("spark.driver.bindAddress", "0.0.0.0 ") .set("spark.kubernetes.namespace", "spark") .set("spark.kubernetes.authenticate.driver.serviceAccountName", "spark") .set("spark.kubernetes.authenticate.serviceAccountName", "spark") .set("spark.executor.instances", "2") .set("spark.kubernetes.container.image.pullPolicy", "IfNotPresent") .set(" www.example.com ", "tutorial_app"))
我创造了这个然后跑了

创建SparkContext

sc = SparkContext（conf = conf）
/home/jovyan/. local/lib/python3.9/site-packages/pyspark/bin/www.example.com：line 68：ps：load-spark-env.sh原因：java. net. ConnectException：无法连接到/127.0.0.1：52750输出以获取群集的概念**Caused by: java.net.ConnectException: Failed to connect to /127.0.0.1:52750output to get the idea of the cluster**

root@TIGER03720:~# k get all -n jupyterhub -o wide
NAME                                  READY   STATUS    RESTARTS      AGE    IP            NODE       NOMINATED NODE   READINESS GATES
pod/continuous-image-puller-qrjr8     1/1     Running   1 (42s ago)   104m   10.244.0.14   minikube   <none>           <none>
pod/hub-6d64d94c89-54vvp              0/1     Error     0             104m   <none>        minikube   <none>           <none>
pod/jupyter-admin                     0/1     Error     0             102m   10.244.0.18   minikube   <none>           <none>
pod/proxy-5c6db96f8-wg9jc             0/1     Running   1 (42s ago)   104m   10.244.0.16   minikube   <none>           <none>
pod/user-scheduler-86cfcff58c-v4m8q   0/1     Running   1 (42s ago)   104m   10.244.0.15   minikube   <none>           <none>
pod/user-scheduler-86cfcff58c-zdv9n   0/1     Running   1 (42s ago)   104m   10.244.0.20   minikube   <none>           <none>

NAME                     TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE    SELECTOR
service/driver-service   ClusterIP      10.110.72.226    <none>        2222/TCP,7777/TCP,4040/TCP   109m   app=jupyterhub,component=singleuser-server
service/hub              ClusterIP      10.106.88.232    <none>        8081/TCP                     104m   app=jupyterhub,component=hub,release=jupyterhub
service/proxy-api        ClusterIP      10.105.217.11    <none>        8001/TCP                     104m   app=jupyterhub,component=proxy,release=jupyterhub
service/proxy-public     LoadBalancer   10.108.107.209   <pending>     80:31888/TCP                 104m   component=proxy,release=jupyterhub

NAME                                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE    CONTAINERS   IMAGES                 SELECTOR
daemonset.apps/continuous-image-puller   1         1         1       1            1           <none>          104m   pause        k8s.gcr.io/pause:3.8   app=jupyterhub,component=continuous-image-puller,release=jupyterhub

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS       IMAGES
                         SELECTOR
deployment.apps/hub              0/1     1            0           104m   hub              jupyterhub/k8s-hub:2.0.0                   app=jupyterhub,component=hub,release=jupyterhub
deployment.apps/proxy            0/1     1            0           104m   chp              jupyterhub/configurable-http-proxy:4.5.3   app=jupyterhub,component=proxy,release=jupyterhub
deployment.apps/user-scheduler   0/2     2            0           104m   kube-scheduler   k8s.gcr.io/kube-scheduler:v1.23.10         app=jupyterhub,component=user-scheduler,release=jupyterhub

NAME                                        DESIRED   CURRENT   READY   AGE    CONTAINERS       IMAGES                                     SELECTOR
replicaset.apps/hub-6d64d94c89              1         1         0       104m   hub              jupyterhub/k8s-hub:2.0.0                   app=jupyterhub,component=hub,pod-template-hash=6d64d94c89,release=jupyterhub
replicaset.apps/proxy-5c6db96f8             1         1         0       104m   chp              jupyterhub/configurable-http-proxy:4.5.3   app=jupyterhub,component=proxy,pod-template-hash=5c6db96f8,release=jupyterhubreplicaset.apps/user-scheduler-86cfcff58c   2         2         0       104m   kube-scheduler   k8s.gcr.io/kube-scheduler:v1.23.10         app=jupyterhub,component=user-scheduler,pod-template-hash=86cfcff58c,release=jupyterhub

NAME                                READY   AGE    CONTAINERS   IMAGES
root@TIGER03720:~#

root@TIGER03720:~# k get all -n spark -o wide
NAME                  READY   STATUS      RESTARTS   AGE    IP           NODE       NOMINATED NODE   READINESS GATES
pod/spark-test1-pi2   0/1     Completed   0          114m   10.244.0.3   minikube   <none>           <none>

忽略POD状态错误!