jupyterlab容器无法连接到端口9092上的kafka localhost服务器

w51jfk4q  于 2021-05-16  发布在  Spark
关注(0)|答案(1)|浏览(685)

我有docker compose包含一个kafka服务器、一个zookeeper、spark和一个jupyter-lab。这是我的jupyter-lab dockerfile代码

  1. ARG debian_buster_image_tag=8-jre-slim
  2. FROM openjdk:${debian_buster_image_tag}
  3. RUN mkdir -p ${shared_workspace} && \
  4. apt-get update -y && \
  5. apt-get install -y python3 && \
  6. ln -s /usr/bin/python3 /usr/bin/python && \
  7. rm -rf /var/lib/apt/lists/* && \
  8. apt-get install -y python3-pip && \
  9. pip3 install pyspark==3.0.1 jupyterlab==2.1.5

这是我的docker-compose.yml代码

  1. version: "3.6"
  2. volumes:
  3. shared-workspace:
  4. name: "hadoop-distributed-file-system"
  5. driver: local
  6. services:
  7. jupyterlab:
  8. image: jupyterlab
  9. container_name: jupyterlab
  10. ports:
  11. - 8888:8888
  12. volumes:
  13. - shared-workspace:/opt/workspace
  14. spark-master:
  15. image: spark-master
  16. container_name: spark-master
  17. ports:
  18. - 8080:8080
  19. - 7077:7077
  20. volumes:
  21. - shared-workspace:/opt/workspace
  22. spark-worker-1:
  23. image: spark-worker
  24. container_name: spark-worker-1
  25. environment:
  26. - SPARK_WORKER_CORES=1
  27. - SPARK_WORKER_MEMORY=512m
  28. ports:
  29. - 8081:8081
  30. volumes:
  31. - shared-workspace:/opt/workspace
  32. depends_on:
  33. - spark-master
  34. spark-worker-2:
  35. image: spark-worker
  36. container_name: spark-worker-2
  37. environment:
  38. - SPARK_WORKER_CORES=1
  39. - SPARK_WORKER_MEMORY=512m
  40. ports:
  41. - 8082:8081
  42. volumes:
  43. - shared-workspace:/opt/workspace
  44. depends_on:
  45. - spark-master
  46. zookeeper:
  47. image: wurstmeister/zookeeper:3.4.6
  48. expose:
  49. - "2181"
  50. kafka:
  51. image: wurstmeister/kafka:latest
  52. depends_on:
  53. - zookeeper
  54. ports:
  55. - "9092:9092"
  56. expose:
  57. - "9093"
  58. environment:
  59. KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://localhost:9092
  60. KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
  61. KAFKA_LISTENERS: INSIDE://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
  62. KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  63. KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE

在jupyter实验室我无法进入kafka:9093, localhost:9092 by sparksession.readstream.format(“kafka”).option(“kafka.bootstrap.servers”,”kafka:9093“”…,显示错误:

  1. 20/12/03 16:10:23 WARN NetworkClient: [Consumer clientId=consumer-spark-kafka-source-4b4422bc-2a6a-456a-b750-4fba8300328c-37706672-driver-0-1, groupId=spark-kafka-source-4b4422bc-2a6a-456a-b750-4fba8300328c-37706672-driver-0] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected

我还安装了telnet通过jupyterlab的bash检查连接,我可以telnet到一些服务器,比如 telnet www.cyberciti.biz 80 ,但无法连接到localhost:9092. 请帮帮我。

368yc8dk

368yc8dk1#

根据@onecricketeer的说法,我认为我之前的答案是一条红鲱鱼。
所以,我查了你的 Dockerfile 以及 docker-compose.yaml 而且,这对我很有效。

  1. ARG debian_buster_image_tag=8-jre-slim
  2. FROM openjdk:${debian_buster_image_tag}
  3. RUN apt-get update -y && \
  4. apt-get install -y python3 python3-pip && \
  5. rm -rf /var/lib/apt/lists/* && \
  6. pip3 install pyspark==3.0.1 jupyterlab==2.1.5
  7. ENTRYPOINT [ "jupyter","notebook","--allow-root","--ip=0.0.0.0"]

笔记
远离的 ${shared_workspace} 因为这看起来 /opt/workspace 重新排序 apt-get install s
补充 ENTRYPOINTjupyter-notebook --allow-root -ip=0.0.0.0 这将公开上的笔记本服务 localhost:8888 但需要一个令牌。
调整了合成:

  1. services:
  2. jupyterlab:
  3. image: jupyterlab
  4. build:
  5. context: .
  6. dockerfile: ./Dockerfile
  7. container_name: jupyterlab
  8. ports:
  9. - 8888:8888
  10. volumes:
  11. - shared-workspace:/opt/workspace

然后:

  1. docker-compose build && docker-compose up

以及 docker-compose ps :

  1. Name Command State Ports
  2. ------------------------------------------------------------------------------------------------------------------------
  3. 65130144_kafka_1 start-kafka.sh Up 0.0.0.0:9092->9092/tcp, 9093/tcp
  4. 65130144_zookeeper_1 /bin/sh -c /usr/sbin/sshd ... Up 2181/tcp, 22/tcp, 2888/tcp, 3888/tcp
  5. jupyterlab jupyter notebook --allow-r ... Up 0.0.0.0:8888->8888/tcp
  6. spark-master /bin/bash /master.sh Up 6066/tcp, 0.0.0.0:7077->7077/tcp, 0.0.0.0:8080->8080/tcp
  7. spark-worker-1 /bin/bash /worker.sh Up 0.0.0.0:8081->8081/tcp
  8. spark-worker-2 /bin/bash /worker.sh Up 0.0.0.0:8082->8081/tcp

你可能想要笔记本的代币,用吧 docker-compose logs jupyterlab :

  1. Attaching to jupyterlab
  2. jupyterlab | [I 17:57:08.412 NotebookApp] Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret
  3. jupyterlab | [I 17:57:08.659 NotebookApp] JupyterLab extension loaded from /usr/local/lib/python3.7/dist-packages/jupyterlab
  4. jupyterlab | [I 17:57:08.659 NotebookApp] JupyterLab application directory is /usr/local/share/jupyter/lab
  5. jupyterlab | [I 17:57:08.662 NotebookApp] Serving notebooks from local directory: /
  6. jupyterlab | [I 17:57:08.662 NotebookApp] Jupyter Notebook 6.1.5 is running at:
  7. jupyterlab | [I 17:57:08.662 NotebookApp] http://1b77e2b01d7b:8888/?token=e1796ee09331bd31459211ce35ad16fc9a7f4ee854870457
  8. jupyterlab | [I 17:57:08.662 NotebookApp] or http://127.0.0.1:8888/?token=e1796ee09331bd31459211ce35ad16fc9a7f4ee854870457
  9. jupyterlab | [I 17:57:08.662 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
  10. jupyterlab | [W 17:57:08.665 NotebookApp] No web browser found: could not locate runnable browser.
  11. jupyterlab | [C 17:57:08.666 NotebookApp]
  12. jupyterlab |
  13. jupyterlab | To access the notebook, open this file in a browser:
  14. jupyterlab | file:///root/.local/share/jupyter/runtime/nbserver-1-open.html
  15. jupyterlab | Or copy and paste one of these URLs:
  16. jupyterlab | http://1b77e2b01d7b:8888/?token=e1796ee09331bd31459211ce35ad16fc9a7f4ee854870457
  17. jupyterlab | or http://127.0.0.1:8888/?token=e1796ee09331bd31459211ce35ad16fc9a7f4ee854870457

使用第二个链接(您需要使用生成的任何令牌),创建一个笔记本并添加:

  1. from pyspark.sql import SparkSession;
  2. spark = SparkSession \
  3. .builder \
  4. .appName("test") \
  5. .getOrCreate()
  6. spark.readStream.format("kafka").option("kafka.bootstrap.servers", "kafka:9093")

然后运行它:

  1. <pyspark.sql.streaming.DataStreamReader at 0x7fbf8defe208>

似乎是(!?)好 啊。

我怀疑 jupyter-lab 以及 jupyterlab 在配置中是相同的。
dockerfile不运行(否 CMD 或者 ENTRYPOINT )所以我也假设你在里面使用了一个shell。这很好只是不寻常。
有几件事 kafka 服务(注意,我只对Kafka略知一二):
kafka 您必须将其引用为 localhost 你有 INSIDE://kafka:9093 但这有两个问题。在服务内部,它应该将自己引用为 localhost 在服务区内我怀疑它在港口 :8082 声明 expose: - "9093" 不是功能性的,而是记录片,除非你的图像也在无声地曝光,否则是不正确的 :9093 ; 这实际上没有暴露任何东西,因此可能是不正确的。

展开查看全部

相关问题