jupyterlab容器无法连接到端口9092上的kafka localhost服务器

w51jfk4q  于 2021-05-16  发布在  Spark
关注(0)|答案(1)|浏览(685)

我有docker compose包含一个kafka服务器、一个zookeeper、spark和一个jupyter-lab。这是我的jupyter-lab dockerfile代码

  1. ARG debian_buster_image_tag=8-jre-slim
  2. FROM openjdk:${debian_buster_image_tag}
  3. RUN mkdir -p ${shared_workspace} && \
  4.     apt-get update -y && \
  5.     apt-get install -y python3 && \
  6.     ln -s /usr/bin/python3 /usr/bin/python && \
  7.     rm -rf /var/lib/apt/lists/* && \
  8.     apt-get install -y python3-pip && \
  9.     pip3 install pyspark==3.0.1 jupyterlab==2.1.5

这是我的docker-compose.yml代码

  1. version: "3.6"
  2. volumes:
  3.   shared-workspace:
  4.     name: "hadoop-distributed-file-system"
  5.     driver: local
  6. services:
  7.   jupyterlab:
  8.     image: jupyterlab
  9.     container_name: jupyterlab
  10.     ports:
  11.       - 8888:8888
  12.     volumes:
  13.       - shared-workspace:/opt/workspace
  15.   spark-master:
  16.     image: spark-master
  17.     container_name: spark-master
  18.     ports:
  19.       - 8080:8080
  20.       - 7077:7077
  21.     volumes:
  22.       - shared-workspace:/opt/workspace
  23.   spark-worker-1:
  24.     image: spark-worker
  25.     container_name: spark-worker-1
  26.     environment:
  27.       - SPARK_WORKER_CORES=1
  28.       - SPARK_WORKER_MEMORY=512m
  29.     ports:
  30.       - 8081:8081
  31.     volumes:
  32.       - shared-workspace:/opt/workspace
  33.     depends_on:
  34.       - spark-master
  35.   spark-worker-2:
  36.     image: spark-worker
  37.     container_name: spark-worker-2
  38.     environment:
  39.       - SPARK_WORKER_CORES=1
  40.       - SPARK_WORKER_MEMORY=512m
  41.     ports:
  42.       - 8082:8081
  43.     volumes:
  44.       - shared-workspace:/opt/workspace
  45.     depends_on:
  46.       - spark-master
  48.   zookeeper:
  49.     image: wurstmeister/zookeeper:3.4.6
  50.     expose:
  51.     - "2181"
  53.   kafka:
  54.     image: wurstmeister/kafka:latest
  55.     depends_on:
  56.     - zookeeper
  57.     ports:
  58.     - "9092:9092"
  59.     expose:
  60.     - "9093"
  61.     environment:
  62.       KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://localhost:9092
  63.       KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
  64.       KAFKA_LISTENERS: INSIDE://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
  65.       KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  66.       KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE

在jupyter实验室我无法进入kafka:9093, localhost:9092 by sparksession.readstream.format（“kafka”）.option（“kafka.bootstrap.servers”，”kafka:9093“”…，显示错误：

  1. 20/12/03 16:10:23 WARN NetworkClient: [Consumer clientId=consumer-spark-kafka-source-4b4422bc-2a6a-456a-b750-4fba8300328c-37706672-driver-0-1, groupId=spark-kafka-source-4b4422bc-2a6a-456a-b750-4fba8300328c-37706672-driver-0] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected

我还安装了telnet通过jupyterlab的bash检查连接，我可以telnet到一些服务器，比如 telnet www.cyberciti.biz 80 ，但无法连接到localhost:9092. 请帮帮我。

apache-kafkaapache-sparkdockerdocker-composejupyter

1条答案

按热度按时间
368yc8dk

368yc8dk1#

根据@onecricketeer的说法，我认为我之前的答案是一条红鲱鱼。
所以，我查了你的 Dockerfile 以及 docker-compose.yaml 而且，这对我很有效。

  1. ARG debian_buster_image_tag=8-jre-slim
  2. FROM openjdk:${debian_buster_image_tag}
  4. RUN apt-get update -y && \
  5.     apt-get install -y python3 python3-pip && \
  6.     rm -rf /var/lib/apt/lists/* && \
  7.     pip3 install pyspark==3.0.1 jupyterlab==2.1.5
  9. ENTRYPOINT [ "jupyter","notebook","--allow-root","--ip=0.0.0.0"]

笔记
远离的 ${shared_workspace} 因为这看起来 /opt/workspace 重新排序 apt-get install s
补充 ENTRYPOINT 跑 jupyter-notebook --allow-root -ip=0.0.0.0 这将公开上的笔记本服务 localhost:8888 但需要一个令牌。
调整了合成：

  1. services:
  2.   jupyterlab:
  3.     image: jupyterlab
  4.     build:
  5.       context: .
  6.       dockerfile: ./Dockerfile
  7.     container_name: jupyterlab
  8.     ports:
  9.       - 8888:8888
  10.     volumes:
  11.       - shared-workspace:/opt/workspace

然后：

  1. docker-compose build && docker-compose up

以及 docker-compose ps :

  1. Name                      Command               State                            Ports                          
  2. ------------------------------------------------------------------------------------------------------------------------
  3. 65130144_kafka_1       start-kafka.sh                   Up      0.0.0.0:9092->9092/tcp, 9093/tcp                        
  4. 65130144_zookeeper_1   /bin/sh -c /usr/sbin/sshd  ...   Up      2181/tcp, 22/tcp, 2888/tcp, 3888/tcp                    
  5. jupyterlab             jupyter notebook --allow-r ...   Up      0.0.0.0:8888->8888/tcp                                  
  6. spark-master           /bin/bash /master.sh             Up      6066/tcp, 0.0.0.0:7077->7077/tcp, 0.0.0.0:8080->8080/tcp
  7. spark-worker-1         /bin/bash /worker.sh             Up      0.0.0.0:8081->8081/tcp                                  
  8. spark-worker-2         /bin/bash /worker.sh             Up      0.0.0.0:8082->8081/tcp

你可能想要笔记本的代币，用吧 docker-compose logs jupyterlab :

  1. Attaching to jupyterlab
  2. jupyterlab        | [I 17:57:08.412 NotebookApp] Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret
  3. jupyterlab        | [I 17:57:08.659 NotebookApp] JupyterLab extension loaded from /usr/local/lib/python3.7/dist-packages/jupyterlab
  4. jupyterlab        | [I 17:57:08.659 NotebookApp] JupyterLab application directory is /usr/local/share/jupyter/lab
  5. jupyterlab        | [I 17:57:08.662 NotebookApp] Serving notebooks from local directory: /
  6. jupyterlab        | [I 17:57:08.662 NotebookApp] Jupyter Notebook 6.1.5 is running at:
  7. jupyterlab        | [I 17:57:08.662 NotebookApp] http://1b77e2b01d7b:8888/?token=e1796ee09331bd31459211ce35ad16fc9a7f4ee854870457
  8. jupyterlab        | [I 17:57:08.662 NotebookApp]  or http://127.0.0.1:8888/?token=e1796ee09331bd31459211ce35ad16fc9a7f4ee854870457
  9. jupyterlab        | [I 17:57:08.662 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
  10. jupyterlab        | [W 17:57:08.665 NotebookApp] No web browser found: could not locate runnable browser.
  11. jupyterlab        | [C 17:57:08.666 NotebookApp] 
  12. jupyterlab        |     
  13. jupyterlab        |     To access the notebook, open this file in a browser:
  14. jupyterlab        |         file:///root/.local/share/jupyter/runtime/nbserver-1-open.html
  15. jupyterlab        |     Or copy and paste one of these URLs:
  16. jupyterlab        |         http://1b77e2b01d7b:8888/?token=e1796ee09331bd31459211ce35ad16fc9a7f4ee854870457
  17. jupyterlab        |      or http://127.0.0.1:8888/?token=e1796ee09331bd31459211ce35ad16fc9a7f4ee854870457

使用第二个链接（您需要使用生成的任何令牌），创建一个笔记本并添加：

  1. from pyspark.sql import SparkSession;
  3. spark = SparkSession \
  4.     .builder \
  5.     .appName("test") \
  6.     .getOrCreate()
  8. spark.readStream.format("kafka").option("kafka.bootstrap.servers", "kafka:9093")

然后运行它：

  1. <pyspark.sql.streaming.DataStreamReader at 0x7fbf8defe208>

似乎是（！？）好 啊。

我怀疑 jupyter-lab 以及 jupyterlab 在配置中是相同的。
dockerfile不运行（否 CMD 或者 ENTRYPOINT )所以我也假设你在里面使用了一个shell。这很好只是不寻常。
有几件事 kafka 服务（注意，我只对Kafka略知一二）：
在 kafka 您必须将其引用为 localhost 你有 INSIDE://kafka:9093 但这有两个问题。在服务内部，它应该将自己引用为 localhost 在服务区内我怀疑它在港口 :8082 声明 expose: - "9093" 不是功能性的，而是记录片，除非你的图像也在无声地曝光，否则是不正确的 :9093 ; 这实际上没有暴露任何东西，因此可能是不正确的。

展开查看全部
赞(0）分享  回复(0）举报  2021-05-16
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com