我正在尝试将zookeeper和kafka设置为共享命名空间中独立的kubernetes部署/pods。我已经通过kubeadm在我的ubuntu沙盒上引导了本地的k8s1.8和calico。。。
对于Zookeeper,我用的是图片zookeeper:3.4 from 我和hub.docker.com创建了一个kubernetes部署和服务,在这里我公开了端口:2181 2888 3888。服务名是zookeeper,我假设我应该能够通过名称空间中的pod中的这个主机名来使用它。
对于Kafka1.0,我创建了自己的容器映像,可以用环境变量控制它。。。我正在设置zookeeper.connect到zookeeper:2181. 我假设kubernetes dns会解决这个问题并打开到服务的连接。
不幸的是我得到:
[2018-01-03 15:48:26,292] INFO Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient)
[2018-01-03 15:48:32,293] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2018-01-03 15:48:46,286] INFO Opening socket connection to server zookeeper.sandbox.svc.cluster.local/10.107.41.148:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2018-01-03 15:48:46,299] INFO Socket connection established to zookeeper.sandbox.svc.cluster.local/10.107.41.148:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2018-01-03 15:48:46,319] INFO Session establishment complete on server zookeeper.sandbox.svc.cluster.local/10.107.41.148:2181, sessionid = 0x10000603c560001, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2018-01-03 15:48:46,331] INFO Session: 0x10000603c560001 closed (org.apache.zookeeper.ZooKeeper)
[2018-01-03 15:48:46,333] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server 'zookeeper:2181' with timeout of 6000 ms
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1233)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:157)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:131)
at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:115)
at kafka.utils.ZkUtils$.withMetrics(ZkUtils.scala:92)
at kafka.server.KafkaServer.initZk(KafkaServer.scala:346)
at kafka.server.KafkaServer.startup(KafkaServer.scala:194)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
at kafka.Kafka$.main(Kafka.scala:92)
at kafka.Kafka.main(Kafka.scala)
所以我假设我的集群中有一个通用的网络问题,然后我注意到一些更让我困惑的事情。。。如果我将zookeeper.connect设置为10.107.41.148:2181(zookeeper服务的当前地址),则连接正常(至少从kafka到zookeeper)。
[2018-01-03 15:51:31,092] INFO Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient)
[2018-01-03 15:51:31,094] INFO Opening socket connection to server 10.107.41.148/10.107.41.148:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2018-01-03 15:51:31,105] INFO Socket connection established to 10.107.41.148/10.107.41.148:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2018-01-03 15:51:31,134] INFO Session establishment complete on server 10.107.41.148/10.107.41.148:2181, sessionid = 0x10000603c560005, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
通过这个设置,我可以使用kubernetes集群主机上的zookeeper服务来执行例如“bin/kafka-topics.sh--list--zookeeper 10.107.41.148:2181”。产生一个信息是行不通的。。。我假设一旦网络正常工作,我需要添加Kafka广告地址。。。
kafka-console-producer.sh --broker-list 10.100.117.196:9092 --topic test1
>test-msg1
>[2018-01-03 17:05:35,689] WARN [Producer clientId=console-producer] Connection to node 0 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
任何提示我的kubernetes网络设置有什么问题,或者至少从哪里开始故障排除?
谢谢并致以最诚挚的问候,帕维尔
1条答案
按热度按时间new9mtju1#
如果使用statefulset,则需要首先部署服务。
这是我的服务。
这里是configmap(稍后使用):
这是我的状态集:
在您部署了一个正常工作的zookeeper配置并且他们选择了一个主机之后,您可以继续kafka部署。
一旦zookeeper部署完毕,您的kafka配置必须通过服务引用zookeeper StatefolSet。在Kafka中,必须定义/覆盖此属性:
从一个豆荚里你应该
ping zookeeper-0.zookeeper
成功。希望这有帮助。