我正在尝试为具有3个节点的kafka connect集群设置我的第3个worker。worker在第3个节点上正常运行,我可以进行rest调用来获取现有的连接器(目前我有2个,每个节点上一个)。但是,当我尝试使用以下命令进行post调用以创建第三个连接器时:
curl -X POST -H "Content-Type: application/json" --data @test-s3-sink-config.json http://<my-host>:<my-port>/connectors
我得到这个timeoutexception响应:
{"error_code":500,"message":"IO Error trying to forward REST request: java.net.SocketTimeoutException: Connect Timeout"}
当我查看worker堆栈跟踪时,它会说:
[2020-10-20 18:27:04,062] INFO AbstractConfig values:
(org.apache.kafka.common.config.AbstractConfig:354)
[2020-10-20 18:27:19,081] ERROR IO error forwarding REST request: (org.apache.kafka.connect.runtime.rest.RestClient:143)
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: Connect Timeout
at org.eclipse.jetty.client.util.FutureResponseListener.getResult(FutureResponseListener.java:118)
at org.eclipse.jetty.client.util.FutureResponseListener.get(FutureResponseListener.java:101)
at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:711)
at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:125)
at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:65)
at org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.completeOrForwardRequest(ConnectorsResource.java:369)
at org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.createConnector(ConnectorsResource.java:164)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.jav
a:52)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherPr
ovider.java:176)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Connect Timeout
at org.eclipse.jetty.io.ManagedSelector$Connect.run(ManagedSelector.java:812)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
跟踪的第一个日志是困扰我的,因为我没有看到任何关于我做错了什么的相关信息,第二个日志只是timeoutexception。我到处寻找有类似问题的人,或者了解“abstractconfig”类,但找不到任何有用的东西,这是kafka的abstractconfig类(我使用的是kafka版本2.0.0)。
最后,这里是我使用的配置文件:
{"name":"s3-connector-orderbooks",
"config":{
"connector.class":"io.confluent.connect.s3.S3SinkConnector",
"file":"snapshots-test",
"format.class":"io.confluent.connect.s3.format.json.JsonFormat",
"flush.size":"1000000",
"tasks.max":"1",
"topics":"binance-full-snaps-test",
"timezone":"UTC",
"storage.class":"io.confluent.connect.s3.storage.S3Storage",
"rotate.schedule.interval.ms":"3600000",
"s3.bucket.name":"pfc-data",
"timestamp.extractor":"Record",
"partitioner.class":"io.confluent.connect.storage.partitioner.HourlyPartitioner",
"locale":"en-US",
"s3.compression.type":"gzip"
}
}
如果你觉得有任何额外的信息,我应该包括随时要求它,我是相当新的堆栈溢出。
我很想知道是否有人遇到过这样的事情,或者是否有人知道是什么导致了这个问题。谢谢!
1条答案
按热度按时间1yjd4xko1#
在kafka连接集群中,负责rest请求服务的节点是leader节点。因此,需要检查leader节点是否可以到达集群中所有可用的worker节点。你可以检查一下你的房间
rest.advertised.host.name
在你的distributed.properties
以确保节点的播发主机名在连接群集中是否可访问。罗宾莫法特在他的博客里有一篇关于这个主题的文章写得很好。请阅读它以获得一个很好的洞察力。