kafka连接:启动s3接收器连接器时出现无法识别的错误

fslejnso  于 2021-06-04  发布在  Kafka
关注(0)|答案(1)|浏览(591)

我正在尝试为具有3个节点的kafka connect集群设置我的第3个worker。worker在第3个节点上正常运行,我可以进行rest调用来获取现有的连接器(目前我有2个,每个节点上一个)。但是,当我尝试使用以下命令进行post调用以创建第三个连接器时:

curl -X POST -H "Content-Type: application/json" --data @test-s3-sink-config.json http://<my-host>:<my-port>/connectors

我得到这个timeoutexception响应:

{"error_code":500,"message":"IO Error trying to forward REST request: java.net.SocketTimeoutException: Connect Timeout"}

当我查看worker堆栈跟踪时,它会说:

[2020-10-20 18:27:04,062] INFO AbstractConfig values:
 (org.apache.kafka.common.config.AbstractConfig:354)
[2020-10-20 18:27:19,081] ERROR IO error forwarding REST request:  (org.apache.kafka.connect.runtime.rest.RestClient:143)
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: Connect Timeout
        at org.eclipse.jetty.client.util.FutureResponseListener.getResult(FutureResponseListener.java:118)
        at org.eclipse.jetty.client.util.FutureResponseListener.get(FutureResponseListener.java:101)
        at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:711)
        at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:125)
        at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:65)
        at org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.completeOrForwardRequest(ConnectorsResource.java:369)
        at org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.createConnector(ConnectorsResource.java:164)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.jav
a:52)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherPr
ovider.java:176)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
        at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:500)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Connect Timeout
        at org.eclipse.jetty.io.ManagedSelector$Connect.run(ManagedSelector.java:812)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more

跟踪的第一个日志是困扰我的,因为我没有看到任何关于我做错了什么的相关信息,第二个日志只是timeoutexception。我到处寻找有类似问题的人,或者了解“abstractconfig”类,但找不到任何有用的东西,这是kafka的abstractconfig类(我使用的是kafka版本2.0.0)。
最后,这里是我使用的配置文件:

{"name":"s3-connector-orderbooks",
"config":{
"connector.class":"io.confluent.connect.s3.S3SinkConnector",
"file":"snapshots-test",
"format.class":"io.confluent.connect.s3.format.json.JsonFormat",
"flush.size":"1000000",
"tasks.max":"1",
"topics":"binance-full-snaps-test",
"timezone":"UTC",
"storage.class":"io.confluent.connect.s3.storage.S3Storage",
"rotate.schedule.interval.ms":"3600000",
"s3.bucket.name":"pfc-data",
"timestamp.extractor":"Record",
"partitioner.class":"io.confluent.connect.storage.partitioner.HourlyPartitioner",
"locale":"en-US",
"s3.compression.type":"gzip"
}
}

如果你觉得有任何额外的信息,我应该包括随时要求它,我是相当新的堆栈溢出。
我很想知道是否有人遇到过这样的事情,或者是否有人知道是什么导致了这个问题。谢谢!

1yjd4xko

1yjd4xko1#

在kafka连接集群中,负责rest请求服务的节点是leader节点。因此,需要检查leader节点是否可以到达集群中所有可用的worker节点。你可以检查一下你的房间 rest.advertised.host.name 在你的 distributed.properties 以确保节点的播发主机名在连接群集中是否可访问。罗宾莫法特在他的博客里有一篇关于这个主题的文章写得很好。请阅读它以获得一个很好的洞察力。

相关问题