2020-11-02 13:52:25.892 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'refreshScope': registering with JMX server as MBean [org.springframework.cloud.context.scope.refresh:name=refreshScope,type=RefreshScope]
2020-11-02 13:52:25.907 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'configurationPropertiesRebinder': registering with JMX server as MBean [org.springframework.cloud.context.properties:name=configurationPropertiesRebinder,context=ee86bcb,type=ConfigurationPropertiesRebinder]
2020-11-02 13:52:25.937 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## start the canal client adapters.
2020-11-02 13:52:25.942 [main] INFO c.a.otter.canal.client.adapter.support.ExtensionLoader - extension classpath dir: /usr/local/adapter/plugin
2020-11-02 13:52:25.962 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: logger succeed
2020-11-02 13:52:25.965 [main] INFO c.a.o.canal.client.adapter.es.config.ESSyncConfigLoader - ## Start loading es mapping config ...
2020-11-02 13:52:26.140 [main] INFO c.a.o.canal.client.adapter.es.config.ESSyncConfigLoader - ## ES mapping config loaded
2020-11-02 13:52:26.706 [main] INFO org.elasticsearch.plugins.PluginsService - no modules loaded
2020-11-02 13:52:26.708 [main] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin]
2020-11-02 13:52:26.708 [main] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.join.ParentJoinPlugin]
2020-11-02 13:52:26.708 [main] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.percolator.PercolatorPlugin]
2020-11-02 13:52:26.708 [main] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.script.mustache.MustachePlugin]
2020-11-02 13:52:26.708 [main] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2020-11-02 13:52:28.341 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: es succeed
2020-11-02 13:52:28.351 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Start adapter for canal instance: example succeed
2020-11-02 13:52:28.351 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## the canal client adapters are running now ......
2020-11-02 13:52:28.354 [Thread-4] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Start to connect destination: example <=============
2020-11-02 13:52:28.363 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8081"]
2020-11-02 13:52:28.364 [main] INFO org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read
2020-11-02 13:52:28.385 [main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8081 (http) with context path ''
2020-11-02 13:52:28.388 [main] INFO c.a.otter.canal.adapter.launcher.CanalAdapterApplication - Started CanalAdapterApplication in 7.508 seconds (JVM running for 8.207)
2020-11-02 13:58:52.149 [Thread-4] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Disconnect destination: example <=============
2020-11-02 13:58:53.149 [Thread-4] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Start to connect destination: example <=============
2020-11-02 14:01:00.404 [Thread-4] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - process error!
com.alibaba.otter.canal.protocol.exception.CanalClientException: java.net.ConnectException: Connection timed out
at com.alibaba.otter.canal.client.impl.SimpleCanalConnector.doConnect(SimpleCanalConnector.java:198)
at com.alibaba.otter.canal.client.impl.SimpleCanalConnector.connect(SimpleCanalConnector.java:115)
at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterWorker.process(CanalAdapterWorker.java:88)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection timed out
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:454)
at sun.nio.ch.Net.connect(Net.java:446)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
at com.alibaba.otter.canal.client.impl.SimpleCanalConnector.doConnect(SimpleCanalConnector.java:150)
... 3 common frames omitted
2020-11-02 14:01:00.405 [Thread-4] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Disconnect destination: example <=============
2020-11-02 14:01:01.405 [Thread-4] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Start to connect destination: example <=============
9条答案
按热度按时间pbpqsu0x1#
阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴
rkue9o1l2#
adapter-1.1.4
application.yml
server:
port: 8081
spring:
jackson:
date-format: yyyy-MM-dd HH:mm:ss
time-zone: GMT+8
default-property-inclusion: non_null
canal.conf:
mode: tcp # kafka rocketMQ
canalServerHost: IP地址:11111
batchSize: 500
syncBatchSize: 1000
retries: 0
timeout:
accessKey:
secretKey:
srcDataSources:
defaultDS:
url: jdbc:mysql://IP地址:3306/test?useUnicode=true
username: root
password: biosan#17
canalAdapters:
groups:
outerAdapters:
hosts: IP地址:9300
properties:
cluster.name: my-application
——————
test.yml
dataSourceKey: defaultDS
#outerAdapterKey: exampleKey
destination: example
groupId: g1
esMapping:
_index: test
_type: _doc
_id: _id
upsert: true
sql: "select a.id as _id,a.name,a.address from test a"
commitBatch: 3000
hsgswve43#
检查一下你的kafka集群是否正常工作
xnifntxz4#
手动同步可以 ，自动不行啊 ； 单机部署 elasticsearch 6.5.4 mysql5.7 jdk 1.8_171
2020-11-02 16:19:26.092 [http-nio-8081-exec-1] DEBUG c.a.otter.canal.client.adapter.es.service.ESEtlService - etl sql : select a.id as _id,a.name,a.address from test a
2020-11-02 16:19:26.103 [http-nio-8081-exec-1] INFO c.a.otter.canal.client.adapter.es.service.ESEtlService - 数据全量导入完成, 一共导入 22 条数据, 耗时: 11
2020-11-02 16:19:45.168 [http-nio-8081-exec-3] INFO c.a.otter.canal.client.adapter.es.service.ESEtlService - start etl to import data to index: test
2020-11-02 16:19:45.168 [http-nio-8081-exec-3] DEBUG c.a.otter.canal.client.adapter.es.service.ESEtlService - etl sql : select a.id as _id,a.name,a.address from test a
2020-11-02 16:19:45.286 [http-nio-8081-exec-3] INFO c.a.otter.canal.client.adapter.es.service.ESEtlService - 数据全量导入完成, 一共导入 24 条数据, 耗时: 118
bq3bfh9z5#
检查一下你的kafka能够正常工作
我是mysql 同步到es 要 kafka 做什么
ercv8c1e6#
烦死啦 。。。 大哥们 怎么回事啊 这问题折腾我好久了
yyhrrdl87#
检查一下你的kafka能够正常工作
我是mysql 同步到es 要 kafka 做什么
emmmm，没注意你用的tcp模式，不是用的kafka模式，你可以试一下kafka模式
zpf6vheq8#
检查一下你的kafka能够正常工作
我是mysql 同步到es 要 kafka 做什么
emmmm，没注意你用的tcp模式，不是用的kafka模式，你可以试一下kafka模式
amework.cloud.context.scope.refresh:name=refreshScope,type=RefreshScope]
2020-11-03 14:53:31.464 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'configurationPropertiesRebinder': registering with JMX server as MBean [org.springframework.cloud.context.properties:name=configurationPropertiesRebinder,context=2631f68c,type=ConfigurationPropertiesRebinder]
2020-11-03 14:53:31.480 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## start the canal client adapters.
2020-11-03 14:53:31.486 [main] INFO c.a.otter.canal.client.adapter.support.ExtensionLoader - extension classpath dir: /usr/local/adapter/plugin
2020-11-03 14:53:31.680 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: logger succeed
2020-11-03 14:53:31.682 [main] INFO c.a.o.canal.client.adapter.es.config.ESSyncConfigLoader - ## Start loading es mapping config ...
2020-11-03 14:53:31.840 [main] INFO c.a.o.canal.client.adapter.es.config.ESSyncConfigLoader - ## ES mapping config loaded
2020-11-03 14:53:32.950 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: es succeed
2020-11-03 14:53:33.011 [main] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## something goes wrong when starting up the canal client adapters:
java.lang.NullPointerException: null
at java.util.Hashtable.put(Hashtable.java:460)
at com.alibaba.otter.canal.client.kafka.KafkaCanalConnector.(KafkaCanalConnector.java:52)
at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterKafkaWorker.(CanalAdapterKafkaWorker.java:38)
at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterLoader.init(CanalAdapterLoader.java:109)
at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterService.init(CanalAdapterService.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:365)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:308)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:135)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:422)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1694)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:579)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:501)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$1(AbstractBeanFactory.java:353)
at org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:390)
vnzz0bqm9#
检查一下你的kafka能够正常工作
我是mysql 同步到es 要 kafka 做什么
emmmm，没注意你用的tcp模式，不是用的kafka模式，你可以试一下kafka模式
amework.cloud.context.scope.refresh:name=refreshScope,type=RefreshScope]
2020-11-03 14:53:31.464 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'configurationPropertiesRebinder': registering with JMX server as MBean [org.springframework.cloud.context.properties:name=configurationPropertiesRebinder,context=2631f68c,type=ConfigurationPropertiesRebinder]
2020-11-03 14:53:31.480 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## start the canal client adapters.
2020-11-03 14:53:31.486 [main] INFO c.a.otter.canal.client.adapter.support.ExtensionLoader - extension classpath dir: /usr/local/adapter/plugin
2020-11-03 14:53:31.680 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: logger succeed
2020-11-03 14:53:31.682 [main] INFO c.a.o.canal.client.adapter.es.config.ESSyncConfigLoader - ## Start loading es mapping config ...
2020-11-03 14:53:31.840 [main] INFO c.a.o.canal.client.adapter.es.config.ESSyncConfigLoader - ## ES mapping config loaded
2020-11-03 14:53:32.950 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: es succeed
2020-11-03 14:53:33.011 [main] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## something goes wrong when starting up the canal client adapters:
java.lang.NullPointerException: null
at java.util.Hashtable.put(Hashtable.java:460)
at com.alibaba.otter.canal.client.kafka.KafkaCanalConnector.(KafkaCanalConnector.java:52)
at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterKafkaWorker.(CanalAdapterKafkaWorker.java:38)
at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterLoader.init(CanalAdapterLoader.java:109)
at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterService.init(CanalAdapterService.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:365)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:308)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:135)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:422)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1694)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:579)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:501)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$1(AbstractBeanFactory.java:353)
at org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:390)
你的配置文件长什么样。等回复效率太低了，建议你下载一份源码自己调试一下