Kafka debezium“快照-最终阶段”上出现空指针异常

6psbrbz9  于 2023-01-20  发布在  Apache
关注(0)|答案(1)|浏览(422)

我使用debezium连接到Oracle数据库版本19 c。遵循debezium examples repository并使用Docker-compose脚本。为了使其工作，我必须修改debezium-with-oracle-jdbc的Docker文件。您可以在this repository中找到我的更改以供复制。
针对该问题：
当连接到数据库时，我得到了下面的异常。最初的步骤看起来是成功的，但我假设当恢复模式时，连接器崩溃了。我在代码中找到了这个地方，但我不能判断这是数据库问题还是从debezium连接到Kafka的问题。崩溃行可以在这里找到。
我在这里读到，通常表模式应该在崩溃阶段打印出来。可能是数据库用户缺少权限？我找不到连接数据库的用户需要哪些权限。

connect_1    | 2021-11-23 15:08:44,773 INFO   Oracle|server1|snapshot  Metrics registered   [io.debezium.pipeline.ChangeEventSourceCoordinator]
connect_1    | 2021-11-23 15:08:44,773 INFO   Oracle|server1|snapshot  Context created   [io.debezium.pipeline.ChangeEventSourceCoordinator]
connect_1    | 2021-11-23 15:08:44,776 INFO   Oracle|server1|snapshot  No previous offset has been found.   [io.debezium.connector.oracle.OracleSnapshotChangeEventSource]
connect_1    | 2021-11-23 15:08:44,776 INFO   Oracle|server1|snapshot  According to the connector configuration only schema will be snapshot.   [io.debezium.connector.oracle.OracleSnapshotChangeEventSource]
connect_1    | 2021-11-23 15:08:44,777 INFO   Oracle|server1|snapshot  Snapshot step 1 - Preparing   [io.debezium.relational.RelationalSnapshotChangeEventSource]
connect_1    | 2021-11-23 15:08:44,777 INFO   Oracle|server1|snapshot  Snapshot step 2 - Determining captured tables   [io.debezium.relational.RelationalSnapshotChangeEventSource]
connect_1    | 2021-11-23 15:08:55,141 INFO   Oracle|server1|snapshot  Snapshot step 3 - Locking captured tables [<Disclosed, but found 323 tables in total>]   [io.debezium.relational.RelationalSnapshotChangeEventSource]
connect_1    | 2021-11-23 15:09:11,187 INFO   Oracle|server1|snapshot  Snapshot step 4 - Determining snapshot offset   [io.debezium.relational.RelationalSnapshotChangeEventSource]
connect_1    | 2021-11-23 15:09:12,023 INFO   Oracle|server1|snapshot  Snapshot step 5 - Reading structure of captured tables   [io.debezium.relational.RelationalSnapshotChangeEventSource]
connect_1    | 2021-11-23 15:09:43,928 INFO   ||  WorkerSourceTask{id=inventory-connector-0} flushing 0 outstanding messages for offset commit   [org.apache.kafka.connect.runtime.WorkerSourceTask]
connect_1    | 2021-11-23 15:09:53,390 INFO   Oracle|server1|snapshot  Snapshot step 6 - Persisting schema history   [io.debezium.relational.RelationalSnapshotChangeEventSource]
connect_1    | 2021-11-23 15:09:53,438 INFO   Oracle|server1|snapshot  Snapshot - Final stage   [io.debezium.pipeline.source.AbstractSnapshotChangeEventSource]
connect_1    | 2021-11-23 15:09:53,439 ERROR  Oracle|server1|snapshot  Producer failure   [io.debezium.pipeline.ErrorHandler]
connect_1    | io.debezium.DebeziumException: java.lang.NullPointerException
connect_1    |  at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:80)
connect_1    |  at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:118)
connect_1    |  at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
connect_1    |  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
connect_1    |  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
connect_1    |  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
connect_1    |  at java.base/java.lang.Thread.run(Thread.java:829)
connect_1    | Caused by: java.lang.NullPointerException
connect_1    |  at io.debezium.relational.RelationalSnapshotChangeEventSource.createSchemaChangeEventsForTables(RelationalSnapshotChangeEventSource.java:277)
connect_1    |  at io.debezium.relational.RelationalSnapshotChangeEventSource.doExecute(RelationalSnapshotChangeEventSource.java:124)
connect_1    |  at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:71)
connect_1    |  ... 6 more
connect_1    | 2021-11-23 15:09:53,445 INFO   Oracle|server1|snapshot  Connected metrics set to 'false'   [io.debezium.pipeline.metrics.StreamingChangeEventSourceMetrics]
connect_1    | 2021-11-23 15:09:53,909 INFO   ||  WorkerSourceTask{id=inventory-connector-0} flushing 0 outstanding messages for offset commit   [org.apache.kafka.connect.runtime.WorkerSourceTask]
connect_1    | 2021-11-23 15:09:53,910 ERROR  ||  WorkerSourceTask{id=inventory-connector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted   [org.apache.kafka.connect.runtime.WorkerTask]
connect_1    | org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
connect_1    |  at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42)
connect_1    |  at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:135)
connect_1    |  at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
connect_1    |  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
connect_1    |  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
connect_1    |  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
connect_1    |  at java.base/java.lang.Thread.run(Thread.java:829)
connect_1    | Caused by: io.debezium.DebeziumException: java.lang.NullPointerException
connect_1    |  at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:80)
connect_1    |  at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:118)
connect_1    |  ... 5 more
connect_1    | Caused by: java.lang.NullPointerException
connect_1    |  at io.debezium.relational.RelationalSnapshotChangeEventSource.createSchemaChangeEventsForTables(RelationalSnapshotChangeEventSource.java:277)
connect_1    |  at io.debezium.relational.RelationalSnapshotChangeEventSource.doExecute(RelationalSnapshotChangeEventSource.java:124)
connect_1    |  at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:71)
connect_1    |  ... 6 more
connect_1    | 2021-11-23 15:09:53,911 INFO   ||  Stopping down connector   [io.debezium.connector.common.BaseSourceTask]
apache-kafka

1条答案

按热度按时间
h5qlskok

h5qlskok1#

我也遇到了同样的问题，但我发现当传递属性“schema.history.internal.store.only.captured.tables.ddl”为false时，错误停止发生，但我需要将其设置为true才能工作：

[2023-01-18 14:49:03,029] INFO Snapshot step 1 - Preparing (io.debezium.relational.RelationalSnapshotChangeEventSource)
[2023-01-18 14:49:03,029] INFO Previous snapshot was cancelled before completion; a new snapshot will be taken. (io.debezium.relational.RelationalSnapshotChangeEventSource)
[2023-01-18 14:49:03,037] INFO Snapshot step 2 - Determining captured tables (io.debezium.relational.RelationalSnapshotChangeEventSource)
[2023-01-18 14:49:54,682] INFO Snapshot step 4 - Determining snapshot offset (io.debezium.relational.RelationalSnapshotChangeEventSource)
[2023-01-18 14:49:54,682] INFO Snapshot step 5 - Reading structure of captured tables (io.debezium.relational.RelationalSnapshotChangeEventSource)
[2023-01-18 14:49:54,682] INFO Only captured tables schema should be captured, capturing: [] (io.debezium.connector.oracle.OracleSnapshotChangeEventSource)
[2023-01-18 14:49:54,682] INFO Snapshot step 6 - Persisting schema history (io.debezium.relational.RelationalSnapshotChangeEventSource)
[2023-01-18 14:49:54,682] INFO Capturing structure of table ORCL.XXXX.XXXX (io.debezium.connector.oracle.OracleSnapshotChangeEventSource)
[2023-01-18 14:49:54,796] INFO Snapshot - Final stage (io.debezium.pipeline.source.AbstractSnapshotChangeEventSource)
[2023-01-18 14:49:54,796] WARN Snapshot was not completed successfully, it will be re-executed upon connector restart (io.debezium.pipeline.source.AbstractSnapshotChangeEventSource)
[2023-01-18 14:49:54,796] ERROR Producer failure (io.debezium.pipeline.ErrorHandler)
io.debezium.DebeziumException: java.lang.NullPointerException
    at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:85)
    at io.debezium.pipeline.ChangeEventSourceCoordinator.doSnapshot(ChangeEventSourceCoordinator.java:155)
    at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:137)
    at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:109)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.NullPointerException
    at io.debezium.connector.oracle.OracleSnapshotChangeEventSource.createSchemaChangeEventsForTables(OracleSnapshotChangeEventSource.java:215)
    at io.debezium.relational.RelationalSnapshotChangeEventSource.doExecute(RelationalSnapshotChangeEventSource.java:122)
    at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:76)
    ... 8 more
[2023-01-18 14:49:54,797] INFO Connected metrics set to 'false' (io.debezium.pipeline.ChangeEventSourceCoordinator)
[2023-01-18 14:49:55,130] INFO WorkerSourceTask{id=connect-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)
[2023-01-18 14:49:55,130] INFO WorkerSourceTask{id=connect-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)
[2023-01-18 14:49:55,130] ERROR WorkerSourceTask{id=connect-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
    at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:53)
    at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:116)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

    at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:85)
    at io.debezium.pipeline.ChangeEventSourceCoordinator.doSnapshot(ChangeEventSourceCoordinator.java:155)
    at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:137)
    at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:109)
    ... 5 more
Caused by: java.lang.NullPointerException
    at io.debezium.connector.oracle.OracleSnapshotChangeEventSource.createSchemaChangeEventsForTables(OracleSnapshotChangeEventSource.java:215)
    at io.debezium.relational.RelationalSnapshotChangeEventSource.doExecute(RelationalSnapshotChangeEventSource.java:122)
    at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:76)
    ... 8 more
[2023-01-18 14:49:55,131] INFO Stopping down connector (io.debezium.connector.common.BaseSourceTask)
[2023-01-18 14:49:55,246] INFO [Producer clientId=connect-schemahistory] Closing the Kafka producer with timeoutMillis = 30000 ms. (org.apache.kafka.clients.producer.KafkaProducer)
[2023-01-18 14:49:55,246] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics)
[2023-01-18 14:49:55,246] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics)
[2023-01-18 14:49:55,246] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics)
[2023-01-18 14:49:55,247] INFO App info kafka.producer for connect-schemahistory unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2023-01-18 14:49:55,247] INFO [Producer clientId=connector-producer-connect-0] Closing the Kafka producer with timeoutMillis = 30000 ms. (org.apache.kafka.clients.producer.KafkaProducer)

“类”：“io.debezium.连接器.oracle. oracle连接器”，“类型”：“来源”、“版本”：“2.0.1.最终版”

"connector.class": "io.debezium.connector.oracle.OracleConnector",
            "database.connection.adapter": "logminer",
            "log.mining.strategy": "online_catalog",
            "snapshot.mode": "initial",
            "tasks.max" : "1",
            "database.hostname" : "hostname",
            "database.port": "1521",
            "database.user" : "user",
            "database.password" : "pass",
            "database.dbname" : "DB",
            "database.server.name": "DB",
            "topic.prefix" : "PREFIX",
            "schema.history.internal.kafka.bootstrap.servers" : "kafka:9092",
            "schema.history.internal.kafka.topic": "PREFIX-INTERNAL-SCHEMA-CHANGES",
            "table.include.list": "TABLE",
            "snapshot.include.collection.list": "TABLE",
            "schema.history.internal.store.only.captured.tables.ddl": true,
            "database.history.kafka.bootstrap.servers" :  "kafka:9092",
            "snapshot.locking.mode": "shared",
赞(0）分享  回复(0）举报  2023-01-20
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com