mariadb Zendesk Maxwell崩溃/下降

vddsk6oq  于 2022-11-08  发布在  其他
关注(0)|答案(1)|浏览(158)

在我们的产品中，我们在docker上运行了maxwell，它被配置为监听MariadDb表上的更改，并将它们推送到Kafka主题。我们的MariadDb有大约10-12个表，我们将maxwell配置为只监听4个表，所有这4个表都被大量使用（写入数量太高）。
最近我们看到我们的Maxwell在下降，下面是我们在Maxwell下降时发现的几个错误日志

  1. Oct 28, 2022 @ 06:10:40.037 06:10:40,037 ERROR MysqlParserListener - (parse SET STATEMENT max_statement_time = 60 FOR flush table)
  3. Oct 28, 2022 @ 06:10:40.040 06:10:40,040 ERROR SchemaChange - Error parsing SQL: 'SET STATEMENT max_statement_time=60 FOR flush table'
  5. Oct 28, 2022 @ 06:10:40.047 06:10:40,045 ERROR AbstractSchemaStore - Error on bin log position Position[BinlogPosition[mysql-bin-changelog.049610:3209], lastHeartbeat=1666898518069]

由于最后一个错误日志显示错误在bin日志位置Position[BinlogPosition[mysql-bin-changelog.049610：3209]，因此为了验证名称为mysql-bin-changelog.049610:3209的bin日志文件是否存在，我们运行SHOW binary logs;，并在上述命令输出的列表中找到该文件
请帮助我了解这里出了什么问题，并帮助解决问题

环境详细信息

  1. OS：Centos
    1.平台- AWS示例（T4 g中型）
  2. CPU - 2个CPU
    1.内存- 4GB

Maxwell版本- v1.37.7
注意：我们不处理DDL，日志保留期为3天

详细的日志记录时，麦克斯韦是去了，对不起张贴巨大的日志：）

  1. Oct 28, 2022 @ 06:10:39.084 06:10:39,082 INFO  ProducerConfig - ProducerConfig values: 
  2. Oct 28, 2022 @ 06:10:39.146 06:10:39,146 INFO  AppInfoParser - Kafka commitId : aaa7af6d4a11b29d
  4. Oct 28, 2022 @ 06:10:39.146 06:10:39,146 INFO  AppInfoParser - Kafka version : 1.0.0
  6. Oct 28, 2022 @ 06:10:39.188 06:10:39,187 INFO  Maxwell - Maxwell v1.37.7 is booting (MaxwellKafkaProducer), starting at Position[BinlogPosition[mysql-bin-changelog.049610:3136], lastHeartbeat=1666898518069]
  8. Oct 28, 2022 @ 06:10:39.377 06:10:39,377 INFO  MysqlSavedSchema - Restoring schema id 22 (last modified at Position[BinlogPosition[mysql-bin-changelog.046940:91534], lastHeartbeat=1666103424529])
  10. Oct 28, 2022 @ 06:10:39.502 06:10:39,502 INFO  MysqlSavedSchema - Restoring schema id 1 (last modified at Position[BinlogPosition[mysql-bin-changelog.040524:418929], lastHeartbeat=0])
  12. Oct 28, 2022 @ 06:10:39.555 06:10:39,555 INFO  MysqlSavedSchema - beginning to play deltas...
  14. Oct 28, 2022 @ 06:10:39.576 06:10:39,576 INFO  MysqlSavedSchema - played 21 deltas in 15ms
  16. Oct 28, 2022 @ 06:10:39.601 06:10:39,601 INFO  BinlogConnectorReplicator - Setting initial binlog pos to: mysql-bin-changelog.049610:3136
  18. Oct 28, 2022 @ 06:10:39.605 06:10:39,605 INFO  MaxwellHTTPServer - Maxwell http server starting
  20. Oct 28, 2022 @ 06:10:39.607 06:10:39,607 INFO  MaxwellHTTPServer - Maxwell http server started on port 8080
  22. Oct 28, 2022 @ 06:10:39.649 06:10:39,649 INFO  BinlogConnectorReplicator - Binlog connected.
  24. Oct 28, 2022 @ 06:10:39.649 06:10:39,646 INFO  BinaryLogClient - Connected to mariadb-nlp.app.engati.local:3306 at mysql-bin-changelog.049610/3136 (sid:6379, cid:1560183)
  26. Oct 28, 2022 @ 06:10:39.669 06:10:39,669 INFO  log - Logging initialized @2288ms to org.eclipse.jetty.util.log.Slf4jLog
  28. Oct 28, 2022 @ 06:10:39.902 06:10:39,902 INFO  Server - jetty-9.4.41.v20210516; built: 2021-05-16T23:56:28.993Z; git: 98607f93c7833e7dc59489b13f3cb0a114fb9f4c; jvm 11.0.15+10
  30. Oct 28, 2022 @ 06:10:40.037 06:10:40,037 ERROR MysqlParserListener - (parse SET STATEMENT max_statement_time = 60 FOR flush table)
  32. Oct 28, 2022 @ 06:10:40.040 06:10:40,040 ERROR SchemaChange - Error parsing SQL: 'SET STATEMENT max_statement_time=60 FOR flush table'
  34. Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.Maxwell.main(Maxwell.java:336)
  36. Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.Maxwell.start(Maxwell.java:227)
  38. Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.Maxwell.startInner(Maxwell.java:301)
  40. Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.util.RunLoopProcess.runLoop(RunLoopProcess.java:34)
  42. Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.work(BinlogConnectorReplicator.java:228)
  44. Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.getRow(BinlogConnectorReplicator.java:723)
  46. Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.processQueryEvent(BinlogConnectorReplicator.java:400)
  48. Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.processQueryEvent(BinlogConnectorReplicator.java:378)
  50. Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.schema.MysqlSchemaStore.processSQL(MysqlSchemaStore.java:102)
  52. Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.schema.AbstractSchemaStore.resolveSQL(AbstractSchemaStore.java:49)
  54. Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:108)
  56. Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.schema.ddl.SchemaChange.parseSQL(SchemaChange.java:94)
  58. Oct 28, 2022 @ 06:10:40.047     at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
  60. Oct 28, 2022 @ 06:10:40.047     at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:17)
  62. Oct 28, 2022 @ 06:10:40.047     at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:93)
  64. Oct 28, 2022 @ 06:10:40.047 com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: SET
  66. Oct 28, 2022 @ 06:10:40.047 06:10:40,045 ERROR AbstractSchemaStore - Error on bin log position Position[BinlogPosition[mysql-bin-changelog.049610:3209], lastHeartbeat=1666898518069]
  68. Oct 28, 2022 @ 06:10:40.047 06:10:40,044 INFO  ContextHandler - Started o.e.j.s.ServletContextHandler@5628733f{/,null,AVAILABLE}
  70. Oct 28, 2022 @ 06:10:40.075 06:10:40,074 INFO  Server - Started @2700ms
  72. Oct 28, 2022 @ 06:10:40.075 06:10:40,074 INFO  AbstractConnector - Started ServerConnector@7152d3c1{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
  74. Oct 28, 2022 @ 06:10:40.129 06:10:40,129 INFO  BinlogConnectorReplicator - Binlog disconnected.
  76. Oct 28, 2022 @ 06:10:40.149 06:10:40,149 INFO  TaskManager - Stopping 5 tasks
  78. Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.schema.ddl.SchemaChange.parseSQL(SchemaChange.java:94) ~[maxwell-1.37.7.jar:1.37.7]
  80. Oct 28, 2022 @ 06:10:40.187     at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.8-1.jar:4.8-1]
  82. Oct 28, 2022 @ 06:10:40.187     at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:17) ~[antlr4-runtime-4.8-1.jar:4.8-1]
  84. Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:93) ~[maxwell-1.37.7.jar:1.37.7]
  86. Oct 28, 2022 @ 06:10:40.187 com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: SET
  88. Oct 28, 2022 @ 06:10:40.187 06:10:40,152 ERROR TaskManager - cause: 
  90. Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.Maxwell.main(Maxwell.java:336) ~[maxwell-1.37.7.jar:1.37.7]
  92. Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.Maxwell.start(Maxwell.java:227) ~[maxwell-1.37.7.jar:1.37.7]
  94. Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.Maxwell.startInner(Maxwell.java:301) ~[maxwell-1.37.7.jar:1.37.7]
  96. Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.util.RunLoopProcess.runLoop(RunLoopProcess.java:34) ~[maxwell-1.37.7.jar:1.37.7]
  98. Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.work(BinlogConnectorReplicator.java:228) ~[maxwell-1.37.7.jar:1.37.7]
  100. Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.getRow(BinlogConnectorReplicator.java:723) ~[maxwell-1.37.7.jar:1.37.7]
  102. Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.processQueryEvent(BinlogConnectorReplicator.java:400) ~[maxwell-1.37.7.jar:1.37.7]
  104. Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.processQueryEvent(BinlogConnectorReplicator.java:378) ~[maxwell-1.37.7.jar:1.37.7]
  106. Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.schema.MysqlSchemaStore.processSQL(MysqlSchemaStore.java:102) ~[maxwell-1.37.7.jar:1.37.7]
  108. Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.schema.AbstractSchemaStore.resolveSQL(AbstractSchemaStore.java:49) ~[maxwell-1.37.7.jar:1.37.7]
  110. Oct 28, 2022 @ 06:10:40.187     at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:108) ~[maxwell-1.37.7.jar:1.37.7]
  112. Oct 28, 2022 @ 06:10:40.189 06:10:40,189 INFO  TaskManager - Stopping: com.zendesk.maxwell.schema.PositionStoreThread@51b3ee1b
  114. Oct 28, 2022 @ 06:10:40.191 06:10:40,189 INFO  TaskManager - Stopping: com.zendesk.maxwell.producer.MaxwellKafkaProducerWorker@5ec6ffb6
  116. Oct 28, 2022 @ 06:10:40.192 06:10:40,192 INFO  KafkaProducer - [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
  118. Oct 28, 2022 @ 06:10:40.201 06:10:40,201 INFO  TaskManager - Stopping: com.zendesk.maxwell.monitoring.MaxwellHTTPServerWorker@1490968e
  120. Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.Maxwell.main(Maxwell.java:336)
  122. Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.Maxwell.start(Maxwell.java:227)
  124. Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.Maxwell.startInner(Maxwell.java:301)
  126. Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.util.RunLoopProcess.runLoop(RunLoopProcess.java:34)
  128. Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.work(BinlogConnectorReplicator.java:228)
  130. Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.getRow(BinlogConnectorReplicator.java:723)
  132. Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.processQueryEvent(BinlogConnectorReplicator.java:400)
  134. Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.replication.BinlogConnectorReplicator.processQueryEvent(BinlogConnectorReplicator.java:378)
  136. Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.schema.MysqlSchemaStore.processSQL(MysqlSchemaStore.java:102)
  138. Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.schema.AbstractSchemaStore.resolveSQL(AbstractSchemaStore.java:49)
  140. Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:108)
  142. Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.schema.ddl.SchemaChange.parseSQL(SchemaChange.java:94)
  144. Oct 28, 2022 @ 06:10:40.201     at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
  146. Oct 28, 2022 @ 06:10:40.201     at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:17)
  148. Oct 28, 2022 @ 06:10:40.201     at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:93)
  150. Oct 28, 2022 @ 06:10:40.201 com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: SET
  152. Oct 28, 2022 @ 06:10:40.201 06:10:40,199 INFO  TaskManager - Stopping: com.zendesk.maxwell.replication.BinlogConnectorReplicator@5a1d68e0
  154. Oct 28, 2022 @ 06:10:40.201 06:10:40,198 INFO  TaskManager - Stopping: com.zendesk.maxwell.bootstrap.BootstrapController@3857c48d
  156. Oct 28, 2022 @ 06:10:40.220 06:10:40,219 INFO  AbstractConnector - Stopped ServerConnector@7152d3c1{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
  158. Oct 28, 2022 @ 06:10:40.223 06:10:40,223 INFO  ContextHandler - Stopped o.e.j.s.ServletContextHandler@5628733f{/,null,STOPPED}
  160. ct 28, 2022 @ 06:10:41.637  06:10:41,636 INFO  TaskManager - Stopped all tasks
mariadb

1条答案

按热度按时间
cu6pst1q

cu6pst1q1#

Java回溯显示这是Maxwell中的错误。
问题是，v1.37.7中的Maxwell没有根据SQL黑名单（如FLUSH）解析SET STATEMENT ... FOR，而这意味着它不采取任何行动。
由于上游的响应速度非常快，这一问题在v1.39.2中得到了修复。

赞(0）分享  回复(0）举报  2022-11-08
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com