并发测试情况下,先插入->后修改id->最后删除数据 偶现数据未删除成功问题,但是load delete日志是存在的,也没有报如何异常?
load insert 日志:
row_load-2019-04-24-0.log-- PairId: 44 , TableId: 13 , EventType : I , Time : 1556042711000
row_load-2019-04-24-0.log-- Consistency : , Mode :
row_load-2019-04-24-0.log------------------
row_load-2019-04-24-0.log----Pks
row_load-2019-04-24-0.log- EventColumn[index=0,columnType=-5,columnName=id,columnValue=390442,isNull=false,isKey=true,isUpdate=true,isReverse=false]
row_load-2019-04-24-0.log----oldPks
row_load-2019-04-24-0.log-
row_load-2019-04-24-0.log----Columns
row_load-2019-04-24-0.log: EventColumn[index=1,columnType=-5,columnName=traceId,columnValue=1997861556042711096,isNull=false,isKey=false,isUpdate=true,isReverse=false]
row_load-2019-04-24-0.log- EventColumn[index=2,columnType=12,columnName=status,columnValue=test,isNull=false,isKey=false,isUpdate=true,isReverse=false]
row_load-2019-04-24-0.log- EventColumn[index=3,columnType=12,columnName=userId,columnValue=dxy_test,isNull=false,isKey=false,isUpdate=true,isReverse=false]
row_load-2019-04-24-0.log- EventColumn[index=4,columnType=93,columnName=createTime,columnValue=2019-04-24 02:05:11,isNull=false,isKey=false,isUpdate=true,isReverse=false]
row_load-2019-04-24-0.log- EventColumn[index=5,columnType=93,columnName=modifyTime,columnValue=2018-07-19 11:05:48,isNull=false,isKey=false,isUpdate=true,isReverse=false]
row_load-2019-04-24-0.log----Sql
row_load-2019-04-24-0.log- insert into `otter_db_1`.`message`(`traceId` , `status` , `userId` , `createTime` , `modifyTime` , `id`) values (? , ? , ? , ? , ? , ?) on duplicate key update `traceId`=values(`traceId`) , `status`=values(`status`) , `userId`=values(`userId`) , `createTime`=values(`createTime`) , `modifyTime`=values(`modifyTime`) , `id`=values(`id`)
load delete日志:
row_load-2019-04-24-0.log------------------
row_load-2019-04-24-0.log-- PairId: 44 , TableId: 13 , EventType : D , Time : 1556042714000
row_load-2019-04-24-0.log-- Consistency : , Mode :
row_load-2019-04-24-0.log------------------
row_load-2019-04-24-0.log----Pks
row_load-2019-04-24-0.log- EventColumn[index=0,columnType=3,columnName=id,columnValue=390442,isNull=false,isKey=true,isUpdate=true,isReverse=false]
row_load-2019-04-24-0.log----oldPks
row_load-2019-04-24-0.log-
row_load-2019-04-24-0.log----Columns
row_load-2019-04-24-0.log: EventColumn[index=1,columnType=-5,columnName=traceId,columnValue=1997861556042711096,isNull=false,isKey=false,isUpdate=true,isReverse=false]
row_load-2019-04-24-0.log- EventColumn[index=2,columnType=12,columnName=status,columnValue=updateId,isNull=false,isKey=false,isUpdate=true,isReverse=false]
row_load-2019-04-24-0.log- EventColumn[index=3,columnType=12,columnName=userId,columnValue=390442,isNull=false,isKey=false,isUpdate=true,isReverse=false]
row_load-2019-04-24-0.log- EventColumn[index=4,columnType=93,columnName=createTime,columnValue=2019-04-24 02:05:11,isNull=false,isKey=false,isUpdate=true,isReverse=false]
row_load-2019-04-24-0.log- EventColumn[index=5,columnType=93,columnName=modifyTime,columnValue=2019-04-24 02:05:13,isNull=false,isKey=false,isUpdate=true,isReverse=false]
row_load-2019-04-24-0.log----Sql
row_load-2019-04-24-0.log- delete from `otter_db_1`.`message` where `id` = ?
row_load-2019-04-24-0.log------------------
执行sql如下:
21-2019-04-24-3.log:2019-04-24 02:06:01.786 [pipelineId = 21 , pipelineName = sync , DbLoadAction] WARN com.alibaba.otter.node.etl.load.loader.db.DbLoadAction - sql23:delete from `otter_db_1`.`message` where `id` = ? --- columns:[EventColumn[index=0,columnType=3,columnName=id,columnValue=390442,isNull=false,isKey=true,isUpdate=true,isReverse=false]]
21-2019-04-24-3.log:2019-04-24 02:06:01.879 [pipelineId = 21 , pipelineName = sync , DbLoadAction] WARN com.alibaba.otter.node.etl.load.loader.db.DbLoadAction - sql23:insert into `otter_db_1`.`message`(`traceId` , `status` , `userId` , `createTime` , `modifyTime` , `id`) values (? , ? , ? , ? , ? , ?) on duplicate key update `traceId`=values(`traceId`) , `status`=values(`status`) , `userId`=values(`userId`) , `createTime`=values(`createTime`) , `modifyTime`=values(`modifyTime`) , `id`=values(`id`) --- columns:[EventColumn[index=1,columnType=-5,columnName=traceId,columnValue=1997861556042711096,isNull=false,isKey=false,isUpdate=true,isReverse=false], EventColumn[index=2,columnType=12,columnName=status,columnValue=test,isNull=false,isKey=false,isUpdate=true,isReverse=false], EventColumn[index=3,columnType=12,columnName=userId,columnValue=dxy_test,isNull=false,isKey=false,isUpdate=true,isReverse=false], EventColumn[index=4,columnType=93,columnName=createTime,columnValue=2019-04-24 02:05:11,isNull=false,isKey=false,isUpdate=true,isReverse=false], EventColumn[index=5,columnType=93,columnName=modifyTime,columnValue=2018-07-19 11:05:48,isNull=false,isKey=false,isUpdate=true,isReverse=false], EventColumn[index=0,columnType=-5,columnName=id,columnValue=390442,isNull=false,isKey=true,isUpdate=true,isReverse=false]]
为啥会先执行delete,然后在执行insert呢?
4条答案
按热度按时间kuarbcqp1#
问题已经查明,我在DbLoadMerger.merge中加入processid合并后的所有日志,代码如下:
日志如下:
从日志中看出,delete语句中columnType=3,但是insert语句中columnType=-5,造成这2个eventdate不是同一pk数据,导致未合并sql,最终执行的时候先执行delete后执行insert造成数据不一致问题。
目前解决方案是重写rowkey中的keys,也就是EventColumn的hashcode和equals:
iyr7buue2#
你这是哪个版本的
lokaqttq3#
@ppj19891020 方便提供下MySQL版本、对源端数据所有操作的SQL吗?columnType为什么会变?
ppcbkaq54#
估计是源端执行了什么ddl语句,然后同步没有配置ddl同步