canal-adapter同步mysql至es,etlCondition无效

vsdwdz23  于 2022-10-22  发布在  Mysql
关注(0)|答案(9)|浏览(632)

Question

我的conf/es/test.yml内容如下:
dataSourceKey: defaultDS
destination: example1
groupId: g1
esMapping:
_index: test6-subaccount_history
_type: _doc
_id: _id
upsert: true
#pk: id
sql: "select f_id as _id, f_id, f_version, f_user_id, f_account_id, f_subaccount_id, f_account_type, f_subaccount_type, f_action_type, f_currency, f_amount, f_balance, f_suspense, f_clearing, f_total, f_action_id, f_source_id, f_source_type, f_source_type_code, f_source_ext, f_created_at, f_updated_at from t_subaccount_history"
#objFields:
#_labels: array:;
etlCondition: "where f_version>3"
commitBatch: 3000

不加etl条件可以正常的同步至es,但加上etl条件,依旧是把所有mysql更新或新增记录都同步到es,仿佛etl条件无效

ebdffaop

ebdffaop1#

我也遇到了同样的问题

gojuced7

gojuced72#

Question

我的conf/es/test.yml内容如下:
dataSourceKey: defaultDS
destination: example1
groupId: g1
esMapping:
_index: test6-subaccount_history
_type: _doc
_id: _id
upsert: true
#pk: id
sql: "select f_id as _id, f_id, f_version, f_user_id, f_account_id, f_subaccount_id, f_account_type, f_subaccount_type, f_action_type, f_currency, f_amount, f_balance, f_suspense, f_clearing, f_total, f_action_id, f_source_id, f_source_type, f_source_type_code, f_source_ext, f_created_at, f_updated_at from t_subaccount_history"
#objFields:
#_labels: array:;
etlCondition: "where f_version>3"
commitBatch: 3000

不加etl条件可以正常的同步至es,但加上etl条件,依旧是把所有mysql更新或新增记录都同步到es,仿佛etl条件无效

请问你这个问题解决了吗?

ny6fqffe

ny6fqffe3#

etlCondition:条件无效

q0qdq0h2

q0qdq0h25#

我也遇到了,1.1.5版本

s3fp2yjn

s3fp2yjn6#

可能是yaml格式的问题,etlCondition字段需要在 esMapping子级

ubof19bj

ubof19bj7#

也是醉了,这文档写的跟啥一样
etlCondition不支持直接写死的方式,如下
etlCondition: "where f_version>3"

只能这样etlCondition: "where f_version>{}"
然后再调用etl全量同步接口的时候
加上参数 ?params=3
这样才会生效

t1qtbnec

t1qtbnec8#

也是醉了,这文档写的跟啥一样 etlCondition不支持直接写死的方式,如下 etlCondition: "where f_version>3"

只能这样etlCondition: "where f_version>{}" 然后再调用etl全量同步接口的时候 加上参数 ?params=3 这样才会生效

请问如果有多个条件呢?etlcondition 怎么写?还有etl指令怎么拼接参数?

b5lpy0ml

b5lpy0ml9#

也是醉了,这文档写的跟啥一样 etlCondition不支持直接写死的方式,如下 etlCondition: "where f_version>3"
只能这样etlCondition: "where f_version>{}" 然后再调用etl全量同步接口的时候 加上参数 ?params=3 这样才会生效

请问如果有多个条件呢?etlcondition 怎么写?还有etl指令怎么拼接参数?

用分号分隔
etlCondition: "where id>={} and age = {}"
curl http://127.0.0.1:8081/etl/rdb/mysql1/mytest_user.yml?params=4;44 -X POST

相关问题