我现在正在尝试创建一个spring引导批处理作业,它从elasticsearch索引中读取所有文档,这样这些文档就可以写入文件系统或其他目的地,如s3或kafka。
我已经找到一个答案张贴之前,但这是不再有效,因为ElasticSearch已删除扫描搜索类型,因为es 3。请参阅:用于elasticsearch扫描和滚动的spring batch itemreader
相反,我们应该使用scroll api。我找到了另一个扩展的 ItemReader
:
public class ElasticsearchReader<T> extends AbstractPaginatedDataItemReader<T> implements InitializingBean {
private final ElasticsearchOperations elasticsearchOperations;
private final SearchQuery query;
private final Class<? extends T> targetType;
public ElasticsearchReader(ElasticsearchOperations elasticsearchOperations, SearchQuery query, Class<? extends T> targetType) {
this.elasticsearchOperations = elasticsearchOperations;
this.query = query;
this.targetType = targetType;
}
@Override
public void afterPropertiesSet() throws Exception {
state(elasticsearchOperations != null, "An ElasticsearchOperations implementation is required.");
state(query != null, "A query is required.");
state(targetType != null, "A target type to convert the input into is required.");
}
@Override
@SuppressWarnings("unchecked")
protected Iterator<T> doPageRead() {
return (Iterator<T>)elasticsearchOperations.queryForList(query, targetType).iterator();
}
}
但是这个似乎没有使用scrollapi,而是使用默认分页。有人知道如何扩展阅读器来使用es scroll api吗?
暂无答案!
目前还没有任何答案,快来回答吧!