缺少Flink保存点数据文件夹

ubof19bj  于 2022-12-09  发布在  Apache
关注(0)|答案(1)|浏览(120)

I was testing Flink savepoint locally.
While taking savepoint, I mentioned the folder, and save point has been taken into it.
Later I restarted the Flink cluster and restored the Flink job from the savepoint and it worked as expected.
My concern is about the save point folder contents.
I am only seeing the _metadata file in the folder.
Where will the savepoint data get saved?
It is clear from the documentation that( https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/savepoints/#triggering-savepoints )
"If you use statebackend: jobmanager, metadata and savepoint state will be stored in the _metadata file, so don’t be confused by the absence of additional data files."
But I have used rocksDb backend for states.

state.backend: rocksdb
state.backend.incremental: true
state.backend.rocksdb.ttl.compaction.filter.enabled: true

Thanks for the helps in advance.

bhmjp9jg

bhmjp9jg1#

If your state is small, it will be stored in the metadata file. The definition of "small" is controlled by state.backend.fs.memory-threshold . See https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/config/#state-storage-fs-memory-threshold for more info.
这样做是为了防止创建大量的小文件,这会导致一些文件系统(如S3)出现问题。

相关问题