incubator-doris [Feature] support TiKV as an option for metadata storage

h9vpoimq  于 2022-04-22  发布在  Java
关注(0)|答案(2)|浏览(187)

Search before asking

  • I had searched in the issues and found no similar issues.

Description

TiKV is a low latency, strongly consistent, highly available key-value database that supports ACID transactions. It has been used to store metadata for some infrastructures and services, for example:

  1. Metadata of the Object Storage Service in JD Cloud: https://www.cncf.io/blog/2019/11/26/case-study-tikv-in-jd-cloud/
  2. Metadata of the JuiceFS, which is a POSIX file system designed for cloud-native environment: https://juicefs.com/blog/en/posts/juicefs-release-v016/

TiKV is also the storage of TiDB, a distributed HTAP database used by 1500+ users in production. Which gives it the following strengths:

  1. Low maintenance overhead for existing TiDB users, the SOPs, KBs of TiDB storage layer can be reused
  2. No need to learn how to maintain another system for metadata storage

Use case

By adopting TiKV, it enables us to solve the following problems:

  1. Doris stores metadata in local disk, caches it in memory in a specific format. It might be inconvenient to view or modify it
  2. As the cluster and data grow, there might be more metadata needed to be stored, the current metadata storage mechanism might be a bottleneck

N/A

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

f4t66c6m

f4t66c6m1#

Currently, doris store rowset meta and tablet meta in local rocksdb. Could you explain more detail why need to store it to a external global KV store?

llycmphe

llycmphe2#

Doris's FE and BE both has metadata, so which part do you want to replace?
Can you offer more detailed design?

相关问题