{"id":26250,"date":"2025-03-31T09:36:03","date_gmt":"2025-03-31T16:36:03","guid":{"rendered":"https:\/\/www.pingcap.com\/?p=26250"},"modified":"2025-04-28T23:58:04","modified_gmt":"2025-04-29T06:58:04","slug":"tikv-write-latency-solved-smoother-performance-without-compromises","status":"publish","type":"post","link":"https:\/\/www.pingcap.com\/ko\/blog\/tikv-write-latency-solved-smoother-performance-without-compromises\/","title":{"rendered":"Write Latency, Solved: TiKV\u2019s Journey to Smoother Performance"},"content":{"rendered":"

Whether you\u2019re processing thousands of concurrent writes per second or scaling out infrastructure to meet a burst in demand, latency spikes can undermine user experience, reliability, and trust. At PingCAP, we obsess over these details. Our mission is to help teams build and scale confidently on distributed SQL<\/a>. This post highlights how a subtle performance challenge deep inside our storage engine\u2014write stalls during data ingestion\u2014could ripple across workloads in production and impact TiKV write latency. <\/p>\n\n\n\n

In this blog, we’ll dive deep into how RocksDB, Raft, and TiKV<\/a> interact, and how we solved this write stall challenge through two key optimizations that now eliminate those stalls without compromising correctness.<\/p>\n\n\n\n

<\/span>When Bulk Data Ingestion Slows Down TiKV Write Latency<\/span><\/h2>\n\n\n\n

\ud2f0DB<\/a>, a distributed SQL database, scales effortlessly with modern applications. Under the hood, it separates compute and storage to optimize for performance and flexibility. TiKV, an open-source distributed key-value engine, serves as the storage layer powering TiDB\u2019s transactional workloads.<\/p>\n\n\n\n

Whenever TiDB needs to move large amounts of data\u2014for example, during Region migration, cluster rebalancing, or scaling operations\u2014it relies on TiKV to load that data quickly and reliably.<\/p>\n\n\n\n

TiKV uses RocksDB<\/a> as its embedded storage engine. When large volumes of data need to be loaded, it uses a method called IngestExternalFile()<\/code>, which directly imports Sorted String Table (SST) files into storage. This approach bypasses the normal write path, reducing memory pressure and minimizing compaction overhead. RocksDB\u2019s ingestion process is designed for speed and efficiency \u2014 especially compared to writing data through the foreground path.<\/p>\n\n\n\n

But this speed comes with a catch. To maintain sequence number consistency in RocksDB\u2019s log-structured merge tree (LSM-tree), the ingestion process temporarily blocks all foreground writes. This temporary pause is known as a write stall.<\/p>\n\n\n\n

In isolation, a short stall might not matter. But all Regions (logical partitions of data) on the same TiKV node share a single RocksDB instance. So when one Region triggers a write stall, every Region on that node is affected\u2014even if they\u2019re unrelated. During large migrations or cluster scaling, this can lead to widespread write latency spikes, affecting the performance of otherwise unrelated workloads.<\/p>\n\n\n

\n
\"Each
Figure 1. Each TiKV node hosts multiple Regions, all backed by a single RocksDB instance. During SST ingestion, a write stall in one Region affects all others on the same node.<\/em><\/figcaption><\/figure>\n<\/div>\n\n\n

<\/span>Why Sequence Numbers Matter to TiKV Write Latency<\/span><\/h2>\n\n\n\n

In RocksDB\u2019s log-structured merge tree (LSM-tree), data exists across multiple levels. To preserve consistency and ensure fast, correct reads, RocksDB assigns a global sequence number to every write.<\/p>\n\n\n\n

These sequence numbers are especially important during SST (Sorted String Table) ingestion. SSTs are immutable, sorted files used by RocksDB to store data efficiently on disk. When new SST files are ingested via IngestExternalFile()<\/code>, they must fit into the sequence number ordering rules that RocksDB enforces.<\/p>\n\n\n\n

Specifically:<\/p>\n\n\n\n