TiDB Log Compaction: Faster Point-in-Time Recovery for Large Clusters

Key Takeaways

• Long recovery windows make PITR restore expensive: fragmented log files, many small writes through TiKV, and RocksDB write amplification afterward.
• Log compaction converts backed-up logs into sorted SST files offline. Restore imports those SSTs and replays logs only for uncompacted ranges.
• Internal benchmarks cut log restore time by 52% to 99%, and made restore feasible on a 206.6 TiB cluster where traditional PITR was not.
• Compacted log backup ships in TiDB v8.5.5. Keep BR version-matched, keep periodic full backups, and check the encryption limitation first.

For a large distributed SQL cluster, backup and restore define whether the business can recover from an accident inside a realistic service objective. As TiDB adoption grows across larger, more write-intensive workloads, many production clusters now hold tens or hundreds of terabytes of data. At that scale, point-in-time recovery (PITR) has to satisfy two goals that pull against each other:

• Low Recovery Point Objective (RPO). Protect recent committed changes.
• Low Recovery Time Objective (RTO). Do not spend hours replaying fragmented logs at restore time.

Traditional PITR log backup handles the first goal well. It continuously captures write changes and persists them to external storage. The second goal is harder. When a restore has to replay a large log window, the restore path must read highly fragmented log files, reconstruct many small write operations, send them through TiKV, and then let RocksDB compact the resulting LSM-tree shape afterward.

TiDB log compaction changes that path. Instead of processing unstructured log records at restore time, TiDB compacts backed-up log data offline into structured SST files. Restore then imports those SST files much like snapshot backup files, which cuts logical replay, Raft write overhead, and post-restore write amplification.

Starting with TiDB v8.5.5, PITR supports recovery from compacted log backups. The same release added table-level PITR filters, checkpoint storage for BR, and faster system-table restore. TiDB v8.5.6 then fixed several PITR and BR restore-point issues, including PITR metadata upgrade failures and restore-point waits on schema reload. This post covers why log compaction matters, how it works, what internal performance tests showed, and how to operate it safely at scale.

Why Log Replay Is Expensive for Long Recovery Windows

TiDB’s PITR model combines snapshot backups and continuous log backups:

• Snapshot backup. Captures the cluster at a point in time as structured SST files.
• Log backup. Captures ongoing changes after the snapshot, so you can recover to a later timestamp.

This is the right architecture for correctness and low RPO. Recovery performance depends on how much log data must be replayed after the snapshot. In short windows, log replay is manageable. In long windows, it becomes expensive for three reasons.

First, log backup files are organized for continuous capture, not for fast bulk ingestion. They are range-based, time-sliced, and optimized for durability and incremental progress, not sorted into restore-friendly SST boundaries. Second, replaying unordered log records turns recovery into many small write operations. Compared with importing sorted SST files, this creates more request overhead, more retries when Region epochs change, and more pressure on Raft and TiKV memory. Third, the restored cluster eventually pays the LSM-tree write amplification cost. If a large amount of log data is replayed as ordinary writes, many of those writes land in L0 and must later be compacted down through RocksDB levels. Recovery might appear complete, but the cluster can still carry significant compaction debt.

Before log compaction, the common mitigation was to increase snapshot backup frequency. That reduces the recovery window and limits log replay, but it has a cost. Full backups scan large amounts of data from the production cluster, and on busy clusters frequent full backups compete for CPU, I/O, and GC safe point progress.

Log compaction provides a better middle ground:

• Keep continuous log backup for low RPO.
• Keep periodic snapshot backup as the durable baseline.
• Add compacted log backup as an intermediate restore layer to reduce RTO without increasing snapshot backup frequency.

How the Full Backup Plus Compacted SST Pipeline Works

The design is a multi-level recovery pipeline:

1. A periodic full backup provides the base snapshot.
2. Log compaction converts the selected log backup range into incremental SST files. This step can run in parallel with the full backup. The compacted range is represented by SST artifacts and compaction metadata, not by a second copy of the original log files.
3. Log files outside the compacted range stay as log files. For a fully compacted recovery range, restore consumes the generated SSTs directly.

During restore, TiDB applies the layers in order:

1. Restore the full snapshot.
2. Import compacted SST files for the selected log backup range.
3. Replay logs only for time ranges that were not compacted, if any. Restore reads the compacted range from SST files.

This shifts most of the heavy work from restore time to an offline background process. The source cluster keeps producing log backup files as before. Once a target range is compacted, that range is materialized as SST artifacts plus metadata in backup storage, and restore treats those SSTs as the recovery input for the covered range rather than expecting the original log files to remain available.

The result is similar in spirit to what TiKV does inside RocksDB. It reorganizes fragmented write history into sorted files. The difference is that log compaction does this before restore, outside the source cluster, over backup files in external object storage.

What Internal Benchmarks Show About Restore Speed

Internal benchmark results showed that log compaction makes PITR restore practical in scenarios where traditional log replay is too slow or resource-intensive.

Scenario	Workload profile	Before log compaction	After log compaction	Result
Customer-like workload (default PITR baseline)	4.2 TiB dataset, read-heavy, ~25K QPS	10h13m total restore, 9h46m log restore	34m57s total, 6m39s log restore	Total time down 94.3%; log phase down 98.9%
Customer-like workload (tuned PITR baseline)	Same workload, tuned PITR concurrency	1h22m total, 20m21s log phase	34m57s total, 6m39s log phase	Total time down 57.7%; log phase down 67.3%
MVCC-heavy workload	3.2 TiB dataset, read-heavy, ~11.3K QPS	50m53s total (tuned), 22m43s log phase	38m20s total, 10m52s log phase	Log phase down 52.2%
Large write-heavy workload	206.6 TiB cluster, 118 TiKV nodes, large-table write-heavy	Not practically comparable: high concurrency caused memory pressure, low concurrency was too slow	1h48m49s full restore, 36m04s log restore	A previously impractical path became operationally feasible
Early 24-hour customer-like test	23 TiB cluster, 9 TiKV nodes, 6K inserts and 18K updates per second	~2h04m log restore	~40m log restore with remote compaction	Log restore time down ~68%

These results came from controlled benchmark environments. Actual results vary with workload shape, object storage throughput, TiKV topology, restore tuning, and the size distribution of generated SST files.

The tests cover different shapes: read-heavy workloads, MVCC-heavy updates, large single-table and wide-table workloads, and high-write scenarios. The improvement varies because log compaction benefits are workload-dependent. The more concentrated and reusable the changed key ranges are, the more SST import can replace logical log replay.

One lesson matters beyond raw speed. Log compaction also changes failure modes. In the large write-heavy benchmark, traditional PITR either put too much pressure on TiKV memory at high concurrency or became too slow at lower concurrency. Compacted SST restore avoided most of that logical replay pressure.

How Log Compaction Works

The compaction process is deliberately offline and stateless. It reads existing log backup files from external storage, groups them into compaction tasks, sorts their MVCC records, generates SST files, and writes SST artifacts and compaction metadata back to external storage. After the metadata is committed, restore treats the covered range as SST-backed. The source cluster does not need to participate in the compaction work.

At a high level:

1. The source cluster continuously writes log backup files to object storage.
2. A compaction worker scans file metadata for a target time range.
3. The worker groups related files, commonly by Region history or key range.
4. The worker reads the selected log files, keeps all required MVCC versions, and sorts records.
5. The worker writes generated SST files to a compaction area in backup storage.
6. The worker writes compaction metadata after the artifacts are complete.
7. Restore discovers the compaction metadata and imports the generated SST files for the covered range. It reads normal log files only for ranges that were not compacted.

Keeping all MVCC versions is a deliberate design choice. PITR must restore to any timestamp in the selected range. If compaction applied GC-like logic too early, correctness would become harder to reason about. By retaining the required MVCC history, the compaction process stays simpler and safer.

The process is also atomic from the restore path’s point of view. A compaction task becomes visible only after its SST artifacts and metadata are complete. If compaction fails before metadata is committed, the range is not advertised as compacted. Once the metadata commits, restore reads the covered range from SST artifacts. Operators should not rely on original log files for that compacted range.

How Log Compaction Preserves PITR Correctness

Log compaction is a performance optimization, not a replacement for the PITR correctness model. Several safeguards matter:

• Committed input only. Before compaction commits, the original log backup remains the input. After the metadata commits, SST artifacts and metadata represent the covered range.
• Mixed inputs at restore. Restore tolerates a mixture of full snapshot files, compacted SST files, and normal log files for uncompacted ranges. It does not expect log files for a range already compacted into SST.
• Metadata defines coverage. Compaction metadata records which source files and time ranges are covered.
• Failures stay invisible. Failed or partial compaction remains invisible to restore until the metadata commits.
• MVCC preserved. The generated SST files retain the MVCC information PITR needs.

For validation, internal designs evaluated file-level checksum strategies and end-to-end consistency checks. In benchmark runs, the team also ran consistency validation on restored clusters after restore.

One subtle point is worth stating. A compaction range is bounded by timestamps, but backup log files can contain data that overlaps or extends beyond the requested range. This is expected. The restore layer uses timestamp visibility rules to recover to the selected point in time.

How to Operate Log Compaction Safely at Scale

Log compaction adds a recovery layer, so operate it deliberately. The latest TiDB Operator can trigger log compaction during the full restore phase, which removes most of the manual coordination.

Align compaction with full backup boundaries. Choose the compaction range so that it starts close to the latest successful full backup timestamp and ends at the intended compacted recovery boundary. Restore then applies the full snapshot, imports the compacted incremental SSTs for that covered range, and replays log files only after the compacted boundary if a newer tail still exists.

Use a matching BR version. For PITR, use a BR version that matches the target TiDB cluster version. The TiDB v8.5.5 release notes warn that running BR v8.5.5 against earlier cluster versions such as v8.5.4 or v8.1.2 can cause log recovery failures.

Plan restore cluster capacity. Compacted SST restore is faster, but it can require significant temporary disk and ingestion capacity. Longer compaction windows produce larger SST artifacts. For very large ranges, insufficient TiKV disk headroom can still cause restore failures. A fresh target cluster with enough storage, import threads, and network bandwidth remains the recommended recovery target.

Understand current limitations. Compacted log backup is not a replacement for full backups. Use it together with periodic full backups. Because compaction retains the MVCC versions PITR needs inside the generated SSTs, very long compaction windows can increase SST artifact size, object-storage usage, and restore risk. Current TiDB documentation also notes that compacting backups with local encryption enabled is not supported. Validate object-storage permissions, credential handling, and encryption requirements before enabling compaction.

What TiDB 8.5 Added for PITR Recovery

TiDB v8.5.5 is the key 8.5 release for this recovery path. The release notes call out that PITR supports recovery from compacted log backups, and describe three benefits:

• Faster recovery. SST files import quickly.
• Lower storage consumption. Compaction removes redundant data.
• Reduced application impact. RPO holds with less frequent snapshot backups.

TiDB v8.5.5 also added adjacent recovery capabilities:

• Table-level PITR from log backups using filters.
• BR --checkpoint-storage.
• BR --fast-load-sys-tables, enabled by default for faster physical restore of system tables on new clusters.
• tidb_advancer_check_point_lag_limit for log backup checkpoint lag control.
• Better compatibility between ongoing log backup and snapshot restore.

TiDB v8.5.6 then fixed several issues that matter for PITR operations, including a PITR metadata upgrade failure, flush_tsbeing 0 in log backup, and br restore point getting stuck while waiting for schema information to finish reloading.

For production use on 8.5, the guidance is simple. Use the latest v8.5 patch release, and keep BR aligned with the target cluster version.

Why Log Compaction Is an Architecture Change, Not a Feature

The core of log compaction is not the new command. It is the change in recovery architecture. Before compaction, teams had to choose between:

• Frequent full backups, which improve RTO but increase production impact.
• Long log replay windows, which preserve low RPO but can make restore too slow.

With compaction, TiDB separates these concerns. The source cluster continuously writes logs for RPO. Offline compaction reorganizes older log data for RTO. Restore imports compacted SSTs for speed and replays logs only for ranges that were not compacted, such as a newer tail after the compacted boundary.

This same compaction layer underpins region-level disaster recovery in TiDB Cloud. In Cross-Region PITR, snapshot restore and log backup compaction run in parallel during a cross-region restore, so a Dedicated cluster can reach hour-level RTO in a second region without running a duplicate production stack.

The net effect is a better set of tradeoffs. Teams can reduce full backup frequency without letting log replay windows grow uncontrollably, keep recent data protection tight while moving heavy data reorganization to elastic compute, and make large-scale PITR recovery more predictable.

Where Log Compaction Goes Next

Log compaction sets up several future improvements:

• Managed scheduling through TiDB Operator or cloud control planes.
• More automated compaction policies based on log volume, checkpoint lag, and restore objectives.
• Better observability for compacted ranges, artifact sizes, and restore path selection.
• Deeper integration with table-level PITR and disaster recovery workflows.
• More cost-aware compaction strategies that balance object-storage I/O, compute, and recovery objectives.

The direction is clear. PITR should not be limited by the cost of replaying fragmented logs. By converting log history into restore-ready SSTs before disaster strikes, TiDB makes point-in-time recovery faster, more predictable, and more practical at scale.

Running large-scale PITR on TiDB v8.5? Read the Compact Log Backup documentation to enable compaction on your existing log backups, or talk to the PingCAP team about a recovery architecture sized for hundred-terabyte clusters.