A distributed database earns trust at the worst possible moment. Not when every node is healthy, but when a leader dies mid-transaction, or an entire region goes dark. The question is what the cluster does in the seconds that follow, and whether a write your application already acknowledged is still there afterward. Data consistency in distributed databases with TiDB rests on two mechanisms working together: Multi-Raft, which replicates every piece of data across nodes and commits only on majority agreement, and the Timestamp Oracle, which stamps every transaction with a globally ordered timestamp that never moves backward. This article walks through both, then follows an actual failover step by step to show why no committed write is lost or reordered.
How Does TiDB Ensure Data Consistency in Distributed Databases?
TiDB ensures data consistency using Multi-Raft for replication and the Timestamp Oracle (TSO) for global ordering. Data is split into Regions, each replicated across nodes as an independent Raft group that commits a write only after a majority of its replicas have persisted it. TSO assigns every transaction a strictly increasing timestamp, so transaction order is defined cluster-wide rather than per node. Together these give strong consistency: a read returns the most recent committed write, not a stale copy that converges later.
The CAP theorem sets the boundary. Under a network partition, a distributed system chooses between consistency and availability, and TiDB chooses consistency. A minority partition stops serving writes rather than accepting writes it cannot reconcile. That is a deliberate trade, and the right one for a ledger, an inventory count, or an order book. The wider framing sits on data consistency in distributed systems.
Multi-Raft: Consistency Without a Single Point of Failure
TiDB splits table data into Regions, contiguous ranges of key-value pairs that form the unit of replication and scheduling. Each Region is replicated to multiple nodes, three by default, and those replicas form their own Raft group. A cluster holding thousands of Regions therefore runs thousands of independent Raft groups at once, which is what Multi-Raft describes.
Within a Raft group, one replica is the leader and the rest are followers. The leader accepts all writes for that Region, appends each one to its Raft log, and replicates the entry to its followers. The entry is committed once a majority of the group has written it to durable storage. Only then does the write become visible. With three replicas, that means two nodes must agree, so a single node failure cannot cost a committed write, and no single node can commit one on its own.
Running many small Raft groups rather than one large one is what makes this scale. Each group elects its own leader and orders its own log independently, so leadership is spread across the cluster and every node carries write traffic for the Regions it leads. Adding nodes moves Region replicas onto them rather than concentrating load, and the Placement Driver (PD) handles that redistribution while the cluster stays online.
Followers do not serve stale reads by default. A read is routed to the Region’s leader, which makes the majority-commit guarantee visible to the application rather than merely true internally.
What Happens During a Regional Failover
The guarantee that matters is what survives when a leader disappears. Two mechanisms answer that, and they answer different halves of the question.
Raft handles which replica takes over. When followers stop receiving heartbeats from their leader, one of them times out and stands for election. The rule that protects committed data is that a candidate cannot win unless its log is at least as current as that of the majority who vote for it. Because a committed entry is by definition already on a majority of replicas, any candidate capable of assembling a majority necessarily holds every committed entry. A replica missing committed writes cannot become leader. Recovery is automatic, completes in seconds, and loses no committed write.
TSO handles ordering. The Placement Driver allocates timestamps to every transaction in the cluster, which is what enables TiDB’s Percolator-based transaction model and MVCC. A TSO timestamp is a single number in two parts: 46 bits of physical time in milliseconds, and an 18-bit logical counter for events falling inside the same millisecond. The logical counter is what keeps the guarantee intact if the physical clock ever stalls or steps backward. In that case the physical part holds steady while the logical part keeps advancing, so the timestamp always moves forward and never regresses.
That monotonicity is why a leader change cannot reorder history. Transaction order is established by TSO before any replica is involved, and it is a property of the cluster rather than of the node that happened to be leading. A new leader inherits an ordering that was already fixed. It does not reconstruct one from its own clock, which is precisely the failure mode that makes wall-clock ordering unsafe across machines.
Worth separating from all of this: uncommitted writes are a different case. An entry that reached the old leader but never reached a majority was never committed, so it was never acknowledged to the application and never visible to a reader. The new leader may discard it. That is correct behavior rather than data loss, and it is the reason a client should treat a write as durable only once it returns successfully.
Multi-Region Consistency in Production
Stretching a cluster across geography means choosing a topology, and the choice sets what you can promise. Two patterns cover most deployments.
The first is one cluster spanning multiple sites. Deployed across three availability zones in a single region, replicas are distributed so that data consistency is unaffected if one AZ fails, giving a recovery point objective of zero. Extending further, the three availability zones in two regions architecture uses five replicas across a production AZ, a disaster recovery AZ in the same region, and a third AZ in another region, which survives a region-level disaster. PingCAP describes this shape as a 2-2-1 architecture when spread across three regions, where the third region holds the replica that lets the remaining sites still reach a majority.
The cost is latency, and it is unavoidable rather than a tuning problem. Every committed write needs majority acknowledgment, and TiDB commits transactions with two-phase commit, so a write spanning distant sites pays multiple network round trips before it returns. The further apart the sites, the higher the write latency floor. Reads can be served locally, but writes cannot escape the speed of light.
The second pattern accepts that trade in the other direction: two separate clusters, with changes replicated asynchronously between them using TiCDC. Write latency stays local because commits only need a majority within the primary cluster. The trade is that replication lag is real, so the recovery point objective is not zero and a failover may lose recently committed transactions.
MNC Bank runs the second pattern. The Indonesian private bank moved its Motion Apps mobile platform, which handles fund transfers and real-time notifications, onto TiDB, and uses TiCDC to replicate changes to a secondary cluster at its disaster recovery site. Before cutover the team ran load tests and failover simulations, and used TiCDC to mirror changes back to the legacy database so both systems stayed consistent during the transition. Migration took two weeks with a three-hour cutover window, and the platform now absorbs a tenfold throughput increase at month-end and payout-day peaks with application latency cut by 50%. Read the full MNC Bank case study.
Choose between the patterns on your recovery point objective. If losing recent transactions is unacceptable, stretch one cluster and budget for the write latency. If local write latency matters more and a small window of loss is survivable, run two clusters with asynchronous replication.
Both patterns are worth testing before you commit to one. Start with TiDB Cloud Starter, free, or build a cluster in TiDB Labs.
Frequently Asked Questions
How does TiDB ensure data consistency across large-scale databases?
TiDB splits data into Regions and replicates each one across nodes as an independent Raft group. A write commits only after a majority of that group has persisted it, so no single node failure can cost a committed write. The Timestamp Oracle assigns every transaction a globally increasing timestamp, which defines transaction order across the whole cluster rather than per node. Running thousands of independent Raft groups is what lets this hold as the cluster grows, because leadership and write traffic spread across nodes instead of concentrating.
What happens if a TiDB node fails during a transaction?
If the failed node was a follower, the Region continues serving from its leader, and the majority requirement is still met by the remaining replicas. If it was a leader, the surviving followers hold an election, and a candidate can only win if its log is at least as current as the majority voting for it. Since a committed entry already sits on a majority, any new leader holds every committed write. Recovery is automatic and typically takes seconds. Writes that never reached a majority were never committed or acknowledged, so discarding them is correct rather than data loss.
What is TSO and why does TiDB need it?
The Timestamp Oracle is the service in the Placement Driver that allocates timestamps to transactions across the cluster. Each timestamp combines 46 bits of physical time in milliseconds with an 18-bit logical counter for events within the same millisecond. If the physical clock stalls or steps backward, the physical part holds while the logical part advances, so timestamps always move forward. TiDB needs this because independent machine clocks drift, and a distributed transaction cannot be ordered safely against a clock that might run backward. TSO also underpins TiDB’s Percolator-based transaction model and MVCC.
Can TiDB stay consistent across multiple regions?
Yes, with a topology choice that determines what is guaranteed. A single cluster stretched across availability zones or regions keeps strong consistency and a recovery point objective of zero, because every commit still needs majority acknowledgment, but cross-region round trips raise the write latency floor. Two clusters linked by asynchronous TiCDC replication keep write latency local, at the cost of replication lag and a non-zero recovery point objective. Decide based on whether losing recently committed transactions during a regional failure is acceptable.