Every generation of distributed database was built to fix what the last one gave up. That is the useful way to read the evolution of distributed databases: not as a march toward a better product, but as a sequence of trades. Scale for consistency, then consistency back at the cost of proprietary hardware, then transactions and analytics in one system. Each step made the previous compromise unnecessary, and the current step is being forced by a workload that did not exist five years ago.
Historical Development and Milestones
The sequence matters more than the individual systems, because each entry is a response to the limitation of the one above it.
| Era | What arrived | What it traded away |
|---|---|---|
| Late 1970s to 1980s | Research prototypes, SDD-1 and IBM’s System R* among them | Proved distributed transactions were possible, but the hardware of the era could not make them fast |
| 2006 to 2007 | Google’s Bigtable and Amazon’s Dynamo papers | Bought scale and availability by giving up strong consistency |
| 2008 to 2010 | The open-source NoSQL wave: Cassandra, HBase, MongoDB | Gave developers horizontal scale, and moved the consistency problem into application code |
| 2012 | Google’s Spanner paper and TrueTime | Restored strong consistency at global scale, on infrastructure only Google had |
| 2015 to 2017 | CockroachDB and TiDB | Brought Spanner’s guarantees to commodity hardware and open source |
| 2020 onward | HTAP, in TiDB’s case through the TiFlash columnar engine | Removed the separate analytics warehouse and the pipeline feeding it |
| 2024 to 2026 | Vector search inside the transactional engine | Removes the standalone vector database and its sync gap |
Two corrections to the version of this history that circulates most often. Spanner was not an early system; it arrived in 2012, after the NoSQL wave and partly in answer to it. And NoSQL did not follow CockroachDB; it preceded both CockroachDB and TiDB by roughly a decade, which is precisely why both were built.
What Each Generation Left Unsolved
Three problems recur at every stage, and how a system handles them is what places it on the timeline above.
- Scale. Spreading a workload across nodes is the easy part. Doing it without the application knowing which node holds what is the part that took thirty years.
- Availability. Replication keeps a system serving through node failure, but only if failover happens without an operator and without losing a committed write.
- Consistency. This is the one the NoSQL generation traded away and the Raft and Paxos generation bought back. A consensus protocol commits a write only once a majority of replicas acknowledge it, so no node serves a version of the data the others have not agreed to.
The general trade-offs behind these three, and when distribution is worth its operational cost at all, are covered in our guide to distributed database use cases.
What TiDB Added: HTAP in One Cluster
TiDB entered the same lineage as CockroachDB in 2015 and made a different bet. Rather than choosing between transactional and analytical processing, it built both into one system: hybrid transactional/analytical processing, or HTAP.
The architecture is three parts. The Placement Driver (PD) manages metadata, schedules where replicas live, and balances load. TiKV is the row-based storage engine that serves transactions, replicating each range of data across nodes as its own Raft group. TiFlash keeps a columnar replica of that same data and serves read-heavy analytical queries, so analytics never competes with transactions for the same engine. The component-level detail is worth reading if you are evaluating the design rather than the history.
Where Distributed Databases Are Headed: The AI and Agentic Era
The next stage is being driven by what AI workloads ask of the database rather than by transaction volume. An agent needs its transactional state, its tool-call history, and its semantic retrieval index to agree with each other at the same instant, not to be synced on separate schedules across separate systems. That requirement did not exist when Spanner or the first NoSQL systems were designed, which is why HTAP is shifting from a differentiator to a baseline expectation.
TiDB extends the same architecture one step further: native vector search runs inside the same Raft-consistent SQL engine as TiKV and TiFlash, rather than in a separate vector database that has to be kept in sync. Because the vector index sits on the columnar replica and validates its log index against TiKV before serving, a retrieval query reads data consistent with what the agent just committed. That closes the same class of synchronization gap that made earlier distributed systems fragile whenever a new workload type arrived.
TiDB in Production
Three deployments across different pressures:
- MNC Bank uses TiDB’s transactional integrity and real-time analytics for financial processing, and reported 10x higher throughput, 50% lower latency, and 85% faster backups after adopting it.
- Flipkart handles high-volume transactions and customer traffic through peak sales without manual resharding.
- Plaid modernized its legacy MySQL estate on TiDB, cutting database-maintenance effort by 96% and gaining zero-downtime upgrades.
The Takeaway
The evolution of distributed databases has been a sequence of reclaimed compromises: scale without consistency, then consistency without commodity hardware, then transactions and analytics without two systems, and now retrieval without a separate store to synchronize. Each generation existed because the last one gave something up.
If you want to see where the current step landed at the architectural level, the TiDB architecture documentation covers how the components actually fit together.
FAQs
When were distributed databases invented?
The first distributed database prototypes are usually credited to research projects of the late 1970s and early 1980s, SDD-1 and IBM’s System R* among them. They demonstrated that transactions could span machines, but the hardware and networks of the period made them too slow for production. The commercially significant wave came much later, starting with Google’s Bigtable paper in 2006.
Is NoSQL a distributed database?
Most NoSQL systems are distributed, but the categories are not the same thing. NoSQL describes the data model and the consistency guarantee, typically non-relational and eventually consistent. Distributed describes how the system spreads data across nodes. A distributed SQL database like TiDB is distributed without being NoSQL: it keeps the relational model and strong consistency.
What problem does Spanner solve that NoSQL did not?
Strong consistency at global scale. The NoSQL generation achieved scale by accepting eventual consistency, which pushed conflict resolution into application code. Spanner, published in 2012, showed that a globally distributed database could offer strict transactional guarantees, using TrueTime to order transactions across regions. Its practical limitation was that it depended on Google’s own infrastructure.
What comes after HTAP?
Retrieval as a first-class part of the same engine. Agentic applications need vector similarity search, transactional state, and memory to reflect the same moment, which means the retrieval index has to live in the same consistency domain as the transactions rather than in a separate vector store synced on its own schedule.