As a rule of thumb, we call an infrastructure scalable if it can maintain flexibility, performance, reliability, and operability as business demands increase. In a truly scalable system, growth has no impact on operations except for the additional cost of resources like storage and compute. <\/p>\n\n\n\n
That\u2019s the ideal. In practice, businesses tend to hit a point where scaling becomes exponentially harder. The problem comes on slowly. Fixing it can take even longer. This is most evident at the data layer. At the heart of the data layer are the application\u2019s sources of truth; its systems of record. These are where application data changes happen first and remain longest. They are typically served by relational, ACID-compliant databases<\/a>, which are notoriously “sticky.\u201d The very properties of these databases that enforce data integrity have historically made them very resistant to changes in the application. <\/p>\n\n\n\n Changes to the data layer come in many forms, most of which are downstream of the business or the application. One form is growth in the number of users, as the business adds more customers. Others may have to do with the amount of data you ingest, read, and manage per user, as applications add features and the business finds more ways to leverage data it collects. It could be any or all of the above. Whatever the scenario, the growth in users, user data, and frequency of queries pressures the data layer to do something it was historically not built to do: expand. If the database can\u2019t easily scale, innovation gets impeded or rejected. Tech debt increases. Maintenance costs rise.<\/p>\n\n\n\n This problem is a failure of architecture. Monolithic ACID databases are, at their core, designed to be run on a single instance. As pressure increases on them, applications incur higher latency, lower availability, and ultimately failure. <\/p>\n\n\n\n So if dynamic scaling is an architectural problem, what does an architectural solution look like? <\/p>\n\n\n\n Distributed SQL<\/a> is the most modern answer to this question. Inspired by innovators at Google<\/a>, distributed SQL architectures are specifically designed to handle scenarios where an organization has outgrown its monolithic relational system. It is an ACID-compliant relational database cluster that scales writes, reads, and data stored horizontally.<\/p>\n\n\n\n To illustrate how distributed SQL differs from a traditional SQL RDBMS, I\u2019ll refer to the open source distributed SQL database \ud2f0DB<\/a>.<\/p>\n\n\n\n The most obvious bottleneck in a system that enforces strong data integrity like ACID compliance is how you safely distribute (scale) data-modifying transactions. This is impossible in traditional application databases. <\/p>\n\n\n\n However, TiDB \u2014 inspired by multiple paradigm-shifting Google papers \u2014 does gracefully support horizontal scale of these transactions. Where one ACID-compliant SQL instance controlled write consistency through a single transaction management process, TiDB allows for unlimited transaction managers simultaneously. TiDB automates database sharding behind the scenes, totally transparent to applications. Developers will never need to think about sharding. <\/p>\n\n\n\n The first problem of scaling transactions is enforcing the write consistency required to ensure durability and availability of all data written. TiDB stores schemas and tables of any size in many small moveable \u201cshards\u201d called \u201cregions\u201d or key ranges, scattered across a cluster of storage nodes. A single node may have hundreds of thousands of these \u201cshards\u201d and a cluster may have many millions. These key ranges are their own raft<\/a> replication groups (\u201craft groups\u201d), guaranteeing that every write is replicated to a majority of replicas. Any raft group may accept writes simultaneously. This is referred to as a \u201cmulti-raft\u201d or \u201cmulti-write\u201d architecture. The number of simultaneous writes against the system scales with the cluster.<\/p>\n\n\n\n This is roughly illustrated below (TiKVs are TiDB storage nodes).<\/p>\n\n\n Write replication consistency is only part of the problem when you distribute important data across many nodes. In relational systems, a single business-driven data change may happen in multiple nodes. The business requires all of the data changes to succeed\u2026or else none succeed. This is called atomicity. TiDB enforces this as well. <\/p>\n\n\n\n To do so, it leverages the Percolator<\/a> 2-phase commit (2PC) protocol, a sophisticated key locking algorithm, to guarantee all or none of a transaction is stored. This achieves atomicity even in cases where large data-modifying transactions are \u201cfanned out\u201d to many storage nodes. The larger the transaction, the more important this is. And TiDB supports any size transaction.<\/p>\n\n\n\n It\u2019s an eye chart but, for the curious, below is a visualization of the 2PC framework implemented in TiDB.<\/p>\n\n\n When you horizontally distribute a transactional workload, many transactions may occur at the same time. Businesses require that each of these transactions see a consistent view of the data the whole time the transaction is processing. In other words, no data change from transaction T2 that occurred after transaction T1 started will be seen by T1. TiDB achieves this by enforcing Snapshot Isolation. <\/p>\n\n\n\n TiDB accomplishes this with multi-version concurrency control (MVCC) to store multiple versions of recently modified data in order by timestamps, enabling time-based snapshots of the entire cluster and providing snapshot isolation<\/a>. <\/p>\n\n\n\n This combo of multi-raft, 2PC, and MVCC allows for extremely high throughput of ACID transactions of all kinds that are horizontally and linearly scalable.<\/p>\n\n\n\n TiDB\u2019s query cost optimizer inherent to the SQL compute layer is distributed-aware. It understands distributed table statistics to optimize plans for how queries will be executed, and it knows locations of key ranges of table data to more optimally execute the cost-optimized query plans. This all keeps query latency low as the cluster scales.<\/p>\n\n\n\n TiDB\u2019s architecture is not only distributed but disaggregated. Compute and storage are separated. The separation of compute from storage allows for an important separation of concerns, mainly of state. TiDB compute nodes are agile and can be added and removed with ease. When added, they become ready for serving traffic almost immediately. Together, these properties make TiDB faster and more cost-effective to scale out.<\/p>\n\n\n\n The aforementioned properties facilitate \uc218\ud3c9\uc801 \ud655\uc7a5<\/a> of data workloads that were not previously scalable. But there\u2019s more reason to consider the true scalability of a data layer than simply increasing throughput and storage of critical data. A system may scale to meet the needs of the workload, sure. But, once in place, can it bend to the needs of a changing business? This is an all too overlooked consideration when choosing scaling systems.<\/p>\n\n\n\n As a system writes and stores more data, the entire paradigm of the database changes. It has to account for:<\/p>\n\n\n\n Databases can get so bogged down by these operations, businesses will actively avoid using them. Obviously, this destroys developer velocity and stifles business growth.<\/p>\n\n\n\n Distributed SQL systems have the opportunity to parallelize these operations so that as the workload scales, the speed of these operations remain largely constant. TiDB takes these opportunities. <\/p>\n\n\n\n Arguably most important, and an often overlooked aspect of scaling \u2014 is how often the database\u2019s schema can change, and what it means for the business if that process slows down \u2013 or worse, can\u2019t be accomplished at all. Typically, the larger the schemas and tables, the more cumbersome the process becomes, and the less likely it is that orgs will try to make those changes in the first place. In the most extreme cases, the difficulty of modifying a database schema can keep developer orgs from providing certain features at all.<\/p>\n\n\n\n In addition to transactions, TiDB leverages its distributed architecture<\/a> for schema changes as well. For instance, when adding an index, data must be read and reorganized. TiDB splits the reading into many workers and splits the reorganizing of the data into many other workers, all scattered across the cluster. This means schema changes do not slow down as the business scales. This mechanism also leverages cloud-native technology like external object storage for storing temporary results, making the entire operation less expensive and more stable.<\/p>\n\n\n\n When a storage node goes down for any reason, the architecture of many mini raft-replicated shards can save the day. Raft leadership is transferred immediately to consistent replicas contained in the remaining live nodes. This enables reads and writes against the keys that were being served by the downed node. As a result, the availability of your application\u2019s most important data is unmatched.<\/p>\n\n\n The above illustrates the multi-raft mechanism during a storage node outage and recovery. A node goes down (upper left), taking down the shard leaders that were on that node. The cluster quickly realizes this and re-elects leaders on remaining nodes (upper right). During the time to \u201crealization\u201d of the outage, there is a blip in availability loss on the affected keys, followed by a very quick recovery to normal throughput picked up by the remaining nodes (bottom).<\/p>\n\n\n\n This means the larger the cluster, the less impactful a node outage. <\/p>\n\n\n\n A many-shard architecture also mitigates the classic challenges of scaling out<\/a> a sharded system and making sure data is still balanced. This architecture makes for easy redistribution to new nodes, balancing the workload as soon after scaling as possible. The larger the cluster, the faster the scale-out. Scale-down operations are similar.<\/p>\n\n\n\n In SaaS<\/a> deployments, scaling up often means scaling up the number of \u201ctenants\u201d: users that are logically walled off from one another for privacy, performance and security purposes. More tenants means more objects like schemas and tables. This in turn means more metadata to manage and more optimizer statistics to collect.<\/p>\n\n\n\n Compared to traditional SQL, distributed SQL architectures can more easily accommodate multi-tenant environments<\/a>. I\u2019ll illustrate with an example.<\/p>\n\n\n\n Atlassian recently launched its fully managed developer platform, Forge SQL, enabling 3rd party developers to extend or entirely build their own businesses on top of any Atlassian product, like Jira. They give their builder customers an interface to essentially work with the app database directly.<\/p>\n\n\n\n Atlassian uses TiDB<\/a> as the scalable backend to serve Forge SQL, providing each of Atlassian\u2019s 3rd party developers a shared database system to store their own customers\u2019 data. This raises several traditionally difficult SaaS backend engineering questions.<\/p>\n\n\n\n To provide each Forge customer their private data, each gets their own schema\/s. But that means potentially a whole lot of those, and potentially a whole lot of trouble. But TiDB is designed with this challenge in mind, as shown below.<\/p>\n\n\n To start, TiDB handles the initialization of these schemas by allowing parallel creation of databases and tables, drastically speeding up object creation at scale. <\/p>\n\n\n\n<\/span>Distributed SQL: A New Architectural Paradigm<\/span><\/h2>\n\n\n\n
<\/span>Inside TiDB: The Architecture of Scaling Throughput<\/span><\/h2>\n\n\n\n
Distributed Write Consistency Enabled By \u201cMulti-Raft\u201d<\/h3>\n\n\n\n
![\"How](\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXeXBjzbmLmQv3JWhHCmUAbrUilqratmNT9aWxWHfLAX9156EY0WmXIDAb_N8WP2ui3Nloeoaz9ncQNRRHpdRXkc3u2QhYJJfS5u8VqucL1Jvbfe899mN77fwJ-spQT3gDOaaqX8-w?key=nOBeO_7AoKYus5v1omPWzxiV\")
Distributed Transaction Atomicity Enabled By 2-Phase Commit<\/h3>\n\n\n\n
![\"A](\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXcMeXH9LcvgghGEb3KksZwztEd_oh0AF7uLzCEewtKVK1qHa13-Y6GRh3QbRIteqrLgBmgLqZMO_Etc95c_XE5AFNzzyWlvCyUIBmWvhSQCkXUaL78-HwCyA65yvnOU014wf9Dj1Q?key=nOBeO_7AoKYus5v1omPWzxiV\")
Transaction Contention Solved By MVCC<\/h3>\n\n\n\nRead Performance Enabled By a Compute Layer Designed For Distributed Data<\/h3>\n\n\n\n
Cost and Complexity of Scaling Optimized By Physical Architecture<\/h3>\n\n\n\n
<\/span>Parallelism in Practice: How Distributed SQL Powers Scalable Operations<\/span><\/h2>\n\n\n\n
\n
Flexibility Served By Fast, Online Schema Changes<\/h3>\n\n\n\n
Availability At Scale Supported By Dynamic Fault Tolerance<\/h3>\n\n\n\n
![\"A](\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXcnO6FW2wuU3-dzBBa7T7zcfiuuRjR24C0fN26azmF7Y-UpgFQg7IyMiEMgkflcY_MwrwzXzVz0EahT29Ij-Sw3j4lKy5_fb_Qo4lHHg5R5T_BH-1tFWmZcyTUH9D-YegFzh9J-Qw?key=nOBeO_7AoKYus5v1omPWzxiV\")
Agility Supported By Efficient Scale-Up\/Down<\/h3>\n\n\n\n
<\/span>Multi-Tenant Scaling: Simplifying SaaS Growth<\/span><\/h2>\n\n\n\n
How Atlassian Scaled 3 Million Tables with TiDB<\/h3>\n\n\n\n
![\"A](\"https:\/\/static.pingcap.com\/files\/2025\/08\/19152325\/Screenshot-2025-08-19-at-6.22.44-PM-1024x746.png\")
<\/figure>\n<\/div>\n\n\n