{"id":31060,"date":"2025-12-16T20:39:47","date_gmt":"2025-12-17T04:39:47","guid":{"rendered":"https:\/\/www.pingcap.com\/?p=31060"},"modified":"2025-12-16T20:41:01","modified_gmt":"2025-12-17T04:41:01","slug":"tidbx-origins-architecture","status":"publish","type":"post","link":"https:\/\/www.pingcap.com\/ko\/blog\/tidbx-origins-architecture\/","title":{"rendered":"The Making of TiDB X: Origins, Architecture, and What\u2019s to Come"},"content":{"rendered":"

When we unveiled TiDB X<\/a>, the new core engine for TiDB Cloud, at our recent TiDB SCaiLE annual event, the response was immediate and enthusiastic. Many people reached out afterward with technical and non-technical questions about where TiDB X came from, why we decided to build it, and how it connects to the future of TiDB Cloud<\/a>. Instead of replying privately to everyone, we decided to write down the story \u2014 not just the architecture, but the reasoning, the missteps, and the years of evolution behind this new engine.<\/p>\n\n\n\n

TiDB X did not appear suddenly. It grew out of years of building TiDB Cloud, hitting painful limits, challenging long-held assumptions, and eventually deciding that incremental thinking would not get us where the future is clearly heading. In many ways, TiDB X is a return to first principles: why we built TiDB, what customers truly value, and what scalability<\/a> really means in a world reshaped by AI and cloud-native development.<\/p>\n\n\n\n

<\/span>Revisiting the Core Question: Why Do People Choose TiDB?<\/span><\/h2>\n\n\n\n

For years, one of our biggest internal challenges was the architectural fragmentation between TiDB Cloud Starter and TiDB Cloud Dedicated. While they shared the TiDB name, they diverged significantly in technology, teams, and evolution. We tried to unify them more than once, without success.<\/p>\n\n\n\n

What ultimately realigned us was a simple question: why do users choose TiDB? Across countless conversations, the answer consistently came down to one word \u2014 scalability.<\/p>\n\n\n\n

Not just in data volume or throughput, but across every dimension that breaks databases in practice: indexes, tenants, tables, connections, metadata, and operational complexity. Systems don\u2019t fail at their strongest points, but at their weakest, and TiDB has proven uniquely balanced across all of them.<\/p>\n\n\n\n

One example came from \uc544\ud2c0\ub77c\uc2dc\uc548<\/a>, which needed to support millions of tables in a single database. The data size was modest, but the metadata demands were extreme. Competing systems failed at a few thousand tables; TiDB scaled to nearly ten million, winning the deployment. It reinforced a core truth we\u2019ve seen repeatedly: scalability is what TiDB does best \u2014 and it\u2019s never static.<\/strong><\/p>\n\n\n\n

<\/span>What Happens at 10\u00d7 or 100\u00d7 Scale?<\/span><\/h2>\n\n\n\n

We\u2019ve stretched the legacy TiDB architecture further than we once thought possible, from comfortable 100 TB clusters to stable 1+ PB clusters. But the natural next question is: What happens at 10 PB and beyond?<\/p>\n\n\n\n

The honest answer is that the traditional shared-nothing architecture<\/a>, built on local storage, simply cannot deliver the elasticity, cost profile, or operational dynamics required for that scale.<\/p>\n\n\n\n

To survive the next decade of workloads \u2014 especially workloads shaped by AI agents generating personalized SQL, exploding long-tail use cases, and a flood of dynamic applications \u2014 a database must fully embrace two cloud principles:<\/p>\n\n\n\n

    \n
  1. Elastic resources<\/strong>: true pay-as-you-go compute<\/li>\n\n\n\n
  2. Object storage<\/strong>: cheap, durable, massively parallel<\/li>\n<\/ol>\n\n\n\n

    If today\u2019s workloads seem manageable, history reminds us that we routinely underestimate the future. Moore\u2019s Law is a humbling reminder: our imagination rarely keeps pace with technological reality. And the last six months of AI progress alone have made this even more obvious.<\/p>\n\n\n\n

    When applications can be created in minutes \u2014 when data collection is trivial \u2014 and when each user may have a personalized workload generated by an AI agent, the combination of SQL patterns, table shapes, and query paths will stress every part of a database system.<\/p>\n\n\n\n

    The question then becomes: How do we design a database that preserves rapid innovation while guaranteeing long-term stability and safety?<\/strong><\/p>\n\n\n\n

    This question marks the true beginning of TiDB X.<\/p>\n\n\n\n

    <\/span>Origins of TiDB X: The Early Technical Seeds<\/span><\/h2>\n\n\n\n

    The earliest sign of TiDB X\u2019s direction dates back to an April 2021 talk I gave at a CCF conference, though the idea had been forming much earlier. Around 2017, Max Liu<\/a> started a small internal project to explore transaction optimizations and accelerate SQL-layer testing. At the same time, we were influenced by the WiscKey paper (FAST \u201916) and BadgerDB, Dgraph\u2019s Go implementation, which explored separating keys and values in LSM-Trees to reduce write amplification.<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    Figure 1:<\/strong><\/em> How TiDB X combines object storage, workload-aware scaling, and task isolation.<\/em><\/p>\n\n\n\n

    As Go enthusiasts, we experimented with a TiKV-compatible distributed KV store built on BadgerDB. It felt lightweight and flexible, though we later learned the hard way that this optimism was misplaced \u2014 TiDB X was eventually rewritten in Rust. Still, the prototype planted a critical idea: storage concerns could be separated more aggressively.<\/p>\n\n\n\n

    In parallel, Snowflake\u2019s compute\u2013storage separation was reshaping expectations for cloud-native data systems. While Snowflake targeted OLAP, it pushed us to ask a provocative question for OLTP: could SST files live on S3, with only caches and Raft logs kept locally? Others, like RockSet and the RocksDB team, were exploring similar directions, but within existing constraints. We wanted to go further \u2014 to build a fully serverless, cloud-native OLTP database.<\/p>\n\n\n\n

    <\/span>The Turning Point: 2022 and the Decision to Break From Incrementalism<\/span><\/h2>\n\n\n\n

    By 2022, we restarted the cloud-native engine project with clear, ambitious goals. They were bold enough that incremental engineering could not possibly deliver them:<\/p>\n\n\n\n