{"id":13966,"date":"2025-11-13T05:20:04","date_gmt":"2025-11-13T13:20:04","guid":{"rendered":"https:\/\/www.pingcap.com\/?p=13966"},"modified":"2025-11-19T07:36:07","modified_gmt":"2025-11-19T15:36:07","slug":"achieving-zero-downtime-upgrades-tidb","status":"publish","type":"post","link":"https:\/\/www.pingcap.com\/ko\/blog\/achieving-zero-downtime-upgrades-tidb\/","title":{"rendered":"Zero-Downtime Upgrades: How TiDB Powers Always-On Databases"},"content":{"rendered":"

In the vast landscape of databases, ensuring zero-downtime upgrades and operation continuity remains a challenge. Due to inherent design limitations, traditional databases often introduce significant downtime during upgrades – a challenge that can spell operational chaos for businesses reliant on real-time data access. <\/p>\n\n\n\n

Enter TiDB, a cutting-edge distributed SQL database that offers a solution to overcome this challenge. Built on a robust, cloud-native<\/a>, and loosely coupled architecture, TiDB introduces online rolling upgrades – a feature that enables zero-downtime upgrades with uninterrupted operations.<\/p>\n\n\n\n

In this post, we will explore the unique and easy-to-use upgrading mechanisms of TiDB with a hands-on demonstration. <\/p>\n\n\n\n

<\/span>Why Zero Downtime Matters in Production Environments<\/span><\/h2>\n\n\n\n

When production goes dark, the meter spins fast. Recent industry surveys show the average cost of an hour of IT downtime now exceeds $300,000 for most mid-to-large enterprises<\/a>, and a significant share report $1\u20135 million per hour depending on the workload and industry. <\/p>\n\n\n\n

Uptime Institute\u2019s outage analyses reinforce the trend: major incidents are costly<\/a> and increasingly scrutinized by regulators and boards, which raises the bar for resilience and change safety. Even conservative calculators place typical downtime between $100k and $540k per hour<\/a>, which aligns with many teams\u2019 internal incident postmortems. In short, every planned maintenance window you eliminate and every risky schema change you can do online directly protects revenue, reputation, and SLOs.<\/p>\n\n\n\n

Industry<\/th>Typical Range (USD\/hr)<\/th>Notes<\/th><\/tr><\/thead>
Financial Services<\/td>$300,000 \u2013 $5,000,000+<\/td>Large enterprises frequently report \u2265$300k\/hr; high-stakes incidents can exceed $1\u20135M\/hr.<\/td><\/tr>
E-commerce \/ Retail<\/td>$140,000 \u2013 $1,000,000<\/td>Common \u201cper minute\u201d averages translate to ~$140k\u2013$540k+\/hr, with peaks higher for major events<\/a>.<\/td><\/tr>
SaaS \/ Cloud<\/td>$300,000 \u2013 $1,000,000+<\/td>Mid-to-large enterprises commonly cite \u2265$300k\/hr; severe incidents can cross $1M.<\/td><\/tr>
Manufacturing<\/td>$260,000 \u2013 $1,360,000+<\/td>Benchmarks cluster around ~$260k\/hr; recent reports show averages >$1.3M\/hr in some regions<\/a>. <\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

What this means for zero-downtime upgrades in TiDB:<\/strong> If your environment runs dozens of services across shared database clusters, the difference between online DDL\/rolling upgrades and even a brief maintenance window can be six figures per incident. Investing in automation for online schema evolution, rolling restarts, and version changes is not a \u201cnice to have.\u201d It is a material cost saver that keeps roadmaps moving on schedule.<\/p>\n\n\n\n

<\/span>Challenges of Traditional Database Upgrades<\/span><\/h2>\n\n\n\n

Legacy database platforms still assume stop-and-wait maintenance windows, which clashes with 24\u00d77 SaaS and fintech realities. Teams struggle with true online schema change: blocking DDL, long locks, and table copies that spike IO and trigger timeouts. To dodge outages they bolt on brittle dual-writes, triggers, or shadow tables, adding risk while slowing delivery. Upgrades also collide with versioning conflicts across ORMs, client drivers, and SQL dialects; planner differences, reserved keywords, and collation quirks cause regressions and rollbacks.<\/p>\n\n\n\n

Without first-class rolling updates, many stacks require cluster-wide restarts or read-only windows, producing scheduled downtime, cache thrash, and retry storms. These stop-and-wait techniques may work for small fleets, but they collapse under multi-region, multi-tenant load, amplifying queue backlogs and replica lag. The takeaway: If your upgrade path depends on blocking DDL, coordinated app outages, or all-at-once node restarts, you are carrying avoidable risk\u2014precisely the gap zero-downtime approaches like TiDB are built to close.<\/p>\n\n\n\n

<\/span>TiDB\u2019s Distributed SQL Architecture for Zero Downtime<\/span><\/h2>\n\n\n\n

Traditional databases often use “stop-and-wait” techniques, freezing all operations for the time-consuming upgrade process. In contrast, TiDB uses an online rolling upgrade strategy. This approach ensures zero-downtime upgrades by upgrading components in a specific sequence: <\/p>\n\n\n\n

    \n
  1. Placement Driver (PD) servers <\/li>\n\n\n\n
  2. The TiKV servers<\/li>\n\n\n\n
  3. The TiDB servers<\/li>\n<\/ol>\n\n\n\n

    Each server upgrades one at a time, ensuring that other servers seamlessly handle the incoming load, resulting in a smooth and uninterrupted upgrade experience.<\/p>\n\n\n\n

    Here’s a closer look at how each key component contributes to the process:<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    Figure 1. Auto-upgrading architecture<\/em><\/p>\n\n\n\n

    Component<\/strong><\/td>Definition<\/strong><\/td>Auto-Upgrading Mechanism<\/strong><\/td><\/tr>
    Placement Driver (PD) Servers<\/td>PD servers act as the cluster manager, managing metadata, scheduling, and load balancing.<\/td>During the upgrade, each PD server is upgraded one at a time. If a PD is the current leader, leadership is transferred first, causing only a brief pause in active TSO requests without affecting ongoing transactions or client connections.<\/td><\/tr>
    TiKV Servers<\/td>TiKV is the distributed transactional key-value storage layer, responsible for data storage and retrieval.<\/td>TiKV servers are upgraded one at a time. Before upgrading, the leader for each Region is transferred to another TiKV server, thereby ensuring that ongoing operations are not disrupted.<\/td><\/tr>
    TiDB Servers (Facilitated by TiProxy)<\/td>TiDB is the stateless SQL server responsible for SQL query processing, maintaining sessions, and handling transactions.<\/td>TiProxy assists in the smooth upgrading of TiDB servers by sitting between the network load balancer and the SQL Layer. It migrates client sessions to other TiDB servers during the upgrade, thereby ensuring zero disruption to client applications.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

    This upgrade mechanism ensures that at each stage of the upgrade, the client experiences zero downtime and continues to interact with the database as if nothing has changed. In TiDB’s world, upgrades are not an interruption but a seamless transition. To learn more about how the upgrade mechanism works, see Maintaining Database Connectivity in Serverless Infrastructure with TiProxy<\/a>.<\/p>\n\n\n\n

    Cloud-Native and Horizontally Scalable by Design<\/h3>\n\n\n\n

    TiDB\u2019s resilience starts with its architecture: a cloud-native<\/a>, shared-nothing design that cleanly separates stateless SQL compute (TiDB) from stateful storage (TiKV) with Raft replication and optional TiFlash for analytics. That separation means you can roll out changes, add capacity, or replace nodes one at a time without taking the service down. When demand spikes or a node fails, the cluster rebalances leaders and regions automatically, so throughput and availability stay steady while upgrades proceed in the background.<\/p>\n\n\n\n

    Here’s why this enables zero-downtime change:<\/p>\n\n\n\n