In this blog post, we’ll unravel how TiDB Dedicated achieves zero downtime for TiDB clusters during Kubernetes node upgrades (especially with EKS), ensuring uninterrupted service.<\/p>\n\n\n\n
<\/span>Challenges with Managed Kubernetes Services<\/strong> and Zero Downtime<\/span><\/h2>\n\n\n\nTiDB Dedicated serves as a foundational infrastructure where operational impacts must be minimized. For instance, latency should not fluctuate too much, and connections and QPS should always remain on. Such requirements present significant challenges in managing \ud2f0DB<\/a> on managed Kubernetes.<\/p>\n\n\n\nWhile Managed Kubernetes services like Amazon EKS or Google GKE simplify the management of the Kubernetes control plane, they also introduce challenges, particularly around the version support policy. EKS<\/a> \uadf8\ub9ac\uace0 GKE<\/a> only support each Kubernetes minor version for about 14 months after release. After the end of the support date, the vendor will initiate a control plane upgrade that involves both the control plane and the nodes. While control plane upgrades usually have little impact on TiDB clusters, node upgrades can be somewhat disruptive as they require node replacement. <\/p>\n\n\n\n<\/span>How TiDB Dedicated Approaches Zero<\/strong> Downtime<\/span><\/h2>\n\n\n\nEnsuring zero downtime during Kubernetes node upgrades is a multifaceted process. Here are the strategies TiDB Dedicated uses to tackle managed Kubernetes services challenges:<\/p>\n\n\n\n
Add an Extra TiDB Server Pod<\/strong><\/h3>\n\n\n\nTiDB Dedicated uses the advanced StatefulSet provided by TiDB Operator, the operation system for TiDB clusters on Kubernetes, to manage TiDB server pods. This setup facilitates rolling updates from the largest ordinal to the smallest, updating one pod at a time<\/strong>. However, this means the number of available TiDB server pods will always be less than the expected replicas during the rolling restart. Consequently, the availability and performance will be affected. Consider a worst-case scenario: if there’s only one TiDB server pod (tidb-0), a restart of tidb-0 will result in downtime as there are no other pods to handle queries. <\/p>\n\n\n\nTo mitigate this, TiDB Dedicated scales out an extra TiDB server before the upgrade to guarantee availability and performance and scales in it after the upgrade. This ensures that there are always at least as many TiDB server pods serving traffic as the desired number of replicas. The process is transparent to users, and TiDB Dedicated covers the cost.<\/p>\n\n\n\n![\"\"](\"https:\/\/static.pingcap.com\/files\/2023\/11\/21010113\/image-6-1024x480.png\")
<\/figure>\n\n\n\nFigure 1. Scale out an extra TiDB server pod<\/em><\/p>\n\n\n\nUse Blue-Green Upgrades<\/strong><\/h3>\n\n\n\nSurge and blue-green upgrades are two prevalent methods for upgrading Kubernetes nodes, each suitable for different circumstances. Surge upgrades are the default upgrade strategy for EKS and GKE. It uses a rolling method where nodes are upgraded individually in an undefined order<\/strong>. <\/p>\n\n\n\nThis works well enough for stateless applications. However, stateful applications, such as a distributed database like TiDB, usually involve extra operational steps before the process shuts down. During the planned maintenance of TiDB, if we instruct the PD and TiKV Raft group to evict the leader proactively without waiting for the Raft election timeout, the cluster can serve the client request steadily during the server restart. TiDB Operator simplifies this by automating the rolling restart of the cluster and handling the complex orchestration seamlessly. However, the TiDB Operator handles the rolling restart sequentially. Random pod and node termination during the EKS upgrade could be a big problem. Furthermore, the inability to roll back to the original version when issues arise on the new nodes adds to the challenge.<\/p>\n\n\n\n
Given the intricacies associated with surge upgrades, we choose the blue-green upgrades for TiDB Dedicated. This strategy entails the following steps:<\/p>\n\n\n\n
\n- Create a new set of desired nodes (“green” nodes).<\/li>\n\n\n\n
- Cordon the original nodes (“blue” nodes).<\/li>\n\n\n\n
- TiDB Operator triggers a rolling restart of the TiDB cluster. This facilitates a graceful migration of pods to “green” nodes.<\/li>\n\n\n\n
- Verify the TiDB cluster’s health after the “blue” nodes are drained.<\/li>\n\n\n\n
- Remove the “blue” nodes.<\/li>\n<\/ol>\n\n\n\n
This approach allows us to stop upgrading and revert to old nodes if any issues arise. It is also more resource-intensive as it doubles the nodes needed during the upgrade. Again, the fee is on TiDB Dedicated, so customers don’t have to worry about extra fees.<\/p>\n\n\n\n
Migrate to Self-Managed Node Groups<\/strong><\/h3>\n\n\n\nWe originally used managed node groups for TiDB clusters on AWS, which simplified many aspects of node maintenance. However, it did not align well with TiDB’s operational requirements. For example, the EKS-managed node groups provide a one-click node upgrades feature, but it is for surge upgrades and does not support blue-green upgrades. Furthermore, to comply with AWS’s policy, we must upgrade nodes annually, as the node version with managed node groups should either follow the control plane’s version or lag behind by one version.<\/p>\n\n\n\n
On the other hand, self-managed node groups allow node versions to be up to two minor versions older than the control plane. Given this flexibility, we migrated from managed node groups to self-managed ones during blue-green upgrades. <\/p>\n\n\n\n
Following this change, TiDB Dedicated can now upgrade the Kubernetes control plane version twice consecutively without the need to upgrade nodes. This significantly reduces the maintenance frequency for TiDB clusters caused by EKS upgrades and minimizes the impact on the customer side.<\/p>\n\n\n\n![\"\"](\"https:\/\/static.pingcap.com\/files\/2023\/11\/21010227\/image-7-1024x198.png\")
<\/figure>\n\n\n\nFigure 2.<\/em> Self-managed node groups<\/em><\/p>\n\n\n\n<\/span>Benefits of TiDB Dedicated\u2019s Zero-Downtime Upgrade Strategy<\/strong><\/span><\/h2>\n\n\n\nThrough rigorous testing under a spectrum of loading conditions\u2014from high to low\u2014we have evaluated the impact of upgrading TiDB nodes on performance and availability. The findings are encouraging:<\/p>\n\n\n\n
\n- Stability and performance: <\/strong>During the upgrade process, the system maintained a stable performance, with only minor fluctuations in key metrics. The Queries Per Second (QPS) experienced a negligible drop, and the query duration saw a slight increase, both within a tight margin of just 5%. The figure below illustrates the stability in QPS and query duration throughout the upgrade process.<\/li>\n<\/ul>\n\n\n\n
![\"\"](\"https:\/\/static.pingcap.com\/files\/2023\/11\/21010545\/image-8-1024x505.png\")
<\/figure>\n\n\n\nFigure 3. Stability and performance gains <\/em><\/p>\n\n\n\n\n- Uninterrupted service: <\/strong>The results underscore that our meticulous upgrade methodology yields virtually no observable impact on TiDB service performance or availability from the user\u2019s perspective.<\/li>\n\n\n\n
- Control over latency and throughput:<\/strong> Even during the upgrades of the underlying Kubernetes infrastructure, we can maintain stringent control over latency and throughput, ensuring a consistent and reliable service for our users.<\/li>\n<\/ul>\n\n\n\n
<\/span>Conclusion<\/strong><\/span><\/h2>\n\n\n\nIn conclusion, TiDB Dedicated’s strategic approach towards Kubernetes node upgrades significantly mitigates the operational challenges commonly associated with managed Kubernetes services. The meticulous employment of extra TiDB server pods, blue-green upgrades, and migration to self-managed node groups (for EKS) provides a robust framework that ensures zero downtime and consistent performance during upgrades.<\/p>\n\n\n\n
As customers, you can rely on the performance and reliability of TiDB Dedicated and focus on your applications without worrying about disruptive Kubernetes upgrades and extra costs.<\/p>","protected":false},"excerpt":{"rendered":"
A key aspect of Database as a Service (DBaaS) solutions is ensuring zero downtime, especially during system upgrades. Zero downtime is paramount as it guarantees uninterrupted service, a critical requirement for mission-critical applications running on the database. TiDB Dedicated is PingCAP’s fully-managed cloud Database-as-a-Service (DBaaS) platform on AWS and GCP. This DBaaS uses managed Kubernetes […]<\/p>\n","protected":false},"author":252,"featured_media":14711,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ub_ctt_via":"","footnotes":""},"categories":[13],"tags":[85,33,31,236,225],"class_list":["post-14688","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-product","tag-cloud-native","tag-kubernetes","tag-tidb-cloud","tag-tidb-dedicated","tag-zero-downtime-upgrade"],"acf":[],"featured_image_src":"https:\/\/static.pingcap.com\/files\/2023\/11\/22011429\/tidbcloud_node_upgrades_banner.jpeg","author_info":{"display_name":"Shuan Deng","author_link":"https:\/\/www.pingcap.com\/ko\/blog\/author\/dengshuan\/"},"yoast_head":"\n
Zero Downtime: How TiDB Performs During Kubernetes Upgrades<\/title>\n<meta name=\"description\" content=\"Find out how TiDB achieves zero downtime for TiDB clusters during Kubernetes node upgrades, ensuring uninterrupted service.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.pingcap.com\/ko\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/\" \/>\n<meta property=\"og:locale\" content=\"ko_KR\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Zero Downtime: How TiDB Performs During Kubernetes Upgrades\" \/>\n<meta property=\"og:description\" content=\"Find out how TiDB achieves zero downtime for TiDB clusters during Kubernetes node upgrades, ensuring uninterrupted service.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.pingcap.com\/ko\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/\" \/>\n<meta property=\"og:site_name\" content=\"TiDB\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/facebook.com\/pingcap2015\" \/>\n<meta property=\"article:published_time\" content=\"2023-11-22T09:17:21+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-03-18T13:57:20+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/static.pingcap.com\/files\/2023\/11\/22011639\/tidbcloud_node_upgrades_social.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1875\" \/>\n\t<meta property=\"og:image:height\" content=\"938\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Shuan Deng\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/static.pingcap.com\/files\/2023\/11\/22011639\/tidbcloud_node_upgrades_social.png\" \/>\n<meta name=\"twitter:creator\" content=\"@PingCAP\" \/>\n<meta name=\"twitter:site\" content=\"@PingCAP\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Shuan Deng\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6\ubd84\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/\"},\"author\":{\"name\":\"Shuan Deng\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/person\/cc74406d0af796d8d0262a3222621813\"},\"headline\":\"How TiDB Dedicated Achieves Zero Downtime During Kubernetes Node Upgrades\",\"datePublished\":\"2023-11-22T09:17:21+00:00\",\"dateModified\":\"2024-03-18T13:57:20+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/\"},\"wordCount\":1196,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.pingcap.com\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/static.pingcap.com\/files\/2023\/11\/22011429\/tidbcloud_node_upgrades_banner.jpeg\",\"keywords\":[\"Cloud-native\",\"Kubernetes\",\"TiDB Cloud\",\"TiDB Dedicated\",\"Zero-downtime Upgrade\"],\"articleSection\":[\"Product\"],\"inLanguage\":\"ko-KR\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/\",\"url\":\"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/\",\"name\":\"Zero Downtime: How TiDB Performs During Kubernetes Upgrades\",\"isPartOf\":{\"@id\":\"https:\/\/www.pingcap.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/static.pingcap.com\/files\/2023\/11\/22011429\/tidbcloud_node_upgrades_banner.jpeg\",\"datePublished\":\"2023-11-22T09:17:21+00:00\",\"dateModified\":\"2024-03-18T13:57:20+00:00\",\"description\":\"Find out how TiDB achieves zero downtime for TiDB clusters during Kubernetes node upgrades, ensuring uninterrupted service.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/#breadcrumb\"},\"inLanguage\":\"ko-KR\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/#primaryimage\",\"url\":\"https:\/\/static.pingcap.com\/files\/2023\/11\/22011429\/tidbcloud_node_upgrades_banner.jpeg\",\"contentUrl\":\"https:\/\/static.pingcap.com\/files\/2023\/11\/22011429\/tidbcloud_node_upgrades_banner.jpeg\",\"width\":1875,\"height\":625},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.pingcap.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How TiDB Dedicated Achieves Zero Downtime During Kubernetes Node Upgrades\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.pingcap.com\/#website\",\"url\":\"https:\/\/www.pingcap.com\/\",\"name\":\"TiDB\",\"description\":\"TiDB | SQL at Scale\",\"publisher\":{\"@id\":\"https:\/\/www.pingcap.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.pingcap.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"ko-KR\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.pingcap.com\/#organization\",\"name\":\"PingCAP\",\"url\":\"https:\/\/www.pingcap.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png\",\"contentUrl\":\"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png\",\"width\":811,\"height\":232,\"caption\":\"PingCAP\"},\"image\":{\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/facebook.com\/pingcap2015\",\"https:\/\/x.com\/PingCAP\",\"https:\/\/linkedin.com\/company\/pingcap\",\"https:\/\/youtube.com\/channel\/UCuq4puT32DzHKT5rU1IZpIA\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/person\/cc74406d0af796d8d0262a3222621813\",\"name\":\"Shuan Deng\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/static.pingcap.com\/files\/2023\/11\/22010434\/dengshuan-150x150.png\",\"contentUrl\":\"https:\/\/static.pingcap.com\/files\/2023\/11\/22010434\/dengshuan-150x150.png\",\"caption\":\"Shuan Deng\"},\"url\":\"https:\/\/www.pingcap.com\/ko\/blog\/author\/dengshuan\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Zero Downtime: How TiDB Performs During Kubernetes Upgrades","description":"Find out how TiDB achieves zero downtime for TiDB clusters during Kubernetes node upgrades, ensuring uninterrupted service.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.pingcap.com\/ko\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/","og_locale":"ko_KR","og_type":"article","og_title":"Zero Downtime: How TiDB Performs During Kubernetes Upgrades","og_description":"Find out how TiDB achieves zero downtime for TiDB clusters during Kubernetes node upgrades, ensuring uninterrupted service.","og_url":"https:\/\/www.pingcap.com\/ko\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/","og_site_name":"TiDB","article_publisher":"https:\/\/facebook.com\/pingcap2015","article_published_time":"2023-11-22T09:17:21+00:00","article_modified_time":"2024-03-18T13:57:20+00:00","og_image":[{"width":1875,"height":938,"url":"https:\/\/static.pingcap.com\/files\/2023\/11\/22011639\/tidbcloud_node_upgrades_social.png","type":"image\/png"}],"author":"Shuan Deng","twitter_card":"summary_large_image","twitter_image":"https:\/\/static.pingcap.com\/files\/2023\/11\/22011639\/tidbcloud_node_upgrades_social.png","twitter_creator":"@PingCAP","twitter_site":"@PingCAP","twitter_misc":{"Written by":"Shuan Deng","Est. reading time":"6\ubd84"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/#article","isPartOf":{"@id":"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/"},"author":{"name":"Shuan Deng","@id":"https:\/\/www.pingcap.com\/#\/schema\/person\/cc74406d0af796d8d0262a3222621813"},"headline":"How TiDB Dedicated Achieves Zero Downtime During Kubernetes Node Upgrades","datePublished":"2023-11-22T09:17:21+00:00","dateModified":"2024-03-18T13:57:20+00:00","mainEntityOfPage":{"@id":"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/"},"wordCount":1196,"commentCount":0,"publisher":{"@id":"https:\/\/www.pingcap.com\/#organization"},"image":{"@id":"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/#primaryimage"},"thumbnailUrl":"https:\/\/static.pingcap.com\/files\/2023\/11\/22011429\/tidbcloud_node_upgrades_banner.jpeg","keywords":["Cloud-native","Kubernetes","TiDB Cloud","TiDB Dedicated","Zero-downtime Upgrade"],"articleSection":["Product"],"inLanguage":"ko-KR","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/","url":"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/","name":"Zero Downtime: How TiDB Performs During Kubernetes Upgrades","isPartOf":{"@id":"https:\/\/www.pingcap.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/#primaryimage"},"image":{"@id":"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/#primaryimage"},"thumbnailUrl":"https:\/\/static.pingcap.com\/files\/2023\/11\/22011429\/tidbcloud_node_upgrades_banner.jpeg","datePublished":"2023-11-22T09:17:21+00:00","dateModified":"2024-03-18T13:57:20+00:00","description":"Find out how TiDB achieves zero downtime for TiDB clusters during Kubernetes node upgrades, ensuring uninterrupted service.","breadcrumb":{"@id":"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/#breadcrumb"},"inLanguage":"ko-KR","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/"]}]},{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/#primaryimage","url":"https:\/\/static.pingcap.com\/files\/2023\/11\/22011429\/tidbcloud_node_upgrades_banner.jpeg","contentUrl":"https:\/\/static.pingcap.com\/files\/2023\/11\/22011429\/tidbcloud_node_upgrades_banner.jpeg","width":1875,"height":625},{"@type":"BreadcrumbList","@id":"https:\/\/www.pingcap.com\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.pingcap.com\/"},{"@type":"ListItem","position":2,"name":"How TiDB Dedicated Achieves Zero Downtime During Kubernetes Node Upgrades"}]},{"@type":"WebSite","@id":"https:\/\/www.pingcap.com\/#website","url":"https:\/\/www.pingcap.com\/","name":"\ud2f0DB","description":"TiDB | SQL at Scale","publisher":{"@id":"https:\/\/www.pingcap.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.pingcap.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"ko-KR"},{"@type":"Organization","@id":"https:\/\/www.pingcap.com\/#organization","name":"PingCAP","url":"https:\/\/www.pingcap.com\/","logo":{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/","url":"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png","contentUrl":"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png","width":811,"height":232,"caption":"PingCAP"},"image":{"@id":"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/facebook.com\/pingcap2015","https:\/\/x.com\/PingCAP","https:\/\/linkedin.com\/company\/pingcap","https:\/\/youtube.com\/channel\/UCuq4puT32DzHKT5rU1IZpIA"]},{"@type":"Person","@id":"https:\/\/www.pingcap.com\/#\/schema\/person\/cc74406d0af796d8d0262a3222621813","name":"Shuan Deng","image":{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/www.pingcap.com\/#\/schema\/person\/image\/","url":"https:\/\/static.pingcap.com\/files\/2023\/11\/22010434\/dengshuan-150x150.png","contentUrl":"https:\/\/static.pingcap.com\/files\/2023\/11\/22010434\/dengshuan-150x150.png","caption":"Shuan Deng"},"url":"https:\/\/www.pingcap.com\/ko\/blog\/author\/dengshuan\/"}]}},"grav_blocks":false,"card_markup":"<a class=\"card-resource bg-white\" href=\"https:\/\/www.pingcap.com\/ko\/blog\/how-tidb-achieves-zero-downtime-during-kubernetes-node-upgrades\/\"><div class=\"card-resource__image-container\"><img class=\"card-resource__image\" alt=\"tidbcloud_node_upgrades_banner\" src=\"https:\/\/static.pingcap.com\/files\/2023\/11\/22011429\/tidbcloud_node_upgrades_banner.jpeg\" loading=\"lazy\" width=1875 height=625 \/><\/div><div class=\"card-resource__content-container\"><div class=\"card-resource__content-head\"><div class=\"card-resource__category\">Product<\/div><\/div><h5 class=\"card-resource__title\">How TiDB Dedicated Achieves Zero Downtime During Kubernetes Node Upgrades<\/h5><\/div><\/a>","_links":{"self":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts\/14688","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/users\/252"}],"replies":[{"embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/comments?post=14688"}],"version-history":[{"count":5,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts\/14688\/revisions"}],"predecessor-version":[{"id":16052,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts\/14688\/revisions\/16052"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/media\/14711"}],"wp:attachment":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/media?parent=14688"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/categories?post=14688"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/tags?post=14688"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
While Managed Kubernetes services like Amazon EKS or Google GKE simplify the management of the Kubernetes control plane, they also introduce challenges, particularly around the version support policy. EKS<\/a> \uadf8\ub9ac\uace0 GKE<\/a> only support each Kubernetes minor version for about 14 months after release. After the end of the support date, the vendor will initiate a control plane upgrade that involves both the control plane and the nodes. While control plane upgrades usually have little impact on TiDB clusters, node upgrades can be somewhat disruptive as they require node replacement. <\/p>\n\n\n\n Ensuring zero downtime during Kubernetes node upgrades is a multifaceted process. Here are the strategies TiDB Dedicated uses to tackle managed Kubernetes services challenges:<\/p>\n\n\n\n TiDB Dedicated uses the advanced StatefulSet provided by TiDB Operator, the operation system for TiDB clusters on Kubernetes, to manage TiDB server pods. This setup facilitates rolling updates from the largest ordinal to the smallest, updating one pod at a time<\/strong>. However, this means the number of available TiDB server pods will always be less than the expected replicas during the rolling restart. Consequently, the availability and performance will be affected. Consider a worst-case scenario: if there’s only one TiDB server pod (tidb-0), a restart of tidb-0 will result in downtime as there are no other pods to handle queries. <\/p>\n\n\n\n To mitigate this, TiDB Dedicated scales out an extra TiDB server before the upgrade to guarantee availability and performance and scales in it after the upgrade. This ensures that there are always at least as many TiDB server pods serving traffic as the desired number of replicas. The process is transparent to users, and TiDB Dedicated covers the cost.<\/p>\n\n\n\n Figure 1. Scale out an extra TiDB server pod<\/em><\/p>\n\n\n\n Surge and blue-green upgrades are two prevalent methods for upgrading Kubernetes nodes, each suitable for different circumstances. Surge upgrades are the default upgrade strategy for EKS and GKE. It uses a rolling method where nodes are upgraded individually in an undefined order<\/strong>. <\/p>\n\n\n\n This works well enough for stateless applications. However, stateful applications, such as a distributed database like TiDB, usually involve extra operational steps before the process shuts down. During the planned maintenance of TiDB, if we instruct the PD and TiKV Raft group to evict the leader proactively without waiting for the Raft election timeout, the cluster can serve the client request steadily during the server restart. TiDB Operator simplifies this by automating the rolling restart of the cluster and handling the complex orchestration seamlessly. However, the TiDB Operator handles the rolling restart sequentially. Random pod and node termination during the EKS upgrade could be a big problem. Furthermore, the inability to roll back to the original version when issues arise on the new nodes adds to the challenge.<\/p>\n\n\n\n Given the intricacies associated with surge upgrades, we choose the blue-green upgrades for TiDB Dedicated. This strategy entails the following steps:<\/p>\n\n\n\n This approach allows us to stop upgrading and revert to old nodes if any issues arise. It is also more resource-intensive as it doubles the nodes needed during the upgrade. Again, the fee is on TiDB Dedicated, so customers don’t have to worry about extra fees.<\/p>\n\n\n\n We originally used managed node groups for TiDB clusters on AWS, which simplified many aspects of node maintenance. However, it did not align well with TiDB’s operational requirements. For example, the EKS-managed node groups provide a one-click node upgrades feature, but it is for surge upgrades and does not support blue-green upgrades. Furthermore, to comply with AWS’s policy, we must upgrade nodes annually, as the node version with managed node groups should either follow the control plane’s version or lag behind by one version.<\/p>\n\n\n\n On the other hand, self-managed node groups allow node versions to be up to two minor versions older than the control plane. Given this flexibility, we migrated from managed node groups to self-managed ones during blue-green upgrades. <\/p>\n\n\n\n Following this change, TiDB Dedicated can now upgrade the Kubernetes control plane version twice consecutively without the need to upgrade nodes. This significantly reduces the maintenance frequency for TiDB clusters caused by EKS upgrades and minimizes the impact on the customer side.<\/p>\n\n\n\n Figure 2.<\/em> Self-managed node groups<\/em><\/p>\n\n\n\n Through rigorous testing under a spectrum of loading conditions\u2014from high to low\u2014we have evaluated the impact of upgrading TiDB nodes on performance and availability. The findings are encouraging:<\/p>\n\n\n\n Figure 3. Stability and performance gains <\/em><\/p>\n\n\n\n In conclusion, TiDB Dedicated’s strategic approach towards Kubernetes node upgrades significantly mitigates the operational challenges commonly associated with managed Kubernetes services. The meticulous employment of extra TiDB server pods, blue-green upgrades, and migration to self-managed node groups (for EKS) provides a robust framework that ensures zero downtime and consistent performance during upgrades.<\/p>\n\n\n\n As customers, you can rely on the performance and reliability of TiDB Dedicated and focus on your applications without worrying about disruptive Kubernetes upgrades and extra costs.<\/p>","protected":false},"excerpt":{"rendered":" A key aspect of Database as a Service (DBaaS) solutions is ensuring zero downtime, especially during system upgrades. Zero downtime is paramount as it guarantees uninterrupted service, a critical requirement for mission-critical applications running on the database. TiDB Dedicated is PingCAP’s fully-managed cloud Database-as-a-Service (DBaaS) platform on AWS and GCP. This DBaaS uses managed Kubernetes […]<\/p>\n","protected":false},"author":252,"featured_media":14711,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ub_ctt_via":"","footnotes":""},"categories":[13],"tags":[85,33,31,236,225],"class_list":["post-14688","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-product","tag-cloud-native","tag-kubernetes","tag-tidb-cloud","tag-tidb-dedicated","tag-zero-downtime-upgrade"],"acf":[],"featured_image_src":"https:\/\/static.pingcap.com\/files\/2023\/11\/22011429\/tidbcloud_node_upgrades_banner.jpeg","author_info":{"display_name":"Shuan Deng","author_link":"https:\/\/www.pingcap.com\/ko\/blog\/author\/dengshuan\/"},"yoast_head":"\n<\/span>How TiDB Dedicated Approaches Zero<\/strong> Downtime<\/span><\/h2>\n\n\n\n
Add an Extra TiDB Server Pod<\/strong><\/h3>\n\n\n\n
<\/figure>\n\n\n\nUse Blue-Green Upgrades<\/strong><\/h3>\n\n\n\n
\n
Migrate to Self-Managed Node Groups<\/strong><\/h3>\n\n\n\n
<\/figure>\n\n\n\n<\/span>Benefits of TiDB Dedicated\u2019s Zero-Downtime Upgrade Strategy<\/strong><\/span><\/h2>\n\n\n\n
\n
<\/figure>\n\n\n\n\n
<\/span>Conclusion<\/strong><\/span><\/h2>\n\n\n\n