{"id":478,"date":"2021-08-10T00:00:00","date_gmt":"2021-08-10T00:00:00","guid":{"rendered":"https:\/\/en.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/"},"modified":"2024-08-22T08:25:14","modified_gmt":"2024-08-22T15:25:14","slug":"how-chaos-mesh-helps-apache-apisix-improve-system-stability","status":"publish","type":"post","link":"https:\/\/www.pingcap.com\/ko\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/","title":{"rendered":"How Chaos Mesh Helps Apache APISIX Improve System Stability"},"content":{"rendered":"<p>Apache APISIX is a cloud-native, high-performance, scaling microservices API gateway. It is one of the Apache Software Foundation&#8217;s top-level projects and serves hundreds of companies around the world, processing their mission-critical traffic, including finance, the Internet, manufacturing, retail, and operators. Our customers include NASA, the European Union&#8217;s digital factory, China Mobile, and Tencent.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"2470\" height=\"1174\" src=\"https:\/\/static.pingcap.com\/files\/2024\/05\/17225800\/apache-apisix-architecture.jpg\" alt=\"Apache APISIX architecture\" class=\"wp-image-16856\" srcset=\"https:\/\/static.pingcap.com\/files\/2024\/05\/17225800\/apache-apisix-architecture.jpg 2470w, https:\/\/static.pingcap.com\/files\/2024\/05\/17225800\/apache-apisix-architecture-300x143.jpg 300w, https:\/\/static.pingcap.com\/files\/2024\/05\/17225800\/apache-apisix-architecture-1024x487.jpg 1024w, https:\/\/static.pingcap.com\/files\/2024\/05\/17225800\/apache-apisix-architecture-768x365.jpg 768w, https:\/\/static.pingcap.com\/files\/2024\/05\/17225800\/apache-apisix-architecture-1536x730.jpg 1536w, https:\/\/static.pingcap.com\/files\/2024\/05\/17225800\/apache-apisix-architecture-2048x973.jpg 2048w, https:\/\/static.pingcap.com\/files\/2024\/05\/17225800\/apache-apisix-architecture-1440x684.jpg 1440w\" sizes=\"auto, (max-width: 2470px) 100vw, 2470px\" \/><figcaption class=\"wp-element-caption\">Apache APISIX architecture<\/figcaption><\/figure>\n<\/div>\n\n\n<p>As our community grows, Apache APISIX&#8217;s features more frequently interact with external components, making our system more complex and increasing the possibility of errors. To identify potential system failures and build confidence in the production environment, we introduced the concept of Chaos Engineering.<\/p>\n\n\n\n<p>In this post, we&#8217;ll share how we use <a href=\"https:\/\/chaos-mesh.org\/docs\/\">Chaos Mesh<\/a> to improve our system stability.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Our_pain_points\"><\/span>Our pain points<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Apache APISIX processes tens of billions of requests a day. At that volume level, our users have noticed a couple of issues:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Scenario #1:<\/strong> In Apache APISIX&#8217;s configuration center, when unexpectedly high network latency occurs between etcd and Apache APISIX, can Apache APISIX still filter and forward traffic normally?<\/li>\n\n\n\n<li><strong>Scenario #2:<\/strong> When a node in the etcd cluster fails and the cluster can still run normally, an error is reported for the node&#8217;s interaction with the Apache APISIX admin API.<\/li>\n<\/ul>\n\n\n\n<p>Although Apache APISIX has covered many scenarios through unit, end-to-end (E2E), and fuzz tests in continuous integration (CI), it has not covered the interaction scenario with external components. If the system behaves abnormally, for example, if the network jitters, a hard disk fails, or a process is killed, can Apache APISIX give appropriate error messages? Can it keep running or restore itself to normal operation?<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Why_we_chose_Chaos_Mesh\"><\/span>Why we chose Chaos Mesh<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>To test these user scenarios and to discover similar problems before our product goes into production, our community decided to use Chaos Mesh for chaos testing.<\/p>\n\n\n\n<p>Chaos Mesh is a cloud-native Chaos Engineering platform that features all-around fault injection methods for complex systems on Kubernetes, covering faults in Pod, the network, file system, and even the kernel. It helps users find weaknesses in the system and ensures that the system can resist out-of-control situations in the production environment.<\/p>\n\n\n\n<p>Like Apache APISIX, Chaos Mesh has an active open source community. We know that an active community can ensure stable software use and rapid iteration. This makes Chaos Mesh more attractive.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"How_we_use_Chaos_Mesh_in_APISIX\"><\/span>How we use Chaos Mesh in APISIX<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Chaos Engineering has grown beyond simple fault injection and now forms a complete methodology. To create a chaos experiment, we determined what the normal operation or &#8220;steady state&#8221; of our application should be. We then introduced potential problems to see how the system responded. If the problems knocked the application out of its steady state, we fixed them.<\/p>\n\n\n\n<p>Now, we&#8217;ll take the two scenarios we mentioned to show you how we use Chaos Mesh in Apache APISIX.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Scenario #1<\/h3>\n\n\n\n<p>We deployed a Chaos Engineering experiment using the following steps:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>We found metrics to measure whether Apache APISIX is running normally. In the test, the most important method is to use Grafana to monitor the Apache APISIX&#8217;s running metrics. We extracted data from Prometheus in CI for comparison. Here, we used the routing and forwarding requests per second (RPS) and etcd connectivity as evaluation metrics. We analyzed the log. For Apache APISIX, we checked Nginx&#8217;s error log to determine whether there was an error and whether the error was in line with our expectations.<\/li>\n\n\n\n<li>We performed a test in the control group. We found that both <code>create route<\/code> \uadf8\ub9ac\uace0 <code>access route<\/code> were successful, and we could connect to etcd. We recorded the RPS.<\/li>\n\n\n\n<li>We used network chaos to add a five second network latency and then retested. This time, <code>set route<\/code> failed, <code>get route<\/code> succeeded, etcd could be connected to, and RPS had no significant change compared to the previous experiment. The experiment met our expectations.<\/li>\n<\/ol>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"2000\" height=\"1169\" src=\"https:\/\/en.pingcap.com\/wp-content\/uploads\/2021\/08\/high-network-latency-between-etcd-and-apache-apisix.jpg\" alt=\"High network latency occurs between etcd and Apache APISIX\" class=\"wp-image-479\" srcset=\"https:\/\/static.pingcap.com\/files\/2021\/08\/high-network-latency-between-etcd-and-apache-apisix.jpg 2000w, https:\/\/static.pingcap.com\/files\/2021\/08\/high-network-latency-between-etcd-and-apache-apisix-300x175.jpg 300w, https:\/\/static.pingcap.com\/files\/2021\/08\/high-network-latency-between-etcd-and-apache-apisix-1024x599.jpg 1024w, https:\/\/static.pingcap.com\/files\/2021\/08\/high-network-latency-between-etcd-and-apache-apisix-768x449.jpg 768w, https:\/\/static.pingcap.com\/files\/2021\/08\/high-network-latency-between-etcd-and-apache-apisix-1536x898.jpg 1536w, https:\/\/static.pingcap.com\/files\/2021\/08\/high-network-latency-between-etcd-and-apache-apisix-1440x842.jpg 1440w\" sizes=\"auto, (max-width: 2000px) 100vw, 2000px\" \/><\/figure>\n\n\n\n<div class=\"caption-center\">High network latency occurs between etcd and Apache APISIX<\/div>\n\n\n\n<h3 class=\"wp-block-heading\">Scenario #2<\/h3>\n\n\n\n<p>After we conducted the same experiment as above in the control group, we introduced pod-kill chaos and reproduced the expected error. When we randomly deleted a small number of etcd nodes in the cluster, sometimes APISIX could connect to etcd and sometimes not, and the log printed a large number of connection rejection errors.<\/p>\n\n\n\n<p>When we deleted the first or third node in the etcd endpoint list, the <code>set route<\/code> returned a result normally. However, when we deleted the second node in the list, the <code>set route<\/code> returned the error &#8220;connection refused.&#8221;<\/p>\n\n\n\n<p>Our troubleshooting revealed that the etcd Lua API used by Apache APISIX selected the endpoint sequentially, not randomly. Therefore, when we created an etcd client, we bound to only one etcd endpoint. This led to continuous failure.<\/p>\n\n\n\n<p>After we fixed this problem, we added a health check to the etcd Lua API to ensure that a large number of requests would not be sent to the disconnected etcd node. To avoid flooding the log with errors, we added a fallback mechanism when the etcd cluster was completely disconnected.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"2000\" height=\"1055\" src=\"https:\/\/en.pingcap.com\/wp-content\/uploads\/2021\/08\/error-reported-from-etcd-node-interaction.jpg\" alt=\"An error is reported\" class=\"wp-image-480\" srcset=\"https:\/\/static.pingcap.com\/files\/2021\/08\/error-reported-from-etcd-node-interaction.jpg 2000w, https:\/\/static.pingcap.com\/files\/2021\/08\/error-reported-from-etcd-node-interaction-300x158.jpg 300w, https:\/\/static.pingcap.com\/files\/2021\/08\/error-reported-from-etcd-node-interaction-1024x540.jpg 1024w, https:\/\/static.pingcap.com\/files\/2021\/08\/error-reported-from-etcd-node-interaction-768x405.jpg 768w, https:\/\/static.pingcap.com\/files\/2021\/08\/error-reported-from-etcd-node-interaction-1536x810.jpg 1536w, https:\/\/static.pingcap.com\/files\/2021\/08\/error-reported-from-etcd-node-interaction-1440x760.jpg 1440w\" sizes=\"auto, (max-width: 2000px) 100vw, 2000px\" \/><\/figure>\n\n\n\n<div class=\"caption-center\">An error is reported from one etcd node&#8217;s interaction with the Apache APISIX admin API<\/div>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Our_future_plans\"><\/span>Our future plans<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Run a chaos test in E2E simulation scenarios<\/h3>\n\n\n\n<p>In Apache APISIX, we manually identify system weaknesses for testing and repair. As in the open source community, we test in CI, so we don&#8217;t need to worry about the impact of Chaos Engineering&#8217;s failure radius on the production environment. But the test cannot cover complicated and comprehensive application scenarios in the production environment.<\/p>\n\n\n\n<p>To cover more scenarios, the community plans to use the existing E2E test to simulate more complete scenarios and conduct chaos tests that are more random and cover a larger range.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Add chaos tests to more Apache APISIX projects<\/h3>\n\n\n\n<p>In addition to finding more vulnerabilities for Apache APISIX, the community plans to add chaos tests to more projects such as Apache APISIX Dashboard and Apache APISIX Ingress Controller.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Add features to Chaos Mesh<\/h3>\n\n\n\n<p>When we deployed Chaos Mesh, some features were temporarily unsupported. For example, we couldn&#8217;t select a service as a network latency target or specify container port injection as network chaos. In the future, the Apache APISIX community will assist Chaos Mesh to add related features.<\/p>\n\n\n\n<p>You&#8217;re welcome to contribute to the <a href=\"https:\/\/github.com\/apache\/apisix\">Apache APISIX project<\/a> on GitHub. If you are interested in Chaos Mesh and would like to improve it, join its <a href=\"https:\/\/slack.cncf.io\/\">Slack channel<\/a> <span style=\"text-decoration: underline;\">(#project-chaos-mesh) <\/span>or submit your pull requests or issues to its <a href=\"https:\/\/github.com\/chaos-mesh\/chaos-mesh\">GitHub repository<\/a>.<\/p>","protected":false},"excerpt":{"rendered":"<p>This post describes how Apache APISIX uses Chaos Mesh to improve their system stability.<\/p>","protected":false},"author":57,"featured_media":482,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"ub_ctt_via":"","footnotes":""},"categories":[18],"tags":[21],"class_list":["post-478","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-community","tag-chaos-engineering"],"acf":[],"featured_image_src":"https:\/\/static.pingcap.com\/files\/2021\/08\/chaos-mesh-helps-apache-apisix-improve-system-stability-scaled.jpg","author_info":{"display_name":"Shuyang Wu","author_link":"https:\/\/www.pingcap.com\/ko\/blog\/author\/shuyang-wu\/"},"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.9 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>How Chaos Mesh Helps Apache APISIX Improve System Stability | TiDB<\/title>\n<meta name=\"description\" content=\"In this post, we&#039;ll share how we use Chaos Mesh to improve the stability of Apache APISIX, a scaling microservices API gateway.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.pingcap.com\/ko\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/\" \/>\n<meta property=\"og:locale\" content=\"ko_KR\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How Chaos Mesh Helps Apache APISIX Improve System Stability | TiDB\" \/>\n<meta property=\"og:description\" content=\"In this post, we&#039;ll share how we use Chaos Mesh to improve the stability of Apache APISIX, a scaling microservices API gateway.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.pingcap.com\/ko\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/\" \/>\n<meta property=\"og:site_name\" content=\"TiDB\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/facebook.com\/pingcap2015\" \/>\n<meta property=\"article:published_time\" content=\"2021-08-10T00:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-08-22T15:25:14+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/static.pingcap.com\/files\/2021\/08\/chaos-mesh-helps-apache-apisix-improve-system-stability-scaled.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"2560\" \/>\n\t<meta property=\"og:image:height\" content=\"854\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Shuyang Wu\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@PingCAP\" \/>\n<meta name=\"twitter:site\" content=\"@PingCAP\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Shuyang Wu\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5\ubd84\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/\"},\"author\":{\"name\":\"Shuyang Wu\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/person\/efb695a5934edf0065581356567ac23d\"},\"headline\":\"How Chaos Mesh Helps Apache APISIX Improve System Stability\",\"datePublished\":\"2021-08-10T00:00:00+00:00\",\"dateModified\":\"2024-08-22T15:25:14+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/\"},\"wordCount\":1067,\"publisher\":{\"@id\":\"https:\/\/www.pingcap.com\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/static.pingcap.com\/files\/2021\/08\/chaos-mesh-helps-apache-apisix-improve-system-stability-scaled.jpg\",\"keywords\":[\"Chaos Engineering\"],\"articleSection\":[\"Community\"],\"inLanguage\":\"ko-KR\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/\",\"url\":\"https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/\",\"name\":\"How Chaos Mesh Helps Apache APISIX Improve System Stability | TiDB\",\"isPartOf\":{\"@id\":\"https:\/\/www.pingcap.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/static.pingcap.com\/files\/2021\/08\/chaos-mesh-helps-apache-apisix-improve-system-stability-scaled.jpg\",\"datePublished\":\"2021-08-10T00:00:00+00:00\",\"dateModified\":\"2024-08-22T15:25:14+00:00\",\"description\":\"In this post, we'll share how we use Chaos Mesh to improve the stability of Apache APISIX, a scaling microservices API gateway.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/#breadcrumb\"},\"inLanguage\":\"ko-KR\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/#primaryimage\",\"url\":\"https:\/\/static.pingcap.com\/files\/2021\/08\/chaos-mesh-helps-apache-apisix-improve-system-stability-scaled.jpg\",\"contentUrl\":\"https:\/\/static.pingcap.com\/files\/2021\/08\/chaos-mesh-helps-apache-apisix-improve-system-stability-scaled.jpg\",\"width\":2560,\"height\":854},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.pingcap.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How Chaos Mesh Helps Apache APISIX Improve System Stability\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.pingcap.com\/#website\",\"url\":\"https:\/\/www.pingcap.com\/\",\"name\":\"TiDB\",\"description\":\"TiDB | SQL at Scale\",\"publisher\":{\"@id\":\"https:\/\/www.pingcap.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.pingcap.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"ko-KR\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.pingcap.com\/#organization\",\"name\":\"PingCAP\",\"url\":\"https:\/\/www.pingcap.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png\",\"contentUrl\":\"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png\",\"width\":811,\"height\":232,\"caption\":\"PingCAP\"},\"image\":{\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/facebook.com\/pingcap2015\",\"https:\/\/x.com\/PingCAP\",\"https:\/\/linkedin.com\/company\/pingcap\",\"https:\/\/youtube.com\/channel\/UCuq4puT32DzHKT5rU1IZpIA\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/person\/efb695a5934edf0065581356567ac23d\",\"name\":\"Shuyang Wu\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg\",\"contentUrl\":\"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg\",\"caption\":\"Shuyang Wu\"},\"url\":\"https:\/\/www.pingcap.com\/ko\/blog\/author\/shuyang-wu\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How Chaos Mesh Helps Apache APISIX Improve System Stability | TiDB","description":"In this post, we'll share how we use Chaos Mesh to improve the stability of Apache APISIX, a scaling microservices API gateway.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.pingcap.com\/ko\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/","og_locale":"ko_KR","og_type":"article","og_title":"How Chaos Mesh Helps Apache APISIX Improve System Stability | TiDB","og_description":"In this post, we'll share how we use Chaos Mesh to improve the stability of Apache APISIX, a scaling microservices API gateway.","og_url":"https:\/\/www.pingcap.com\/ko\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/","og_site_name":"TiDB","article_publisher":"https:\/\/facebook.com\/pingcap2015","article_published_time":"2021-08-10T00:00:00+00:00","article_modified_time":"2024-08-22T15:25:14+00:00","og_image":[{"width":2560,"height":854,"url":"https:\/\/static.pingcap.com\/files\/2021\/08\/chaos-mesh-helps-apache-apisix-improve-system-stability-scaled.jpg","type":"image\/jpeg"}],"author":"Shuyang Wu","twitter_card":"summary_large_image","twitter_creator":"@PingCAP","twitter_site":"@PingCAP","twitter_misc":{"Written by":"Shuyang Wu","Est. reading time":"5\ubd84"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/#article","isPartOf":{"@id":"https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/"},"author":{"name":"Shuyang Wu","@id":"https:\/\/www.pingcap.com\/#\/schema\/person\/efb695a5934edf0065581356567ac23d"},"headline":"How Chaos Mesh Helps Apache APISIX Improve System Stability","datePublished":"2021-08-10T00:00:00+00:00","dateModified":"2024-08-22T15:25:14+00:00","mainEntityOfPage":{"@id":"https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/"},"wordCount":1067,"publisher":{"@id":"https:\/\/www.pingcap.com\/#organization"},"image":{"@id":"https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/#primaryimage"},"thumbnailUrl":"https:\/\/static.pingcap.com\/files\/2021\/08\/chaos-mesh-helps-apache-apisix-improve-system-stability-scaled.jpg","keywords":["Chaos Engineering"],"articleSection":["Community"],"inLanguage":"ko-KR"},{"@type":"WebPage","@id":"https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/","url":"https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/","name":"How Chaos Mesh Helps Apache APISIX Improve System Stability | TiDB","isPartOf":{"@id":"https:\/\/www.pingcap.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/#primaryimage"},"image":{"@id":"https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/#primaryimage"},"thumbnailUrl":"https:\/\/static.pingcap.com\/files\/2021\/08\/chaos-mesh-helps-apache-apisix-improve-system-stability-scaled.jpg","datePublished":"2021-08-10T00:00:00+00:00","dateModified":"2024-08-22T15:25:14+00:00","description":"In this post, we'll share how we use Chaos Mesh to improve the stability of Apache APISIX, a scaling microservices API gateway.","breadcrumb":{"@id":"https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/#breadcrumb"},"inLanguage":"ko-KR","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/"]}]},{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/#primaryimage","url":"https:\/\/static.pingcap.com\/files\/2021\/08\/chaos-mesh-helps-apache-apisix-improve-system-stability-scaled.jpg","contentUrl":"https:\/\/static.pingcap.com\/files\/2021\/08\/chaos-mesh-helps-apache-apisix-improve-system-stability-scaled.jpg","width":2560,"height":854},{"@type":"BreadcrumbList","@id":"https:\/\/www.pingcap.com\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.pingcap.com\/"},{"@type":"ListItem","position":2,"name":"How Chaos Mesh Helps Apache APISIX Improve System Stability"}]},{"@type":"WebSite","@id":"https:\/\/www.pingcap.com\/#website","url":"https:\/\/www.pingcap.com\/","name":"\ud2f0DB","description":"TiDB | SQL at Scale","publisher":{"@id":"https:\/\/www.pingcap.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.pingcap.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"ko-KR"},{"@type":"Organization","@id":"https:\/\/www.pingcap.com\/#organization","name":"PingCAP","url":"https:\/\/www.pingcap.com\/","logo":{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/","url":"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png","contentUrl":"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png","width":811,"height":232,"caption":"PingCAP"},"image":{"@id":"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/facebook.com\/pingcap2015","https:\/\/x.com\/PingCAP","https:\/\/linkedin.com\/company\/pingcap","https:\/\/youtube.com\/channel\/UCuq4puT32DzHKT5rU1IZpIA"]},{"@type":"Person","@id":"https:\/\/www.pingcap.com\/#\/schema\/person\/efb695a5934edf0065581356567ac23d","name":"Shuyang Wu","image":{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/www.pingcap.com\/#\/schema\/person\/image\/","url":"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg","contentUrl":"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg","caption":"Shuyang Wu"},"url":"https:\/\/www.pingcap.com\/ko\/blog\/author\/shuyang-wu\/"}]}},"grav_blocks":false,"card_markup":"<a class=\"card-resource bg-white\" href=\"https:\/\/www.pingcap.com\/ko\/blog\/how-chaos-mesh-helps-apache-apisix-improve-system-stability\/\"><div class=\"card-resource__image-container\"><img class=\"card-resource__image\" alt=\"chaos-mesh-helps-apache-apisix-improve-system-stability.jpg\" src=\"https:\/\/static.pingcap.com\/files\/2021\/08\/chaos-mesh-helps-apache-apisix-improve-system-stability-scaled.jpg\" loading=\"lazy\" width=2560 height=854 \/><\/div><div class=\"card-resource__content-container\"><div class=\"card-resource__content-head\"><div class=\"card-resource__category\">Community<\/div><\/div><h5 class=\"card-resource__title\">How Chaos Mesh Helps Apache APISIX Improve System Stability<\/h5><\/div><\/a>","_links":{"self":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts\/478","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/users\/57"}],"replies":[{"embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/comments?post=478"}],"version-history":[{"count":5,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts\/478\/revisions"}],"predecessor-version":[{"id":19082,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts\/478\/revisions\/19082"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/media\/482"}],"wp:attachment":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/media?parent=478"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/categories?post=478"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/tags?post=478"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}