{"id":470,"date":"2020-11-26T00:00:00","date_gmt":"2020-11-26T00:00:00","guid":{"rendered":"https:\/\/en.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/"},"modified":"2025-03-04T06:35:35","modified_gmt":"2025-03-04T14:35:35","slug":"how-a-top-game-company-uses-chaos-engineering-to-improve-testing","status":"publish","type":"post","link":"https:\/\/www.pingcap.com\/ko\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/","title":{"rendered":"How a Top Game Company Uses Chaos Engineering to Improve Testing"},"content":{"rendered":"<p>NetEase Fuxi AI Lab is China&#8217;s first professional game AI research institution. Researchers use our Kubernetes-based Danlu platform for algorithm development, training and tuning, and online publishing. Thanks to the integration with Kubernetes, our platform is much more efficient. However, due to Kubernetes- and microservices-related issues, we are constantly testing and improving our platform to make it more stable.<\/p>\n<p>In this article, I&#8217;ll discuss one of our most valuable testing tools, <a href=\"https:\/\/github.com\/chaos-mesh\/chaos-mesh\">Chaos Mesh<\/a>. Chaos Mesh is an open-source Chaos Engineering tool that provides a wide range of fault injections and excellent fault monitoring through its Dashboard.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Why_Chaos_Mesh\"><\/span>Why Chaos Mesh<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>We started our search for a Chaos Engineering tool in 2018. We were looking for a tool with:<\/p>\n<ul>\n<li>Cloud-native support. Kubernetes is practically the de facto standard for service orchestration and scheduling, and the application runtime has been fully standardized. For applications that run entirely on K8s, cloud-native support is a must for any tools that go with them.<\/li>\n<li>Sufficient fault injection types. For stateful services, network failure simulation is particularly important. The platform must be able to simulate failures at different levels, such as Pods, network, and I\/O.<\/li>\n<li>Good observability. Knowing when a fault is injected and when it can be recovered is vital for us to tell whether there is an abnormality in the application.<\/li>\n<li>Active community support. We want to use an open-source project that is thoroughly tested and consistently maintained. That&#8217;s why we value sustained and timely community support.<\/li>\n<li>No intrusion on existing applications, with no domain knowledge required.<\/li>\n<li>Actual use cases for us to evaluate and build upon.<\/li>\n<\/ul>\n<p>In 2019, when Chaos Mesh, a Chaos Engineering platform for Kubernetes was open-sourced, we found the tool we were looking for. It was still in its early stage; however, we were immediately struck with the richness of fault types it supported. This was a big advantage over other chaos engineering tools, because, to a certain degree, it determines the number of issues that we can locate in the system. We instantly realized that Chaos Mesh met our expectations in almost every way.<\/p>\n<img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-25499\" src=\"https:\/\/static.pingcap.com\/files\/2020\/11\/04063523\/cloud-native-chaos-mesh-architecture.png\" alt=\"cloud-native-chaos-mesh-architecture\" width=\"1639\" height=\"889\" srcset=\"https:\/\/static.pingcap.com\/files\/2020\/11\/04063523\/cloud-native-chaos-mesh-architecture.png 1639w, https:\/\/static.pingcap.com\/files\/2020\/11\/04063523\/cloud-native-chaos-mesh-architecture-300x163.png 300w, https:\/\/static.pingcap.com\/files\/2020\/11\/04063523\/cloud-native-chaos-mesh-architecture-1024x555.png 1024w, https:\/\/static.pingcap.com\/files\/2020\/11\/04063523\/cloud-native-chaos-mesh-architecture-768x417.png 768w, https:\/\/static.pingcap.com\/files\/2020\/11\/04063523\/cloud-native-chaos-mesh-architecture-1536x833.png 1536w, https:\/\/static.pingcap.com\/files\/2020\/11\/04063523\/cloud-native-chaos-mesh-architecture-1440x781.png 1440w\" sizes=\"auto, (max-width: 1639px) 100vw, 1639px\" \/>\n<div class=\"caption-center\">Chaos Mesh architecture<\/div>\n<h2><span class=\"ez-toc-section\" id=\"Our_journey_with_Chaos_Mesh\"><\/span>Our journey with Chaos Mesh<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Chaos Mesh has helped us find several important bugs. For example, it detected a brain-split issue in <a href=\"https:\/\/www.rabbitmq.com\/\">rabbitMQ<\/a>, the open-source message-queueing software for Danlu. According to <a href=\"https:\/\/en.wikipedia.org\/wiki\/Split-brain\">Wikipedia<\/a>, &#8220;a split-brain condition indicates data or availability inconsistencies originating from the maintenance of two separate data sets with overlap in scope.&#8221; When a rabbitMQ cluster has a brain split error, there will be data write conflicts or errors, which cause more serious problems such as data inconsistencies in the messaging service. As shown in our architecture below, when brain split happens, consumers do not function normally and keep reporting server exceptions.<\/p>\n<img loading=\"lazy\" decoding=\"async\" width=\"2000\" height=\"911\" class=\"wp-image-471\" src=\"https:\/\/en.pingcap.com\/wp-content\/uploads\/2020\/11\/rabbitmq-cluster-architecture.jpg\" alt=\"Architecture of a RabbitMQ cluster\" srcset=\"https:\/\/static.pingcap.com\/files\/2020\/11\/rabbitmq-cluster-architecture.jpg 2000w, https:\/\/static.pingcap.com\/files\/2020\/11\/rabbitmq-cluster-architecture-300x137.jpg 300w, https:\/\/static.pingcap.com\/files\/2020\/11\/rabbitmq-cluster-architecture-1024x466.jpg 1024w, https:\/\/static.pingcap.com\/files\/2020\/11\/rabbitmq-cluster-architecture-768x350.jpg 768w, https:\/\/static.pingcap.com\/files\/2020\/11\/rabbitmq-cluster-architecture-1536x700.jpg 1536w, https:\/\/static.pingcap.com\/files\/2020\/11\/rabbitmq-cluster-architecture-1440x656.jpg 1440w\" sizes=\"auto, (max-width: 2000px) 100vw, 2000px\" \/>\n<div class=\"caption-center\">Architecture of a RabbitMQ cluster<\/div>\n<p>With Chaos Mesh, we could stably reproduce this issue by injecting <code>pod-kill<\/code> faults into our container instances cloud.<\/p>\n<p>Chaos Mesh also found several other issues including a startup failure, a join failure for crashed broker clusters, a heartbeat timeout, and a connection channel shutdown. Over time, our development team fixed these issues and greatly improved the stability of the Danlu platform.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"A_fast-growing_project\"><\/span>A fast-growing project<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Chaos Mesh is constantly updated and improved. When we first adopted it, it hadn&#8217;t even reached a stable version. It didn&#8217;t have a debugging or log collection tool, and the Dashboard component only applied to TiDB. The only way we could use Chaos Mesh to test other applications was to execute the YAML configuration file via <code>kubectl apply<\/code>.<\/p>\n<p><a href=\"https:\/\/chaos-mesh.org\/blog\/chaos-mesh-1.0-chaos-engineering-on-kubernetes-made-easier\">Chaos Mesh 1.0<\/a> fixed or improved most of these limitations. It offers more fine-grained and powerful chaos support, a generally-available Chaos Dashboard, enhanced observability, and more accurate chaos scope control. These are all driven by an open, collaborative, and vibrant community.<\/p>\n<img loading=\"lazy\" decoding=\"async\" width=\"1280\" height=\"720\" class=\"wp-image-472\" src=\"https:\/\/en.pingcap.com\/wp-content\/uploads\/2020\/11\/chaos-dashboard-is-generally-available.gif\" alt=\"Chaos Dashboard is now generally available\" \/>\n<div class=\"caption-center\">Chaos Dashboard is now generally available<\/div>\n<h2><span class=\"ez-toc-section\" id=\"Looking_forward\"><\/span>Looking forward<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>It&#8217;s amazing to see how much Chaos Mesh has grown and how much traction it&#8217;s gaining. We&#8217;re also happy with what we have achieved with it.<\/p>\n<p>However, Chaos Engineering is a big area to work on. In the future, we&#8217;d like to see the following features:<\/p>\n<ul>\n<li>Atomic fault injection<\/li>\n<li>Unattended fault inject that combines customized fault types with standardized methods to validate experimental objects<\/li>\n<li>Standard test cases for general components such as MySQL, Redis, and Kafka<\/li>\n<\/ul>\n<p>We&#8217;ve discussed these features with those who maintain Chaos Mesh, and they said these features are on the Chaos Mesh 2.0 roadmap.<\/p>\n<p>If you are interested, join the Chaos Mesh community via <a href=\"https:\/\/join.slack.com\/t\/cloud-native\/shared_invite\/zt-fyy3b8up-qHeDNVqbz1j8HDY6g1cY4w\">Slack<\/a> (#project-chaos-mesh) or <a href=\"https:\/\/github.com\/chaos-mesh\/chaos-mesh\">GitHub<\/a>.<\/p>","protected":false},"excerpt":{"rendered":"<p>NetEase Fuxi AI Lab is China&#8217;s first professional game AI research institution. In their search for a Chaos Engineering tool to test their Kubernetes-based AI training platform, they chose Chaos Mesh and have improved their system resiliency ever since.<\/p>","protected":false},"author":55,"featured_media":474,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"ub_ctt_via":"","footnotes":""},"categories":[18],"tags":[21],"class_list":["post-470","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-community","tag-chaos-engineering"],"acf":[],"featured_image_src":"https:\/\/static.pingcap.com\/files\/2020\/11\/cloud-native-chaos-mesh-scaled.jpg","author_info":{"display_name":"Hui Zhang","author_link":"https:\/\/www.pingcap.com\/ko\/blog\/author\/hui-zhang\/"},"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.9 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>How a Top Game Company Uses Chaos Engineering to Improve Testing | TiDB<\/title>\n<meta name=\"description\" content=\"Learn why Fuxi AI Lab, the China&#039;s first professional game AI research institution chose Chaos Mesh to improve their system resiliency.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.pingcap.com\/ko\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/\" \/>\n<meta property=\"og:locale\" content=\"ko_KR\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How a Top Game Company Uses Chaos Engineering to Improve Testing | TiDB\" \/>\n<meta property=\"og:description\" content=\"Learn why Fuxi AI Lab, the China&#039;s first professional game AI research institution chose Chaos Mesh to improve their system resiliency.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.pingcap.com\/ko\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/\" \/>\n<meta property=\"og:site_name\" content=\"TiDB\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/facebook.com\/pingcap2015\" \/>\n<meta property=\"article:published_time\" content=\"2020-11-26T00:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-03-04T14:35:35+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/static.pingcap.com\/files\/2020\/11\/cloud-native-chaos-mesh-scaled.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"2560\" \/>\n\t<meta property=\"og:image:height\" content=\"854\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Hui Zhang\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@PingCAP\" \/>\n<meta name=\"twitter:site\" content=\"@PingCAP\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Hui Zhang\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4\ubd84\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/\"},\"author\":{\"name\":\"Hui Zhang\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/person\/b6b229e73053234cffecb0b6d80c8837\"},\"headline\":\"How a Top Game Company Uses Chaos Engineering to Improve Testing\",\"datePublished\":\"2020-11-26T00:00:00+00:00\",\"dateModified\":\"2025-03-04T14:35:35+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/\"},\"wordCount\":754,\"publisher\":{\"@id\":\"https:\/\/www.pingcap.com\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/static.pingcap.com\/files\/2020\/11\/cloud-native-chaos-mesh-scaled.jpg\",\"keywords\":[\"Chaos Engineering\"],\"articleSection\":[\"Community\"],\"inLanguage\":\"ko-KR\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/\",\"url\":\"https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/\",\"name\":\"How a Top Game Company Uses Chaos Engineering to Improve Testing | TiDB\",\"isPartOf\":{\"@id\":\"https:\/\/www.pingcap.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/static.pingcap.com\/files\/2020\/11\/cloud-native-chaos-mesh-scaled.jpg\",\"datePublished\":\"2020-11-26T00:00:00+00:00\",\"dateModified\":\"2025-03-04T14:35:35+00:00\",\"description\":\"Learn why Fuxi AI Lab, the China's first professional game AI research institution chose Chaos Mesh to improve their system resiliency.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/#breadcrumb\"},\"inLanguage\":\"ko-KR\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/#primaryimage\",\"url\":\"https:\/\/static.pingcap.com\/files\/2020\/11\/cloud-native-chaos-mesh-scaled.jpg\",\"contentUrl\":\"https:\/\/static.pingcap.com\/files\/2020\/11\/cloud-native-chaos-mesh-scaled.jpg\",\"width\":2560,\"height\":854},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.pingcap.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How a Top Game Company Uses Chaos Engineering to Improve Testing\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.pingcap.com\/#website\",\"url\":\"https:\/\/www.pingcap.com\/\",\"name\":\"TiDB\",\"description\":\"TiDB | SQL at Scale\",\"publisher\":{\"@id\":\"https:\/\/www.pingcap.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.pingcap.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"ko-KR\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.pingcap.com\/#organization\",\"name\":\"PingCAP\",\"url\":\"https:\/\/www.pingcap.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png\",\"contentUrl\":\"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png\",\"width\":811,\"height\":232,\"caption\":\"PingCAP\"},\"image\":{\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/facebook.com\/pingcap2015\",\"https:\/\/x.com\/PingCAP\",\"https:\/\/linkedin.com\/company\/pingcap\",\"https:\/\/youtube.com\/channel\/UCuq4puT32DzHKT5rU1IZpIA\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/person\/b6b229e73053234cffecb0b6d80c8837\",\"name\":\"Hui Zhang\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg\",\"contentUrl\":\"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg\",\"caption\":\"Hui Zhang\"},\"url\":\"https:\/\/www.pingcap.com\/ko\/blog\/author\/hui-zhang\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How a Top Game Company Uses Chaos Engineering to Improve Testing | TiDB","description":"Learn why Fuxi AI Lab, the China's first professional game AI research institution chose Chaos Mesh to improve their system resiliency.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.pingcap.com\/ko\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/","og_locale":"ko_KR","og_type":"article","og_title":"How a Top Game Company Uses Chaos Engineering to Improve Testing | TiDB","og_description":"Learn why Fuxi AI Lab, the China's first professional game AI research institution chose Chaos Mesh to improve their system resiliency.","og_url":"https:\/\/www.pingcap.com\/ko\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/","og_site_name":"TiDB","article_publisher":"https:\/\/facebook.com\/pingcap2015","article_published_time":"2020-11-26T00:00:00+00:00","article_modified_time":"2025-03-04T14:35:35+00:00","og_image":[{"width":2560,"height":854,"url":"https:\/\/static.pingcap.com\/files\/2020\/11\/cloud-native-chaos-mesh-scaled.jpg","type":"image\/jpeg"}],"author":"Hui Zhang","twitter_card":"summary_large_image","twitter_creator":"@PingCAP","twitter_site":"@PingCAP","twitter_misc":{"Written by":"Hui Zhang","Est. reading time":"4\ubd84"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/#article","isPartOf":{"@id":"https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/"},"author":{"name":"Hui Zhang","@id":"https:\/\/www.pingcap.com\/#\/schema\/person\/b6b229e73053234cffecb0b6d80c8837"},"headline":"How a Top Game Company Uses Chaos Engineering to Improve Testing","datePublished":"2020-11-26T00:00:00+00:00","dateModified":"2025-03-04T14:35:35+00:00","mainEntityOfPage":{"@id":"https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/"},"wordCount":754,"publisher":{"@id":"https:\/\/www.pingcap.com\/#organization"},"image":{"@id":"https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/#primaryimage"},"thumbnailUrl":"https:\/\/static.pingcap.com\/files\/2020\/11\/cloud-native-chaos-mesh-scaled.jpg","keywords":["Chaos Engineering"],"articleSection":["Community"],"inLanguage":"ko-KR"},{"@type":"WebPage","@id":"https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/","url":"https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/","name":"How a Top Game Company Uses Chaos Engineering to Improve Testing | TiDB","isPartOf":{"@id":"https:\/\/www.pingcap.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/#primaryimage"},"image":{"@id":"https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/#primaryimage"},"thumbnailUrl":"https:\/\/static.pingcap.com\/files\/2020\/11\/cloud-native-chaos-mesh-scaled.jpg","datePublished":"2020-11-26T00:00:00+00:00","dateModified":"2025-03-04T14:35:35+00:00","description":"Learn why Fuxi AI Lab, the China's first professional game AI research institution chose Chaos Mesh to improve their system resiliency.","breadcrumb":{"@id":"https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/#breadcrumb"},"inLanguage":"ko-KR","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/"]}]},{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/#primaryimage","url":"https:\/\/static.pingcap.com\/files\/2020\/11\/cloud-native-chaos-mesh-scaled.jpg","contentUrl":"https:\/\/static.pingcap.com\/files\/2020\/11\/cloud-native-chaos-mesh-scaled.jpg","width":2560,"height":854},{"@type":"BreadcrumbList","@id":"https:\/\/www.pingcap.com\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.pingcap.com\/"},{"@type":"ListItem","position":2,"name":"How a Top Game Company Uses Chaos Engineering to Improve Testing"}]},{"@type":"WebSite","@id":"https:\/\/www.pingcap.com\/#website","url":"https:\/\/www.pingcap.com\/","name":"\ud2f0DB","description":"TiDB | SQL at Scale","publisher":{"@id":"https:\/\/www.pingcap.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.pingcap.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"ko-KR"},{"@type":"Organization","@id":"https:\/\/www.pingcap.com\/#organization","name":"PingCAP","url":"https:\/\/www.pingcap.com\/","logo":{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/","url":"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png","contentUrl":"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png","width":811,"height":232,"caption":"PingCAP"},"image":{"@id":"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/facebook.com\/pingcap2015","https:\/\/x.com\/PingCAP","https:\/\/linkedin.com\/company\/pingcap","https:\/\/youtube.com\/channel\/UCuq4puT32DzHKT5rU1IZpIA"]},{"@type":"Person","@id":"https:\/\/www.pingcap.com\/#\/schema\/person\/b6b229e73053234cffecb0b6d80c8837","name":"Hui Zhang","image":{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/www.pingcap.com\/#\/schema\/person\/image\/","url":"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg","contentUrl":"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg","caption":"Hui Zhang"},"url":"https:\/\/www.pingcap.com\/ko\/blog\/author\/hui-zhang\/"}]}},"grav_blocks":false,"card_markup":"<a class=\"card-resource bg-white\" href=\"https:\/\/www.pingcap.com\/ko\/blog\/how-a-top-game-company-uses-chaos-engineering-to-improve-testing\/\"><div class=\"card-resource__image-container\"><img class=\"card-resource__image\" alt=\"cloud-native-chaos-mesh.jpg\" src=\"https:\/\/static.pingcap.com\/files\/2020\/11\/cloud-native-chaos-mesh-scaled.jpg\" loading=\"lazy\" width=2560 height=854 \/><\/div><div class=\"card-resource__content-container\"><div class=\"card-resource__content-head\"><div class=\"card-resource__category\">Community<\/div><\/div><h5 class=\"card-resource__title\">How a Top Game Company Uses Chaos Engineering to Improve Testing<\/h5><\/div><\/a>","_links":{"self":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts\/470","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/users\/55"}],"replies":[{"embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/comments?post=470"}],"version-history":[{"count":3,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts\/470\/revisions"}],"predecessor-version":[{"id":25500,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts\/470\/revisions\/25500"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/media\/474"}],"wp:attachment":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/media?parent=470"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/categories?post=470"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/tags?post=470"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}