{"id":1492,"date":"2019-12-04T00:00:00","date_gmt":"2019-12-04T00:00:00","guid":{"rendered":"https:\/\/en.pingcap.com\/case-study\/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions\/"},"modified":"2024-07-16T23:07:23","modified_gmt":"2024-07-17T06:07:23","slug":"why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions","status":"publish","type":"case-study","link":"https:\/\/www.pingcap.com\/ko\/case-study\/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions\/","title":{"rendered":"NetEase Games: Why We Chose TiDB over Other MySQL-based and NewSQL Storage Solutions"},"content":{"rendered":"<p><strong>Author:<\/strong> Wenjie Li (Senior Database Administrator at NetEase Games Billing Team, TiDB User Group Ambassador)<\/p>\n\n\n\n<p>NetEase Games, affiliated with NetEase, Inc., is a leading provider of self\u2010developed PC\u2010client and mobile games to worldwide users. As one of the biggest players in China&#8217;s online gaming market, we currently have over 250 games in operation, some of which maintain millions of daily active users. NetEase Games has achieved more than $1 billion in revenue for the fifth consecutive quarter.<\/p>\n\n\n\n<p>The billing team is a support department that provides unified login and payment solutions for NetEase Games products. Our billing system unifies storage for more than 280 game clients and servers and more than 300 channel vendors at home and abroad. We also provide valuable data services for business units such as the product operations team and strategy research team.<\/p>\n\n\n\n<p>As our business boomed, the soaring data size became a nightmare for us. Every day, our billing applications processed over 3.6 billion requests, generated over 5 terabytes of log data, and wrote dozens of gigabytes of data into the database. This exposed the bottlenecks of our standalone MySQL architecture in scalability, data isolation, performance, and other areas.<\/p>\n\n\n\n<p>In this post, I&#8217;ll dive deep into why we chose <a href=\"\/ko\/tidb\/\">\ud2f0DB<\/a>, an open-source MySQL-compatible distributed Hybrid Transactional\/Analytical Processing (HTAP) database, over some other MySQL-based and NewSQL storage solutions. I&#8217;ll describe how we tested TiDB against its competitors, why TiDB won, and how we are using it now.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Our_pain_points\"><\/span>Our pain points<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>In this section, I&#8217;ll introduce the MySQL architecture for our billing applications and the bottlenecks we identify. As you&#8217;ll see, <strong>database scalability<\/strong> \uadf8\ub9ac\uace0 <strong>data isolation<\/strong> are our two major pain points.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">MySQL architecture for our applications<\/h3>\n\n\n\n<p>We deploy more than 10 billing applications to provide services for different game products. A big product employs a dedicated billing application, while multiple small products share the same application. Each billing application applies the standalone MySQL architecture as shown below. (The direction of the arrows indicates the flow of the data or request.)<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"390\" src=\"https:\/\/www.pingcap.com\/core\/uploads\/2021\/11\/standalone-mysql-architecture.png\" alt=\"\" class=\"wp-image-2726\" srcset=\"https:\/\/static.pingcap.com\/files\/2021\/11\/standalone-mysql-architecture.png 600w, https:\/\/static.pingcap.com\/files\/2021\/11\/standalone-mysql-architecture-300x195.png 300w\" sizes=\"auto, (max-width: 600px) 100vw, 600px\" \/><figcaption class=\"wp-element-caption\">Standalone MySQL architecture<\/figcaption><\/figure>\n\n\n\n<p>The architecture has the following characteristics:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Load balancing<\/strong> <p>Each billing application is deployed on multiple machines through Keepalived, so the traffic is load-balanced. This guarantees the high availability of the application service.<\/p><\/li>\n\n\n\n<li><strong>MySQL primary-secondary structure<\/strong>\n<p>The database layer employs the MySQL primary-secondary structure. Its semi-synchronous replication enables data from one MySQL server (the primary instance) to be copied automatically to one or more MySQL servers (the secondary instances). This feature addresses latency and consistency issues.<\/p>\n<\/li>\n\n\n\n<li><strong>Failover<\/strong>\n<p>The billing application accesses the back-end database through a virtual IP address (VIP). If the primary instance is down, the application service automatically drifts to the secondary through a VIP, ensuring that the service is unaffected.<\/p>\n<\/li>\n\n\n\n<li><strong>Data backup<\/strong>\n<p>Through the primary-secondary semi-synchronous replication, the secondary collects data from the online billing application. Then, it dumps the data files to a data warehouse through full and incremental replication. Online and offline computing tasks are performed there.<\/p>\n<\/li>\n<\/ul>\n\n\n\n<p>There are over 50 MySQL primary-secondary structures like this, involving 200~400 servers. Gigabytes of data are inserted on a daily basis.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Bottlenecks\"><\/span>Bottlenecks<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>With our applications&#8217; traffic and data volume growing rapidly, our standalone MySQL approach reached the following bottlenecks:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Capacity<\/strong> <p>A standalone MySQL instance has limited storage space. To maintain the existing architecture, we had to delete and rotate old data to release space.<\/p><span style=\"font-size: revert; color: initial; font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Oxygen-Sans, Ubuntu, Cantarell, &quot;Helvetica Neue&quot;, sans-serif;\">For example, a single table stored more than 700 GB of data because we needed to persist game users&#8217; purchase orders. With more users jumping in, the generated data surged, and the standalone MySQL instance would reach its storage limit.<\/span><\/li>\n\n\n\n<li><strong>Performance<\/strong> <p>Our largest single MySQL table stored 1.5 billion rows of data. With such a large number of rows, the read and write performance were severely affected.<\/p><span style=\"font-size: revert; color: initial; font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Oxygen-Sans, Ubuntu, Cantarell, &quot;Helvetica Neue&quot;, sans-serif;\">When we persisted the purchase orders&#8217; data, we also had to ensure that online users could query the data in real time. The standalone MySQL servers could not deliver a satisfactory query performance.<\/span><\/li>\n\n\n\n<li><strong>\ud655\uc7a5\uc131<\/strong>\n<p>MySQL could not elastically scale while staying online, so the capacity bottleneck could not be easily resolved.<\/p>\n<\/li>\n\n\n\n<li><strong>SQL statement complexity<\/strong>\n<p>Rotating a large table meant moving the older data from it into a newly created sub-table. It was performed on a daily basis. This would produce multiple sub-tables that needed to be joined when we ran a union query. The SQL statement could be too complicated and difficult to maintain. For this reason, it was not easy to analyze large amounts of data in a standalone MySQL architecture.<\/p>\n<\/li>\n\n\n\n<li><strong>Data barriers<\/strong>\n<p>We deployed databases independently for different game products. Data was scattered on data silos (that is, isolated islands of data), which made it difficult to gain insights from data analysis. When performing cross-product calculations, we needed to maintain multiple heterogeneous data sources, and the methods to access these sources were complicated.<\/p>\n<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1545\" height=\"843\" src=\"https:\/\/www.pingcap.com\/core\/uploads\/2019\/12\/data-silos.png\" alt=\"Data silos\" class=\"wp-image-1493\" srcset=\"https:\/\/static.pingcap.com\/files\/2019\/12\/data-silos.png 1545w, https:\/\/static.pingcap.com\/files\/2019\/12\/data-silos-300x164.png 300w, https:\/\/static.pingcap.com\/files\/2019\/12\/data-silos-1024x559.png 1024w, https:\/\/static.pingcap.com\/files\/2019\/12\/data-silos-768x419.png 768w, https:\/\/static.pingcap.com\/files\/2019\/12\/data-silos-1536x838.png 1536w, https:\/\/static.pingcap.com\/files\/2019\/12\/data-silos-1440x786.png 1440w\" sizes=\"auto, (max-width: 1545px) 100vw, 1545px\" \/><\/figure>\n\n\n\n<div class=\"caption-center\" style=\"text-align: center;\"><em>Data silos<\/em><\/div>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"New_database_exploration\"><\/span>New database exploration<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>In this section, I&#8217;ll describe the ideal database for our applications, and what we did to find it.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">The ideal database for our applications<\/h3>\n\n\n\n<p>Considering our billing applications were highly coupled with MySQL, to resolve our issues we needed a new storage solution with the following features:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Compatible with the MySQL protocol.<\/li>\n\n\n\n<li>Support for transactions\u2014tasks are executed as transactions or rolled back in the case of error.<\/li>\n\n\n\n<li>Support for indexes\u2014especially for secondary indexes.<\/li>\n\n\n\n<li>Horizontal scalability\u2014the system can elastically scale while online, including performance scaling and capacity expansion.<\/li>\n\n\n\n<li>Stability and reliability.<\/li>\n\n\n\n<li>Backup and recovery.<\/li>\n\n\n\n<li>Disaster tolerance.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Our options<\/h3>\n\n\n\n<p>To meet the requirements above, we listed several optional solutions. These solutions can be grouped into two categories: MySQL-based solutions and NewSQL-based solutions (CockroachDB and TiDB).<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><td><strong>\uc194\ub8e8\uc158<\/strong><\/td><td><strong>Notes<\/strong><\/td><\/tr><tr><td>MySQL sharding<\/td><td>Based on MySQL<\/td><\/tr><tr><td>MySQL InnoDB cluster<\/td><td>Based on MySQL<\/td><\/tr><tr><td>MySQL + Vitess<\/td><td>Based on MySQL<\/td><\/tr><tr><td>MySQL + MyCAT<\/td><td>Based on MySQL<\/td><\/tr><tr><td>CockroachDB (CRDB for short)<\/td><td>NewSQL<\/td><\/tr><tr><td>\ud2f0DB<\/td><td>NewSQL<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Testing<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\">MySQL versus NewSQL<\/h4>\n\n\n\n<p>At the beginning, we preferred to use MySQL-based solutions such as MySQL InnoDB clusters or MySQL with middleware. We designed some tests to verify each solution&#8217;s practicality.<\/p>\n\n\n\n<p>In MySQL clusters of two different versions\u20145.7.25 and 8.0.12\u2014we performed writes on 10 tables (each had 10 million rows) with 128 concurrent threads. The same test was run for clusters with a single node (that is, a standalone architecture), three nodes, and five nodes. The results are shown below:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"2492\" height=\"1148\" src=\"https:\/\/www.pingcap.com\/core\/uploads\/2019\/12\/results-of-testing-mysql-clusters-of-different-node-sizes.png\" alt=\"Results of testing MySQL clusters of different node sizes\" class=\"wp-image-1494\" srcset=\"https:\/\/static.pingcap.com\/files\/2019\/12\/results-of-testing-mysql-clusters-of-different-node-sizes.png 2492w, https:\/\/static.pingcap.com\/files\/2019\/12\/results-of-testing-mysql-clusters-of-different-node-sizes-300x138.png 300w, https:\/\/static.pingcap.com\/files\/2019\/12\/results-of-testing-mysql-clusters-of-different-node-sizes-1024x472.png 1024w, https:\/\/static.pingcap.com\/files\/2019\/12\/results-of-testing-mysql-clusters-of-different-node-sizes-768x354.png 768w, https:\/\/static.pingcap.com\/files\/2019\/12\/results-of-testing-mysql-clusters-of-different-node-sizes-1536x708.png 1536w, https:\/\/static.pingcap.com\/files\/2019\/12\/results-of-testing-mysql-clusters-of-different-node-sizes-2048x943.png 2048w, https:\/\/static.pingcap.com\/files\/2019\/12\/results-of-testing-mysql-clusters-of-different-node-sizes-1440x663.png 1440w\" sizes=\"auto, (max-width: 2492px) 100vw, 2492px\" \/><\/figure>\n\n\n\n<div class=\"caption-center\">Results of testing MySQL clusters of different node sizes<\/div>\n\n\n\n<p>The tests showed that a multi-node MySQL cluster performed about 30% worse under write workloads than a standalone MySQL node. Other read and write tests had similar results.<\/p>\n\n\n\n<p>We tested MySQL with middleware solutions, and this architecture didn&#8217;t meet our expectations. To sum up, we did not adopt these solutions or the MySQL sharding solution for two reasons:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The performance did not satisfy our requirements.<\/li>\n\n\n\n<li>Migrating to the new application architecture was complicated. We would need to modify large amounts of application code.<\/li>\n<\/ul>\n\n\n\n<p>In fact, solutions such as MySQL InnoDB clusters and MySQL with middleware are essentially an extension of the MySQL primary-secondary structure. They are not a real distributed system, but achieve horizontal scalability in a way like adding &#8220;patches&#8221;\u2014that&#8217;s why many of their characteristics did not meet our expectations.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">CockroachDB versus TiDB<\/h4>\n\n\n\n<p>In the distributed NewSQL world, TiDB and CockroachDB (CRDB for short) are well known to many, and both are the open-source implementation of the Google Spanner and F1 papers. Our research showed that:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>TiDB is compatible with the MySQL protocol, while CRDB is compatible with the PostgreSQL protocol.<\/li>\n\n\n\n<li>If the application is based on the MySQL database, TiDB may be a better choice; if the application is based on PostgreSQL, CRDB may be the preferred choice.<\/li>\n<\/ul>\n\n\n\n<p>We also conducted comprehensive tests for TiDB and CRDB. For example, in July, 2018, we created two clusters of 10 machines each. Five of the nodes in each cluster were dedicated storage nodes. We deployed CRDB 2.1.0 on one cluster and TiDB 2.0.5 on the other. We performed reads and writes on a single table of 200 million rows with 160 concurrent threads. (Both CRDB and TiDB clusters used the default configuration and were not tuned.)<\/p>\n\n\n\n<p>Here are some testing statements we ran:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><td><strong>Query type<\/strong><\/td><td><strong>Testing statements<\/strong><\/td><\/tr><tr><td>Range query<\/td><td>SELECT c FROM sbtest%u WHERE id BETWEEN ? AND ? <br><span style=\"font-size: revert; font-family: inherit; font-weight: inherit; color: initial;\">SELECT SUM(k) FROM sbtest%u WHERE id BETWEEN ? AND ?<\/span> <br><span style=\"font-size: revert; font-family: inherit; font-weight: inherit; color: initial;\">SELECT c FROM sbtest WHERE id BETWEEN ? AND ? ORDER BY c<\/span> <br><span style=\"font-size: revert; font-family: inherit; font-weight: inherit; color: initial;\">SELECT DISTINCT c FROM sbtest%u WHERE id BETWEEN ? AND ? ORDER BY c<\/span><\/td><\/tr><tr><td>Random `IN` query<\/td><td>SELECT id, k, c, pad FROM sbtest1 WHERE k IN (?)<\/td><\/tr><tr><td>Random range query<\/td><td>SELECT count(k) FROM sbtest1 WHERE k BETWEEN ? AND ? OR k BETWEEN ? AND ?<\/td><\/tr><tr><td>Update indexed column<\/td><td>UPDATE sbtest%u SET k=k+1 WHERE id=?<\/td><\/tr><tr><td>Update non-indexed column<\/td><td>UPDATE sbtest%u SET c=? WHERE id=?<\/td><\/tr><tr><td>Mixed read and write<\/td><td>Range query + create\/delete\/update queries<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>The following figure shows one of our important testing results: In general, CRDB 2.1.0 and TiDB 2.0.5 had a similar performance in most of the read and write workloads.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"2256\" height=\"1088\" src=\"https:\/\/www.pingcap.com\/core\/uploads\/2019\/12\/performance-comparison-of-reads-and-writes-on-a-large-table.png\" alt=\"Performance comparison of reads and writes on a large table\" class=\"wp-image-1495\" srcset=\"https:\/\/static.pingcap.com\/files\/2019\/12\/performance-comparison-of-reads-and-writes-on-a-large-table.png 2256w, https:\/\/static.pingcap.com\/files\/2019\/12\/performance-comparison-of-reads-and-writes-on-a-large-table-300x145.png 300w, https:\/\/static.pingcap.com\/files\/2019\/12\/performance-comparison-of-reads-and-writes-on-a-large-table-1024x494.png 1024w, https:\/\/static.pingcap.com\/files\/2019\/12\/performance-comparison-of-reads-and-writes-on-a-large-table-768x370.png 768w, https:\/\/static.pingcap.com\/files\/2019\/12\/performance-comparison-of-reads-and-writes-on-a-large-table-1536x741.png 1536w, https:\/\/static.pingcap.com\/files\/2019\/12\/performance-comparison-of-reads-and-writes-on-a-large-table-2048x988.png 2048w, https:\/\/static.pingcap.com\/files\/2019\/12\/performance-comparison-of-reads-and-writes-on-a-large-table-1440x694.png 1440w\" sizes=\"auto, (max-width: 2256px) 100vw, 2256px\" \/><\/figure>\n\n\n\n<div class=\"caption-center\">Performance comparison of reads and writes on a large table<\/div>\n\n\n\n<p>Conclusion:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>CRDB and TiDB were comparable in performance.\n<p>Note: The above is the testing result for TiDB 2.0.5. In July 2019, PingCAP announced that <a href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-3.0-announcement\/\">TiDB 3.0 reached general availability<\/a>, achieving a major performance boost. Our recent tests showed that in most scenarios TiDB 3.0&#8217;s performance was multiple times better than TiDB 2.1. The following test results compare TiDB 3.0.3 and TiDB 2.1.15 in the areas of peak <a href=\"https:\/\/en.wikipedia.org\/wiki\/Online_transaction_processing\">online transaction processing<\/a> (OLTP) and <a href=\"http:\/\/www.tpc.org\/tpcc\/\">TPC Benchmark C<\/a> (TPC-C):<\/p>\n<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"2452\" height=\"1172\" src=\"https:\/\/www.pingcap.com\/core\/uploads\/2019\/12\/peak-oltp-performance-comparison.png\" alt=\"Peak OLTP performance comparison\" class=\"wp-image-1496\" srcset=\"https:\/\/static.pingcap.com\/files\/2019\/12\/peak-oltp-performance-comparison.png 2452w, https:\/\/static.pingcap.com\/files\/2019\/12\/peak-oltp-performance-comparison-300x143.png 300w, https:\/\/static.pingcap.com\/files\/2019\/12\/peak-oltp-performance-comparison-1024x489.png 1024w, https:\/\/static.pingcap.com\/files\/2019\/12\/peak-oltp-performance-comparison-768x367.png 768w, https:\/\/static.pingcap.com\/files\/2019\/12\/peak-oltp-performance-comparison-1536x734.png 1536w, https:\/\/static.pingcap.com\/files\/2019\/12\/peak-oltp-performance-comparison-2048x979.png 2048w, https:\/\/static.pingcap.com\/files\/2019\/12\/peak-oltp-performance-comparison-1440x688.png 1440w\" sizes=\"auto, (max-width: 2452px) 100vw, 2452px\" \/><\/figure>\n\n\n\n<div class=\"caption-center\">Peak OLTP performance comparison<\/div>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"2670\" height=\"864\" src=\"https:\/\/www.pingcap.com\/core\/uploads\/2019\/12\/tpc-c-performance-comparison.png\" alt=\"TPC-C performance comparison\" class=\"wp-image-1497\" srcset=\"https:\/\/static.pingcap.com\/files\/2019\/12\/tpc-c-performance-comparison.png 2670w, https:\/\/static.pingcap.com\/files\/2019\/12\/tpc-c-performance-comparison-300x97.png 300w, https:\/\/static.pingcap.com\/files\/2019\/12\/tpc-c-performance-comparison-1024x331.png 1024w, https:\/\/static.pingcap.com\/files\/2019\/12\/tpc-c-performance-comparison-768x249.png 768w, https:\/\/static.pingcap.com\/files\/2019\/12\/tpc-c-performance-comparison-1536x497.png 1536w, https:\/\/static.pingcap.com\/files\/2019\/12\/tpc-c-performance-comparison-2048x663.png 2048w, https:\/\/static.pingcap.com\/files\/2019\/12\/tpc-c-performance-comparison-1440x466.png 1440w\" sizes=\"auto, (max-width: 2670px) 100vw, 2670px\" \/><\/figure>\n\n\n\n<div class=\"caption-center\">TPC-C performance comparison<\/div>\n\n\n\n<ul class=\"wp-block-list\">\n<li>CRDB is compatible with the PostgreSQL protocol. If we migrate our application architecture to CRDB, we have to change the protocol, which is complicated and costly.<\/li>\n\n\n\n<li>TiDB is compatible with the MySQL protocol, which means few code modifications and low migration costs.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Choosing TiDB<\/h4>\n\n\n\n<p>Finally, it was time to make our decision. For each solution in the discussion, we evaluated the factors that were most important to us:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><td><strong>Solution<\/strong><\/td><td><strong>\ud655\uc7a5\uc131<\/strong><\/td><td><strong>OLTP<\/strong><\/td><td><strong>OLAP<\/strong><\/td><td><strong>Open Source<\/strong><\/td><\/tr><tr><td>MySQL sharding<\/td><td>No<\/td><td>Yes<\/td><td>No<\/td><td>Yes<\/td><\/tr><tr><td>MySQL InnoDB cluster<\/td><td>No<\/td><td>Yes<\/td><td>Yes<\/td><td>Yes<\/td><\/tr><tr><td>MySQL + Vitess<\/td><td>No<\/td><td>Yes<\/td><td>No<\/td><td>Yes<\/td><\/tr><tr><td>MySQL + MyCAT<\/td><td>Yes<\/td><td>Yes<\/td><td>No<\/td><td>Yes<\/td><\/tr><tr><td>CockroachDB (CRDB)<\/td><td>Yes<\/td><td>Yes<\/td><td>Yes<\/td><td>BSL and CCL<\/td><\/tr><tr><td>\ud2f0DB<\/td><td>Yes<\/td><td>Yes<\/td><td>Yes<\/td><td>Yes<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>It seemed like TiDB has a competitive edge over the others. Then, we listed what TiDB empowered us to do:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>MySQL compatibility saves application migration efforts and costs.<\/strong>\n<p>TiDB is highly compatible with MySQL and supports indexes (including secondary indexes) and ACID transactions. It also has an ecosystem of tools such as <a href=\"https:\/\/www.pingcap.com\/ko\/docs\/stable\/reference\/tools\/syncer\/\">Syncer<\/a> \uadf8\ub9ac\uace0 <a href=\"https:\/\/docs.pingcap.com\/tidb-data-migration\/dev\/overview\">TiDB Data Migration<\/a> (TiDB DM) for seamless migration from MySQL.<\/p>\n<\/li>\n\n\n\n<li><strong>Horizontal scalability brings large capacity storage and lower operation costs.<\/strong>\n<p>TiDB expands storage simply by adding new nodes. The database can elastically scale out in real time without taking any node offline. This makes our infrastructure capacity planning much easier and more cost-effective than traditional relational databases that only scale vertically. In addition, our database administrators are finally free from the drudgery of frequently purging older data from MySQL.<\/p>\n<\/li>\n\n\n\n<li><strong>Horizontal scalability makes storing data in a large database pool and big data mining possible.<\/strong>\n<p>Horizontal scalability ensures large capacity storage in TiDB. This enables us to store different game products&#8217; data in the same database pool. This removes data barriers and is very important for business analytics.<\/p>\n<\/li>\n\n\n\n<li><strong>No need for sharding.<\/strong>\n<p>TiDB supports efficient reads and writes on super large tables.<\/p>\n<\/li>\n\n\n\n<li><strong>A stateless computing engine guarantees high queries per second (QPS) performance.<\/strong>\n<p>As the computing layer, TiDB server is stateless and supports multi-node reads and writes. It is easy to increase QPS by adding new TiDB servers as long as the storage layer is not under-powered.<\/p>\n<\/li>\n\n\n\n<li><strong>The Raft consensus algorithm ensures high availability and strong consistency.<\/strong> <p>TiDB uses the Raft consensus algorithm to ensure that data is replicated to multiple replicas throughout storage in Raft groups. If a node fails, a Raft group automatically elects a new leader for the failed member and self-heals the TiDB cluster. Thus, one node failure does not affect application service.<\/p><\/li>\n\n\n\n<li><strong>TiDB&#8217;s ecosystem of tools simplifies data backup and recovery.<\/strong>\n<p>You can deploy TiDB across data centers or across regions. We can use <a href=\"https:\/\/www.pingcap.com\/ko\/docs\/stable\/reference\/tools\/mydumper\/\">Mydumper<\/a> \uadf8\ub9ac\uace0 <a href=\"https:\/\/www.pingcap.com\/ko\/docs\/stable\/reference\/tools\/loader\/\">Loader<\/a> to concurrently export and restore data with multiple threads. TiDB also has the <a href=\"https:\/\/www.pingcap.com\/ko\/docs\/stable\/reference\/tidb-binlog\/overview\/\">TiDB Binlog<\/a> tool, which collects the logical changes made to a TiDB cluster and provides incremental backup and replication to the downstream (TiDB, Kafka, or MySQL).<\/p>\n<\/li>\n<\/ul>\n\n\n\n<p>As far as we were concerned, TiDB was the most cost-effective solution to fit in our scenarios, and we wanted to give it a try!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Using_TiDB_for_our_applications\"><\/span>Using TiDB for our applications<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Throughout our journey with TiDB, we&#8217;ve used multiple versions for testing or production, including versions 1.0, 2.0.5, 2.0.11, 2.1.5, 2.1.15, and 3.0.3. We&#8217;ve witnessed the great improvements TiDB has made over the years. In this section, I&#8217;ll describe how we&#8217;re currently using TiDB for our billing applications.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">TiDB architecture<\/h3>\n\n\n\n<p>We deploy the TiDB cluster in a highly-layered structure as shown below. <a href=\"https:\/\/www.nginx.com\/\">NGINX<\/a> is used for load balancing in the front end.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"2704\" height=\"1450\" src=\"https:\/\/www.pingcap.com\/core\/uploads\/2019\/12\/tidb-architecture-for-billing-applications.png\" alt=\"TiDB architecture for billing applications\" class=\"wp-image-1498\" srcset=\"https:\/\/static.pingcap.com\/files\/2019\/12\/tidb-architecture-for-billing-applications.png 2704w, https:\/\/static.pingcap.com\/files\/2019\/12\/tidb-architecture-for-billing-applications-300x161.png 300w, https:\/\/static.pingcap.com\/files\/2019\/12\/tidb-architecture-for-billing-applications-1024x549.png 1024w, https:\/\/static.pingcap.com\/files\/2019\/12\/tidb-architecture-for-billing-applications-768x412.png 768w, https:\/\/static.pingcap.com\/files\/2019\/12\/tidb-architecture-for-billing-applications-1536x824.png 1536w, https:\/\/static.pingcap.com\/files\/2019\/12\/tidb-architecture-for-billing-applications-2048x1098.png 2048w, https:\/\/static.pingcap.com\/files\/2019\/12\/tidb-architecture-for-billing-applications-1440x772.png 1440w\" sizes=\"auto, (max-width: 2704px) 100vw, 2704px\" \/><\/figure>\n\n\n\n<div class=\"caption-center\">TiDB architecture for billing applications<\/div>\n\n\n\n<p>Inside the TiDB cluster, there are three main components:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>TiDB server is a stateless SQL layer that processes SQL requests, accesses data from the storage layer, and returns the computing results to the application. It is MySQL-compatible and sits on top of TiKV.<\/li>\n\n\n\n<li>TiKV server is the distributed transactional key-value storage layer where the data persists. It uses the Raft consensus protocol to ensure that data has multiple replicas and will not be lost if a node fails.<\/li>\n\n\n\n<li>PD (Placement Driver) server is a metadata cluster powered by etcd that manages and schedules TiKV nodes.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">TiDB use cases<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\">Application scenarios<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>TiDB mirrors the MySQL databases in the production environment. TiDB collects data from online MySQL databases, assembles it into the data pool, and manages the data intensively.<\/li>\n\n\n\n<li>TiDB supports data platform services, including reporting, monitoring, operations, generating user profiles, and computing big data.<\/li>\n\n\n\n<li>HTAP scenarios that involve both OLTP and online analytical processing (OLAP) workloads.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Cluster<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>In our testing environment, we use a TiDB 3.0.3 cluster to evaluate new features.<\/li>\n\n\n\n<li>In our production environment, we also have a TiDB 3.0.3 cluster that serves two kinds of workloads: 80% of them are offline big data computing, while the other 20% are online services.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Scale<\/h4>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><td>Environment<\/td><td>41 servers <br><span style=\"font-size: revert; font-family: inherit; font-weight: inherit; color: initial;\">88 instances<\/span> <br><span style=\"font-size: revert; font-family: inherit; font-weight: inherit; color: initial;\">38 Syncer real-time replication streams (will upgrade to <\/span><a style=\"font-size: revert; font-family: inherit; font-weight: inherit;\" href=\"https:\/\/docs.pingcap.com\/tidb-data-migration\/dev\/overview\">TiDB Data Migration<\/a><span style=\"font-size: revert; font-family: inherit; font-weight: inherit; color: initial;\">)<\/span><\/td><\/tr><tr><td>Storage<\/td><td>20 TB data (50 TB total) <br><span style=\"font-size: revert; font-family: inherit; font-weight: inherit; color: initial;\">2.3 million Regions<\/span><\/td><\/tr><tr><td>Performance<\/td><td>4 k\/s average QPS <br><span style=\"font-size: revert; font-family: inherit; font-weight: inherit; color: initial;\">10 k\/s QPS at peak time (sustained for 4 hours)<\/span> <br><span style=\"font-size: revert; font-family: inherit; font-weight: inherit; color: initial;\">1:5 read\/write ratio<\/span><\/td><\/tr><tr><td>Latency<\/td><td>80% of latencies below 8 ms <br><span style=\"font-size: revert; font-family: inherit; font-weight: inherit; color: initial;\">95% of latencies below 125 ms<\/span> <br><span style=\"font-size: revert; font-family: inherit; font-weight: inherit; color: initial;\">99.9% of latencies below 500 ms<\/span><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Conclusion\"><\/span>Conclusion<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>In this post, I discussed why we adopted a new storage solution for our billing applications and why we chose TiDB over other alternatives. Our current success with using TiDB gives us the confidence to move more of our services to TiDB in the future. This scale-out distributed SQL database enables us to gain insights from data and create value for our users.<\/p>\n\n\n\n<p>In my next post, I&#8217;ll share some first-hand experiences on how we migrated applications from MySQL to TiDB, as well as how we managed and maintained a TiDB cluster. Stay tuned.<\/p>","protected":false},"excerpt":{"rendered":"<p>As business boomed, NetEase Games met bottlenecks in database scalability and data isolation. This post introduces why they chose TiDB over other MySQL-based and NewSQL storage solutions to solve their pain points.<\/p>","protected":false},"author":149,"featured_media":17241,"template":"","tags":[41,14,9],"customer":[102],"industry":[50],"class_list":["post-1492","case-study","type-case-study","status-publish","has-post-thumbnail","hentry","tag-big-data","tag-mysql","tag-scalability","customer-netease-games","industry-gaming"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.9 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Why NetEase Games Chose TiDB over Other Storage Solutions<\/title>\n<meta name=\"description\" content=\"Explore why NetEase Games chose TiDB, an open-source distributed HTAP database, over some other MySQL-based and NewSQL storage solutions.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.pingcap.com\/ko\/case-study\/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions\/\" \/>\n<meta property=\"og:locale\" content=\"ko_KR\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Why NetEase Games Chose TiDB over Other Storage Solutions\" \/>\n<meta property=\"og:description\" content=\"Explore why NetEase Games chose TiDB, an open-source distributed HTAP database, over some other MySQL-based and NewSQL storage solutions.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.pingcap.com\/ko\/case-study\/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions\/\" \/>\n<meta property=\"og:site_name\" content=\"TiDB\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/facebook.com\/pingcap2015\" \/>\n<meta property=\"article:modified_time\" content=\"2024-07-17T06:07:23+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/static.pingcap.com\/files\/2024\/05\/27004048\/neteasegames.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1500\" \/>\n\t<meta property=\"og:image:height\" content=\"500\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@PingCAP\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"13\ubd84\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.pingcap.com\/case-study\/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions\/\",\"url\":\"https:\/\/www.pingcap.com\/case-study\/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions\/\",\"name\":\"Why NetEase Games Chose TiDB over Other Storage Solutions\",\"isPartOf\":{\"@id\":\"https:\/\/www.pingcap.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.pingcap.com\/case-study\/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.pingcap.com\/case-study\/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/static.pingcap.com\/files\/2024\/05\/27004048\/neteasegames.jpg\",\"datePublished\":\"2019-12-04T00:00:00+00:00\",\"dateModified\":\"2024-07-17T06:07:23+00:00\",\"description\":\"Explore why NetEase Games chose TiDB, an open-source distributed HTAP database, over some other MySQL-based and NewSQL storage solutions.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.pingcap.com\/case-study\/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions\/#breadcrumb\"},\"inLanguage\":\"ko-KR\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.pingcap.com\/case-study\/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/www.pingcap.com\/case-study\/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions\/#primaryimage\",\"url\":\"https:\/\/static.pingcap.com\/files\/2024\/05\/27004048\/neteasegames.jpg\",\"contentUrl\":\"https:\/\/static.pingcap.com\/files\/2024\/05\/27004048\/neteasegames.jpg\",\"width\":1500,\"height\":500,\"caption\":\"NetEaseGames\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.pingcap.com\/case-study\/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.pingcap.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Case Studies\",\"item\":\"https:\/\/www.pingcap.com\/customers\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"NetEase Games: Why We Chose TiDB over Other MySQL-based and NewSQL Storage Solutions\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.pingcap.com\/#website\",\"url\":\"https:\/\/www.pingcap.com\/\",\"name\":\"TiDB\",\"description\":\"TiDB | SQL at Scale\",\"publisher\":{\"@id\":\"https:\/\/www.pingcap.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.pingcap.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"ko-KR\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.pingcap.com\/#organization\",\"name\":\"PingCAP\",\"url\":\"https:\/\/www.pingcap.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png\",\"contentUrl\":\"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png\",\"width\":811,\"height\":232,\"caption\":\"PingCAP\"},\"image\":{\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/facebook.com\/pingcap2015\",\"https:\/\/x.com\/PingCAP\",\"https:\/\/linkedin.com\/company\/pingcap\",\"https:\/\/youtube.com\/channel\/UCuq4puT32DzHKT5rU1IZpIA\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Why NetEase Games Chose TiDB over Other Storage Solutions","description":"Explore why NetEase Games chose TiDB, an open-source distributed HTAP database, over some other MySQL-based and NewSQL storage solutions.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.pingcap.com\/ko\/case-study\/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions\/","og_locale":"ko_KR","og_type":"article","og_title":"Why NetEase Games Chose TiDB over Other Storage Solutions","og_description":"Explore why NetEase Games chose TiDB, an open-source distributed HTAP database, over some other MySQL-based and NewSQL storage solutions.","og_url":"https:\/\/www.pingcap.com\/ko\/case-study\/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions\/","og_site_name":"TiDB","article_publisher":"https:\/\/facebook.com\/pingcap2015","article_modified_time":"2024-07-17T06:07:23+00:00","og_image":[{"width":1500,"height":500,"url":"https:\/\/static.pingcap.com\/files\/2024\/05\/27004048\/neteasegames.jpg","type":"image\/jpeg"}],"twitter_card":"summary_large_image","twitter_site":"@PingCAP","twitter_misc":{"Est. reading time":"13\ubd84"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.pingcap.com\/case-study\/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions\/","url":"https:\/\/www.pingcap.com\/case-study\/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions\/","name":"Why NetEase Games Chose TiDB over Other Storage Solutions","isPartOf":{"@id":"https:\/\/www.pingcap.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.pingcap.com\/case-study\/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions\/#primaryimage"},"image":{"@id":"https:\/\/www.pingcap.com\/case-study\/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions\/#primaryimage"},"thumbnailUrl":"https:\/\/static.pingcap.com\/files\/2024\/05\/27004048\/neteasegames.jpg","datePublished":"2019-12-04T00:00:00+00:00","dateModified":"2024-07-17T06:07:23+00:00","description":"Explore why NetEase Games chose TiDB, an open-source distributed HTAP database, over some other MySQL-based and NewSQL storage solutions.","breadcrumb":{"@id":"https:\/\/www.pingcap.com\/case-study\/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions\/#breadcrumb"},"inLanguage":"ko-KR","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.pingcap.com\/case-study\/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions\/"]}]},{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/www.pingcap.com\/case-study\/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions\/#primaryimage","url":"https:\/\/static.pingcap.com\/files\/2024\/05\/27004048\/neteasegames.jpg","contentUrl":"https:\/\/static.pingcap.com\/files\/2024\/05\/27004048\/neteasegames.jpg","width":1500,"height":500,"caption":"NetEaseGames"},{"@type":"BreadcrumbList","@id":"https:\/\/www.pingcap.com\/case-study\/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.pingcap.com\/"},{"@type":"ListItem","position":2,"name":"Case Studies","item":"https:\/\/www.pingcap.com\/customers\/"},{"@type":"ListItem","position":3,"name":"NetEase Games: Why We Chose TiDB over Other MySQL-based and NewSQL Storage Solutions"}]},{"@type":"WebSite","@id":"https:\/\/www.pingcap.com\/#website","url":"https:\/\/www.pingcap.com\/","name":"\ud2f0DB","description":"TiDB | SQL at Scale","publisher":{"@id":"https:\/\/www.pingcap.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.pingcap.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"ko-KR"},{"@type":"Organization","@id":"https:\/\/www.pingcap.com\/#organization","name":"PingCAP","url":"https:\/\/www.pingcap.com\/","logo":{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/","url":"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png","contentUrl":"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png","width":811,"height":232,"caption":"PingCAP"},"image":{"@id":"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/facebook.com\/pingcap2015","https:\/\/x.com\/PingCAP","https:\/\/linkedin.com\/company\/pingcap","https:\/\/youtube.com\/channel\/UCuq4puT32DzHKT5rU1IZpIA"]}]}},"card_markup":"<div class=\"card-case-study\"><div class=\"card-case-study__image-container\"><img class=\"card-case-study__image\" alt=\"NetEase Games\" src=\"https:\/\/static.pingcap.com\/files\/2023\/12\/14203609\/NetEase-Games.png\" loading=\"lazy\" width=2000 height=722 \/><\/div><span class=\"card-case-study__title\">NetEase Games: Why We Chose TiDB over Other MySQL-based and NewSQL Storage Solutions<\/span><div class=\"card-case-study__button\"><a class=\"button--secondary\" href=\"https:\/\/www.pingcap.com\/ko\/case-study\/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions\/\" target=\"_blank\">View Case Study<\/a><\/div><\/div>","_links":{"self":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/case-study\/1492","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/case-study"}],"about":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/types\/case-study"}],"author":[{"embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/users\/149"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/media\/17241"}],"wp:attachment":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/media?parent=1492"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/tags?post=1492"},{"taxonomy":"customer","embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/customer?post=1492"},{"taxonomy":"industry","embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/industry?post=1492"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}