{"id":4529,"date":"2022-02-03T01:29:31","date_gmt":"2022-02-03T09:29:31","guid":{"rendered":"https:\/\/en.pingcap.com\/?post_type=case-study&p=4529"},"modified":"2025-05-07T19:33:48","modified_gmt":"2025-05-08T02:33:48","slug":"reducing-p99-latency-to-150-%ce%bcs-and-hardware-cost-by-75-with-a-scale-out-dbms","status":"publish","type":"case-study","link":"https:\/\/www.pingcap.com\/ko\/case-study\/reducing-p99-latency-to-150-%ce%bcs-and-hardware-cost-by-75-with-a-scale-out-dbms\/","title":{"rendered":"Reducing P99 Latency to 150 \u03bcs and Hardware Cost by 75% with a Scale-Out DBMS"},"content":{"rendered":"

Tuya Smart is a global Internet of Things (IoT) development platform. It builds interconnectivity standards to bridge the intelligent needs of brands, original equipment manufacturers, developers, and retail chains across a broad range of smart devices and industries. By the end of June 2021, the Tuya IoT Development Platform served 384,000+ developers around the world. Now, smart devices \u201cPowered by Tuya\u201d are available in 200+ countries and regions in 100,000+ stores all over the world.<\/p>\n\n\n\n

As our business developed, our data volume grew sharply. We needed to ensure that our average query response time was less than 10 milliseconds. We tried AWS Aurora and Apache Ignite, but they didn\u2019t meet our business requirements. Thanks to TiKV, a highly scalable, low latency, key-value database, we solved our database problem. With TiKV,\u00a0 we reduced our hardware cost by 75%. Our P99 query latency was 150 microseconds, and the write latency was 360 microseconds.<\/strong><\/p>\n\n\n\n

In this post, I\u2019ll share our business pain point, why we chose TiKV, our challenge with using TiKV, and our future plans. <\/p>\n\n\n\n

<\/span>Our business challenge: real-time responses to massive data<\/span><\/h2>\n\n\n\n

Our equipment processes 84 billion requests every day around the world. The average peak transactions per second (TPS) reached 1.5 million. We need to ensure that our average response time for queries is less than 10 milliseconds. We\u2019re in the IoT industry where there is no off-peak traffic, and the amount of writes is very large. We spent six years looking for the most suitable data architecture solution.<\/p>\n\n\n\n

<\/span>Database exploration<\/span><\/h2>\n\n\n\n

We tried Aurora for three years. But as our data size grew, it didn\u2019t meet our business requirements. Then, we switched to Ignite and used it for two years. But it was not ideal, either.<\/p>\n\n\n\n

AWS Aurora couldn\u2019t withstand our data volume surge<\/h3>\n\n\n\n

Previously, we used AWS Aurora. Its architecture separated storage and computing layers. Our application ran stably on Aurora for three years. In these years, Aurora fully met our application demands. This is because six or seven years ago IoT was unpopular, and smart home devices were not widely used.<\/p>\n\n\n\n

However, as our business developed, in recent years, our devices have increased exponentially. Every year, the number of our devices increases by three to five times. Aurora couldn\u2019t withstand the data volume surge. Not to mention that IoT devices\u2019 response times should be within 10 milliseconds. Even if we sharded our database and split the cluster, we couldn\u2019t meet our business needs.<\/strong><\/p>\n\n\n\n

<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

The AWS Aurora-based architecture<\/em><\/p>\n\n\n\n

Apache Ignite scaling risked data loss<\/h3>\n\n\n\n

We also tried Apache Ignite, a key-value system similar to TiKV, but it couldn’t meet our business needs, either. Its partition size was large; one partition stored 1 GB of data. Unlike TiKV, its scalability was not linear. When our business volume doubled and we needed to scale out our database, we had to shut down our machines. There was a risk of data loss<\/strong>; this is unacceptable to IoT devices. To solve this issue, we used Aurora behind an Ignite server for disaster recovery, and data was written to Aurora synchronously.<\/p>\n\n\n\n

<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

The Apache Ignite-based architecture<\/em><\/p>\n\n\n\n

<\/p>\n\n\n\n

TiKV is an optimal solution<\/h3>\n\n\n\n

\ud2f0DB<\/a> is an open-source distributed SQL database built by PingCAP<\/a> and its open-source community. We tested TiDB 3.0 and TiDB 4.0, but they didn\u2019t meet our requirements for low query latency and high throughput. The PingCAP team analyzed these problems and found that the SQL parser layer consumed most of the time, while TiKV<\/a>, TiDB\u2019s underlying storage engine, was completely idle. <\/p>\n\n\n\n

We thought we could remove the SQL layer (\ud2f0DB<\/a>) and write directly to TiKV, because:<\/p>\n\n\n\n