{"id":14827,"date":"2023-12-04T06:00:00","date_gmt":"2023-12-04T14:00:00","guid":{"rendered":"https:\/\/www.pingcap.com\/?p=14827"},"modified":"2023-12-04T15:25:18","modified_gmt":"2023-12-04T23:25:18","slug":"presenting-tidb-7-5-improved-stability-scale-data-intensive-apps","status":"publish","type":"post","link":"https:\/\/www.pingcap.com\/ko\/blog\/presenting-tidb-7-5-improved-stability-scale-data-intensive-apps\/","title":{"rendered":"Presenting TiDB 7.5: Improved Stability at Scale for Data-Intensive Applications"},"content":{"rendered":"

While \ud2f0DB<\/a> is already the trusted system of record for some of the most data-intensive applications ever, there is still room to improve. Subsequently, TiDB 7.5, our second long-term support (LTS) release of 2023, emphasizes stability at scale.<\/p>\n\n\n\n

As application data continues to grow, we see more and more scaling challenges. The most common antidote is sharding, but the simplest and most future-proof antidote is distributed SQL<\/a>. In distributed SQL databases, scaling issues with single applications are mostly in our rearview. However, transactional consistency, durability, and massive scale begets new challenges, like application performance stability. <\/p>\n\n\n\n

In this blog, we\u2019ll discuss the key enhancements TiDB 7.5 brings to the general stability of large resource-hungry background jobs. We\u2019ll also explore the operator-controlled stability of those same jobs, as well as a key feature for controlling \u201cbad\u201d queries.<\/p>\n\n\n

\n
\"A<\/figure>\n<\/div>\n\n\n

<\/p>\n\n\n\n

Figure 1. A high-level view of TiDB 7.5 enhancements. For all other details on the delta between TiDB 7.1 and TiDB 7.5, please see the<\/em> release notes<\/em><\/a>.<\/em><\/p>\n\n\n\n

<\/span>TiDB 7.5 Reduces I\/O and Compute Overhead of Background Jobs<\/span><\/h2>\n\n\n\n

In TiDB 7.1, the distributed SQL database received a big background task optimization with a new framework<\/a> for distributing work across TiDB nodes. In doing so, we saw huge speed-ups for large data re-organization tasks like adding 10 billion entry indexes and backdoor importing of 10TB tables. However, this new framework comes with a new challenge.<\/p>\n\n\n\n

Data re-orgs require sorting. But sorting data on many TiDB nodes simultaneously means temporarily storing out-of-order data in TiKV<\/a>, TiDB\u2019s row-oriented storage layer. Out-of-order results lead to storing ranges of keys that overlap significantly. This overlap kicks off several costly operations in TiKV, sometimes leading to killed nodes. <\/p>\n\n\n\n

To prevent this, TiDB 7.4 added a global sort<\/a> feature. This feature leverages external shared storage to write intermediate results, along with metadata to identify overlap. As some nodes\u2019 workers write new results to shared storage, other nodes\u2019 workers read from this shared storage. They then merge sort that data until there is no overlap. Once complete, TiKV loads these sorted data files, avoiding the costly overhead previously experienced.<\/p>\n\n\n\n

Stabilize Your Most Critical Workloads During Large Background Jobs<\/h3>\n\n\n\n

Online databases supporting critical applications feel justifiably vulnerable. Marrying many of them in the same backend is a scary thought, but it\u2019s hypothetically ideal. TiDB 7.1, laid the groundwork for this paradigm by way of Resource Control<\/a>, a new feature detailed in this blog<\/a>. This laid the foundation for combining workloads with differing patterns and criticality into a single cluster, simplifying infrastructure. At that point in time, however, Resource Control cannot control background tasks (i.e., DDL, analyze, import).<\/p>\n\n\n\n

Background tasks each have their own default priority level. In TiDB 7.5, operators of TiDB can choose whether to leave each at their default priority or add them to the default resource group, which can have its \u201cmedium\u201d priority level altered to \u201clow\u201d. This enhancement allows operators to alter the priority level of specific background tasks. <\/p>\n\n\n\n

In the future, this feature will allow for each resource group to change the priority level of background tasks issued by users of that resource group. This will give users much more control over the behavior of background tasks across the cluster.<\/p>\n\n\n\n

The below table illustrates the impact on a foreground workload of having the \u201canalyze\u201d background task running with default priority versus running with background\/low priority:<\/p>\n\n\n\n

<\/p>\n\n\n\n

case<\/strong><\/td>threads<\/strong><\/td>qps<\/strong><\/td>P999<\/strong><\/td>P99<\/strong><\/td>P95<\/strong><\/td>P80<\/strong><\/td><\/tr>
oltp_read_only<\/td>50<\/td>32900<\/td>15.6ms<\/td>7.92ms<\/td>6.74ms<\/td>1.85ms<\/td><\/tr>
oltp_read_only + analyze (default)<\/td>50<\/td>18800<\/td>31.2ms<\/td>23.9ms<\/td>13.7ms<\/td>2.7ms<\/td><\/tr>
oltp_read_only + analyze (background)<\/td>50<\/td>27200<\/td>26.6ms<\/td>14.7ms<\/td>7.56ms<\/td>1.97ms<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

In the above example, the first row shows the performance when all cluster resources are available to the foreground workload. The second row reveals what happens when you introduce the analyze task in the background. Finally, the third row shows what happens when you throttle that task automatically with this new feature.<\/p>\n\n\n\n

This is a huge win for further achieving stability at scale.<\/p>\n\n\n\n

<\/span>TiDB 7.5 Isolates Background Tasks to Designated Nodes<\/span><\/h2>\n\n\n\n

Staying with the theme, the Resource Control feature now allows operators to isolate the compute resources for background tasks by node. You can simply set the system variable `tidb_service_scope` to `background` on the nodes you want to run background tasks. Any nodes not designated this way will not be eligible for computing background tasks.<\/p>\n\n\n\n

The implications here are significant. While you could designate existing nodes for this work \u2013 perhaps larger ones \u2013 the big win here is being able to dynamically add compute nodes with these designations to handle ad hoc background tasks. Want to import a huge table? Add 5 TiDB nodes to your cluster to do it and incur zero extra cost on existing compute nodes. Want to add an index? Same thing. You can tear them down when done and automated background tasks will begin to run on the original nodes again, as none will have the background designation anymore. <\/p>\n\n\n\n

This enhancement should be a seamless way to do big jobs on production clusters without headache.<\/p>\n\n\n\n

<\/span>TiDB 7.5 Catches and Acts on Unexpected Costly Queries<\/span><\/h2>\n\n\n\n

Resource Control does not solve all stability problems. Workloads within resource groups can still affect other workloads in the same group, especially since most workloads have a degree of unpredictability. To solve this, TiDB 7.2 and TiDB 7.3 added capabilities for catching and acting on unexpectedly expensive (\u201crunaway\u201d) queries. These capabilities add stability protection in a way Resource Control cannot. <\/p>\n\n\n\n

Operators can now set any number of query \u201cwatchers\u201d (observable in a \u201cwatchlist\u201d) per resource group with a few critical properties. The first property is a query duration threshold. Any query that breaks this threshold is labeled a runaway. The second property is a declaration of what action to take on queries deemed runaways\u2014either cool them down or cancel them. <\/p>\n\n\n\n

The last property solves the big remaining problem: That waiting for queries to break a duration threshold still uses a ton of resources while they\u2019re under that threshold. For that, watchers also have a rule for matching incoming queries\u2014by exact SQL or their plan digests\u2014to currently labeled runaways. Matches will be labeled as runaway and also treated by the configured action. <\/p>\n\n\n\n

Please note that if you already know of bad queries or digests that haven\u2019t come through yet (i.e., haven\u2019t been caught), you can manually watch for those as well. This is similar to having a database-level SQL blocklist.<\/p>\n\n\n\n

This feature is ideal for data infrastructure and platform teams who have internal customers with queries that can and should be reworked. This is a guardrail against those. <\/p>\n\n\n\n

Combining this feature with Resource Control means more stability between workloads and within them, minimizing any workload consolidation vulnerabilities.<\/p>\n\n\n\n

<\/span>Get Started and Learn More<\/span><\/h2>\n\n\n\n

We\u2019re excited to deliver these new innovative enterprise-grade features and enhancements in TiDB 7.5, the latest LTS version of our flagship database. Discover more, or try TiDB today:<\/p>\n\n\n\n