In this post, we\u2019ll dive into why we built Pipelined DML, how it works, and the transformational benefits it brings to managing modern data workloads.<\/p>\n\n\n\n
<\/span>Tackling Large Transactions: Challenges and Opportunities<\/span><\/h2>\n\n\n\nLarge-scale transactions are the backbone of many critical operations\u2014think bulk data updates, system migrations, or ETL workflows where millions of rows need processing. While TiDB excels as a distributed SQL database, handling such transactions at scale brought two significant challenges:<\/p>\n\n\n\n
\n- Memory Limits:<\/strong> Before TiDB 8.1, all transaction mutations were held in memory throughout the transaction\u2019s lifecycle. For operations touching millions of rows, this could lead to high memory usage and, in some cases, Out of Memory (OOM) errors if there weren\u2019t available resources.<\/li>\n\n\n\n
- Performance Slowdowns:<\/strong> Managing large in-memory buffers relied on red-black trees, introducing computational overhead. As buffers grew, their operations slowed due to the O(NlogN) complexity inherent in these structures.<\/li>\n<\/ul>\n\n\n\n
Take a common example: Archiving historical sales data into a separate table:<\/p>\n\n\n\n
INSERT INTO sales_archive \nSELECT * FROM sales \nWHERE sale_date < '2023-01-01';<\/code><\/pre>\n\n\n\nIn such scenarios, holding millions of rows in memory until the transaction commits not only strains resources but also impacts speed. These challenges highlighted a clear opportunity to improve scalability, reduce complexity, and enhance reliability. With the rise of modern data workloads, the TiDB team developed a bold solution: Pipelined DML, designed to transform how large transactions are handled.<\/p>\n\n\n\n
Existing Workarounds: Steps Toward a Complete Solution for Large Transactions<\/h3>\n\n\n\n
Before Pipelined DML was available, TiDB implemented several workarounds to bypass the large transaction problem. These interim solutions, while helpful in specific scenarios, came with trade-offs:<\/p>\n\n\n\n
\n- Batch<\/strong>–<\/strong>DML <\/strong>(now deprecated):<\/strong> Introduced in TiDB v4.0, Batch-DML allowed splitting transactions into smaller parts for individual commits, enabling support for larger data operations. However, as TiDB has evolved with more robust solutions, Batch-DML has been deprecated to ensure higher reliability and maintain data integrity. It is no longer recommended for use.<\/li>\n\n\n\n
- Non-Transactional DML<\/a><\/strong>:<\/strong> The feature was first introduced in v6.1 and became GA in v6.5. TiDB splits a statement into multiple ones and executes them in sequence. It is safe for data integrity, but lacks atomicity and requires users to modify their statements, which often creates additional complexity for users.<\/li>\n<\/ol>\n\n\n\n
These methods showcased TiDB\u2019s flexibility and focus on user needs, even under challenging circumstances. However, they also underscored the need for a more seamless, built-in solution that is atomic, performant, and easy to use<\/strong>. This realization paved the way for Pipelined DML.<\/p>\n\n\n\n<\/span>Pipelined DML: Revolutionizing Large Transactions<\/span><\/h2>\n\n\n\nTo address the growing demands of large-scale data operations, the TiDB team developed Pipelined DML, a transformative enhancement to the original Percolator protocol. This feature changes the game by enabling continuous write flushing to TiKV<\/a>, TiDB\u2019s storage layer, rather than relying solely on in-memory buffers until the commit phase. This shift ensures efficient, scalable, and reliable transaction management. Here\u2019s what makes Pipelined DML a breakthrough:<\/p>\n\n\n\n1. Continuous Flushing: Incremental Writes for Better Memory Management<\/strong><\/h3>\n\n\n\nPipelined DML writes data to TiKV in small, manageable batches, drastically reducing memory usage. This ensures smooth processing, even for large transactions, while eliminating the risk of out-of-memory (OOM) errors.<\/p>\n\n\n\n
Example:<\/em> Picture migrating a massive sales dataset to an archive table in TiDB. Previously, this would require storing the entire dataset in memory before committing, risking resource exhaustion. Now, Pipelined DML writes the data incrementally to TiKV as processed, keeping memory usage steady and workflows uninterrupted.<\/p>\n\n\n\n2. Asynchronous Buffer Management: Parallel Processing for Reduced Latency<\/strong><\/h3>\n\n\n\nBy decoupling transaction execution from storage writes, Pipelined DML allows TiDB to process and write data simultaneously. This parallelism reduces transaction latency and optimizes resource utilization.<\/p>\n\n\n\n
Example:<\/em> Imagine processing millions of real-time log entries for an analytics system. With Pipelined DML, the system can process new entries while simultaneously writing completed logs to storage, enabling faster throughput and seamless handling of high-volume workloads.<\/p>\n\n\n\n3. Smoothed CPU and I\/O Utilization: Consistent Resource Efficiency<\/strong><\/h3>\n\n\n\nTraditional batch processing often leads to spikes in resource usage, which can slow down other operations. Pipelined DML spreads the workload evenly, ensuring TiKV processes data at a steady rate.<\/p>\n\n\n\n
Example:<\/em> Updating global pricing data for a retail platform\u2019s products can be resource-intensive. Pipelined DML ensures these updates happen gradually, avoiding sudden CPU or I\/O bottlenecks and maintaining system stability.<\/p>\n\n\n\nThese innovations make Pipelined DML a cornerstone of TiDB\u2019s ability to meet modern data workload demands. It allows businesses to manage massive transactions with confidence, delivering a scalable, efficient, and reliable solution.<\/p>\n\n\n\n![\"Traditional](\"https:\/\/static.pingcap.com\/files\/2024\/11\/22112643\/image-1024x436.png\")
<\/figure>\n\n\n\nFigure 1: Traditional 2PC processes store all writes in memory until the commit phase (left), while Pipelined DML incrementally flushes writes to TiKV as generated, ensuring minimal memory usage (right).<\/em><\/p>\n\n\n\n<\/span>How Pipelined DML Works: A Step-by-Step Breakdown for Managing Large Transactions<\/span><\/h2>\n\n\n\nPipelined DML integrates seamlessly into TiDB\u2019s architecture, enhancing transaction processing without disrupting established workflows. Let\u2019s break down how it optimizes each phase of the transaction lifecycle:<\/p>\n\n\n\n
1. Execution Phase: Efficient Start to Every Transaction<\/strong><\/h3>\n\n\n\nLike traditional Two-Phase Commit (2PC), the transaction begins with parsing, planning, and executing the SQL statement. However, Pipelined DML introduces a critical difference: instead of holding all writes in memory, it immediately flushes them to TiKV as generated. This approach prevents memory buildup and keeps processing efficient, even for transactions involving millions of rows.<\/p>\n\n\n\n
2. Flushing Mechanism: Incremental Writes for Stability<\/strong><\/h3>\n\n\n\nWrites are sent to TiKV in small, manageable batches. This mechanism serves two vital functions:<\/p>\n\n\n\n
\n- Persistence: <\/strong>By progressively storing writes in TiKV, Pipelined DML significantly reduces memory usage, ensuring stable operations for large transactions.<\/li>\n\n\n\n
- Rate Limiting:<\/strong> If TiDB\u2019s executor generates writes faster than TiKV can process them, the system dynamically slows down the producer. This ensures smooth operation, avoiding memory overloads or processing bottlenecks.<\/li>\n<\/ul>\n\n\n\n
3. Commit Phase: Consistent and Reliable Finalization<\/strong><\/h3>\n\n\n\nTiDB moves to the commit phase after all flushed writes. At this stage, it scans and commits locks associated with the flushed data, ensuring transactional consistency. This approach maintains TiDB’s ACID guarantees, even with highly complex or large-scale workloads.<\/p>\n\n\n\n
Memory Buffer Enhancements: Supporting Continuous Flushing<\/h3>\n\n\n\n
To make Pipelined DML possible, TiDB\u2019s in-memory database (MemDB)\u2014responsible for managing transaction writes\u2014was upgraded to support continuous flushing.<\/p>\n\n\n\n
The enhanced architecture consists of seven principles:<\/p>\n\n\n\n
\n- Dual MemDB States<\/strong>: Each transaction uses one mutable MemDB and at most one immutable MemDB.<\/li>\n\n\n\n
- Writes<\/strong>: All write operations are directed to the mutable MemDB.<\/li>\n\n\n\n
- Reads<\/strong>: Read operations aggregate data from all active MemDBs for accurate results.<\/li>\n\n\n\n
- Flushing<\/strong>: The immutable MemDB handles flushes exclusively.<\/li>\n\n\n\n
- Transition<\/strong>: When no immutable MemDB exists, the mutable MemDB becomes immutable, and a new mutable MemDB is created.<\/li>\n\n\n\n
- Memory release<\/strong>: The immutable MemDB is discarded after flushing completes.<\/li>\n\n\n\n
- Flow Control<\/strong>: Incoming writes are paused when the mutable MemDB grows too large to maintain stability.<\/li>\n<\/ol>\n\n\n\n
These enhancements enable TiDB to efficiently manage concurrent read, write, and flush operations while keeping memory usage under control.<\/p>\n\n\n
\n![\"The](\"https:\/\/static.pingcap.com\/files\/2024\/11\/22112706\/image-1.png\")
<\/figure>\n<\/div>\n\n\nFigure 2: The updated buffer structure supporting continuous flushing, designed to replace the original memory buffer<\/em>.<\/p>\n\n\n\n<\/span>Overcoming Challenges in Pipelined DML<\/span><\/h2>\n\n\n\nIntroducing a feature as transformative as Pipelined DML requires tackling unique technical hurdles. The TiDB team identified and addressed three primary challenges to ensure seamless and reliable operation.<\/p>\n\n\n\n
Challenge 1: Out-of-Order Flush Operations<\/h3>\n\n\n\n
In distributed systems, network instability can cause lost, delayed, or arrive in the wrong order flush operations, risking data inconsistency.<\/p>\n\n\n\n
How <\/strong>\ud2f0DB<\/strong> Solves This: Generation Numbers<\/strong><\/h4>\n\n\n\nEach flush operation is assigned a unique, incrementing generation number to ensure:<\/p>\n\n\n\n
\n- Correct Ordering:<\/strong> TiKV processes writes in sequential order based on their generation numbers.<\/li>\n\n\n\n
- Stale Write Prevention: <\/strong>If a delayed operation arrives after a more recent one, TiKV rejects it, preserving data integrity.<\/li>\n\n\n\n
- Example: <\/em>Imagine Flush Operation 1 (Gen 1) and Flush Operation 2 (Gen 2) are sent correctly but arrive out of sequence. TiKV processes Gen 2 while discarding the outdated Gen 1, ensuring data consistency.<\/li>\n<\/ul>\n\n\n\n
Challenge 2: Slow Point-Read Operations<\/h3>\n\n\n\n
When a requested key-value pair has already been flushed to TiKV and is no longer in TiDB’s memory, retrieving it requires a remote procedure call (RPC) to TiKV, increasing latency.<\/p>\n\n\n\n
How <\/strong>Does TiDB Solve This?<\/strong><\/h4>\n\n\n\n\n- Lazy Check:<\/strong> For cases requiring verification of key non-existence, checks are deferred to the actual write in TiKV. The new
buffer.GetLocal(key)<\/code> method limits lookups to in-memory data, avoiding unnecessary RPCs.<\/li>\n\n\n\n- Prefetching:<\/strong> For scenarios like bulk updates, TiDB preloads key-value pairs into a cache using
BatchGet<\/code>. This cache ensures subsequent buffer.Get(key)<\/code> operations hit the in-memory cache rather than initiating costly RPCs.<\/li>\n<\/ol>\n\n\n\nChallenge 3: Staging Across Nodes<\/h3>\n\n\n\n
TiDB\u2019s staging mechanism allows temporary buffers (stages) to hold committed or rolled back changes. While this is simple in a memory-only setup, extending it across TiDB and TiKV nodes adds complexity.<\/p>\n\n\n\n
How <\/strong>Does TiDB Simplify Staging<\/strong>?<\/h4>\n\n\n\nBy limiting staging operations to occur between flush points, the system ensures:<\/p>\n\n\n\n
\n- All staged changes resolve (committed or rolled back) before TiKV flushes data.<\/li>\n\n\n\n
- Memory-only staging is preserved, simplifying implementation while retaining essential features like
ON DUPLICATE KEY UPDATE<\/code>.<\/li>\n<\/ul>\n\n\n\n<\/span>Performance Enhancements: TiDB Reaches New Heights for Large Transactions<\/span><\/h2>\n\n\n\nWith Pipelined DML, TiDB sets a new standard for processing large transactions, significantly improving speed, memory efficiency, and throughput. Extensive benchmarks of TiDB 8.4 against TiDB 7.5 highlight these advancements in practical, real-world scenarios.<\/p>\n\n\n\n
Highlights of Testing Environment<\/h3>\n\n\n\n
All benchmarks follow the below parameters:<\/p>\n\n\n\n
\n- Workload:<\/strong> YCSB table with 10 million rows.<\/li>\n\n\n\n
- Test Environment:<\/strong> GCP n2-standard-16 machines.<\/li>\n\n\n\n
- Cluster Size: 3 TiKV nodes.<\/li>\n<\/ul>\n\n\n\n
Results demonstrate the practical application of Pipelined DML for scaling operations with TiDB.<\/p>\n\n\n\n
Key Improvements<\/h3>\n\n\n\n
Take a common example: Archiving historical sales data into a separate table:<\/p>\n\n\n\n
INSERT INTO sales_archive \nSELECT * FROM sales \nWHERE sale_date < '2023-01-01';<\/code><\/pre>\n\n\n\nIn such scenarios, holding millions of rows in memory until the transaction commits not only strains resources but also impacts speed. These challenges highlighted a clear opportunity to improve scalability, reduce complexity, and enhance reliability. With the rise of modern data workloads, the TiDB team developed a bold solution: Pipelined DML, designed to transform how large transactions are handled.<\/p>\n\n\n\n
Existing Workarounds: Steps Toward a Complete Solution for Large Transactions<\/h3>\n\n\n\n
Before Pipelined DML was available, TiDB implemented several workarounds to bypass the large transaction problem. These interim solutions, while helpful in specific scenarios, came with trade-offs:<\/p>\n\n\n\n
\n- Batch<\/strong>–<\/strong>DML <\/strong>(now deprecated):<\/strong> Introduced in TiDB v4.0, Batch-DML allowed splitting transactions into smaller parts for individual commits, enabling support for larger data operations. However, as TiDB has evolved with more robust solutions, Batch-DML has been deprecated to ensure higher reliability and maintain data integrity. It is no longer recommended for use.<\/li>\n\n\n\n
- Non-Transactional DML<\/a><\/strong>:<\/strong> The feature was first introduced in v6.1 and became GA in v6.5. TiDB splits a statement into multiple ones and executes them in sequence. It is safe for data integrity, but lacks atomicity and requires users to modify their statements, which often creates additional complexity for users.<\/li>\n<\/ol>\n\n\n\n
These methods showcased TiDB\u2019s flexibility and focus on user needs, even under challenging circumstances. However, they also underscored the need for a more seamless, built-in solution that is atomic, performant, and easy to use<\/strong>. This realization paved the way for Pipelined DML.<\/p>\n\n\n\n<\/span>Pipelined DML: Revolutionizing Large Transactions<\/span><\/h2>\n\n\n\nTo address the growing demands of large-scale data operations, the TiDB team developed Pipelined DML, a transformative enhancement to the original Percolator protocol. This feature changes the game by enabling continuous write flushing to TiKV<\/a>, TiDB\u2019s storage layer, rather than relying solely on in-memory buffers until the commit phase. This shift ensures efficient, scalable, and reliable transaction management. Here\u2019s what makes Pipelined DML a breakthrough:<\/p>\n\n\n\n1. Continuous Flushing: Incremental Writes for Better Memory Management<\/strong><\/h3>\n\n\n\nPipelined DML writes data to TiKV in small, manageable batches, drastically reducing memory usage. This ensures smooth processing, even for large transactions, while eliminating the risk of out-of-memory (OOM) errors.<\/p>\n\n\n\n
Example:<\/em> Picture migrating a massive sales dataset to an archive table in TiDB. Previously, this would require storing the entire dataset in memory before committing, risking resource exhaustion. Now, Pipelined DML writes the data incrementally to TiKV as processed, keeping memory usage steady and workflows uninterrupted.<\/p>\n\n\n\n2. Asynchronous Buffer Management: Parallel Processing for Reduced Latency<\/strong><\/h3>\n\n\n\nBy decoupling transaction execution from storage writes, Pipelined DML allows TiDB to process and write data simultaneously. This parallelism reduces transaction latency and optimizes resource utilization.<\/p>\n\n\n\n
Example:<\/em> Imagine processing millions of real-time log entries for an analytics system. With Pipelined DML, the system can process new entries while simultaneously writing completed logs to storage, enabling faster throughput and seamless handling of high-volume workloads.<\/p>\n\n\n\n3. Smoothed CPU and I\/O Utilization: Consistent Resource Efficiency<\/strong><\/h3>\n\n\n\nTraditional batch processing often leads to spikes in resource usage, which can slow down other operations. Pipelined DML spreads the workload evenly, ensuring TiKV processes data at a steady rate.<\/p>\n\n\n\n
Example:<\/em> Updating global pricing data for a retail platform\u2019s products can be resource-intensive. Pipelined DML ensures these updates happen gradually, avoiding sudden CPU or I\/O bottlenecks and maintaining system stability.<\/p>\n\n\n\nThese innovations make Pipelined DML a cornerstone of TiDB\u2019s ability to meet modern data workload demands. It allows businesses to manage massive transactions with confidence, delivering a scalable, efficient, and reliable solution.<\/p>\n\n\n\n<\/figure>\n\n\n\nFigure 1: Traditional 2PC processes store all writes in memory until the commit phase (left), while Pipelined DML incrementally flushes writes to TiKV as generated, ensuring minimal memory usage (right).<\/em><\/p>\n\n\n\n<\/span>How Pipelined DML Works: A Step-by-Step Breakdown for Managing Large Transactions<\/span><\/h2>\n\n\n\nPipelined DML integrates seamlessly into TiDB\u2019s architecture, enhancing transaction processing without disrupting established workflows. Let\u2019s break down how it optimizes each phase of the transaction lifecycle:<\/p>\n\n\n\n
1. Execution Phase: Efficient Start to Every Transaction<\/strong><\/h3>\n\n\n\nLike traditional Two-Phase Commit (2PC), the transaction begins with parsing, planning, and executing the SQL statement. However, Pipelined DML introduces a critical difference: instead of holding all writes in memory, it immediately flushes them to TiKV as generated. This approach prevents memory buildup and keeps processing efficient, even for transactions involving millions of rows.<\/p>\n\n\n\n
2. Flushing Mechanism: Incremental Writes for Stability<\/strong><\/h3>\n\n\n\nWrites are sent to TiKV in small, manageable batches. This mechanism serves two vital functions:<\/p>\n\n\n\n
\n- Persistence: <\/strong>By progressively storing writes in TiKV, Pipelined DML significantly reduces memory usage, ensuring stable operations for large transactions.<\/li>\n\n\n\n
- Rate Limiting:<\/strong> If TiDB\u2019s executor generates writes faster than TiKV can process them, the system dynamically slows down the producer. This ensures smooth operation, avoiding memory overloads or processing bottlenecks.<\/li>\n<\/ul>\n\n\n\n
3. Commit Phase: Consistent and Reliable Finalization<\/strong><\/h3>\n\n\n\nTiDB moves to the commit phase after all flushed writes. At this stage, it scans and commits locks associated with the flushed data, ensuring transactional consistency. This approach maintains TiDB’s ACID guarantees, even with highly complex or large-scale workloads.<\/p>\n\n\n\n
Memory Buffer Enhancements: Supporting Continuous Flushing<\/h3>\n\n\n\n
To make Pipelined DML possible, TiDB\u2019s in-memory database (MemDB)\u2014responsible for managing transaction writes\u2014was upgraded to support continuous flushing.<\/p>\n\n\n\n
The enhanced architecture consists of seven principles:<\/p>\n\n\n\n
\n- Dual MemDB States<\/strong>: Each transaction uses one mutable MemDB and at most one immutable MemDB.<\/li>\n\n\n\n
- Writes<\/strong>: All write operations are directed to the mutable MemDB.<\/li>\n\n\n\n
- Reads<\/strong>: Read operations aggregate data from all active MemDBs for accurate results.<\/li>\n\n\n\n
- Flushing<\/strong>: The immutable MemDB handles flushes exclusively.<\/li>\n\n\n\n
- Transition<\/strong>: When no immutable MemDB exists, the mutable MemDB becomes immutable, and a new mutable MemDB is created.<\/li>\n\n\n\n
- Memory release<\/strong>: The immutable MemDB is discarded after flushing completes.<\/li>\n\n\n\n
- Flow Control<\/strong>: Incoming writes are paused when the mutable MemDB grows too large to maintain stability.<\/li>\n<\/ol>\n\n\n\n
These enhancements enable TiDB to efficiently manage concurrent read, write, and flush operations while keeping memory usage under control.<\/p>\n\n\n
\n<\/figure>\n<\/div>\n\n\nFigure 2: The updated buffer structure supporting continuous flushing, designed to replace the original memory buffer<\/em>.<\/p>\n\n\n\n<\/span>Overcoming Challenges in Pipelined DML<\/span><\/h2>\n\n\n\nIntroducing a feature as transformative as Pipelined DML requires tackling unique technical hurdles. The TiDB team identified and addressed three primary challenges to ensure seamless and reliable operation.<\/p>\n\n\n\n
Challenge 1: Out-of-Order Flush Operations<\/h3>\n\n\n\n
In distributed systems, network instability can cause lost, delayed, or arrive in the wrong order flush operations, risking data inconsistency.<\/p>\n\n\n\n
How <\/strong>\ud2f0DB<\/strong> Solves This: Generation Numbers<\/strong><\/h4>\n\n\n\nEach flush operation is assigned a unique, incrementing generation number to ensure:<\/p>\n\n\n\n
\n- Correct Ordering:<\/strong> TiKV processes writes in sequential order based on their generation numbers.<\/li>\n\n\n\n
- Stale Write Prevention: <\/strong>If a delayed operation arrives after a more recent one, TiKV rejects it, preserving data integrity.<\/li>\n\n\n\n
- Example: <\/em>Imagine Flush Operation 1 (Gen 1) and Flush Operation 2 (Gen 2) are sent correctly but arrive out of sequence. TiKV processes Gen 2 while discarding the outdated Gen 1, ensuring data consistency.<\/li>\n<\/ul>\n\n\n\n
Challenge 2: Slow Point-Read Operations<\/h3>\n\n\n\n
When a requested key-value pair has already been flushed to TiKV and is no longer in TiDB’s memory, retrieving it requires a remote procedure call (RPC) to TiKV, increasing latency.<\/p>\n\n\n\n
How <\/strong>Does TiDB Solve This?<\/strong><\/h4>\n\n\n\n\n- Lazy Check:<\/strong> For cases requiring verification of key non-existence, checks are deferred to the actual write in TiKV. The new
buffer.GetLocal(key)<\/code> method limits lookups to in-memory data, avoiding unnecessary RPCs.<\/li>\n\n\n\n- Prefetching:<\/strong> For scenarios like bulk updates, TiDB preloads key-value pairs into a cache using
BatchGet<\/code>. This cache ensures subsequent buffer.Get(key)<\/code> operations hit the in-memory cache rather than initiating costly RPCs.<\/li>\n<\/ol>\n\n\n\nChallenge 3: Staging Across Nodes<\/h3>\n\n\n\n
TiDB\u2019s staging mechanism allows temporary buffers (stages) to hold committed or rolled back changes. While this is simple in a memory-only setup, extending it across TiDB and TiKV nodes adds complexity.<\/p>\n\n\n\n
How <\/strong>Does TiDB Simplify Staging<\/strong>?<\/h4>\n\n\n\nBy limiting staging operations to occur between flush points, the system ensures:<\/p>\n\n\n\n
\n- All staged changes resolve (committed or rolled back) before TiKV flushes data.<\/li>\n\n\n\n
- Memory-only staging is preserved, simplifying implementation while retaining essential features like
ON DUPLICATE KEY UPDATE<\/code>.<\/li>\n<\/ul>\n\n\n\n<\/span>Performance Enhancements: TiDB Reaches New Heights for Large Transactions<\/span><\/h2>\n\n\n\nWith Pipelined DML, TiDB sets a new standard for processing large transactions, significantly improving speed, memory efficiency, and throughput. Extensive benchmarks of TiDB 8.4 against TiDB 7.5 highlight these advancements in practical, real-world scenarios.<\/p>\n\n\n\n
Highlights of Testing Environment<\/h3>\n\n\n\n
All benchmarks follow the below parameters:<\/p>\n\n\n\n
\n- Workload:<\/strong> YCSB table with 10 million rows.<\/li>\n\n\n\n
- Test Environment:<\/strong> GCP n2-standard-16 machines.<\/li>\n\n\n\n
- Cluster Size: 3 TiKV nodes.<\/li>\n<\/ul>\n\n\n\n
Results demonstrate the practical application of Pipelined DML for scaling operations with TiDB.<\/p>\n\n\n\n
Key Improvements<\/h3>\n\n\n\n
![\"Steady](\"https:\/\/static.pingcap.com\/files\/2024\/11\/22112736\/image-2-1024x565.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/static.pingcap.com\/files\/2024\/11\/22112850\/image-4-1024x568.png\")
<\/figure>\n\n\n\n