{"id":189,"date":"2020-04-13T00:00:00","date_gmt":"2020-04-13T00:00:00","guid":{"rendered":"https:\/\/en.pingcap.com\/blog\/back-up-and-restore-a-10-tb-cluster-at-1-gb-per-second\/"},"modified":"2025-03-04T06:12:00","modified_gmt":"2025-03-04T14:12:00","slug":"back-up-and-restore-a-10-tb-cluster-at-1-gb-per-second","status":"publish","type":"post","link":"https:\/\/www.pingcap.com\/ko\/blog\/back-up-and-restore-a-10-tb-cluster-at-1-gb-per-second\/","title":{"rendered":"How to Back Up and Restore a 10-TB Cluster at 1+ GB\/s"},"content":{"rendered":"

Backing up or restoring large-scale distributed databases is time-consuming. When it takes a lot of time to backup or restore a database, Garbage Collection might break the snapshot used in the backup or restore process. Thus, some changes might be missing. This threatens data safety.<\/p>\n

As an open-source, distributed SQL database, \ud2f0DB<\/a> fulfills the requirement for backing up and restoring large-scale clusters<\/strong>. TiDB 4.0 release candidate (RC)<\/a> introduced Backup & Restore<\/a> (BR), a distributed backup and restore tool, that offers high backup and restore speeds<\/strong>\u20141 GB\/s or more for 10 TB of data.<\/p>\n

In this post, I’ll share how we use BR to improve backup and restore speeds for large-scale TiDB clusters. If you specialize in database development, this post might inspire your backup and restore design work. If your application deals with huge amounts of data and you also attach great importance to data security and the efficiency of backup and restore, you can try \ud2f0DB<\/a> with BR<\/a>.<\/p>\n

<\/span>Why Backup & Restore?<\/span><\/h2>\n

TiDB is an open-source, MySQL-compatible, NewSQL<\/a> database that supports Hybrid Transactional\/Analytical Processing<\/a> (HTAP) workloads. Backup and restore has long been a challenging problem for large-scale TiDB clusters. Because data is distributed in different nodes, TiDB doesn’t have a consistent physical snapshot.<\/p>\n

Mydumper and myloader are widely used as backup and restore tools in the MySQL community. Since TiDB is compatible with the MySQL protocol<\/a>, we used them too.<\/p>\n

However, mydumper does not work well with very large TiDB clusters. It cannot reasonably utilize cluster resources to improve backup speed. This seriously affects application requests. It might even cause TiDB’s out of memory (OOM) errors.<\/p>\n

We also had issues restoring data. We optimized Loader<\/a>, a tool similar to myloader, for TiDB. According to our test<\/a>, it took about 19 hours for Loader to restore 1 TB of data. This speed does not meet our requirement for high restore efficiency. The main reason is that we execute SQL INSERT statements to restore data. The whole process is long and involves too many unnecessary calculations. Therefore, resources cannot be fully utilized.<\/p>\n

In short, although we can use mydumper and Loader, they do not perfectly meet our needs. Therefore, we developed a new backup and restore tool, BR.<\/p>\n

<\/span>Backup & Restore’s key features<\/span><\/h2>\n

BR is a command-line tool for fast distributed backup and restore of the TiDB cluster data. Compared with mydumper<\/a> and Loader, BR is more suitable for TiDB clusters with 1TB of data or more.<\/p>\n

In this section, I’ll briefly introduce BR’s key features: horizontal scalability and strong consistency.<\/p>\n

First, however, it’s helpful to learn a bit about TiDB’s architecture<\/a>. Inside the TiDB platform, the main components are as follows:<\/p>\n