{"id":3691,"date":"2021-12-14T19:18:00","date_gmt":"2021-12-14T19:18:00","guid":{"rendered":"https:\/\/en.pingcap.com\/?p=3691"},"modified":"2024-07-02T09:55:25","modified_gmt":"2024-07-02T16:55:25","slug":"tidb-operator-source-code-reading-v-backup-and-restore","status":"publish","type":"post","link":"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/","title":{"rendered":"TiDB Operator Source Code Reading (V): Backup and Restore"},"content":{"rendered":"<p><strong>Author:<\/strong>&nbsp;<a href=\"https:\/\/github.com\/dragonly\">Elon Li<\/a>&nbsp;(Software Engineer at PingCAP)<\/p>\n\n\n\n<p><strong>Transcreator:<\/strong>&nbsp;<a href=\"https:\/\/github.com\/ran-huang\">Ran Huang<\/a>;&nbsp;<strong>Editor:<\/strong>&nbsp;Tom Dewan<\/p>\n\n\n\n<p><em>Previous articles in this series:<\/em><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><em><a href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-1-overview\/\" target=\"_blank\" rel=\"noreferrer noopener\">TiDB Operator Source Code Reading (I): Overview<\/a><\/em><\/li>\n\n\n\n<li><em><a href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-2-operator-pattern\/\" target=\"_blank\" rel=\"noreferrer noopener\">TiDB Operator Source Code Reading (II): Operator Pattern<\/a><\/em><\/li>\n\n\n\n<li><em><a href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-3-component-control-loop\/\" target=\"_blank\" rel=\"noreferrer noopener\">TiDB Operator Source Code Reading (III): The Component Control Loop<\/a><\/em><\/li>\n\n\n\n<li><em><a href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/\" target=\"_blank\" rel=\"noreferrer noopener\">TiDB Operator Source Code Reading (IV): Implementing a Component Control Loop<\/a><\/em><\/li>\n<\/ul>\n\n\n\n<p>In our last article, we learned how to&nbsp;<a href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/\" target=\"_blank\" rel=\"noreferrer noopener\">implement a component control loop<\/a>&nbsp;in TiDB Operator. This time, I&#8217;ll move on to a new but important topic: backup and restore.<\/p>\n\n\n\n<p>Backup and restore are two of the most important and frequently used operations when you maintain a database. To ensure data safety, database maintainers usually need a set of scripts that automatically back up the data and recover the dataset when data is corrupted. A well-designed backup and restore platform should allow you to:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Choose different data sources and target storage.<\/li>\n\n\n\n<li>Execute backup jobs on scheduled time.<\/li>\n\n\n\n<li>Maintain the backup and restore history for auditing.<\/li>\n\n\n\n<li>Clean up obsolete backup files to save storage space.<\/li>\n<\/ul>\n\n\n\n<p>TiDB Operator provides CustomResourceDefinitions (CRDs) for all these requirements. In this post,&nbsp;<strong>I&#8217;ll walk you through the core design logic of TiDB Operator&#8217;s backup and restore features<\/strong>, leaving out the trivial implementation details. Let&#8217;s get started.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"controllers\"><span class=\"ez-toc-section\" id=\"Controllers\"><\/span><a href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#controllers\"><\/a>Controllers<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>TiDB Operator performs ad-hoc backup, restore, and scheduled backup via custom resources (CRs) such as&nbsp;<code>Backup<\/code>,&nbsp;<code>Restore<\/code>, \uadf8\ub9ac\uace0&nbsp;<code>BackupSchedule<\/code>, so we implement three corresponding controllers to execute the control loops.<\/p>\n\n\n\n<p>When a user needs to start a backup job, they can create a YAML file as follows and submit it to Kubernetes. For example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># An example backup job\napiVersion: pingcap.com\/v1alpha1\nkind: Backup\nmetadata:\n  name: demo-backup-gcp\n  namespace: test1\nspec:\n  br:\n    cluster: mycluster\n  gcs:\n    projectId: gcp\n    location: us-west2\n    bucket: backup\n    prefix: test1-demo1\n    secretName: gcp-secret<\/code><\/pre>\n\n\n\n<p>When the backup controller receives an event that creates the&nbsp;<code>Backup<\/code>&nbsp;resource, it creates a job to do the configured backup operation. In the case above, it backs up data in the&nbsp;<code>mycluster<\/code>&nbsp;database in the&nbsp;<code>test1<\/code>&nbsp;namespace and stores the data in the GCP storage specified in the&nbsp;<code>gcs<\/code>&nbsp;field.<\/p>\n\n\n\n<p>In the following sections, I&#8217;ll explain the internal logic of the three controllers.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"the-backup-controller\"><a href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#the-backup-controller\"><\/a>The backup controller<\/h3>\n\n\n\n<p>The backup controller manages the&nbsp;<code>Backup<\/code>&nbsp;CR. Based on the configuration in the&nbsp;<code>spec<\/code>&nbsp;field, the controller uses&nbsp;<a href=\"https:\/\/docs.pingcap.com\/tidb\/stable\/backup-and-restore-tool\" target=\"_blank\" rel=\"noreferrer noopener\">BR<\/a>&nbsp;\ub610\ub294&nbsp;<a href=\"https:\/\/docs.pingcap.com\/tidb\/stable\/dumpling-overview\" target=\"_blank\" rel=\"noreferrer noopener\">Dumpling<\/a>&nbsp;to perform the backup task and deletes the corresponding backup file when the user deletes the&nbsp;<code>Backup<\/code>&nbsp;CR.<\/p>\n\n\n\n<p>Similar to that of other controllers,&nbsp;<strong>the core of the backup controller is a control loop<\/strong>, which listens to the&nbsp;<code>Backup<\/code>&nbsp;CR events (create, update, and delete) and runs the required operations.<\/p>\n\n\n\n<p>In this section, I&#8217;ll skip the generic control loop logic and focus on the core backup logic.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"core-logic\"><a href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#core-logic\"><\/a>Core logic<\/h4>\n\n\n\n<p>The core logic of the backup controller is implemented in the&nbsp;<code>syncBackupJob<\/code>&nbsp;function in the&nbsp;<code>pkg\/backup\/backup\/backup_manager.go<\/code>&nbsp;file. The actual code processes many corner cases; to facilitate understanding, we removed the unimportant details, so you may see some function signatures are inconsistent. The core logic code is as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>backuputil.ValidateBackup(backup)\nif err := JobLister.Jobs(ns).Get(backupJobName); err == nil {\n    return nil\n} else if !errors.IsNotFound(err) {\n    return err\n}\nif backup.Spec.BR == nil {\n    \/\/ create Job Spec which will use Dumpling to do the work\n    job = bm.makeExportJob(backup)\n} else {\n    \/\/ create Job Spec which will use BR to do the work\n    job = bm.makeBackupJob(backup)\n}\nif err := bm.deps.JobControl.CreateJob(backup, job); err != nil {\n    \/\/ update Backup.Status with error message\n}<\/code><\/pre>\n\n\n\n<p>In the code block above,&nbsp;<code>backup<\/code>&nbsp;is the Go struct converted from the&nbsp;<code>Backup<\/code>&nbsp;YAML created by the user. We use a&nbsp;<code>ValidateBackup<\/code>&nbsp;function to check the validity of the fields in&nbsp;<code>backup<\/code>.<\/p>\n\n\n\n<p>Because the backup task is executed as a Kubernetes-native job and because the controller must ensure idempotency\u2014duplicated executions don&#8217;t affect the end result\u2014it is possible that a job already exists. Therefore, we try to find if there is an existing backup job in the same namespace:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>If a job is found, the&nbsp;<code>if<\/code>&nbsp;statement returns&nbsp;<code>nil<\/code>&nbsp;and stops processing the current backup object.<\/li>\n\n\n\n<li>If a job is not found, the controller proceeds to the next step.<\/li>\n<\/ul>\n\n\n\n<p>Next, the controller decides whether to use BR or Dumpling to perform the backup task and executes the corresponding function to create the Job spec. In this step, if you have configured the&nbsp;<code>br<\/code>&nbsp;field, the controller chooses BR; otherwise, it goes with Dumpling.<\/p>\n\n\n\n<p>Finally, the controller uses&nbsp;<code>CreateJob<\/code>&nbsp;to create the job object and executes the backup operation in the job Pod.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"create-job\"><a href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#create-job\"><\/a>Create Job<\/h4>\n\n\n\n<p>The actual backup job is created by two functions:&nbsp;<code>makeExportJob<\/code>&nbsp;\uadf8\ub9ac\uace0&nbsp;<code>makeBackupJob<\/code>. I&#8217;ll take&nbsp;<code>makeBackupJob<\/code>&nbsp;as an example and explain its core code (which is simplified below):<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>tc := bm.deps.TiDBClusterLister.TidbClusters(backupNamespace).Get(backup.Spec.BR.Cluster)\nenvVars := backuputil.GenerateTidbPasswordEnv(ns, name, backup, bm)\nenvVars = append(envVars, backuputil.GenerateStorageCertEnv(ns, backup, bm))\nargs := &#91;]string{\"backup\", fmt.Sprintf(\"--namespace=%s\", ns), fmt.Sprintf(\"--backupName=%s\", name)}\npodSpec := &amp;corev1.PodTemplateSpec{\n    Spec: corev1.PodSpec{\n        InitContainers: &#91;]corev1.Container{{\n            Image:   \"pingcap\/br\",\n            Command: &#91;]string{\"\/bin\/sh\", \"-c\"},\n            Args:    &#91;]string{fmt.Sprintf(\"cp \/br %s\/br; echo 'BR copy finished'\", util.BRBinPath)},\n        }},\n        Containers: &#91;]corev1.Container{{\n            Image:           \"pingcap\/tidb-backup-manager\",\n            Args:            args,\n            Env:             util.AppendEnvIfPresent(envVars, \"TZ\"),\n        }},\n        Volumes:          volumes,\n    },\n}\njob := &amp;batchv1.Job{\n    Spec: batchv1.JobSpec{\n        Template:     *podSpec,\n    },\n}<\/code><\/pre>\n\n\n\n<p>In the code above, the controller:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Obtains the&nbsp;<code>TidbCluster<\/code>&nbsp;resources according to the backup information.<\/li>\n\n\n\n<li>Sets the environment variables and command-line arguments.<\/li>\n\n\n\n<li>Constructs a job using&nbsp;<code>pingcap\/tidb-backup-manager<\/code>&nbsp;as the image to perform the backup task. In this process, the&nbsp;<code>pingcap\/br<\/code>&nbsp;image is used as the init container so that BR is copied to the actual working container.<\/li>\n\n\n\n<li>Executes the&nbsp;<a href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#heading=h.g8pxm2t1s6sp\" target=\"_blank\" rel=\"noreferrer noopener\">backup-manager<\/a>&nbsp;logic after the job is created.<\/li>\n<\/ol>\n\n\n\n<p>The&nbsp;<code>makeExportJob<\/code>&nbsp;function takes similar steps, except that it uses Dumpling rather than BR.<\/p>\n\n\n\n<div class=\"wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex\">\n<div class=\"wp-block-button\"><a class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/www.pingcap.com\/ko\/download\/\">Download TiDB<\/a><\/div>\n\n\n\n<div class=\"wp-block-button\"><a class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/share.hsforms.com\/1e2W03wLJQQKPd1d9rCbj_Q2npzm\">Subscribe to Blog<\/a><\/div>\n<\/div>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"clean-backup-files\"><a href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#clean-backup-files\"><\/a>Clean backup files<\/h4>\n\n\n\n<p>After a user deletes the&nbsp;<code>Backup<\/code>&nbsp;resource, the backup controller deletes the backup files and releases the storage space.<\/p>\n\n\n\n<p>As mentioned in the previous section, the backup controller creates a job when it receives the create event. Meanwhile, the controller adds a string (<code>tidb.pingcap.com\/backup-protection<\/code>) to&nbsp;<code>finalizers<\/code>&nbsp;of the&nbsp;<code>Backup<\/code>&nbsp;resource, which signals that the resource needs special treatment when deleted. When a user deletes the&nbsp;<code>Backup<\/code>&nbsp;resource, the API server sets a value for the&nbsp;<code>metadata.deletionTimestamp<\/code>&nbsp;field. The backup controller then checks the clean policy of the&nbsp;<code>Backup<\/code>&nbsp;resource. If the clean policy isn&#8217;t&nbsp;<code>Retain<\/code>, the controller creates a clean job to delete the backup files and reclaim the storage space.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"the-restore-controller\"><a href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#the-restore-controller\"><\/a>The restore controller<\/h3>\n\n\n\n<p>Restoration is the reverse process of backup, and their core logic is similar.<\/p>\n\n\n\n<p>The restore controller checks the validity of the&nbsp;<code>Restore<\/code>&nbsp;field and then calls the&nbsp;<code>makeImportJob<\/code>&nbsp;function (using TiDB Lightning) or the&nbsp;<code>makeRestoreJob<\/code>&nbsp;function (using BR) to create the restore job. The restore job also uses the backup-manager as a basic image.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"the-backupschedule-controller\"><a href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#the-backupschedule-controller\"><\/a>The backupschedule controller<\/h3>\n\n\n\n<p>To ensure data safety, we usually need to automatically back up important data on a scheduled basis. The backupschedule controller implements such functionality.<\/p>\n\n\n\n<p>A user can schedule a backup job in Cron format and configure backup details similar to ad-hoc backup. After the configuration is submitted to the API server, TiDB Operator runs a scheduled backup with this configuration.<\/p>\n\n\n\n<p>To avoid taking up a lot of storage, a user can also decide on the maximum number of copies to save or the maximum time to keep them. TiDB Operator deletes the outdated backup files.<\/p>\n\n\n\n<p>The design logic of the backupschedule controller uses the existing functionality of the backup controller and encapsulates an abstraction layer for scheduled execution. When TiDB Operator runs a scheduled backup task, the backupschedule controller simply creates a&nbsp;<code>Backup<\/code>&nbsp;resource and has the backup controller do the rest.<\/p>\n\n\n\n<p>You may have noticed that this design is similar to the relationship between&nbsp;<a href=\"https:\/\/kubernetes.io\/docs\/concepts\/workloads\/controllers\/cron-jobs\/\" target=\"_blank\" rel=\"noreferrer noopener\">CronJob<\/a>&nbsp;\uadf8\ub9ac\uace0&nbsp;<a href=\"https:\/\/kubernetes.io\/docs\/concepts\/workloads\/controllers\/job\/\" target=\"_blank\" rel=\"noreferrer noopener\">the Job controller<\/a>. In fact, the implementation of the backupschedule controller is borrowed from CronJob, especially the logic that determines the next execution time.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"core-logic-1\"><a href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#core-logic-1\"><\/a>Core logic<\/h4>\n\n\n\n<p>The core logic of the backupschedule controller is implemented in the&nbsp;<code>Sync<\/code>&nbsp;function in the&nbsp;<code>pkg\/backup\/backupschedule\/backup_schedule_manager.go<\/code>&nbsp;file. The main logic is as follows. For implementation details, you can refer to the source code.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>func (bm *backupScheduleManager) Sync(bs *v1alpha1.BackupSchedule) error {\n    defer bm.backupGC(bs)\n    if err := bm.canPerformNextBackup(bs); err != nil { \n        return err \n    }    \n    scheduledTime, err := getLastScheduledTime(bs, bm.now)\n    if err := bm.deleteLastBackupJob(bs); err != nil { \n        return nil \n    }\n    backup, err := createBackup(bm.deps.BackupControl, bs, *scheduledTime)\n    bs.Status.LastBackup = backup.GetName()\n    bs.Status.LastBackupTime = &amp;metav1.Time{Time: *scheduledTime}\n    bs.Status.AllBackupCleanTime = nil\n    return nil\n}<\/code><\/pre>\n\n\n\n<p>The code first calls the&nbsp;<code>canPerformNextBackup<\/code>&nbsp;function to determine if a new&nbsp;<code>Backup<\/code>&nbsp;resource should be created to perform a new backup task. If the previous backup has completed or the previous backup has failed, the function agrees to execute the next backup; otherwise, the request is rejected.<\/p>\n\n\n\n<p>After deciding to execute the backup task, the controller call&nbsp;<code>getLastScheduledTime<\/code>&nbsp;to get the next backup execution time.&nbsp;<code>getLastScheduledTime<\/code>&nbsp;calculates the last Cron time just before the current time, based on the current time and the Cron settings.&nbsp;<code>getLastScheduledTime<\/code>&nbsp;also handles a lot of boundary conditions; you can check the source code if you&#8217;re interested to know more.<\/p>\n\n\n\n<p>When the controller gets the backup time, it calls the&nbsp;<code>createBackup<\/code>&nbsp;function to create the&nbsp;<code>Backup<\/code>&nbsp;resource, thus leaving the actual backup operation to the backup controller.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"backup-manager\"><span class=\"ez-toc-section\" id=\"backup-manager\"><\/span><a href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#backup-manager\"><\/a>backup-manager<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>To adapt TiDB Operator to the Kubernetes execution environment, we abstracted a backup-manager on top of BR, Dumpling, and TiDB Lightning. The backup-manager provides a uniform encapsulation of entry parameters for the tools. When each controller creates a job resource, it uses the backup-manager as an image. The backup-manager starts the corresponding tools to perform backup or restore tasks with the container start parameters and the&nbsp;<code>Backup<\/code>\/<code>Restore<\/code>&nbsp;resource specs. Moreover, the backup-manager syncs the status of the&nbsp;<code>Backup<\/code>&nbsp;\uadf8\ub9ac\uace0&nbsp;<code>Restore<\/code>&nbsp;resources and updates their progress.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"backuprestore-using-br\"><a href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#backuprestore-using-br\"><\/a>Backup\/restore using BR<\/h3>\n\n\n\n<p>This section explains how the backup-manager implements the main logic of backup and restore. When the backup controller calls the&nbsp;<code>makeBackupJob<\/code>&nbsp;function to create a backup job, the Job controller starts a Pod to run the task, using the backup-manager as the image. The first container start parameter passed to&nbsp;<code>makeBackupJob<\/code>&nbsp;is&nbsp;<code>backup<\/code>; the&nbsp;<code>backup<\/code>&nbsp;processing logic is located in&nbsp;<strong><code>cmd\/backup-manager\/app\/cmd\/backup.go<\/code><\/strong>.<\/p>\n\n\n\n<p>Similar to controllers, when the Pod is started, the backup-manager constructs a series of common Kubernetes client objects, such as Informer, Lister, and Updater. Then it calls the&nbsp;<code>ProcessBackup<\/code>&nbsp;function and performs the backup. The simplified main logic is as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>    backup, err := bm.backupLister.Backups(bm.Namespace).Get(bm.ResourceName)\n    if backup.Spec.From == nil {\n        return bm.performBackup(ctx, backup.DeepCopy(), nil)\n    }\nfunc (bm *Manager) performBackup(ctx context.Context, backup *v1alpha1.Backup, db *sql.DB) error {\n    \/\/ update status to BackupRunning\n    backupFullPath, err := util.GetStoragePath(backup)\n    backupErr := bm.backupData(ctx, backup)\n    \/\/ update status to BackupComplete\n}\nfunc (bo *Options) backupData(ctx context.Context, backup *v1alpha1.Backup) error {\n    clusterNamespace := backup.Spec.BR.ClusterNamespace\n    args := make(&#91;]string, 0)\n    args = append(args, fmt.Sprintf(\"--pd=%s-pd.%s:2379\", backup.Spec.BR.Cluster, clusterNamespace))\n    dataArgs, err := constructOptions(backup)\n    args = append(args, dataArgs...)\n    fullArgs := &#91;]string{\"backup\", backupType}\n    fullArgs = append(fullArgs, args...)\n    klog.Infof(\"Running br command with args: %v\", fullArgs)\n    bin := path.Join(util.BRBinPath, \"br\")\n    cmd := exec.CommandContext(ctx, bin, fullArgs...)\n    \/\/ parse error messages\n}<\/code><\/pre>\n\n\n\n<p><code>ProcessBackup<\/code>&nbsp;runs on a simple logic:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>It obtains the&nbsp;<code>Backup<\/code>&nbsp;object in the corresponding namespace and calls&nbsp;<code>performBackup<\/code>.<\/li>\n\n\n\n<li>The&nbsp;<code>performBackup<\/code>&nbsp;function gets the backup storage path and calls the&nbsp;<code>backupData<\/code>&nbsp;function. We support three backup paths:&nbsp;<code>s3<\/code>,&nbsp;<code>gcs<\/code>, \uadf8\ub9ac\uace0&nbsp;<code>local<\/code>.&nbsp;<code>local<\/code>&nbsp;refers to the local path mounted by persistent volumes (PVs).<\/li>\n\n\n\n<li><code>backupData<\/code>&nbsp;uses BR to perform the backup. This function combines the command-line parameters BR needs and uses&nbsp;<code>backup<\/code>&nbsp;as the command to run the BR binary and parse error messages.<\/li>\n<\/ol>\n\n\n\n<p>After the&nbsp;<code>backup<\/code>&nbsp;command runs successfully, the backup task is complete.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"importexport-using-dumpling-and-tidb-lightning\"><a href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#importexport-using-dumpling-and-tidb-lightning\"><\/a>Import\/export using Dumpling and TiDB Lightning<\/h3>\n\n\n\n<p>Import and export are similar to backup and restore using BR, except that the tools involved are Dumpling and TiDB Lightning. You can check the previous section for details.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"clean\"><a href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#clean\"><\/a>Clean<\/h3>\n\n\n\n<p>When the backup controller creates a job to execute the&nbsp;<strong>clean<\/strong>&nbsp;command, the backup-manager executes the clean logic to delete the corresponding backup files. In&nbsp;<code>cmd\/backup-manager\/app\/cmd\/clean.go<\/code>, the clean logic calls the&nbsp;<code>ProcessCleanBackup<\/code>&nbsp;function to start the clean process.<\/p>\n\n\n\n<p>After a series of checks, the controller calls&nbsp;<code>cleanBRRemoteBackupData<\/code>&nbsp;\ub610\ub294&nbsp;<code>cleanRemoteBackupData<\/code>&nbsp;to delete the BR or Dumpling backup files from remote storage.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"summary\"><span class=\"ez-toc-section\" id=\"Summary\"><\/span><a href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#summary\"><\/a>Summary<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>In this post, we talked about the design and implementation of TiDB Operator&#8217;s backup and restore features. When a user creates a backup or restore task, the corresponding backup, restore, or backupschedule controller calls the backup-manager to run the actual operations. The backupschedule controller encapsulates the timing task logic based on the backup controller, while the backup-manager encapsulates the specific tools into a unified portal for each controller to call.<\/p>\n\n\n\n<p>If you are interested in learning more about TiDB Operator, feel free to&nbsp;<a href=\"https:\/\/slack.tidb.io\/invite?team=tidb-community&amp;channel=sig-k8s&amp;ref=pingcap-blog\">join our Slack channel<\/a>&nbsp;or join our discussions at&nbsp;<a href=\"https:\/\/github.com\/pingcap\/tidb-operator\">pingcap\/tidb-operator<\/a>.<\/p>","protected":false},"excerpt":{"rendered":"<p>Author:&nbsp;Elon Li&nbsp;(Software Engineer at PingCAP) Transcreator:&nbsp;Ran Huang;&nbsp;Editor:&nbsp;Tom Dewan Previous articles in this series: In our last article, we learned how to&nbsp;implement a component control loop&nbsp;in TiDB Operator. This time, I&#8217;ll move on to a new but important topic: backup and restore. Backup and restore are two of the most important and frequently used operations when [&hellip;]<\/p>\n","protected":false},"author":8,"featured_media":3692,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ub_ctt_via":"","footnotes":""},"categories":[6],"tags":[33,84],"class_list":["post-3691","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-engineering","tag-kubernetes","tag-tidb-operator"],"acf":[],"featured_image_src":"https:\/\/static.pingcap.com\/files\/2021\/12\/tidb-operator-source-code-reading-v-backup-and-restore.png","author_info":{"display_name":"TiDB Team","author_link":"https:\/\/www.pingcap.com\/ko\/blog\/author\/pingcap\/"},"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.9 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>TiDB Operator Source Code Reading (V): Backup and Restore | TiDB<\/title>\n<meta name=\"description\" content=\"In this post,\u00a0we will walk you through the core design logic of TiDB Operator&#039;s backup and restore features.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/\" \/>\n<meta property=\"og:locale\" content=\"ko_KR\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"TiDB Operator Source Code Reading (V): Backup and Restore | TiDB\" \/>\n<meta property=\"og:description\" content=\"In this post,\u00a0we will walk you through the core design logic of TiDB Operator&#039;s backup and restore features.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/\" \/>\n<meta property=\"og:site_name\" content=\"TiDB\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/facebook.com\/pingcap2015\" \/>\n<meta property=\"article:published_time\" content=\"2021-12-14T19:18:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-07-02T16:55:25+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/static.pingcap.com\/files\/2021\/12\/tidb-operator-source-code-reading-v-backup-and-restore.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1501\" \/>\n\t<meta property=\"og:image:height\" content=\"501\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"TiDB Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@PingCAP\" \/>\n<meta name=\"twitter:site\" content=\"@PingCAP\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"TiDB Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"10\ubd84\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/\"},\"author\":{\"name\":\"TiDB Team\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/person\/b17c1fde961eebd318de8729d595df74\"},\"headline\":\"TiDB Operator Source Code Reading (V): Backup and Restore\",\"datePublished\":\"2021-12-14T19:18:00+00:00\",\"dateModified\":\"2024-07-02T16:55:25+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/\"},\"wordCount\":1850,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.pingcap.com\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/static.pingcap.com\/files\/2021\/12\/tidb-operator-source-code-reading-v-backup-and-restore.png\",\"keywords\":[\"Kubernetes\",\"TiDB Operator\"],\"articleSection\":[\"Engineering\"],\"inLanguage\":\"ko-KR\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/\",\"url\":\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/\",\"name\":\"TiDB Operator Source Code Reading (V): Backup and Restore | TiDB\",\"isPartOf\":{\"@id\":\"https:\/\/www.pingcap.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/static.pingcap.com\/files\/2021\/12\/tidb-operator-source-code-reading-v-backup-and-restore.png\",\"datePublished\":\"2021-12-14T19:18:00+00:00\",\"dateModified\":\"2024-07-02T16:55:25+00:00\",\"description\":\"In this post,\u00a0we will walk you through the core design logic of TiDB Operator's backup and restore features.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#breadcrumb\"},\"inLanguage\":\"ko-KR\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#primaryimage\",\"url\":\"https:\/\/static.pingcap.com\/files\/2021\/12\/tidb-operator-source-code-reading-v-backup-and-restore.png\",\"contentUrl\":\"https:\/\/static.pingcap.com\/files\/2021\/12\/tidb-operator-source-code-reading-v-backup-and-restore.png\",\"width\":1501,\"height\":501},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.pingcap.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"TiDB Operator Source Code Reading (V): Backup and Restore\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.pingcap.com\/#website\",\"url\":\"https:\/\/www.pingcap.com\/\",\"name\":\"TiDB\",\"description\":\"TiDB | SQL at Scale\",\"publisher\":{\"@id\":\"https:\/\/www.pingcap.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.pingcap.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"ko-KR\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.pingcap.com\/#organization\",\"name\":\"PingCAP\",\"url\":\"https:\/\/www.pingcap.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png\",\"contentUrl\":\"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png\",\"width\":811,\"height\":232,\"caption\":\"PingCAP\"},\"image\":{\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/facebook.com\/pingcap2015\",\"https:\/\/x.com\/PingCAP\",\"https:\/\/linkedin.com\/company\/pingcap\",\"https:\/\/youtube.com\/channel\/UCuq4puT32DzHKT5rU1IZpIA\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/person\/b17c1fde961eebd318de8729d595df74\",\"name\":\"TiDB Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg\",\"contentUrl\":\"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg\",\"caption\":\"TiDB Team\"},\"url\":\"https:\/\/www.pingcap.com\/ko\/blog\/author\/pingcap\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"TiDB Operator Source Code Reading (V): Backup and Restore | TiDB","description":"In this post,\u00a0we will walk you through the core design logic of TiDB Operator's backup and restore features.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/","og_locale":"ko_KR","og_type":"article","og_title":"TiDB Operator Source Code Reading (V): Backup and Restore | TiDB","og_description":"In this post,\u00a0we will walk you through the core design logic of TiDB Operator's backup and restore features.","og_url":"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/","og_site_name":"TiDB","article_publisher":"https:\/\/facebook.com\/pingcap2015","article_published_time":"2021-12-14T19:18:00+00:00","article_modified_time":"2024-07-02T16:55:25+00:00","og_image":[{"width":1501,"height":501,"url":"https:\/\/static.pingcap.com\/files\/2021\/12\/tidb-operator-source-code-reading-v-backup-and-restore.png","type":"image\/png"}],"author":"TiDB Team","twitter_card":"summary_large_image","twitter_creator":"@PingCAP","twitter_site":"@PingCAP","twitter_misc":{"Written by":"TiDB Team","Est. reading time":"10\ubd84"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#article","isPartOf":{"@id":"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/"},"author":{"name":"TiDB Team","@id":"https:\/\/www.pingcap.com\/#\/schema\/person\/b17c1fde961eebd318de8729d595df74"},"headline":"TiDB Operator Source Code Reading (V): Backup and Restore","datePublished":"2021-12-14T19:18:00+00:00","dateModified":"2024-07-02T16:55:25+00:00","mainEntityOfPage":{"@id":"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/"},"wordCount":1850,"commentCount":0,"publisher":{"@id":"https:\/\/www.pingcap.com\/#organization"},"image":{"@id":"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#primaryimage"},"thumbnailUrl":"https:\/\/static.pingcap.com\/files\/2021\/12\/tidb-operator-source-code-reading-v-backup-and-restore.png","keywords":["Kubernetes","TiDB Operator"],"articleSection":["Engineering"],"inLanguage":"ko-KR","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/","url":"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/","name":"TiDB Operator Source Code Reading (V): Backup and Restore | TiDB","isPartOf":{"@id":"https:\/\/www.pingcap.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#primaryimage"},"image":{"@id":"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#primaryimage"},"thumbnailUrl":"https:\/\/static.pingcap.com\/files\/2021\/12\/tidb-operator-source-code-reading-v-backup-and-restore.png","datePublished":"2021-12-14T19:18:00+00:00","dateModified":"2024-07-02T16:55:25+00:00","description":"In this post,\u00a0we will walk you through the core design logic of TiDB Operator's backup and restore features.","breadcrumb":{"@id":"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#breadcrumb"},"inLanguage":"ko-KR","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/"]}]},{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#primaryimage","url":"https:\/\/static.pingcap.com\/files\/2021\/12\/tidb-operator-source-code-reading-v-backup-and-restore.png","contentUrl":"https:\/\/static.pingcap.com\/files\/2021\/12\/tidb-operator-source-code-reading-v-backup-and-restore.png","width":1501,"height":501},{"@type":"BreadcrumbList","@id":"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.pingcap.com\/"},{"@type":"ListItem","position":2,"name":"TiDB Operator Source Code Reading (V): Backup and Restore"}]},{"@type":"WebSite","@id":"https:\/\/www.pingcap.com\/#website","url":"https:\/\/www.pingcap.com\/","name":"\ud2f0DB","description":"TiDB | SQL at Scale","publisher":{"@id":"https:\/\/www.pingcap.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.pingcap.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"ko-KR"},{"@type":"Organization","@id":"https:\/\/www.pingcap.com\/#organization","name":"PingCAP","url":"https:\/\/www.pingcap.com\/","logo":{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/","url":"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png","contentUrl":"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png","width":811,"height":232,"caption":"PingCAP"},"image":{"@id":"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/facebook.com\/pingcap2015","https:\/\/x.com\/PingCAP","https:\/\/linkedin.com\/company\/pingcap","https:\/\/youtube.com\/channel\/UCuq4puT32DzHKT5rU1IZpIA"]},{"@type":"Person","@id":"https:\/\/www.pingcap.com\/#\/schema\/person\/b17c1fde961eebd318de8729d595df74","name":"TiDB Team","image":{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/www.pingcap.com\/#\/schema\/person\/image\/","url":"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg","contentUrl":"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg","caption":"TiDB Team"},"url":"https:\/\/www.pingcap.com\/ko\/blog\/author\/pingcap\/"}]}},"grav_blocks":false,"card_markup":"<a class=\"card-resource bg-white\" href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-v-backup-and-restore\/\"><div class=\"card-resource__image-container\"><img class=\"card-resource__image\" alt=\"tidb-operator-source-code-reading-v-backup-and-restore\" src=\"https:\/\/static.pingcap.com\/files\/2021\/12\/tidb-operator-source-code-reading-v-backup-and-restore.png\" loading=\"lazy\" width=1501 height=501 \/><\/div><div class=\"card-resource__content-container\"><div class=\"card-resource__content-head\"><div class=\"card-resource__category\">Engineering<\/div><\/div><h5 class=\"card-resource__title\">TiDB Operator Source Code Reading (V): Backup and Restore<\/h5><\/div><\/a>","_links":{"self":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts\/3691","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/comments?post=3691"}],"version-history":[{"count":6,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts\/3691\/revisions"}],"predecessor-version":[{"id":18023,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts\/3691\/revisions\/18023"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/media\/3692"}],"wp:attachment":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/media?parent=3691"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/categories?post=3691"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/tags?post=3691"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}