{"id":2321,"date":"2021-10-28T08:00:00","date_gmt":"2021-10-28T08:00:00","guid":{"rendered":"https:\/\/en.pingcap.com\/?p=2321"},"modified":"2025-11-14T06:22:48","modified_gmt":"2025-11-14T14:22:48","slug":"tidb-operator-source-code-reading-4-implement-component-control-loop","status":"publish","type":"post","link":"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/","title":{"rendered":"TiDB Operator Source Code Reading (IV): Implementing a Component Control Loop"},"content":{"rendered":"\n<p class=\"has-cyan-bluish-gray-color has-text-color\"><em>Previous articles in this series:<\/em><\/p>\n\n\n\n<ul class=\"wp-block-list has-cyan-bluish-gray-color has-text-color\">\n<li><em><a href=\"https:\/\/pingcap.com\/blog\/tidb-operator-source-code-reading-1-overview\">TiDB Operator Source Code Reading (I): Overview<\/a><\/em><\/li>\n\n\n\n<li><em><a href=\"https:\/\/pingcap.com\/blog\/tidb-operator-source-code-reading-2-operator-pattern\">TiDB Operator Source Code Reading (II): Operator Pattern<\/a><\/em><\/li>\n\n\n\n<li><em><a href=\"https:\/\/pingcap.com\/blog\/tidb-operator-source-code-reading-3-component-control-loop\">TiDB Operator Source Code Reading (III): The Component Control Loop<\/a><\/em><\/li>\n<\/ul>\n\n\n\n<p>In our&nbsp;<a href=\"https:\/\/pingcap.com\/blog\/tidb-operator-source-code-reading-3-component-control-loop\">last article<\/a>, we introduced how TiDB Operator orchestrates control loop events to manage the lifecycles of TiDB components. The&nbsp;<code>TidbCluster<\/code>&nbsp;controller manages TiDB components&#8217; lifecycles, and the member manager of each TiDB component encapsulates that component&#8217;s specific management logic.<\/p>\n\n\n\n<p>In this post, I&#8217;ll explain in detail&nbsp;<strong>how we implement a component control loop by taking PD as an example<\/strong>. You&#8217;ll learn about the PD member manager and its lifecycle management operations. I&#8217;ll also compare other components with PD and show their differences.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"the-pd-control-loop\"><span class=\"ez-toc-section\" id=\"The_PD_control_loop\"><\/span><a href=\"#the-pd-control-loop\">The PD control loop<\/a><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p><a href=\"https:\/\/docs.pingcap.com\/tidb\/stable\/tidb-scheduling\">Placement Driver<\/a>&nbsp;(PD) manages a TiDB cluster and schedules Regions in the cluster. The PD member manager maintains the PD lifecycle management logic. Most of its code resides in&nbsp;<code>pkg\/manager\/member\/pd_member_manager.go<\/code>, and other code related to scaling, upgrade, and failover is located in&nbsp;<code>pd_scaler.go<\/code>,&nbsp;<code>pd_upgrader.go<\/code>, and&nbsp;<code>pd_failover.go<\/code>&nbsp;respectively.<\/p>\n\n\n\n<p>As is illustrated in&nbsp;<a href=\"https:\/\/pingcap.com\/blog\/tidb-operator-source-code-reading-3-component-control-loop#call-the-component-control-loop\">Part III<\/a>, it takes the following tasks to manage a component&#8217;s lifecycle:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Sync Service<\/li>\n\n\n\n<li>Start syncing StatefulSet<\/li>\n\n\n\n<li>Sync status<\/li>\n\n\n\n<li>Sync ConfigMap<\/li>\n\n\n\n<li>Rolling update<\/li>\n\n\n\n<li>Scaling in and <a href=\"https:\/\/www.pingcap.com\/solutions\/modernize-mysql-workloads\/\">scaling out<\/a><\/li>\n\n\n\n<li>Failover<\/li>\n\n\n\n<li>Finish syncing StatefulSet<\/li>\n<\/ul>\n\n\n\n<p>Syncing StatefulSet is the major logic. Other tasks, like syncing status and syncing ConfigMap, are defined as sub-functions and called by the PD member manager when it syncs StatefulSet.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"sync-statefulset\">Sync StatefulSet<\/h3>\n\n\n\n<p>To sync StatefulSet, the PD member manager:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Obtains the current PD StatefulSet using&nbsp;<code>StatefulSetLister<\/code>.<br><code>oldPDSetTmp, err := m.deps.StatefulSetLister.StatefulSets(ns).Get(controller.PDMemberName(tcName))<\/code><br><br><code>if err != nil &amp;&amp; !errors.IsNotFound(err) {<br><br>return fmt.Errorf(\"syncPDStatefulSetForTidbCluster: fail to get sts %s for cluster %s\/%s, error: %s\", controller.PDMemberName(tcName), ns, tcName, err)<br>}<br><br>setNotExist := errors.IsNotFound(err)<br>oldPDSet := oldPDSetTmp.DeepCopy()<\/code><\/li>\n\n\n\n<li>Gets the latest status using&nbsp;<code>m.syncTidbClusterStatus(tc, oldPDSet).<\/code><br><code>if err := m.syncTidbClusterStatus(tc, oldPDSet); err != nil {<\/code><br><br><code>klog.Errorf(\"failed to sync TidbCluster: [%s\/%s]'s status, error: %v\", ns, tcName, err)<\/code><br><br><code>}<\/code><\/li>\n\n\n\n<li>Checks whether the TiDB cluster pauses the synchronization. If so, terminate the following reconciliation.<br><code>if tc.Spec.Paused {<\/code><br><br><code>klog.V(4).Infof(\"tidb cluster %s\/%s is paused, skip syncing for pd statefulset\", tc.GetNamespace(), tc.GetName())<\/code><br><br><code>return nil<\/code><br><br><code>}<\/code><br><br><code>Syncs ConfigMap according to the latest&nbsp;tc.Spec.<\/code><br><br><code>cm, err := m.syncPDConfigMap(tc, oldPDSet)<\/code><\/li>\n\n\n\n<li>Generates the latest StatefulSet template according to the latest&nbsp;<code>tc.Spec,<\/code>&nbsp;<code>tc.Status<\/code>, and ConfigMap obtained in the last step.<br><code>newPDSet, err := getNewPDSetForTidbCluster(tc, cm)<\/code><\/li>\n\n\n\n<li><code>If the new PD StatefulSet hasn't been created yet, first create the StatefulSet.<\/code><\/li>\n\n\n\n<li><br><code>if setNotExist {<\/code><br><br><code>if err := SetStatefulSetLastAppliedConfigAnnotation(newPDSet); err != nil {<\/code><br><br><code>return err<\/code><br><br><code>}<\/code><br><br><code>if err := m.deps.StatefulSetControl.CreateStatefulSet(tc, newPDSet); err != nil {<\/code><br><br><code>return err<\/code><br><br><code>}<\/code><br><br><code>tc.Status.PD.StatefulSet = &amp;apps.StatefulSetStatus{}<\/code><br><br><code>return controller.RequeueErrorf(\"TidbCluster: [%s\/%s], waiting for PD cluster running\", ns, tcName)<\/code><br><br><code>}<\/code><\/li>\n\n\n\n<li>If a user configures a forced upgrade using&nbsp;annotations, this step configures the StatefulSet to perform a rolling upgrade directly. This is to avoid an upgrade failure if the reconciliation loop is blocked.<br><code>if !tc.Status.PD.Synced &amp;&amp; NeedForceUpgrade(tc.Annotations) {<\/code><br><br><code>tc.Status.PD.Phase = v1alpha1.UpgradePhase<\/code><br><br><code>setUpgradePartition(newPDSet, 0)<\/code><br><br><code>errSTS := UpdateStatefulSet(m.deps.StatefulSetControl, tc, newPDSet, oldPDSet)<\/code><br><br><code>return controller.RequeueErrorf(\"tidbcluster: [%s\/%s]'s pd needs force upgrade, %v\", ns, tcName, errSTS)<\/code><br><br><code>}<\/code><br><br><code>Calls the scaling logic implemented in&nbsp;pd_scaler.go.<\/code><br><br><code>if err := m.scaler.Scale(tc, oldPDSet, newPDSet); err != nil {<\/code><br><br><code>return err<\/code><br><br><code>}<\/code><\/li>\n\n\n\n<li>Calls the failover logic implemented in&nbsp;pd_failover.go. The function first checks if the cluster needs to recover, then checks if all Pods are started and all members are healthy, and finally determines whether it needs to begin failover. <code>if m.deps.CLIConfig.AutoFailover {<\/code><br><br><code>if m.shouldRecover(tc) {<\/code><br><br><code>m.failover.Recover(tc)<\/code><br><br><code>} else if tc.PDAllPodsStarted() &amp;&amp; !tc.PDAllMembersReady() || tc.PDAutoFailovering() {<\/code><br><br><code>if err := m.failover.Failover(tc); err != nil {<\/code><br><br><code>return err<\/code><br><br><code>}<\/code><br><br><code>}<\/code><br><br><code>}<\/code> Calls the upgrade logic implemented in&nbsp;pd_upgrader.go. When the newly generated PD StatefulSet is inconsistent with the existing one or when the two StatefulSets are consistent but&nbsp;tc.Status.PD.Phase&nbsp;is&nbsp;upgrade, the PD Member Manager enters&nbsp;upgrader&nbsp;to process the rolling upgrade logic.<br>if !templateEqual(newPDSet, oldPDSet) || tc.Status.PD.Phase == v1alpha1.UpgradePhase {<br>if err := m.upgrader.Upgrade(tc, oldPDSet, newPDSet); err != nil {<br>return err<br>}<br>}<br>The PD StatefulSet is synced, and the new StatefulSet is updated to the Kubernetes cluster.<\/li>\n<\/ol>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"sync-service\">Sync Service<\/h3>\n\n\n\n<p>PD uses \u200b\u200bboth&nbsp;<a href=\"https:\/\/kubernetes.io\/docs\/concepts\/services-networking\/service\/\">Services<\/a>&nbsp;and&nbsp;<a href=\"https:\/\/kubernetes.io\/docs\/concepts\/services-networking\/service\/#headless-services\">headless Services<\/a>, managed by&nbsp;<code>syncPDServiceForTidbCluster<\/code>&nbsp;and&nbsp;<code>syncPDHeadlessServiceForTidbCluster<\/code>.<\/p>\n\n\n\n<p>The Service address is used in TiDB, TiKV, and TiFlash to configure the PD endpoint. For example, TiDB uses&nbsp;<code>--path=${CLUSTER_NAME}-pd:2379<\/code>&nbsp;as its PD service address in the startup parameter:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ARGS=\"--store=tikv \\\n--advertise-address=${POD_NAME}.${HEADLESS_SERVICE_NAME}.${NAMESPACE}.svc \\\n--path=${CLUSTER_NAME}-pd:2379 \\<\/code><\/pre>\n\n\n\n<p>The headless Service provides a unique identifier for each Pod. When PD starts up, the PD Pod registers its endpoint in PD members as&nbsp;<code>\"${POD_NAME}.${PEER_SERVICE_NAME}.${NAMESPACE}.svc\"<\/code>:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>domain=\"${POD_NAME}.${PEER_SERVICE_NAME}.${NAMESPACE}.svc\"\nARGS=\"--data-dir=\/var\/lib\/pd \\\n--name=${POD_NAME} \\\n--peer-urls=http:\/\/0.0.0.0:2380 \\\n--advertise-peer-urls=http:\/\/${domain}:2380 \\\n--client-urls=http:\/\/0.0.0.0:2379 \\\n--advertise-client-urls=http:\/\/${domain}:2379 \\\n--config=\/etc\/pd\/pd.toml \\\n\"<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"sync-configmap\">Sync ConfigMap<\/h3>\n\n\n\n<p>PD uses ConfigMap to manage the configurations and the start script. The&nbsp;<code>syncPDConfigMap<\/code>&nbsp;function calls&nbsp;<code>getPDConfigMap<\/code>&nbsp;to get the latest ConfigMap and apply it to the Kubernetes cluster. ConfigMap handles the following tasks:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><span data-mce-type=\"bookmark\" id=\"mce_1_start\" data-mce-style=\"overflow:hidden;line-height:0px\" style=\"overflow:hidden;line-height:0px\"><span data-mce-type=\"bookmark\" id=\"mce_1_end\" data-mce-style=\"overflow:hidden;line-height:0px\" style=\"overflow:hidden;line-height:0px\">Get&nbsp;<code>PD.Config<\/code>&nbsp;for the follow-up sync. To remain compatible with earlier versions that use Helm, when the config object is empty, ConfigMap is not synced.<\/span><\/span><br><code>config := tc.Spec.PD.Config<\/code><\/li>\n\n\n\n<li><code>if config == nil {<\/code>\n<ul class=\"wp-block-list\">\n<li><code>return nil, nil<\/code><\/li>\n\n\n\n<li><code>}<\/code><\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code><\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>\/\/ override CA if tls enabled\nif tc.IsTLSClusterEnabled() {\n    config.Set(\"security.cacert-path\", path.Join(pdClusterCertPath, tlsSecretRootCAKey))\n    config.Set(\"security.cert-path\", path.Join(pdClusterCertPath, corev1.TLSCertKey))\n    config.Set(\"security.key-path\", path.Join(pdClusterCertPath, corev1.TLSPrivateKeyKey))\n}\n\/\/ Versions below v4.0 do not support Dashboard\nif tc.Spec.TiDB != nil &amp;&amp; tc.Spec.TiDB.IsTLSClientEnabled() &amp;&amp; !tc.SkipTLSWhenConnectTiDB() &amp;&amp; clusterVersionGE4 {\n    config.Set(\"dashboard.tidb-cacert-path\", path.Join(tidbClientCertPath, tlsSecretRootCAKey))\n    config.Set(\"dashboard.tidb-cert-path\", path.Join(tidbClientCertPath, corev1.TLSCertKey))\n    config.Set(\"dashboard.tidb-key-path\", path.Join(tidbClientCertPath, corev1.TLSPrivateKeyKey))\n}<\/code><\/pre>\n\n\n\n<ol start=\"2\" class=\"wp-block-list\">\n<li>Modify TLS-related configuration. Because TiDB 4.0 and earlier versions don&#8217;t support TiDB Dashboard, PD member manager skips configuring Dashboard certificates for PD in earlier versions.<\/li>\n<\/ol>\n\n\n\n<ol start=\"3\" class=\"wp-block-list\">\n<li>Transform the configuration to TOML format so that PD can read it.<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>confText, err := config.MarshalTOML()<\/code><\/pre>\n\n\n\n<ol start=\"4\" class=\"wp-block-list\">\n<li>Generate the PD start script using&nbsp;<code>RenderPDStartScript<\/code>. The script template is stored in the&nbsp;<code>pdStartScriptTpl<\/code>&nbsp;variable in&nbsp;<code>pkg\/manager\/member\/template.go<\/code>. The PD start script is a Bash script.&nbsp;<code>RenderPDStartScript<\/code>&nbsp;inserts some variables and annotations configured by the&nbsp;<code>TidbCluster<\/code>&nbsp;object into the start script. These variables and annotations are used for PD startup and debugging.<\/li>\n<\/ol>\n\n\n\n<ol start=\"5\" class=\"wp-block-list\">\n<li>Assemble the PD configurations and the start script as the Kubernetes ConfigMap object, and return the ConfigMap to the&nbsp;<code>syncPDConfigMap<\/code>&nbsp;function.<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>cm := &amp;corev1.ConfigMap{\n    ObjectMeta: metav1.ObjectMeta{\n        Name:            controller.PDMemberName(tc.Name),\n        Namespace:       tc.Namespace,\n        Labels:          pdLabel,\n        OwnerReferences: &#91;]metav1.OwnerReference{controller.GetOwnerRef(tc)},\n    },\n    Data: map&#91;string]string{\n        \"config-file\":    string(confText),\n        \"startup-script\": startScript,\n    },\n}<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"scaling\">Scaling<\/h3>\n\n\n\n<p>The scaling logic is implemented&nbsp;<code>in pkg\/manager\/member\/pd_scaler.go<\/code>. The StatefulSet calls the&nbsp;<code>Scale<\/code>&nbsp;function to scale out or scale in the PD cluster.<\/p>\n\n\n\n<p>Before scaling, TiDB Operator needs to perform some preliminary operations. For example, to scale in the PD cluster, TiDB Operator needs to transfer Leaders, take members offline, and add annotations to&nbsp;<a href=\"https:\/\/kubernetes.io\/docs\/concepts\/storage\/persistent-volumes\/#introduction\">PersistentVolumeClaims<\/a>&nbsp;(PVCs) for deferred deletion. To scale out the cluster, TiDB Operator needs to delete the previously retained PVCs.<\/p>\n\n\n\n<p>After the preliminary operations are completed, TiDB Operator adjusts the number of StatefulSet replicas and begins the scaling process. The preliminary operations minimize the impact of the scaling operation.<\/p>\n\n\n\n<p>The&nbsp;<code>Scale<\/code>&nbsp;function acts like a router. Based on the scaling direction, step, and length, it determines which scaling plan to use. If the&nbsp;<code>scaling<\/code>&nbsp;variable is a positive number, PD is scaled out; otherwise, PD is scaled in. In each scaling operation, PD is only scaled by one step. The specific plan is implemented by the&nbsp;<code>ScaleIn<\/code>&nbsp;and&nbsp;<code>ScaleOut<\/code>&nbsp;functions.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>func (s *pdScaler) Scale(meta metav1.Object, oldSet *apps.StatefulSet, newSet *apps.StatefulSet) error {\n    scaling, _, _, _ := scaleOne(oldSet, newSet)\n    if scaling &gt; 0 {\n        return s.ScaleOut(meta, oldSet, newSet)\n    } else if scaling &amp;lt; 0 {\n        return s.ScaleIn(meta, oldSet, newSet)\n    }\n    return s.SyncAutoScalerAnn(meta, oldSet)\n}<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"scaling-in\">Scaling in<\/h4>\n\n\n\n<p>Before TiDB Operator takes a PD member offline, it must&nbsp;<strong>transfer the Leader<\/strong>. Otherwise, when the Leader member is taken offline, the rest of the PD members are forced into a Leader election, which affects cluster performance. Therefore, TiDB Operator needs to transfer the Leader to the PD member with the smallest ID. This makes sure that the PD Leader is only transferred once.<\/p>\n\n\n\n<p>To transfer the PD Leader, first get the PD client and the PD Leader:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>pdClient := controller.GetPDClient(s.deps.PDControl, tc)\nleader, err := pdClient.GetPDLeader()<\/code><\/pre>\n\n\n\n<p>When the Leader name equals the member name, TiDB Operator transfers the Leader. If there is only one member left, no other member is available for transfer, so TiDB Operator skips transferring the Leader.<\/p>\n\n\n\n<p>After the Leader is transferred, the&nbsp;<code>ScaleIn<\/code>&nbsp;function calls PD&#8217;s&nbsp;<code>DeleteMember<\/code>&nbsp;API and deletes the member from PD members. The PD member is then taken offline. Finally, the function calls&nbsp;<code>setReplicasAndDeleteSlots<\/code>&nbsp;to adjust the number of StatefulSet replicas, and the scaling-in process is completed.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"scaling-out\">Scaling out<\/h4>\n\n\n\n<p>Before TiDB Operator scales out the PD cluster, it needs to delete deferred PVCs by calling&nbsp;<code>deleteDeferDeletingPVC<\/code>. When TiDB Operator previously scaled in PD, these PVCs were retained for deferred deletion to achieve data reliability; now, TiDB Operator must delete them to avoid the cluster using old data. After these PVCs are deleted, TiDB Operator only has to adjust the number of StatefulSet replicas and scale out the PD cluster.<\/p>\n\n\n\n<p>Whether you scale in or scale out the PD cluster, the operation is performed by configuring the number of StatefulSet replicas. Therefore, when TiDB Operator supports the&nbsp;<a href=\"https:\/\/docs.pingcap.com\/tidb-in-kubernetes\/stable\/advanced-statefulset\">advanced StatefulSet<\/a>, it must take into account the empty slots when counting the replicas.<\/p>\n\n\n\n\n\n<h3 class=\"wp-block-heading\" id=\"rolling-upgrade\"><a href=\"#rolling-update\">Rolling upgrade<\/a><\/h3>\n\n\n\n<p>The upgrade logic is implemented in&nbsp;<code>pkg\/manager\/member\/pd_upgrader.go<\/code>. The&nbsp;<code>UpdateStrategy<\/code>&nbsp;in the StatefulSet performs the rolling upgrade on PD. To mitigate the impact of upgrade on the PD cluster,&nbsp;<code>pd_upgrader.go<\/code>&nbsp;performs some preliminary operations when it updates&nbsp;<code>UpdateStrategy<\/code>&nbsp;in the StatefulSet. For details on how to control&nbsp;<code>UpdateStrategy<\/code>, refer to&nbsp;<a href=\"https:\/\/pingcap.com\/blog\/tidb-operator-source-code-reading-3-component-control-loop#rolling-update\">our last article<\/a>.<\/p>\n\n\n\n<p>Before upgrade, TiDB Operator completes the following status check:<\/p>\n\n\n\n<ol>\n<li>\nIt checks whether there are other ongoing operations. In particular, it checks whether TiCDC and TiFlash is being upgraded and whether PD is being scaled out or scaled in.\n\n<pre><code>if tc.Status.TiCDC.Phase == v1alpha1.UpgradePhase ||\n    tc.Status.TiFlash.Phase == v1alpha1.UpgradePhase ||\n    tc.PDScaling()\n<\/code><\/pre>\n\n<\/li>\n<li>The PD member manager determines to enter <code>Upgrader<\/code> on two circumstances:\n<ol>\n<li>\nWhen the newly generated PD StatefulSet is inconsistent with the existing one, the status check returns <code>nil<\/code>, and TiDB Operator updates the StatefulSet. There is no need to perform the following checks in steps 3 and 4.\n<pre><code>if !templateEqual(newSet, oldSet) {\n    return nil\n}\n<\/code><\/pre>\n<\/li>\n<li>\nWhen the two StatefulSets are consistent but <code>tc.Status.PD.Phase<\/code> is <code>upgrade<\/code>, <code>newSet<\/code> and <code>oldSet<\/code> have the same template spec, and TiDB Operator continues the following two checks.\n<\/li>\n<\/ol>\nFor more information on these two situations, see <a href=\"#sync-statefulset\">Sync StatefulSet<\/a>, Step 10.\n<\/li>\n<li>\nIt compares <code>tc.Status.PD.StatefulSet.UpdateRevision<\/code> and <code>tc.Status.PD.StatefulSet.CurrentRevision<\/code>. If the two variables are equal, the rolling upgrade operation is already completed. Then, TiDB Operator ends the upgrade process.<p><\/p>\n<pre><code>if tc.Status.PD.StatefulSet.UpdateRevision == tc.Status.PD.StatefulSet.CurrentRevision\n<\/code><\/pre>\n<\/li>\n<li>\nIt checks if the <code>UpdateStrategy<\/code> in the StatefulSet is manually modified. If yes, follow the modified strategy.\n<pre><code>if oldSet.Spec.UpdateStrategy.Type == apps.OnDeleteStatefulSetStrategyType || oldSet.Spec.UpdateStrategy.RollingUpdate == nil {\n    newSet.Spec.UpdateStrategy = oldSet.Spec.UpdateStrategy\n    klog.Warningf(\"tidbcluster: [%s\/%s] pd statefulset %s UpdateStrategy has been modified manually\", ns, tcName, oldSet.GetName())\n    return nil\n}\n<\/code><\/pre>\n<\/li>\n<\/ol>\nAfter all checks are passed, TiDB Operator begins processing each Pod and rolling upgrades the PD cluster:\n<ol>\n<li>\nIt checks whether the PD Pod is updated. By comparing the <code>controller-revision-hash<\/code> value in the Pod Label with the <code>UpdateRevision<\/code> value of the StatefulSet, TiDB Operator can tell whether this Pod is updated or still unprocessed. If the Pod is updated, TiDB Operator then checks whether the PD member is healthy. If the PD member is healthy, TiDB Operator moves on to the next Pod. If not, TiDB Operator returns an error and checks the health state in the next sync.\n<pre><code>revision, exist := pod.Labels[apps.ControllerRevisionHashLabelKey]\n    if !exist {\n        return controller.RequeueErrorf(\"tidbcluster: [%s\/%s]'s pd pod: [%s] has no label: %s\", ns, tcName, podName, apps.ControllerRevisionHashLabelKey)\n    }\n\n    if revision == tc.Status.PD.StatefulSet.UpdateRevision {\n        if member, exist := tc.Status.PD.Members[PdName(tc.Name, i, tc.Namespace, tc.Spec.ClusterDomain)]; !exist || !member.Health {\n            return controller.RequeueErrorf(\"tidbcluster: [%s\/%s]'s pd upgraded pod: [%s] is not ready\", ns, tcName, podName)\n        }\n        continue\n    }\n<\/code><\/pre>\n<\/li>\n<li>\nIf <code>Pod revision != tc.Status.PD.StatefulSet.UpdateRevision<\/code>, the Pod is not updated yet. TiDB Operator calls the <code>upgradePDPod<\/code> function to process this Pod. When TiDB Operator processes the PD Leader Pod, it transfers the Leader before it continues the upgrade process.\n<pre><code>if tc.Status.PD.Leader.Name == upgradePdName || tc.Status.PD.Leader.Name == upgradePodName {\n    var targetName string\n    targetOrdinal := helper.GetMaxPodOrdinal(*newSet.Spec.Replicas, newSet)\n    if ordinal == targetOrdinal {\n        targetOrdinal = helper.GetMinPodOrdinal(*newSet.Spec.Replicas, newSet)\n    }\n    targetName = PdName(tcName, targetOrdinal, tc.Namespace, tc.Spec.ClusterDomain)\n    if _, exist := tc.Status.PD.Members[targetName]; !exist {\n        targetName = PdPodName(tcName, targetOrdinal)\n    }\n\n    if len(targetName) &gt; 0 {\n        err := u.transferPDLeaderTo(tc, targetName)\n        if err != nil {\n            klog.Errorf(\"pd upgrader: failed to transfer pd leader to: %s, %v\", targetName, err)\n            return err\n        }\n        klog.Infof(\"pd upgrader: transfer pd leader to: %s successfully\", targetName)\n        return controller.RequeueErrorf(\"tidbcluster: [%s\/%s]'s pd member: [%s] is transferring leader to pd member: [%s]\", ns, tcName, upgradePdName, targetName)\n    }\n}\nsetUpgradePartition(newSet, ordinal)\n<\/code><\/pre>\n<\/li>\n<\/ol>\n\n\n\n<h3 class=\"wp-block-heading\">Failover<\/h3>\n\n\n\n<p>The PD failover logic is implemented in&nbsp;<code>pd_failover.go<\/code>. PD deletes failed Pods to fix the failure, which is different from other component failover logic.<\/p>\n\n\n\n<p>Before performing failover, TiDB Operator also carries out health checks. If the PD cluster is unavailable, which means half of the PD members are unhealthy, recreating PD members cannot recover the cluster. Under such circumstances, TiDB Operator doesn&#8217;t perform failover.<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><span data-mce-type=\"bookmark\" id=\"mce_1_start\" data-mce-style=\"overflow:hidden;line-height:0px\" style=\"overflow:hidden;line-height:0px\"><span data-mce-type=\"bookmark\" id=\"mce_1_end\" data-mce-style=\"overflow:hidden;line-height:0px\" style=\"overflow:hidden;line-height:0px\">TiDB Operator traverses the PD member health status obtained from the PD client. When the PD member is unhealthy and the duration between its last transition time and now is longer than&nbsp;<code>failoverDeadline<\/code>, the member is marked as unhealthy. Its Pod information and PVC information are recorded in&nbsp;<code>tc.Status.PD.FailureMembers<\/code>.<\/span><\/span><\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>for pdName, pdMember := range tc.Status.PD.Members {\n    podName := strings.Split(pdName, \".\")&#91;0]\n\n    failoverDeadline := pdMember.LastTransitionTime.Add(f.deps.CLIConfig.PDFailoverPeriod)\n    _, exist := tc.Status.PD.FailureMembers&#91;pdName]\n\n    if pdMember.Health || time.Now().Before(failoverDeadline) || exist {\n        continue\n    }\n\n    pod, _ := f.deps.PodLister.Pods(ns).Get(podName)\n\n    pvcs, _ := util.ResolvePVCFromPod(pod, f.deps.PVCLister)\n\n    f.deps.Recorder.Eventf(tc, apiv1.EventTypeWarning, \"PDMemberUnhealthy\", \"%s\/%s(%s) is unhealthy\", ns, podName, pdMember.ID)\n\n    pvcUIDSet := make(map&#91;types.UID]struct{})\n    for _, pvc := range pvcs {\n        pvcUIDSet&#91;pvc.UID] = struct{}{}\n    }\n    tc.Status.PD.FailureMembers&#91;pdName] = v1alpha1.PDFailureMember{\n        PodName:       podName,\n        MemberID:      pdMember.ID,\n        PVCUIDSet:     pvcUIDSet,\n        MemberDeleted: false,\n        CreatedAt:     metav1.Now(),\n    }\n    return controller.RequeueErrorf(\"marking Pod: %s\/%s pd member: %s as failure\", ns, podName, pdMember.Name)\n}<\/code><\/pre>\n\n\n\n<ol start=\"2\" class=\"wp-block-list\">\n<li><span data-mce-type=\"bookmark\" id=\"mce_1_start\" data-mce-style=\"overflow:hidden;line-height:0px\" style=\"overflow:hidden;line-height:0px\"><span data-mce-type=\"bookmark\" id=\"mce_1_end\" data-mce-style=\"overflow:hidden;line-height:0px\" style=\"overflow:hidden;line-height:0px\">TiDB Operator calls the&nbsp;<code>tryToDeleteAFailureMember<\/code>&nbsp;function to process the&nbsp;<code>FailureMembers<\/code>. It traverses all failure members, and when it encounters a member whose&nbsp;<code>MemberDeleted<\/code>&nbsp;is&nbsp;<code>False<\/code>, it calls the PD client to delete the member and tries to recover the Pod.<\/span><\/span><\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>func (f *pdFailover) tryToDeleteAFailureMember(tc *v1alpha1.TidbCluster) error {\n    ns := tc.GetNamespace()\n    tcName := tc.GetName()\n    var failureMember *v1alpha1.PDFailureMember\n    var failurePodName string\n    var failurePDName string\n\n    for pdName, pdMember := range tc.Status.PD.FailureMembers {\n        if !pdMember.MemberDeleted {\n            failureMember = &amp;pdMember\n            failurePodName = strings.Split(pdName, \".\")&#91;0]\n            failurePDName = pdName\n            break\n        }\n    }\n    if failureMember == nil {\n        klog.Infof(\"No PD FailureMembers to delete for tc %s\/%s\", ns, tcName)\n        return nil\n    }\n\n    memberID, err := strconv.ParseUint(failureMember.MemberID, 10, 64)\n    if err != nil {\n        return err\n    }\n\n    if err := controller.GetPDClient(f.deps.PDControl, tc).DeleteMemberByID(memberID); err != nil {\n        klog.Errorf(\"pd failover&#91;tryToDeleteAFailureMember]: failed to delete member %s\/%s(%d), error: %v\", ns, failurePodName, memberID, err)\n        return err\n    }\n    klog.Infof(\"pd failover&#91;tryToDeleteAFailureMember]: delete member %s\/%s(%d) successfully\", ns, failurePodName, memberID)\n...<\/code><\/pre>\n\n\n\n<p>Delete the failed Pod.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>pod, err := f.deps.PodLister.Pods(ns).Get(failurePodName)\n    if err != nil &amp;&amp; !errors.IsNotFound(err) {\n        return fmt.Errorf(\"pd failover&#91;tryToDeleteAFailureMember]: failed to get pod %s\/%s for tc %s\/%s, error: %s\", ns, failurePodName, ns, tcName, err)\n    }\n    if pod != nil {\n        if pod.DeletionTimestamp == nil {\n            if err := f.deps.PodControl.DeletePod(tc, pod); err != nil {\n                return err\n            }\n        }\n    } else {\n        klog.Infof(\"pd failover&#91;tryToDeleteAFailureMember]: failure pod %s\/%s not found, skip\", ns, failurePodName)\n    }<\/code><\/pre>\n\n\n\n<p>Delete the PVC.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>for _, pvc := range pvcs {\n    _, pvcUIDExist := failureMember.PVCUIDSet&#91;pvc.GetUID()]\n    \/\/ for backward compatibility, if there exists failureMembers and user upgrades operator to newer version\n    \/\/ there will be failure member structures with PVCUID set from api server, we should handle this as pvcUIDExist == true\n    if pvc.GetUID() == failureMember.PVCUID {\n        pvcUIDExist = true\n    }\n    if pvc.DeletionTimestamp == nil &amp;&amp; pvcUIDExist {\n        if err := f.deps.PVCControl.DeletePVC(tc, pvc); err != nil {\n            klog.Errorf(\"pd failover&#91;tryToDeleteAFailureMember]: failed to delete PVC: %s\/%s, error: %s\", ns, pvc.Name, err)\n            return err\n        }\n        klog.Infof(\"pd failover&#91;tryToDeleteAFailureMember]: delete PVC %s\/%s successfully\", ns, pvc.Name)\n    }\n}<\/code><\/pre>\n\n\n\n<p>Mark the status of&nbsp;<code>tc.Status.PD.FailureMembers<\/code>&nbsp;as&nbsp;<code>Deleted<\/code>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>setMemberDeleted(tc, failurePDName)<\/code><\/pre>\n\n\n\n<ol start=\"3\" class=\"wp-block-list\">\n<li><span data-mce-type=\"bookmark\" id=\"mce_1_start\" data-mce-style=\"overflow:hidden;line-height:0px\" style=\"overflow:hidden;line-height:0px\"><span data-mce-type=\"bookmark\" id=\"mce_1_start\" data-mce-style=\"overflow:hidden;line-height:0px\" style=\"overflow:hidden;line-height:0px\"><span data-mce-type=\"bookmark\" id=\"mce_1_end\" data-mce-style=\"overflow:hidden;line-height:0px\" style=\"overflow:hidden;line-height:0px\"><span data-mce-type=\"bookmark\" id=\"mce_1_end\" data-mce-style=\"overflow:hidden;line-height:0px\" style=\"overflow:hidden;line-height:0px\"><code>tc.PDStsDesiredReplicas()<\/code>&nbsp;obtains the number of PD StatefulSet replicas, which equals the number of StatefulSet replicas plus the deleted failure members. When syncing the StatefulSet, TiDB Operator calls the scaling-out logic to add a PD Pod for failover.<\/span><\/span><\/span><\/span><\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>func (tc *TidbCluster) GetPDDeletedFailureReplicas() int32 {\n    var deletedReplicas int32 = 0\n    for _, failureMember := range tc.Status.PD.FailureMembers {\n        if failureMember.MemberDeleted {\n            deletedReplicas++\n        }\n    }\n    return deletedReplicas\n}\n\nfunc (tc *TidbCluster) PDStsDesiredReplicas() int32 {\n    return tc.Spec.PD.Replicas + tc.GetPDDeletedFailureReplicas()\n}<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"other-components-control-loop\"><span class=\"ez-toc-section\" id=\"Other_components_control_loop\"><\/span>Other component&#8217;s control loop<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Now that you know how TiDB Operator manages PD&#8217;s lifecycle, I&#8217;ll move on to the differences between PD and other components.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"tikv-and-tiflash\"><a href=\"https:\/\/pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop#tikv-and-tiflash\"><\/a>TiKV and TiFlash<\/h3>\n\n\n\n<p><a href=\"https:\/\/docs.pingcap.com\/tidb\/stable\/tikv-overview\">TiKV<\/a>&nbsp;and&nbsp;<a href=\"https:\/\/docs.pingcap.com\/tidb\/stable\/tiflash-overview\">TiFlash<\/a>&nbsp;are two storage engines of TiDB and have a similar lifecycle management mechanism. In this section, I&#8217;ll take TiKV as an example.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>When syncing the StatefulSet, the TiKV member manager needs to set TiKV store labels using the&nbsp;<code>setStoreLabelsForTiKV<\/code>&nbsp;function. This function gets the node labels and sets labels for TiKV stores via the PD client&#8217;s&nbsp;<code>SetStoreLables<\/code>&nbsp;function.<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>for _, store := range storesInfo.Stores {\n    nodeName := pod.Spec.NodeName\n    ls, _ := getNodeLabels(m.deps.NodeLister, nodeName, storeLabels)\n\n    if !m.storeLabelsEqualNodeLabels(store.Store.Labels, ls) {\n        set, err := pdCli.SetStoreLabels(store.Store.Id, ls)\n        if err != nil {\n            continue\n        }\n        if set {\n            setCount++\n            klog.Infof(\"pod: &#91;%s\/%s] set labels: %v successfully\", ns, podName, ls)\n        }\n    }\n}<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>When syncing Status, the TiKV member manager calls the PD client&#8217;s&nbsp;<code>GetStores<\/code>&nbsp;function and obtains the TiKV store information from PD. It then categorizes the store information for later synchronization. This operation is similar to PD&#8217;s syncing status, in which the PD member manager calls the&nbsp;<code>GetMembers<\/code>&nbsp;function and records PD member information.<\/li>\n\n\n\n<li>When syncing Service, the TiKV member manager only creates headless Services for parsing TiKV Pod DNS.<\/li>\n\n\n\n<li>When syncing ConfigMap, the TiKV member manager uses a template in the&nbsp;<code>templates.go<\/code>&nbsp;file, calls the&nbsp;<code>RenderTiKVStartScript<\/code>&nbsp;function to generate the TiKV start script, and calls the&nbsp;<code>transformTiKVConfigMap<\/code>&nbsp;function to get the TiKV configuration file.<\/li>\n\n\n\n<li>When performing scaling operations, the TiKV member manager must safely take TiKV stores offline. It calls the PD client&#8217;s&nbsp;<code>DeleteStore<\/code>&nbsp;function to delete the corresponding store from the TiKV Pod.<\/li>\n\n\n\n<li>In terms of rolling upgrade, the TiKV member manager must ensure that the Region Leader is not on the Pod to be upgraded. Before the rolling upgrade, TiKV Upgrader checks whether the Pod annotation contains&nbsp;<code>EvictLeaderBeginTime<\/code>&nbsp;and determines whether it has performed&nbsp;<code>EvictLeader<\/code>&nbsp;on this Pod. If not, it calls the&nbsp;<code>BeginEvictLeader<\/code>&nbsp;function from the PD client and adds an evict leader scheduler to the TiKV store. The Region Leader is then evicted from the TiKV store.<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>_, evicting := upgradePod.Annotations&#91;EvictLeaderBeginTime]\nif !evicting {\n    return u.beginEvictLeader(tc, storeID, upgradePod)\n}<\/code><\/pre>\n\n\n\n<p>In the&nbsp;<code>readyToUpgrade<\/code>&nbsp;function, when the Region Leader is zero, or when the evict Leader duration exceeds&nbsp;<code>tc.Spec.TiKV.EvictLeaderTimeout<\/code>, the partition configuration in the StatefulSet&nbsp;<code>UpdateStrategy<\/code>&nbsp;is updated, which triggers Pod upgrade. When PD finishes the upgrade, it calls the&nbsp;<code>endEvictLeaderbyStoreID<\/code>&nbsp;function to end the operation.<\/p>\n\n\n\n<p>When performing failover, the TiKV member manager records the timestamp of the last store state change.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>status.LastTransitionTime = metav1.Now()\nif exist &amp;&amp; status.State == oldStore.State {\n    status.LastTransitionTime = oldStore.LastTransitionTime\n}<\/code><\/pre>\n\n\n\n<p>When the store state is&nbsp;<code>v1alpha1.TiKVStateDown<\/code>&nbsp;and the downtime exceeds the maximum failover timeout, the TiKV Pod is added to&nbsp;<code>FailureStores<\/code>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>if store.State == v1alpha1.TiKVStateDown &amp;&amp; time.Now().After(deadline) &amp;&amp; !exist {\n    if tc.Status.TiKV.FailureStores == nil {\n        tc.Status.TiKV.FailureStores = map&#91;string]v1alpha1.TiKVFailureStore{}\n    }\n    if tc.Spec.TiKV.MaxFailoverCount != nil &amp;&amp; *tc.Spec.TiKV.MaxFailoverCount &gt; 0 {\n        maxFailoverCount := *tc.Spec.TiKV.MaxFailoverCount\n        if len(tc.Status.TiKV.FailureStores) &gt;= int(maxFailoverCount) {\n            klog.Warningf(\"%s\/%s failure stores count reached the limit: %d\", ns, tcName, tc.Spec.TiKV.MaxFailoverCount)\n            return nil\n        }\n        tc.Status.TiKV.FailureStores&#91;storeID] = v1alpha1.TiKVFailureStore{\n            PodName:   podName,\n            StoreID:   store.ID,\n            CreatedAt: metav1.Now(),\n        }\n        msg := fmt.Sprintf(\"store&#91;%s] is Down\", store.ID)\n        f.deps.Recorder.Event(tc, corev1.EventTypeWarning, unHealthEventReason, fmt.Sprintf(unHealthEventMsgPattern, \"tikv\", podName, msg))\n    }\n}<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>When syncing TiKV StatefulSet, the number of replicas includes the number of&nbsp;<code>FailureStores<\/code>, which triggers the scaling-out logic and completes the failover operation.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"tidb-ticdc-and-pump\">TiDB, TiCDC, and Pump<a href=\"https:\/\/pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop#tidb-ticdc-and-pump\"><\/a><\/h3>\n\n\n\n<p>TiDB, TiCDC, and Pump have a similar lifecycle management mechanism. Compared to other components, their rolling upgrade is only carried out when the member is healthy.<\/p>\n\n\n\n<p>During scaling, TiDB Operator needs to pay extra attention to PVC usage. When scaling in the cluster, TiDB Operator needs to add\u00a0<code>deferDeleting<\/code>\u00a0to ensure data safety; when <a href=\"https:\/\/www.pingcap.com\/solutions\/modernize-mysql-workloads\/\">scaling out<\/a> the cluster, TiDB Operator also needs to remove the PVC.<\/p>\n\n\n\n<p>In addition, TiDB Operator only supports failover logic for TiDB, but not for TiCDC or Pump.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"summary\"><span class=\"ez-toc-section\" id=\"Summary\"><\/span>Summary<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>In this post, taking PD as an example, we have learned the implementation of PD&#8217;s control loop and compared PD with other components. By now you should be familiar with the member manager design of major TiDB components and how TiDB Operator implements TiDB lifecycle management.<\/p>\n\n\n\n<p>In the next article, we&#8217;ll discuss&nbsp;<strong>how TiDB Operator implements backup and restore for TiDB clusters<\/strong>. Stay tuned!<\/p>\n\n\n\n<p>If you are interested in learning more about TiDB Operator, feel free to&nbsp;<a href=\"https:\/\/slack.tidb.io\/invite?team=tidb-community&amp;channel=sig-k8s&amp;ref=pingcap-blog\">join our Slack channel<\/a>&nbsp;or join our discussions at&nbsp;<a href=\"https:\/\/github.com\/pingcap\/tidb-operator\">pingcap\/tidb-operator<\/a>.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<hr class=\"wp-block-separator has-text-color has-cyan-bluish-gray-color has-css-opacity has-cyan-bluish-gray-background-color has-background is-style-wide\"\/>\n\n\n\n<p>If you&#8217;ve missed an article in our series, you can read them here:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/pingcap.com\/blog\/tidb-operator-source-code-reading-1-overview\">TiDB Operator Source Code Reading (I): Overview<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/pingcap.com\/blog\/tidb-operator-source-code-reading-2-operator-pattern\">TiDB Operator Source Code Reading (II): Operator Pattern<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/pingcap.com\/blog\/tidb-operator-source-code-reading-3-component-control-loop\">TiDB Operator Source Code Reading (III): The Component Control Loop<\/a><\/li>\n<\/ul>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This post explains how TiDB Operator implements a component control loop by taking PD as an example. You&#8217;ll learn PD and other component&#8217;s lifecycle management.<\/p>","protected":false},"author":67,"featured_media":2323,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ub_ctt_via":"","footnotes":""},"categories":[6],"tags":[33,84],"class_list":["post-2321","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-engineering","tag-kubernetes","tag-tidb-operator"],"acf":[],"featured_image_src":"https:\/\/static.pingcap.com\/files\/2021\/11\/tidb-operator-source-code-reading-4.jpg","author_info":{"display_name":"Yiwen Chen","author_link":"https:\/\/www.pingcap.com\/ko\/blog\/author\/yiwen-chen\/"},"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.9 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>TiDB Operator Source Code Reading (IV)<\/title>\n<meta name=\"description\" content=\"Explore how we implement a component control loop by taking PD as an example, and compare other components with PD and show the differences.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/\" \/>\n<meta property=\"og:locale\" content=\"ko_KR\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"TiDB Operator Source Code Reading (IV)\" \/>\n<meta property=\"og:description\" content=\"Explore how we implement a component control loop by taking PD as an example, and compare other components with PD and show the differences.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/\" \/>\n<meta property=\"og:site_name\" content=\"TiDB\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/facebook.com\/pingcap2015\" \/>\n<meta property=\"article:published_time\" content=\"2021-10-28T08:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-11-14T14:22:48+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/static.pingcap.com\/files\/2021\/11\/tidb-operator-source-code-reading-4.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1500\" \/>\n\t<meta property=\"og:image:height\" content=\"501\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Yiwen Chen\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@PingCAP\" \/>\n<meta name=\"twitter:site\" content=\"@PingCAP\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Yiwen Chen\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"15\ubd84\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/\"},\"author\":{\"name\":\"Yiwen Chen\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/person\/5e8e0624ab59c4f01eb367e8c9869ab2\"},\"headline\":\"TiDB Operator Source Code Reading (IV): Implementing a Component Control Loop\",\"datePublished\":\"2021-10-28T08:00:00+00:00\",\"dateModified\":\"2025-11-14T14:22:48+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/\"},\"wordCount\":2369,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.pingcap.com\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/static.pingcap.com\/files\/2021\/11\/tidb-operator-source-code-reading-4.jpg\",\"keywords\":[\"Kubernetes\",\"TiDB Operator\"],\"articleSection\":[\"Engineering\"],\"inLanguage\":\"ko-KR\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/\",\"url\":\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/\",\"name\":\"TiDB Operator Source Code Reading (IV)\",\"isPartOf\":{\"@id\":\"https:\/\/www.pingcap.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/static.pingcap.com\/files\/2021\/11\/tidb-operator-source-code-reading-4.jpg\",\"datePublished\":\"2021-10-28T08:00:00+00:00\",\"dateModified\":\"2025-11-14T14:22:48+00:00\",\"description\":\"Explore how we implement a component control loop by taking PD as an example, and compare other components with PD and show the differences.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/#breadcrumb\"},\"inLanguage\":\"ko-KR\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/#primaryimage\",\"url\":\"https:\/\/static.pingcap.com\/files\/2021\/11\/tidb-operator-source-code-reading-4.jpg\",\"contentUrl\":\"https:\/\/static.pingcap.com\/files\/2021\/11\/tidb-operator-source-code-reading-4.jpg\",\"width\":1500,\"height\":501},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.pingcap.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"TiDB Operator Source Code Reading (IV): Implementing a Component Control Loop\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.pingcap.com\/#website\",\"url\":\"https:\/\/www.pingcap.com\/\",\"name\":\"TiDB\",\"description\":\"TiDB | SQL at Scale\",\"publisher\":{\"@id\":\"https:\/\/www.pingcap.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.pingcap.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"ko-KR\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.pingcap.com\/#organization\",\"name\":\"PingCAP\",\"url\":\"https:\/\/www.pingcap.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png\",\"contentUrl\":\"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png\",\"width\":811,\"height\":232,\"caption\":\"PingCAP\"},\"image\":{\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/facebook.com\/pingcap2015\",\"https:\/\/x.com\/PingCAP\",\"https:\/\/linkedin.com\/company\/pingcap\",\"https:\/\/youtube.com\/channel\/UCuq4puT32DzHKT5rU1IZpIA\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/person\/5e8e0624ab59c4f01eb367e8c9869ab2\",\"name\":\"Yiwen Chen\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg\",\"contentUrl\":\"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg\",\"caption\":\"Yiwen Chen\"},\"url\":\"https:\/\/www.pingcap.com\/ko\/blog\/author\/yiwen-chen\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"TiDB Operator Source Code Reading (IV)","description":"Explore how we implement a component control loop by taking PD as an example, and compare other components with PD and show the differences.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/","og_locale":"ko_KR","og_type":"article","og_title":"TiDB Operator Source Code Reading (IV)","og_description":"Explore how we implement a component control loop by taking PD as an example, and compare other components with PD and show the differences.","og_url":"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/","og_site_name":"TiDB","article_publisher":"https:\/\/facebook.com\/pingcap2015","article_published_time":"2021-10-28T08:00:00+00:00","article_modified_time":"2025-11-14T14:22:48+00:00","og_image":[{"width":1500,"height":501,"url":"https:\/\/static.pingcap.com\/files\/2021\/11\/tidb-operator-source-code-reading-4.jpg","type":"image\/jpeg"}],"author":"Yiwen Chen","twitter_card":"summary_large_image","twitter_creator":"@PingCAP","twitter_site":"@PingCAP","twitter_misc":{"Written by":"Yiwen Chen","Est. reading time":"15\ubd84"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/#article","isPartOf":{"@id":"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/"},"author":{"name":"Yiwen Chen","@id":"https:\/\/www.pingcap.com\/#\/schema\/person\/5e8e0624ab59c4f01eb367e8c9869ab2"},"headline":"TiDB Operator Source Code Reading (IV): Implementing a Component Control Loop","datePublished":"2021-10-28T08:00:00+00:00","dateModified":"2025-11-14T14:22:48+00:00","mainEntityOfPage":{"@id":"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/"},"wordCount":2369,"commentCount":0,"publisher":{"@id":"https:\/\/www.pingcap.com\/#organization"},"image":{"@id":"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/#primaryimage"},"thumbnailUrl":"https:\/\/static.pingcap.com\/files\/2021\/11\/tidb-operator-source-code-reading-4.jpg","keywords":["Kubernetes","TiDB Operator"],"articleSection":["Engineering"],"inLanguage":"ko-KR","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/","url":"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/","name":"TiDB Operator Source Code Reading (IV)","isPartOf":{"@id":"https:\/\/www.pingcap.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/#primaryimage"},"image":{"@id":"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/#primaryimage"},"thumbnailUrl":"https:\/\/static.pingcap.com\/files\/2021\/11\/tidb-operator-source-code-reading-4.jpg","datePublished":"2021-10-28T08:00:00+00:00","dateModified":"2025-11-14T14:22:48+00:00","description":"Explore how we implement a component control loop by taking PD as an example, and compare other components with PD and show the differences.","breadcrumb":{"@id":"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/#breadcrumb"},"inLanguage":"ko-KR","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/"]}]},{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/#primaryimage","url":"https:\/\/static.pingcap.com\/files\/2021\/11\/tidb-operator-source-code-reading-4.jpg","contentUrl":"https:\/\/static.pingcap.com\/files\/2021\/11\/tidb-operator-source-code-reading-4.jpg","width":1500,"height":501},{"@type":"BreadcrumbList","@id":"https:\/\/www.pingcap.com\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.pingcap.com\/"},{"@type":"ListItem","position":2,"name":"TiDB Operator Source Code Reading (IV): Implementing a Component Control Loop"}]},{"@type":"WebSite","@id":"https:\/\/www.pingcap.com\/#website","url":"https:\/\/www.pingcap.com\/","name":"\ud2f0DB","description":"TiDB | SQL at Scale","publisher":{"@id":"https:\/\/www.pingcap.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.pingcap.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"ko-KR"},{"@type":"Organization","@id":"https:\/\/www.pingcap.com\/#organization","name":"PingCAP","url":"https:\/\/www.pingcap.com\/","logo":{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/","url":"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png","contentUrl":"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png","width":811,"height":232,"caption":"PingCAP"},"image":{"@id":"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/facebook.com\/pingcap2015","https:\/\/x.com\/PingCAP","https:\/\/linkedin.com\/company\/pingcap","https:\/\/youtube.com\/channel\/UCuq4puT32DzHKT5rU1IZpIA"]},{"@type":"Person","@id":"https:\/\/www.pingcap.com\/#\/schema\/person\/5e8e0624ab59c4f01eb367e8c9869ab2","name":"Yiwen Chen","image":{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/www.pingcap.com\/#\/schema\/person\/image\/","url":"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg","contentUrl":"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg","caption":"Yiwen Chen"},"url":"https:\/\/www.pingcap.com\/ko\/blog\/author\/yiwen-chen\/"}]}},"grav_blocks":false,"card_markup":"<a class=\"card-resource bg-white\" href=\"https:\/\/www.pingcap.com\/ko\/blog\/tidb-operator-source-code-reading-4-implement-component-control-loop\/\"><div class=\"card-resource__image-container\"><img class=\"card-resource__image\" alt=\"tidb-operator-source-code-reading-4\" src=\"https:\/\/static.pingcap.com\/files\/2021\/11\/tidb-operator-source-code-reading-4.jpg\" loading=\"lazy\" width=1500 height=501 \/><\/div><div class=\"card-resource__content-container\"><div class=\"card-resource__content-head\"><div class=\"card-resource__category\">Engineering<\/div><\/div><h5 class=\"card-resource__title\">TiDB Operator Source Code Reading (IV): Implementing a Component Control Loop<\/h5><\/div><\/a>","_links":{"self":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts\/2321","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/users\/67"}],"replies":[{"embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/comments?post=2321"}],"version-history":[{"count":6,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts\/2321\/revisions"}],"predecessor-version":[{"id":30503,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts\/2321\/revisions\/30503"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/media\/2323"}],"wp:attachment":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/media?parent=2321"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/categories?post=2321"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/tags?post=2321"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}