{"id":1178,"date":"2019-05-08T00:00:00","date_gmt":"2019-05-08T00:00:00","guid":{"rendered":"https:\/\/en.pingcap.com\/blog\/tidb-binlog-tutorial\/"},"modified":"2023-09-12T03:52:57","modified_gmt":"2023-09-12T10:52:57","slug":"tidb-binlog-tutorial","status":"publish","type":"post","link":"https:\/\/www.pingcap.com\/ko\/blog\/tidb-binlog-tutorial\/","title":{"rendered":"TiDB Binlog Tutorial"},"content":{"rendered":"

This tutorial will start with a very simple TiDB Binlog deployment with a single node of each component (Placement Driver, TiKV Server, TiDB Server, Pump, and Drainer), set up to push data into a MariaDB Server instance.<\/p>\n

This tutorial is targeted toward users who have some familiarity with the TiDB Architecture<\/a>, who may have already set up a TiDB cluster (though that is not mandatory), and who wants to gain hands-on familiarity with the features and functionality of TiDB Binlog. This tutorial is a good way to “kick the tires” of TiDB Binlog and to familiarize yourself with the concepts of its architecture.<\/p>\n

Warning:<\/strong><\/p>\n

The methodology used to deploy TiDB in this tutorial should not<\/strong> be used to deploy TiDB in a production or development setting.<\/p><\/blockquote>\n

This tutorial assumes you’re using a modern Linux distribution on x86-64. I’ll use a minimal CentOS 7 installation running in VMware for the examples. It’ll be easiest if you start from a clean install, so that you aren’t impacted by quirks of your existing environment. If you don’t want to use local virtualization, you can easily and inexpensively start a CentOS 7 VM in your favorite cloud provider.<\/p>\n

<\/span>Overview<\/span><\/h2>\n

TiDB Binlog is a solution to collect binary log data from TiDB server and provide real-time data backup and replication. It pushes incremental data updates made to a TiDB Server cluster into any of various downstream platforms.<\/p>\n

You can use TiDB Binlog for incremental backups, to replicate data from one TiDB cluster to another, or to send TiDB updates through Kafka to a downstream platform of your choice.<\/p>\n

TiDB Binlog is particularly useful when you migrate data from MySQL or MariaDB to TiDB, in which case you may use the TiDB DM (Data Migration) platform to get data from a MySQL\/MariaDB cluster into TiDB, and then use TiDB Binlog to keep a separate, downstream MySQL\/MariaDB instance\/cluster in sync with your TiDB cluster. TiDB Binlog enables application traffic to TiDB to be pushed to a downstream MySQL or MariaDB instance\/cluster, which reduces the risk of a migration to TiDB because you can easily revert the application to MySQL or MariaDB without downtime or data loss.<\/p>\n

See TiDB Binlog Cluster Overview<\/a> for more information.<\/p>\n

<\/span>Architecture<\/span><\/h2>\n

TiDB Binlog comprises two components: the Pump<\/strong> and the Drainer<\/strong>. Several Pump nodes make up a Pump cluster. Each Pump node connects to TiDB Server instances and receives updates made to each of the TiDB Server instances in a cluster. A Drainer connects to the Pump cluster and transforms updates into the correct format for a particular downstream destination, be it Kafka or another TiDB Cluster or a MySQL\/MariaDB server.<\/p>\n

The clustered architecture of the Pump component ensures that updates won’t be lost as new TiDB Server instances join or leave the TiDB Cluster or Pump nodes join or leave the Pump cluster.<\/p>\n

<\/span>Installation<\/span><\/h2>\n

We’re using MariaDB Server in this case instead of MySQL Server because RHEL\/CentOS 7 includes MariaDB Server in their default package repositories. We’ll need the client as well as the server for later, so let’s install them now:<\/p>\n

sudo yum install -y mariadb-server\n<\/code><\/pre>\n
Even if you’ve already started a TiDB cluster, it will be easier to follow along with this tutorial if you set up a new, very simple cluster. We will install from a tarball, using a simplified form of the Local Deployment<\/a> guide. You may also wish to consult Testing Deployment from Binary Tarball<\/a> for best practices establishing a real testing deployment that goes beyond the scope of this tutorial.<\/p>\n
curl -L http:\/\/download.pingcap.org\/tidb-latest-linux-amd64.tar.gz | tar xzf -\ncd tidb-latest-linux-amd64\n<\/code><\/pre>\n
Expect this output:<\/p>\n
[kolbe@localhost ~]$ curl -LO http:\/\/download.pingcap.org\/tidb-latest-linux-amd64.tar.gz | tar xzf -\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n100  368M  100  368M    0     0  8394k      0  0:00:44  0:00:44 --:--:-- 11.1M\n[kolbe@localhost ~]$ cd tidb-latest-linux-amd64\n[kolbe@localhost tidb-latest-linux-amd64]$\n<\/code><\/pre>\n
<\/span>Configuration<\/span><\/h2>\n
Now we’ll start a very simple TiDB cluster, with a single instance each of pd-server<\/code>, tikv-server<\/code>, and tidb-server<\/code>.<\/p>\n
First, let’s populate the config files we’ll use:<\/p>\n
printf > pd.toml %s\\n 'log-file=\"pd.log\"' 'data-dir=\"pd.data\"'\nprintf > tikv.toml %s\\n 'log-file=\"tikv.log\"' '[storage]' 'data-dir=\"tikv.data\"' '[pd]' 'endpoints=[\"127.0.0.1:2379\"]' '[rocksdb]' max-open-files=1024 '[raftdb]' max-open-files=1024\nprintf > pump.toml %s\\n 'log-file=\"pump.log\"' 'data-dir=\"pump.data\"' 'addr=\"127.0.0.1:8250\"' 'advertise-addr=\"127.0.0.1:8250\"' 'pd-urls=\"http:\/\/127.0.0.1:2379\"'\nprintf > tidb.toml %s\\n 'store=\"tikv\"' 'path=\"127.0.0.1:2379\"' '[log.file]' 'filename=\"tidb.log\"' '[binlog]' 'enable=true'\nprintf > drainer.toml %s\\n 'log-file=\"drainer.log\"' '[syncer]' 'db-type=\"mysql\"' '[syncer.to]' 'host=\"127.0.0.1\"' 'user=\"root\"' 'password=\"\"' 'port=3306'\n<\/code><\/pre>\n
This will allow you to see the contents of the config files:<\/p>\n
for f in *.toml; do echo \"$f:\"; cat \"$f\"; echo; done\n<\/code><\/pre>\n
Expect this output:<\/p>\n
drainer.toml:\nlog-file=\"drainer.log\"\n[syncer]\ndb-type=\"mysql\"\n[syncer.to]\nhost=\"127.0.0.1\"\nuser=\"root\"\npassword=\"\"\nport=3306\n\npd.toml:\nlog-file=\"pd.log\"\ndata-dir=\"pd.data\"\n\npump.toml:\nlog-file=\"pump.log\"\ndata-dir=\"pump.data\"\naddr=\"127.0.0.1:8250\"\nadvertise-addr=\"127.0.0.1:8250\"\npd-urls=\"http:\/\/127.0.0.1:2379\"\n\ntidb.toml:\nstore=\"tikv\"\npath=\"127.0.0.1:2379\"\n[log.file]\nfilename=\"tidb.log\"\n[binlog]\nenable=true\n\ntikv.toml:\nlog-file=\"tikv.log\"\n[storage]\ndata-dir=\"tikv.data\"\n[pd]\nendpoints=[\"127.0.0.1:2379\"]\n[rocksdb]\nmax-open-files=1024\n[raftdb]\nmax-open-files=1024\n<\/code><\/pre>\n
<\/span>Bootstrapping<\/span><\/h2>\n
Now we can start each component. This is best done in a specific order, first bringing up the PD (Placement Driver), then TiKV Server (the backend key\/value store used by TiDB Platform), then Pump (because TiDB must connect to the Pump service to send the binary log), and finally TiDB Server (the frontend that speaks the MySQL protocol to your applications).<\/p>\n
Start all the services:<\/p>\n
.\/bin\/pd-server --config=pd.toml &>pd.out &\n.\/bin\/tikv-server --config=tikv.toml &>tikv.out &\n.\/bin\/pump --config=pump.toml &>pump.out &\nsleep 3\n.\/bin\/tidb-server --config=tidb.toml &>tidb.out &\n<\/code><\/pre>\n
Expect this output:<\/p>\n
[kolbe@localhost tidb-latest-linux-amd64]$ .\/bin\/pd-server --config=pd.toml &>pd.out &\n[1] 20935\n[kolbe@localhost tidb-latest-linux-amd64]$ .\/bin\/tikv-server --config=tikv.toml &>tikv.out &\n[2] 20944\n[kolbe@localhost tidb-latest-linux-amd64]$ .\/bin\/pump --config=pump.toml &>pump.out &\n[3] 21050\n[kolbe@localhost tidb-latest-linux-amd64]$ sleep 3\n[kolbe@localhost tidb-latest-linux-amd64]$ .\/bin\/tidb-server --config=tidb.toml &>tidb.out &\n[4] 21058\n<\/code><\/pre>\n
And if you execute jobs<\/code>, you should see the list of running daemons:<\/p>\n
[kolbe@localhost tidb-latest-linux-amd64]$ jobs\n[1]   Running                 .\/bin\/pd-server --config=pd.toml &>pd.out &\n[2]   Running                 .\/bin\/tikv-server --config=tikv.toml &>tikv.out &\n[3]-  Running                 .\/bin\/pump --config=pump.toml &>pump.out &\n[4]+  Running                 .\/bin\/tidb-server --config=tidb.toml &>tidb.out &\n<\/code><\/pre>\n
If one of the services has failed to start (if you see “Exit 1<\/code>” instead of “Running<\/code>“, for example), try to restart that individual service.<\/p>\n