{"id":29676,"date":"2025-10-21T06:16:45","date_gmt":"2025-10-21T13:16:45","guid":{"rendered":"https:\/\/www.pingcap.com\/?p=29676"},"modified":"2025-11-03T07:13:57","modified_gmt":"2025-11-03T15:13:57","slug":"kafka-streaming-data-integrate-with-tidb","status":"publish","type":"post","link":"https:\/\/www.pingcap.com\/ko\/blog\/kafka-streaming-data-integrate-with-tidb\/","title":{"rendered":"How to Stream Data from Kafka to TiDB"},"content":{"rendered":"

Modern applications generate enormous amounts of event data with user actions, transactions, logs, and metrics all happening in real time. To handle this scale, many teams rely on Apache Kafka<\/a>, a distributed messaging system that decouples applications from their data pipelines<\/a> and ensures reliable, high-throughput data delivery<\/a>.<\/p>\n\n\n\n

On the storage side, TiDB provides a distributed SQL database that scales horizontally, handles both transactional and analytical queries, and maintains low-latency performance<\/a> even under heavy load.<\/p>\n\n\n\n

Together, Kafka and TiDB form a powerful foundation for real-time workloads where high write throughput and fast data processing<\/a> are critical.<\/p>\n\n\n\n

This two-part blog tutorial explores how to integrate Kafka with TiDB. Part 1 covers the basics around how to stream data from Kafka to TiDB and why this architecture is becoming increasingly popular. Part 2 will examine how TiDB performs when Kafka processes millions of messages per second and how to monitor TiDB\u2019s internal performance<\/a>.<\/p>\n\n\n\n

<\/span>Why Stream Data through Kafka?<\/strong><\/span><\/h2>\n\n\n\n

A recent customer project involved an application that sent messages<\/a> directly to Kafka. From Kafka, data flowed into a persistent storage layer that included systems such as SQL Server and Cassandra.<\/p>\n\n\n\n

This design choice is common for systems that handle large volumes of writes. Sending data directly to a database under heavy load can lead to latency issues, slowing down the entire application. Kafka helps mitigate this by acting as a buffer between the application and the database, ensuring that high-frequency writes are first collected and processed asynchronously before reaching the storage layer.<\/p>\n\n\n\n

By decoupling ingestion from persistence, Kafka maintains consistent performance and reliability even during spikes in traffic.<\/p>\n\n\n\n

\"How<\/figure>\n\n\n\n

Fig. 1: How Kafka decouples application data streams<\/em><\/p>\n\n\n\n

<\/span>What is TiDB?<\/strong><\/span><\/h2>\n\n\n\n

TiDB is an open-source, distributed SQL database<\/a> designed for horizontal scalability, strong consistency, and high availability. It uses a decoupled compute and storage architecture, allowing each layer to scale independently \u2014 a key advantage for cost and performance optimization.<\/p>\n\n\n\n

TiDB is MySQL \ud638\ud658<\/a>, which means existing applications, drivers, and SQL syntax can often be reused with minimal modification. This compatibility also makes it easier to migrate from other databases such as MySQL, PostgreSQL, or MongoDB.<\/p>\n\n\n\n

\"How<\/figure>\n\n\n\n

Fig. 2: How TiDB complements Kafka for unified workloads<\/em><\/p>\n\n\n\n

Let’s start with an example from a test TiDB instance running on the cloud:
<\/p>\n\n\n\n

ankitkapoor@Ankits-MacBook-Air bin % .\/mysql -uankit -hxxx -P 4000 -p\n\n\nmysql> show databases;\n+--------------------+\n| Database           |\n+--------------------+\n| INFORMATION_SCHEMA |\n| PERFORMANCE_SCHEMA |\n| ankit              |\n| kafka              |\n| mysql              |\n| test               |\n+--------------------+\n6 rows in set (0.10 sec)<\/code><\/pre>\n\n\n\n
<\/span>What About TiDB to Kafka?<\/strong><\/span><\/h2>\n\n\n\n
\ud2f0DB<\/a> can also stream data to Kafka using TiCDC<\/a>, which is documented \uc5ec\uae30<\/a>.<\/p>\n\n\n\n
TiCDC (TiDB Change Data Capture) is a component that captures real-time changes from TiDB and replicates them downstream. It reads Raft logs, internal records that track every change in the TiDB cluster, and pushes those changes to external systems like Kafka, another TiDB cluster, or cloud storage.<\/p>\n\n\n\n
For reference, TiDB Raft log files<\/a> typically look like this:<\/p>\n\n\n\n
-rw-r--r--  1 ankitkapoor  cc    69B 11 Sep 02:29 0000000000000001.rewrite\n-rw-r--r--  1 ankitkapoor  cc     0B 11 Sep 02:39 LOCK\n-rw-r--r--  1 ankitkapoor  cc   869K 11 Sep 02:39 0000000000000001.raftlog  <\u2014 Raft log\n<\/code><\/pre>\n\n\n\n
While TiCDC handles data from TiDB to Kafka, this article focuses on the reverse, Kafka to TiDB.<\/p>\n\n\n\n
<\/span>Kafka to TiDB: Overview<\/span><\/h2>\n\n\n\n
Streaming data from Kafka to TiDB is often achieved using Kafka Connect, an open-source framework for building scalable and reliable data pipelines. While other tools like PySpark can accomplish this, Kafka Connect provides a simpler and more performant approach, especially for production environments.<\/p>\n\n\n\n
Since TiDB is MySQL-compatible, existing MySQL JDBC drivers can be used to set up the data stream between Kafka and TiDB.<\/p>\n\n\n\n
Requirements<\/strong><\/h3>\n\n\n\n
To follow this guide, the following components are required:<\/p>\n\n\n\n
    \n
  1. Kafka<\/li>\n\n\n\n
  2. Zookeeper<\/li>\n\n\n\n
  3. Kafka-topics<\/li>\n\n\n\n
  4. Kafka-console-producer<\/li>\n\n\n\n
  5. Kafka-console-consumer<\/li>\n\n\n\n
  6. Kafka Sink connector<\/li>\n\n\n\n
  7. MySQL client<\/li>\n\n\n\n
  8. TiDB cluster<\/li>\n<\/ol>\n\n\n\n
    Test Environment<\/h3>\n\n\n\n