Real-Time Streaming: Integrating TiDB Cloud Changefeed with StreamNative Cloud

At PingCAP, our mission with TiDB Cloud is to make data easier, faster, and more powerful for developers and enterprises. Today, we’re taking that vision one step further with a new capability: streaming real-time data changes directly from TiDB Cloud into the StreamNative Cloud ecosystem.

This integration, powered by TiDB Cloud’s Changefeed feature, gives you the ability to capture and propagate database events the moment they happen. Combined with StreamNative’s Apache Pulsar–based platform, this enables low-latency data distribution and event-driven architectures that keep businesses responsive in an always-on world.

TiDB Cloud offers a powerful, distributed SQL database that excels at handling large-scale, high-concurrency workloads. StreamNative’s ONE platform, powered by Apache Pulsar, on the other hand, provides a highly scalable, low-latency messaging and streaming platform designed for real-time data ingestion and distribution. 

The new changefeed feature bridges these two powerful platforms, enabling seamless data flow from TiDB Cloud to StreamNative Cloud.

This integration allows users to capture data changes in TiDB Cloud and stream them directly into Apache Pulsar topics on StreamNative Cloud. This opens up a multitude of use cases, including:

• Real-time Analytics: Feed operational data from TiDB Cloud into Pulsar for immediate processing and analytical insights.
• Event-Driven Applications: Build reactive applications that respond instantly to database changes.
• Data Synchronization: Keep various data systems consistent by propagating changes efficiently.
• Data Lake Ingestion: Stream TiDB Cloud data into data lakes for historical analysis and machine learning.

Real-Time Streaming: Getting Started with TiDB Cloud Changefeed and StreamNative Cloud

Getting started with this powerful integration is straightforward. Here’s a quick guide to help you set up your first changefeed:

Prerequisites

• A TiDB Cloud account
• A StreamNative Cloud account
• Basic understanding of Apache Pulsar topics and subscriptions

Step 1: Prepare Your StreamNative Cloud Environment

Before configuring the changefeed, you need to set up the destination topic and get authentication details from your StreamNative Cloud console.

1. Log in to your StreamNative Cloud console. You can create a serverless cluster to get started quickly.
2. Create a Topic: Navigate to the Topics section, select your desired tenant and namespace, and create a new topic (e.g., tidb-changes). Note the full topic name, as you will need it later.
3. Get Authentication Key: Navigate to ‘Pulsar Clients‘, select your service account, then click Create API Key to generate a token. Copy this token securely.

Fig 1: Prepare Your StreamNative Cloud Environment

Step 2: Configure TiDB Cloud Changefeed

With your StreamNative Cloud access APIKey ready, you can now create the changefeed in TiDB Cloud.

1. Log in to your TiDB Cloud console.
2. Navigate to the Changefeed section and create a new changefeed.
3. Select Pulsar as the destination.
4. Under Connection, provide your Pulsar Broker URL and select the Token Auth Type.
5. Enter the broker Pulsar Broker URL and API key you obtained in the previous step.

Fig 2: Configure TiDB Cloud Changefeed

Step 3: Create a Table and Insert Data in TiDB Cloud

Now that the changefeed is set up, you can generate change events by executing SQL commands in the TiDB Cloud Shell.

1. Open the TiDB Cloud Shell for your cluster.
2. Run the following SQL statements to create a users table and insert two records:

CREATE TABLE users (
    id BIGINT PRIMARY KEY AUTO_RANDOM,
    name VARCHAR(255),
    email VARCHAR(255)
);

INSERT INTO users (name, email) VALUES 
('Bob', 'bob@example.com'),
('Charlie', 'charlie@example.com');

Step 4: Verify Data in StreamNative Cloud

The SQL command change events stream directly to your StreamNative Apache Pulsar topic. A Pulsar consumer can then verify that the data has been received correctly.

The consumer will receive the change events in a structured JSON format. The first event will be the DDL for the CREATE TABLE command, followed by separate DML events for each INSERT operation.

{"id":0,"database":"test","table":"users","pkNames":null,"isDdl":true,"type":"CREATE","es":1754465697775,"ts":1754465697959,"sql":"CREATE TABLE `users` (`id` BIGINT PRIMARY KEY /*T![auto_rand] AUTO_RANDOM */,`name` VARCHAR(255),`email` VARCHAR(255))","sqlType":null,"mysqlType":null,"data":null,"old":null}

{"id":0,"database":"test","table":"users","pkNames":["id"],"isDdl":false,"type":"INSERT","es":1754465706724,"ts":1754465707051,"sql":"","sqlType":{"id":-5,"name":12,"email":12},"mysqlType":{"id":"bigint","name":"varchar","email":"varchar"},"old":null,"data":[{"id":"3458764513820540929","name":"Bob","email":"bob@example.com"}]}

{"id":0,"database":"test","table":"users","pkNames":["id"],"isDdl":false,"type":"INSERT","es":1754465706724,"ts":1754465707051,"sql":"","sqlType":{"id":-5,"name":12,"email":12},"mysqlType":{"name":"varchar","email":"varchar","id":"bigint"},"old":null,"data":[{"id":"3458764513820540930","name":"Charlie","email":"charlie@example.com"}]}

For detailed instructions and advanced configurations, please refer to the official documentation:

• TiDB Cloud Documentation
• StreamNative Documentation

Real-Time Streaming: Scaling from Data to Decisions

The addition of changefeed integration underscores TiDB Cloud’s role as the backbone for modern, data-intensive applications. By bridging TiDB Cloud’s distributed SQL power with StreamNative’s real-time streaming platform, we’re enabling developers to unify their operational and analytical data flows without added complexity.

Whether you’re building real-time analytics pipelines, event-driven microservices, or seamless data lake ingestion, TiDB Cloud gives you the flexibility to scale with confidence.

We invite you to try this feature, experience TiDB Cloud and StreamNative Cloud together to build applications that are real-time, resilient, and future-ready.