Supercharging Real-Time Applications with TiDB and DragonflyDB

Data-intensive applications demand scalability, low latency, and resilience. However, traditional databases often struggle to handle both transactional consistency and fast in-memory caching at scale. But that’s where TiDB and DragonflyDB shine together:

• TiDB : a distributed, MySQL-compatible database designed for massive scalability and mixed workload processing.
• DragonflyDB: a modern, Redis compatible in-memory data store that delivers sub-millisecond caching and queuing performance with much better efficiency than Redis.

In this tutorial, we’ll walk through setting up a TiDB + Dragonfly stack, show how they complement each other, and build a hands-on example for a real-time leaderboard system.

TiDB gives you the scale and consistency of a distributed SQL database, while DragonflyDB delivers blazing-fast, Redis-compatible caching. Together, they power real-time use cases like leaderboards, payments, and feeds.

Think of TiDB as your engine and DragonflyDB as your turbocharger.

Why TiDB + DragonflyDB?

Both TiDB (like MySQL) and DragonflyDB are multi-threaded by design. This is critical for data-intensive workloads.

TiDB’s architecture distributes queries and transactions across multiple nodes and threads for parallel execution.

• TiDB Strengths: Horizontal scalability, strong consistency, mixed workload processing.

On the other hand, DragonflyDB can fully utilize multi-core CPUs, unlike Redis which is single-threaded by default. This means DragonflyDB can handle far more throughput per instance with the same hardware.

• DragonflyDB Strengths: Lightning-fast cache, Redis drop-in replacement, multi-threaded scalability. DragonflyDB is also horizontally scalable with Dragonfly Swarm, a multi-node clustering solution.

Together, TiDB serves as the durable source of truth, while DragonflyDB accelerates reads, queues, and ephemeral data.

DragonflyDB: Drop-In Redis Replacement

DragonflyDB is 100% Redis-compatible. Migrating from Redis usually just requires changing the endpoint in your application code — no rewrites needed. For example:

# Before: using Redis
r = redis.Redis(host="redis-host", port=6379)

# After: using Dragonfly
r = redis.Redis(host="dragonfly-host", port=6379)

That’s it. Your Redis commands will just work, but with significantly better performance and efficiency.

Next, we’ll walk you through how to get started running TiDB and DragronflyDB together in Docker.

Step 1. Setting Up with Colima: Install Colima + Docker CLI

brew install colima docker docker-compose
colima start --cpu 4 --memory 8 --disk 20
docker context use colima

Step 2. Run TiDB

If you don’t already have a TiDB cluster, the easiest path is TiDB Cloud Starter.
You can also run it locally with Docker:

docker run -d --name tidb -p 4000:4000 pingcap/tidb:latest

Step 3. Run DragonflyDB

DragonflyDB runs easily in Docker as well:

docker run -d --name dragonfly -p 6379:6379 docker.dragonflydb.io/dragonflydb/dragonfly

Now you have a Redis-compatible cache at port 6379.

At this point, both TiDB and Dragonfly should be running inside Colima. Let’s confirm:

docker ps

Next, confirm TiDB & DragonflyDB are both reachable:

mysql -h 127.0.0.1 -P 4000 -uroot -e "SELECT VERSION();"
redis-cli -h 127.0.0.1 -p 6379 PING

Step 4. Create Schema in TiDB

Let’s create a durable leaderboard table in TiDB.

CREATE DATABASE game;
USE game;

CREATE TABLE leaderboard (
  user_id BIGINT PRIMARY KEY,
  username VARCHAR(50),
  score INT,
  updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
);

Step 5. Building a Read-Through Leaderboard with Python (Example)

You’ll need to set up a Virtual Environment (PEP 668 safe). On macOS/Linux, don’t install Python libraries globally. Instead:

python3 -m venv venv
source venv/bin/activate

You’ll see (venv) in your shell prompt. Now install dependencies:

pip install mysql-connector-python redis

Python Demo Code (Read-Through Cache)

Here’s the heart of our demo: a read-through cache pattern in Python.

Save as leaderboard_demo.py:

# leaderboard_demo.py
import mysql.connector
import redis
from datetime import datetime, timedelta

# --- Connections ---
db = mysql.connector.connect(
  host="127.0.0.1", port=4000, user="root", password="", database="game"
)
cursor = db.cursor(dictionary=True)  # dict rows for convenience
r = redis.Redis(host="127.0.0.1", port=6379)

CACHE_TTL_SECONDS = 300  # 5m; tune for your app

def _lb_key(period: str | None = None) -> str:
  # Examples: leaderboard:alltime, leaderboard:2025-09
  if period in (None, "alltime"):
      return "leaderboard:alltime"
  return f"leaderboard:{period}"

def add_score(user_id: int, username: str, score: int):
  # Write-through: TiDB first (source of truth)
  cursor.execute(
      "INSERT INTO leaderboard (user_id, username, score) VALUES (%s,%s,%s) "
      "ON DUPLICATE KEY UPDATE username=VALUES(username), score=VALUES(score)",
      (user_id, username, score),
  )
  db.commit()

  # Then update Dragonfly cache(s). For simplicity we update all-time only here.
  r.zadd(_lb_key("alltime"), {username: score})
  r.expire(_lb_key("alltime"), CACHE_TTL_SECONDS)

def get_top_players(n: int = 10, period: str | None = "alltime"):
  """
  Read-through: try Dragonfly; on miss, query TiDB, rehydrate Dragonfly, return data.
  period:
    - "alltime" (default)
    - "YYYY-MM" for monthly boards (e.g., "2025-09")
  """
  key = _lb_key(period)
  results = r.zrevrange(key, 0, n - 1, withscores=True)

  if results:  # cache hit
      return results

  # --- Cache miss: rebuild from TiDB ---
  if period in (None, "alltime"):
      sql = (
          "SELECT username, score FROM leaderboard "
          "ORDER BY score DESC LIMIT %s"
      )
      params = (n,)
  else:
      # Monthly board using updated_at; adjust to your business rule
      start = datetime.strptime(period, "%Y-%m")
      end = (start.replace(day=28) + timedelta(days=4)).replace(day=1)
      sql = (
          "SELECT username, score FROM leaderboard "
          "WHERE updated_at >= %s AND updated_at < %s "
          "ORDER BY score DESC LIMIT %s"
      )
      params = (start, end, n)

  cursor.execute(sql, params)
  rows = cursor.fetchall()

  # Rehydrate Dragonfly (pipeline for speed)
  pipe = r.pipeline()
  pipe.delete(key)
  for row in rows:
      pipe.zadd(key, {row["username"]: int(row["score"])})
  pipe.expire(key, CACHE_TTL_SECONDS)
  pipe.execute()

  # Return in the same shape as ZREVRANGE ... WITHSCORES
  return [(row["username"].encode("utf-8"), float(row["score"])) for row in rows]

# --- demo ---
if __name__ == "__main__":
  add_score(1, "Alice", 1500)
  add_score(2, "Bob", 2000)
  print("Top Players (all-time):", get_top_players())

• get_top_players() implements a read-through cache. It calls ZREVRANGE; if the list is empty (key missing/expired), it queries TiDB, rehydrates DragonflyDB with ZADD, sets a short TTL, and returns the results.
• Why it matters: This shows TiDB on both the write path and the read path while keeping hot reads sub-ms from DragonflyDB.

Monthly Leaderboards (Period-Based Queries)

Use period=”YYYY-MM” to fetch a monthly leaderboard using the updated_at column. Example:

python -c 'from leaderboard_demo import get_top_players; print(get_top_players(10, period="2025-09"))'

This function computes the month window and rebuilds the cache on a miss.

Step 6. Testing Cache Miss & Rehydration:

Next, delete the cache key to start clean:

redis-cli -h 127.0.0.1 -p 6379 DEL leaderboard:2025-09
redis-cli -h 127.0.0.1 -p 6379 CONFIG RESETSTAT

First read -> miss + rebuild from TiDB (filtered by updated_at month window):

python -c 'from leaderboard_demo import get_top_players; print(get_top_players(10, period="2025-09"))'

Check stats (expect a miss):

redis-cli -h 127.0.0.1 -p 6379 INFO stats | egrep "keyspace_hits|keyspace_misses"

Second read -> hit:

python -c 'from leaderboard_demo import get_top_players; print(get_top_players(10, period="2025-09"))'

Check stats again (hits should increase):

redis-cli -h 127.0.0.1 -p 6379 INFO stats | egrep "keyspace_hits|keyspace_misses"

See the month key content:

redis-cli -h 127.0.0.1 -p 6379 ZREVRANGE leaderboard:2025-09 0 9 WITHSCORES

Production Notes

• TTL & freshness: Tune CACHE_TTL_SECONDS per workload; add jitter (±20%) to avoid thundering herds.
• Cache stampede control: Use lightweight locks (SETNX / SET lock:key token NX EX 10) so only one process rebuilds on expiry.
• Indexes: Add INDEX(updated_at) in TiDB if you’ll query monthly leaderboards often.

Conclusion

By combining TiDB’s distributed, multi-threaded architecture with DragonflyDB’s drop-in Redis replacement and multi-threaded caching engine, you get the best of both worlds:

• Strong durability, MySQL compatibility, and scalable analytics with TiDB.
• Ultra-fast cache, queues, and leaderboards with DragonflyDB without rewriting Redis code.

This architecture fits perfectly for gaming, fintech, ad tech, and social apps where speed and reliability matter equally.

If you’re building apps that need to scale without compromise, try pairing TiDB and DragonflyDB. It’s like giving your database rocket fuel.

You can also check out more free, hands-on tutorials by heading over to TiDB Labs.