The NFT market is booming
An NFT, or non-fungible token, is a blockchain-based digital asset. Unlike Bitcoin and Ether, two world leading cryptocurrencies that are interchangeable and fungible, an NFT represents the unique ownership of a digital item such as a painting, a song, a video, and even a tweet. It cannot be copied, substituted, or subdivided, but it can be held, transferred, and traded.
The recent years have witnessed the booming of the NFT market. In 2021 alone, the NFT trading volume hit more than $17 billion, a 21,000% increase over 2020. Tens of thousands of artists have produced and traded NFT artworks. NFT marketplaces such as Opensea and Coinbase that offer platforms for NFT transactions are springing up like mushrooms after a rain.
The traditional data architecture of an NFT marketplace
An NFT marketplace consists of multiple components including digital wallets, smart contracts, databases, and blockchains. One of the most crucial components is the database. It stores, processes, and analyzes massive amounts of blockchain data. If you are a new participant in the Web3 or NFT ecosystem and plan to build an NFT platform yourself, how will you design your data architecture and choose your database?
Traditional data architectures need at least the following elements:
- A blockchain subscriber
Because there are huge amounts of NFT transactions happening per second on multiple blockchains such as Ethereum, Solana, Polygon, and BNB Chain, you’ll need a blockchain subscriber to collect all those NFT transactions.
- A message queue (MQ) system like Kafka and an MQ reader
A message queue system like Kafka helps you receive and channel the NFT transaction data you collect from blockchains. Then, an MQ reader consumes the messages asynchronously in the message queue and bulk inserts them into an analytical database.
- An OLAP database
An Online Analytical Processing (OLAP) database such as the combination of Apache Hive and Apache Spark helps you analyze the NFT transactions and respond to analytical queries.
- An OLTP database
An Online Transactional Processing (OTLP) database like MySQL stores the analytical results returned by the OLAP database and handles the requests from the web server of your NFT platform.
- A web server
You’ll also need a web server to run your NFT platform.
Traditional data architecture for an NFT marketplace
Pain points you’ll probably encounter
When you implement, run, and operate the traditional data system for your NFT marketplace, you’ll probably meet the following pain points.
- You can hardly get real-time NFT deals. Every second, numerous NFT items are created and traded on each blockchain. Your NFT marketplace must be agile enough to get the most up-to-date information in the NFT market. The faster your platform reacts, the more accurate NFT deals your users can get. But the existing data architecture has a long data pipeline with a complex technology stack, which undermines the agility of your platform.
- The size of the NFT data stored in blockchains is enormous and is growing continuously. If you store all existing NFT transactions from mainstream blockchains along with their metadata, you’ll need at least three terabytes of storage space, even after the data is compressed. Although most data warehouses can hold such huge amounts of data, most OLTP databases struggle to do so.
- The technology stack is too complex. If you choose the traditional data architecture, you’ll need a big data system (let’s say Apache Hive) to import the historical NFT data from blockchains to your NFT marketplace storage system. You also need to build an extra program to subscribe to real-time transaction data separately from blockchains or the NFT API, write it to a message queue, and then batch write it to the big data system. What’s more? You need an OLTP database to store the analytical results from Apache Spark. Not only is this expensive, but it also undermines your platform agility.
TiDB is an open-source, distributed, MySQL compatible database that supports Hybrid Transactional and Analytical Processing (HTAP) workloads. It is also horizontally scalable, strongly consistent, and highly availabilable.
TiDB can solve the pain points of NFT marketplaces because it can:
- Handle both real-time writes and analytics. TiDB stores data in both row and column format, and the data is strongly consistent. TiDB can analyze real-time data while accepting high-workload writes. It can calculate analytical information such as NFT rankings in real time.
- Scale out easily. TiDB can horizontally scale out on demand to adapt to your rapidly growing volume of the blockchain data.
- Simplify the technology stack. TiDB can replace the Apache Kafka and Hive parts of the traditional data architecture, accept high QPS writes, and store large amounts of data. It is also a perfect replacement of MySQL, which can directly query the analytical results in real time and display them on the NFT marketplace. With TiDB, you can quickly develop and release your minimum viable products.
- Support JSON format. The most common format for NFT metadata is JSON. TiDB fully supports the JSON format, so it can fetch and store the blockchain data directly without predefining the metadata schema.
The new data architecture with TiDB for an NFT marketplace
In addition to the on-premises version of TiDB, you can also try TiDB Cloud, a fully managed Database-as-a-Service (DBaas) that can bring everything great about TiDB to your cloud. TiDB Cloud now is available on AWS and Google Cloud.
TiDB Cloud architecture and how TiDB Cloud supports applications
How a leading NFT participant uses TiDB
One of TiDB adopters is a leading participant in the NFT market. They track NFT transactions on dozens of public chains and help NFT users discover investment opportunities with real-time NFT information.
They previously used a data warehouse to store all their NFT transaction data, made daily batch analytics, and then wrote the analytical results to an OLTP database. With this data solution, their data latency was as long as one day. Since they migrated data to TiDB, the data latency was reduced from one day to seconds. Their data volume has tripled to almost 20 terabytes. Thanks to its elastic scalability, TiDB could easily support their huge data increase.
In addition, by adopting TiDB, they get a much more simplified data architecture, which greatly reduces their engineering, operation, and maintenance costs.
Try TiDB Cloud Free Now!
Get the power of a cloud-native, distributed SQL database built for real-time
analytics in a fully managed service.
Try our NFT demo now!
This demo can:
- Scan historical NFT transactions on Ethereum and track real-time transactions.
- Browse the hottest items in the NFT market.
An NFT demo: TiDB Cloud NFT Quickstart
If you are interested in the demo source code, you can find it here on GitHub.
Keep in contact with us!
If you’re interested in TiDB or TiDB Cloud, you’re welcome to contact our sales staff for more information. You can also join our community on Slack and TiDB Internals to share your thoughts with us. Don’t forget to follow us on Twitter, LinkedIn, or GitHub for the latest information.
How to Efficiently Choose the Right Database for Your Applications
Technical Paths to HTAP: GreenPlum and TiDB as Examples
How to Process and Analyze Billions of Rows of Data in a Single Table with TiDB on AWS
Edited by: Fendy Feng, Tom Dewan
A fully-managed cloud DBaaS for predictable workloads
A fully-managed cloud DBaaS for auto-scaling workloads