AmzTrends’ Cloud-Native Transformation: 80% Cost Saving with TiDB Serverless

Through cloud-native transformation, AmzTrends unlocked unparalleled cost saving, slashing their overall expenses by 80% through a shift to TiDB Serverless.

AmzTrends is a software-as-a-service (SaaS) company that specializes in providing data analytics for Amazon sellers. It offers insights into brand performance, advertising strategies, and market trends. The core of AmzTrends’ offering is the analysis of Amazon’s Brand Analytics (ABA) data, updated weekly via Amazon’s API. This analysis involves comparing current and historical datasets to identify new growth opportunities. So far, AmzTrends has served hundreds of e-commerce companies across the United States, Japan, China, and other regions.  T

The Challenges: Navigating Massive Data Volume and Infrastructure Complexity 

Managing an expansive dataset of over 2 billion data points accumulated from ABA over years was a formidable challenge for AmzTrends. The sheer volume demanded a sophisticated, scalable infrastructure capable of efficient processing and analysis.

Initially, AmzTrends constructed a self-managed TiDB cluster on a public cloud service, comprising:

• 1 TiDB node for SQL layer query processing.
• 1 PD (Placement Driver) node- a single-point setup that risked high availability for cost savings against best practices.
• 3 TiKV nodes for data storage and transaction processing.

Figure 1. AmzTrends Original Architecture

Additionally, AmzTrends employed 3 separate servers to run Spark for full-volume data analysis. It was a choice that increased operational complexity and costs due to Spark’s specific maintenance requirement. This setup, while functional in the beginning, introduced several inefficiencies:

• Complex maintenance: The original deployment was both costly and complex, requiring significant manual intervention for operations, maintenance, and scaling, 
• Resource redundancy: The separate Spark servers added complexity, leading to underutilized resources and further cost implications.
• Scalability Concerns: Handling the growing data volume efficiently became increasingly challenging, given the manual processes involved.

The Solution: Embracing TiDB Serverless for a Scalable, Cost-Efficient Solution 

Seeking to address these challenges, AmzTrends evaluated TiDB Serverless, a fully managed DBaaS solution offering auto-scaling, cost efficiency, and simplified operations. 

Why TiDB Serverless?

TiDB Serverless aligns with AmzTrends’ business and technical goals:

Cost-Effective Auto-scaling: AmzTrends features a variable workload that peaks during business hours and idles during off-hours. TiDB Serverless automatically scales in seconds to match your various needs for storage, transaction processing, and analytics. This, combined with a pay-for-what-you-use pricing model, avoids unnecessary resource redundancy and ensures AmzTrends pays only for what’s needed.

Simplification of database operations: The transition to TiDB Serverless eliminated the need for manual data sharding and complex maintenance tasks. AmzTrends could now effectively manage its vast dataset like a single table. This simplification supports queries over extended periods without the repetitive task of data reorganization. 

Hassle-free operation and maintenance: TiDB Serverless is a fully managed service that takes care of server setup, maintenance, and recovery from failures. This allows the AmzTrends team to focus on strategic tasks instead of managing servers.

Developer-friendly interface: TiDB Serverless is MySQL compatible with the familiar SQL interface,  facilitating a seamless transition for AmzTrends’ development team without additional training. 

Implementation Details 

The implementation involved replacing the self-managed TiDB cluster with TiDB Serverless and optimizing the existing data schema. The key steps involved are:

Figure 2. AmzTrends’s Cloud-Native Architecture Based on TiDB Serverless

• Data migration: Securely transferring over 2 billion data points to the TiDB Serverless environment.
• Architecture optimization: Simplifying the overall infrastructure by eliminating the need for separate Spark servers and consolidating data analysis within the TiDB Serverless framework.
• Index and query optimization: Analyzing and removing redundant indexes to significantly cut data storage costs by 50% and improve query performance.  
• Cloud-native transformation: Leveraging the fully managed capabilities of TiDB Serverless on AWS to adopt a cloud-native approach, enhancing scalability and resilience. 

Benefits of TiDB Serverless

The migration to TiDB Serverless has resulted in several immediate and long-term benefits:

• 80% overall cost saving: AmzTrends achieved 50% savings in data storage costs by optimizing indexing and query performance, transforming into an AWS-based cloud-native architecture, and a 30% reduction in service resource expenses.  
• Operational efficiency: The fully managed nature of TiDB Serverless drastically reduced the operational burden, enabling the team to focus on innovation rather than maintenance.
• Performance gains: Query performance improvements and index optimizations led to faster data access and analysis capabilities, directly enhancing the user experience.
• Scalability and flexibility: The ability to dynamically scale resources in response to demand ensured that AmzTrends could efficiently handle data growth without incurring unnecessary costs.

Conclusion

The migration to TiDB Serverless transformed AmzTrends’ data handling and analysis capabilities, simplifying its architecture with substantial cost saving. The transition enabled AmzTrends to focus more on innovation and development, ensuring higher system stability and availability. This case study demonstrates the power of TiDB Serverless in providing scalable, cost-efficient solutions for startups, even those without extensive professional operation and maintenance teams.