Transforming Financial Operations: Beijing Bank’s Journey with TiDB

Author: Zhenhua Yu, Senior Architect of Software Development at Beijing Bank

In recent years, the rapid advancement of technology has significantly transformed various industries, with the financial sector being no exception. This evolution, often called the Fourth Industrial Revolution, has introduced new challenges and demands for banking systems. 

The banking industry faces increasing pressure from the rapid growth of emerging businesses, the flourishing of inclusive finance, and the explosive expansion of internet businesses. These trends demand systems that can handle high concurrency, scale quickly, and deliver new products at unprecedented speeds. Additionally, the need for financial security and autonomy has become paramount, driving the shift towards more agile and self-controlled information systems.

Beijing Bank, recognizing the need to stay ahead, has actively embraced these changes by integrating TiDB into critical real-time transaction systems, including UnionPay cardless payments and financial interconnection service platforms.

Since its production deployment in 2018, TiDB has proven stable and reliable, suggesting that distributed database technology has matured enough to handle the complexities of modern finance.

In this blog, we’ll explore how Beijing Bank’s use of TiDB addresses these challenges, offering insights into the future of financial technology.

The Background 

Beijing Bank thoroughly analyzed traditional technological architectures from a global perspective. The bank identified several architectural patterns that have long supported financial operations. These included architectures such as monolithic applications, traditional databases like DB2 and Oracle, single-server or clustered deployment models, waterfall development methodologies, and conventional operations and maintenance practices.

While these traditional architectures have served their purpose well, they now face challenges in meeting the demands of modern financial operations, particularly in areas like high concurrency, scalability, cost management, and product delivery speed. Recognizing these limitations, Beijing Bank started looking at adopting distributed databases.

Beijing Bank’s collaboration with PingCAP has been instrumental in this transformation. Over the past year, the bank’s team engaged in detailed discussions with PingCAP, covering technical details, solutions, workflows, and more. Three key principles have guided this collaborative effort:

1. Pragmatism: Beijing Bank’s architectural transformation is not just about adopting new technologies or products but about genuinely enhancing efficiency in business development, application development, and operations.
2. Quick Wins: Achieving quick wins has been crucial in the transformation process. Technological upgrades often encounter resistance in any enterprise, particularly in traditional sectors. By demonstrating immediate value through quick wins, Beijing Bank has been able to smoothen the transition and gain support from within the organization.
3. Holistic Approach: The bank’s transformation strategy is comprehensive. It focuses on constructing a platform that delivers overall value rather than solely on isolated improvements.

Progress Update

Before diving into specific details, let me update everyone on our current progress. In March 2018, we launched the industry’s first distributed database tailored for core financial operations, utilizing a two-site, three-center, five-replica architecture. Building upon this distributed database foundation, in May, we deployed the Internet UnionPay settlement platform, a critical real-time transaction system involving fund transactions. In June, we launched the UnionPay cardless payment platform. The diagram (Figure 3) may be slightly outdated; currently, our deployments also include the Financial Interconnection Service Platform and the IFRS9 impairment system. Looking ahead, our future initiatives align closely with what Liu Qi just discussed, including HTAP and container cloud solutions, which represent our most urgent needs.

Specialized Evaluation

Reflecting on its journey, Beijing Bank’s early adoption of distributed database technologies was a pioneering move within the industry. This initiative presented numerous challenges and uncertainties, primarily due to the industry’s predominant focus on traditional databases like DB2 and Oracle. Mastering distributed databases required a significant learning curve.

Beijing Bank developed a tailored evaluation system to ensure product availability and suitability for financial operations. The evaluation process focused on several critical aspects:

1. Functionality Testing: Key considerations included assessing whether the database ensured high availability, could scale linearly, and supported online upgrades.
2. Performance Testing: Traditional tools like TPCC and Sysbench were found inadequate for financial scenarios. Therefore, Beijing Bank developed a financial performance evaluation system specifically designed to measure the applicability and effectiveness of distributed databases. This custom system provided a transparent, measurable way to evaluate functionality and performance.

Deployment 

Beijing Bank’s data storage architecture features a two-site, three-center, five-replica deployment model.

Building a cross-city, long-distance distributed database presents considerable challenges. For example, the distance between Beijing and Xi’an exceeds a thousand kilometers, leading to relatively high latency. Measured latency is approximately seventeen milliseconds. With a single SQL query, this results in a round-trip time exceeding thirty milliseconds, which is unacceptable.

Beijing Bank implemented a five-replica model to address this issue: two Internet Data Centers (IDCs) in Beijing, each with two replicas, and one IDC in Xi’an with a single replica. This 2:2:1 configuration minimizes reliance on the network between Beijing and Xi’an. When a frontend application sends a request, it can be served by any of the three replicas in Beijing without depending on the distant IDC in Xi’an. As a result, the average SQL latency is reduced to around 1.2 milliseconds, with a 95th percentile latency of approximately 5 milliseconds. This performance is incredible, especially given the complexity of UnionPay and Internet Union business operations compared to typical internet services.

A real-world incident illustrates how Beijing Bank’s multi-replica strategy significantly enhanced service stability. One Saturday at noon, the maintenance staff reported a failure in one of the TiKV storage servers. The immediate concern was whether the service was affected, but the team assured it was not. This confirmation allowed for a prompt repair without any disruption to service.

The following Sunday, another server failure was reported, raising concerns about a potential issue with the hardware batch. Despite this second failure, the service remained unaffected. Beijing Bank’s ability to handle two consecutive server failures without impacting service underscored the effectiveness of its multi-replica strategy, demonstrating its robustness and reliability in maintaining service stability even under challenging conditions.

Issues Encountered

During their transition from traditional databases to distributed databases, Beijing Bank encountered several issues, illustrating that there is no one-size-fits-all technical solution. Here are some of the challenges faced:

1. Isolation Levels: Beijing Bank previously used the READ COMMITTED isolation level in DB2 and Oracle. Adapting to TiDB’s Repeatable Read isolation level required adjustments. Initially, the team faced issues where data inserted on one side was not immediately visible on the other due to TiDB’s snapshot isolation level.
2. Execution Plans: Problems arose when execution plans did not select the correct index. Even when an index was available, it was not always chosen accurately, leading to slow SQL execution and high memory usage. Temporary solutions, such as manually adding hints, helped address this issue. However, it is anticipated that future versions of TiDB will improve execution plan accuracy.
3. Hot Data Handling: Relying solely on the database to solve hot data problems proved unrealistic. Whether traditional or distributed databases are used, additional application caching components are necessary. Previously, various technical solutions like hashing were employed to distribute hot data. With caching, the team can now address these issues more effectively.
4. Microservices Demands: The application architecture’s shift to a microservices approach increased the database’s demands compared to traditional monolithic applications. In monolithic setups, transactions involve multiple SQL statements. When divided into microservices, both application logic and database processing become more granular, leading to a significant increase in transaction commits. This shift put pressure on MVCC’s optimistic concurrency control model, resulting in poorer performance during tests with finer granularity.

These issues highlight the complexities of adapting to new technologies and underscore the need for ongoing adjustments and improvements.

Differences in Application Practices Between Finance and Internet Industries

Implementing distributed databases in the financial industry presents unique challenges compared to other internet-based industries due to differences in development periods, organizational structures, and historical IT assets.

Banks typically operate with well-established hardware systems, deployment models, software design patterns, and operational models. Introducing new technologies into such a mature environment is more complex. The high costs associated with migrating large legacy systems often require parallel operation of both new and old systems. This parallel operation involves managing new and existing business logic and data, a challenge for which ready-made solutions are not always available. Currently, banks are developing graceful parallel operation solutions to address these issues.

Organizational structures also differ significantly. In traditional monolithic applications, roles and responsibilities are clear, with well-defined technical leads, business owners, and departmental duties. However, when transitioning to microservices, accountability can become ambiguous as applications are broken down into smaller components. Adapting organizational structures to fit new system architectures can be challenging, especially in organizations with established processes.

Moreover, banks have more extensive historical IT assets and complex business operations than internet companies. This historical context and complexity add additional difficulty when integrating new technologies, whereas internet companies often face different challenges due to their more recent technological advancements and simpler legacy systems.

The Future 

In conclusion, Beijing Bank’s journey with TiDB highlights the challenges and advancements of adopting distributed databases in the financial sector. The bank’s approach underscores the importance of comprehensive validation to ensure the new architecture delivers high availability, scalability, and cost-effectiveness. As Beijing Bank expands its application scope, the gradual migration of fast-growing and high-concurrency systems will be a critical next step.

A key focus is the development of tailored application standards and guidelines for financial-grade development with TiDB, including middleware for seamless integration between traditional databases and TiDB. This initiative is expected to enhance application expansion and operational efficiency significantly.

Beijing Bank’s pursuit of Hybrid Transactional/Analytical Processing (HTAP) with TiFlash represents a forward-looking approach that offers improvements over traditional data analysis solutions. The bank aims to leverage advanced technologies for better business processes by continuously improving performance and reducing network dependencies.

As Beijing Bank continues to refine its use of TiDB and share its valuable insights, it offers a compelling example of the benefits distributed database technology can bring to complex financial operations.

For organizations looking to enhance their database infrastructure and explore the transformative potential of distributed databases, TiDB provides a powerful and flexible solution. Discover how TiDB can support your business’s growth and innovation by scheduling a demo. Embrace the future of data management with TiDB today!