{"id":121,"date":"2018-08-03T00:00:00","date_gmt":"2018-08-03T00:00:00","guid":{"rendered":"https:\/\/en.pingcap.com\/blog\/adding-built-in-functions-to-tikv\/"},"modified":"2024-03-18T06:31:29","modified_gmt":"2024-03-18T13:31:29","slug":"adding-built-in-functions-to-tikv","status":"publish","type":"post","link":"https:\/\/www.pingcap.com\/ko\/blog\/adding-built-in-functions-to-tikv\/","title":{"rendered":"Landing Your First Rust Pull Request in TiKV"},"content":{"rendered":"

Note: This post is out of date.<\/em><\/p>\n

This guide is intended to show how you can land your first Pull Request (PR) in Rust to contribute to TiKV in less than 30 minutes. But before we do that, here’s some helpful background.<\/p>\n

TiDB<\/a> (“Ti” = Titanium) is an open-source distributed scalable Hybrid Transactional and Analytical Processing (HTAP<\/a>) database, built by the company PingCAP (that’s us!) and its active open-source community (that’s you!). It’s designed to provide infinite horizontal scalability, strong consistency, and high availability with MySQL compatibility. It serves as a one-stop data warehouse for both OLTP (Online Transactional Processing) and OLAP (Online Analytical Processing) workloads.<\/p>\n

What powers this experience is TiKV<\/a>, a distributed transactional key-value store (all built in Rust!), which is now deployed in more than 200 companies in production. One key reason why TiDB can process complex SQL queries so quickly is a Coprocessor API layer between TiDB and TiKV, which takes advantage of the distributed nature of a distributed database to “push down” partial queries in parallel, where partial results are generated and reassembled for the client. This is a key differentiator between TiDB and other distributed databases.<\/p>\n

So far, TiDB can only push down some simple expressions to TiKV to be processed, e.g. fetching the value in a column and doing comparison or arithmetic operations on simple data structures. To get more juice out of distributed computing resources, we need to include more expressions to push down. The first type is MySQL built-in functions. How do we accomplish that in short order? That’s where you<\/em>–our intrepid systems hacker, Rust lover, and distributed system geek–come in!<\/p>\n

So follow along this guide to contribute a built-in MySQL function to TiKV in Rust in 30 minutes. And when you do, you will receive some special gifts from us reserved just for our beloved contributors, as a small token of our gratitude. Let’s get started!<\/p>\n

How the Coprocessor Works<\/h2>\n

Before diving into our step-by-step guide on how to contribute, it’s worth understanding how TiDB’s Coprocessor works at a high-level. After TiDB receives a SQL statement, it parses the statement into an abstract syntax tree (AST), then generates an optimal execution plan using its Cost-Based Optimizer. (Learn more details on how TiDB generates a query plan HERE<\/a>.) The execution plan is split into multiple subtasks and the Coprocessor API pushes down these subtasks to different TiKV nodes to be processed in parallel.<\/p>\n

Here’s an illustration on how a statement like select count(*) from t where a + b > 5<\/code> gets pushed down:<\/p>\n

After TiKV receives these subtask expressions, the following steps are performed in a loop:<\/p>\n

    \n
  1. Obtain the complete data of the next row, parse and decode the data record based on the requested columns.<\/li>\n
  2. Use the predicate specified in the where<\/code> clause to filter data.<\/li>\n
  3. If the data passes the filter predicate, the aggregation result will be computed.<\/li>\n<\/ol>\n

    After different TiKV nodes compute and return results of their respective subtasks, they are returned to TiDB. TiDB then aggregates on all the results sent from TiKV and sends the final result to the client.<\/p>\n

    How to add a MySQL built-in function to TiKV<\/h2>\n

    Now that you have an overview of how Coprocessor in TiDB\/TiKV works, here’s how to contribute MySQL built-in functions to further strengthen TiKV’s coprocessing power!<\/p>\n

    Step 1: Select a function for pushdown<\/h3>\n

    Go to the push down scalar functions<\/code> issue page<\/a>, choose a function you like from the unimplemented function signature list, then tell us so we can create an issue and assign it to you to prevent duplication of work.<\/p>\n

    Step 2: Find the logic of corresponding implementation in TiDB<\/h3>\n

    Search the related builtinXXXSig<\/code> (XXX is the function signature you want to implement) in the expression<\/code> directory of TiDB.<\/p>\n

    Take MultiplyIntUnsigned<\/code><\/a> as an example, which we will use throughout this guide, you can find the corresponding function signature (builtinArithmeticMultiplyIntUnsignedSig<\/code>) and its implementation.<\/p>\n

    Step 3: Define the function<\/h3>\n
      \n
    1. The name of the file where the built-in function exists in TiKV should correspond to the same name in TiDB.For example, since all the pushdown files in the expression<\/code> directory in TiDB are named builtin_XXX<\/code>, in TiKV the corresponding file name should be builtin_XXX.rs<\/code>. In this example, the current function is in the builtin_arithmetic.go file in TiDB, so the function should be placed in builtin_arithmetic.rs in TiKV.Note:<\/strong> If the corresponding file in TiKV does not exist, you need to create a new file in the corresponding directory with the same name as in TiDB.<\/li>\n
    2. The function name should follow the Rust snake_case naming conventions.For this example, MultiplyIntUnsigned<\/code> will be defined as multiply_int_unsigned<\/code>.<\/li>\n
    3. For the return value, you can refer to the Eval<\/code> functions which are implemented in TiDB and their corresponding return value types, as shown in the following table:
      \n\n\n\n\n\n\n\n\n\n\n
      Eval<\/code> Function in TiDB<\/strong><\/td>\nReturn Value Type in TiKV<\/strong><\/td>\n<\/tr>\n
      evalInt<\/td>\nResult<Option<i64>><\/code><\/td>\n<\/tr>\n
      evalReal<\/td>\nResult<Option<i64>><\/code><\/td>\n<\/tr>\n
      evalString<\/td>\nResult<Option<Cow<'a, [u8]>>><\/code><\/td>\n<\/tr>\n
      evalDecimal<\/td>\nResult<Option<Cow<'a, Decimal>>><\/code><\/td>\n<\/tr>\n
      evalTime<\/td>\nResult<Option<Cow<'a, Time>>><\/code><\/td>\n<\/tr>\n
      evalDuration<\/td>\n Result<Option<Cow<'a, Duration>>><\/code><\/td>\n<\/tr>\n
      evalJSON<\/td>\nResult<Option<Cow<'a, Json>>><\/code><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

      Thus, in TiDB’s builtinArithmeticMultiplyIntUnsignedSig<\/code>, it implements the evalInt<\/code> method, so the return value type of this function multiply_int_unsigned<\/code> should be Result<Option<i64>><\/code>.<\/li>\n

    4. All the arguments of the builtin-in<\/code> function should be consistent with that of the eval<\/code> function of the expression:\n