{"id":617,"date":"2019-08-26T00:00:00","date_gmt":"2019-08-26T07:00:00","guid":{"rendered":"https:\/\/en.pingcap.com\/blog\/how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment\/"},"modified":"2024-08-21T06:20:29","modified_gmt":"2024-08-21T13:20:29","slug":"how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment","status":"publish","type":"post","link":"https:\/\/www.pingcap.com\/ko\/blog\/how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment\/","title":{"rendered":"How TSQ Becomes a Performance Bottleneck for TiKV in AWS ARM Environment"},"content":{"rendered":"<h2><span class=\"ez-toc-section\" id=\"Background\"><\/span>Background<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Although Intel x86-64 is still the most widely used architecture in the servers, <a href=\"https:\/\/en.wikipedia.org\/wiki\/ARM_architecture\">ARM<\/a> is attracting more and more attention, and many companies have already been migrating their products from x86-64 to ARM. At PingCAP, we would love to see <a href=\"https:\/\/en.wikipedia.org\/wiki\/TiDB\">TiDB<\/a> run on ARM and decided to test its performance on the AWS ARM platform.<\/p>\n<p><a href=\"http:\/\/www.tikv.org\">TiKV<\/a> is the distributed transactional Key-Value storage engine for TiDB. When we tested TiKV on the AWS ARM server, we optimized the network queue according to the <a href=\"https:\/\/aws.amazon.com\/cn\/blogs\/compute\/optimizing-network-intensive-workloads-on-amazon-ec2-a1-instances\/\">AWS official document<\/a> and found that there were still two cores with a high usage rate of soft interrupts, which became a performance bottleneck. Further investigations revealed that they were two tasklet soft interrupts.<\/p>\n<p>As we know, tasklets of the same type are always serialized and cannot run concurrently on multiple CPUs. This was consistent with the issue we encountered, so our initial guess was that the tasklet function was possibly implemented by the <a href=\"https:\/\/github.com\/amzn\/amzn-drivers\">ENA NIC driver<\/a>. However, the network subsystem is very time sensitive, and two soft interrupts (NET_TX and NET_RX of softirq) are actually defined in the kernel. This was against the guess of using the tasklet in the NIC driver to handle network requests. What were the two tasklets doing? We decided to set up the environment again to find it out. This article aims to record our exploration into Linux kernel with the following sections:<\/p>\n<ul>\n<li><a href=\"#system-configuration\">System configuration<\/a><\/li>\n<li><a href=\"#reproduction\">Reproduction<\/a><\/li>\n<li><a href=\"#analysis\">Analysis<\/a><\/li>\n<li><a href=\"#conclusion\">Conclusion<\/a><\/li>\n<\/ul>\n<h2><span class=\"ez-toc-section\" id=\"System_configuration\"><\/span>System configuration<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<h3>Operating System<\/h3>\n<p>Red Hat Enterprise Linux Server release 7.6 (Maipo)<\/p>\n<h3>Kernel<\/h3>\n<p>4.14.0-115.2.2.el7a.aarch64<\/p>\n<h3>NIC info<\/h3>\n<p>Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)<\/p>\n<p>Two RX\/TX Channels, with 1024 ring size<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Reproduction\"><\/span>Reproduction<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>To reproduce the environment consistently, we did the following:<\/p>\n<ol>\n<li>Set interrupt affinity and RPS.\n<pre><code>grep eth0 \/proc\/interrupts | awk '{print $1}' | tr -d :\n44\n45\n\nsudo sh -c \"echo 0 &gt; \/proc\/irq\/44\/smp_affinity_list\"\nsudo sh -c \"echo 8 &gt; \/proc\/irq\/45\/smp_affinity_list\"\n\nsudo sh -c 'echo fe &gt; \/sys\/class\/net\/eth0\/queues\/rx-0\/rps_cpus'\nsudo sh -c 'echo fe00 &gt; \/sys\/class\/net\/eth0\/queues\/rx-1\/rps_cpus'\n<\/code><\/pre>\n<\/li>\n<li>Generate the workload using wrk + nginx.<\/li>\n<li>Confirm successful reproduction by monitoring the CPU and interrupts.\n<pre><code>watch -d cat \/proc\/softirqs\nmpstat -P ALL 2\n<\/code><\/pre>\n<\/li>\n<\/ol>\n<div class=\"caption-center\">Reproducing the environment<\/div>\n<h2><span class=\"ez-toc-section\" id=\"Analysis\"><\/span>Analysis<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>It is known that tasklet needs to evoke scheduling through tasklet_schedule. To get the runtime call chain of <code>tasklet_schedule<\/code>, we intended to write a trace tool which, combined with the kernel code, should be able to help me find the definition of the tasklet function. Because the Linux we used was a 4.14 kernel, BCC (BPF Compiler Collection) seemed to be a fitting choice for the tool.<\/p>\n<p>However, the BCC we installed could not run. Checked the kernel configuration file, we found that <code>CONFIG_BPF_SYSCALL<\/code> was not enabled. It was not clear whether the BPF (Berkeley Packet Filter) of this version of the kernel supported the Aarch64 platform. Therefore, for the time being we decided to write the kprobe kernel module directly to grab the <code>tasklet_schedule<\/code> function, so that we could see in which process tasklet was woken. The following is the trace log.<\/p>\n<pre><code>Jul 17 07:30:50 ip-172-31-4-116 kernel: CPU: 0 PID: 7 Comm: ksoftirqd\/0 Kdump: loaded Tainted: G           OE  ------------   4.14.0-115.2.2.el7a.aarch64 #1\nJul 17 07:30:50 ip-172-31-4-116 kernel: Hardware name: Amazon EC2 a1.4xlarge\/, BIOS 1.0 11\/1\/2018\nJul 17 07:30:50 ip-172-31-4-116 kernel: Call trace:\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff000008089d64&gt;] dump_backtrace+0x0\/0x23c\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff000008089fc4&gt;] show_stack+0x24\/0x2c\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff0000088148fc&gt;] dump_stack+0x84\/0xa8\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff000000cc00cc&gt;] on___tasklet_schedule_ent+0x44\/0x88 [debug_events]\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff0000087522c0&gt;] tcp_wfree+0x15c\/0x168\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff0000086cd3f0&gt;] skb_release_head_state+0x6c\/0xf0\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff0000086cd494&gt;] skb_release_all+0x20\/0x3c\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff0000086cdd50&gt;] consume_skb+0x48\/0xcc\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff0000008e4f2c&gt;] ena_io_poll+0x54c\/0x8f0 [ena]\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff0000086e6df0&gt;] net_rx_action+0x314\/0x470\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff000008081a7c&gt;] __do_softirq+0x11c\/0x2f0\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff0000080db074&gt;] run_ksoftirqd+0x48\/0x5c\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff0000080fbcf8&gt;] smpboot_thread_fn+0x1b4\/0x1c4\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff0000080f72e8&gt;] kthread+0x10c\/0x138\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff000008084ea4&gt;] ret_from_fork+0x10\/0x18\nJul 17 07:30:50 ip-172-31-4-116 kernel: comm: ksoftirqd\/0, pid: 7, state: 0\nJul 17 07:30:50 ip-172-31-4-116 kernel: ctx: bh\n<\/code><\/pre>\n<p>As can be seen from the trace information, the <code>ksoftirqd<\/code> thread running in the lower half of the interrupt is processing the <code>NET_RX<\/code> soft interrupt, attempting to initiate the tasklet scheduling. Let&#8217;s take a look at the NIC driver code (version 2.0.1k not found, so here we used version 2.0.2 instead).<\/p>\n<pre><code>static int ena_io_poll(struct napi_struct *napi, int budget)\n{\n        ... ...\n        tx_work_done = ena_clean_tx_irq(tx_ring, tx_budget);\n        rx_work_done = ena_clean_rx_irq(rx_ring, napi, budget);\n        ... ...\n}\n\nstatic int ena_clean_tx_irq(struct ena_ring *tx_ring, u32 budget)\n{\n  ... ...\n  dev_kfree_skb(skb);\n  ... ...\n}\n<\/code><\/pre>\n<p>It can be noted that the NIC driver responds to the transmit completion interrupt and the data reception interrupt by multiplexing <code>NET_RX<\/code>, while the context captured is in the lower half of the transmit completion interrupt. The NIC driver calls the <code>dev_kfree_skb<\/code> macro (<code>#define dev_kfree_skb(a) consume_skb(a)<\/code>) to release <code>sk_buffer<\/code> when performing post-transfer cleanup. Then it checks the TSQ_THROTTLED flag to determine where there is data waiting for the qdisc space when executing the <code>tcp_wfree<\/code> function. If there is any packet waiting, the sock object corresponding to the packet will be added to the TCP small queues (TSQ)<a href=\"#reference\"><sup>1<\/sup><\/a> of the current CPU, as shown below:<\/p>\n<pre><code>struct tsq_tasklet {\n  struct tasklet_struct tasklet;\n  struct list_head head; \/* queue of tcp sockets *\/\n};\nstatic DEFINE_PER_CPU(struct tsq_tasklet, tsq_tasklet);\n\nstruct tcp_sock *tp = tcp_sk(sk);\n... ...\nlist_add(&amp;tp-&gt;tsq_node, &amp;tsq-&gt;head);\n<\/code><\/pre>\n<p>Then the TSQ tasklet is scheduled, and the data packets in the sock are sent in the context of the next soft interrupt. The corresponding kernel code is as follows:<\/p>\n<pre><code>\/*\n * Write buffer destructor automatically called from kfree_skb.\n * We can't xmit new skbs from this context, as we might already\n * hold qdisc lock.\n *\/\nvoid tcp_wfree(struct sk_buff *skb)\n{\n  struct sock *sk = skb-&gt;sk;\n   struct tcp_sock *tp = tcp_sk(sk);\n\n  if (test_and_clear_bit(TSQ_THROTTLED, &amp;tp-&gt;tsq_flags) &amp;&amp;\n      !test_and_set_bit(TSQ_QUEUED, &amp;tp-&gt;tsq_flags)) {\n        unsigned long flags;\n        struct tsq_tasklet *tsq;\n\n      \/* Keep a ref on socket.\n       * This last ref will be released in tcp_tasklet_func()\n         *\/\n      atomic_sub(skb-&gt;truesize - 1, &amp;sk-&gt;sk_wmem_alloc);\n\n      \/* queue this socket to tasklet queue *\/\n      local_irq_save(flags);\n      tsq = &amp;__get_cpu_var(tsq_tasklet);\n      list_add(&amp;tp-&gt;tsq_node, &amp;tsq-&gt;head);\n      tasklet_schedule(&amp;tsq-&gt;tasklet);\n      local_irq_restore(flags);\n  } else {\n      sock_wfree(skb);\n }\n}\n<\/code><\/pre>\n<div class=\"trackable-btns\"><a href=\"\/download\"><button>Download TiDB<\/button><\/a><br \/>\n<a href=\"https:\/\/share.hsforms.com\/1e2W03wLJQQKPd1d9rCbj_Q2npzm\"><button>Subscribe to Blog<\/button><\/a><\/div>\n<p>Now we can view the <code>tsq_tasklet_func<\/code> definition:<\/p>\n<pre><code>\/*\n * One tasklet per cpu tries to send more skbs.\n * We run in tasklet context but need to disable irqs when\n * transferring tsq-&gt;head because tcp_wfree() might\n * interrupt us (non NAPI drivers)\n *\/\nstatic void tcp_tasklet_func(unsigned long data)\n{\n  struct tsq_tasklet *tsq = (struct tsq_tasklet *)data;\n  LIST_HEAD(list);\n  unsigned long flags;\n  struct list_head *q, *n;\n  struct tcp_sock *tp;\n  struct sock *sk;\n\n  local_irq_save(flags);\n  list_splice_init(&amp;tsq-&gt;head, &amp;list);\n  local_irq_restore(flags);\n\n  list_for_each_safe(q, n, &amp;list) {\n      tp = list_entry(q, struct tcp_sock, tsq_node);\n      list_del(&amp;tp-&gt;tsq_node);\n\n      sk = (struct sock *)tp;\n      bh_lock_sock(sk);\n\n      if (!sock_owned_by_user(sk)) {\n          tcp_tsq_handler(sk);\n      } else {\n          \/* defer the work to tcp_release_cb() *\/\n          set_bit(TCP_TSQ_DEFERRED, &amp;tp-&gt;tsq_flags);\n      }\n      bh_unlock_sock(sk);\n\n      clear_bit(TSQ_QUEUED, &amp;tp-&gt;tsq_flags);\n      sk_free(sk);\n  }\n}\n<\/code><\/pre>\n<p>The core logic is:<\/p>\n<pre><code>if (!sock_owned_by_user(sk)) {\n    tcp_tsq_handler(sk);\n} else {\n    \/* defer the work to tcp_release_cb() *\/\n    set_bit(TCP_TSQ_DEFERRED, &amp;tp-&gt;tsq_flags);\n}\n<\/code><\/pre>\n<p>Sock processing is delayed if the sock is held by another user process; when the sock is released, <code>tcp_tsq_handler<\/code> is called to send data.<\/p>\n<pre><code>void tcp_release_cb(struct sock *sk)\n{\n    struct tcp_sock *tp = tcp_sk(sk);\n    unsigned long flags, nflags;\n\n    ... ...\n    if (flags &amp; (1UL &lt;&lt; TCP_TSQ_DEFERRED))\n        tcp_tsq_handler(sk);\n    ... ...\n}\n<\/code><\/pre>\n<p>Otherwise, <code>tcp_tsq_handler<\/code> is called directly to send the data.<\/p>\n<pre><code>static void tcp_tsq_handler(struct sock *sk)\n{\n     if ((1 &lt;&lt; sk-&gt;sk_state) &amp;\n         (TCPF_ESTABLISHED | TCPF_FIN_WAIT1 | TCPF_CLOSING |\n          TCPF_CLOSE_WAIT  | TCPF_LAST_ACK))\n          tcp_write_xmit(sk, tcp_current_mss(sk), tcp_sk(sk)-&gt;nonagle,\n                  0, GFP_ATOMIC);\n}\n<\/code><\/pre>\n<p>We modified the trace module to trace <code>tcp_tasklet_func<\/code> to make sure it was actually awakened and running on CPU 0:<\/p>\n<pre><code>Jul 17 07:30:50 ip-172-31-4-116 kernel: CPU: 0 PID: 7 Comm: ksoftirqd\/0 Kdump: loaded Tainted: G           OE  ------------   4.14.0-115.2.2.el7a.aarch64 #1\nJul 17 07:30:50 ip-172-31-4-116 kernel: Hardware name: Amazon EC2 a1.4xlarge\/, BIOS 1.0 11\/1\/2018\nJul 17 07:30:50 ip-172-31-4-116 kernel: Call trace:\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff000008089d64&gt;] dump_backtrace+0x0\/0x23c\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff000008089fc4&gt;] show_stack+0x24\/0x2c\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff0000088148fc&gt;] dump_stack+0x84\/0xa8\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff000000cc0044&gt;] on_tcp_tasklet_func_ent+0x44\/0x88 [debug_events]\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff0000080db8d4&gt;] tasklet_action+0x88\/0x10c\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff000008081a7c&gt;] __do_softirq+0x11c\/0x2f0\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff0000080db074&gt;] run_ksoftirqd+0x48\/0x5c\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff0000080fbcf8&gt;] smpboot_thread_fn+0x1b4\/0x1c4\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff0000080f72e8&gt;] kthread+0x10c\/0x138\nJul 17 07:30:50 ip-172-31-4-116 kernel: [&lt;ffff000008084ea4&gt;] ret_from_fork+0x10\/0x18\nJul 17 07:30:50 ip-172-31-4-116 kernel: comm: ksoftirqd\/0, pid: 7, state: 0\nJul 17 07:30:50 ip-172-31-4-116 kernel: ctx: bh\n<\/code><\/pre>\n<p>To wrap up, here is a summary of the execution flow above:<\/p>\n<pre><code>Transmit completion IRQ handler  (IRQ ctx on CPU 0)\n    \u2192 raise napi_shedule (SOFTIRQ ctx on CPU 0)\n        \u2192 NET_RX (SOFTIRQ ctx on CPU 0)\n            \u2192 ena_io_poll (driver in SOFTIRQ ctx on CPU 0)\n                 \u2192 ena_clean_tx_irq (driver in SOFTIRQ ctx on CPU 0)\n                     \u2192 dev_kfree_skb (in SOFTIRQ ctx on CPU 0)\n                          \u2192 tcp_wfree (in SOFTIRQ ctx on CPU 0 DO TSQ flag CHECK)\n                              \u2192 get TSQ tasklet from CPU 0 (per-cpu val)\n                               \u2192 raise tsq tasklet (on CPU 0)\n<\/code><\/pre>\n<p>Next time CPU 0&#8217;s SOTRIRQ scheduled:<\/p>\n<pre><code>    \u2192 run tsq_tasklet_func (on CPU 0)\n<\/code><\/pre>\n<p>ENA&#8217;s NIC hardware queue only contains 2 hard interrupts, which means only two TSQ tasklets are allowed to work according to the above process. Therefore, when the network load is high, the corresponding CPU cores become performance bottlenecks.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Conclusion\"><\/span>Conclusion<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Based on the observations and analysis in the previous section, we can conclude that for a network card running on a Linux kernel that supports TSQs, if the number of hardware queues is much smaller than the number of CPU cores, the corresponding CPU is likely to become a bottleneck when network card interrupts occur.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Reference\"><\/span>Reference<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<ul>\n<li>TCP small queues is a mechanism designed to fight bufferbloat. TCP Small Queues goal is to reduce number of TCP packets in <code>xmit<\/code> queues (qdisc &amp; device queues), to reduce RTT and cwnd bias, part of the bufferbloat problem. See <a href=\"https:\/\/lwn.net\/Articles\/506237\/\">TCP Small Queues<\/a>.<\/li>\n<li><a href=\"https:\/\/lwn.net\/Articles\/576263\/\">tcp: auto corking<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Explore into Linux kernel to find out how TSQ becomes a performance bottleneck for TiKV in an AWS ARM environment.<\/p>","protected":false},"author":226,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"ub_ctt_via":"","footnotes":""},"categories":[6],"tags":[70,22],"class_list":["post-617","post","type-post","status-publish","format-standard","hentry","category-engineering","tag-linux","tag-tikv"],"acf":[],"featured_image_src":null,"author_info":{"display_name":"Wenbo Zhang","author_link":"https:\/\/www.pingcap.com\/ko\/blog\/author\/zhangwenbo\/"},"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.9 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>How TSQ Becomes a Bottleneck for TiKV in AWS ARM Environment<\/title>\n<meta name=\"description\" content=\"Explore into Linux kernel to find out how TSQ becomes a performance bottleneck for TiKV in an AWS ARM environment.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.pingcap.com\/ko\/blog\/how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment\/\" \/>\n<meta property=\"og:locale\" content=\"ko_KR\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How TSQ Becomes a Bottleneck for TiKV in AWS ARM Environment\" \/>\n<meta property=\"og:description\" content=\"Explore into Linux kernel to find out how TSQ becomes a performance bottleneck for TiKV in an AWS ARM environment.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.pingcap.com\/ko\/blog\/how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment\/\" \/>\n<meta property=\"og:site_name\" content=\"TiDB\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/facebook.com\/pingcap2015\" \/>\n<meta property=\"article:published_time\" content=\"2019-08-26T07:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-08-21T13:20:29+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/static.pingcap.com\/files\/2024\/09\/11005522\/Homepage-Ad.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1440\" \/>\n\t<meta property=\"og:image:height\" content=\"714\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Wenbo Zhang\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@PingCAP\" \/>\n<meta name=\"twitter:site\" content=\"@PingCAP\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Wenbo Zhang\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"10\ubd84\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment\/\"},\"author\":{\"name\":\"Wenbo Zhang\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/person\/3a46a13870b11ccb3f59734ca38ed634\"},\"headline\":\"How TSQ Becomes a Performance Bottleneck for TiKV in AWS ARM Environment\",\"datePublished\":\"2019-08-26T07:00:00+00:00\",\"dateModified\":\"2024-08-21T13:20:29+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment\/\"},\"wordCount\":860,\"publisher\":{\"@id\":\"https:\/\/www.pingcap.com\/#organization\"},\"keywords\":[\"Linux\",\"TiKV\"],\"articleSection\":[\"Engineering\"],\"inLanguage\":\"ko-KR\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment\/\",\"url\":\"https:\/\/www.pingcap.com\/blog\/how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment\/\",\"name\":\"How TSQ Becomes a Bottleneck for TiKV in AWS ARM Environment\",\"isPartOf\":{\"@id\":\"https:\/\/www.pingcap.com\/#website\"},\"datePublished\":\"2019-08-26T07:00:00+00:00\",\"dateModified\":\"2024-08-21T13:20:29+00:00\",\"description\":\"Explore into Linux kernel to find out how TSQ becomes a performance bottleneck for TiKV in an AWS ARM environment.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment\/#breadcrumb\"},\"inLanguage\":\"ko-KR\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.pingcap.com\/blog\/how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.pingcap.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How TSQ Becomes a Performance Bottleneck for TiKV in AWS ARM Environment\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.pingcap.com\/#website\",\"url\":\"https:\/\/www.pingcap.com\/\",\"name\":\"TiDB\",\"description\":\"TiDB | SQL at Scale\",\"publisher\":{\"@id\":\"https:\/\/www.pingcap.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.pingcap.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"ko-KR\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.pingcap.com\/#organization\",\"name\":\"PingCAP\",\"url\":\"https:\/\/www.pingcap.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png\",\"contentUrl\":\"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png\",\"width\":811,\"height\":232,\"caption\":\"PingCAP\"},\"image\":{\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/facebook.com\/pingcap2015\",\"https:\/\/x.com\/PingCAP\",\"https:\/\/linkedin.com\/company\/pingcap\",\"https:\/\/youtube.com\/channel\/UCuq4puT32DzHKT5rU1IZpIA\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/person\/3a46a13870b11ccb3f59734ca38ed634\",\"name\":\"Wenbo Zhang\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg\",\"contentUrl\":\"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg\",\"caption\":\"Wenbo Zhang\"},\"description\":\"Linux Kernel Engineer\",\"url\":\"https:\/\/www.pingcap.com\/ko\/blog\/author\/zhangwenbo\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How TSQ Becomes a Bottleneck for TiKV in AWS ARM Environment","description":"Explore into Linux kernel to find out how TSQ becomes a performance bottleneck for TiKV in an AWS ARM environment.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.pingcap.com\/ko\/blog\/how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment\/","og_locale":"ko_KR","og_type":"article","og_title":"How TSQ Becomes a Bottleneck for TiKV in AWS ARM Environment","og_description":"Explore into Linux kernel to find out how TSQ becomes a performance bottleneck for TiKV in an AWS ARM environment.","og_url":"https:\/\/www.pingcap.com\/ko\/blog\/how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment\/","og_site_name":"TiDB","article_publisher":"https:\/\/facebook.com\/pingcap2015","article_published_time":"2019-08-26T07:00:00+00:00","article_modified_time":"2024-08-21T13:20:29+00:00","og_image":[{"width":1440,"height":714,"url":"https:\/\/static.pingcap.com\/files\/2024\/09\/11005522\/Homepage-Ad.png","type":"image\/png"}],"author":"Wenbo Zhang","twitter_card":"summary_large_image","twitter_creator":"@PingCAP","twitter_site":"@PingCAP","twitter_misc":{"Written by":"Wenbo Zhang","Est. reading time":"10\ubd84"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.pingcap.com\/blog\/how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment\/#article","isPartOf":{"@id":"https:\/\/www.pingcap.com\/blog\/how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment\/"},"author":{"name":"Wenbo Zhang","@id":"https:\/\/www.pingcap.com\/#\/schema\/person\/3a46a13870b11ccb3f59734ca38ed634"},"headline":"How TSQ Becomes a Performance Bottleneck for TiKV in AWS ARM Environment","datePublished":"2019-08-26T07:00:00+00:00","dateModified":"2024-08-21T13:20:29+00:00","mainEntityOfPage":{"@id":"https:\/\/www.pingcap.com\/blog\/how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment\/"},"wordCount":860,"publisher":{"@id":"https:\/\/www.pingcap.com\/#organization"},"keywords":["Linux","TiKV"],"articleSection":["Engineering"],"inLanguage":"ko-KR"},{"@type":"WebPage","@id":"https:\/\/www.pingcap.com\/blog\/how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment\/","url":"https:\/\/www.pingcap.com\/blog\/how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment\/","name":"How TSQ Becomes a Bottleneck for TiKV in AWS ARM Environment","isPartOf":{"@id":"https:\/\/www.pingcap.com\/#website"},"datePublished":"2019-08-26T07:00:00+00:00","dateModified":"2024-08-21T13:20:29+00:00","description":"Explore into Linux kernel to find out how TSQ becomes a performance bottleneck for TiKV in an AWS ARM environment.","breadcrumb":{"@id":"https:\/\/www.pingcap.com\/blog\/how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment\/#breadcrumb"},"inLanguage":"ko-KR","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.pingcap.com\/blog\/how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.pingcap.com\/blog\/how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.pingcap.com\/"},{"@type":"ListItem","position":2,"name":"How TSQ Becomes a Performance Bottleneck for TiKV in AWS ARM Environment"}]},{"@type":"WebSite","@id":"https:\/\/www.pingcap.com\/#website","url":"https:\/\/www.pingcap.com\/","name":"\ud2f0DB","description":"TiDB | SQL at Scale","publisher":{"@id":"https:\/\/www.pingcap.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.pingcap.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"ko-KR"},{"@type":"Organization","@id":"https:\/\/www.pingcap.com\/#organization","name":"PingCAP","url":"https:\/\/www.pingcap.com\/","logo":{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/","url":"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png","contentUrl":"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png","width":811,"height":232,"caption":"PingCAP"},"image":{"@id":"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/facebook.com\/pingcap2015","https:\/\/x.com\/PingCAP","https:\/\/linkedin.com\/company\/pingcap","https:\/\/youtube.com\/channel\/UCuq4puT32DzHKT5rU1IZpIA"]},{"@type":"Person","@id":"https:\/\/www.pingcap.com\/#\/schema\/person\/3a46a13870b11ccb3f59734ca38ed634","name":"Wenbo Zhang","image":{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/www.pingcap.com\/#\/schema\/person\/image\/","url":"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg","contentUrl":"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg","caption":"Wenbo Zhang"},"description":"Linux Kernel Engineer","url":"https:\/\/www.pingcap.com\/ko\/blog\/author\/zhangwenbo\/"}]}},"grav_blocks":false,"card_markup":"<a class=\"card-resource bg-white\" href=\"https:\/\/www.pingcap.com\/ko\/blog\/how-tsq-becomes-a-performance-bottleneck-for-tikv-in-aws-arm-environment\/\"><div class=\"card-resource__content-container\"><div class=\"card-resource__content-head\"><div class=\"card-resource__category\">Engineering<\/div><\/div><h5 class=\"card-resource__title\">How TSQ Becomes a Performance Bottleneck for TiKV in AWS ARM Environment<\/h5><\/div><\/a>","_links":{"self":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts\/617","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/users\/226"}],"replies":[{"embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/comments?post=617"}],"version-history":[{"count":3,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts\/617\/revisions"}],"predecessor-version":[{"id":19004,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts\/617\/revisions\/19004"}],"wp:attachment":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/media?parent=617"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/categories?post=617"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/tags?post=617"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}