{"id":607,"date":"2020-12-24T00:00:00","date_gmt":"2020-12-24T08:00:00","guid":{"rendered":"https:\/\/en.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/"},"modified":"2024-08-21T06:10:21","modified_gmt":"2024-08-21T13:10:21","slug":"how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance","status":"publish","type":"post","link":"https:\/\/www.pingcap.com\/ko\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/","title":{"rendered":"Trace Linux System Calls with Least Impact on Performance in Production"},"content":{"rendered":"<p>If you need to dynamically trace Linux process system calls, you might first consider strace. strace is simple to use and works well for issues such as &#8220;Why can&#8217;t the software run on this machine?&#8221; However, if you&#8217;re running a trace in a production environment, strace is NOT a good choice. It introduces a substantial amount of overhead. According to a performance test conducted by Arnaldo Carvalho de Melo, a senior software engineer at Red Hat, <strong>the process traced using strace ran 173 times slower, which is disastrous for a production environment<\/strong>.<\/p>\n\n\n\n<p>So are there any tools that excel at tracing system calls in a production environment? The answer is YES. <strong>This blog post introduces perf and traceloop, two commonly used command-line tools, to help you trace system calls in a production environment.<\/strong><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"perf_a_performance_profiler_for_Linux\"><\/span>perf, a performance profiler for Linux<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>perf is a powerful Linux profiling tool, refined and upgraded by Linux kernel developers. In addition to common features such as analyzing Performance Monitoring Unit (PMU) hardware events and kernel events, perf has the following subcomponents:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>sched: Analyzes scheduler actions and latencies.<\/li>\n\n\n\n<li>timechart: Visualizes system behaviors based on the workload.<\/li>\n\n\n\n<li>c2c: Detects the potential for false sharing. Red Hat once tested the c2c prototype on a number of Linux applications and found many cases of false sharing and cache lines on hotspots.<\/li>\n\n\n\n<li>trace: Traces system calls with acceptable overheads. It performs only <strong>1.36<\/strong> times slower with workloads specified in the <code>dd<\/code> command.<\/li>\n<\/ul>\n\n\n\n<p>Let&#8217;s look at some common uses of perf.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>To see which commands made the most system calls: <br><br> <pre><code class=\"language-shell\">perf top -F 49 -e raw_syscalls:sys_enter --sort comm,dso --show-nr-samples <\/code><\/pre> <img loading=\"lazy\" decoding=\"async\" width=\"723\" height=\"174\" class=\"wp-image-17687\" style=\"\" src=\"https:\/\/static.pingcap.com\/files\/2020\/12\/14192946\/system-call-counts.jpg\" alt=\"system-call-counts\" srcset=\"https:\/\/static.pingcap.com\/files\/2020\/12\/14192946\/system-call-counts.jpg 723w, https:\/\/static.pingcap.com\/files\/2020\/12\/14192946\/system-call-counts-300x72.jpg 300w\" sizes=\"auto, (max-width: 723px) 100vw, 723px\" \/><div class=\"caption-center\">System call counts<\/div><p>From the output, you can see that the <code>kube-apiserver<\/code> command had the most system calls during sampling.<\/p><\/li>\n\n\n\n<li> To see system calls that have latencies longer than a specific duration. In the following example, this duration is 200 milliseconds:  <pre><code class=\"language-shell\">perf trace --duration 200 <\/code><\/pre> <figure><img loading=\"lazy\" decoding=\"async\" width=\"776\" height=\"191\" class=\"wp-image-608\" src=\"https:\/\/en.pingcap.com\/wp-content\/uploads\/2020\/12\/system-calls-longer-than-200-ms.jpg\" alt=\"System calls longer than 200 ms\" srcset=\"https:\/\/static.pingcap.com\/files\/2020\/12\/system-calls-longer-than-200-ms.jpg 776w, https:\/\/static.pingcap.com\/files\/2020\/12\/system-calls-longer-than-200-ms-300x74.jpg 300w, https:\/\/static.pingcap.com\/files\/2020\/12\/system-calls-longer-than-200-ms-768x189.jpg 768w\" sizes=\"auto, (max-width: 776px) 100vw, 776px\" \/><\/figure> <div class=\"caption-center\">System calls longer than 200 ms<\/div> <p>From the output, you can see the process names, process IDs (PIDs), the specific system calls that exceed 200 ms, and the returned values.<\/p> <\/li>\n\n\n\n<li><p>To see the processes that had system calls within a period of time and a summary of their overhead:<code class=\"language-shell\">perf trace -p $PID  -s <\/code><\/p> <figure><img loading=\"lazy\" decoding=\"async\" width=\"501\" height=\"581\" class=\"wp-image-609\" src=\"https:\/\/en.pingcap.com\/wp-content\/uploads\/2020\/12\/system-call-overheads-by-process.jpg\" alt=\"System call overheads by process\" srcset=\"https:\/\/static.pingcap.com\/files\/2020\/12\/system-call-overheads-by-process.jpg 501w, https:\/\/static.pingcap.com\/files\/2020\/12\/system-call-overheads-by-process-259x300.jpg 259w\" sizes=\"auto, (max-width: 501px) 100vw, 501px\" \/><\/figure> <div class=\"caption-center\">System call overheads by process<\/div> <p>From the output, you can see the times of each system call, the times of the errors, the total latency, the average latency, and so on.<\/p><\/li>\n\n\n\n<li> <p>To analyze the stack information of calls that have a high latency:<\/p>  <pre><code class=\"language-shell\">perf trace record --call-graph dwarf -p $PID -- sleep 10 <\/code><\/pre> <figure><img loading=\"lazy\" decoding=\"async\" width=\"484\" height=\"374\" class=\"wp-image-610\" src=\"https:\/\/en.pingcap.com\/wp-content\/uploads\/2020\/12\/stack-information-of-system-calls-with-high-latency.jpg\" alt=\"Stack information of system calls with high latency\" srcset=\"https:\/\/static.pingcap.com\/files\/2020\/12\/stack-information-of-system-calls-with-high-latency.jpg 484w, https:\/\/static.pingcap.com\/files\/2020\/12\/stack-information-of-system-calls-with-high-latency-300x232.jpg 300w\" sizes=\"auto, (max-width: 484px) 100vw, 484px\" \/><\/figure> <div class=\"caption-center\">Stack information of system calls with high latency<\/div> <\/li>\n\n\n\n<li> <p>To trace a group of tasks. For example, two BPF tools are running in the background. To see their system call information, you can add them to a <code>perf_event<\/code> cgroup and then execute <code>per trace<\/code>:<\/p>  <pre><code class=\"language-shell\">mkdir \/sys\/fs\/cgroup\/perf_event\/bpftools\/ echo 22542 &gt;&gt; \/sys\/fs\/cgroup\/perf_event\/bpftools\/tasks echo 20514 &gt;&gt; \/sys\/fs\/cgroup\/perf_event\/bpftools\/tasks perf trace -G bpftools -a -- sleep 10 <\/code><\/pre> <figure><img loading=\"lazy\" decoding=\"async\" width=\"908\" height=\"253\" class=\"wp-image-611\" src=\"https:\/\/en.pingcap.com\/wp-content\/uploads\/2020\/12\/trace-a-group-of-tasks.jpg\" alt=\"Trace a group of tasks\" srcset=\"https:\/\/static.pingcap.com\/files\/2020\/12\/trace-a-group-of-tasks.jpg 908w, https:\/\/static.pingcap.com\/files\/2020\/12\/trace-a-group-of-tasks-300x84.jpg 300w, https:\/\/static.pingcap.com\/files\/2020\/12\/trace-a-group-of-tasks-768x214.jpg 768w\" sizes=\"auto, (max-width: 908px) 100vw, 908px\" \/><\/figure> <div class=\"caption-center\">Trace a group of tasks<\/div> <\/li>\n<\/ul>\n\n\n\n<p>Those are some of the most common uses of perf. If you&#8217;d like to know more (especially about perf-trace), see the <a href=\"https:\/\/man7.org\/linux\/man-pages\/man1\/perf-trace.1.html\">Linux manual page<\/a>. From the manual pages, you will learn that perf-trace can filter tasks based on PIDs or thread IDs (TIDs), but that it has no convenient support for containers and the Kubernetes (K8s) environments. Don&#8217;t worry. Next, we&#8217;ll discuss a tool that can easily trace system calls in containers and in K8s environments that uses cgroup v2.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Traceloop_a_performance_profiler_for_cgroup_v2_and_K8s\"><\/span>Traceloop, a performance profiler for cgroup v2 and K8s<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Traceloop provides better support for tracing Linux system calls in the containers or K8s environments that use cgroup v2. You might be unfamiliar with traceloop but know BPF Compiler Collection (BCC) pretty well. (Its front-end is implemented using Python or C++.) In the IO Visor Project, BCC&#8217;s parent project, there is another project named gobpf that provides Golang bindings for the BCC framework. Based on gobpf, traceloop is developed for environments of containers and K8s. The following illustration shows the traceloop architecture:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"588\" height=\"342\" src=\"https:\/\/en.pingcap.com\/wp-content\/uploads\/2020\/12\/traceloop-architecture.jpg\" alt=\"traceloop architecture\" class=\"wp-image-612\" srcset=\"https:\/\/static.pingcap.com\/files\/2020\/12\/traceloop-architecture.jpg 588w, https:\/\/static.pingcap.com\/files\/2020\/12\/traceloop-architecture-300x174.jpg 300w\" sizes=\"auto, (max-width: 588px) 100vw, 588px\" \/><\/figure>\n\n\n\n<div class=\"caption-center\">traceloop architecture<\/div>\n\n\n\n<p>We can further simplify this illustration into the following key procedures. Note that these procedures are implementation details, not operations to perform:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><code>bpf helper<\/code> gets the cgroup ID. Tasks are filtered based on the cgroup ID rather than on the PID and TID.<\/li>\n\n\n\n<li>Each cgroup ID corresponds to a bpf tail call that can call and execute another eBPF program and replace the execution context. Syscall events are written through a bpf tail call to a perf ring buffer with the same cgroup ID.<\/li>\n\n\n\n<li>The user space reads the perf ring buffer based on this cgroup ID.<\/li>\n<\/ol>\n\n\n\n<p><strong>Note:<\/strong><\/p>\n\n\n\n<p>Currently, you can get the cgroup ID only by executing <code>bpf helper: bpf_get_current_cgroup_id<\/code>, and this ID is available only in cgroup v2. Therefore, before you use traceloop, make sure that cgroup v2 is enabled in your environment.<\/p>\n\n\n\n<p>In the following demo (on the CentOS 8 4.18 kernel), when traceloop exits, the system call information is traced:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo -E .\/traceloop cgroups --dump-on-exit \/sys\/fs\/cgroup\/system.slice\/sshd.service\n<\/code><\/pre>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"623\" height=\"427\" src=\"https:\/\/en.pingcap.com\/wp-content\/uploads\/2020\/12\/traceloop-tracing-system-calls.jpg\" alt=\"traceloop tracing system calls\" class=\"wp-image-613\" srcset=\"https:\/\/static.pingcap.com\/files\/2020\/12\/traceloop-tracing-system-calls.jpg 623w, https:\/\/static.pingcap.com\/files\/2020\/12\/traceloop-tracing-system-calls-300x206.jpg 300w\" sizes=\"auto, (max-width: 623px) 100vw, 623px\" \/><\/figure>\n\n\n\n<div class=\"caption-center\">traceloop tracing system calls<\/div>\n\n\n\n<p>As the results show, the traceloop output is similar to that of strace or perf-trace except for the cgroup-based task filtering. Note that CentOS 8 mounts cgroup v2 directly on the <code>\/sys\/fs\/cgroup<\/code> path instead of on <code>\/sys\/fs\/cgroup\/unified<\/code> as Ubuntu does. Therefore, before you use traceloop, you should run <code>mount -t cgroup2<\/code> to determine the mount information.<\/p>\n\n\n\n<p>The team behind traceloop has integrated it with the Inspektor Gadget project, so you can run traceloop on the K8s platform using kubectl. See the demos in <a href=\"https:\/\/github.com\/kinvolk\/inspektor-gadget#how-to-use\">Inspektor Gadget &#8211; How to use<\/a> and, if you like, try it on your own.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Benchmark_with_system_calls_traced\"><\/span>Benchmark with system calls traced<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>We conducted a sysbench test in which system calls were either traced using multiple tracers (traceloop, strace, and perf-trace) or not traced. The benchmark results are as follows:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"518\" height=\"236\" src=\"https:\/\/en.pingcap.com\/wp-content\/uploads\/2020\/12\/sysbench-results-with-system-calls-traced-and-untraced.jpg\" alt=\"Sysbench results with system calls traced and untraced\" class=\"wp-image-614\" srcset=\"https:\/\/static.pingcap.com\/files\/2020\/12\/sysbench-results-with-system-calls-traced-and-untraced.jpg 518w, https:\/\/static.pingcap.com\/files\/2020\/12\/sysbench-results-with-system-calls-traced-and-untraced-300x137.jpg 300w\" sizes=\"auto, (max-width: 518px) 100vw, 518px\" \/><\/figure>\n\n\n\n<div class=\"caption-center\">Sysbench results with system calls traced and untraced<\/div>\n\n\n\n<p>As the benchmark shows, strace caused the biggest decrease in application performance. perf-trace caused a smaller decrease, and traceloop caused the smallest.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Summary_of_Linux_profilers\"><\/span>Summary of Linux profilers<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>For issues such as &#8220;Why can&#8217;t the software run on this machine,&#8221; strace is still a powerful system call tracer in Linux. But to trace the latency of system calls, the BPF-based perf-trace is a better option. In containers or K8s environments that use cgroup v2, traceloop is the easiest to use.<\/p>","protected":false},"excerpt":{"rendered":"<p>To trace system calls in Linux effectively, you can use perf to analyze system calls that have latency in general scenarios. For containers or Kubernetes that use cgroup v2, traceloop is more convenient.<\/p>","protected":false},"author":226,"featured_media":616,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"ub_ctt_via":"","footnotes":""},"categories":[6],"tags":[70,35],"class_list":["post-607","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-engineering","tag-linux","tag-performance-tuning"],"acf":[],"featured_image_src":"https:\/\/static.pingcap.com\/files\/2020\/12\/how-to-trace-linux-syscalls.png","author_info":{"display_name":"Wenbo Zhang","author_link":"https:\/\/www.pingcap.com\/ko\/blog\/author\/zhangwenbo\/"},"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.9 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Trace Linux System Calls with Least Impact on Performance<\/title>\n<meta name=\"description\" content=\"This post introduces perf and traceloop, two commonly command-line tools, to help you trace system calls in a production environment.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.pingcap.com\/ko\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/\" \/>\n<meta property=\"og:locale\" content=\"ko_KR\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Trace Linux System Calls with Least Impact on Performance\" \/>\n<meta property=\"og:description\" content=\"This post introduces perf and traceloop, two commonly command-line tools, to help you trace system calls in a production environment.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.pingcap.com\/ko\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/\" \/>\n<meta property=\"og:site_name\" content=\"TiDB\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/facebook.com\/pingcap2015\" \/>\n<meta property=\"article:published_time\" content=\"2020-12-24T08:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-08-21T13:10:21+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/static.pingcap.com\/files\/2020\/12\/how-to-trace-linux-syscalls.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1999\" \/>\n\t<meta property=\"og:image:height\" content=\"667\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Wenbo Zhang\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@PingCAP\" \/>\n<meta name=\"twitter:site\" content=\"@PingCAP\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Wenbo Zhang\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7\ubd84\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/\"},\"author\":{\"name\":\"Wenbo Zhang\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/person\/3a46a13870b11ccb3f59734ca38ed634\"},\"headline\":\"Trace Linux System Calls with Least Impact on Performance in Production\",\"datePublished\":\"2020-12-24T08:00:00+00:00\",\"dateModified\":\"2024-08-21T13:10:21+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/\"},\"wordCount\":992,\"publisher\":{\"@id\":\"https:\/\/www.pingcap.com\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/static.pingcap.com\/files\/2020\/12\/how-to-trace-linux-syscalls.png\",\"keywords\":[\"Linux\",\"Performance tuning\"],\"articleSection\":[\"Engineering\"],\"inLanguage\":\"ko-KR\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/\",\"url\":\"https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/\",\"name\":\"Trace Linux System Calls with Least Impact on Performance\",\"isPartOf\":{\"@id\":\"https:\/\/www.pingcap.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/static.pingcap.com\/files\/2020\/12\/how-to-trace-linux-syscalls.png\",\"datePublished\":\"2020-12-24T08:00:00+00:00\",\"dateModified\":\"2024-08-21T13:10:21+00:00\",\"description\":\"This post introduces perf and traceloop, two commonly command-line tools, to help you trace system calls in a production environment.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/#breadcrumb\"},\"inLanguage\":\"ko-KR\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/#primaryimage\",\"url\":\"https:\/\/static.pingcap.com\/files\/2020\/12\/how-to-trace-linux-syscalls.png\",\"contentUrl\":\"https:\/\/static.pingcap.com\/files\/2020\/12\/how-to-trace-linux-syscalls.png\",\"width\":1999,\"height\":667},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.pingcap.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Trace Linux System Calls with Least Impact on Performance in Production\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.pingcap.com\/#website\",\"url\":\"https:\/\/www.pingcap.com\/\",\"name\":\"TiDB\",\"description\":\"TiDB | SQL at Scale\",\"publisher\":{\"@id\":\"https:\/\/www.pingcap.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.pingcap.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"ko-KR\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.pingcap.com\/#organization\",\"name\":\"PingCAP\",\"url\":\"https:\/\/www.pingcap.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png\",\"contentUrl\":\"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png\",\"width\":811,\"height\":232,\"caption\":\"PingCAP\"},\"image\":{\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/facebook.com\/pingcap2015\",\"https:\/\/x.com\/PingCAP\",\"https:\/\/linkedin.com\/company\/pingcap\",\"https:\/\/youtube.com\/channel\/UCuq4puT32DzHKT5rU1IZpIA\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/person\/3a46a13870b11ccb3f59734ca38ed634\",\"name\":\"Wenbo Zhang\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ko-KR\",\"@id\":\"https:\/\/www.pingcap.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg\",\"contentUrl\":\"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg\",\"caption\":\"Wenbo Zhang\"},\"description\":\"Linux Kernel Engineer\",\"url\":\"https:\/\/www.pingcap.com\/ko\/blog\/author\/zhangwenbo\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Trace Linux System Calls with Least Impact on Performance","description":"This post introduces perf and traceloop, two commonly command-line tools, to help you trace system calls in a production environment.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.pingcap.com\/ko\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/","og_locale":"ko_KR","og_type":"article","og_title":"Trace Linux System Calls with Least Impact on Performance","og_description":"This post introduces perf and traceloop, two commonly command-line tools, to help you trace system calls in a production environment.","og_url":"https:\/\/www.pingcap.com\/ko\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/","og_site_name":"TiDB","article_publisher":"https:\/\/facebook.com\/pingcap2015","article_published_time":"2020-12-24T08:00:00+00:00","article_modified_time":"2024-08-21T13:10:21+00:00","og_image":[{"width":1999,"height":667,"url":"https:\/\/static.pingcap.com\/files\/2020\/12\/how-to-trace-linux-syscalls.png","type":"image\/png"}],"author":"Wenbo Zhang","twitter_card":"summary_large_image","twitter_creator":"@PingCAP","twitter_site":"@PingCAP","twitter_misc":{"Written by":"Wenbo Zhang","Est. reading time":"7\ubd84"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/#article","isPartOf":{"@id":"https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/"},"author":{"name":"Wenbo Zhang","@id":"https:\/\/www.pingcap.com\/#\/schema\/person\/3a46a13870b11ccb3f59734ca38ed634"},"headline":"Trace Linux System Calls with Least Impact on Performance in Production","datePublished":"2020-12-24T08:00:00+00:00","dateModified":"2024-08-21T13:10:21+00:00","mainEntityOfPage":{"@id":"https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/"},"wordCount":992,"publisher":{"@id":"https:\/\/www.pingcap.com\/#organization"},"image":{"@id":"https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/#primaryimage"},"thumbnailUrl":"https:\/\/static.pingcap.com\/files\/2020\/12\/how-to-trace-linux-syscalls.png","keywords":["Linux","Performance tuning"],"articleSection":["Engineering"],"inLanguage":"ko-KR"},{"@type":"WebPage","@id":"https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/","url":"https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/","name":"Trace Linux System Calls with Least Impact on Performance","isPartOf":{"@id":"https:\/\/www.pingcap.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/#primaryimage"},"image":{"@id":"https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/#primaryimage"},"thumbnailUrl":"https:\/\/static.pingcap.com\/files\/2020\/12\/how-to-trace-linux-syscalls.png","datePublished":"2020-12-24T08:00:00+00:00","dateModified":"2024-08-21T13:10:21+00:00","description":"This post introduces perf and traceloop, two commonly command-line tools, to help you trace system calls in a production environment.","breadcrumb":{"@id":"https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/#breadcrumb"},"inLanguage":"ko-KR","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/"]}]},{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/#primaryimage","url":"https:\/\/static.pingcap.com\/files\/2020\/12\/how-to-trace-linux-syscalls.png","contentUrl":"https:\/\/static.pingcap.com\/files\/2020\/12\/how-to-trace-linux-syscalls.png","width":1999,"height":667},{"@type":"BreadcrumbList","@id":"https:\/\/www.pingcap.com\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.pingcap.com\/"},{"@type":"ListItem","position":2,"name":"Trace Linux System Calls with Least Impact on Performance in Production"}]},{"@type":"WebSite","@id":"https:\/\/www.pingcap.com\/#website","url":"https:\/\/www.pingcap.com\/","name":"\ud2f0DB","description":"TiDB | SQL at Scale","publisher":{"@id":"https:\/\/www.pingcap.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.pingcap.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"ko-KR"},{"@type":"Organization","@id":"https:\/\/www.pingcap.com\/#organization","name":"PingCAP","url":"https:\/\/www.pingcap.com\/","logo":{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/","url":"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png","contentUrl":"https:\/\/static.pingcap.com\/files\/2021\/11\/pingcap-logo.png","width":811,"height":232,"caption":"PingCAP"},"image":{"@id":"https:\/\/www.pingcap.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/facebook.com\/pingcap2015","https:\/\/x.com\/PingCAP","https:\/\/linkedin.com\/company\/pingcap","https:\/\/youtube.com\/channel\/UCuq4puT32DzHKT5rU1IZpIA"]},{"@type":"Person","@id":"https:\/\/www.pingcap.com\/#\/schema\/person\/3a46a13870b11ccb3f59734ca38ed634","name":"Wenbo Zhang","image":{"@type":"ImageObject","inLanguage":"ko-KR","@id":"https:\/\/www.pingcap.com\/#\/schema\/person\/image\/","url":"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg","contentUrl":"https:\/\/static.pingcap.com\/files\/2022\/10\/17234942\/avatar.jpg","caption":"Wenbo Zhang"},"description":"Linux Kernel Engineer","url":"https:\/\/www.pingcap.com\/ko\/blog\/author\/zhangwenbo\/"}]}},"grav_blocks":false,"card_markup":"<a class=\"card-resource bg-white\" href=\"https:\/\/www.pingcap.com\/ko\/blog\/how-to-trace-linux-system-calls-in-production-with-minimal-impact-on-performance\/\"><div class=\"card-resource__image-container\"><img class=\"card-resource__image\" alt=\"how-to-trace-linux-syscalls.jpg\" src=\"https:\/\/static.pingcap.com\/files\/2020\/12\/how-to-trace-linux-syscalls.png\" loading=\"lazy\" width=1999 height=667 \/><\/div><div class=\"card-resource__content-container\"><div class=\"card-resource__content-head\"><div class=\"card-resource__category\">Engineering<\/div><\/div><h5 class=\"card-resource__title\">Trace Linux System Calls with Least Impact on Performance in Production<\/h5><\/div><\/a>","_links":{"self":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts\/607","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/users\/226"}],"replies":[{"embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/comments?post=607"}],"version-history":[{"count":11,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts\/607\/revisions"}],"predecessor-version":[{"id":17689,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/posts\/607\/revisions\/17689"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/media\/616"}],"wp:attachment":[{"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/media?parent=607"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/categories?post=607"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.pingcap.com\/ko\/wp-json\/wp\/v2\/tags?post=607"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}