After extensive investigation and diagnosis, we’ve identified ways to handle these bugs. In this post, I’ll share with you how we tackled these problems. However, as useful as these solutions are, we consider them workarounds, and we believe that more elegant solutions are possible. The goal of this post is to inspire the K8s community, RHEL<\/a>, \uadf8\ub9ac\uace0 CentOS<\/a> so that they can help fix these bugs thoroughly in the near future.<\/p>\n Keyword:<\/strong> SLUB: Unable to allocate memory on node -1<\/p>\n Related issues in the community:<\/strong><\/p>\n When we tested TiKV<\/a>‘s online transaction processing (OLTP<\/a>) performance in K8s, I\/O performance occasionally jittered. However, following items appeared normal:<\/p>\n The only negative indicator was the message, “SLUB: Unable to allocate memory on node -1” in the output of the We used funcslower<\/a> in perf-tools<\/a> to trace kernel functions that were executed slowly and adjusted the threshold value of the Based on the information above, we could see that the I\/O jitter was related to the file system executing We analyzed the call stack information output from If this operation failed, the kernel fell back to allocate the When we created the Docker container, we didn’t set Because we knew that the kmem accounting was unstable in the RHEL 3.10 kernel, we suspected that a kernel bug caused the SLUB allocation failure. We searched for kernel patch information, and found that it was a kernel bug, and that it had been fixed in Linux kernel version 4.x: slub: make dead caches discard free slabs immediately<\/a>. There was also a namespace leak issue associated with kmem accounting: mm: memcontrol: fix cgroup creation failure after many small jobs<\/a>.<\/p>\n So how was kmem accounting enabled? We used the opensnoop<\/a> tool in bcc<\/a> to monitor the kmem configuration file and captured runc<\/a> as the file modifier. From the K8s code, we found that kmem accounting was enabled by default in the K8s-dependent Based on our issue analysis, we can fix the bug in either of the following ways:<\/p>\n Now We need to disable the kmem accounting feature on both kubelet and Docker.<\/p>\n Then recompile kubelet.<\/li>\n If the returned result is as follows, it means that the kmem accounting feature is disabled successfully:<\/p>\n Keyword:<\/strong> kernel:unregister_netdevice: waiting for eth0 to become free. Usage count = 1<\/p>\n Related issues in the community:<\/strong><\/p>\n <\/p>\n After the K8s platform ran for a time, the following message was displayed: “Kernel:unregister_netdevice: waiting for eth0 to become free. Usage count = 1”. In addition, multiple processes would be in an uninterruptible state. The only workaround for this problem was to restart the server.<\/p>\n We used the crash tool to analyze vmcore. We found that the To avoid restarting the server every time this issue occurred, we developed a kernel module. When the However, this solution had the following defects:<\/p>\n As we prepared to drill down into the callback function logic registered by each subscriber, we also followed up on the progress of kernel patches and RHEL. We found that solutions:3659011<\/a> of RHEL had an update that mentioned a patch submitted by the upstream link: route: set the deleted fnhe fnhe_daddr to 0 in ip_del_fnhe to fix a race<\/a>.<\/p>\n We applied this patch to the kernel as a hotfix and tested for a week. The issue didn’t reappear. We reported our test information to RHEL, and they told us they’d started backporting the patch.<\/p>\n We recommend that you use CentOS 7.6 kernel-3.10.0-957 or later.<\/p>\n We’ve fixed these kernel bugs expediently. However, there are more effective long-term solutions. For Bug #1, we hope that the K8s community can provide an argument for kubelet to allow users to disable or enable the kmem accounting feature.<\/p>\n For Bug #2, the optimal solution is for RHEL and CentOS to fix the kernel bug. After TiDB users upgrade to a new version of CentOS, they won’t have to worry about this problem.<\/p>","protected":false},"excerpt":{"rendered":" This post introduces how the EE team at PingCAP tackled two Linux kernel bugs while testing TiDB Operator in K8s. PingCAP engineers also hope the K8s community, RHEL and CentOS can help fix these bugs thoroughly in the near future.<\/p>","protected":false},"author":226,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"ub_ctt_via":"","footnotes":""},"categories":[6],"tags":[68,33,70],"class_list":["post-1441","post","type-post","status-publish","format-standard","hentry","category-engineering","tag-bug-fix","tag-kubernetes","tag-linux"],"acf":[],"featured_image_src":null,"author_info":{"display_name":"Wenbo Zhang","author_link":"https:\/\/www.pingcap.com\/ko\/blog\/author\/zhangwenbo\/"},"yoast_head":"\n<\/span>Bug #1: Unstable kmem accounting<\/span><\/h2>\n
\n
Issue description<\/h3>\n
\n
dmesg<\/code> command.<\/p>\nIssue analysis<\/h3>\n
hung_task_timeout_secs<\/code> kernel parameter. When we performed the write operations in TiKV, we found the following kernel path information:<\/p>\n![\"Kernel](\"https:\/\/en.pingcap.com\/wp-content\/uploads\/2019\/05\/kernel-path-information-2.png\")
\nwritepage<\/code>. At the same time, before and after the performance jitter was captured, the dmesg<\/code> output contained a large amount of information about “SLUB: Unable to allocate memory on node -1”. This message occurred even when the node had sufficient memory.<\/p>\nhung_task<\/code> and the kernel code. We found that the kernel tried to allocate the bio_vec<\/code> object via kmem_cache_alloc<\/code> when executing the bvec_alloc<\/code> function to allocate the bio_vec<\/code> object.<\/p>\nbio_vec<\/code> object from the mempool. However, inside the mempool, the system first tried to execute the pool->alloc<\/code> callback for allocation. If this allocation failed, the kernel set the process to an uninterruptible state and put the process in a wait queue. After other processes returned memory resources to the mempool or the timer timed out (after 5 seconds), the process scheduler invoked the process for retry. The wait time of the wait queue was consistent with the jitter delay of our application monitoring.<\/p>\nmemory.kmem.limit_in_bytes<\/code>. But why was kmem insufficient? To determine whether memory.kmem.limit_in_bytes<\/code> was set, we went to the cgroup memory controller to check the container’s kmem information. The kmem statistics were enabled, and the kemem limit was set to a very large value.<\/p>\nrunc<\/code> project.<\/p>\nSolution<\/h3>\n
\n
runc<\/code> includes the conditional compilation option, and you can disable kmem accounting via Build Tags<\/a>. After we disabled this feature, our test result showed that the jitter disappeared, and so did the namespace leak and SLUB allocation failure problems.<\/p>\nOperation steps<\/h4>\n
\n
\n
make BUILDTAGS=\"nokmem\"\n<\/code><\/pre>\n\n
func EnableKernelMemoryAccounting(path string) error {\n return nil\n}\n<\/code><\/pre>\nfunc setKernelMemory(path string, kernelMemoryLimit int64) error {\n return nil\n}\n<\/code><\/pre>\ndocker-ce<\/code> to v18.09.1 or later. In these versions, kmem accounting of runc<\/code> is disabled in Docker.<\/li>\ncat \/sys\/fs\/cgroup\/memory\/kubepods\/burstable\/pod<pod-uid>\/<container-id>\/memory.kmem.slabinfo\n<\/code><\/pre>\n> cat: memory.kmem.slabinfo: Input\/output error\n<\/code><\/pre>\n<\/li>\n<\/ol>\n<\/span>Bug #2: Network device reference count leak<\/span><\/h2>\n
\n
Issue description<\/h3>\n
Issue analysis<\/h3>\n
netdev_wait_allrefs<\/code> function blocked the kernel thread, and that the function was in an infinite loop waiting for dev->refcnt<\/code> to drop to 0. Because the pod had been released, we suspected that this issue was due to a reference count leak. After analyzing K8s issues, we found that the kernel was the crux of the issue, but there was no simple, stable, and reliable method to reproduce the issue. Also, this issue still occurred in the later community version of the kernel.<\/p>\nnet_device<\/code> reference count was found to leak, this kernel module would be removed after the reference count was dropped to 0 (to avoid accidentally deleting other NICs that weren’t referenced by the leak). Besides, to avoid manual cleanup, we wrote a monitoring script that periodically automated this operation.<\/p>\n\n
netdev_wait_allrefs<\/code> would continually retry releasing the NETDEV_UNREGISTER<\/code> \uadf8\ub9ac\uace0 NETDEV_UNREGISTER_FINAL<\/code> messages to all message subscribers via notification chains. We traced this issue and found there were up to 22 subscribers. It was difficult to access the processing logic of all the callback functions registered by these subscribers and therefore difficult to decide whether we had a way to avoid misjudgement.<\/li>\n<\/ul>\nSolution<\/h3>\n
Operation steps<\/h4>\n
\n
UNAME=$(uname -r)\nsudo yum install gcc kernel-devel-${UNAME%.*} elfutils elfutils-devel\nsudo yum install pesign yum-utils zlib-devel \n binutils-devel newt-devel python-devel perl-ExtUtils-Embed \n audit-libs audit-libs-devel numactl-devel pciutils-devel bison\n\n# enable CentOS 7 debug repo\nsudo yum-config-manager --enable debug\n\nsudo yum-builddep kernel-${UNAME%.*}\nsudo debuginfo-install kernel-${UNAME%.*}\n\n# optional, but highly recommended - enable EPEL 7\nsudo yum install ccache\nccache --max-size=5G\n<\/code><\/pre>\n<\/li>\ngit clone https:\/\/github.com\/dynup\/kpatch && cd kpatch\nmake\nsudo make install\nsystemctl enable kpatch\n<\/code><\/pre>\n<\/li>\ncurl -SOL https:\/\/raw.githubusercontent.com\/pingcap\/kdt\/master\/kpatchs\/route.patch\nkpatch-build -t vmlinux route.patch (Compiles and generates the kernel module)\nmkdir -p \/var\/lib\/kpatch\/${UNAME}\ncp -a livepatch-route.ko \/var\/lib\/kpatch\/${UNAME}\nsystemctl restart kpatch (Loads the kernel module)\nkpatch list (Checks the loaded module)\n<\/code><\/pre>\n<\/li>\n<\/ol>\n<\/span>Conclusion<\/span><\/h2>\n