Linux’s memory management system is transparent to the user. However, if you’re not familiar with its working principles, you might meet unexpected performance issues. That’s especially true for sophisticated software like databases. When databases are running in Linux, even small system variations might impact performance.<\/p>\n\n\n\n
After an in-depth investigation, we found that Transparent Huge Page (THP), a Linux memory management feature, often slows down database performance. In this post, I’ll describe how THP causes performance to fluctuate, the typical symptoms, and how to disable transparent huge pages in your Linux system.<\/p>\n\n\n\n
THP is an important feature of the Linux kernel. It maps page table entries to larger page sizes to reduce page faults. This improves the translation lookaside buffer (TLB) hit ratio. TLB is a memory cache used by the memory management unit to improve the translation speed from virtual memory addresses to physical memory addresses.<\/p>\n\n\n\n
When the application data being accessed is contiguous, THP often boosts performance. In contrast, if the memory access patterns are not contiguous, THP can’t fulfill its duty, and it may even cause system instability.<\/p>\n\n\n\n
Unfortunately, database workloads are known to have sparse rather than contiguous memory access. Therefore, you should disable THP for your database.<\/p>\n\n\n\n
To understand the harm THP can cause, let’s consider how Linux manages its physical memory.<\/p>\n\n\n\n
For different architectures, the Linux kernel employs different memory mapping approaches. Among them, the user space maps the memory via multi-level paging to save space, while the kernel space uses linear mapping to achieve simplicity and high efficiency.<\/p>\n\n\n\n
When the kernel starts, it adds physical pages to the buddy system. Every time the user applies for memory, the buddy system allocates the desired pages. When the user releases memory, the buddy system deallocates the pages.<\/p>\n\n\n\n
To accommodate low-speed devices and various workloads, Linux divides the memory pages into anonymous pages and file-based pages. Linux uses page cache to cache files for low-speed devices. When memory is insufficient, users can employ swap cache and swappiness to specify a proportion of the two types of pages to be released.<\/p>\n\n\n\n
To respond to the user’s memory application as soon as possible and guarantee that the system runs normally when the memory resources are insufficient, Linux defines three watermarks: During the direct reclaim, if the pages are clean, the blockage caused by synchronous reclaim is short; otherwise, it might result in tens of milliseconds of latency, and, depending on the back-end devices, sometimes even seconds.<\/p>\n\n\n\n Apart from the watermarks, another mechanism may also cause direct memory reclaim. Sometimes, a thread applies for a large section of continuous memory pages. If there is enough physical memory, but it’s fragmented, the kernel performs memory compaction. This might also trigger a direct memory reclaim.<\/p>\n\n\n\n To sum up, when threads apply for memory, the major causes of latency are direct memory reclaim and memory compaction. For workloads whose memory access is not very contiguous, such as databases, THP may trigger the two tasks and thus cause fluctuating performance.<\/p>\n\n\n\n If your system performance fluctuates, how can you be sure THP is the cause? I’d like to share three symptoms that we’ve found are related to THP.<\/p>\n\n\n\n Based on our customer support experience, the most typical symptom of THP-caused performance fluctuation is sharply rising system CPU utilization.<\/p>\n\n\n\n In such cases, if you create an on-cpu flame graph using perf, you’ll see that all the service threads that are in the runnable state are performing memory compaction. In addition, the page fault exception handler is high<\/code>, low<\/code>, \uadf8\ub9ac\uace0 min<\/code>.<\/p>\n\n\n\n\n
low<\/code> and more than min<\/code>, when the user applies for memory, the page replacement daemon kswapd<\/code> asynchronously frees memory until the available physical memory is higher than high<\/code>.<\/li>\n\n\n\n<\/span>When THP causes performance fluctuation<\/span><\/h2>\n\n\n\n
The most typical symptom:
sys cpu<\/code> rises<\/h3>\n\n\n\ndo_huge_pmd_anonymous_page<\/code>. This means that the current system doesn’t have 2 MB of contiguous physical memory and that triggers the direct memory compaction. The direct memory compaction is time-consuming, so it leads to high system CPU utilization.<\/p>\n\n\n\n