Using Linux Kernel Tiering with Compute Express Link (CXL) Memory
- Steve Scargall
- Cxl
- May 9, 2024
In this blog post, we will walk through the process of enabling the Linux Kernel Transparent Page Placement (TPP) feature with CXL memory mapped as NUMA nodes using the system-ram namespace. This feature allows the kernel to automatically place pages in different types of memory based on their usage patterns.
Prerequisites
This guide assumes that you are using a Fedora 36 system with Kernel 5.19.13, and that your system has a Samsung CXL device installed. You can confirm the presence of the CXL device with the following command:
lspci | grep CXL
Step 1: Verify Automatic Memory Onlining
First, we need to verify if the OS automatically onlines memory. This can be done with the following command:
grep CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE /boot/config-$(uname -r)
If the output is CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y, then the OS is configured to automatically online memory.
Step 2: Change the Default Memory Zone
Next, we change the default memory zone when memory is onlined to ZONE_MOVABLE. This can be done with the following command:
sudo echo online_movable > /sys/devices/system/memory/auto_online_blocks
Step 3: Convert the Namespace
We then use daxctl to convert the namespace from devdax to system-ram for all CXL Devices. This can be done with the following command:
daxctl reconfigure-device --mode=system-ram --force all
Step 4: Verify NUMA Output
At this point, you should be able to see the single-CPU (NODE0) and Samsung CXL device (NODE1) in the NUMA output. You can check this with the following command:
numactl -H
Step 5: Display Memory Blocks by NUMA Node and Zone
You can display the memory blocks by NUMA node and Zone with the following command:
lsmem -o +NODE,ZONES
Step 6: Enable Kernel Transparent Page Placement (TPP)
Finally, we can enable Kernel Transparent Page Placement (TPP). First, check the default setting for page demotions:
cat /sys/kernel/mm/numa/demotion_enabled
If the output is false, enable it with the following command:
echo true > /sys/kernel/mm/numa/demotion_enabled
Then, enable promotions:
echo 2 > /proc/sys/kernel/numa_balancing
Lastly, do reclaim for each zone. This makes sure that demotion is run to maintain a minimum set of free pages in each NUMA node:
echo 1 > /proc/sys/vm/zone_reclaim_mode
And that’s it! You have now enabled the Linux Kernel Transparent Page Placement (TPP) feature with CXL memory mapped as NUMA nodes using the system-ram namespace.
Please note that this guide is based on a specific system configuration and may need to be adjusted based on your specific hardware and software setup. Always refer to the official documentation for the most accurate and up-to-date information.
