DAMON is a data access monitoring framework subsystem for the Linux kernel. DAMON (Data Access MONitor) tool monitors memory access patterns specific to user-space processes introduced in Linux kernel 5.15 LTS, such as operation schemes, physical memory monitoring, and proactive memory reclamation. It was designed and implemented by Amazon AWS Labs and upstreamed into the 5.15 Kernel, but it was not enabled by default.cd /boot
Keen to try this new feature to identify the working set size (Active Memory) of a server or process, this post documents the steps I took to build a custom Kernel with DAMON enabled using Fedora Server 35.
To use this feature, you should first ensure your system is running on a kernel that is built with CONFIG_DAMON_RECLAIM=y. To check if your kernel was built with the DAMON feature, run the following:
$ grep CONFIG_DAMON /boot/config-$(uname -r)
If you see one or more entries, congratulations, your Kernel does have the feature and you do not need to continue following this post.
If no output is returned, your Kernel does not have the feature, and you should continue reading.
At the time of writing – 23rd December 2021 – neither the latest stable Fedora Kernel (5.15.10-200.fc35.x86_64) nor the Linus Torvalds mainline Kernel (5.16.0-0.rc6.20211222git2f47a9a4dfa3.43.vanilla.1.fc35.x86_64) had the feature enabled. So I needed to build a custom Kernel. If you’ve never built a Kernel, the process isn’t as daunting as one might imagine or used to be. I simply followed some of the instructions on https://fedoraproject.org/wiki/Building_a_custom_kernel and https://sjp38.github.io/post/damon/#install.
Install the dependencies for building Kernels
sudo dnf install fedpkg fedora-packager rpmdevtools ncurses-devel pesign grubby
Create a build directory
mkdir ~/downloads cd ~/downloads
Clone the Linus Torvalds kernel main branch from kernel.org. This is the bleeding edge release. This will clone the entire upstream tree and may take a while depending on your Internet connection speed.
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git - or - git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git cd linux
If you prefer a stable Kernel release, use:
git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git - or - git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git cd linux
Create a Kernel configuration file
.config in the current directory
Add the following DAMON entries to the
.config file. Open the file for editing and locate the DAMON section which should look something like this:
# # Data Access Monitoring # # CONFIG_DAMON is not set # end of Data Access Monitoring # end of Memory Management options
Add the following entries so your .config file now looks like this:
# # Data Access Monitoring # CONFIG_DAMON=y CONFIG_DAMON_KUNIT_TEST=y CONFIG_DAMON_VADDR=y CONFIG_DAMON_PADDR=y CONFIG_DAMON_PGIDLE=y CONFIG_DAMON_VADDR_KUNIT_TEST=y CONFIG_DAMON_DBGFS=y CONFIG_DAMON_DBGFS_KUNIT_TEST=y CONFIG_DAMON_RECLAIM=y # end of Data Access Monitoring # end of Memory Management options
Start the building process using all available CPUs
make -j$(nproc) make bzImage make modules
Assuming no errors were reported during the build phase, install the new Kernel
sudo make modules_install sudo make install
Reboot the host. The kernel is unlikely to be the default option in GRUB, so make sure to select the new Kernel when prompted.
Copy the config file to /boot for future reference
cp ~/downloads/linux/.config /boot/config-$(uname -r)
Once you have tested the Kernel, you may choose to remove the source code to free space on your storage.
rm -rf ~/downloads/linux
To let sysadmins enable or disable it and tune for the given system, DAMON_RECLAIM utilizes module parameters. That is, you can put
damon_reclaim.<parameter>=<value> on the kernel boot command line or write proper values to
# ls /sys/module/damon_reclaim/parameters/ aggr_interval max_nr_regions monitor_region_end quota_reset_interval_ms wmarks_high wmarks_mid enabled min_age monitor_region_start quota_sz wmarks_interval kdamond_pid min_nr_regions quota_ms sample_interval wmarks_low
A description of each parameter can be found in the documentation.
DAMO – A User-Space Tool
To use the latest development release from GitHub, clone the repository:
mkdir ~/downloads cd ~/downloads git clone https://github.com/awslabs/damo cd damo
# ./damo version 1.0.9 # ./damo -h usage: damo [-h] <command> ... options: -h, --help show this help message and exit command: record record data accesses schemes apply operation schemes report report the recorded data accesses in the specified form monitor repeat the recording and the reporting of data accesses adjust adjust the record results with different monitoring attributes reclaim control DAMON_RECLAIM features list supported DAMON features in the kernel validate validate a given record result file version print the version number # ./damo features init_regions: Supported paddr: Supported record: Unsupported schemes: Supported schemes_prioritization: Supported schemes_quotas: Supported schemes_speed_limit: Supported schemes_wmarks: Supported # ./damo report -h usage: damo report [-h] <report type> ... options: -h, --help show this help message and exit report type: <report type> the type of the report to generate raw human readable raw data heats heats of regions wss working set size nr_regions number of regions
In this post, I showed how to build a custom Linux Kernel with the DAMON feature enabled.