January 4, 2023

How To Build a custom Linux Kernel to test Data Access Monitor (DAMON)

Posted on January 4, 2023  •  5 minutes

DAMON is a data access monitoring framework subsystem for the Linux kernel. DAMON (Data Access MONitor) tool monitors memory access patterns specific to user-space processes introduced in Linux kernel 5.15 LTS, such as operation schemes, physical memory monitoring, and proactive memory reclamation. It was designed and implemented by Amazon AWS Labs and upstreamed into the 5.15 Kernel , but it was not enabled by default.cd /boot

Keen to try this new feature to identify the working set size (Active Memory) of a server or process, this post documents the steps I took to build a custom Kernel with DAMON enabled using Fedora Server 35.

To use this feature, you should first ensure your system is running on a kernel that is built with CONFIG_DAMON_RECLAIM=y. To check if your kernel was built with the DAMON feature, run the following:

$ grep CONFIG_DAMON /boot/config-$(uname -r)

If you see one or more entries, congratulations, your Kernel does have the feature and you do not need to continue following this post.
If no output is returned, your Kernel does not have the feature, and you should continue reading.

At the time of writing - 23rd December 2021 - neither the latest stable Fedora Kernel (5.15.10-200.fc35.x86_64) nor the Linus Torvalds mainline Kernel (5.16.0-0.rc6.20211222git2f47a9a4dfa3.43.vanilla.1.fc35.x86_64) had the feature enabled. So I needed to build a custom Kernel. If you’ve never built a Kernel, the process isn’t as daunting as one might imagine or used to be. I simply followed some of the instructions on https://fedoraproject.org/wiki/Building_a_custom_kernel and https://sjp38.github.io/post/damon/#install .

Install the dependencies for building Kernels

sudo dnf install fedpkg fedora-packager rpmdevtools ncurses-devel pesign grubby

Create a build directory

mkdir ~/downloads
cd ~/downloads

Clone the Linus Torvalds kernel main branch from kernel.org. This is the bleeding edge release. This will clone the entire upstream tree and may take a while depending on your Internet connection speed.

git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
- or -
git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

cd linux

If you prefer a stable Kernel release, use:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
- or -
git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git

cd linux

Create a Kernel configuration file .config in the current directory

make oldconfig

Add the following DAMON entries to the .config file. Open the file for editing and locate the DAMON section which should look something like this:

# Data Access Monitoring
# CONFIG_DAMON is not set
# end of Data Access Monitoring
# end of Memory Management options

Add the following entries so your .config file now looks like this:

# Data Access Monitoring
# end of Data Access Monitoring
# end of Memory Management options

Start the building process using all available CPUs

make -j$(nproc)
make bzImage
make modules

Assuming no errors were reported during the build phase, install the new Kernel

sudo make modules_install
sudo make install

Reboot the host. The kernel is unlikely to be the default option in GRUB, so make sure to select the new Kernel when prompted.

Copy the config file to /boot for future reference

cp ~/downloads/linux/.config /boot/config-$(uname -r)

Once you have tested the Kernel, you may choose to remove the source code to free space on your storage.

rm -rf ~/downloads/linux

To let sysadmins enable or disable it and tune for the given system, DAMON_RECLAIM utilizes module parameters. That is, you can put **damon**_reclaim.<parameter>=<value> on the kernel boot command line or write proper values to /sys/modules/damon_reclaim/parameters/<parameter> files.

# ls /sys/module/damon_reclaim/parameters/
aggr_interval  max_nr_regions  monitor_region_end    quota_reset_interval_ms  wmarks_high      wmarks_mid
enabled        min_age         monitor_region_start  quota_sz                 wmarks_interval
kdamond_pid    min_nr_regions  quota_ms              sample_interval          wmarks_low

A description of each parameter can be found in the documentation .

DAMO - A User-Space Tool

A user-space tool for DAMON, called DAMO  is available. It’s also available at PyPi . You may start using DAMON by following the Getting Started  of the tool for start.

To use the latest development release from GitHub, clone the repository:

mkdir ~/downloads
cd ~/downloads
git clone https://github.com/awslabs/damo
cd damo

Getting help:

# ./damo version

# ./damo -h
usage: damo [-h] <command> ...

  -h, --help  show this help message and exit

    record    record data accesses
    schemes   apply operation schemes
    report    report the recorded data accesses in the specified form
    monitor   repeat the recording and the reporting of data accesses
    adjust    adjust the record results with different monitoring attributes
    reclaim   control DAMON_RECLAIM
    features  list supported DAMON features in the kernel
    validate  validate a given record result file
    version   print the version number

# ./damo features
init_regions: Supported
paddr: Supported
record: Unsupported
schemes: Supported
schemes_prioritization: Supported
schemes_quotas: Supported
schemes_speed_limit: Supported
schemes_wmarks: Supported

# ./damo report -h
usage: damo report [-h] <report type> ...

  -h, --help     show this help message and exit

report type:
  <report type>  the type of the report to generate
    raw          human readable raw data
    heats        heats of regions
    wss          working set size
    nr_regions   number of regions


In this post, I showed how to build a custom Linux Kernel with the DAMON feature enabled.


Related Posts

How To Enable Debug Logging in ipmctl

The ipmctl utility is used for configuring and managing Intel Optane Persistent Memory modules (DCPMM/PMem). It supports the functionality to: Discover Persistent Memory on the server Provision the persistent memory configuration View and update the firmware on the persistent memory modules Configure data-at-rest security Track health and performance of the persistent memory modules Debug and troubleshoot persistent memory modules I wrote the IPMCTL User Guide showing how to use the tool, but what if ipmctl returns an error or something you’re not expecting?

Read More

How To Monitor Persistent Memory Performance on Linux using PCM, Prometheus, and Grafana

In a previous article, I showed How To Install Prometheus and Grafana on Fedora Server . This article demonstrates how to use the open-source Process Counter Monitor (PCM) utility to collect DRAM and Intel® Optane™ Persistent Memory statistics, and visualize the data in Grafana. Processor Counter Monitor is an application programming interface (API) and a set of tools based on the API to monitor performance and energy metrics of Intel® Core™, Xeon®, Atom™ and Xeon Phi™ processors.

Read More

Using ltrace to see what ipmctl and ndctl are doing

Occasionally, it is necessary to debug commands that are slow. Or you may simply be interested in learning how the tools work. While there are many strategies, here are some simple methods that show code flow and timing information. To show a high-level view of where the time is being spent within libipmctl, use: # ltrace -c -o ltrace_library_count.out -l '*ipmctl*' ipmctl show -memoryresources To show a high-level view of where the time is being spent within libndctl, use:

Read More