Blog Posts

How To map VMWare vSphere/ESXi PMem devices from the Host to Guest VM

In this post, we’ll use VMWare ESXi 7.0u3 to create a Guest VM running Ubuntu 21.10 with two Virtual Persistent Memory (vPMem) devices, then show how we can map the vPMem device in the host (ESXi) to “nmem” devices in the guest VM as shown by the ndctl utility.

If you’re new to using vPMem or need a refresher, start with the VMWare Persistent Memory documentation.

Create a Guest VM with vPMem Devices
Install the Guest OS
Install ndctl
Mapping the NVDIMMs in the guest to devices in the ESXi Host
Summary

Create a Guest VM with vPMem Devices

The procedure you use may be different from the one shown below if you use vSphere or an automated procedure.

How To Emulate CXL Devices using KVM and QEMU

What is CXL?

Compute Express Link (CXL) is an open standard for high-speed central processing unit-to-device and CPU-to-memory connections, designed for high-performance data center computers. CXL is built on the PCI Express physical and electrical interface with protocols in three areas: input/output, memory, and cache coherence.

CXL is designed to be an industry open standard interface for high-speed communications, as accelerators are increasingly used to complement CPUs in support of emerging applications such as Artificial Intelligence and Machine Learning.

How To Build a custom Linux Kernel to test Data Access Monitor (DAMON)

DAMON is a data access monitoring framework subsystem for the Linux kernel. DAMON (Data Access MONitor) tool monitors memory access patterns specific to user-space processes introduced in Linux kernel 5.15 LTS, such as operation schemes, physical memory monitoring, and proactive memory reclamation. It was designed and implemented by Amazon AWS Labs and upstreamed into the 5.15 Kernel , but it was not enabled by default.cd /boot

Keen to try this new feature to identify the working set size (Active Memory) of a server or process, this post documents the steps I took to build a custom Kernel with DAMON enabled using Fedora Server 35.

Resolving commands 'Killed' on GCP f1-micro Compute Engine instances

When I want to perform a quick task, I generally spin up a Google GCP Compute Engine instance as they’re cheap. However, they have limited resources, particularly memory. When refreshing the package repositories, it’s quite easy to encounter an Out-of-Memory (OOM) situation which results in the command - yum or dnf - is ‘killed’. For example:

$ sudo dnf update 
CentOS Stream 8 - AppStream                                                                                                  8.3 MB/s |  18 MB     00:02    
CentOS Stream 8 - BaseOS                                                                                                      13 MB/s |  16 MB     00:01    
CentOS Stream 8 - Extras                                                                                                      69 kB/s |  16 kB     00:00    
Google Compute Engine                                                                                                         20 kB/s | 9.4 kB     00:00    
Google Cloud SDK                                                                                                              24 MB/s |  43 MB     00:01    
Killed

dmesg has a lot of information about the situation, but the key line to confirm dnf caused the OOM event, is:

How To Enable Debug Logging in ipmctl

The ipmctl utility is used for configuring and managing Intel Optane Persistent Memory modules (DCPMM/PMem). It supports the functionality to:

Discover Persistent Memory on the server
Provision the persistent memory configuration
View and update the firmware on the persistent memory modules
Configure data-at-rest security
Track health and performance of the persistent memory modules
Debug and troubleshoot persistent memory modules

I wrote the IPMCTL User Guide showing how to use the tool, but what if ipmctl returns an error or something you’re not expecting? How do you debug the debugger? On Linux, ipmctl relies on libndctl to help perform communication to the BIOS and persistent memory modules themselves. This is a complicated stack involving multiple kernel drivers and the physical hardware itself. Anything along this path could be causing a problem.

How To Install Prometheus and Grafana on Fedora Server

[Updated] This article was updated on 03/13/2021 using Fedora Server 33, Prometheus v2.25.0, Grafana v7.4.3, and Node Exporter v1.1.2.

In this article, we will show how to install Prometheus and Grafana to collect and display system performance metrics.

Prometheus is an open source monitoring and alerting toolkit for bare metal systems, virtual machines, containers, and microservices. Grafana allows you to query, visualize, and alert on metrics using fully customizable dashboards .

How To Monitor Persistent Memory Performance on Linux using PCM, Prometheus, and Grafana

In a previous article, I showed How To Install Prometheus and Grafana on Fedora Server . This article demonstrates how to use the open-source Process Counter Monitor (PCM) utility to collect DRAM and Intel® Optane™ Persistent Memory statistics, and visualize the data in Grafana.

Processor Counter Monitor is an application programming interface (API) and a set of tools based on the API to monitor performance and energy metrics of Intel® Core™, Xeon®, Atom™ and Xeon Phi™ processors. It can also show memory bandwidth for DRAM and Intel Optane Persistent Memory devices. PCM works on Linux, Windows, Mac OS X, FreeBSD and DragonFlyBSD operating systems.

How to , Linux , Performance , Troubleshooting

Occasionally, it is necessary to debug commands that are slow. Or you may simply be interested in learning how the tools work. While there are many strategies, here are some simple methods that show code flow and timing information.

To show a high-level view of where the time is being spent within libipmctl, use:

# ltrace -c -o ltrace_library_count.out -l '*ipmctl*' ipmctl show -memoryresources

To show a high-level view of where the time is being spent within libndctl, use: