Troubleshooting

How To Enable Debug Logging in ipmctl

How To Enable Debug Logging in ipmctl

The ipmctl utility is used for configuring and managing Intel Optane Persistent Memory modules (DCPMM/PMem). It supports the functionality to:

  • Discover Persistent Memory on the server
  • Provision the persistent memory configuration
  • View and update the firmware on the persistent memory modules
  • Configure data-at-rest security
  • Track health and performance of the persistent memory modules
  • Debug and troubleshoot persistent memory modules

I wrote the IPMCTL User Guide showing how to use the tool, but what if ipmctl returns an error or something you’re not expecting? How do you debug the debugger? On Linux, ipmctl relies on libndctl to help perform communication to the BIOS and persistent memory modules themselves. This is a complicated stack involving multiple kernel drivers and the physical hardware itself. Anything along this path could be causing a problem.

Read More

Using ltrace to see what ipmctl and ndctl are doing

Occasionally, it is necessary to debug commands that are slow. Or you may simply be interested in learning how the tools work. While there are many strategies, here are some simple methods that show code flow and timing information.

To show a high-level view of where the time is being spent within libipmctl, use:

# ltrace -c -o ltrace_library_count.out -l '*ipmctl*' ipmctl show -memoryresources

To show a high-level view of where the time is being spent within libndctl, use:

Read More
"ipmctl show -memoryresources" returns "Error: GetMemoryResourcesInfo Failed"

"ipmctl show -memoryresources" returns "Error: GetMemoryResourcesInfo Failed"

Issue:

Running ipmctl show -memoryresources returns an error similar to the following:

# ipmctl show -memoryresources

Error: GetMemoryResourcesInfo Failed

Applies To:

  • Linux & Microsoft Windows

  • Intel Optane Persistent Memory

  • ipmctl utility

Cause:

The Platform Configuration Data (PCD) is invalid or has been erased using a previously executed ipmctl delete -dimm -pcd command or the system has new persistent memory modules that have not been initialized yet.

A module with an empty PCD will show information similar to the following. This shows an example of PCD of DIMM ID 0x0001. To review the PCD for all modules in the system use ipmctl show -dimm -pcd.

Read More

Intel Optane Persistent Memory Modules report "Non-functional" state in ipmctl

Issue

Executing ipmctl show-dimm to get device information shows the persistent memory modules in a ‘Non-functional’ health state, eg:

# ipmctl show -dimm

 DimmID | Capacity | HealthState    | ActionRequired | LockState | FWVersion
=============================================================================
 0x0001 | 0.0 GiB  | Non-functional | N/A            | N/A       | N/A
 0x0011 | 0.0 GiB  | Non-functional | N/A            | N/A       | N/A
 0x0021 | 0.0 GiB  | Non-functional | N/A            | N/A       | N/A
 0x0101 | 0.0 GiB  | Non-functional | N/A            | N/A       | N/A
 0x0111 | 0.0 GiB  | Non-functional | N/A            | N/A       | N/A
 0x0121 | 0.0 GiB  | Non-functional | N/A            | N/A       | N/A
 0x1001 | 0.0 GiB  | Non-functional | N/A            | N/A       | N/A
 0x1011 | 0.0 GiB  | Non-functional | N/A            | N/A       | N/A
 0x1021 | 0.0 GiB  | Non-functional | N/A            | N/A       | N/A
 0x1101 | 0.0 GiB  | Non-functional | N/A            | N/A       | N/A
 0x1111 | 0.0 GiB  | Non-functional | N/A            | N/A       | N/A
 0x1121 | 0.0 GiB  | Non-functional | N/A            | N/A       | N/A

Other ipmctl commands may fail and return “No functional DIMMs in the system.”, eg:

Read More