How do I make sure Variorum can use msr_batch? #542

brandonbiggs · 2024-05-02T14:39:35Z

brandonbiggs
May 2, 2024

Hi,

I was able to build Variorum and so far it seems like most things are working. However I've noticed when running some python examples, I often see a message like:

/opt/variorum/src/variorum/msr/msr_core.c:do_batch_op():215: _ERROR_VARIORUM_MSR_BATCH: IOctl failed, does /dev/cpu/msr_batch exist?

I did build msr_safe and rebuilt variorum, but I may not be linking them correctly? Apparently I'm using variorum version 0.5.0.. Seems like git clone is pulling an older version? Or maybe the version number hasn't been updated?

# python variorum-get-current-version-python-example.py

=== Running Variorum Get Current Version:
Variorum version is: 0.5.0

/dev/cpu/msr_batch does exist:

ls /dev/cpu/
0  10  12  14  16  18  2   21  23  25  27  29  30  4  6  8  msr_allowlist  msr_safe_version
1  11  13  15  17  19  20  22  24  26  28  3   31  5  7  9  msr_batch

Here's how I built variorum:

git clone https://github.com/llnl/variorum

cd variorum
mkdir build install
cd build

cmake -DHWLOC_DIR=/usr/include/ -DJANSSON_DIR=/usr/include/ -DENABLE_FORTRAN=OFF -DCMAKE_INSTALL_PREFIX=../install ../src

make -j8
make install

Here's the full output of a few of the python example commands:

# python variorum-print-thermals-python-example.py

=== Running Variorum Print Thermals:
server:/opt/variorum/src/variorum/msr/msr_core.c:do_batch_op():215: _ERROR_VARIORUM_MSR_BATCH: IOctl failed, does /dev/cpu/msr_batch exist?
    CPU   0, MSR 0x1b1, ERR (Unknown error -13)
    CPU   8, MSR 0x1b1, ERR (Unknown error -13)
server:/opt/variorum/src/variorum/msr/msr_core.c:do_batch_op():215: _ERROR_VARIORUM_MSR_BATCH: IOctl failed, does /dev/cpu/msr_batch exist?
    CPU   0, MSR 0x1a2, ERR (Unknown error -13)
    CPU   8, MSR 0x1a2, ERR (Unknown error -13)
server:/opt/variorum/src/variorum/msr/msr_core.c:do_batch_op():215: _ERROR_VARIORUM_MSR_BATCH: IOctl failed, does /dev/cpu/msr_batch exist?
    CPU   0, MSR 0x19c, ERR (Unknown error -13)
# all 32 cores had this error
    CPU  31, MSR 0x19c, ERR (Unknown error -13)
_THERMALS Socket PhysicalCore LogicalThread TCC_C Pkg_Reading_C Pkg_Actual_C Thread_Reading_C Thread_Actual_C Thread_DigitalReadingValid
_THERMALS 0 0 0 0 0 0 0 0 0
# All 32 cores had some thermal response here
_THERMALS 1 7 31 0 0 0 0 0 0

# python3 variorum-print-available-frequencies-python-example.py

=== Running Variorum Print Available Frequencies:
=== Turbo Schedule ===
server:/opt/variorum/src/variorum/msr/msr_core.c:do_batch_op():215: _ERROR_VARIORUM_MSR_BATCH: IOctl failed, does /dev/cpu/msr_batch exist?
    CPU   0, MSR 0x1ad, ERR (Unknown error -13)
    CPU   8, MSR 0x1ad, ERR (Unknown error -13)
server:/opt/variorum/src/variorum/msr/msr_core.c:do_batch_op():215: _ERROR_VARIORUM_MSR_BATCH: IOctl failed, does /dev/cpu/msr_batch exist?
    CPU   0, MSR 0x1ae, ERR (Unknown error -13)
    CPU   8, MSR 0x1ae, ERR (Unknown error -13)
 1C = 0 MHz
 2C = 0 MHz
 3C = 0 MHz
 4C = 0 MHz
 5C = 0 MHz
 6C = 0 MHz
 7C = 0 MHz
 8C = 0 MHz
 9 C = 0 MHz
10 C = 0 MHz
11 C = 0 MHz
12 C = 0 MHz
13 C = 0 MHz
14 C = 0 MHz
15 C = 0 MHz

tpatki · 2024-05-02T16:19:26Z

tpatki
May 2, 2024
Maintainer

Hi @brandonbiggs :

I can reproduce the incorrect version reported by Variorum at my end, I'll open an issue for fixing this.

Couple of questions to help debug:

What are the permissions on your msr_batch and msr_allowlist files?
Can you run the C examples first to make sure you can access the MSRs? The simplest one to try is the variorum-print-power-example. Let me know if you see the same error there.

1 reply

brandonbiggs May 2, 2024
Author

Thanks @tpatki !

What are the permissions on your msr_batch and msr_allowlist files?

crw-------  1 root root 236, 0 May  1 22:03 msr_allowlist
crw-------  1 root root 237, 0 May  1 22:03 msr_batch
cr--------  1 root root 235, 0 May  1 22:03 msr_safe_version

I am running variorum as root.

Can you run the C examples first to make sure you can access the MSRs? The simplest one to try is the variorum-print-power-example. Let me know if you see the same error there.

Sure. I'm trying to build the C example. I'm probably missing something here. Sorry!

# gcc variorum-print-power-example.c -o variorum_test -I /opt/variorum/install/include/
/tmp/ccWuMU3f.o: In function `main':
variorum-print-power-example.c:(.text+0x68): undefined reference to `variorum_get_current_version'
variorum-print-power-example.c:(.text+0xa5): undefined reference to `variorum_print_power'
collect2: error: ld returned 1 exit status

tpatki · 2024-05-02T16:49:24Z

tpatki
May 2, 2024
Maintainer

Hi @brandonbiggs:

As you are running with root, things should have worked. Did you try running the example with sudo python variorum-print-power-python-example.py, just in case?
In terms of msr-safe permissions in general, take a look at the setup section in the README for msr-safe (https://github.com/LLNL/msr-safe#setup). We recommend creating the msr-user and msr-admin groups, and giving rw permissions to the group (chmod g+rw /dev/cpu/*/msr_safe /dev/cpu/msr_*). You could try that, but you ideally shouldn't need that as root.
cmake should have built the examples in your build folder and installed them as well. Your Variorum build looks fine to me, did the examples not get installed for some reason? Could you post a verbose cmake output in that case?

0 replies

brandonbiggs · 2024-05-02T16:59:03Z

brandonbiggs
May 2, 2024
Author

As you are running with root, things should have worked. Did you try running the example with sudo python variorum-print-power-python-example.py, just in case?

# sudo LD_LIBRARY_PATH=/opt/variorum/install/lib/ python variorum-print-power-python-example.py

=== Running Variorum Print Power:
server:/opt/variorum/src/variorum/msr/msr_core.c:do_batch_op():215: _ERROR_VARIORUM_MSR_BATCH: IOctl failed, does /dev/cpu/msr_batch exist?
    CPU   0, MSR 0x611, ERR (Unknown error -13)
    CPU   8, MSR 0x611, ERR (Unknown error -13)
    CPU   0, MSR 0x619, ERR (Unknown error -13)
    CPU   8, MSR 0x619, ERR (Unknown error -13)
server:/opt/variorum/src/variorum/msr/msr_core.c:do_batch_op():215: _ERROR_VARIORUM_MSR_BATCH: IOctl failed, does /dev/cpu/msr_batch exist?
    CPU   0, MSR 0x606, ERR (Unknown error -13)
    CPU   8, MSR 0x606, ERR (Unknown error -13)
_PACKAGE_ENERGY_STATUS Offset Host Socket Bits Energy_J Power_W Elapsed_sec Timestamp_sec
_PACKAGE_ENERGY_STATUS 0x611 server 0 0 0.000000 0.000000 0.000000 0.000005
_PACKAGE_ENERGY_STATUS 0x611 server 1 0 0.000000 0.000000 0.000000 0.000005
_DRAM_ENERGY_STATUS Offset Host Socket Bits Energy_J Power_W Elapsed_sec Timestamp_sec
_DRAM_ENERGY_STATUS 0x619 server 0 0 0.000000 0.000000 0.000000 0.000005
_DRAM_ENERGY_STATUS 0x619 server 1 0 0.000000 0.000000 0.000000 0.000005

In terms of msr-safe permissions in general, take a look at the setup section in the README for msr-safe (https://github.com/LLNL/msr-safe#setup). We recommend creating the msr-user and msr-admin groups, and giving rw permissions to the group (chmod g+rw /dev/cpu//msr_safe /dev/cpu/msr_). You could try that, but you ideally shouldn't need that as root.

I did see this, but I figured since I'm running as root I may not need to set those groups up yet. I wanted to try to get this running as root first to eliminate some of the potential layers of complex, i.e. if I had permissions messed up running this as a non root user.

cmake should have built the examples in your build folder and installed them as well. Your Variorum build looks fine to me, did the examples not get installed for some reason? Could you post a verbose cmake output in that case?

You were right! Sorry. Yes, the examples did get installed. I was looking in src and not in build.

[root@server: (dev) /opt/variorum/build/examples ]
# ./variorum-print-power-example
server:/opt/variorum/src/variorum/msr/msr_core.c:do_batch_op():215: _ERROR_VARIORUM_MSR_BATCH: IOctl failed, does /dev/cpu/msr_batch exist?
    CPU   0, MSR 0x611, ERR (Unknown error -13)
    CPU   8, MSR 0x611, ERR (Unknown error -13)
    CPU   0, MSR 0x619, ERR (Unknown error -13)
    CPU   8, MSR 0x619, ERR (Unknown error -13)
server:/opt/variorum/src/variorum/msr/msr_core.c:do_batch_op():215: _ERROR_VARIORUM_MSR_BATCH: IOctl failed, does /dev/cpu/msr_batch exist?
    CPU   0, MSR 0x606, ERR (Unknown error -13)
    CPU   8, MSR 0x606, ERR (Unknown error -13)
_PACKAGE_ENERGY_STATUS Offset Host Socket Bits Energy_J Power_W Elapsed_sec Timestamp_sec
_PACKAGE_ENERGY_STATUS 0x611 server 0 0 0.000000 0.000000 0.000000 0.000004
_PACKAGE_ENERGY_STATUS 0x611 server 1 0 0.000000 0.000000 0.000000 0.000004
_DRAM_ENERGY_STATUS Offset Host Socket Bits Energy_J Power_W Elapsed_sec Timestamp_sec
_DRAM_ENERGY_STATUS 0x619 server 0 0 0.000000 0.000000 0.000000 0.000004
_DRAM_ENERGY_STATUS 0x619 server 1 0 0.000000 0.000000 0.000000 0.000004
Final result: inf
server:/opt/variorum/src/variorum/msr/msr_core.c:do_batch_op():215: _ERROR_VARIORUM_MSR_BATCH: IOctl failed, does /dev/cpu/msr_batch exist?
    CPU   0, MSR 0x611, ERR (Unknown error -13)
    CPU   8, MSR 0x611, ERR (Unknown error -13)
    CPU   0, MSR 0x619, ERR (Unknown error -13)
    CPU   8, MSR 0x619, ERR (Unknown error -13)
_PACKAGE_ENERGY_STATUS 0x611 server 0 0 0.000000 0.000000 0.167754 0.167732
_PACKAGE_ENERGY_STATUS 0x611 server 1 0 0.000000 0.000000 0.167754 0.167732
_DRAM_ENERGY_STATUS 0x619 server 0 0 0.000000 0.000000 0.167754 0.167732
_DRAM_ENERGY_STATUS 0x619 server 1 0 0.000000 0.000000 0.167754 0.167732

0 replies

tpatki · 2024-05-02T17:04:14Z

tpatki
May 2, 2024
Maintainer

@brandonbiggs
I'm not entirely sure why you're seeing this as a root user.

Could you, just for the sake of trying, set up your group permissions for the root user? Essentially, chmod g+rw /dev/cpu/*/msr_safe /dev/cpu/msr_*, without creating any other groups.

Essentially, make it look like:
crw-rw---- 1 root root msr_allowlist.

If that doesn't work, try setting both group and other permissions for root.

crw-rw-rw- 1 root root msr_allowlist.

Let me know if none of those ideas work...

5 replies

brandonbiggs May 2, 2024
Author

[root@server: (dev) /opt/variorum/build/examples ]
# ls -al /dev/cpu/msr*
crw------- 1 root root 236, 0 May  1 22:03 /dev/cpu/msr_allowlist
crw------- 1 root root 237, 0 May  1 22:03 /dev/cpu/msr_batch
cr-------- 1 root root 235, 0 May  1 22:03 /dev/cpu/msr_safe_version
[root@server: (dev) /opt/variorum/build/examples ]
# chmod g+rw /dev/cpu/msr_* /dev/cpu/*/msr_safe
[root@server: (dev) /opt/variorum/build/examples ]
# ls -al /dev/cpu/msr*
crw-rw---- 1 root root 236, 0 May  1 22:03 /dev/cpu/msr_allowlist
crw-rw---- 1 root root 237, 0 May  1 22:03 /dev/cpu/msr_batch
cr--rw---- 1 root root 235, 0 May  1 22:03 /dev/cpu/msr_safe_version
[root@server: (dev) /opt/variorum/build/examples ]
# ./variorum-print-power-example
server:/opt/variorum/src/variorum/msr/msr_core.c:do_batch_op():215: _ERROR_VARIORUM_MSR_BATCH: IOctl failed, does /dev/cpu/msr_batch exist?
    CPU   0, MSR 0x611, ERR (Unknown error -13)
    CPU   8, MSR 0x611, ERR (Unknown error -13)
    CPU   0, MSR 0x619, ERR (Unknown error -13)
    CPU   8, MSR 0x619, ERR (Unknown error -13)
server:/opt/variorum/src/variorum/msr/msr_core.c:do_batch_op():215: _ERROR_VARIORUM_MSR_BATCH: IOctl failed, does /dev/cpu/msr_batch exist?
    CPU   0, MSR 0x606, ERR (Unknown error -13)
    CPU   8, MSR 0x606, ERR (Unknown error -13)
_PACKAGE_ENERGY_STATUS Offset Host Socket Bits Energy_J Power_W Elapsed_sec Timestamp_sec
_PACKAGE_ENERGY_STATUS 0x611 server 0 0 0.000000 0.000000 0.000000 0.000003
_PACKAGE_ENERGY_STATUS 0x611 server 1 0 0.000000 0.000000 0.000000 0.000003
_DRAM_ENERGY_STATUS Offset Host Socket Bits Energy_J Power_W Elapsed_sec Timestamp_sec
_DRAM_ENERGY_STATUS 0x619 server 0 0 0.000000 0.000000 0.000000 0.000003
_DRAM_ENERGY_STATUS 0x619 server 1 0 0.000000 0.000000 0.000000 0.000003
Final result: inf
server:/opt/variorum/src/variorum/msr/msr_core.c:do_batch_op():215: _ERROR_VARIORUM_MSR_BATCH: IOctl failed, does /dev/cpu/msr_batch exist?
    CPU   0, MSR 0x611, ERR (Unknown error -13)
    CPU   8, MSR 0x611, ERR (Unknown error -13)
    CPU   0, MSR 0x619, ERR (Unknown error -13)
    CPU   8, MSR 0x619, ERR (Unknown error -13)
_PACKAGE_ENERGY_STATUS 0x611 server 0 0 0.000000 0.000000 0.167456 0.167431
_PACKAGE_ENERGY_STATUS 0x611 server 1 0 0.000000 0.000000 0.167456 0.167431
_DRAM_ENERGY_STATUS 0x619 server 0 0 0.000000 0.000000 0.167456 0.167431
_DRAM_ENERGY_STATUS 0x619 server 1 0 0.000000 0.000000 0.167456 0.167431

Then with other permissions set:

[root@server: (dev) /opt/variorum/build/examples ]
# ls -al /dev/cpu/msr*
crw-rw-rw- 1 root root 236, 0 May  1 22:03 /dev/cpu/msr_allowlist
crw-rw-rw- 1 root root 237, 0 May  1 22:03 /dev/cpu/msr_batch
cr--rw-rw- 1 root root 235, 0 May  1 22:03 /dev/cpu/msr_safe_version
[root@server: (dev) /opt/variorum/build/examples ]
# ls -al /dev/cpu/*/msr_safe
crw-rw-rw- 1 root root 234,  0 May  1 22:03 /dev/cpu/0/msr_safe
crw-rw-rw- 1 root root 234, 10 May  1 22:03 /dev/cpu/10/msr_safe
crw-rw-rw- 1 root root 234, 11 May  1 22:03 /dev/cpu/11/msr_safe
crw-rw-rw- 1 root root 234, 12 May  1 22:03 /dev/cpu/12/msr_safe
# didn't print all but they all look like this
[root@server: (dev) /opt/variorum/build/examples ]
# ./variorum-print-power-example
server:/opt/variorum/src/variorum/msr/msr_core.c:do_batch_op():215: _ERROR_VARIORUM_MSR_BATCH: IOctl failed, does /dev/cpu/msr_batch exist?
    CPU   0, MSR 0x611, ERR (Unknown error -13)
    CPU   8, MSR 0x611, ERR (Unknown error -13)
    CPU   0, MSR 0x619, ERR (Unknown error -13)
    CPU   8, MSR 0x619, ERR (Unknown error -13)
server:/opt/variorum/src/variorum/msr/msr_core.c:do_batch_op():215: _ERROR_VARIORUM_MSR_BATCH: IOctl failed, does /dev/cpu/msr_batch exist?
    CPU   0, MSR 0x606, ERR (Unknown error -13)
    CPU   8, MSR 0x606, ERR (Unknown error -13)
_PACKAGE_ENERGY_STATUS Offset Host Socket Bits Energy_J Power_W Elapsed_sec Timestamp_sec
_PACKAGE_ENERGY_STATUS 0x611 server 0 0 0.000000 0.000000 0.000000 0.000004
_PACKAGE_ENERGY_STATUS 0x611 server 1 0 0.000000 0.000000 0.000000 0.000004
_DRAM_ENERGY_STATUS Offset Host Socket Bits Energy_J Power_W Elapsed_sec Timestamp_sec
_DRAM_ENERGY_STATUS 0x619 server 0 0 0.000000 0.000000 0.000000 0.000004
_DRAM_ENERGY_STATUS 0x619 server 1 0 0.000000 0.000000 0.000000 0.000004
Final result: inf
server:/opt/variorum/src/variorum/msr/msr_core.c:do_batch_op():215: _ERROR_VARIORUM_MSR_BATCH: IOctl failed, does /dev/cpu/msr_batch exist?
    CPU   0, MSR 0x611, ERR (Unknown error -13)
    CPU   8, MSR 0x611, ERR (Unknown error -13)
    CPU   0, MSR 0x619, ERR (Unknown error -13)
    CPU   8, MSR 0x619, ERR (Unknown error -13)
_PACKAGE_ENERGY_STATUS 0x611 server 0 0 0.000000 0.000000 0.167525 0.167485
_PACKAGE_ENERGY_STATUS 0x611 server 1 0 0.000000 0.000000 0.167525 0.167485
_DRAM_ENERGY_STATUS 0x619 server 0 0 0.000000 0.000000 0.167525 0.167485
_DRAM_ENERGY_STATUS 0x619 server 1 0 0.000000 0.000000 0.167525 0.167485

tpatki May 2, 2024
Maintainer

Couple more things to check:

Could you also share the /proc/cpu output if possible? Is the architecture supported by Variorum (I am assuming you checked this, but just another sanity check)?
Can you install msr-tools and test whether you can actually access MSRs? I am trying to figure out if this is a msr_batch or ioctl issue or if access to MSRs is restricted.
To install msr-tools, you can look here: https://github.com/intel/msr-tools and use autotools to build. I know this is not maintained anymore by Intel. Once you build, can you try ./rdmsr -a 0x610 and see if you are able to access the registers themselves.
Another thing to try is to use the msr module instead of the msr_safe module. The goal of the msr-safe module is to allow user-space access to MSRs and also support batching which results in better performance. With root access, you can also try with the msr module (from what I recall that doesn't use batching).

Let me also poke @rountree to see if I missed something here or if he has some suggestions.

rountree May 2, 2024
Collaborator

@brandonbiggs @tpatki
The msr-safe documentation doesn't quite close the loop. You're getting a return code of -13 on the ioctl() call. Running errno will tell you what that is (the documentation should tell you to do that):

$ errno 13
EACCES 13 Permission denied

From the REAME.md file

EACCES An individual operation requested an MSR that is not present in the allowlist.

So that raises the question of whether the allowlist should be ignored if you're running as root. Let me take a look.

brandonbiggs May 2, 2024
Author

Thank you both. Really appreciate your help with this!

rountree May 2, 2024
Collaborator

After a quick skim, I think we took out the ability for root to ignore the allowlist for any access: read, write, and ioctl. I'll ask my colleagues if they remember that happening.

For now, just enable MSRs 0x611 and 0x619 in the /dev/cpu/msr_allowlist file and see if that works when running as root.

brandonbiggs · 2024-05-02T18:13:02Z

brandonbiggs
May 2, 2024
Author

Could you also share the /proc/cpu output if possible? Is the architecture supported by Variorum (I am assuming you checked this, but just another sanity check)?

I did check. I thought it was supported as a broadwell cpu.

# lscpu | grep -i model
Model:               79
Model name:          Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz
BIOS Model name:     Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz

Here's the first group. The remaining 32 look similar, just different IDs.

# cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 79
model name	: Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz
stepping	: 1
microcode	: 0xb00002e
cpu MHz		: 1824.207
cache size	: 25600 KB
physical id	: 0
siblings	: 16
core id		: 0
cpu cores	: 8
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 20
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts flush_l1d
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit mmio_stale_data
bogomips	: 6399.93
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

Can you install msr-tools and test whether you can actually access MSRs? I am trying to figure out if this is a msr_batch or ioctl issue or if access to MSRs is restricted.
To install msr-tools, you can look here: https://github.com/intel/msr-tools and use autotools to build. I know this is not maintained anymore by Intel. Once you build, can you try ./rdmsr -a 0x610 and see if you are able to access the registers themselves.

This went through all 32, I just cut the output. The registers were the same for all of them.

# ./rdmsr  -a 0x610
CPU 0: 7851000158438
CPU 1: 7851000158438
CPU 2: 7851000158438

# ./variorum-print-power-example
server:/opt/variorum/src/variorum/msr/msr_core.c:do_batch_op():215: _ERROR_VARIORUM_MSR_BATCH: IOctl failed, does /dev/cpu/msr_batch exist?

Another thing to try is to use the msr module instead of the msr_safe module. The goal of the msr-safe module is to allow user-space access to MSRs and also support batching which results in better performance. With root access, you can also try with the msr module (from what I recall that doesn't use batching).

Something like this right?

[root@server: (dev) /opt/variorum/build/examples ]
# rmmod msr-safe
[root@server: (dev) /opt/variorum/build/examples ]
# ./variorum-print-power-example
Warning: <variorum> Could not stat /dev/cpu/msr_allowlist: stat_module(): No such file or directory: server:/opt/variorum/src/variorum/msr/msr_core.c::321
/dev/cpu/msr_batch: No such file or directory
Warning: <variorum> No /dev/cpu/msr_batch, using compatibility batch: compatibility_batch(): No such file or directory: server:/opt/variorum/src/variorum/msr/msr_core.c::103
Warning: <variorum> No /dev/cpu/msr_batch, using compatibility batch: compatibility_batch(): No such file or directory: server:/opt/variorum/src/variorum/msr/msr_core.c::103
_PACKAGE_ENERGY_STATUS Offset Host Socket Bits Energy_J Power_W Elapsed_sec Timestamp_sec
_PACKAGE_ENERGY_STATUS 0x611 server 0 0xe11342d 14404.815247 0.000000 0.000000 0.000005
_PACKAGE_ENERGY_STATUS 0x611 server 1 0x67fac00c 106475.000732 0.000000 0.000000 0.000005
_DRAM_ENERGY_STATUS Offset Host Socket Bits Energy_J Power_W Elapsed_sec Timestamp_sec
_DRAM_ENERGY_STATUS 0x619 server 0 0x6a02b726 27138.715424 0.000000 0.000000 0.000005
_DRAM_ENERGY_STATUS 0x619 server 1 0xecd6863c 60630.524353 0.000000 0.000000 0.000005
Final result: inf
Warning: <variorum> Could not stat /dev/cpu/msr_allowlist: stat_module(): No such file or directory: server:/opt/variorum/src/variorum/msr/msr_core.c::321
Warning: <variorum> No /dev/cpu/msr_batch, using compatibility batch: compatibility_batch(): No such file or directory: server:/opt/variorum/src/variorum/msr/msr_core.c::103
_PACKAGE_ENERGY_STATUS 0x611 server 0 0xe11d10f 14407.266541 14.620014 0.167667 0.167659
_PACKAGE_ENERGY_STATUS 0x611 server 1 0x67fbb6a8 106478.854004 22.981693 0.167667 0.167659
_DRAM_ENERGY_STATUS 0x619 server 0 0x6a03924f 27139.571518 5.105920 0.167667 0.167659
_DRAM_ENERGY_STATUS 0x619 server 1 0xecd78af8 60631.542847 6.074503 0.167667 0.167659

1 reply

tpatki May 2, 2024
Maintainer

@brandonbiggs , looks like you can access MSRs and use the msr module. So this is most likely an allowlist issue as @rountree pointed out (thanks @rountree for catching that!).

You need to have all the registers you want access to in the allowlist along with the correct permissions. Can you paste a line of your allowlist here?

Did you copy the allowlist from here: https://github.com/LLNL/msr-safe/blob/main/allowlists/al_06_4F? In that case, you will need to uncomment the MSR of interest to allow for reading.

brandonbiggs · 2024-05-02T18:33:33Z

brandonbiggs
May 2, 2024
Author

@tpatki Is the link you shared the allowlist I should be using? Is there a way to generate an appropriate allowlist if that's not an appropriate one to use?

0 replies

tpatki · 2024-05-02T18:45:38Z

tpatki
May 2, 2024
Maintainer

Hi @brandonbiggs

The one I pointed to is a good starting point.
https://github.com/LLNL/msr-safe/blob/main/allowlists/al_06_4F

Go ahead and copy that, uncomment say 0x611 and 0x619 and let's start there. You should be able to read the values at that point from the power/energy MSRs. Based on your needs, you can uncomment other MSRs too.

We have typically left the options for setting write masks to the site and to the system administrators, as every site has different policies on which registers users should have write permissions to. I cannot as a result give you direct recommendations on what to use there.

For your individual testing purposes only, if you want to set power caps, you can use the 0x00FFFFFF00FFFFFF write mask for the 610h register. This will allow you to set a power cap, but you will have to make sure that you reset it to the original value in the MSR. You can do this by using resetting it with Variorum power capping example or with wrmsr. We don't recommend testing power caps until you have a good understanding on how to reset MSRs.

We also recommend reading MSRs first with Variorum/msr-safe and learning about MSRs in detail. The Intel Software Development Manual is the best resource for learning about MSRs. On the Variorum website, we have some information on some of the MSRs in our documentation: https://variorum.readthedocs.io/en/latest/Intel.html.

Intel SDMs are here: https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html

I got 06_4F from:

cpu family	: 6    <- 6h
model		: 79 <- 4Fh

I'll create an issue for adding this to our docs, we should have more details in our documentation around allowlists for new users.

5 replies

brandonbiggs May 2, 2024
Author

Okay, I believe I've followed along.

I uncommented those two lines.

# cat /dev/cpu/msr_allowlist
#MSR       Write mask
0x00000611 0x0000000000000000
0x00000619 0x0000000000000000
# ./variorum-print-power-example
server:/opt/variorum/src/variorum/msr/msr_core.c:do_batch_op():215: _ERROR_VARIORUM_MSR_BATCH: IOctl failed, does /dev/cpu/msr_batch exist?
    CPU   0, MSR 0x606, ERR (Unknown error -13)
    CPU   8, MSR 0x606, ERR (Unknown error -13)
_PACKAGE_ENERGY_STATUS Offset Host Socket Bits Energy_J Power_W Elapsed_sec Timestamp_sec
_PACKAGE_ENERGY_STATUS 0x611 server 0 0x1c0af49f 470480031.000000 0.000000 0.000000 0.000004
_PACKAGE_ENERGY_STATUS 0x611 server 1 0x7ad0f950 2060515664.000000 0.000000 0.000000 0.000004
_DRAM_ENERGY_STATUS Offset Host Socket Bits Energy_J Power_W Elapsed_sec Timestamp_sec
_DRAM_ENERGY_STATUS 0x619 server 0 0x8a847132 35460.442169 0.000000 0.000000 0.000004
_DRAM_ENERGY_STATUS 0x619 server 1 0x6d8684c 1752.407410 0.000000 0.000000 0.000004
Final result: inf
_PACKAGE_ENERGY_STATUS 0x611 server 0 0x1c0b905f 470519903.000000 237987.799783 0.167538 0.167465
_PACKAGE_ENERGY_STATUS 0x611 server 1 0x7ad1ee58 2060578392.000000 374410.581480 0.167538 0.167465
_DRAM_ENERGY_STATUS 0x619 server 0 0x8a854019 35461.250381 4.824054 0.167538 0.167465
_DRAM_ENERGY_STATUS 0x619 server 1 0x6d96198 1753.381226 5.812508 0.167538 0.167465

It looks like for the first error, I'd need to also uncomment # 0x00000606 0x0000000000000000 # "MSR_RAPL_POWER_UNIT (Table: 2-36)"?

I've got a lot to read about, but this has been super helpful. Thanks again for your help with this!

tpatki May 2, 2024
Maintainer

It should be okay to uncomment 606h. You will see the correct power values then (it uses the MSR_RAPL_POWER_UNIT to convert the units correctly).

brandonbiggs May 2, 2024
Author

That's looking great!

# ./variorum-print-power-example
_PACKAGE_ENERGY_STATUS Offset Host Socket Bits Energy_J Power_W Elapsed_sec Timestamp_sec
_PACKAGE_ENERGY_STATUS 0x611 server 0 0x1e52bc32 31050.940552 0.000000 0.000000 0.000008
_PACKAGE_ENERGY_STATUS 0x611 server 1 0x7de3bbe8 128910.936035 0.000000 0.000000 0.000008
_DRAM_ENERGY_STATUS Offset Host Socket Bits Energy_J Power_W Elapsed_sec Timestamp_sec
_DRAM_ENERGY_STATUS 0x619 server 0 0x8fcb7af4 36811.480286 0.000000 0.000000 0.000008
_DRAM_ENERGY_STATUS 0x619 server 1 0xb14359a 2836.209381 0.000000 0.000000 0.000008
Final result: inf
_PACKAGE_ENERGY_STATUS 0x611 server 0 0x1e5359d0 31053.403320 14.691080 0.167637 0.167600
_PACKAGE_ENERGY_STATUS 0x611 server 1 0x7de4b616 128914.845093 23.318585 0.167637 0.167600
_DRAM_ENERGY_STATUS 0x619 server 0 0x8fcc3141 36812.192398 4.247943 0.167637 0.167600
_DRAM_ENERGY_STATUS 0x619 server 1 0xb1533cb 2837.202316 5.923127 0.167637 0.167600

rountree May 2, 2024
Collaborator

msr-genie is useful for generating allowlists.

braw-lee Oct 1, 2024

looks like capping-core-frequency is not supported for Broadwell (06_4F) architecture
is there any guide/docs to implement it ? @tpatki

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How do I make sure Variorum can use msr_batch? #542

{{title}}

Replies: 7 comments 12 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

How do I make sure Variorum can use msr_batch? #542

brandonbiggs May 2, 2024

Replies: 7 comments · 12 replies

tpatki May 2, 2024 Maintainer

brandonbiggs May 2, 2024 Author

tpatki May 2, 2024 Maintainer

brandonbiggs May 2, 2024 Author

tpatki May 2, 2024 Maintainer

brandonbiggs May 2, 2024 Author

tpatki May 2, 2024 Maintainer

rountree May 2, 2024 Collaborator

brandonbiggs May 2, 2024 Author

rountree May 2, 2024 Collaborator

brandonbiggs May 2, 2024 Author

tpatki May 2, 2024 Maintainer

brandonbiggs May 2, 2024 Author

tpatki May 2, 2024 Maintainer

brandonbiggs May 2, 2024 Author

tpatki May 2, 2024 Maintainer

brandonbiggs May 2, 2024 Author

rountree May 2, 2024 Collaborator

braw-lee Oct 1, 2024

brandonbiggs
May 2, 2024

Replies: 7 comments 12 replies

tpatki
May 2, 2024
Maintainer

brandonbiggs May 2, 2024
Author

tpatki
May 2, 2024
Maintainer

brandonbiggs
May 2, 2024
Author

tpatki
May 2, 2024
Maintainer

brandonbiggs May 2, 2024
Author

tpatki May 2, 2024
Maintainer

rountree May 2, 2024
Collaborator

brandonbiggs May 2, 2024
Author

rountree May 2, 2024
Collaborator

brandonbiggs
May 2, 2024
Author

tpatki May 2, 2024
Maintainer

brandonbiggs
May 2, 2024
Author

tpatki
May 2, 2024
Maintainer

brandonbiggs May 2, 2024
Author

tpatki May 2, 2024
Maintainer

brandonbiggs May 2, 2024
Author

rountree May 2, 2024
Collaborator