Events' information is missing

[nikosT]

Hi,

It seems that the current version dd66063 does not provide an output of the events that are being monitored. A working instance seems to be on (4 of August 2022): 74dde21. Both cases have been tested with the same configuration (i.e. env variables).

Env variables:

export LD_PRELOAD=/<path>/countdown/build/src/libcntd.so

export CNTD_DISABLE_POWER_MONITOR=on
export CNTD_PERF_ENABLE=on

# For each perf event
codes=(0xc4 0xc5 0x8108 0x149 0x412e 0x110 0x185)
for i in "${!codes[@]}"; do
        export CNTD_PERF_EVENT_$i=${codes[$i]}
done

export CNTD_ENABLE_REPORT=on
export CNTD_ENABLE_TIMESERIES_REPORT=on

The current version creates an output that contains:

...
MAX Memory usage:   	192.06 MByte
AVG IPC:            	1.46
AVG CPU frequency:      0 MHz
Cycles:                 352587934310
Instructions retired:   513596918758
##################### MPI TIMING #####################
APP time: 125.694 sec (99.04%)
MPI time: 1.221 sec (0.96%)
TOT time: 126.915 sec (100.00%)
##################### MPI REPORTING ##################
...

While the older commit creates an output that contains:

...
MAX Memory usage:   	189.16 MByte
AVG IPC:            	1.46
AVG CPU frequency:      2800 MHz
Cycles:                 351634069514
Instructions retired:   513562913509
Perf event 0:           31412920360
Perf event 1:           972966157
Perf event 2:           1719990
Perf event 3:           14324
Perf event 4:           9535
Perf event 5:           4313396880
Perf event 6:           0
...

[danielecesarini]

@f-tesser can help you

[f-tesser]

Good afternoon @nikosT, and thank you for using Countdown.

I have replicated your environmental set, but with the benchmark I have used, I have not encountered any issues.

What are the architecture and microarchitecture with which you are currently working with?

Regards,

Federico

[nikosT]

Hi @f-tesser, the benchmark is the known NAS Parallel Benchmark suite (NPB). The machines that the script was submitted to are the so-called "thin" nodes as they are defined here: https://doc.aris.grnet.gr/system/hardware/#thin-nodes

[f-tesser]

Good afternoon @nikosT.

Here I report to you the output of Countdown, on a node of our system and using your events, executing NPB "ft" class "D".

######################################################
##################### COUNTDOWN ######################
######################################################
EXE time: 228.774 sec
#################### GENERAL INFO ####################
Number of MPI Ranks: 32
Number of Nodes: 1
Number of Sockets: 2
Number of CPUs: 48
##################### ENERGY #########################
PKG: 62275 J
DRAM: 10316 J
##################### AVG POWER ######################
PKG: 272.21 W
DRAM: 45.09 W
################## PERFORMANCE INFO ##################
MPI network - SENT: 864.00 GByte
MPI network - RECV: 864.00 GByte
MPI network - TOT: 1.69 TByte
MPI file - WRITE: 0 Byte
MPI file - READ: 0 Byte
MPI file - TOT: 0 Byte
MAX Memory usage: 112.99 GByte
AVG IPC: 1.46
AVG CPU frequency: 2727 MHz
Cycles: 19495414716506
Instructions retired: 28486493784012
DP FLOPs: 9434266033225
DP FLOPs 64: 9111983546177
DP FLOPs 128: 322282487048
DP FLOPs 256: 0
DP FLOPs 512: 0
DP UOPs (TIME_EN/TIME_RUN): 9273124789701 ( 29272230407923/ 17888520599144)
DP UOPs 64 (TIME_EN/TIME_RUN): 9111983546177 ( 7318057620149/ 4472167090858)
DP UOPs 128 (TIME_EN/TIME_RUN): 161141243524 ( 7318057638357/ 4472131040797)
DP UOPs 256 (TIME_EN/TIME_RUN): 0 ( 7318057617538/ 4472126763004)
DP UOPs 512 (TIME_EN/TIME_RUN): 0 ( 7318057531879/ 4472095704485)
SP FLOPs: 1
SP FLOPs 32: 1
SP FLOPs 128: 0
SP FLOPs 256: 0
SP FLOPs 512: 0
SP UOPs (TIME_EN/TIME_RUN): 1 ( 29272230337859/ 17888451533545)
SP UOPs 32 (TIME_EN/TIME_RUN): 1 ( 7318057633682/ 4472131742652)
SP UOPs 128 (TIME_EN/TIME_RUN): 0 ( 7318057631667/ 4472123365274)
SP UOPs 256 (TIME_EN/TIME_RUN): 0 ( 7318057581497/ 4472096141980)
SP UOPs 512 (TIME_EN/TIME_RUN): 0 ( 7318057491013/ 4472100283639)
MEM UOPs (TIME_EN/TIME_RUN): 301906390654 ( 2745350208537/ 2745350208537)
MEM GLOBAL DATA: 19322009001856
Perf event 0: 1651535020067
Perf event 1: 2524441600
Perf event 2: 104198716
Perf event 3: 57586334
Perf event 4: 167288155976
Perf event 5: 0
Perf event 6: 14006883
##################### MPI TIMING #####################
APP time: 6423.972 sec (87.73%)
MPI time: 898.669 sec (12.27%)
TOT time: 7322.641 sec (100.00%)
##################### MPI REPORTING ##################
MPI_INIT: 32 - 0.000 Sec (0.00%)
MPI_ALLTOALL: 864 - 878.270 Sec (97.73%) - SEND 864.00 GByte - RECV 864.00 GByte
MPI_BARRIER: 32 - 2.949 Sec (0.33%)
MPI_BCAST: 160 - 1.881 Sec (0.21%) - SEND 640 Byte - RECV 620 Byte
MPI_COMM_RANK: 32 - 0.000 Sec (0.00%)
MPI_COMM_SIZE: 32 - 0.002 Sec (0.00%)
MPI_COMM_SPLIT: 64 - 0.025 Sec (0.00%)
MPI_REDUCE: 800 - 13.348 Sec (1.49%) - SEND 12.11 KByte - RECV 12.50 KByte
MPI_FINALIZE: 32 - 2.194 Sec (0.24%)
################## COUNTDOWN REPORTING ###############
MPI_ALLTOALL: 864 - 877.838 Sec (97.68%)
MPI_BARRIER: 31 - 2.934 Sec (0.33%)
MPI_BCAST: 31 - 1.861 Sec (0.21%)
MPI_COMM_SIZE: 1 - 0.001 Sec (0.00%)
MPI_COMM_SPLIT: 8 - 0.002 Sec (0.00%)
MPI_REDUCE: 307 - 13.159 Sec (1.46%)
MPI_FINALIZE: 31 - 2.178 Sec (0.24%)
################### COUNTDOWN SUMMARY ################
MPIs: 1273 - 897.972 Sec - MPI: 99.92% - TOT: 12.26%
######################################################

As you see, the events are present. We do not have any Ivy Bridge where to test Countdown. Could it be possible to have a temporary account on your system, to replicate your issues, and find a possible solution?

Regards,

Federico

[nikosT]

Hi @f-tesser, I'm afraid is not possible due to limited access to the system. As a workaround, we will continue to use the older commit (74dde21) which was working. I hope this will not affect our measurements.