[DLIO_PROFILER ERROR]: signal caught 6 if benchmark is run with RDMA mount and data_loader: dali

[alexander272272]

mount:
IP:/ifs on /mnt/1/ifs type nfs (rw,relatime,vers=3,rsize=131072,wsize=524288,namlen=255,hard,forcerdirplus,proto=rdma,nconnect=24,port=20049,timeo=600,retrans=2,sec=sys,mountaddr=IP,mountvers=3,mountproto=tcp,local_lock=none,addr=IP)

h# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

h# ./benchmark.sh run --hosts HOST --workload resnet50 --accelerator-type h100 --num-accelerators 2 --results-dir resultsdir-$(date +"%d-%m-%Y") --param data
set.num_files_train=1200 --param dataset.data_folder=/mnt/1/ifs/data/rosnet50_05_04_2024_x02
[INFO] 2024-04-08T16:08:03.406678 Profiling DLIO /root/aan/storage/resultsdir-08-04-2024/trace-0-of-2.pfw [/root/aan/storage/dlio_benchmark/dlio_benchmark/utils/config.py:189]
[INFO] 2024-04-08T16:08:03.407010 Running DLIO with 2 process(es) [/root/aan/storage/dlio_benchmark/dlio_benchmark/main.py:98]
[INFO] 2024-04-08T16:08:03.635388 Max steps per epoch: 1876 = 1251 * 1200 / 400 / 2 (samples per file * num files / batch size / comm size) [/root/aan/storage/dlio_benchmark/dlio_benchm
ark/main.py:322]
[INFO] 2024-04-08T16:08:07.053113 Starting epoch 1: 1876 steps expected [/root/aan/storage/dlio_benchmark/dlio_benchmark/utils/statscounter.py:128]
[INFO] 2024-04-08T16:08:07.053269 Starting block 1 [/root/aan/storage/dlio_benchmark/dlio_benchmark/utils/statscounter.py:198]
A process has executed an operation involving a call
to the fork() system call to create a child process.

As a result, the libfabric EFA provider is operating in
a condition that could result in memory corruption or
other system errors.

For the libfabric EFA provider to work safely when fork()
is called, you will need to set the following environment
variable:
RDMAV_FORK_SAFE

However, setting this environment variable can result in
signficant performance impact to your application due to
increased cost of memory registration.

You may want to check with your application vendor to see
if an application-level alternative (of not using fork)
exists.

Your job will now abort.

Your job will now abort.
[DLIO_PROFILER ERROR]: signal caught 6
/usr/local/lib/python3.10/dist-packages/dlio_profiler_py.cpython-310-x86_64-linux-gnu.so(+0x30325) [0x7f8db87ca325]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f8e0f8fd520]
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c) [0x7f8e0f9519fc]
/lib/x86_64-linux-gnu/libc.so.6(raise+0x16) [0x7f8e0f8fd476]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3) [0x7f8e0f8e37f3]
/lib/x86_64-linux-gnu/libfabric.so.1(+0x76b4e) [0x7f8db8a76b4e]
/lib/x86_64-linux-gnu/libc.so.6(+0xeaf48) [0x7f8e0f9a5f48]
/lib/x86_64-linux-gnu/libc.so.6(__libc_fork+0x71) [0x7f8e0f9a5711]
python3(+0x287a6e) [0x56136209ba6e]
python3(+0x157a3e) [0x561361f6ba3e]
python3(_PyEval_EvalFrameDefault+0x614a) [0x561361f5ccfa]
python3(_PyFunction_Vectorcall+0x7c) [0x561361f6e9fc]
python3(_PyEval_EvalFrameDefault+0x8ac) [0x561361f5745c]
python3(_PyObject_FastCallDictTstate+0xc4) [0x561361f63c14]
python3(+0x164a64) [0x561361f78a64]
python3(_PyObject_MakeTpCall+0x1fc) [0x561361f64a1c]
python3(_PyEval_EvalFrameDefault+0x64e6) [0x561361f5d096]
python3(_PyFunction_Vectorcall+0x7c) [0x561361f6e9fc]
python3(_PyEval_EvalFrameDefault+0x614a) [0x561361f5ccfa]
python3(_PyFunction_Vectorcall+0x7c) [0x561361f6e9fc]
python3(_PyEval_EvalFrameDefault+0x8ac) [0x561361f5745c]
python3(_PyFunction_Vectorcall+0x7c) [0x561361f6e9fc]
python3(_PyEval_EvalFrameDefault+0x8ac) [0x561361f5745c]
python3(_PyObject_FastCallDictTstate+0xc4) [0x561361f63c14]
python3(+0x164b05) [0x561361f78b05]
python3(_PyObject_MakeTpCall+0x1fc) [0x561361f64a1c]
python3(_PyEval_EvalFrameDefault+0x64e6) [0x561361f5d096]
python3(+0x1687f1) [0x561361f7c7f1]
python3(_PyEval_EvalFrameDefault+0x614a) [0x561361f5ccfa]
python3(+0x1687f1) [0x561361f7c7f1]
python3(_PyEval_EvalFrameDefault+0x198c) [0x561361f5853c]
python3(_PyFunction_Vectorcall+0x7c) [0x561361f6e9fc]
python3(_PyEval_EvalFrameDefault+0x8ac) [0x561361f5745c]
python3(_PyFunction_Vectorcall+0x7c) [0x561361f6e9fc]
python3(_PyEval_EvalFrameDefault+0x8ac) [0x561361f5745c]
python3(_PyFunction_Vectorcall+0x7c) [0x561361f6e9fc]
python3(PyObject_Call+0x122) [0x561361f7d492]
python3(_PyEval_EvalFrameDefault+0x2a27) [0x561361f595d7]
python3(+0x1687f1) [0x561361f7c7f1]
python3(_PyEval_EvalFrameDefault+0x198c) [0x561361f5853c]
^C[mpiexec@mpl078d] Sending Ctrl-C to processes as requested
[mpiexec@mpl078d] Press Ctrl-C again to force abort
[DLIO_PROFILER ERROR]: signal caught 2
[DLIO_PROFILER ERROR]: signal caught 2
^CCtrl-C caught... cleaning up processes

with RDMAV_FORK_SAFE=1 benchmark is running without exception but no load is generated