NA OFI: inadvertent activation of psm3 instead of psm2

[carns]

Describe the bug

After upgrading libfabric (via external repo at https://github.com/mochi-hpc/mochi-spack-packages/tree/main/packages/libfabric) from 1.11.1 to 1.13.0, psm2 performance on the LCRC Bebop system dropped by an order of magnitude. Performance with 1.13.0 returns to normal if libfabric is compiled with an explicit --disable-psm3 argument. The libfabric package linked above now has an explicit, hardcoded --disable-psm3 configure argument for this reason. At the moment we are not running mochi code on any native psm3 environments.

To Reproduce

Reproducing with the margo-p2p-bw and margo-p2p-latency tests in https://github.com/mochi-hpc-experiments/mochi-tests/blob/main/perf-regression/bebop/margo-regression.sbatch .

The package list looks like this:

[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]

Expected behavior

I didn't expect the psm3 provider being enabled to impact psm2 performance. Are we activating the former by accident, and if so is there a way to prevent it from happening?

[carns]

@marcvef heads up