RDMA stability

[maximilianheer]

Description
Our RDMA-stack is not yet 100% stable, especially when transferring huge files from one node to another (batched transmissions as in throughput-tests). However, it's very hard rn to pinpoint a specific issue that one could trigger on during testing and development. Thus, we consider the current version of the stack with the fixed retrans-issue to be "good enough" for all intents and purposes. If there's some time and more insight in specific problems in the future, I'll return to debugging these.

Steps to reproduce
Build the rdma-example and benchmark for throughput (batched transmissions).

Expected behavior
If instability occurs, you will see that the RDMA-example stalls at some point. Printouts from the client-side for both sending messages and receiving completions for local WRITES can help to further determine where such a stall takes place.

Screenshots & Logs
Nah, nothing available yet.

Environment:

• Tested on the hacc (u55c-nodes)

Additional information
Procedure discussed with @bo3z