UltraGrid Network Check

At this page you can find several tips how to check your connectivity.

PCIe utilization

The Myrinet card uses 8 lanes in PCIe. To check, whether the port it is connected to provides all 8, enter the command:

lspci -vv | fgrep -A 50 Myri | egrep '(LnkCap|LnkSta)'

The output should look like this:

 LnkCap: Port #0, Speed 2.5GT/s, Width x8, ASPM L0s, Latency L0 unlimited, L1 unlimited
 LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk- DLActive + BWMgmt- ABWMgmt-

MSI Interrupts

Check the MSI interrupts:

 lspci -vv | fgrep -A 50 Myri | egrep '(MSI)'

Look for the output

 Subsystem: MYRICOM Inc. 10G-PCIE-8A (MSI-X firmware)
 Capabilities: [44] MSI: Enable+ Count=1/1 Maskable- 64bit+

DMA troughput

Throughput of your DMA must be at least 1.3GBps (note that the output is in Bytes, not bits):

 ethtool -S eth2 | fgrep dma
 read_dma_bw_MBs: 1757
 write_dma_bw_MBs: 1736
 read_write_dma_bw_MBs: 3436

Write-combining

It is a good idea to have the write-combining set, although for fast processors it is not a critical point

How to do so you can find at this page.

Jumbo frames

Check whether the path between sender and receiver supports jumbo frames.

 ping -M do -s 8500 <target-ip>

Additional

• you may try to fiddle with flow control (myri10ge_flow_control during load of module via modprobe), althrough usually the correct setting is the default one (i.e. flow control is on)
• check whether the buffers are set correctly with sysctl. The directions to set them can be found at the appropriate setup page
• To measure the network, use Iperf 2.0.2 with patches by Andrew Gallatin. For more informations about the iperf glitches, follow this link.