Same subnet multipathing

[PwrBank]

Hello, not directly related to the plugin, but maybe something the plugin can help setup.

Currently, when using multipathing, it is only using one available interface to do so. Is there a way to get same-subnet multipathing working easily?

Following this guide: https://www.suse.com/c/configure-multiple-iscsi-ifaces-and-use-multipathing/
(And this) https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/6/html/storage_administration_guide/iface-config-software-iscsi#iface-config-software-iscsi

I have successfully got multipath to see both interfaces, but I'm not 100% confident it is actually utilizing them. As IOPS and bandwidth both are the same before and after making the change.

I'd imagine a significant performance increase if this worked. We have a physically identical machine we are using to test ESXi vs Proxmox, and due to the way ESXi can do port binding, it's almost a 2x speed and IOPS increase over Proxmox. Identical hardware and workload tests.

[timansky]

Multipath not related to network setup. It is used for FC and iSCSI at the same time.
iscsiadm is main app to configure how it will work

iscsiadm can be configured in 2 way:

1. default route and manual network config

Example:

cat /etc/network/interfaces

auto ens1f2np2
iface ens1f2np2 inet manual

auto ens1f3np3
iface ens1f3np3 inet manual

auto bond0
iface bond0 inet manual
  bond-slaves ens1f2np2 ens1f3np3
  bond-miimon 100
  bond-mode 802.3ad
  bond-xmit-hash-policy layer3+4
  mtu 9000
  bond_lacp_rate fast
  bond_updelay 200
  bond_downdelay 200

auto bond0.1
iface bond0.1 inet manual
  mtu 9000

auto bond0.2
iface bond0.2 inet manual
  mtu 9000

auto bond0.3
iface bond0.3 inet manual
  mtu 9000

auto bond0.23
iface bond0.23 inet manual
  mtu 9000

auto vmbr1
iface vmbr10 inet static
  address xx.xx.1.1/24
  bridge-ports bond0.1
  bridge-stp off
  bridge-fd 0
  mtu 9000

auto vmbr2
iface vmbr11 inet static
  address xx.xx.2.1/24
  bridge-ports bond0.2
  bridge-stp off
  bridge-fd 0
  mtu 9000

auto vmbr3
iface vmbr12 inet static
  address xx.xx.3.1/24
  bridge-ports bond0.22
  bridge-stp off
  bridge-fd 0
  mtu 9000

auto vmbr4
iface vmbr13 inet static
  address xx.xx.4.1/24
  bridge-ports bond0.23
  bridge-stp off
  bridge-fd 0
  mtu 9000

PS vmbr can be replaced with alias or virtual interface

2. iscsiadm can handle network config by it self
   Examples from your links

Is there a way to get same-subnet multipathing

It is better to make different networks for each interface it is recommendation. There are many issues if it is one subnet. It is hard for network device to calculate this kind of traffic.

but I'm not 100% confident it is actually utilizing them

iptables -A OUTPUT -d {portal_ip} -p tcp --dport 3260 -j DROP can help with checking connectivity

I'd imagine a significant performance increase if this worked. We have a physically identical machine we are using to test ESXi vs Proxmox, and due to the way ESXi can do port binding, it's almost a 2x speed and IOPS increase over Proxmox. Identical hardware and workload tests.

1. Check session number. This parameter creates different amount of connections per interface. Recomended from documentation is 4, but it is optional and need tests.

2. It is important how iops and bandwidth are tested.
   ESXi != Qemu.
   Qemu has different mechanisms of cachenig. For example writecache can be enabled on proxmox level, on qemu level, inside qemu. Writecache depends on memory (Huge pages)
   Enable or disable discard also can increase speed and decrease consistency.
   Async IO is another parameter to play with.

PSS
i'm using fio to get more stable results:

fio -ioengine=libaio -direct=1 -sync=1 -name=test -bs=4k -iodepth=1 -rw=randwrite -runtime=600 -filename=/dev/disk/by-id/wwn-0x624a93702xxxxxxxx --size=100G

i'm always testing speed on hypervisor and inside qemu because of this layers and how disk semaphores working

PSSS
Do not forget there is also caches on pure storage side

[PwrBank]

Feeling kind of stupid here. Trying to run FIO to one of the mounted iSCSI disks for a VM, directly from Proxmox.

fio --name=read_throughput -filename=/dev/disk/by-id/wwn-0x624a937096622a35a07e4a33000125e1 --numjobs=1 --size=100G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=1M --iodepth=64 --rw=read --group_reporting=1
I'm not seeing any IO or data being written to the array, it looks like it's just writing it to the local PVE.

[timansky]

ls -l /dev/disk/by-id/wwn-0x624a937096622a35a07e4a33000125e1 must be link to dm device
or find thru multipath -ll or lsblk

if vm is off disk become unmapped. This is made for live migration and to avoid lun limit.

[PwrBank]

Okay, I found the correct dm device and ran some tests to it.

I have confirmed that both interfaces are being used after configuring the hwaddress on each iface

[Screenshot from nmon while running a fio test to the dm block device]