From fe36949cb7398f069207e438c600688789ac8feb Mon Sep 17 00:00:00 2001 From: Piotr Misiak Date: Fri, 29 Mar 2024 10:53:02 +0100 Subject: [PATCH 1/3] Update the L3 HA document in the Howtos section --- .../10.l3-high-availability/docs.en.md | 288 +++++++++++++++++- 1 file changed, 285 insertions(+), 3 deletions(-) diff --git a/user/pages/03.Howtos/10.l3-high-availability/docs.en.md b/user/pages/03.Howtos/10.l3-high-availability/docs.en.md index 5cb8eb7b..d1c428f8 100644 --- a/user/pages/03.Howtos/10.l3-high-availability/docs.en.md +++ b/user/pages/03.Howtos/10.l3-high-availability/docs.en.md @@ -1,11 +1,12 @@ --- title: 'Building Layer 3 High Availability' published: true -date: '16-09-2018 16:20' +date: '29-03-2024 10:20' taxonomy: category: - docs --- +! This solution is deprecated, we highly recommend to use Octavia LBaaS for this purpose: [Create a Load Balancer](../../02.Tutorials/05.lbaas/docs.en.md) ## Problem statement @@ -20,13 +21,21 @@ taxonomy: * `keepalived` is an open source VRRP implementation -## Considerations +## Legacy vs New regions + +Our legacy regions (cbk, dbl) have a different SDN backed than our new regions (fes, dus2). +Each SDN requires a dedicated `keepalived` configuration - both are presented below. + +[ui-tabs position="top-left" active="0" theme="lite"] +[ui-tab title="Legacy regions (cbk, dbl)"] + +### Considerations * VRRP must discover routers (in our case - application servers) in unicast mode, since broadcast is not available in cloud networking. * Virtual IP address must resolve to virtual MAC (00:00:5E:00:01:XX) address, so there is no need in Gratuitous ARP, which does not always work correctly with our SDN controller. * Instances, that share virtual IP address, must be connected to the same OpenStack subnet and must have same subnet and mask configuration. -## Configuration +### Configuration Let's say we have three instances running: @@ -288,6 +297,279 @@ There is confirmation in log: Aug 07 09:06:18 ha-second Keepalived_vrrp[8665]: VRRP_Instance(vrrp_1) Transition to MASTER STATE Aug 07 09:06:19 ha-second Keepalived_vrrp[8665]: VRRP_Instance(vrrp_1) Entering MASTER STATE ``` +[/ui-tab] +[ui-tab title="New regions (fes, dus2)"] + +### Considerations + +* VRRP must discover routers (in our case - application servers) in unicast mode, since broadcast is not available in cloud networking. +* Virtual IP address must resolve to MASTER node interface's MAC address, so there is a need to generate Gratuitous ARP packets every few seconds to update ARP tables. +* Instances, that share virtual IP address, must be connected to the same OpenStack subnet and must have same subnet and mask configuration. + +### Configuration + +Let's say we have three instances running: + +```shell ++--------------------------------------+-----------+--------+-------------------------------------+--------+---------+ +| ID | Name | Status | Networks | Image | Flavor | ++--------------------------------------+-----------+--------+-------------------------------------+--------+---------+ +| 1f9badd7-13a2-4553-86d1-3aae10ca29f9 | observer | ACTIVE | ha_lab=10.200.51.34, 185.56.135.125 | bionic | m1.tiny | +| da2c4c9c-2581-474e-8caf-abc443a1d29e | ha_second | ACTIVE | ha_lab=10.200.51.33, 185.56.135.122 | bionic | m1.tiny | +| 685b6cdc-ff7f-4d82-8993-a8ca1ebf95b1 | ha_first | ACTIVE | ha_lab=10.200.51.32, 185.56.135.121 | bionic | m1.tiny | ++--------------------------------------+-----------+--------+-------------------------------------+--------+---------+ +``` + +Two of them, `ha_first` and `ha_second` will share single virtual IP address from 10.200.51.0/24 between them. +The third instance, `observer`, will be used by us to check connectivity to virtual IP. + +`ha_lab` network ports list: + +```shell +$ openstack port list --network ha_lab ++--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+ +| ID | Name | MAC Address | Fixed IP Addresses | Status | ++--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+ +| db8fa14d-5800-4db5-8ad7-fe9ad1eaef37 | | fa:16:3e:08:e6:15 | ip_address='10.200.51.34', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE | +| 913c4131-0cc3-4df4-9f7e-b4cf4ae885ca | | fa:16:3e:8d:dc:bb | ip_address='10.200.51.32', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE | +| a8389d5e-6bd0-4368-a6fb-249127f9cdca | | fa:16:3e:c9:e0:b1 | ip_address='10.200.51.1', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE | +| e220e2d9-f4de-4941-92e0-640ee352aa4f | | fa:16:3e:f3:a9:a3 | ip_address='10.200.51.33', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE | ++--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+ +``` + +Now we need to create a network port reserved for the virtual address. +For this example, we will stick to `10.200.51.10` and we use virtual MAC address 00:00:5e:00:01:01, which according to RFC is the virtual mac address assigned to VRRP virtual router 1. + +```shell +openstack port create --network ha_lab --fixed-ip subnet=9a027a94-e688-45cb-a653-e223ccc9f72f,ip-address=10.200.51.10 --mac-address 00:00:5e:00:01:01 --security-group --security-group vip +``` + +`ha_lab` network ports list with new 'vip' port: + +```shell +$ openstack port list --network ha_lab ++--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+ +| ID | Name | MAC Address | Fixed IP Addresses | Status | ++--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+ +| db8fa14d-5800-4db5-8ad7-fe9ad1eaef37 | | fa:16:3e:08:e6:15 | ip_address='10.200.51.34', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE | +| 913c4131-0cc3-4df4-9f7e-b4cf4ae885ca | | fa:16:3e:8d:dc:bb | ip_address='10.200.51.32', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE | +| a8389d5e-6bd0-4368-a6fb-249127f9cdca | | fa:16:3e:c9:e0:b1 | ip_address='10.200.51.1', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE | +| d320ed31-afae-334f-fe10-522ac148bb1a | vip | 00:00:5e:00:01:01 | ip_address='10.200.51.10', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE | +| e220e2d9-f4de-4941-92e0-640ee352aa4f | | fa:16:3e:f3:a9:a3 | ip_address='10.200.51.33', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE | ++--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+ +``` + +Let's install and configure keepalived on `ha_first` and `ha_second` VMs: + +```shell +ubuntu@ha-first:~$ sudo apt install keepalived +``` + +Now, let's create the keepalived configuration. + +`/etc/keepalived/keepalived.conf` on `ha_first`: + +```shell +vrrp_instance vrrp_1 { + debug 2 + interface ens3 + state MASTER + virtual_router_id 1 + garp_master_refresh 5 + garp_master_refresh_repeat 2 + priority 200 + unicast_src_ip 10.200.51.32 dev ens3 + unicast_peer { + 10.200.51.33 dev ens3 + } + virtual_ipaddress { + 10.200.51.10 + } +} +``` + +`/etc/keepalived/keepalived.conf` on `ha_second`: + +```shell +vrrp_instance vrrp_1 { + debug 2 + interface ens3 + state MASTER + virtual_router_id 1 + garp_master_refresh 5 + garp_master_refresh_repeat 2 + priority 100 + unicast_src_ip 10.200.51.33 dev ens3 + unicast_peer { + 10.200.51.32 dev ens3 + } + virtual_ipaddress { + 10.200.51.10 + } +} +``` + +Some of these options need special attention in cloud environments: + +* `garp_master_refresh 5` - forces MASTER VRRP node to generate GARP packets every 5 seconds to update ARP tables in SDN and on all VMs connected to the network. +* `garp_master_refresh_repeat 2` - secifies how many GARP packets will be generated at GARP refresh event. +* `unicast_src_ip` and `unicast_peer` - since the cloud does not support broadcast, all communication must be unicast. + +Now, let's start keepalived and ensure that instances recognize each other, and state of `ha_second` is transitioned to +BACKUP. Let's check some logs for that. + +`journalctl -u keepalived` on `ha_first`: + +```shell +Aug 07 08:46:33 ha-first systemd[1]: Starting Keepalive Daemon (LVS and VRRP)... +Aug 07 08:46:33 ha-first Keepalived[8765]: Starting Keepalived v1.3.9 (10/21,2017) +Aug 07 08:46:33 ha-first Keepalived[8765]: Opening file '/etc/keepalived/keepalived.conf'. +Aug 07 08:46:33 ha-first systemd[1]: Started Keepalive Daemon (LVS and VRRP). +Aug 07 08:46:33 ha-first Keepalived[8777]: Starting Healthcheck child process, pid=8779 +Aug 07 08:46:33 ha-first Keepalived[8777]: Starting VRRP child process, pid=8780 +Aug 07 08:46:33 ha-first Keepalived_healthcheckers[8779]: Opening file '/etc/keepalived/keepalived.conf'. +Aug 07 08:46:33 ha-first Keepalived_vrrp[8780]: Registering Kernel netlink reflector +Aug 07 08:46:33 ha-first Keepalived_vrrp[8780]: Registering Kernel netlink command channel +Aug 07 08:46:33 ha-first Keepalived_vrrp[8780]: Registering gratuitous ARP shared channel +Aug 07 08:46:33 ha-first Keepalived_vrrp[8780]: Opening file '/etc/keepalived/keepalived.conf'. +Aug 07 08:46:33 ha-first Keepalived_vrrp[8780]: NOTICE: setting sysctl net.ipv4.conf.all.rp_filter from 1 to 0 +Aug 07 08:46:33 ha-first Keepalived_vrrp[8780]: Using LinkWatch kernel netlink reflector... +Aug 07 08:46:34 ha-first Keepalived_vrrp[8780]: VRRP_Instance(vrrp_1) Transition to MASTER STATE +Aug 07 08:46:35 ha-first Keepalived_vrrp[8780]: VRRP_Instance(vrrp_1) Entering MASTER STATE +Aug 07 08:46:38 ha-first Keepalived_vrrp[8780]: VRRP_Instance(vrrp_1) Received advert with lower priority 100, ours 200, forcing new election +``` + +`journalctl -u keepalived` on `ha_second`: + +```shell +Aug 07 08:46:38 ha-second systemd[1]: Starting Keepalive Daemon (LVS and VRRP)... +Aug 07 08:46:38 ha-second Keepalived[8647]: Starting Keepalived v1.3.9 (10/21,2017) +Aug 07 08:46:38 ha-second Keepalived[8647]: Opening file '/etc/keepalived/keepalived.conf'. +Aug 07 08:46:38 ha-second systemd[1]: Started Keepalive Daemon (LVS and VRRP). +Aug 07 08:46:38 ha-second Keepalived[8662]: Starting Healthcheck child process, pid=8664 +Aug 07 08:46:38 ha-second Keepalived[8662]: Starting VRRP child process, pid=8665 +Aug 07 08:46:38 ha-second Keepalived_healthcheckers[8664]: Opening file '/etc/keepalived/keepalived.conf'. +Aug 07 08:46:38 ha-second Keepalived_vrrp[8665]: Registering Kernel netlink reflector +Aug 07 08:46:38 ha-second Keepalived_vrrp[8665]: Registering Kernel netlink command channel +Aug 07 08:46:38 ha-second Keepalived_vrrp[8665]: Registering gratuitous ARP shared channel +Aug 07 08:46:38 ha-second Keepalived_vrrp[8665]: Opening file '/etc/keepalived/keepalived.conf'. +Aug 07 08:46:38 ha-second Keepalived_vrrp[8665]: NOTICE: setting sysctl net.ipv4.conf.all.rp_filter from 1 to 0 +Aug 07 08:46:38 ha-second Keepalived_vrrp[8665]: Using LinkWatch kernel netlink reflector... +Aug 07 08:46:38 ha-second Keepalived_vrrp[8665]: VRRP_Instance(vrrp_1) Transition to MASTER STATE +Aug 07 08:46:38 ha-second Keepalived_vrrp[8665]: VRRP_Instance(vrrp_1) Received advert with higher priority 200, ours 100 +Aug 07 08:46:38 ha-second Keepalived_vrrp[8665]: VRRP_Instance(vrrp_1) Entering BACKUP STATE +``` + +Looks like everything is correct. We can confirm that by seeing the virtual IP address on `ha_first` instance: + +```shell +ubuntu@ha-first:~$ ip -4 a +1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 + inet 127.0.0.1/8 scope host lo + valid_lft forever preferred_lft forever +2: ens3: mtu 576 qdisc fq_codel state UP group default qlen 1000 + inet 10.200.51.32/24 brd 10.200.51.255 scope global dynamic ens3 + valid_lft 86037sec preferred_lft 86037sec + inet 10.200.51.10/32 scope global ens3 + valid_lft forever preferred_lft forever +ubuntu@ha-first:~$ +``` + +Let's check if we can reach this IP address from our observer instance. + +```shell +ubuntu@observer:~$ ping 10.200.51.10 -c 3 +PING 10.200.51.10 (10.200.51.10) 56(84) bytes of data. +From 10.200.51.34 icmp_seq=1 Destination Host Unreachable +From 10.200.51.34 icmp_seq=2 Destination Host Unreachable +From 10.200.51.34 icmp_seq=3 Destination Host Unreachable + +--- 10.200.51.10 ping statistics --- +3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2043ms +pipe 3 + +ubuntu@observer:~$ ip neigh +10.200.51.10 dev ens3 FAILED +10.200.51.1 dev ens3 lladdr fa:16:3e:c9:e0:b1 REACHABLE +ubuntu@observer:~$ +``` + +The reason why we can't ping or even arp a virtual IP address, is neutron port security. We need to allow this IP address +on ports, that connect instances `ha_first` and `ha_second` to the network 10.200.51.0/24. + +First, let's find port IDs: + +```shell +$ openstack port list --network ha_lab ++--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+ +| ID | Name | MAC Address | Fixed IP Addresses | Status | ++--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+ +| db8fa14d-5800-4db5-8ad7-fe9ad1eaef37 | | fa:16:3e:08:e6:15 | ip_address='10.200.51.34', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE | +| 913c4131-0cc3-4df4-9f7e-b4cf4ae885ca | | fa:16:3e:8d:dc:bb | ip_address='10.200.51.32', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE | +| a8389d5e-6bd0-4368-a6fb-249127f9cdca | | fa:16:3e:c9:e0:b1 | ip_address='10.200.51.1', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE | +| d320ed31-afae-334f-fe10-522ac148bb1a | vip | 00:00:5e:00:01:01 | ip_address='10.200.51.10', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE | +| e220e2d9-f4de-4941-92e0-640ee352aa4f | | fa:16:3e:f3:a9:a3 | ip_address='10.200.51.33', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE | ++--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+ +``` + +Second and fifth lines are ports, connecting `ha_first` and `ha_second` accordingly. Now, we must allow `10.200.51.10` on these ports: + +```shell +$ openstack port set --allowed-address ip-address=10.200.51.10 913c4131-0cc3-4df4-9f7e-b4cf4ae885ca +(no output) +$ openstack port set --allowed-address ip-address=10.200.51.10 e220e2d9-f4de-4941-92e0-640ee352aa4f +``` + +Let's check connectivity again: + +```shell +ubuntu@observer:~$ ping 10.200.51.10 -c 3 +PING 10.200.51.10 (10.200.51.10) 56(84) bytes of data. +64 bytes from 10.200.51.10: icmp_seq=1 ttl=64 time=3.24 ms +64 bytes from 10.200.51.10: icmp_seq=2 ttl=64 time=1.17 ms +64 bytes from 10.200.51.10: icmp_seq=3 ttl=64 time=1.03 ms + +--- 10.200.51.10 ping statistics --- +3 packets transmitted, 3 received, 0% packet loss, time 2003ms +rtt min/avg/max/mdev = 1.035/1.818/3.243/1.009 ms + +ubuntu@observer:~$ ip neigh +10.200.51.10 dev ens3 lladdr fa:16:3e:8d:dc:bb REACHABLE +10.200.51.1 dev ens3 lladdr fa:16:3e:c9:e0:b1 REACHABLE +ubuntu@observer:~$ +``` + +We can now successfully reach the virtual IP address. It has the MAC address fa:16:3e:8d:dc:bb, which is `ha_first` ens3 interface MAC address. + +Let's now bring down keepalived process on `ha_first` and check if virtual IP address will now be served by `ha_second`: + +```shell +ubuntu@ha-first:~$ sudo systemctl stop keepalived.service +``` + +Now we can see that instance `ha_second` took the IP address: + +```shell +ubuntu@ha-second:~$ ip -4 a +1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 + inet 127.0.0.1/8 scope host lo + valid_lft forever preferred_lft forever +2: ens3: mtu 576 qdisc fq_codel state UP group default qlen 1000 + inet 10.200.51.33/24 brd 10.200.51.255 scope global dynamic ens3 + valid_lft 85234sec preferred_lft 85234sec + inet 10.200.51.10/32 scope global ens3 + valid_lft forever preferred_lft forever +``` + +There is confirmation in log: + +```shell +Aug 07 09:06:18 ha-second Keepalived_vrrp[8665]: VRRP_Instance(vrrp_1) Transition to MASTER STATE +Aug 07 09:06:19 ha-second Keepalived_vrrp[8665]: VRRP_Instance(vrrp_1) Entering MASTER STATE +``` + +[/ui-tab] +[/ui-tabs] ## Conclusion From 38116c1e1675d9802523052d97f1d465bc0847c8 Mon Sep 17 00:00:00 2001 From: Piotr Misiak Date: Fri, 29 Mar 2024 11:03:10 +0100 Subject: [PATCH 2/3] Syntax fix in the L3 HA document --- user/pages/03.Howtos/10.l3-high-availability/docs.en.md | 1 + 1 file changed, 1 insertion(+) diff --git a/user/pages/03.Howtos/10.l3-high-availability/docs.en.md b/user/pages/03.Howtos/10.l3-high-availability/docs.en.md index d1c428f8..e55e5864 100644 --- a/user/pages/03.Howtos/10.l3-high-availability/docs.en.md +++ b/user/pages/03.Howtos/10.l3-high-availability/docs.en.md @@ -297,6 +297,7 @@ There is confirmation in log: Aug 07 09:06:18 ha-second Keepalived_vrrp[8665]: VRRP_Instance(vrrp_1) Transition to MASTER STATE Aug 07 09:06:19 ha-second Keepalived_vrrp[8665]: VRRP_Instance(vrrp_1) Entering MASTER STATE ``` + [/ui-tab] [ui-tab title="New regions (fes, dus2)"] From 3596fc11505c8e28e6df198908e64ff694794513 Mon Sep 17 00:00:00 2001 From: Piotr Misiak Date: Wed, 3 Apr 2024 14:52:49 +0200 Subject: [PATCH 3/3] Change regions naming wording in the l3-high-availability doc --- .../pages/03.Howtos/10.l3-high-availability/docs.en.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/user/pages/03.Howtos/10.l3-high-availability/docs.en.md b/user/pages/03.Howtos/10.l3-high-availability/docs.en.md index e55e5864..c11b7382 100644 --- a/user/pages/03.Howtos/10.l3-high-availability/docs.en.md +++ b/user/pages/03.Howtos/10.l3-high-availability/docs.en.md @@ -6,7 +6,7 @@ taxonomy: category: - docs --- -! This solution is deprecated, we highly recommend to use Octavia LBaaS for this purpose: [Create a Load Balancer](../../02.Tutorials/05.lbaas/docs.en.md) +! Please consider using Octavia LBaaS solution provided by the SysEleven Stack before you decide to create a VIP VRRP cluster on your own: [Create a Load Balancer](../../02.Tutorials/05.lbaas/docs.en.md) ## Problem statement @@ -21,13 +21,13 @@ taxonomy: * `keepalived` is an open source VRRP implementation -## Legacy vs New regions +## Cloud regions specifics -Our legacy regions (cbk, dbl) have a different SDN backed than our new regions (fes, dus2). +CBK and DBL cloud regions have a different SDN backend than FES region. Each SDN requires a dedicated `keepalived` configuration - both are presented below. [ui-tabs position="top-left" active="0" theme="lite"] -[ui-tab title="Legacy regions (cbk, dbl)"] +[ui-tab title="CBK and DBL regions"] ### Considerations @@ -299,7 +299,7 @@ Aug 07 09:06:19 ha-second Keepalived_vrrp[8665]: VRRP_Instance(vrrp_1) Entering ``` [/ui-tab] -[ui-tab title="New regions (fes, dus2)"] +[ui-tab title="FES region"] ### Considerations