Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update the L3 HA document in the Howtos section #273

Merged
merged 3 commits into from
Apr 12, 2024
Merged

Update the L3 HA document in the Howtos section #273

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
fe36949
Update the L3 HA document in the Howtos section
Mar 29, 2024
38116c1
Syntax fix in the L3 HA document
Mar 29, 2024
3596fc1
Change regions naming wording in the l3-high-availability doc
Apr 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
289 changes: 286 additions & 3 deletions user/pages/03.Howtos/10.l3-high-availability/docs.en.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
---
title: 'Building Layer 3 High Availability'
published: true
date: '16-09-2018 16:20'
date: '29-03-2024 10:20'
taxonomy:
category:
- docs
---
! Please consider using Octavia LBaaS solution provided by the SysEleven Stack before you decide to create a VIP VRRP cluster on your own: [Create a Load Balancer](../../02.Tutorials/05.lbaas/docs.en.md)

## Problem statement

Expand All @@ -20,13 +21,21 @@ taxonomy:

* `keepalived` is an open source VRRP implementation

## Considerations
## Cloud regions specifics

CBK and DBL cloud regions have a different SDN backend than FES region.
Each SDN requires a dedicated `keepalived` configuration - both are presented below.

[ui-tabs position="top-left" active="0" theme="lite"]
[ui-tab title="CBK and DBL regions"]

### Considerations

* VRRP must discover routers (in our case - application servers) in unicast mode, since broadcast is not available in cloud networking.
* Virtual IP address must resolve to virtual MAC (00:00:5E:00:01:XX) address, so there is no need in Gratuitous ARP, which does not always work correctly with our SDN controller.
* Instances, that share virtual IP address, must be connected to the same OpenStack subnet and must have same subnet and mask configuration.

## Configuration
### Configuration

Let's say we have three instances running:

Expand Down Expand Up @@ -289,6 +298,280 @@ Aug 07 09:06:18 ha-second Keepalived_vrrp[8665]: VRRP_Instance(vrrp_1) Transitio
Aug 07 09:06:19 ha-second Keepalived_vrrp[8665]: VRRP_Instance(vrrp_1) Entering MASTER STATE
```

[/ui-tab]
[ui-tab title="FES region"]

### Considerations

* VRRP must discover routers (in our case - application servers) in unicast mode, since broadcast is not available in cloud networking.
* Virtual IP address must resolve to MASTER node interface's MAC address, so there is a need to generate Gratuitous ARP packets every few seconds to update ARP tables.
* Instances, that share virtual IP address, must be connected to the same OpenStack subnet and must have same subnet and mask configuration.

### Configuration

Let's say we have three instances running:

```shell
+--------------------------------------+-----------+--------+-------------------------------------+--------+---------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-----------+--------+-------------------------------------+--------+---------+
| 1f9badd7-13a2-4553-86d1-3aae10ca29f9 | observer | ACTIVE | ha_lab=10.200.51.34, 185.56.135.125 | bionic | m1.tiny |
| da2c4c9c-2581-474e-8caf-abc443a1d29e | ha_second | ACTIVE | ha_lab=10.200.51.33, 185.56.135.122 | bionic | m1.tiny |
| 685b6cdc-ff7f-4d82-8993-a8ca1ebf95b1 | ha_first | ACTIVE | ha_lab=10.200.51.32, 185.56.135.121 | bionic | m1.tiny |
+--------------------------------------+-----------+--------+-------------------------------------+--------+---------+
```

Two of them, `ha_first` and `ha_second` will share single virtual IP address from 10.200.51.0/24 between them.
The third instance, `observer`, will be used by us to check connectivity to virtual IP.

`ha_lab` network ports list:

```shell
$ openstack port list --network ha_lab
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+
| ID | Name | MAC Address | Fixed IP Addresses | Status |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+
| db8fa14d-5800-4db5-8ad7-fe9ad1eaef37 | | fa:16:3e:08:e6:15 | ip_address='10.200.51.34', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE |
| 913c4131-0cc3-4df4-9f7e-b4cf4ae885ca | | fa:16:3e:8d:dc:bb | ip_address='10.200.51.32', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE |
| a8389d5e-6bd0-4368-a6fb-249127f9cdca | | fa:16:3e:c9:e0:b1 | ip_address='10.200.51.1', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE |
| e220e2d9-f4de-4941-92e0-640ee352aa4f | | fa:16:3e:f3:a9:a3 | ip_address='10.200.51.33', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+
```

Now we need to create a network port reserved for the virtual address.
For this example, we will stick to `10.200.51.10` and we use virtual MAC address 00:00:5e:00:01:01, which according to RFC is the virtual mac address assigned to VRRP virtual router 1.

```shell
openstack port create --network ha_lab --fixed-ip subnet=9a027a94-e688-45cb-a653-e223ccc9f72f,ip-address=10.200.51.10 --mac-address 00:00:5e:00:01:01 --security-group <sec_group_1> --security-group <sec_group_2> vip
```

`ha_lab` network ports list with new 'vip' port:

```shell
$ openstack port list --network ha_lab
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+
| ID | Name | MAC Address | Fixed IP Addresses | Status |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+
| db8fa14d-5800-4db5-8ad7-fe9ad1eaef37 | | fa:16:3e:08:e6:15 | ip_address='10.200.51.34', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE |
| 913c4131-0cc3-4df4-9f7e-b4cf4ae885ca | | fa:16:3e:8d:dc:bb | ip_address='10.200.51.32', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE |
| a8389d5e-6bd0-4368-a6fb-249127f9cdca | | fa:16:3e:c9:e0:b1 | ip_address='10.200.51.1', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE |
| d320ed31-afae-334f-fe10-522ac148bb1a | vip | 00:00:5e:00:01:01 | ip_address='10.200.51.10', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE |
| e220e2d9-f4de-4941-92e0-640ee352aa4f | | fa:16:3e:f3:a9:a3 | ip_address='10.200.51.33', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+
```

Let's install and configure keepalived on `ha_first` and `ha_second` VMs:

```shell
ubuntu@ha-first:~$ sudo apt install keepalived
```

Now, let's create the keepalived configuration.

`/etc/keepalived/keepalived.conf` on `ha_first`:

```shell
vrrp_instance vrrp_1 {
debug 2
interface ens3
state MASTER
virtual_router_id 1
garp_master_refresh 5
garp_master_refresh_repeat 2
priority 200
unicast_src_ip 10.200.51.32 dev ens3
unicast_peer {
10.200.51.33 dev ens3
}
virtual_ipaddress {
10.200.51.10
}
}
```

`/etc/keepalived/keepalived.conf` on `ha_second`:

```shell
vrrp_instance vrrp_1 {
debug 2
interface ens3
state MASTER
virtual_router_id 1
garp_master_refresh 5
garp_master_refresh_repeat 2
priority 100
unicast_src_ip 10.200.51.33 dev ens3
unicast_peer {
10.200.51.32 dev ens3
}
virtual_ipaddress {
10.200.51.10
}
}
```

Some of these options need special attention in cloud environments:

* `garp_master_refresh 5` - forces MASTER VRRP node to generate GARP packets every 5 seconds to update ARP tables in SDN and on all VMs connected to the network.
benj200 marked this conversation as resolved.
Show resolved Hide resolved
* `garp_master_refresh_repeat 2` - secifies how many GARP packets will be generated at GARP refresh event.
* `unicast_src_ip` and `unicast_peer` - since the cloud does not support broadcast, all communication must be unicast.

Now, let's start keepalived and ensure that instances recognize each other, and state of `ha_second` is transitioned to
BACKUP. Let's check some logs for that.

`journalctl -u keepalived` on `ha_first`:

```shell
Aug 07 08:46:33 ha-first systemd[1]: Starting Keepalive Daemon (LVS and VRRP)...
Aug 07 08:46:33 ha-first Keepalived[8765]: Starting Keepalived v1.3.9 (10/21,2017)
Aug 07 08:46:33 ha-first Keepalived[8765]: Opening file '/etc/keepalived/keepalived.conf'.
Aug 07 08:46:33 ha-first systemd[1]: Started Keepalive Daemon (LVS and VRRP).
Aug 07 08:46:33 ha-first Keepalived[8777]: Starting Healthcheck child process, pid=8779
Aug 07 08:46:33 ha-first Keepalived[8777]: Starting VRRP child process, pid=8780
Aug 07 08:46:33 ha-first Keepalived_healthcheckers[8779]: Opening file '/etc/keepalived/keepalived.conf'.
Aug 07 08:46:33 ha-first Keepalived_vrrp[8780]: Registering Kernel netlink reflector
Aug 07 08:46:33 ha-first Keepalived_vrrp[8780]: Registering Kernel netlink command channel
Aug 07 08:46:33 ha-first Keepalived_vrrp[8780]: Registering gratuitous ARP shared channel
Aug 07 08:46:33 ha-first Keepalived_vrrp[8780]: Opening file '/etc/keepalived/keepalived.conf'.
Aug 07 08:46:33 ha-first Keepalived_vrrp[8780]: NOTICE: setting sysctl net.ipv4.conf.all.rp_filter from 1 to 0
Aug 07 08:46:33 ha-first Keepalived_vrrp[8780]: Using LinkWatch kernel netlink reflector...
Aug 07 08:46:34 ha-first Keepalived_vrrp[8780]: VRRP_Instance(vrrp_1) Transition to MASTER STATE
Aug 07 08:46:35 ha-first Keepalived_vrrp[8780]: VRRP_Instance(vrrp_1) Entering MASTER STATE
Aug 07 08:46:38 ha-first Keepalived_vrrp[8780]: VRRP_Instance(vrrp_1) Received advert with lower priority 100, ours 200, forcing new election
```

`journalctl -u keepalived` on `ha_second`:

```shell
Aug 07 08:46:38 ha-second systemd[1]: Starting Keepalive Daemon (LVS and VRRP)...
Aug 07 08:46:38 ha-second Keepalived[8647]: Starting Keepalived v1.3.9 (10/21,2017)
Aug 07 08:46:38 ha-second Keepalived[8647]: Opening file '/etc/keepalived/keepalived.conf'.
Aug 07 08:46:38 ha-second systemd[1]: Started Keepalive Daemon (LVS and VRRP).
Aug 07 08:46:38 ha-second Keepalived[8662]: Starting Healthcheck child process, pid=8664
Aug 07 08:46:38 ha-second Keepalived[8662]: Starting VRRP child process, pid=8665
Aug 07 08:46:38 ha-second Keepalived_healthcheckers[8664]: Opening file '/etc/keepalived/keepalived.conf'.
Aug 07 08:46:38 ha-second Keepalived_vrrp[8665]: Registering Kernel netlink reflector
Aug 07 08:46:38 ha-second Keepalived_vrrp[8665]: Registering Kernel netlink command channel
Aug 07 08:46:38 ha-second Keepalived_vrrp[8665]: Registering gratuitous ARP shared channel
Aug 07 08:46:38 ha-second Keepalived_vrrp[8665]: Opening file '/etc/keepalived/keepalived.conf'.
Aug 07 08:46:38 ha-second Keepalived_vrrp[8665]: NOTICE: setting sysctl net.ipv4.conf.all.rp_filter from 1 to 0
Aug 07 08:46:38 ha-second Keepalived_vrrp[8665]: Using LinkWatch kernel netlink reflector...
Aug 07 08:46:38 ha-second Keepalived_vrrp[8665]: VRRP_Instance(vrrp_1) Transition to MASTER STATE
Aug 07 08:46:38 ha-second Keepalived_vrrp[8665]: VRRP_Instance(vrrp_1) Received advert with higher priority 200, ours 100
Aug 07 08:46:38 ha-second Keepalived_vrrp[8665]: VRRP_Instance(vrrp_1) Entering BACKUP STATE
```

Looks like everything is correct. We can confirm that by seeing the virtual IP address on `ha_first` instance:

```shell
ubuntu@ha-first:~$ ip -4 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 576 qdisc fq_codel state UP group default qlen 1000
inet 10.200.51.32/24 brd 10.200.51.255 scope global dynamic ens3
valid_lft 86037sec preferred_lft 86037sec
inet 10.200.51.10/32 scope global ens3
valid_lft forever preferred_lft forever
ubuntu@ha-first:~$
```

Let's check if we can reach this IP address from our observer instance.

```shell
ubuntu@observer:~$ ping 10.200.51.10 -c 3
PING 10.200.51.10 (10.200.51.10) 56(84) bytes of data.
From 10.200.51.34 icmp_seq=1 Destination Host Unreachable
From 10.200.51.34 icmp_seq=2 Destination Host Unreachable
From 10.200.51.34 icmp_seq=3 Destination Host Unreachable

--- 10.200.51.10 ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2043ms
pipe 3

ubuntu@observer:~$ ip neigh
10.200.51.10 dev ens3 FAILED
10.200.51.1 dev ens3 lladdr fa:16:3e:c9:e0:b1 REACHABLE
ubuntu@observer:~$
```

The reason why we can't ping or even arp a virtual IP address, is neutron port security. We need to allow this IP address
on ports, that connect instances `ha_first` and `ha_second` to the network 10.200.51.0/24.

First, let's find port IDs:

```shell
$ openstack port list --network ha_lab
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+
| ID | Name | MAC Address | Fixed IP Addresses | Status |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+
| db8fa14d-5800-4db5-8ad7-fe9ad1eaef37 | | fa:16:3e:08:e6:15 | ip_address='10.200.51.34', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE |
| 913c4131-0cc3-4df4-9f7e-b4cf4ae885ca | | fa:16:3e:8d:dc:bb | ip_address='10.200.51.32', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE |
| a8389d5e-6bd0-4368-a6fb-249127f9cdca | | fa:16:3e:c9:e0:b1 | ip_address='10.200.51.1', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE |
| d320ed31-afae-334f-fe10-522ac148bb1a | vip | 00:00:5e:00:01:01 | ip_address='10.200.51.10', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE |
| e220e2d9-f4de-4941-92e0-640ee352aa4f | | fa:16:3e:f3:a9:a3 | ip_address='10.200.51.33', subnet_id='9a027a94-e688-45cb-a653-e223ccc9f72f' | ACTIVE |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+
```

Second and fifth lines are ports, connecting `ha_first` and `ha_second` accordingly. Now, we must allow `10.200.51.10` on these ports:

```shell
$ openstack port set --allowed-address ip-address=10.200.51.10 913c4131-0cc3-4df4-9f7e-b4cf4ae885ca
(no output)
$ openstack port set --allowed-address ip-address=10.200.51.10 e220e2d9-f4de-4941-92e0-640ee352aa4f
```

Let's check connectivity again:

```shell
ubuntu@observer:~$ ping 10.200.51.10 -c 3
PING 10.200.51.10 (10.200.51.10) 56(84) bytes of data.
64 bytes from 10.200.51.10: icmp_seq=1 ttl=64 time=3.24 ms
64 bytes from 10.200.51.10: icmp_seq=2 ttl=64 time=1.17 ms
64 bytes from 10.200.51.10: icmp_seq=3 ttl=64 time=1.03 ms

--- 10.200.51.10 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 1.035/1.818/3.243/1.009 ms

ubuntu@observer:~$ ip neigh
10.200.51.10 dev ens3 lladdr fa:16:3e:8d:dc:bb REACHABLE
10.200.51.1 dev ens3 lladdr fa:16:3e:c9:e0:b1 REACHABLE
ubuntu@observer:~$
```

We can now successfully reach the virtual IP address. It has the MAC address fa:16:3e:8d:dc:bb, which is `ha_first` ens3 interface MAC address.

Let's now bring down keepalived process on `ha_first` and check if virtual IP address will now be served by `ha_second`:

```shell
ubuntu@ha-first:~$ sudo systemctl stop keepalived.service
```

Now we can see that instance `ha_second` took the IP address:

```shell
ubuntu@ha-second:~$ ip -4 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 576 qdisc fq_codel state UP group default qlen 1000
inet 10.200.51.33/24 brd 10.200.51.255 scope global dynamic ens3
valid_lft 85234sec preferred_lft 85234sec
inet 10.200.51.10/32 scope global ens3
valid_lft forever preferred_lft forever
```

There is confirmation in log:

```shell
Aug 07 09:06:18 ha-second Keepalived_vrrp[8665]: VRRP_Instance(vrrp_1) Transition to MASTER STATE
Aug 07 09:06:19 ha-second Keepalived_vrrp[8665]: VRRP_Instance(vrrp_1) Entering MASTER STATE
```

[/ui-tab]
[/ui-tabs]

## Conclusion

With this configuration, it takes approximately 2 seconds to discover failure and update MAC forwarding tables.
Expand Down