feat: ha_cluster_node_options allows per-node addresses and SBD options to be set

.gitlab-ci.yml

@@ -59,8 +59,12 @@ variables:
   - qnetd_and_cluster
   - qnetd_disabled
   - qnetd
-  - sbd_all_options
-  - sbd_check_devices_count
+  - sbd_all_options_combined
+  - sbd_all_options_inventory
+  - sbd_all_options_play
+  - sbd_check_devices_count_combined
+  - sbd_check_devices_count_inventory
+  - sbd_check_devices_count_play
   - sbd_defaults_disabled
   - sbd_defaults
   - sbd_delay_start

README.md

@@ -484,26 +484,15 @@ options are:
 
 You may take a look at [an example](#configuring-cluster-to-use-sbd).
 
-Watchdog and SBD devices are configured on a node to node basis in
-[inventory](#sbd-watchdog-and-devices).
+Watchdog and SBD devices can be configured on a node to node basis in two
+variables:
 
-#### `ha_cluster_cluster_properties`
-
-structure, default: no properties
-
-```yaml
-ha_cluster_cluster_properties:
-  - attrs:
-      - name: property1_name
-        value: property1_value
-      - name: property2_name
-        value: property2_value
-```
-
-List of sets of cluster properties - Pacemaker cluster-wide configuration.
-Currently, only one set is supported.
-
-You may take a look at [an example](#configuring-cluster-properties).
+* [`ha_cluster_node_options`](#ha_cluster_node_options) is a single variable
+  expected to have the same value for all cluster nodes. It is a list of
+  dictionaries, each dictionary defines options for one node.
+* [`ha_cluster`](#sbd-watchdog-and-devices) dictionary defines options for one
+  node only. To set different values for each node, you define the variable
+  separately for each node.
 
 #### `ha_cluster_node_options`
 
@@ -512,13 +501,38 @@ structure, default: no node options
 ```yaml
 ha_cluster_node_options:
   - node_name: node1
+    pcs_address: node1-address
+    corosync_addresses:
+      - 192.168.1.11
+      - 192.168.2.11
+    sbd_watchdog_modules:
+      - module1
+      - module2
+    sbd_watchdog_modules_blocklist:
+      - module3
+    sbd_watchdog: /dev/watchdog2
+    sbd_devices:
+      - /dev/vdx
+      - /dev/vdy
     attributes:
       - attrs:
           - name: attribute1
             value: value1_node1
           - name: attribute2
             value: value2_node1
   - node_name: node2
+    pcs_address: node2-address:2224
+    corosync_addresses:
+      - 192.168.1.12
+      - 192.168.2.12
+    sbd_watchdog_modules:
+      - module1
+    sbd_watchdog_modules_blocklist:
+      - module3
+    sbd_watchdog: /dev/watchdog1
+    sbd_devices:
+      - /dev/vdw
+      - /dev/vdz
     attributes:
       - attrs:
           - name: attribute1
@@ -528,18 +542,54 @@ ha_cluster_node_options:
 ```
 
 This variable defines various settings which vary from cluster node to cluster
-node. **Note:** Use an inventory or playbook hosts to specify which nodes form
-the cluster. This variable merely sets options for the specified nodes. The
-items are as follows:
+node.
+
+**Note:** Use an inventory or playbook hosts to specify which nodes form
+the cluster. This variable merely sets options for the specified nodes.
 
-* `node_name` (mandatory) - Node name.
+The items are as follows:
+
+* `node_name` (mandatory) - Node name. It must match a name defined for a node.
+  See also [`ha_cluster.node_name`](#nodes-names-and-addresses).
+* `pcs_address` (optional) - Address used by pcs to communicate with the node,
+  it can be a name, a FQDN or an IP address. Port can be specified as well.
+* `corosync_addresses` (optional) - List of addresses used by Corosync, all
+  nodes must have the same number of addresses and the order of the addresses
+  matters.
+* `sbd_watchdog_modules` (optional) - Watchdog kernel modules to be loaded
+  (creates `/dev/watchdog*` devices). Defaults to empty list if not set.
+* `sbd_watchdog_modules_blocklist` (optional) - Watchdog kernel modules to be
+  unloaded and blocked. Defaults to empty list if not set.
+* `sbd_watchdog` (optional) - Watchdog device to be used by SBD. Defaults to
+  `/dev/watchdog` if not set.
+* `sbd_devices` (optional) - Devices to use for exchanging SBD messages and for
+  monitoring. Defaults to empty list if not set.
 * `attributes` (optional) - List of sets of Pacemaker node attributes for the
   node. Currently, no more than one set for each node is supported.
 
 You may take a look at examples:
 
+* [configuring cluster to use SBD](#configuring-cluster-to-use-sbd)
 * [configuring node attributes](#configuring-node-attributes)
 
+#### `ha_cluster_cluster_properties`
+
+structure, default: no properties
+
+```yaml
+ha_cluster_cluster_properties:
+  - attrs:
+      - name: property1_name
+        value: property1_value
+      - name: property2_name
+        value: property2_value
+```
+
+List of sets of cluster properties - Pacemaker cluster-wide configuration.
+Currently, only one set is supported.
+
+You may take a look at [an example](#configuring-cluster-properties).
+
 #### `ha_cluster_resource_primitives`
 
 structure, default: no resources
@@ -1202,7 +1252,7 @@ ha_cluster_constraints_ticket:
 You may take a look at
 [an example](#creating-a-cluster-with-resource-constraints).
 
-#### ha_cluster_acls
+#### `ha_cluster_acls`
 
 structure, default: no ACLs
 
@@ -1303,8 +1353,11 @@ example](#configuring-a-cluster-using-a-quorum-device).
 
 #### Nodes' names and addresses
 
-Nodes' names and addresses can be configured in inventory. This is optional. If
-no names or addresses are configured, play's targets will be used.
+Nodes' names and addresses can be configured in `ha_cluster` variable, for
+example in inventory. This is optional.
+Addresses configured in [`ha_cluster_node_options`](#ha_cluster_node_options)
+override those configured in `ha_cluster`.
+If no names or addresses are configured, play's targets will be used.
 
 Example inventory with targets `node1` and `node2`:
 
@@ -1336,10 +1389,13 @@ all:
 #### SBD watchdog and devices
 
 When using SBD, you may optionally configure watchdog and SBD devices for each
-node in inventory. Even though all SBD devices must be shared to and accessible
-from all nodes, each node may use different names for the devices. The loaded
-watchdog modules and used devices may also be different for each node. See also
-[SBD variables](#ha_cluster_sbd_enabled).
+node in `ha_cluster` variable, for example in inventory.
+Even though all SBD devices must be shared to and accessible from all nodes,
+each node may use different names for the devices. The loaded watchdog modules
+and used devices may also be different for each node.
+SBD settings defined in [`ha_cluster_node_options`](#ha_cluster_node_options)
+override those defined in `ha_cluster`.
+See also [SBD variables](#ha_cluster_sbd_enabled).
 
 Example inventory with targets `node1` and `node2`:
 
@@ -1484,10 +1540,70 @@ in /var/lib/pcsd with the file name FILENAME.crt and FILENAME.key, respectively.
 
 ### Configuring cluster to use SBD
 
-#### inventory
+#### Using `ha_cluster_node_options` variable
 
-These variables need to be set in inventory or via `host_vars`. Of course
-the SBD kernel modules and device path might differ depending on your setup.
+```yaml
+- hosts: node1 node2
+  vars:
+    ha_cluster_cluster_name: my-new-cluster
+    ha_cluster_hacluster_password: password
+    ha_cluster_sbd_enabled: true
+    ha_cluster_sbd_options:
+      - name: delay-start
+        value: 'no'
+      - name: startmode
+        value: always
+      - name: timeout-action
+        value: 'flush,reboot'
+      - name: watchdog-timeout
+        value: 30
+    ha_cluster_node_options:
+      - node_name: node1
+        sbd_watchdog_modules:
+          - iTCO_wdt
+        sbd_watchdog_modules_blocklist:
+          - ipmi_watchdog
+        sbd_watchdog: /dev/watchdog1
+        sbd_devices:
+          - /dev/vdx
+          - /dev/vdy
+          - /dev/vdz
+      - node_name: node2
+        sbd_watchdog_modules:
+          - iTCO_wdt
+        sbd_watchdog_modules_blocklist:
+          - ipmi_watchdog
+        sbd_watchdog: /dev/watchdog1
+        sbd_devices:
+          - /dev/vdx
+          - /dev/vdy
+          - /dev/vdz
+    # Best practice for setting SBD timeouts:
+    # watchdog-timeout * 2 = msgwait-timeout (set automatically)
+    # msgwait-timeout * 1.2 = stonith-timeout
+    ha_cluster_cluster_properties:
+      - attrs:
+          - name: stonith-timeout
+            value: 72
+    ha_cluster_resource_primitives:
+      - id: fence_sbd
+        agent: 'stonith:fence_sbd'
+        instance_attrs:
+          - attrs:
+              # taken from host_vars
+              - name: devices
+                value: "{{ ha_cluster.sbd_devices | join(',') }}"
+              - name: pcmk_delay_base
+                value: 30
+
+  roles:
+    - linux-system-roles.ha_cluster
+```
+
+#### Using `ha_cluster` variable
+
+The same result can be achieved by specifying node-specific options in inventory
+like this:
 
 ```yaml
 all:
@@ -1516,11 +1632,7 @@ all:
           - /dev/vdz
 ```
 
-#### playbook
-
-After setting the inventory correctly, use this playbook to configure a
-complete SBD setup including loading watchdog modules and creating the
-SBD stonith resource.
+Variables specified in inventory can be omitted when writing the playbook:
 
 ```yaml
 - hosts: node1 node2
@@ -1559,6 +1671,9 @@ SBD stonith resource.
     - linux-system-roles.ha_cluster
 ```
 
+If both the `ha_cluster_node_options` and `ha_cluster` variables contain SBD
+options, those in `ha_cluster_node_options` have precedence.
+
 ### Configuring cluster properties
 
 ```yaml

examples/sbd.yml

@@ -2,7 +2,6 @@
 ---
 - name: Example ha_cluster role invocation - cluster with SBD
   hosts: node1 node2
-  # do not forget to also set variables via inventory or host_vars.
   vars:
     ha_cluster_manage_firewall: true
     ha_cluster_manage_selinux: true
@@ -18,6 +17,27 @@
         value: 'flush,reboot'
       - name: watchdog-timeout
         value: 30
+    ha_cluster_node_options:
+      - node_name: node1
+        sbd_watchdog_modules:
+          - iTCO_wdt
+        sbd_watchdog_modules_blocklist:
+          - ipmi_watchdog
+        sbd_watchdog: /dev/watchdog1
+        sbd_devices:
+          - /dev/vdx
+          - /dev/vdy
+          - /dev/vdz
+      - node_name: node2
+        sbd_watchdog_modules:
+          - iTCO_wdt
+        sbd_watchdog_modules_blocklist:
+          - ipmi_watchdog
+        sbd_watchdog: /dev/watchdog1
+        sbd_devices:
+          - /dev/vdx
+          - /dev/vdy
+          - /dev/vdz
     # Best practice for setting SBD timeouts:
     # watchdog-timeout * 2 = msgwait-timeout (set automatically)
     # msgwait-timeout * 1.2 = stonith-timeout