Unable to join LXD Cluster on 6.1 #13793

TorbenMartin · 2024-07-22T08:53:07Z

Hey.

Running Ubunu Server 24.04 on HOST System with LXD 6.1
Created 2 VMs with Ubuntu 24.04 and installed LXD 6.1 and want to join Cluster for VXLAN Tests behind OpenWRT LXC

lxc network create net1 \
    tunnel.net.protocol=vxlan \
    tunnel.net.id=1 \
    ipv6.address=none \
    ipv4.address=none

But already Problems on Beginning... with joining the Cluster after fresh Install:

snap remove --purge lxd
snap install lxd --channel=6.1/stable
snap set lxd ui.enable=true
snap refresh --hold lxd
snap restart --reload lxd
reboot

Server 1:

root@lxd1:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.0.3.223]: 
Are you joining an existing cluster? (yes/no) [default=no]: 
What member name should be used to identify this server in the cluster? [default=lxd1]: 
Do you want to configure a new local storage pool? (yes/no) [default=yes]: 
Name of the storage backend to use (dir, lvm, zfs, btrfs) [default=zfs]: 
Create a new ZFS pool? (yes/no) [default=yes]: 
Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]: 
Size in GiB of the new loop device (1GiB minimum) [default=5GiB]: 
Do you want to configure a new remote storage pool? (yes/no) [default=no]: 
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]: 
Would you like to create a new Fan overlay network? (yes/no) [default=yes]: 
What subnet should be used as the Fan underlay? [default=auto]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 
root@lxd1:~# 
root@lxd1:~# lxc cluster add lxd2
To start your first container, try: lxc launch ubuntu:24.04
Or for a virtual machine: lxc launch ubuntu:24.04 --vm

Member lxd2 join token:
eyJzZXJ2ZXJfbmFtZSI6Imx4ZDIiLCJmaW5nZXJwcmludCI6IjQyN2VjYTUzMDRkMTNlOTQ0Mjc3OGJmNmM2NWRkZTU5MmU3MGEwY2M0ZjdlMjQyOTQwZTY5OTFkNjgxNTUxYzIiLCJhZGRyZXNzZXMiOlsiMTAuMC4zLjIyMzo4NDQzIl0sInNlY3JldCI6IjhlZmVjNjEyNWVlMzhmNDg3ZDhlMTBlNWE2NzhlOTExNmNkMjMyYWE1YzZkMzQ0MzA4YWM1YTA3MTAwMzJlYTQiLCJleHBpcmVzX2F0IjoiMjAyNC0wNy0yMlQxMTozMzoyNC45OTYyOTgwNVoifQ==
root@lxd1:~#

Server 2:

root@lxd2:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.0.3.224]: 
Are you joining an existing cluster? (yes/no) [default=no]: yes
Please provide join token: eyJzZXJ2ZXJfbmFtZSI6Imx4ZDIiLCJmaW5nZXJwcmludCI6IjQyN2VjYTUzMDRkMTNlOTQ0Mjc3OGJmNmM2NWRkZTU5MmU3MGEwY2M0ZjdlMjQyOTQwZTY5OTFkNjgxNTUxYzIiLCJhZGRyZXNzZXMiOlsiMTAuMC4zLjIyMzo4NDQzIl0sInNlY3JldCI6IjhlZmVjNjEyNWVlMzhmNDg3ZDhlMTBlNWE2NzhlOTExNmNkMjMyYWE1YzZkMzQ0MzA4YWM1YTA3MTAwMzJlYTQiLCJleHBpcmVzX2F0IjoiMjAyNC0wNy0yMlQxMTozMzoyNC45OTYyOTgwNVoifQ==
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "size" property for storage pool "default": 
Choose "source" property for storage pool "default": 
Choose "zfs.pool_name" property for storage pool "default": 
Choose "size" property for storage pool "local": 
Choose "source" property for storage pool "local": 
Choose "zfs.pool_name" property for storage pool "local": 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: yes
config: {}
networks: []
storage_pools: []
storage_volumes: []
profiles: []
projects: []
cluster:
  server_name: lxd2
  enabled: true
  member_config:
  - entity: storage-pool
    name: default
    key: size
    value: ""
    description: '"size" property for storage pool "default"'
  - entity: storage-pool
    name: default
    key: source
    value: ""
    description: '"source" property for storage pool "default"'
  - entity: storage-pool
    name: default
    key: zfs.pool_name
    value: ""
    description: '"zfs.pool_name" property for storage pool "default"'
  - entity: storage-pool
    name: local
    key: size
    value: ""
    description: '"size" property for storage pool "local"'
  - entity: storage-pool
    name: local
    key: source
    value: ""
    description: '"source" property for storage pool "local"'
  - entity: storage-pool
    name: local
    key: zfs.pool_name
    value: ""
    description: '"zfs.pool_name" property for storage pool "local"'
  cluster_address: 10.0.3.223:8443
  cluster_certificate: |
    -----BEGIN CERTIFICATE-----
    MIIB3jCCAWSgAwIBAgIRANF+sSzifc3aNEBnZV4bwg0wCgYIKoZIzj0EAwMwIjEM
    MAoGA1UEChMDTFhEMRIwEAYDVQQDDAlyb290QGx4ZDEwHhcNMjQwNzIyMDgyOTMw
    WhcNMzQwNzIwMDgyOTMwWjAiMQwwCgYDVQQKEwNMWEQxEjAQBgNVBAMMCXJvb3RA
    bHhkMTB2MBAGByqGSM49AgEGBSuBBAAiA2IABCgImyl74WbTyZYZnvD9nGZHiy9R
    xwIkvKg7GIHs+lYTP3OmCsDG5rlx9dfHgcnhWM/Ucf8GpAMwdS/rkuF6wrKKfHkU
    zeXl6PqHyWX3STNUuCwn50XEBRtRVcobAthjVaNeMFwwDgYDVR0PAQH/BAQDAgWg
    MBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwJwYDVR0RBCAwHoIE
    bHhkMYcEfwAAAYcQAAAAAAAAAAAAAAAAAAAAATAKBggqhkjOPQQDAwNoADBlAjEA
    +m/vsUhONh8CpDsVoHfIX6kJOxz/0NmyK0uECRuNDrYQwaVEn3OEMusgEkOOsG8Q
    AjBnH0jH5VjE6eS4qOPNSb8z2Je9BpTfWoZn923dpEbfHpm5APmRpgeqz87KWdu3
    R8o=
    -----END CERTIFICATE-----
  server_address: 10.0.3.224:8443
  cluster_password: ""
  cluster_token: ""
  cluster_certificate_path: ""

Error: Failed to join cluster: Failed to initialize member: Failed to initialize storage pools and networks: Failed to update storage pool "default": Config key "size" is cluster member specific

The text was updated successfully, but these errors were encountered:

TorbenMartin · 2024-07-22T18:22:24Z

One Try again.... Cluster join works with minimal Initialization.

root@lxd1:~# lxd init --minimal
root@lxd1:~# lxd init --auto

(finaly this both work)

root@lxd1:~# lxc config set core.https_address 10.0.3.223:8443
root@lxd1:~# lxc cluster enable lxd1
Clustering enabled
root@lxd1:~# lxc cluster add lxd2
Member lxd2 join token:
eyJzZXJ2ZXJfbmFtZSI6Imx4ZDIiLCJmaW5nZXJwcmludCI6IjdjMTlkOTYyNGVmZmJkZWQ0ZTRlNTYyNmZkZmM3NTJjMTZiNTI0NmRmZjE5MjU2MTYyZDc3OTExNjAyZDBmYjYiLCJhZGRyZXNzZXMiOlsiMTAuMC4zLjIyMzo4NDQzIl0sInNlY3JldCI6IjdhYzg4Mjc0ODM4M2YyZmE1ZGJjNTkxZDUwOWU4M2Y0YTNiMDE3NjQ2NTIxM2U1OWFiNDA1NGRkMDdkZDQzNTkiLCJleHBpcmVzX2F0IjoiMjAyNC0wNy0yMlQyMToxODoyNi4wMzM1NDAxMzRaIn0=

root@lxd2:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.0.3.224]: 
Are you joining an existing cluster? (yes/no) [default=no]: yes
Please provide join token: eyJzZXJ2ZXJfbmFtZSI6Imx4ZDIiLCJmaW5nZXJwcmludCI6IjdjMTlkOTYyNGVmZmJkZWQ0ZTRlNTYyNmZkZmM3NTJjMTZiNTI0NmRmZjE5MjU2MTYyZDc3OTExNjAyZDBmYjYiLCJhZGRyZXNzZXMiOlsiMTAuMC4zLjIyMzo4NDQzIl0sInNlY3JldCI6IjdhYzg4Mjc0ODM4M2YyZmE1ZGJjNTkxZDUwOWU4M2Y0YTNiMDE3NjQ2NTIxM2U1OWFiNDA1NGRkMDdkZDQzNTkiLCJleHBpcmVzX2F0IjoiMjAyNC0wNy0yMlQyMToxODoyNi4wMzM1NDAxMzRaIn0=
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "source" property for storage pool "default": 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 
root@lxd2:~#

root@lxd2:~# lxc cluster list
To start your first container, try: lxc launch ubuntu:24.04
Or for a virtual machine: lxc launch ubuntu:24.04 --vm

+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| NAME |           URL           |      ROLES       | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE  |      MESSAGE      |
+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| lxd1 | https://10.0.3.223:8443 | database-leader  | x86_64       | default        |             | ONLINE | Fully operational |
|      |                         | database         |              |                |             |        |                   |
+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| lxd2 | https://10.0.3.224:8443 | database-standby | x86_64       | default        |             | ONLINE | Fully operational |
+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+

TorbenMartin · 2024-07-22T20:58:09Z

Another Error.
Why this works on Single Servers but not on a Cluster

root@lxd1:~# lxc cluster list
+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| NAME |           URL           |      ROLES       | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE  |      MESSAGE      |
+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| lxd1 | https://10.0.3.223:8443 | database-leader  | x86_64       | default        |             | ONLINE | Fully operational |
|      |                         | database         |              |                |             |        |                   |
+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| lxd2 | https://10.0.3.224:8443 | database-standby | x86_64       | default        |             | ONLINE | Fully operational |
+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
root@lxd1:~# lxc network create net1 \
    tunnel.net.protocol=vxlan \
    tunnel.net.id=1 \
    ipv6.address=none \
    ipv4.address=none
Error: Network not pending on any node (use --target <node> first)

TorbenMartin · 2024-07-22T21:04:07Z

joining lxd3 not work.

root@lxd3:~# 
snap remove --purge lxd
snap install lxd --channel=6.1/stable
snap set lxd ui.enable=true
snap refresh --hold lxd
snap restart --reload lxd
snap "lxd" is not installed
2024-07-22T21:01:20Z INFO Waiting for automatic snapd restart...
Warning: /snap/bin was not found in your $PATH. If you've not restarted your session since you
         installed snapd, try doing that. Please see https://forum.snapcraft.io/t/9469 for more
         details.

lxd (6.1/stable) 6.1-c14927a from Canonical✓ installed
General refreshes of "lxd" held indefinitely
Restarted.
root@lxd3:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.0.3.225]: 
Are you joining an existing cluster? (yes/no) [default=no]: yes
Please provide join token: eyJzZXJ2ZXJfbmFtZSI6Imx4ZDMiLCJmaW5nZXJwcmludCI6IjRlY2JkZTBlYjE0YTVhMGZiYmU4NWYwM2UzYTAyMjY3N2U2Y2IwYjA0NDljMmU2MjU3YTJiNzcyZTNkZDNiOWIiLCJhZGRyZXNzZXMiOlsiMTAuMC4zLjIyMzo4NDQzIiwiMTAuMC4zLjIyNDo4NDQzIl0sInNlY3JldCI6IjhkYWQxZWZkMTVkNjE2Y2RkYWU3MzE5YWUxN2Q2NzNiOGMwYzU0NzA0MDkxMjc1ZTM4NTRmOGRlMWFlZTU2NjQiLCJleHBpcmVzX2F0IjoiMjAyNC0wNy0yM1QwMDowMzowMS45NTg4MzI0NTdaIn0=
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "size" property for storage pool "default": 
Choose "source" property for storage pool "default": 
Choose "zfs.pool_name" property for storage pool "default": 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 
Error: Failed to join cluster: Failed to initialize member: Failed to initialize storage pools and networks: Failed to update storage pool "default": Config key "size" is cluster member specific
root@lxd3:~#

boltmark · 2024-07-23T02:52:02Z

Thanks @TorbenMartin, I am able to reproduce this with 6.1/stable and Ubuntu 24.04 VMs. We are looking into this.

boltmark · 2024-07-23T02:53:07Z

Another Error. Why this works on Single Servers but not on a Cluster

root@lxd1:~# lxc cluster list
+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| NAME |           URL           |      ROLES       | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE  |      MESSAGE      |
+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| lxd1 | https://10.0.3.223:8443 | database-leader  | x86_64       | default        |             | ONLINE | Fully operational |
|      |                         | database         |              |                |             |        |                   |
+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| lxd2 | https://10.0.3.224:8443 | database-standby | x86_64       | default        |             | ONLINE | Fully operational |
+------+-------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
root@lxd1:~# lxc network create net1 \
    tunnel.net.protocol=vxlan \
    tunnel.net.id=1 \
    ipv6.address=none \
    ipv4.address=none
Error: Network not pending on any node (use --target <node> first)

When defining networks in a cluster, you first need to use lxc network create with --target <node-name> for each of your nodes, while passing in any node-specific config. Once you have done this for each of the nodes, you can use the non-targeted command to create the network for the cluster.

For your reference: https://documentation.ubuntu.com/lxd/en/latest/howto/cluster_config_networks/.

TorbenMartin · 2024-07-23T05:22:51Z

I was waiting one night.
Set lxc config set core.https_address 0.0.0.0:8443

And now can join Cluster with LXD3

root@lxd3:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.0.3.225]: 
Are you joining an existing cluster? (yes/no) [default=no]: yes
Please provide join token: eyJzZXJ2ZXJfbmFtZSI6Imx4ZDMiLCJmaW5nZXJwcmludCI6IjRlY2JkZTBlYjE0YTVhMGZiYmU4NWYwM2UzYTAyMjY3N2U2Y2IwYjA0NDljMmU2MjU3YTJiNzcyZTNkZDNiOWIiLCJhZGRyZXNzZXMiOlsiMTAuMC4zLjIyMzo4NDQzIiwiMTAuMC4zLjIyNDo4NDQzIl0sInNlY3JldCI6IjkwNjc3NzY4ZDFiYzliZjUwY2UyYmU4ZDgzZGJlYWM5M2U4YTJlOWRlMGZjZTliYWFkZTA0MzRiYjZmMzljZGIiLCJleHBpcmVzX2F0IjoiMjAyNC0wNy0yM1QwODoxOToyMS4wMDQ2NTYzMTlaIn0=
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "size" property for storage pool "default": 
Choose "source" property for storage pool "default": 
Choose "zfs.pool_name" property for storage pool "default": 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 
root@lxd3:~# lxc cluster list
To start your first container, try: lxc launch ubuntu:24.04
Or for a virtual machine: lxc launch ubuntu:24.04 --vm

+------+-------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| NAME |           URL           |      ROLES      | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE  |      MESSAGE      |
+------+-------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| lxd1 | https://10.0.3.223:8443 | database-leader | x86_64       | default        |             | ONLINE | Fully operational |
|      |                         | database        |              |                |             |        |                   |
+------+-------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| lxd2 | https://10.0.3.224:8443 | database        | x86_64       | default        |             | ONLINE | Fully operational |
+------+-------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| lxd3 | https://10.0.3.225:8443 | database        | x86_64       | default        |             | ONLINE | Fully operational |
+------+-------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+

tomponline · 2024-07-23T15:09:04Z

Please can you provide output of sudo snap list on each server. Thanks

tomponline · 2024-07-23T15:14:26Z

I've not been able to reproduce this issue with:

lxd 6.1-c14927a 29551 6.1/stable canonical✓ -

host 1

root@vc1:~# snap install lxd --channel=6.1/stable
2024-07-23T15:15:21Z INFO Waiting for automatic snapd restart...
lxd (6.1/stable) 6.1-c14927a from Canonical✓ installed
root@vc1:~# lxd init 
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.21.203.10]: 
Are you joining an existing cluster? (yes/no) [default=no]:  
What member name should be used to identify this server in the cluster? [default=vc1]: 
Do you want to configure a new local storage pool? (yes/no) [default=yes]: 
Name of the storage backend to use (dir, lvm, zfs, btrfs) [default=zfs]: 
Create a new ZFS pool? (yes/no) [default=yes]: 
Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]: 
Size in GiB of the new loop device (1GiB minimum) [default=5GiB]: 
Do you want to configure a new remote storage pool? (yes/no) [default=no]: 
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]: 
Would you like to create a new Fan overlay network? (yes/no) [default=yes]: 
What subnet should be used as the Fan underlay? [default=auto]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 

root@vc1:~# lxc cluster add vc2
To start your first container, try: lxc launch ubuntu:24.04
Or for a virtual machine: lxc launch ubuntu:24.04 --vm

Member vc2 join token:
eyJzZXJ2ZXJfbmFtZSI6InZjMiIsImZpbmdlcnByaW50IjoiOGUwNmI4Mzk4YzI0YWQ5MzVhMzdjYjE5MGVkM2E2OTVhMWVlOTRhM2ZiMzcwY2M5MjI0YWRiOTA0ZGFmMTA0NyIsImFkZHJlc3NlcyI6WyIxMC4yMS4yMDMuMTA6ODQ0MyJdLCJzZWNyZXQiOiJhNjM4OWM0ZjJkM2VhMjliYWJmZTZlNmM3OTc0MzI5YzMzZjJiNzI5MTE4OTZmNTU4YWNkMjIwZmI4MTIyNjkxIiwiZXhwaXJlc19hdCI6IjIwMjQtMDctMjNUMTg6MTY6MzYuNDc3NjAwNzZaIn0=

host 2

root@vc2:~# snap install lxd --channel=6.1/stable
2024-07-23T15:17:00Z INFO Waiting for automatic snapd restart...
2024-07-23T15:17:01Z INFO Waiting for automatic snapd restart...
lxd (6.1/stable) 6.1-c14927a from Canonical✓ installed
root@vc2:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.21.203.5]: 
Are you joining an existing cluster? (yes/no) [default=no]: yes
Please provide join token: eyJzZXJ2ZXJfbmFtZSI6InZjMiIsImZpbmdlcnByaW50IjoiOGUwNmI4Mzk4YzI0YWQ5MzVhMzdjYjE5MGVkM2E2OTVhMWVlOTRhM2ZiMzcwY2M5MjI0YWRiOTA0ZGFmMTA0NyIsImFkZHJlc3NlcyI6WyIxMC4yMS4yMDMuMTA6ODQ0MyJdLCJzZWNyZXQiOiJhNjM4OWM0ZjJkM2VhMjliYWJmZTZlNmM3OTc0MzI5YzMzZjJiNzI5MTE4OTZmNTU4YWNkMjIwZmI4MTIyNjkxIiwiZXhwaXJlc19hdCI6IjIwMjQtMDctMjNUMTg6MTY6MzYuNDc3NjAwNzZaIn0=
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "size" property for storage pool "local": 
Choose "source" property for storage pool "local": 
Choose "zfs.pool_name" property for storage pool "local": 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: yes
config: {}
networks: []
storage_pools: []
storage_volumes: []
profiles: []
projects: []
cluster:
  server_name: vc2
  enabled: true
  member_config:
  - entity: storage-pool
    name: local
    key: size
    value: ""
    description: '"size" property for storage pool "local"'
  - entity: storage-pool
    name: local
    key: source
    value: ""
    description: '"source" property for storage pool "local"'
  - entity: storage-pool
    name: local
    key: zfs.pool_name
    value: ""
    description: '"zfs.pool_name" property for storage pool "local"'
  cluster_address: 10.21.203.10:8443
  cluster_certificate: |
    -----BEGIN CERTIFICATE-----
    MIIB2zCCAWGgAwIBAgIRAMpQKSjwQY1LaGUEFEihzEgwCgYIKoZIzj0EAwMwITEM
    MAoGA1UEChMDTFhEMREwDwYDVQQDDAhyb290QHZjMTAeFw0yNDA3MjMxNTE1MTJa
    Fw0zNDA3MjExNTE1MTJaMCExDDAKBgNVBAoTA0xYRDERMA8GA1UEAwwIcm9vdEB2
    YzEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAATuijHtSMuYa2oSNK8gpd1pjajoo3eJ
    IkhtsXAgVTmZnm/s6muC7JR7D96yjKnU6I9cpImDJ1v79isElghdO911Nzn6nre6
    NbSeqjdRNrIJtPdfVQnOc4w/mzdsAnSyigqjXTBbMA4GA1UdDwEB/wQEAwIFoDAT
    BgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMCYGA1UdEQQfMB2CA3Zj
    MYcEfwAAAYcQAAAAAAAAAAAAAAAAAAAAATAKBggqhkjOPQQDAwNoADBlAjBOsMO/
    HyQHw4zqwm8xXdSicRAWnJsybKjCDeiFDtyFxBiJuorHBYhLANiRJz2cPy8CMQC7
    BF/e6GnS/zTOCc7dHZmnkwfHDEj49Zd5cgiVIFI2ooKOctuHJw801Fer5ra6DcI=
    -----END CERTIFICATE-----
  server_address: 10.21.203.5:8443
  cluster_password: ""
  cluster_token: ""
  cluster_certificate_path: ""

tomponline · 2024-07-23T15:22:37Z

@boltmark do you have a reproducer?

TorbenMartin · 2024-07-23T15:52:19Z

sudo snap lis

root@lxd1:~# sudo snap list
Name    Version      Rev    Tracking       Publisher   Notes
core22  20240408     1380   latest/stable  canonical✓  base
lxd     6.1-c14927a  29551  6.1/stable     canonical✓  held
snapd   2.63         21759  latest/stable  canonical✓  snapd

[root@lxd2:~# sudo snap list
Name    Version      Rev    Tracking       Publisher   Notes
core22  20240408     1380   latest/stable  canonical✓  base
lxd     6.1-c14927a  29551  6.1/stable     canonical✓  held
snapd   2.63         21759  latest/stable  canonical✓  snapd
root@lxd2:~# ]

root@lxd3:~# sudo snap list
Name    Version      Rev    Tracking       Publisher   Notes
core22  20240408     1380   latest/stable  canonical✓  base
lxd     6.1-c14927a  29551  6.1/stable     canonical✓  held
snapd   2.63         21759  latest/stable  canonical✓  snapd

tomponline · 2024-07-23T15:54:10Z

@TorbenMartin and its working now?

boltmark · 2024-07-23T16:30:05Z

@boltmark do you have a reproducer?

To reproduce, initialize lxd on a node and add a storage pool other than the default. Then, run lxd init again and attempt to add the node to a cluster. With fresh VMs that have not previously initialized lxd, this error does not occur. I believe this is expected behavior, but please correct me if I am wrong @tomponline.

@TorbenMartin Can you confirm that this works when using fresh VMs that have not previously initialized lxd or do not have any existing storage volumes other than default?

tomponline · 2024-07-23T16:52:06Z

@boltmark please can you post your reproducer steps in full

TorbenMartin · 2024-07-23T17:03:24Z

@boltmark do you have a reproducer?

To reproduce, initialize lxd on a node and add a storage pool other than the default. Then, run lxd init again and attempt to add the node to a cluster. With fresh VMs that have not previously initialized lxd, this error does not occur. I believe this is expected behavior, but please correct me if I am wrong @tomponline.

@TorbenMartin Can you confirm that this works when using fresh VMs that have not previously initialized lxd or do not have any existing storage volumes other than default?

the process is remove lxd first then install 6.1
so there is no old data from another init...
on a fresh install on 2 new VMs with 24.04

i not initialize lxd many times.
but when initialize lxd with all the default options...-> it will create "default" and "local" storage with both zfs driver.
so now for another test i run init but i choose:

Do you want to configure a new local storage pool? (yes/no) [default=yes]: no

----->

root@lxd1v2:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.0.3.228]: 
Are you joining an existing cluster? (yes/no) [default=no]: 
What member name should be used to identify this server in the cluster? [default=lxd1v2]: 
Do you want to configure a new local storage pool? (yes/no) [default=yes]: no
Do you want to configure a new remote storage pool? (yes/no) [default=no]: 
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]: 
Would you like to create a new Fan overlay network? (yes/no) [default=yes]: 
What subnet should be used as the Fan underlay? [default=auto]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 
root@lxd1v2:~# lxc cluster add lxd2v2
To start your first container, try: lxc launch ubuntu:24.04
Or for a virtual machine: lxc launch ubuntu:24.04 --vm

root@lxd1v2:~# lxc cluster add lxd2v2
Member lxd2v2 join token:
eyJzZXJ2ZXJfbmFtZSI6Imx4ZDJ2MiIsImZpbmdlcnByaW50IjoiMmU2ZWIzNjc5YzY0NGNkYTJkNDEyYWNjZjk0YzBkNmJmNjdiMmE0MzY2NTgzOTExN2NjYjVlZDAwMGVjNzMwMiIsImFkZHJlc3NlcyI6WyIxMC4wLjMuMjI4Ojg0NDMiXSwic2VjcmV0IjoiYjA3N2Y5YWVkNjg4ZmU5ODRkZjMzZGNkMjZlYmJkNjMzNzU0MDdlMjY1NDU2OGNmYzNjNThjYTUyZTE0YzU4NCIsImV4cGlyZXNfYXQiOiIyMDI0LTA3LTIzVDIwOjAwOjQ2LjcwNDU5OTIwM1oifQ==
root@lxd1v2:~# lxc storage list
+---------+--------+-------------+---------+---------+
|  NAME   | DRIVER | DESCRIPTION | USED BY |  STATE  |
+---------+--------+-------------+---------+---------+
| default | zfs    |             | 1       | CREATED |
+---------+--------+-------------+---------+---------+
root@lxd1v2:~#

Here LXD2v2:

root@lxd2v2:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.0.3.230]: 
Are you joining an existing cluster? (yes/no) [default=no]: yes
Please provide join token: eyJzZXJ2ZXJfbmFtZSI6Imx4ZDJ2MiIsImZpbmdlcnByaW50IjoiMmU2ZWIzNjc5YzY0NGNkYTJkNDEyYWNjZjk0YzBkNmJmNjdiMmE0MzY2NTgzOTExN2NjYjVlZDAwMGVjNzMwMiIsImFkZHJlc3NlcyI6WyIxMC4wLjMuMjI4Ojg0NDMiXSwic2VjcmV0IjoiYjA3N2Y5YWVkNjg4ZmU5ODRkZjMzZGNkMjZlYmJkNjMzNzU0MDdlMjY1NDU2OGNmYzNjNThjYTUyZTE0YzU4NCIsImV4cGlyZXNfYXQiOiIyMDI0LTA3LTIzVDIwOjAwOjQ2LjcwNDU5OTIwM1oifQ==
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "source" property for storage pool "default": 
Choose "zfs.pool_name" property for storage pool "default": 
Choose "size" property for storage pool "default": 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 
Error: Failed to join cluster: Failed to initialize member: Failed to initialize storage pools and networks: Failed to update storage pool "default": Config key "size" is cluster member specific
root@lxd2v2:~#

TorbenMartin · 2024-07-23T17:11:32Z

@TorbenMartin and its working now?

it was working add server2 to server1 in cluster when i init the first Node with --minimal
it was not working when i add server3 first.
after change the core.https_address may i can add server3

this test VMs are a fresh auto/cloud-init installation by the Host.

Host + VMs have same Kernel.

root@lxd2v2:~# uname -r
6.8.0-36-generic

root@bitmaster:/home/serveradmin# uname -r
6.8.0-36-generic

The VMs are connected into a Network that i created in LXD on the Host.
This Network is attached to a pfsense Firewall VM.

I just upgraded the Host a few days ago from LXD 5.21 to 6.1

tomponline · 2024-07-23T17:15:52Z

Im still quite confused what the reproducer steps are fot a fresh system. What are you doing differently compared to my working example above?

tomponline · 2024-07-23T17:16:19Z

Its sounding like you have some existing state.

TorbenMartin · 2024-07-23T17:18:57Z

Its sounding like you have some existing state.
no i not run another init before this.

Did you run this Ubuntu 24.04 VMs on a LXD Hypervisor or do you run it on another Virtualization Platform?

Maybe something is wrong with the cloud-init / cloud disk image?!

Just tested on Ubuntu 22.04 VMs. Run LXD 6.1
Same Problem.

boltmark · 2024-07-23T17:35:24Z

@boltmark please can you post your reproducer steps in full

Below is the only way that I can reproduce a similar error, but this seems different from what @TorbenMartin is running into as he is not using lxd init multiple times.

Host 1:

root@vm1:~# snap install lxd --channel=6.1/stable
2024-07-23T16:57:04Z INFO Waiting for automatic snapd restart...
lxd (6.1/stable) 6.1-c14927a from Canonical✓ installed
root@vm1:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.252.72.173]: 
Are you joining an existing cluster? (yes/no) [default=no]: 
What member name should be used to identify this server in the cluster? [default=vm1]: 
Do you want to configure a new local storage pool? (yes/no) [default=yes]: 
Name of the storage backend to use (dir, lvm, zfs, btrfs) [default=zfs]: 
Create a new ZFS pool? (yes/no) [default=yes]: 
Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]: 
Size in GiB of the new loop device (1GiB minimum) [default=5GiB]: 
Do you want to configure a new remote storage pool? (yes/no) [default=no]: 
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]: 
Would you like to create a new Fan overlay network? (yes/no) [default=yes]: 
What subnet should be used as the Fan underlay? [default=auto]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 

root@vm1:~# lxc cluster add vm2
To start your first container, try: lxc launch ubuntu:24.04
Or for a virtual machine: lxc launch ubuntu:24.04 --vm

Member vm2 join token:
eyJzZXJ2ZXJfbmFtZSI6InZtMyIsImZpbmdlcnByaW50IjoiZWE1ZDY5ZDEyYmMwZTBiZWIwZGVkNDAxZGI0OTVjZmViN2Y3YTVjNzMyYjYxOTNiNjI4NDQ0NGNiNjVlZWE4NyIsImFkZHJlc3NlcyI6WyIxMC4yNTIuNzIuMTczOjg0NDMiXSwic2VjcmV0IjoiNGNkMDhiODQwNTExNWFjOTEwMzk0ZDlmOTE4ZjNlZThlODkwOWZlOTQ5ZWZlNGFkZTM1MjBmMjY0OTlhNzc4YSIsImV4cGlyZXNfYXQiOiIyMDI0LTA3LTIzVDIwOjIzOjU3Ljk2MDgwMDcxWiJ9

Host 2:

root@vm2:~# snap install lxd --channel=6.1/stable
2024-07-23T17:21:58Z INFO Waiting for automatic snapd restart...
lxd (6.1/stable) 6.1-c14927a from Canonical✓ installed
root@vm2:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: 
Do you want to configure a new storage pool? (yes/no) [default=yes]: 
Name of the new storage pool [default=default]: 
Name of the storage backend to use (btrfs, ceph, dir, lvm, powerflex, zfs) [default=zfs]: 
Create a new ZFS pool? (yes/no) [default=yes]: 
Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]: 
Size in GiB of the new loop device (1GiB minimum) [default=5GiB]: 
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to create a new local network bridge? (yes/no) [default=yes]: 
What should the new bridge be called? [default=lxdbr0]: 
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
Would you like the LXD server to be available over the network? (yes/no) [default=no]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 

root@vm2:~# lxc storage create local zfs
To start your first container, try: lxc launch ubuntu:24.04
Or for a virtual machine: lxc launch ubuntu:24.04 --vm

Storage pool local created

root@vm2:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.252.72.128]: 
Are you joining an existing cluster? (yes/no) [default=no]: yes
Please provide join token: eyJzZXJ2ZXJfbmFtZSI6InZtMyIsImZpbmdlcnByaW50IjoiZWE1ZDY5ZDEyYmMwZTBiZWIwZGVkNDAxZGI0OTVjZmViN2Y3YTVjNzMyYjYxOTNiNjI4NDQ0NGNiNjVlZWE4NyIsImFkZHJlc3NlcyI6WyIxMC4yNTIuNzIuMTczOjg0NDMiXSwic2VjcmV0IjoiNGNkMDhiODQwNTExNWFjOTEwMzk0ZDlmOTE4ZjNlZThlODkwOWZlOTQ5ZWZlNGFkZTM1MjBmMjY0OTlhNzc4YSIsImV4cGlyZXNfYXQiOiIyMDI0LTA3LTIzVDIwOjIzOjU3Ljk2MDgwMDcxWiJ9
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "size" property for storage pool "local": 
Choose "source" property for storage pool "local": 
Choose "zfs.pool_name" property for storage pool "local": 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: yes
config: {}
networks: []
storage_pools: []
storage_volumes: []
profiles: []
projects: []
cluster:
  server_name: vm3
  enabled: true
  member_config:
  - entity: storage-pool
    name: local
    key: size
    value: ""
    description: '"size" property for storage pool "local"'
  - entity: storage-pool
    name: local
    key: source
    value: ""
    description: '"source" property for storage pool "local"'
  - entity: storage-pool
    name: local
    key: zfs.pool_name
    value: ""
    description: '"zfs.pool_name" property for storage pool "local"'
  cluster_address: 10.252.72.173:8443
  cluster_certificate: |
    -----BEGIN CERTIFICATE-----
    MIIB3DCCAWGgAwIBAgIRANF5Oq8sSdMLrAG0IN3fHCYwCgYIKoZIzj0EAwMwITEM
    MAoGA1UEChMDTFhEMREwDwYDVQQDDAhyb290QHZtMTAeFw0yNDA3MjMxNjU3MzRa
    Fw0zNDA3MjExNjU3MzRaMCExDDAKBgNVBAoTA0xYRDERMA8GA1UEAwwIcm9vdEB2
    bTEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAARe1Igyywf00FAxI6FRUMSbm2+oVMHz
    fOSW6EjdhhUCrCSoagvXZ94Ig8cbpayO33UtD1+H3UJVSQGVXfrVAAlFLBYDuT09
    VcqTOts26oHLbD68C92T67I9RBdGh61u0JOjXTBbMA4GA1UdDwEB/wQEAwIFoDAT
    BgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMCYGA1UdEQQfMB2CA3Zt
    MYcEfwAAAYcQAAAAAAAAAAAAAAAAAAAAATAKBggqhkjOPQQDAwNpADBmAjEAxUrp
    cGKk4X/Hw1yvrWA0fk7VKFT76NOfUKj1xhbP5pGqqUgg6D0OzGF3oMvc0sFNAjEA
    5D9cyYsNt0LRstiTRFXD/E7IixaAzjR+GXG+B+oo003tK49qCn6J9GlsNdjXhwXk
    -----END CERTIFICATE-----
  server_address: 10.252.72.128:8443
  cluster_password: ""
  cluster_token: ""
  cluster_certificate_path: ""

Error: Failed to join cluster: Failed to initialize member: Failed to initialize storage pools and networks: Failed to update storage pool "local": Config key "size" is cluster member specific

tomponline · 2024-07-23T17:48:59Z

Its sounding like you have some existing state.
no i not run another init before this.

Did you run this Ubuntu 24.04 VMs on a LXD Hypervisor or do you run it on another Virtualization Platform?

Maybe something is wrong with the cloud-init / cloud disk image?!

Just tested on Ubuntu 22.04 VMs. Run LXD 6.1 Same Problem.

I ran it inside 2 fresh ubuntu:24.04 lxd vms

TorbenMartin · 2024-07-23T17:49:41Z

I think i found out.

lxd1v4

root@lxd1v4:~# snap install lxd --channel=6.1/stable
root@lxd1v4:~# lxd init
root@lxd1v4:~# lxc cluster add lxd2v4

lxd2v4

root@lxd2v4:~# snap install lxd --channel=6.1/stable
root@lxd2v4:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.0.3.234]: 
Are you joining an existing cluster? (yes/no) [default=no]: yes
Please provide join token: eyJzZXJ2ZXJfbmFtZSI6Imx4ZDJ2NCIsImZpbmdlcnByaW50IjoiZjU0MzM0MmMxMjBkNjY1N2Q2NDQxYzk2MGJkMzJhY2M1MjlhMGRjYjE0MjJlNzBiM2UyYzJjZjYwNjVlMjEzMSIsImFkZHJlc3NlcyI6WyIxMC4wLjMuMjMzOjg0NDMiXSwic2VjcmV0IjoiZDE0OTk2MGQ5Y2IyYjU4OGVhMTg1YmRhN2M3YjZmMGE5NWZhYjA5YzJlODA5ZWQ5MzM0MzFkYjU1OGZiOGVhMiIsImV4cGlyZXNfYXQiOiIyMDI0LTA3LTIzVDIwOjQyOjM1Ljk0MjU5OTAzNFoifQ==
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "size" property for storage pool "local": 
Choose "source" property for storage pool "local": 
Choose "zfs.pool_name" property for storage pool "local": 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]

my first Process was.

apt-get update
snap remove --purge lxd
snap install lxd --channel=6.1/stable
snap set lxd ui.enable=true
snap refresh --hold lxd
snap restart --reload lxd

Its sounding like you have some existing state.

@boltmark do you have a reproducer?

To reproduce, initialize lxd on a node and add a storage pool other than the default. Then, run lxd init again and attempt to add the node to a cluster. With fresh VMs that have not previously initialized lxd, this error does not occur. I believe this is expected behavior, but please correct me if I am wrong @tomponline.

@TorbenMartin Can you confirm that this works when using fresh VMs that have not previously initialized lxd or do not have any existing storage volumes other than default?

....

maybe this not work.
will "snap set lxd ui.enable=true" init LXD before i run "lxd init"?

tomponline · 2024-07-23T17:51:56Z

Ah this might be the issue:

snap restart --reload lxd

There is s bug in snapd that incorrectly initializes lxd on reload.

Try doing:

sudo systemctl reload snap.lxd.daemon instead

tomponline · 2024-07-23T17:55:40Z

@boltmark please can you post your reproducer steps in full

Below is the only way that I can reproduce a similar error, but this seems different from what @TorbenMartin is running into as he is not using lxd init multiple times.

Host 1:

root@vm1:~# snap install lxd --channel=6.1/stable
2024-07-23T16:57:04Z INFO Waiting for automatic snapd restart...
lxd (6.1/stable) 6.1-c14927a from Canonical✓ installed
root@vm1:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.252.72.173]: 
Are you joining an existing cluster? (yes/no) [default=no]: 
What member name should be used to identify this server in the cluster? [default=vm1]: 
Do you want to configure a new local storage pool? (yes/no) [default=yes]: 
Name of the storage backend to use (dir, lvm, zfs, btrfs) [default=zfs]: 
Create a new ZFS pool? (yes/no) [default=yes]: 
Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]: 
Size in GiB of the new loop device (1GiB minimum) [default=5GiB]: 
Do you want to configure a new remote storage pool? (yes/no) [default=no]: 
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]: 
Would you like to create a new Fan overlay network? (yes/no) [default=yes]: 
What subnet should be used as the Fan underlay? [default=auto]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 

root@vm1:~# lxc cluster add vm2
To start your first container, try: lxc launch ubuntu:24.04
Or for a virtual machine: lxc launch ubuntu:24.04 --vm

Member vm2 join token:
eyJzZXJ2ZXJfbmFtZSI6InZtMyIsImZpbmdlcnByaW50IjoiZWE1ZDY5ZDEyYmMwZTBiZWIwZGVkNDAxZGI0OTVjZmViN2Y3YTVjNzMyYjYxOTNiNjI4NDQ0NGNiNjVlZWE4NyIsImFkZHJlc3NlcyI6WyIxMC4yNTIuNzIuMTczOjg0NDMiXSwic2VjcmV0IjoiNGNkMDhiODQwNTExNWFjOTEwMzk0ZDlmOTE4ZjNlZThlODkwOWZlOTQ5ZWZlNGFkZTM1MjBmMjY0OTlhNzc4YSIsImV4cGlyZXNfYXQiOiIyMDI0LTA3LTIzVDIwOjIzOjU3Ljk2MDgwMDcxWiJ9

Host 2:

root@vm2:~# snap install lxd --channel=6.1/stable
2024-07-23T17:21:58Z INFO Waiting for automatic snapd restart...
lxd (6.1/stable) 6.1-c14927a from Canonical✓ installed
root@vm2:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: 
Do you want to configure a new storage pool? (yes/no) [default=yes]: 
Name of the new storage pool [default=default]: 
Name of the storage backend to use (btrfs, ceph, dir, lvm, powerflex, zfs) [default=zfs]: 
Create a new ZFS pool? (yes/no) [default=yes]: 
Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]: 
Size in GiB of the new loop device (1GiB minimum) [default=5GiB]: 
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to create a new local network bridge? (yes/no) [default=yes]: 
What should the new bridge be called? [default=lxdbr0]: 
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
Would you like the LXD server to be available over the network? (yes/no) [default=no]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 

root@vm2:~# lxc storage create local zfs
To start your first container, try: lxc launch ubuntu:24.04
Or for a virtual machine: lxc launch ubuntu:24.04 --vm

Storage pool local created

root@vm2:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.252.72.128]: 
Are you joining an existing cluster? (yes/no) [default=no]: yes
Please provide join token: eyJzZXJ2ZXJfbmFtZSI6InZtMyIsImZpbmdlcnByaW50IjoiZWE1ZDY5ZDEyYmMwZTBiZWIwZGVkNDAxZGI0OTVjZmViN2Y3YTVjNzMyYjYxOTNiNjI4NDQ0NGNiNjVlZWE4NyIsImFkZHJlc3NlcyI6WyIxMC4yNTIuNzIuMTczOjg0NDMiXSwic2VjcmV0IjoiNGNkMDhiODQwNTExNWFjOTEwMzk0ZDlmOTE4ZjNlZThlODkwOWZlOTQ5ZWZlNGFkZTM1MjBmMjY0OTlhNzc4YSIsImV4cGlyZXNfYXQiOiIyMDI0LTA3LTIzVDIwOjIzOjU3Ljk2MDgwMDcxWiJ9
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "size" property for storage pool "local": 
Choose "source" property for storage pool "local": 
Choose "zfs.pool_name" property for storage pool "local": 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: yes
config: {}
networks: []
storage_pools: []
storage_volumes: []
profiles: []
projects: []
cluster:
  server_name: vm3
  enabled: true
  member_config:
  - entity: storage-pool
    name: local
    key: size
    value: ""
    description: '"size" property for storage pool "local"'
  - entity: storage-pool
    name: local
    key: source
    value: ""
    description: '"source" property for storage pool "local"'
  - entity: storage-pool
    name: local
    key: zfs.pool_name
    value: ""
    description: '"zfs.pool_name" property for storage pool "local"'
  cluster_address: 10.252.72.173:8443
  cluster_certificate: |
    -----BEGIN CERTIFICATE-----
    MIIB3DCCAWGgAwIBAgIRANF5Oq8sSdMLrAG0IN3fHCYwCgYIKoZIzj0EAwMwITEM
    MAoGA1UEChMDTFhEMREwDwYDVQQDDAhyb290QHZtMTAeFw0yNDA3MjMxNjU3MzRa
    Fw0zNDA3MjExNjU3MzRaMCExDDAKBgNVBAoTA0xYRDERMA8GA1UEAwwIcm9vdEB2
    bTEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAARe1Igyywf00FAxI6FRUMSbm2+oVMHz
    fOSW6EjdhhUCrCSoagvXZ94Ig8cbpayO33UtD1+H3UJVSQGVXfrVAAlFLBYDuT09
    VcqTOts26oHLbD68C92T67I9RBdGh61u0JOjXTBbMA4GA1UdDwEB/wQEAwIFoDAT
    BgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMCYGA1UdEQQfMB2CA3Zt
    MYcEfwAAAYcQAAAAAAAAAAAAAAAAAAAAATAKBggqhkjOPQQDAwNpADBmAjEAxUrp
    cGKk4X/Hw1yvrWA0fk7VKFT76NOfUKj1xhbP5pGqqUgg6D0OzGF3oMvc0sFNAjEA
    5D9cyYsNt0LRstiTRFXD/E7IixaAzjR+GXG+B+oo003tK49qCn6J9GlsNdjXhwXk
    -----END CERTIFICATE-----
  server_address: 10.252.72.128:8443
  cluster_password: ""
  cluster_token: ""
  cluster_certificate_path: ""

Error: Failed to join cluster: Failed to initialize member: Failed to initialize storage pools and networks: Failed to update storage pool "local": Config key "size" is cluster member specific

Ah ok so ur initializing both hosts first and then trying to cluster them. This is quite sn unusual scenario, but lxd should be doing some consistency checks iirc to check they are compatible.

In this case it looks like the two local storage pool settings are conflicting.

TorbenMartin · 2024-07-23T18:38:14Z

Ah this might be the issue:

snap restart --reload lxd

There is s bug in snapd that incorrectly initializes lxd on reload.

Try doing:

sudo systemctl reload snap.lxd.daemon instead

My Process now:
snap install lxd --channel=6.1/stable
snap set lxd ui.enable=true
snap refresh --hold lxd
systemctl reload snap.lxd.daemon

Result:
snap.lxd.daemon.service is not active, cannot reload.
So may when it is not active... may this is the Problem.

root@lxd1v5:~# systemctl restart snap.lxd.daemon
root@lxd1v5:~# systemctl status snap.lxd.daemon
● snap.lxd.daemon.service - Service for snap application lxd.daemon
     Loaded: loaded (/etc/systemd/system/snap.lxd.daemon.service; static)
     Active: active (running) since Tue 2024-07-23 18:33:18 UTC; 5s ago
TriggeredBy: ● snap.lxd.daemon.unix.socket
   Main PID: 2410 (daemon.start)
      Tasks: 0 (limit: 4643)
     Memory: 37.0M (peak: 56.1M)
        CPU: 1.070s
     CGroup: /system.slice/snap.lxd.daemon.service
             ‣ 2410 /bin/sh /snap/lxd/29551/commands/daemon.start

Jul 23 18:33:19 lxd1v5 lxd.daemon[2556]: - proc_stat
Jul 23 18:33:19 lxd1v5 lxd.daemon[2556]: - proc_swaps
Jul 23 18:33:19 lxd1v5 lxd.daemon[2556]: - proc_uptime
Jul 23 18:33:19 lxd1v5 lxd.daemon[2556]: - proc_slabinfo
Jul 23 18:33:19 lxd1v5 lxd.daemon[2556]: - shared_pidns
Jul 23 18:33:19 lxd1v5 lxd.daemon[2556]: - cpuview_daemon
Jul 23 18:33:19 lxd1v5 lxd.daemon[2556]: - loadavg_daemon
Jul 23 18:33:19 lxd1v5 lxd.daemon[2556]: - pidfds
Jul 23 18:33:20 lxd1v5 lxd.daemon[2410]: => Starting LXD
Jul 23 18:33:21 lxd1v5 lxd.daemon[2566]: time="2024-07-23T18:33:21Z" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network prio>
lines 1-21/21 (END)

Finaly i can join the Cluster.
I accepted all default Options on "lxd init"

root@lxd1v5:~# lxc cluster add lxd2v5
To start your first container, try: lxc launch ubuntu:24.04
Or for a virtual machine: lxc launch ubuntu:24.04 --vm

Member lxd2v5 join token:
eyJzZXJ2ZXJfbmFtZSI6Imx4ZDJ2NSIsImZpbmdlcnByaW50IjoiZGQ3M2M4ZTZhNjUyMzRjNGNmNjk2NTRlMmIyYzRhN2YzNmIyZGNiN2NlNWZlZWM5OWU0MmExYTA2NGU5ZjEyYyIsImFkZHJlc3NlcyI6WyIxMC4wLjMuMjM1Ojg0NDMiXSwic2VjcmV0IjoiYTdkYzZjNGFjNWIwMjFlMWUyM2EzN2RiZDRhMGQ5ZTZkMjZmMzNlNTdmNTU4MzU1ZTY1MjM4MjVjMTNmMGYyOCIsImV4cGlyZXNfYXQiOiIyMDI0LTA3LTIzVDIxOjM0OjQ1LjYzOTI0MDM5NFoifQ==

Join success:

root@lxd2v5:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.0.3.236]: 
Are you joining an existing cluster? (yes/no) [default=no]: yes
Please provide join token: eyJzZXJ2ZXJfbmFtZSI6Imx4ZDJ2NSIsImZpbmdlcnByaW50IjoiZGQ3M2M4ZTZhNjUyMzRjNGNmNjk2NTRlMmIyYzRhN2YzNmIyZGNiN2NlNWZlZWM5OWU0MmExYTA2NGU5ZjEyYyIsImFkZHJlc3NlcyI6WyIxMC4wLjMuMjM1Ojg0NDMiXSwic2VjcmV0IjoiYTdkYzZjNGFjNWIwMjFlMWUyM2EzN2RiZDRhMGQ5ZTZkMjZmMzNlNTdmNTU4MzU1ZTY1MjM4MjVjMTNmMGYyOCIsImV4cGlyZXNfYXQiOiIyMDI0LTA3LTIzVDIxOjM0OjQ1LjYzOTI0MDM5NFoifQ==
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "size" property for storage pool "local": 
Choose "source" property for storage pool "local": 
Choose "zfs.pool_name" property for storage pool "local": 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:

tomponline · 2024-07-23T18:54:15Z

snap.lxd.daemon.service is not active, cannot reload.
So may when it is not active... may this is the Problem.

If you've not started lxd you dont need to reload it.

tomponline · 2024-07-23T18:55:35Z

Finaly i can join the Cluster.

Ok great so the main issue is that "snap restart lxd" causes lxd to be initialized if not already which then reveals the issue @boltmark has identified.

TorbenMartin · 2024-07-23T19:11:17Z

snap.lxd.daemon.service is not active, cannot reload.
So may when it is not active... may this is the Problem.

If you've not started lxd you dont need to reload it.

https://ubuntu.com/blog/lxd_ui

i didnt know that it need to be active started or initialize by myself before i enable the UI

tomponline · 2024-07-23T21:01:58Z

It doesn't since 5.21 onwards

tomponline · 2024-07-23T21:03:51Z

The snap restart issue is here btw https://bugs.launchpad.net/snapd/+bug/2028141

tomponline added the Bug Confirmed to be a bug label Jul 22, 2024

tomponline added this to the lxd-6.2 milestone Jul 22, 2024

tomponline assigned boltmark Jul 22, 2024

tomponline added Incomplete Waiting on more information from reporter and removed Bug Confirmed to be a bug labels Jul 23, 2024

boltmark closed this as completed Jul 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to join LXD Cluster on 6.1 #13793

Unable to join LXD Cluster on 6.1 #13793

TorbenMartin commented Jul 22, 2024

TorbenMartin commented Jul 22, 2024 •

edited

Loading

TorbenMartin commented Jul 22, 2024 •

edited

Loading

TorbenMartin commented Jul 22, 2024

boltmark commented Jul 23, 2024

boltmark commented Jul 23, 2024 •

edited by tomponline

Loading

TorbenMartin commented Jul 23, 2024

tomponline commented Jul 23, 2024

tomponline commented Jul 23, 2024 •

edited

Loading

tomponline commented Jul 23, 2024

TorbenMartin commented Jul 23, 2024

tomponline commented Jul 23, 2024

boltmark commented Jul 23, 2024 •

edited

Loading

tomponline commented Jul 23, 2024

TorbenMartin commented Jul 23, 2024

TorbenMartin commented Jul 23, 2024

tomponline commented Jul 23, 2024

tomponline commented Jul 23, 2024

TorbenMartin commented Jul 23, 2024 •

edited

Loading

boltmark commented Jul 23, 2024

tomponline commented Jul 23, 2024

TorbenMartin commented Jul 23, 2024

tomponline commented Jul 23, 2024

tomponline commented Jul 23, 2024

TorbenMartin commented Jul 23, 2024

tomponline commented Jul 23, 2024

tomponline commented Jul 23, 2024

TorbenMartin commented Jul 23, 2024 •

edited

Loading

tomponline commented Jul 23, 2024

tomponline commented Jul 23, 2024

Unable to join LXD Cluster on 6.1 #13793

Unable to join LXD Cluster on 6.1 #13793

Comments

TorbenMartin commented Jul 22, 2024

TorbenMartin commented Jul 22, 2024 • edited Loading

TorbenMartin commented Jul 22, 2024 • edited Loading

TorbenMartin commented Jul 22, 2024

boltmark commented Jul 23, 2024

boltmark commented Jul 23, 2024 • edited by tomponline Loading

TorbenMartin commented Jul 23, 2024

tomponline commented Jul 23, 2024

tomponline commented Jul 23, 2024 • edited Loading

tomponline commented Jul 23, 2024

TorbenMartin commented Jul 23, 2024

tomponline commented Jul 23, 2024

boltmark commented Jul 23, 2024 • edited Loading

tomponline commented Jul 23, 2024

TorbenMartin commented Jul 23, 2024

TorbenMartin commented Jul 23, 2024

tomponline commented Jul 23, 2024

tomponline commented Jul 23, 2024

TorbenMartin commented Jul 23, 2024 • edited Loading

boltmark commented Jul 23, 2024

tomponline commented Jul 23, 2024

TorbenMartin commented Jul 23, 2024

tomponline commented Jul 23, 2024

tomponline commented Jul 23, 2024

TorbenMartin commented Jul 23, 2024

tomponline commented Jul 23, 2024

tomponline commented Jul 23, 2024

TorbenMartin commented Jul 23, 2024 • edited Loading

tomponline commented Jul 23, 2024

tomponline commented Jul 23, 2024

TorbenMartin commented Jul 22, 2024 •

edited

Loading

TorbenMartin commented Jul 22, 2024 •

edited

Loading

boltmark commented Jul 23, 2024 •

edited by tomponline

Loading

tomponline commented Jul 23, 2024 •

edited

Loading

boltmark commented Jul 23, 2024 •

edited

Loading

TorbenMartin commented Jul 23, 2024 •

edited

Loading

TorbenMartin commented Jul 23, 2024 •

edited

Loading