Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ceph Version / ansible instructions #6

Merged
merged 1 commit into from
May 3, 2024
Merged

Ceph Version / ansible instructions #6

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 44 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
This is a collection of Ansible playbooks, Terraform configurations and scripts to deploy and operate Incus clusters.

## How to get the test setup run:

### Install incus and OpenTofu
Install incus stable or LTS on your system from the [zabbly/incus](https://github.com/zabbly/incus) release and initialize it on your local machine.

Expand Down Expand Up @@ -45,3 +44,47 @@ Run the Playbooks:
```
ansible-playbook deploy.yaml
```

NOTE: When re-deploying the same cluster (e.g. following a `terraform
destroy`), you need to make sure to also clear any local state from the
`data` directory, failure to do so will cause Ceph/OVN to attempt
connection to the previously deployed systems which will cause the
deployment to get stuck.

## Deploying against production systems
### Requirements (when using Incus with both Ceph and OVN)

- At least 3 servers
- One main network interface (or bond/VLAN), referred to as `enp5s0` in the examples
- One additional network interface (or bond/VLAN) to use for ingress into OVN, referred to as `enp6s0` in the examples
- Configured IPv4/IPv6 subnets on that additional network interface, in the examples, we're using:
- IPv4 subnet: `172.31.254.0/24`
- IPv4 gaterway: `172.31.254.1`
- IPv6 subnet: `fd00:1e4d:637d:1234::/64`
- IPv6 gateway: `fd00:1e4d:637d:1234::1`
- DNS server: `1.1.1.1`
- A minimum of 3 disks (or partitions) on distinct servers across the cluster for consumption by Ceph, in the examples, we're using (on each server):
- `nvme-QEMU_NVMe_Ctrl_incus_disk1`
- `nvme-QEMU_NVMe_Ctrl_incus_disk2`
- A minimum of 1 disk (or partition) on each server for local storage, that's `/dev/disk/by-id/nvme-QEMU_NVMe_Ctrl_incus_disk3` in the examples

### Configuring Ansible

With a deployment against physical servers, Terraform isn't currently used at all.
Ansible will be used to deploy Ceph, OVN and Incus on the servers.

You'll need to create a new `hosts.yaml` which you can base on the example one provided.

You'll then need to do the following changes at minimum:
- Generate a new `ceph_fsid` (use `uuidgen`)
- Set a new `incus_name`
- Set a new `ovn_name`
- Update the number and name of servers to match the FQDN of your machines
- Ensure that you have 3 servers with the `mon` `ceph_role` and 3 servers with the `central` `ovn_role`
- Update the connection details to fit your deployment:
- Unset `ansible_connection`, `ansible_incus_remote`, `ansible_user` and `ansible_become` as those are specific to our test environment
- Set the appropriate connection information to access your servers (`ansible_connection`, `ansible_user`, SSH key, ...)
- Update the list of ceph and local disks for each servers (look at `/dev/disk/by-id` for identifiers)
- Tweak the `incus_init` variable to match your environment

You'll find more details about the Ansible configuration options in [ansible/README.md](ansible/README.md).
Loading