Skip to content

Commit

Permalink
user-vms.md: clarify some points, tidy up (#39)
Browse files Browse the repository at this point in the history
  • Loading branch information
wizzdom authored Apr 4, 2024
1 parent 96c5a88 commit 070ddd1
Showing 1 changed file with 15 additions and 10 deletions.
25 changes: 15 additions & 10 deletions docs/services/user-vms.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ The host needs to be configured to allow the VMs to communicate with each other.

### Create a Bridge

To create a bridge that qemu can use to place the guest (vm) onto the same network as the host, follow the instructions listed [here](https://wiki.archlinux.org/title/Network_bridge#With_iproute2) for iproute2, summarised below.
To create a bridge that qemu can use to place the guest (VM) onto the same network as the host, follow the instructions listed [here](https://wiki.archlinux.org/title/Network_bridge#With_iproute2) for `iproute2`, summarised below.

We need to create a bridge interface on the host.

Expand All @@ -36,27 +36,27 @@ We'll be adding a physical interface to this bridge to allow it to communicate w
sudo ip link set eno1 master br0
```

You'll need to assign an IP address to the bridge interface. This will be used as the default address for the host. You can do this with DHCP or by assigning a static IP address. The best way to do this is to create a DHCP static lease on the UDM for the bridge interface MAC address.
You'll need to assign an IP address to the bridge interface. This will be used as the default address for the host. You can do this with DHCP or by assigning a static IP address. The best way to do this is to create a DHCP static lease on the [UDM](../hardware/network/mordor.md) for the bridge interface MAC address.

> [!NOTE] TODO
> [!NOTE]
> TODO: Find out why connectivity seems to be lost when the bridge interface receives an address before the physical interface.
> If connectivity is lost, release the addresses from both the bridge and the physical interface (in that order) with `sudo dhclient -v -r <iface>` and then run `sudo dhclient -v <iface>` to assign the bridge interface an address.
### Add the VMs to the Bridge

The configuration of the qemu network options in the job file will create a new tap interface and add it to the bridge and the VM. I advise you for your own sanity to never touch the network options, they will only cause you pain.

For others looking, this configuration is specific to QEMU only.
For others looking, this configuration is specific to *QEMU only*.

```bash
qemu-system-x86_64 ... -netdev bridge,id=hn0 -device virtio-net-pci,netdev=hn0,id=nic1
```

This will assign the VM an address on the external network. The VM will be able to communicate with the host and other VMs in the network.

You must also add `allow br0` to `/etc/qemu/bridge.conf` to allow qemu to add the tap interfaces tothe bridge. [Source](https://wiki.qemu.org/Features/HelperNetworking)
You must also add `allow br0` to `/etc/qemu/bridge.conf` to allow qemu to add the tap interfaces to the bridge. [Source](https://wiki.qemu.org/Features/HelperNetworking)

The VMs, once connected to the bridge, will be assigned an address via DHCP. You can assign a static IP address to the VMs by adding a DHCP static lease on the UDM for the VMs MAC address. You can get the address of a VM by checking the nomad alloc logs for that VM and searching for `ens3`.
The VMs, once connected to the bridge, will be assigned an address via DHCP. You can assign a static IP address to the VMs by adding a DHCP static lease on the [UDM](../hardware/network/mordor.md) for the VMs MAC address. You can get the address of a VM by checking the `nomad alloc logs` for that VM and searching for `ens3`.

```bash
nomad job status distro-vm | grep "Node ID" -A 1 | tail -n 1 | cut -d " " -f 1
Expand All @@ -67,21 +67,26 @@ nomad alloc logs <alloc-id> | grep -E "ens3.*global" | cut -d "|" -f 4 | xargs

## Configuring the VMs

The VMs are configured with cloud-init. Their docs are pretty good, so I won't repeat them here. The files can be served by any HTTP server, and the address is placed into the job file in the QEMU options.
The VMs are configured with cloud-init. Their [docs](https://cloudinit.readthedocs.io/en/latest/) are pretty good, so I won't repeat them here. The files can be served by any HTTP server, and the address is placed into the job file in the QEMU options.

```hcl title="Nomad"
...
args = [
...
"virtio-net-pci,netdev=hn0,id=nic1,mac=52:54:84:ba:49:22",
"virtio-net-pci,netdev=hn0,id=nic1,mac=52:54:84:ba:49:22", # make sure this MAC address is unique!!
"-smbios",
"type=1,serial=ds=nocloud-net;s=http://136.206.16.5:8000/",
]
...
```

> [!NOTE] Note!
> If you're running multiple VMS on the same host make sure to set different MAC addresses for each VM, otherwise you'll have a bad time.
Here in the args block:

- we define that the VM will have a network device using the `virtio` driver, we pass it an `id` and a random ***unique*** MAC address
- we tell it to use `smbios` type 1 and to grab its `cloud-init` configs from `http://136.206.16.5:8000/`

> [!NOTE]
> If you're running multiple VMs on the same network make sure to set different MAC addresses for each VM, otherwise you'll have a bad time.
## Creating a New VM

Expand Down

0 comments on commit 070ddd1

Please sign in to comment.