diff --git a/docs/services/user-vms.md b/docs/services/user-vms.md index 4b51052e..f72bfb2f 100644 --- a/docs/services/user-vms.md +++ b/docs/services/user-vms.md @@ -21,7 +21,7 @@ The host needs to be configured to allow the VMs to communicate with each other. ### Create a Bridge -To create a bridge that qemu can use to place the guest (vm) onto the same network as the host, follow the instructions listed [here](https://wiki.archlinux.org/title/Network_bridge#With_iproute2) for iproute2, summarised below. +To create a bridge that qemu can use to place the guest (VM) onto the same network as the host, follow the instructions listed [here](https://wiki.archlinux.org/title/Network_bridge#With_iproute2) for `iproute2`, summarised below. We need to create a bridge interface on the host. @@ -36,9 +36,9 @@ We'll be adding a physical interface to this bridge to allow it to communicate w sudo ip link set eno1 master br0 ``` -You'll need to assign an IP address to the bridge interface. This will be used as the default address for the host. You can do this with DHCP or by assigning a static IP address. The best way to do this is to create a DHCP static lease on the UDM for the bridge interface MAC address. +You'll need to assign an IP address to the bridge interface. This will be used as the default address for the host. You can do this with DHCP or by assigning a static IP address. The best way to do this is to create a DHCP static lease on the [UDM](../hardware/network/mordor.md) for the bridge interface MAC address. -> [!NOTE] TODO +> [!NOTE] > TODO: Find out why connectivity seems to be lost when the bridge interface receives an address before the physical interface. > If connectivity is lost, release the addresses from both the bridge and the physical interface (in that order) with `sudo dhclient -v -r ` and then run `sudo dhclient -v ` to assign the bridge interface an address. @@ -46,7 +46,7 @@ You'll need to assign an IP address to the bridge interface. This will be used a The configuration of the qemu network options in the job file will create a new tap interface and add it to the bridge and the VM. I advise you for your own sanity to never touch the network options, they will only cause you pain. -For others looking, this configuration is specific to QEMU only. +For others looking, this configuration is specific to *QEMU only*. ```bash qemu-system-x86_64 ... -netdev bridge,id=hn0 -device virtio-net-pci,netdev=hn0,id=nic1 @@ -54,9 +54,9 @@ qemu-system-x86_64 ... -netdev bridge,id=hn0 -device virtio-net-pci,netdev=hn0,i This will assign the VM an address on the external network. The VM will be able to communicate with the host and other VMs in the network. -You must also add `allow br0` to `/etc/qemu/bridge.conf` to allow qemu to add the tap interfaces tothe bridge. [Source](https://wiki.qemu.org/Features/HelperNetworking) +You must also add `allow br0` to `/etc/qemu/bridge.conf` to allow qemu to add the tap interfaces to the bridge. [Source](https://wiki.qemu.org/Features/HelperNetworking) -The VMs, once connected to the bridge, will be assigned an address via DHCP. You can assign a static IP address to the VMs by adding a DHCP static lease on the UDM for the VMs MAC address. You can get the address of a VM by checking the nomad alloc logs for that VM and searching for `ens3`. +The VMs, once connected to the bridge, will be assigned an address via DHCP. You can assign a static IP address to the VMs by adding a DHCP static lease on the [UDM](../hardware/network/mordor.md) for the VMs MAC address. You can get the address of a VM by checking the `nomad alloc logs` for that VM and searching for `ens3`. ```bash nomad job status distro-vm | grep "Node ID" -A 1 | tail -n 1 | cut -d " " -f 1 @@ -67,21 +67,26 @@ nomad alloc logs | grep -E "ens3.*global" | cut -d "|" -f 4 | xargs ## Configuring the VMs -The VMs are configured with cloud-init. Their docs are pretty good, so I won't repeat them here. The files can be served by any HTTP server, and the address is placed into the job file in the QEMU options. +The VMs are configured with cloud-init. Their [docs](https://cloudinit.readthedocs.io/en/latest/) are pretty good, so I won't repeat them here. The files can be served by any HTTP server, and the address is placed into the job file in the QEMU options. ```hcl title="Nomad" ... args = [ ... - "virtio-net-pci,netdev=hn0,id=nic1,mac=52:54:84:ba:49:22", + "virtio-net-pci,netdev=hn0,id=nic1,mac=52:54:84:ba:49:22", # make sure this MAC address is unique!! "-smbios", "type=1,serial=ds=nocloud-net;s=http://136.206.16.5:8000/", ] ... ``` -> [!NOTE] Note! -> If you're running multiple VMS on the same host make sure to set different MAC addresses for each VM, otherwise you'll have a bad time. +Here in the args block: + +- we define that the VM will have a network device using the `virtio` driver, we pass it an `id` and a random ***unique*** MAC address +- we tell it to use `smbios` type 1 and to grab its `cloud-init` configs from `http://136.206.16.5:8000/` + +> [!NOTE] +> If you're running multiple VMs on the same network make sure to set different MAC addresses for each VM, otherwise you'll have a bad time. ## Creating a New VM