From 2c72153f06f66494696aefcaecb646a50e4e8f64 Mon Sep 17 00:00:00 2001 From: wizzdom Date: Fri, 27 Sep 2024 14:19:29 +0100 Subject: [PATCH] procedures: update post-powercut doc (#45) --- docs/procedures/post-powercut.md | 31 ++++++++++++++++++++----------- docs/services/user-vms.md | 2 +- 2 files changed, 21 insertions(+), 12 deletions(-) diff --git a/docs/procedures/post-powercut.md b/docs/procedures/post-powercut.md index 856905a3..3e9d67f2 100644 --- a/docs/procedures/post-powercut.md +++ b/docs/procedures/post-powercut.md @@ -2,24 +2,33 @@ id: post-powercut aliases: - Post-powercut Todo List -tags: [] +tags: + - powercut + - todo created: 2023-12-05T01:36:11 -modified: 2024-01-31T08:23:37 -title: Post-Powercut Todo List +modified: 2024-09-27T13:35:28 +title: post-powercut --- # Post-powercut Todo List A list of things that should be done/checked immediately after a power cut: -- Check KVM, hit ctrl+D on minerva to make sure it boots. -- Check KVM, hit F1 on sprout to make sure it boots -- Check KVM, sometimes you need to press F1 on carbon for it to boot -- Stop Exim on the mail server (Morpheus) until minerva (NFS) is online. -- If LDAP is down, you'll need to use the ALOM to do the next step. -- Check that ldapclient started (svcs -xv). If it didn't, run svcadm clear ldap/client to make it start. This usually happens because murphy comes back before morpheus does, and the LDAP client won't start due to lack of an LDAP server. -- Apache on [hardcase](../hardware/nix/hardcase.md) sometimes tries to start before networking is finished starting. To fix it, disable/re-enable it a few times. This usually makes it turn on. -- [paphos](../hardware/paphos.md) is old and sometimes its time will become out of sync. To make sure its time is accurate, run: +- Ensure the [`aperture`](../hardware/aperture/index.md) servers have the correct IP addresses: + - `eno1` should have the internal IP address (`10.10.0.0/24`) - this should be reserved by DHCP on [`mordor`](../hardware/network/mordor.md) + - `eno2` should have *no IP address* + - `br0` should have the external IP address (`136.206.16.0/24`) - this should also be reserved by DHCP on [`mordor`](../hardware/network/mordor.md) +- If the [`bastion-vm`](../services/bastion-vm.md) fails to start, check: + - `/storage` is mounted `rw` on each [`aperture`](docs/hardware/aperture/index.md) server + - `br0` is present and configured on each [`aperture`](docs/hardware/aperture/index.md) server + - `vm-resources.service.consul` is running and `http://vm-resources.service.consul:8000/bastion/bastion-vm-latest.qcow2` is accessible + - if the `latest` symlink points to a corrupted image, `ln -sf` it to an earlier one +- All the [`nixos`](docs/procedures/nixos.md) boxes rely on [`DNS`](docs/services/bind.md) for [`ldap`](docs/services/ldap.md) and [`nfs`](docs/services/nfs.md): + - Make sure bind is running on [`paphos`](docs/hardware/paphos.md) + - mount `/storage` + - `systemctl restart` `httpd`, `php-fpm-rbusers-*` and `ldap` +- Apache on [`hardcase`](../hardware/nix/hardcase.md) sometimes tries to start before networking is finished starting. To fix it, disable/re-enable it a few times. This usually makes it turn on. +- [`paphos`](../hardware/paphos.md) is old and sometimes its time will become out of sync. To make sure its time is accurate, run: ```bash sudo service ntp restart diff --git a/docs/services/user-vms.md b/docs/services/user-vms.md index 34d95fad..774206f3 100644 --- a/docs/services/user-vms.md +++ b/docs/services/user-vms.md @@ -38,7 +38,7 @@ sudo ip link set dev br0 up We'll be adding a physical interface to this bridge to allow it to communicate with the external ([UDM](../hardware/network/mordor.md)) network. ```bash -sudo ip link set eno1 master br0 +sudo ip link set eno2 master br0 ``` You'll need to assign an IP address to the bridge interface. This will be used as the default address for the host. You can do this with DHCP or by assigning a static IP address. The best way to do this is to create a DHCP static lease on the [UDM](../hardware/network/mordor.md) for the bridge interface MAC address.