When the PR environments are no longer being created, and the
lagoon-core-broker-<n> pods are missing or not running, and the container logs
contain errors like Error while waiting for Mnesia tables: {timeout_waiting_for_tables.
This situation is caused by the RabbitMQ broker not starting correctly.
- A dplsh session with DPLPLAT_ENV exported .
You are going to exec into the pod and stop the RabbitMQ application, and then
start it with the force_boot
feature, so that it can
perform its Mnesia sync correctly.
Exec into the pod:
dplsh:~/host_mount$ kubectl -n lagoon-core exec -ti pod/lagoon-core-broker-0 -- shStop RabbitMQ:
/ $ rabbitmqctl stop_app
Stopping rabbit application on node [email protected]
core-broker-headless.lagoon-core.svc.cluster.local ...Start it immediately after using the force_boot flag:
/ $ rabbitmqctl force_bootThen exit the shell and check the container logs for one of the broker pods. It should start without errors.