From 9579378ba063237baa5b73380eb8e9f0a28529ee Mon Sep 17 00:00:00 2001 From: Jusong Yu Date: Thu, 11 Jul 2024 15:22:44 +0200 Subject: [PATCH] Docker: Replace sleep with `s6-notifyoncheck` (#6475) For the `aiida-core-with-services` image where the services are part of the image, we cannot rely on the health check for the services provided by docker-build as is used for the `aiida-core-base` case. Instead, a simple sleep was added to the `aiida-prepare.sh` script that sets up the profile, to make sure the services are up before the profile is created. This solution is neither elegant nor robust. Here the sleep approach is replaced by `s6-notifyoncheck`. This hook allows blocking the startup from continuing until a script returns a 0 exit code. The script in question first calls `rabbitmq-diagnostics ping` to make sure the RabbitMQ server is even up, followed by a call to `rabbitmq-diagnostics check_running`. If the latter returns 0, it means RabbitMQ is up and running and the script returns 0 as well, which will trigger `s6-notifyoncheck` to continue with the rest of the startup. Note that `rabbitmq-diagnostics is_running` could not be used as that command sometimes returns 0 even if the service is not ready at all. Co-authored-by: Sebastiaan Huber --- .../s6-assets/init/aiida-prepare.sh | 5 ----- .../s6-assets/s6-rc.d/rabbitmq/data/check | 15 +++++++++++++++ .../s6-assets/s6-rc.d/rabbitmq/notification-fd | 1 + .../s6-assets/s6-rc.d/rabbitmq/run | 13 +++++++++++-- 4 files changed, 27 insertions(+), 7 deletions(-) create mode 100755 .docker/aiida-core-with-services/s6-assets/s6-rc.d/rabbitmq/data/check create mode 100644 .docker/aiida-core-with-services/s6-assets/s6-rc.d/rabbitmq/notification-fd diff --git a/.docker/aiida-core-base/s6-assets/init/aiida-prepare.sh b/.docker/aiida-core-base/s6-assets/init/aiida-prepare.sh index 30eefc3999..a9e54142fa 100755 --- a/.docker/aiida-core-base/s6-assets/init/aiida-prepare.sh +++ b/.docker/aiida-core-base/s6-assets/init/aiida-prepare.sh @@ -18,11 +18,6 @@ verdi config set warnings.development_version False # If the environment variable `SETUP_DEFAULT_AIIDA_PROFILE` is not set, set it to `true`. if [[ ${SETUP_DEFAULT_AIIDA_PROFILE:-true} == true ]] && ! verdi profile show ${AIIDA_PROFILE_NAME} &> /dev/null; then - # For the container that includes the services, this script is called as soon as the RabbitMQ startup script has - # been launched, but it can take a while for the service to come up. If ``verdi presto`` is called straight away - # it is possible it tries to connect to the service before that and it will configure the profile without a broker. - sleep 5 - # Create AiiDA profile. verdi presto \ --verbosity info \ diff --git a/.docker/aiida-core-with-services/s6-assets/s6-rc.d/rabbitmq/data/check b/.docker/aiida-core-with-services/s6-assets/s6-rc.d/rabbitmq/data/check new file mode 100755 index 0000000000..46eb70ea89 --- /dev/null +++ b/.docker/aiida-core-with-services/s6-assets/s6-rc.d/rabbitmq/data/check @@ -0,0 +1,15 @@ +#!/bin/bash + +rabbitmq-diagnostics ping + +if [ $? -ne 0 ]; then + exit 1 +fi + +rabbitmq-diagnostics check_running + +if [ $? -ne 0 ]; then + exit 1 +fi + +exit 0 diff --git a/.docker/aiida-core-with-services/s6-assets/s6-rc.d/rabbitmq/notification-fd b/.docker/aiida-core-with-services/s6-assets/s6-rc.d/rabbitmq/notification-fd new file mode 100644 index 0000000000..00750edc07 --- /dev/null +++ b/.docker/aiida-core-with-services/s6-assets/s6-rc.d/rabbitmq/notification-fd @@ -0,0 +1 @@ +3 diff --git a/.docker/aiida-core-with-services/s6-assets/s6-rc.d/rabbitmq/run b/.docker/aiida-core-with-services/s6-assets/s6-rc.d/rabbitmq/run index e5752294ff..8a35acd20f 100644 --- a/.docker/aiida-core-with-services/s6-assets/s6-rc.d/rabbitmq/run +++ b/.docker/aiida-core-with-services/s6-assets/s6-rc.d/rabbitmq/run @@ -2,5 +2,14 @@ with-contenv -foreground { s6-echo "Calling /etc/init/rabbitmq.sh" } -rabbitmq-server +foreground { s6-echo "Starting RMQ server and notifying back when the service is ready" } + + +# For the container that includes the services, aiida-prepare.sh script is called as soon as the RabbitMQ startup script has +# been launched, but it can take a while for the RMQ service to come up. If ``verdi presto`` is called straight away +# it is possible it tries to connect to the service before that and it will configure the profile without a broker. +# Here we use s6-notifyoncheck to do the polling healthy check of the readyness of RMQ service. +# +# -w 500: 500 ms between two invocations of ./data/check + +s6-notifyoncheck -w 500 rabbitmq-server