Race condition between the k8s-cms containers and database init.

[mrzzy]

There is a race condition between:

• the initdb operation on the database, which setups the schema, db credentials etc.
• the CMS containers trying to connect to the database.

This can been seen in a clean docker-compose up after a docker-compose down -v to clear any prior state: Shown in this pipeline run , we can observe the race condition in action:

• the initdb operations to create cmsuser requires the database to be running.
• the database runs, signalling to the wait-for-it scripts to start running the cms containers.
• the initdb operations to create the cmsuser are not yet complete.
• the k8s-cms containers attempt connect to the database with the credentials and fail.

The problem is mostly clearly observed in the docker-compose setup as it does not have a built in way to restart faulty containers. The health probes in the Kubernetes setup will detect that the container is not functioning correctly due the DB auth failure and restart the container. By then the initdb should already completed and the problem automatically resolved.

We should find a way to resolve this race condition as docker-compose setup requires two docker-compose ups in order to function correctly.

Possible solutions:

• changing to the initdb DB port temporary to something else, so that wait-for-it scripts will continue to wait for when the DB is actually read for external connections.