Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker seems to kill the container instead of terminating it #68

Closed
LeSpocky opened this issue Feb 15, 2021 · 5 comments · Fixed by #69
Closed

docker seems to kill the container instead of terminating it #68

LeSpocky opened this issue Feb 15, 2021 · 5 comments · Fixed by #69

Comments

@LeSpocky
Copy link
Contributor

Although #65 claims to have fixed signal handling, it does not work for me. I start the container with a little script like this:

#!/bin/sh
docker pull prosody/prosody:0.11
docker run -d --name prosody --rm \
        -p 8080:80 -p 443:443 -p 5222:5222 -p 5269:5269 \
        -v /srv/prosody/etc/prosody:/etc/prosody \
        -v /srv/prosody/var/lib/prosody:/var/lib/prosody \
        -v /srv/prosody/var/log/prosody:/var/log/prosody \
        -v /var/lib/dehydrated:/var/lib/dehydrated:ro \
        prosody/prosody:0.11

If prosody is shut down gracefully with SIGTERM log output should be like this:

2021-02-15T20:24:34.203097650Z mod_posix                  warn  Received SIGTERM
2021-02-15T20:24:34.203143988Z startup                    info  Shutting down: Received SIGTERM
2021-02-15T20:24:34.203216985Z general                    info  Shutting down...
2021-02-15T20:24:34.203230176Z general                    info  Shutdown status: Cleaning up
2021-02-15T20:24:34.203254906Z general                    info  Shutdown complete

When calling docker stop however it takes the usual 10 seconds and then prosody only gets SIGKILL and there's no log output anymore:

alex@miraculix /srv/prosody % time docker stop prosody
prosody
docker stop prosody  0,06s user 0,02s system 0% cpu 11,536 total

The process list:

alex@miraculix /srv/prosody % docker exec prosody ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 20:36 ?        00:00:00 /bin/bash /entrypoint.sh prosody -F
root        11     1  0 20:36 ?        00:00:00 runuser -u prosody -- prosody -F
prosody     12    11  1 20:36 ?        00:00:00 lua5.2 /usr/bin/prosody -F
root        13     0  0 20:36 ?        00:00:00 ps -ef
@Zash
Copy link
Member

Zash commented Mar 18, 2021

Thanks for the report. I've confirmed this to be the case but I have no idea what to do about it.

@penguineer
Copy link

I believe the problem is the way the entrypoint script is written.

Docker sends the SIGTERM to PID 1 in the container, which must be able to handle this signal. When a shell script starts another process, it must make sure to propagate this signal.

The fact that after 10s SIGKILL is issues is a hint that the signal either didn't get to the prosody process or wasn't processed there.

See also https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#entrypoint

@maz3max
Copy link

maz3max commented Mar 20, 2021

runuser could be the culprit here. It opens a new shell and the signal does not reach prosody. Maybe try setpriv?

@LeSpocky
Copy link
Contributor Author

runuser could be the culprit here. It opens a new shell and the signal does not reach prosody. Maybe try setpriv?

Confirmed with the following small test script:

#!/bin/bash -e

echo "pid: $$"
echo "whoami: $(whoami)"

exec runuser -u alex -- sleep 30

Sending SIGTERM to the pid printed by the script, does not terminated the sleep. The signal is sent to the runuser process which actually has that PID, but that also had started sleep as another child process with a different PID.

horazont added a commit to horazont/prosody-docker that referenced this issue Mar 20, 2021
tini [1] is a minimalistic PID 1 process. It correctly handles
the special jobs which PID 1 (or a reaper process in general)
needs to take care of in addition to correctly processing the
relevant signals.

Fixes prosody#68.

   [1]: https://github.com/krallin/tini
@Zash Zash closed this as completed in #69 Mar 20, 2021
Zash pushed a commit that referenced this issue Mar 20, 2021
tini [1] is a minimalistic PID 1 process. It correctly handles
the special jobs which PID 1 (or a reaper process in general)
needs to take care of in addition to correctly processing the
relevant signals.

Fixes #68.

   [1]: https://github.com/krallin/tini
@LeSpocky
Copy link
Contributor Author

LeSpocky commented Mar 21, 2021

The problem reported here, was indeed fixed, but already with cc88073 and #65. As you can see in the process list in the top post, there's still entrypoint.sh as PID 1. The only logical conclusion ist, the image prosody/prosody:0.11 on which my image is based on, is not build with the recent master of prosody-docker, but with some old state before cc88073. I could prove that by modifying (and copying) entrypoint.sh in my own image.

I'm not happy with the graceful shutdown yet, but that would be off topic here, since the actual problem reported was invalid in the first place.

LeSpocky added a commit to LeSpocky/prosody-docker that referenced this issue Mar 21, 2021
This reverts commit 31d6d84.

While tini successfully forwards signals, this leads to `runuser`
killing prosody now. The container does terminate in 10 seconds, so
Docker is happy and you could argue that actually fixes prosody#68, but it's no
graceful shutdown. The revert is done because it's easier to apply a
real fix without tini.
Zash pushed a commit that referenced this issue Mar 27, 2021
This reverts commit 31d6d84.

While tini successfully forwards signals, this leads to `runuser`
killing prosody now. The container does terminate in 10 seconds, so
Docker is happy and you could argue that actually fixes #68, but it's no
graceful shutdown. The revert is done because it's easier to apply a
real fix without tini.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants