Sending a signal to the process inside the container

[vrenjith]

We are looking at receiving a signal for the process started by containerpilot (this is in the context of nomad re-rendering a template inside the docker image and sends CP a signal of our choice. We need to inform the process somehow about this).

I read through #283 and related ones, but was unable to find a clear way to achieve this. Any hints on how to go about doing this?

[vrenjith]

@tgross Could you please help?

[jwreagor]

Hello @vrenjith, without being able to look an example Nomad job definition, Dockerfile, and/or ContainerPilot configuration, it's hard to completely recommend a solution. But I take it you're referencing Nomad's template configuration stanza and would like to send a change_signal after the template has been re-rendered, correct?

This might step outside ContainerPilot's existing use case as it is normally used by itself to render configuration templates not unlike how Nomad is achieving this same functionality using consul-template internally. You probably could do the same thing only using ContainerPilot, but with that said, they might not mix well together and may be redundant.

IMHO Nomad supersedes ContainerPilot as it shares many of the same features.

[misterbisson]

@vrenjith what signals can your application accept? ContainerPilot can run any executable you want, and that can include kill -1 <PID> or whatever signal is necessary.

[jwreagor]

As @misterbisson duly noted in an offline chat, #195 contains much of the existing thought around our lack of signals in ContainerPilot v3.x. That said, I'm still interested in exploring/defining features and use cases around this, of which I'm looking into now with signal processing.

[vrenjith]

The app can accept any signal and we can code for it. The key part is that container pilot will receive a signal from docker and I am not sure how can we write an event to act on it and forward it to our app by running a kill. -- Regards, Renjith

[vrenjith]

I was trying to follow the suggestion from @cheapRoc and reading through https://github.com/autopilotpattern/nats/blob/master/etc/containerpilot.json5.

I got a question regarding the way containerpilot handles the watches. Are the watches implemented by using the blocking calls supported by consul or is the containerpilot doing a poll at the interval specified?

[tgross]

The key part is that container pilot will receive a signal from docker and I am not sure how can we write an event to act on it and forward it to our app by running a kill.

I want to make sure that the current behavior is being described accurately to you: currently ContainerPilot handles SIGTERM and SIGINT only, for graceful shutdown (and SIGCHLD for supervision if run as PID1). This graceful shutdown sends SIGTERM to all the child processes. All other signals to the ContainerPilot process are unhandled.

That being said:

this is in the context of nomad re-rendering a template inside the docker image

This really feels like something is off with this approach. Why is the scheduler reaching inside the container to update things on disk? Can you describe this in more detail?

I got a question regarding the way containerpilot handles the watches. Are the watches implemented by using the blocking calls supported by consul or is the containerpilot doing a poll at the interval specified?

This sounds like an unrelated issue. But using blocking calls is proposed in #461 but the current behavior is to poll.

[jwreagor]

That said, I'm still interested in exploring/defining features and use cases around this, of which I'm looking into now with signal processing.

Having researched this a bit further, I don't believe we can easily handle something like SIGHUP at PID 1 given that the new supervisor model would require some sort of IPC with the worker in order for (a hypothetical feature) specific worker/app jobs to be reloaded through the supervisor.

I'm now interested in how Nomad and ContainerPilot both utilize Consul and whether or not the template could be generated under ContainerPilot itself, as I first alluded to.

[jwreagor]

I opened #505 for a feature that came to mind after my last comment. We'll keep this issue going for the Nomad discussion (as that is quite valuable).

[jwreagor]

To simplify the discussion, I'll enumerate communication channels that a normal Docker container can receive. It should be noted that these are primarily applicable to a Docker host not a Docker container running on Triton (which are more decentralized and distributed).

• Ways for something running on the host that's running the container to communicate to a container process.

  • docker exec into the container.
  • TCP port opened within the container.
  • A shared file or socket that is mounted into the container (shared socket).
  • A signal sent to the container via docker kill -s <signal> <container>.

• Various ways that ContainerPilot can pass this communication to jobs in the container.

  • ContainerPilot can start with environment variables which point to a path or IP address of a shared file, socket, or TCP port. Jobs will receive this environment variable and can take periodic action on them.
  • Configure a ContainerPilot job that watches a mounted file and sends a signal to a process run by ContainerPilot when a file's mtime change has occurred.
  • Configure ContainerPilot through some sort of custom process bound to a TCP port (not unlike shared file/socket).
  • docker exec into the container and send a signal to a process, which is applicable if available.
  • Send SIGTERM or SIGINT to ContainerPilot running as PID 1 in the container and trigger a job on graceful shutdown.
  • Configure a job to trigger when a service in Consul is healthy, unhealthy, or has changed. This job can then take action inside the container, like regenerating files.

• Hypothetical ways that ContainerPilot might be able to communicate events with jobs in the future.

  • Watch and react to a k/v change within Consul (does not exist, proposed by Add ability to watch a key in Consul and trigger jobs based on changes to the value for that key #506).
  • Introduce a way for ContainerPilot to handle certain signals and trigger
    jobs based on those signals (again, does not exist).

[tgross]

I think we're trying to tackle a general case, but it just so happens I'm looking into setting up a Nomad cluster with jobs that use the template feature myself and I'm realizing that this may not be the approach @vrenjith will be best served by if he's looking to reach past PID1 into the container.

Instead, I'd run consul-template as a ContainerPilot job (or on the host if that's available) and have it update the ContainerPilot job's configuration rather than have Nomad do it. A lot of the functionality of consul-template is available in the Nomad template config but not all of it.

[misterbisson]

@vrenjith can you share more background on the use case you have in mind? As @cheapRoc points out, there are some aspects of Nomad that are duplicative of ContainerPilot, but I assume you're using Nomad on your host to run the Docker containers, and ContainerPilot in the container as well. Perhaps you've got a volume mapped into the container for the config files?

This also may related to a proposal in #506 to have ContainerPilot watch a key in Consul for the use case that the config files or templates for the app are stored as the value of that key. That's another path to injecting config info into the container, but doesn't involve any dependency on the host.

[jwreagor]

@vrenjith If you're still looking for a solution here, I'm merging in I merged and am releasing signal events shortly. See #513 for more details. They should allow you to send a signal and perform a job within the container using strictly UNIX signals SIGHUP and/or SIGUSR2 (SIGUSR1 reopens the log file).

We're also looking into #506 which may/may not help as well.

I'm going to close this issue by end of week if there are no other updates.

[jwreagor]

UNIX signal events are released in 3.6.0.

[vrenjith]

That's awesome. Thank you so much. I will check out the signal events feature. -- Regards, Renjith