Enhance centurion API to allow for gated deployment

[cory-johannsen]

We need to override the deployment stages to allow centurion to enable controlled container shutdown during a deploy.

For example, we have a service container that is running a set of background tasks. On a deploy, we need centurion to notify the service that a shutdown is requested. Then, centurion should monitor a status endpoint until the service indicates that all background processes have terminated. At that time, centurion will be clear to bring down the container and deploy the new image.

[benders]

@cory-johannsen Could this be handled by a configurable (very long?) timeout between an initial TERM to the background processes and a final KILL?

[cory-johannsen]

@benders Our use case is that we have a Ruby/Sinatra service that utilizes a pool of unicorn background workers. The background unicorns are performing file operations that are variable in duration, and are fed from a queue. The queue is itself populated from HTTP POSTs to the sinatra service.

When it is time to shut down the container, we need to notify the service that it should cleanly cease operations. This tells the service to:
a) Stop accepting incoming requests
b) Allow the background unicorns to continue working until the queue is empty
c) Shut down the service

We need to make sure that the items in the queue are all consumed, such that no data is lost.

Our service has a status check endpoint /status/shutdown that instructs the service to begin the shutdown process. We also provide /status/idle which indicates whether the unicorns are still working.

To ensure deterministic behavior, we'd like to hook into the centurion deployment cycle such that we could insert the logic to trigger the shutdown and wait for coalescence prior to tearing down the container.

[relistan]

We should make an API available so that the shutdown method can be overridden in some reasonable way. I worked with @cory-johannsen and @bettse to put something in place for them for now. But it's fragile.

[kremso]

I think this (together with #110) would be solvable via before tasks/after tasks/task replacement that are supported directly by rake (although the API is very ugly). The only problem is rolling_deploy which combines stopping, starting, health check and cleanup into a single rake task, so you cannot hook into a specific phase. However, I think that breaking it down into smaller tasks would be pretty simple.

I see that @relistan doesn't like capistrano-style hooks and I can see why, but given that centurion is built on rake, this seems natural.

[relistan]

@kremso you are right this is exactly the right way to do this. If we break up rolling_deploy it's trivial to hook into it. I worry about a Capistrano-like mess, but this is the native behavior of Centurion so let's use it. Awesome!