Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Always refresh Resque's redis connection in spawned workers #154

Closed
wants to merge 1 commit into from

Conversation

benweint
Copy link

@benweint benweint commented Jun 3, 2016

When the initial batch of workers is spawned by the resque-pool master process, the master will not generally be in possession of a Redis connection. However, as soon as one of the workers dies and needs to be respawned, the resque-pool master will establish a Redis connection in order to enumerate the resque queues, via the call to Worker#queues in reap_all_workers.

When this happens, the first worker after the initial batch that's spawned by the resque-pool master will inherit resque's Redis connection from the resque-pool master, and will try to use it.

With older versions of the redis-rb gem (pre 3.1.0), or with the inherit_socket option, this will cause an exception in the newly-spawned worker, and make the resque-pool master get into a loop wherein it tries and fails repeatedly to spawn a new worker.

This change ensures that a new Redis connection is always established inside of newly-spawned resque worker processes, avoiding the bad state. While it would be possible to do this manually via an after_prefork hook manually, the manifestation of this problem is pretty non-obvious, because of the fact that it doesn't happen for the first batch of spawned workers, so it seems considerate to do the right thing out of the box, and avoid the unpleasant surprise when a worker dies.

When the initial batch of workers is spawned by the resque-pool master process,
the master will not generally be in possession of a Redis connection. However,
as soon as one of the workers dies and needs to be respawned, the resque-pool
master will establish a Redis connection in order to enumerate the resque
queues, via the call to Worker#queues in reap_all_workers.

When this happens, the first worker after the initial batch that's spawned by
the resque-pool master will inherit resque's Redis connection from the
resque-pool master, and will try to use it.

With older versions of the redis-rb gem (pre 3.1.0), or with the inherit_socket
option, this will cause an exception in the newly-spawned worker, and make the
resque-pool master get into a loop wherein it tries and fails repeatedly to
spawn a new worker.

This change ensures that a new Redis connection is always established inside of
newly-spawned resque worker processes, avoiding the bad state. While it would be
possible to do this manually via an after_prefork hook manually, the
manifestation of this problem is pretty non-obvious, because of the fact that it
doesn't happen for the first batch of spawned workers, so it seems considerate
to do the right thing out of the box, and avoid the unpleasant surprise when
a worker dies.
@codyrobbins
Copy link

This would be awesome to merge but it looks like #135 is a more general solution that should obviate it.

@benweint benweint closed this Jan 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants