Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handling of connection failures #196

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

Drahflow
Copy link
Contributor

This PR is the next iteration of failure handling, taking the discussion of #172 into account.

  • Includes useful CircuitBreaker logic to deal with various failure scenarios
  • Uses Connection.phi as a realtime estimate of connection reliability
  • Disconnects from continually unavailable endpoints (including lymph requests)

Drahflow added 6 commits May 26, 2015 13:27
This adds circuit breaking behaviour to lymph when selecting endpoints
to serve a request. The breaker tries to discern global failures,
instance failures and failures due to method implementations correctly.
lymph.core.connection.Connection pings the endpoint regularly
to determine whether it is still live. `__init__` takes a
`unresponsive_disconnect` parameter, which was however ignored
so far. Similarly, the `idle_disconnect` parameter did not have
any effect.

This commit uses thees two parameters to do what they advertise,
i.e. close the connection once the respective timeouts have been
reached.

A control-flow bug which caused the monitoring greenlet to commit
suicide before deregistering the connection from the server was
also fixed, and a comment to that effect added.
Before this commit, connections which were never successfully
connected remained UNRESPONSIVE indefinitely, repeatedly trying
to ping the (non-connected) endpoint.

Such connections are now moved to UNRESPONSIVE and unresponsive_timeout
triggers as usual.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant