Logging exceptions that happen during a command/query #3884
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We spent weeks trying to track down networking issues between Heroku and Mongolab because we were not seeing answers from commands like:
MOPED: 127.0.0.1:27017 COMMAND database=admin command={:ismaster=>1} runtime: 3.2830ms
They just kept bouncing between nodes asking "ismaster". The main underlying cause was connection pool saturation, which was compounded by the fact that every uncaught exception (Timeout::Error) ends in the Failure class trying to Disconnect (which in turn tries to check out another connection to disconnect that isn't even the offending connection and of course doesn't make a lot of sense with pool saturation, Moped PR coming for that).
I've added a few lines to the log_subscriber so that if the payload has an exception, it shows it along with the command or query etc that it was TRYING to run, instead of making it look like it completed successfully. In most cases, Mongoid/Moped will still handle the exception/retry if it needs to, but surfacing these exceptions is extremely useful for optimizing our cluster and figuring out when and where we are having any sort of issue (connectivity, node failure)