ActiveJob::DeserializationError is not reported to Sentry

[jonleighton]

Issue Description

We had some jobs that were failing because by the time they were processed, the records they operated on no longer existed. The exception in the job was a ActiveJob::DeserializationError, and we were confused about why these errors did not appear in Sentry.

It turns out that internally, Active Job is calling GlobalID::Locator.locate, which is calling Model.find(...), which is raising ActiveRecord::RecordNotFound.

The exception is then rescued in Active Job, and re-raised as an ActiveJob::DeserializationError.

But because the Exception#cause is an ActiveRecord::RecordNotFound, and this error type is in the default excluded_exceptions added by sentry-rails, and config.inspect_exception_causes_for_exclusion is enabled by default, the exception wasn't reported.

This seemed highly surprising to us and it took a long time to understand what was going on. Initially we assumed our exception handling code was buggy in some way.

I'm not sure if there could be some "better" defaults here, so I don't know if this is really a bug, but I thought it's worth discussing.

I couldn't find any information about why ActiveRecord::RecordNotFound is excluded by default. The earliest commit that adds it to the list is 343cc90, which looks like it imports code from elsewhere, so I don't have access to the history before that.

I assume that ActiveRecord::RecordNotFound is excluded because it is commonly raised in web endpoints. I'm not sure if this is justified, as it's easy to add a rescue_from to controllers to handle this without re-raising (e.g. to render a 404). In any case, it seemed very surprising that this ended up impacting an exception raised during job processing.

Reproduction Steps

1. Queue an Active Job with a fake id, e.g. MyJob.perform_later(user: User.new(id: 999999))
2. Observe that the job fails, but the exception is not reported to Sentry

Expected Behavior

The exception is reported to Sentry

Actual Behavior

The exception is silently discarded

Ruby Version

3.3.5

SDK Version

5.15.2 (but the latest version has the same behaviour)

Integration and Its Version

Rails

Sentry Config

No response

[sl0thentr0py]

hmm yeah that's a really unfortunate combination of defaults that causes undesired behaviour.

cc @st0012 @solnic can you think of better ways of handling such scenarios?

ActiveRecord::RecordNotFound is definitely very noisy in most cases but there are also many cases where suppressing it is actually harmful rather than useful.

[st0012]

I think the most ideal solution is that defaults will vary depending on the context in which the exception is raised (maybe through the mechanism option?).

For example:

• Excluded exceptions for mechanism: "rack": ActiveRecord::RecordNotFound, ActionController::UnknownFormat....etc.
• Excluded exceptions for mechanism: "background_job": noisy exceptions specific to the background job context.

We can NOT use integration for this because integrations like sentry-rails can have multiple context.

The value for config.excluded_exceptions will become something ike:

{
  "rack": ["ActiveRecord::RecordNotFound"`, # .... ],
  "background_job: []
}

[sl0thentr0py]

yeah that's a good idea but now thinking more about it, I don't even know if it's about rack vs jobs.
Like there are also cases where in rack/rails, you want to surface those, so the only right answer is that the user disable inspect-exception-causes-for-exclusion.

[jonleighton]

the only right answer is that the user disable inspect-exception-causes-for-exclusion.

Or remove ActiveRecord::RecordInvalid from the default excluded exceptions. I'm not sure of the implications of this, but it's not a given to me that it "should" be excluded by default.

[sl0thentr0py]

Or remove ActiveRecord::RecordInvalid from the default excluded exceptions.

I think this will make a lot of our customers blow their quota much faster, so I don't think we're going to do this, sorry!
It is configurable, so individual users can always remove it from excluded_exceptions.

[solnic]

Not pretty, but we can just look at the trace and see if it's coming from the ActiveJob stack and act accordingly.