You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
LogManager has a method to determine whether an error associated with a processed item is "unrecoverable".
This method should imho be completely revisited. E.g. it is considering OverloadedException as unrecoverable, but it arguably is in most cases.
We should instead proceed as follows:
For malformed records, mapping errors, and load errors: the error is always recoverable since it affects only one record. The record should be written to the bad file and the processing should continue.
For unload and count errors: every error is unrecoverable, because an entire token range won't be read. It's similar to when a connect emits an IOException because a file cannot be read.
Update: actually, it's probably best to take the opposite direction: every error captured as an ErrorRecord, UnmappableStatement, or unsuccessful Result should be recoverable, including range read failures.
For unloads, this means we keep the current behavior of reading the remaining token ranges. With the resume-failed feature (#432), it will be very easy to resume and only read the failed token ranges.
For loads, we should then implement #90 and make IOExceptions recoverable.
LogManager has a method to determine whether an error associated with a processed item is "unrecoverable".
This method should imho be completely revisited. E.g. it is considering OverloadedException as unrecoverable, but it arguably is in most cases.
We should instead proceed as follows:
┆Issue is synchronized with this Jira Task by Unito
The text was updated successfully, but these errors were encountered: