Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revisit unrecoverable errors #450

Open
adutra opened this issue Aug 11, 2022 · 1 comment
Open

Revisit unrecoverable errors #450

adutra opened this issue Aug 11, 2022 · 1 comment
Milestone

Comments

@adutra
Copy link
Contributor

adutra commented Aug 11, 2022

LogManager has a method to determine whether an error associated with a processed item is "unrecoverable".

This method should imho be completely revisited. E.g. it is considering OverloadedException as unrecoverable, but it arguably is in most cases.

We should instead proceed as follows:

  • For malformed records, mapping errors, and load errors: the error is always recoverable since it affects only one record. The record should be written to the bad file and the processing should continue.
  • For unload and count errors: every error is unrecoverable, because an entire token range won't be read. It's similar to when a connect emits an IOException because a file cannot be read.

┆Issue is synchronized with this Jira Task by Unito

@adutra adutra added this to the 1.10.0 milestone Aug 11, 2022
@adutra
Copy link
Contributor Author

adutra commented Aug 11, 2022

Update: actually, it's probably best to take the opposite direction: every error captured as an ErrorRecord, UnmappableStatement, or unsuccessful Result should be recoverable, including range read failures.

For unloads, this means we keep the current behavior of reading the remaining token ranges. With the resume-failed feature (#432), it will be very easy to resume and only read the failed token ranges.

For loads, we should then implement #90 and make IOExceptions recoverable.

@weideng1 weideng1 modified the milestones: 1.10.0, 1.11.0 Aug 12, 2022
@absurdfarce absurdfarce modified the milestones: 1.11.0, 1.12.0 Jul 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants