Normalize concurrently with sync flows #893
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
serprex
force-pushed
the
normalize-split
branch
from
abf99dd to
e785a96
Compare
serprex
force-pushed
the
normalize-split
branch
5 times, most recently
from
f05a857 to
e1976c4
Compare
This was referenced Jan 3, 2024
Closed
serprex
force-pushed
the
normalize-split
branch
4 times, most recently
from
1c351fa to
dd40b2a
Compare
serprex
force-pushed
the
normalize-split
branch
4 times, most recently
from
06613bd to
47ca6ac
Compare
serprex
added a commit
that referenced
this pull request
Jan 15, 2024
A sync batch should not be considered complete until its schema changes are processed, this avoids failures after commit causing schema changes to be dropped, & when decoupling normalize/sync in #893 was causing normalization to be missing values
serprex
force-pushed
the
normalize-split
branch
from
2a3405b to
85c9ee3
Compare
serprex
force-pushed
the
normalize-split
branch
from
85c9ee3 to
758e10e
Compare
serprex
force-pushed
the
normalize-split
branch
10 times, most recently
from
179b4ca to
f2a5b7f
Compare
serprex
force-pushed
the
normalize-split
branch
from
6eb8b93 to
fdf6dce
Compare
serprex
force-pushed
the
normalize-split
branch
from
e390015 to
746f4dc
Compare
iskakaushik
reviewed
Jan 24, 2024
iskakaushik
reviewed
Jan 24, 2024
serprex
force-pushed
the
normalize-split
branch
from
d72dce1 to
5075598
Compare
serprex
force-pushed
the
normalize-split
branch
from
7033c90 to
bcea38b
Compare
iskakaushik
approved these changes
Jan 25, 2024
serprex
force-pushed
the
normalize-split
branch
from
b23c6db to
b70abbb
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously after each sync we'd pause reading slot to process table schema deltas & normalize
This has two problems:
Now NormalizeFlow is created as a child workflow at start of cdc flow & a signal is sent after each sync flow with schema updates. Normalize consumes all signals since it last checked, merging their processing in parallel with sync flows
NormalizeFlow only reads up to the signal's batch id to avoid potentially syncing a batch without its schema. This creates a range
(normid..syncid]in which normid is always catching up to syncid as we normalizenormid+1tosyncid. Normalize logic already handled this, so it goes untouched in this changePEERDB_ENABLE_PARALLEL_SYNC_NORMALIZEneeds to be set to true, for now keep this change behind feature flag to avoid potentially increasing data warehouse costs