BigQuery Add Column: Account for update limit #1662
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
iskakaushik
approved these changes
May 2, 2024
serprex
reviewed
May 2, 2024
| if field.Name == addedColumn.Name { | ||
| c.logger.Info(fmt.Sprintf("[schema delta replay] column %s already exists in table %s", | ||
| addedColumn.Name, schemaDelta.DstTableName)) | ||
| continue |
There was a problem hiding this comment.
need to continue enclosing loop using loop label
serprex
added a commit
that referenced
this pull request
May 2, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
BigQuery has a quota of 5 metadata update operations per table per 10 seconds. So currently if many columns are added in a PG to BQ CDC mirror, sync records fails with this error from BigQuery - even though we do
ADD COLUMN IF NOT EXISTS(it still counts as an update operation)This PR accounts for this by manually checking the table metadata for the column before adding it. This will atleast ensure that on sync flow retry, the remaining columns get added
Functionally tested