Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optional Iceberg commit locks #47

Merged
merged 3 commits into from
Jun 18, 2024
Merged

Conversation

sauliusvl
Copy link
Contributor

@sauliusvl sauliusvl commented Jun 17, 2024

When multiple threads attempt committing Iceberg transactions to the same table at the same time they can end up conflicting with each other resulting it multiple retries. Even though commits eventually succeed, this can effectively prolong commit times and result in lots of warning messages in logs. Hence we add an option to specify a lock to use when performing commits.

Also:

  • fix flaky Iceberg test that randomly failed to download the extension, download it locally once on startup instead
  • add a builder for IcebergRecordBatchStorage
  • bump all libraries
  • override zstd dependency scheme so that we can bump the patch version

@shivam247 shivam247 merged commit 952ecae into adform:master Jun 18, 2024
1 check passed
@sauliusvl sauliusvl deleted the iceberg-locks branch July 3, 2024 08:08
@sauliusvl sauliusvl restored the iceberg-locks branch July 29, 2024 07:55
@sauliusvl sauliusvl deleted the iceberg-locks branch July 29, 2024 07:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants