Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add optimize_manifests table procedure to Iceberg #24678

Merged
merged 1 commit into from
Feb 4, 2025

Conversation

ebyhr
Copy link
Member

@ebyhr ebyhr commented Jan 10, 2025

Description

Add a new procedure that rewrites manifest files like rewrite_manifests in Spark.

ALTER TABLE {table} EXECUTE optimize_manifests;

Fixes #14821

Release notes

## Iceberg
* Add `optimize_manifests` table procedure. ({issue}`14821`)

@cla-bot cla-bot bot added the cla-signed label Jan 10, 2025
@github-actions github-actions bot added docs iceberg Iceberg connector labels Jan 10, 2025
@ebyhr ebyhr force-pushed the ebi/iceberg-rewrite-manifests branch from 16f7f2a to f099255 Compare January 15, 2025 05:28
@ebyhr ebyhr marked this pull request as ready for review January 15, 2025 05:28
@martint
Copy link
Member

martint commented Feb 4, 2025

What does the function do? "rewrite manifests" is too vague, IMO, and doesn't convey what its purpose is. Why would one want to rewrite manifests? In what way are they rewritten?

@ebyhr
Copy link
Member Author

ebyhr commented Feb 4, 2025

@martint This is "compaction" about manifest files, e.g. merging 10 manifest files into one. Too many manifest files increase planning time because we must open each file.

@martint
Copy link
Member

martint commented Feb 4, 2025

In that case, I would rename the function to either "compact_manifests" (if we think that's the only thing the function will do going forward), or "optimize_manifests".

@ebyhr
Copy link
Member Author

ebyhr commented Feb 4, 2025

Let me adopt "optimize_manifests".

@ebyhr ebyhr changed the title Add rewrite_manifests table procedure to Iceberg Add optimize_manifests table procedure to Iceberg Feb 4, 2025
@ebyhr ebyhr force-pushed the ebi/iceberg-rewrite-manifests branch 2 times, most recently from 3a66f6b to 8b85db6 Compare February 4, 2025 05:43
@ebyhr ebyhr force-pushed the ebi/iceberg-rewrite-manifests branch from 8b85db6 to 1b12a66 Compare February 4, 2025 22:26
@ebyhr ebyhr merged commit 1f808b1 into trinodb:master Feb 4, 2025
5 of 15 checks passed
@ebyhr ebyhr deleted the ebi/iceberg-rewrite-manifests branch February 4, 2025 22:26
@github-actions github-actions bot added this to the 470 milestone Feb 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

Successfully merging this pull request may close these issues.

Add the functionality of the Iceberg rewrite_manifests procedure (e.g. in OPTIMIZE)
3 participants