Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(iceberg): add support for validating nested iceberg schemas #20551

Merged
merged 4 commits into from
Feb 21, 2025
Merged

feat(iceberg): add support for validating nested iceberg schemas #20551

merged 4 commits into from
Feb 21, 2025

Conversation

jayakasadev
Copy link
Contributor

@jayakasadev jayakasadev commented Feb 20, 2025 •
edited
Loading

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

  • modified try_matches_arrow_schema to traverse embedded structs for validation

Checklist

  • I have written necessary rustdoc comments.
  • I have added necessary unit tests and integration tests.
  • I have added test labels as necessary.
  • I have added fuzzing tests or opened an issue to track them.
  • My PR contains breaking changes.
  • My PR changes performance-critical code, so I will run (micro) benchmarks and present the results.
  • My PR contains critical fixes that are necessary to be merged into the latest release.

Documentation

  • My PR needs documentation updates.
Release note

@jayakasadev
Copy link
Contributor Author

jayakasadev commented Feb 20, 2025 •
edited
Loading

related to #20552

@jayakasadev jayakasadev force-pushed the iceberg_validate_nested_arrow_schema branch from ae8ed9a to 1e99ba9 Compare February 20, 2025 18:28
@jayakasadev jayakasadev force-pushed the iceberg_validate_nested_arrow_schema branch from 1e99ba9 to e673d92 Compare February 21, 2025 00:55
@jayakasadev jayakasadev marked this pull request as ready for review February 21, 2025 00:56
@chenzl25
Copy link
Contributor

@jayakasadev Thanks for the quick fix. I think we can add an e2e test. Here we have a full type test.

risingwave/e2e_test/iceberg/test_case/iceberg_engine.slt

Line 49 in 367636b

CREATE TABLE full_type_t (

chenzl25
chenzl25 approved these changes Feb 21, 2025
View reviewed changes
Copy link
Contributor

@chenzl25 chenzl25 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for the contribution

@chenzl25 chenzl25 enabled auto-merge February 21, 2025 13:11
@chenzl25 chenzl25 added this pull request to the merge queue Feb 21, 2025
Merged via the queue into risingwavelabs:main with commit bf43ee8 Feb 21, 2025
30 checks passed
@jayakasadev jayakasadev deleted the iceberg_validate_nested_arrow_schema branch February 21, 2025 13:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chenzl25 chenzl25 chenzl25 approved these changes

@ZENOTME ZENOTME Awaiting requested review from ZENOTME

@xxchan xxchan Awaiting requested review from xxchan

Assignees
No one assigned
Labels
ci/run-e2e-iceberg-engine-tests ci/run-e2e-iceberg-sink-v2-tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jayakasadev @chenzl25