Datalake 0.15.0 071124 #12

remeajayi2022 · 2024-11-19T13:45:33Z

Change Logs

Describe context and summary for this change. Highlight if any code was copied.

Impact

Describe any public API or user-facing feature change or any performance impact.

Risk level (write none, low medium or high below)

If medium or high, explain what verification was done to mitigate the risks.

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

…llow for marking a newly created column as nullable." (apache#10782)

Co-authored-by: jp0317 <[email protected]>

…bility (apache#10794)

…che#10772)

Co-authored-by: Y Ethan Guo <[email protected]>

… handle CheckPointComplete not finished (apache#10767)

… data (apache#10594)

…apache#10795)

…che#10728) Co-authored-by: Vova Kolmakov <[email protected]>

apache#10214)

Co-authored-by: Shawn Chang <[email protected]>

Co-authored-by: wuzhiping <[email protected]>

…he#10802)

…e#10799)

…pache#10786)

…che#10565) --------- Co-authored-by: Krishen Bhan <“[email protected]”>

… schema exceptions (apache#10677) Co-authored-by: Jonathan Vexler <=>

…s and add tests for filtering out extensions (apache#10724) Co-authored-by: rmahindra123 <[email protected]>

…che#10818)

… flags (apache#10810)

apache#10698) Co-authored-by: Vova Kolmakov <[email protected]>

…che#10280)

…keys (apache#7153)

…schema changed (apache#10830)

…che#10832)

…mmit metadata (apache#10837) * Fixing handling null values of extra metadata in clean commit metadata * fixing tests

…1270)

…pache#11283)

…oodieStorage abstraction (apache#11280)

…he#11290)

…fig (apache#11294)

…ache#11298) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…e directly (apache#11303)

…che#11304)

Co-authored-by: sivabalan <[email protected]>

…mplementation (apache#11310)

…serde (apache#11356) * [HUDI-7809] Use Spark SerializableConfiguration to avoid NPE in Kryo serde * Revert changes in HoodieBaseRelation

…es (apache#11363)

geserdugarov and others added 30 commits May 14, 2024 13:11

Revert "[HUDI-6438] Config parameter 'MAKE_NEW_COLUMNS_NULLABLE' to a…

7a4f86a

…llow for marking a newly created column as nullable." (apache#10782)

[HUDI-7459] Update hudi-gcp-bundle pom (apache#10790)

e358530

Co-authored-by: jp0317 <[email protected]>

[HUDI-7462] Refactor checkTopicCheckpoint in KafkaOffsetGen for reusa…

56c4937

…bility (apache#10794)

[HUDI-7464] Fix minor bugs in kafka post-processing related code (apa…

d36e526

…che#10772)

[MINOR] Fix violations of Sonarqube rule java:S2184 (apache#10444)

0a4bed6

Co-authored-by: Y Ethan Guo <[email protected]>

[HUDI-7447] Fix not bootstrap when subTask restart when OPCoordinator…

d81f2bb

… handle CheckPointComplete not finished (apache#10767)

[HUDI-9424] Support using local timezone when writing flink TIMESTAMP…

99d52b7

… data (apache#10594)

[HUDI-7465] Split tests in CI further to reduce total CI elapsed time (…

68659a1

…apache#10795)

[HUDI-6089] Handle default insert behaviour to ingest duplicates (apa…

4bdb98b

…che#10728) Co-authored-by: Vova Kolmakov <[email protected]>

[HUDI-7469] Reduce redundant tests with Hudi record types (apache#10800)

53725e8

[HUDI-6953] Adding test for composite keys with bulk insert row writer (

7c3437f

apache#10214)

[HUDI-3625] Update RFC-60 (apache#9462)

e2ec366

Co-authored-by: Shawn Chang <[email protected]>

[MINOR] Clean code of FileSystemViewManager (apache#10797)

734e0cf

Co-authored-by: wuzhiping <[email protected]>

[HUDI-7471] Use existing util method to get Spark conf in tests (apac…

05e16b7

…he#10802)

[MINOR] Add PR description validation on documentation updates (apach…

e35fa8d

…e#10799)

[HUDI-7479] SQL confs don't propagate to spark row writer properly (a…

a4aa005

…pache#10786)

[HUDI-7337] Implement MetricsReporter that reports metrics to M3 (apa…

5deb196

…che#10565) --------- Co-authored-by: Krishen Bhan <“[email protected]”>

[HUDI-7413] Fix schema exception types and error messages thrown with…

78bf676

… schema exceptions (apache#10677) Co-authored-by: Jonathan Vexler <=>

[HUDI-7418] Create a common method for filtering in S3 and GCS source…

4538fb2

…s and add tests for filtering out extensions (apache#10724) Co-authored-by: rmahindra123 <[email protected]>

[MINOR] Fix Azure publishing of JUnit results (apache#10817)

81fe5ad

[MINOR] Publish test results from the containerized job to Azure (apa…

111d138

…che#10818)

[HUDI-7473] Rebalance CI (apache#10805)

3d5d274

[HUDI-6947] Refactored HoodieSchemaUtils.deduceWriterSchema with many…

45923f3

… flags (apache#10810)

[HUDI-7356] Passing configs to file reader constructor for flexibility (

695577b

apache#10698) Co-authored-by: Vova Kolmakov <[email protected]>

[HUDI-7197] Adding mis fixes related with table services testing (apa…

4680cb4

…che#10280)

[HUDI-5167] Reducing total test run time: reducing tests for virtual …

9f00f6d

…keys (apache#7153)

[HUDI-7488] The BigQuerySyncTool can't work well when the hudi table …

8ed8a20

…schema changed (apache#10830)

[MINOR] Separate HoodieSparkWriterTestBase to reduce duplication (apa…

06584c6

…che#10832)

[HUDI-7491] Fixing handling null values of extra metadata in clean co…

8e6eff9

…mmit metadata (apache#10837) * Fixing handling null values of extra metadata in clean commit metadata * fixing tests

[HUDI-7411] Meta sync should consider cleaner commit (apache#10676)

dbe16f3

yihua and others added 29 commits May 23, 2024 09:24

[HUDI-7784] Fix serde of HoodieHadoopConfiguration in Spark (apache#1…

ed2dc91

…1270)

[HUDI-7786] Fix roaring bitmap dependency in hudi-integ-test-bundle (a…

0de2e80

…pache#11283)

[HUDI-7785] Keep public APIs in utilities module the same as before H…

b71da93

…oodieStorage abstraction (apache#11280)

[HUDI-7775] Remove unused APIs in HoodieStorage (apache#11281)

670f131

[HUDI-7788] Fixing exception handling in AverageRecordSizeUtils (apac…

b350a26

…he#11290)

[HUDI-7776] Simplify HoodieStorage instance fetching (apache#11292)

77b1ded

[HUDI-7792] Bump h2 from 1.4.200 to 2.2.220 (apache#11296)

2e763c9

[HUDI-7790] Revert changes in DFSPathSelector and UtilHelpers.readCon…

72dd518

…fig (apache#11294)

[HUDI-7794] Bump org.apache.hive:hive-service from 2.3.1 to 2.3.4 (ap…

b607899

…ache#11298) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

[HUDI-7777] Allow HoodieTableMetaClient to take HoodieStorage instanc…

f5b8088

…e directly (apache#11303)

[HUDI-7796] Gracefully cast file system instance in Avro writers (apa…

b9ffa97

…che#11304)

[HUDI-7778] Fixing global index for duplicate updates (apache#11305)

86552da

Co-authored-by: sivabalan <[email protected]>

[HUDI-7798] Mark configs included in 0.15.0 release (apache#11307)

b4d52c0

[HUDI-7797] Use HoodieIOFactory to return pluggable FileFormatUtils i…

b8796d0

…mplementation (apache#11310)

[MINOR] Fix bundle validation script on branch-0.x (apache#11331)

bd4256b

[HUDI-7707] Enable bundle validation on Java 8 and 11 (apache#11313)

bbebda4

[HUDI-7802] Fix bundle validation scripts (apache#11332)

27e45ac

Bumping release candidate number 2

7100227

[MINOR] Change release candidate validation target

f80c416

[MINOR] Disable release candidate validation by default (apache#11339)

b9ae51e

[MINOR] Fix Flink version in release candidate validation (apache#11341)

27df817

[HUDI-7809] Use Spark SerializableConfiguration to avoid NPE in Kryo …

88d057f

…serde (apache#11356) * [HUDI-7809] Use Spark SerializableConfiguration to avoid NPE in Kryo serde * Revert changes in HoodieBaseRelation

[HUDI-7807] Fixing spark-sql for pk less tables (apache#11354)

fe08b6f

[HUDI-7812] Disabling row writer for clustering (apache#11360)

9e79996

[HUDI-7655] Ensuring clean action executor cleans up all intended fil…

c009895

…es (apache#11363)

[MINOR] Remove thrift gen in staging deploy script

d90c690

Bumping release candidate number 3

d0df1d4

[MINOR] Update release version to reflect published version 0.15.0

3883285

merging custom datalake changes

7b6fbab

remeajayi2022 changed the base branch from master to datalake November 19, 2024 13:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Datalake 0.15.0 071124 #12

Datalake 0.15.0 071124 #12

remeajayi2022 commented Nov 19, 2024

Datalake 0.15.0 071124 #12

Are you sure you want to change the base?

Datalake 0.15.0 071124 #12

Conversation

remeajayi2022 commented Nov 19, 2024

Change Logs

Impact

Risk level (write none, low medium or high below)

Documentation Update

Contributor's checklist