Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create 2024-08-12-hudi-acid-transactions.md #11760

Closed
wants to merge 5,246 commits into from
Closed

Create 2024-08-12-hudi-acid-transactions.md #11760

wants to merge 5,246 commits into from

Conversation

alberttwong
Copy link
Contributor

@alberttwong alberttwong commented Aug 12, 2024 •
edited
Loading

Change Logs

adding article on hudi acid transactions

Impact

no impact

Risk level (write none, low medium or high below)

none

Documentation Update

none

Contributor's checklist

  • Read through contributor's guide
  • Change Logs and Impact were stated clearly
  • Adequate tests were added if applicable
  • CI passed

jonvex and others added 30 commits May 10, 2024 14:19
@jonvex @yihua
[HUDI-7726] Restructure TableSchemaResolver to separate Hadoop logic …
23b283a 
…and use BaseFileUtils (#11185)

Co-authored-by: Jonathan Vexler <=>
Co-authored-by: Y Ethan Guo <[email protected]>
@yihua
[HUDI-7742] Move Hadoop-dependent reader util classes to hudi-hadoop-…
badcca2 
…common module (#11190)
@nsivabalan
[HUDI-7673] Fixing false positive validation failure for RLI with MDT…
0d0e27e 
… validation tool (#11098)
@jonvex
[HUDI-7731] Fix usage of new Configuration() in production code (#11191)
c7c636c 
Co-authored-by: Jonathan Vexler <=>
@Zouxxyy
[HUDI-7739] Shudown asyncDetectorExecutor in AsyncTimelineServerBased…
86f7a65 
…DetectionStrategy (#11182)
@vinishjail97 @yihua
[HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoo…
49072d1 
…dieRecords and JsonKafkaSource mapPartitions (#10872)

Co-authored-by: Y Ethan Guo <[email protected]>
@yihua
[HUDI-7745] Move Hadoop-dependent util methods to hudi-hadoop-common (#…
61f54a0 
…11193)
@the-other-tim-brown @yihua
[HUDI-4732] Add support for confluent schema registry with proto (#11070
aa5bb0d 
)

Co-authored-by: Y Ethan Guo <[email protected]>
@vinishjail97 @yihua
[HUDI-7501] Use source profile for S3 and GCS sources (#10861)
be0a660 
Co-authored-by: Y Ethan Guo <[email protected]>
@vinishjail97 @yihua @codope
[HUDI-7523] Add HOODIE_SPARK_DATASOURCE_OPTIONS to be used in HoodieI…
ce08875 
…ncrSource (#10900)

Co-authored-by: Y Ethan Guo <[email protected]>
Co-authored-by: Sagar Sumit <[email protected]>
@jonvex @yihua
[HUDI-7743] Improve StoragePath usages (#11189)
4b59b20 
Co-authored-by: Jonathan Vexler <=>
Co-authored-by: Y Ethan Guo <[email protected]>
@jonvex @yihua
[HUDI-7744] Introduce IOFactory and a config to set the factory (#11192)
ea4f14c 
Co-authored-by: Jonathan Vexler <=>
Co-authored-by: Y Ethan Guo <[email protected]>
@yihua
[HUDI-7750] Move HoodieLogFormatWriter class to hoodie-hadoop-common …
6627218 
…module (#11207)
@jonvex
remove a few classes from hudi-common (#11209)
c15bdb3 
Co-authored-by: Jonathan Vexler <=>
@jonvex
[HUDI-7589] Add API to create HoodieStorage in HoodieIOFactory (#11208)
076a9ad 
Co-authored-by: Jonathan Vexler <=>
@nsivabalan @yihua
[HUDI-7549] Reverting spurious log block deduction with LogRecordRead…
45ad35b 
…er (#10922)

Co-authored-by: Y Ethan Guo <[email protected]>
@vinishjail97 @nsivabalan
[HUDI-7617] Fix issues for bulk insert user defined partitioner in St…
bbf0280 
…reamSync (#11014)

Co-authored-by: sivabalan <[email protected]>
@vinishjail97 @yihua
[HUDI-7535] Add metrics for sourceParallelism and Refresh profile in …
d9eb7b4 
…S3/GCS (#10918)

Co-authored-by: Y Ethan Guo <[email protected]>
@codope
[HUDI-7749] Bump Spark version 3.3.1 to 3.3.4 (#11198)
7f837d0 
* [HUDI-7749] Bump Spark version 3.3.1 to 3.3.4

* cdcFileReader should return batches for CDC reads only when batch read is supported for the schema
@nsivabalan @yihua
[HUDI-7712] Fixing RLI initialization to account for file slices inst…
d74e642 
…ead of just base files while initializing (#11153)

Co-authored-by: Y Ethan Guo <[email protected]>
@nsivabalan @yihua
[HUDI-7624] Fixing index tagging duration (#11035)
8d103ad 
Co-authored-by: Y Ethan Guo <[email protected]>
@yihua
[HUDI-7752] Abstract serializeRecords for log writing (#11210)
a8da2fd
@nsivabalan @yihua
[HUDI-7429] Fixing average record size estimation for delta commits (#…
c8dec0e 
…10763)

Co-authored-by: Y Ethan Guo <[email protected]>
Co-authored-by: Jonathan Vexler <=>
@yihua
[HUDI-7759] Remove Hadoop dependencies in hudi-common module (#11220)
7e5ff1e 
Co-authored-by: Jonathan Vexler <=>
@nsivabalan
[HUDI-7532] Include only compaction instants for lastCompaction in ge…
9e9bb90 
…tDeltaCommitsSinceLatestCompaction (#10915)

* Fixing schedule compaction bug

* Addressing comments

* Fixing CDC tests
@yihua
[MINOR] Rebalance CI with tests in hudi-hadoop-common module (#11230)
5bf5090
@geserdugarov
[HUDI-7717] Disable row writer for bulk insert if combining before in…
7fc5ada 
…sert is set (#11216)
@xicm
[MINOR] Remove legacy code and add try catch to listStatus of partiti…
d93e4eb 
…on (#11250)
@codope
[HUDI-7652] Add new HoodieMergeKey API to support simple and compos…
e0ca6dd 
…ite keys (#11077)

Introduce a new abstract class `BaseHoodieMergedLogRecordScanner` which 
subclasses from `AbstractHoodieLogRecordReader`. The new abstract class 
holds the `records` map as `ExternalSpillableMap<K, HoodieRecord>` and exposes 
`public abstract Map<K, HoodieRecord> getRecords()` API. The existing 
`HoodieMergedLogRecordScanner` now derives from the new abstract class 
(instead of `AbstractHoodieLogRecordReader`) and uses String keys.
@ad1happy2go
[MINOR] Added condition to check default value to fix extracting pass…
e4b56b0 
…word from credential store (#11246)
wombatu-kun and others added 21 commits July 29, 2024 17:30
@wombatu-kun
[HUDI-7948] Claiming RFC-80 for Column families (#11699)
e315b16 
Co-authored-by: Vova Kolmakov <[email protected]>
@parisni @CTTY
[HUDI-6453] Cascade Glue schema changes to partitions (#11670)
2024d17 
Co-authored-by: Shawn Chang <[email protected]>
@Davis-Zhang-Onehouse
[HUDI-8029] Enforce s3 base path format for partition key calculation (…
2aa60f2 
…#11693)
@TheR1sing3un
[MINOR] Support MOR table type for HoodieJavaWriteClientExample (#11700)
9f21c2d 
1. Support MOR table type for HoodieJavaWriteClientExample
2. Typo Fix

Signed-off-by: ther1sing3un <[email protected]>
@usberkeley
[HUDI-8033] Claiming RFC-81 for Log Compaction with Merge Sort (#11703)
4bbc04b
@lokeshj1703
[HUDI-8035] Fetching commit metadata from timeline fails during upgra…
4dbe5b8 
…de (#11711)
@wombatu-kun
[HUDI-7948] RFC-80: Support column families for wide tables (#11559)
c858d56 
Co-authored-by: Vova Kolmakov <[email protected]>
@dongtingting
[HUDI-8040] Fix SimpleConcurrentFileWritesConflictResolutionStrategy …
0fbf312 
…get clustering instant contains insert overwrite (#11691)
@hbgstc123
[HUDI-8041] Support projection push down for lookup join (#11722)
06d3fad 
Co-authored-by: hbg <[email protected]>
@lokeshj1703 @codope
[HUDI-7996] Store partition type with partition fields in table confi…
af80809 
…gs (#11638)

Store partition type as well in `hoodie.table.partition.fields`. 
Currently this config stores the commas separated partition fields 
without the partition type. Partition type here corresponds to the 
partition type supported in Custom key generators like `simple` 
and `timestamp`.

Co-authored-by: Sagar Sumit <[email protected]>
@codope
[HUDI-7692] Extract metadata record type to MetadataPartitionType enum (
c12bac4 
#11597)
@TheR1sing3un
[MINOR] Reduce unnecessary code paths (#11729)
7f60243
@the-other-tim-brown
[MINOR] don't run scheduled workflow on forks (#11736)
5da405e
@pkgajulapalli @yihua
[HUDI-7881] Compare bigquery table base path via source-uris if hive …
fb2361f 
…partitioning options are not available (#11679)

Co-authored-by: Y Ethan Guo <[email protected]>
@Davis-Zhang-Onehouse
[HUDI-8043] Fix DDB Lock provider bug (#11728)
5da41a1
@vamsikarnika @codope
[HUDI-8015] Fix Glue Meta Sync Failure with recreating the table (#11677
36242ff 
)

Updated AwsGlueCatalaogSyncClient createOrReplace method to drop and 
create the table, instead of updating the table. Updated the check to compare 
the base of the hudi table and metastore table location to ignore file schemas.

---------

Co-authored-by: Vamsi <[email protected]>
Co-authored-by: Sagar Sumit <[email protected]>
@jonvex
[HUDI-6948] Fix schema evolution for non row writer (#11731)
f1ecdf6 
Co-authored-by: Jonathan Vexler <=>
@kamronis
[MINOR] Improve performance of generateBucketKey (#11748)
cbd2573 
Co-authored-by: Sergey Troshkov <[email protected]>
@imhy
[MINOR] HoodieAvroUtils performance optimization of createFullName me…
0cd28e5 
…thod (#11747)
@jonvex
[HUDI-8067] Use exec to run the IT (#11751)
b7e11b1 
Co-authored-by: Jonathan Vexler <=>
@yihua
[HUDI-8012] Update checkstyle.xml and fix checkstyle errors (#11672)
c71b2da 
* [HUDI-8012] Update checkstyle.xml based on the new release

* Upgrade checkstyle and plugin

* Fix checkstyle errors

* Suppress checkstyle error for sql
@github-actions github-actions bot added docs size:S PR with lines of changes in (10, 100] size:XL PR with lines of changes > 1000 and removed size:S PR with lines of changes in (10, 100] labels Aug 12, 2024
@alberttwong alberttwong deleted the patch-1 branch August 12, 2024 22:56
@alberttwong alberttwong restored the patch-1 branch August 12, 2024 23:10
@alberttwong alberttwong deleted the patch-1 branch August 12, 2024 23:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
docs size:XL PR with lines of changes > 1000
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.