…llow for marking a newly created column as nullable." (apache#10782)

Co-authored-by: jp0317 <[email protected]>

…bility (apache#10794)

…che#10772)

Co-authored-by: Y Ethan Guo <[email protected]>

… handle CheckPointComplete not finished (apache#10767)

… data (apache#10594)

…apache#10795)

…che#10728)

Co-authored-by: Vova Kolmakov <[email protected]>

apache#10214)

Co-authored-by: Shawn Chang <[email protected]>

Co-authored-by: wuzhiping <[email protected]>

…he#10802)

…e#10799)

…pache#10786)

…che#10565)


---------

Co-authored-by: Krishen Bhan <“[email protected]”>

… schema exceptions (apache#10677)

Co-authored-by: Jonathan Vexler <=>

…s and add tests for filtering out extensions (apache#10724)

Co-authored-by: rmahindra123 <[email protected]>

…che#10818)

… flags (apache#10810)

apache#10698)

Co-authored-by: Vova Kolmakov <[email protected]>

…che#10280)

…keys (apache#7153)

…schema changed (apache#10830)

…che#10832)

…mmit metadata (apache#10837)

* Fixing handling null values of extra metadata in clean commit metadata

* fixing tests

…reserveCommitMetadata field (apache#8393)

…submit job (apache#7074)

Co-authored-by: Y Ethan Guo <[email protected]>

…platform-service/hudi-metaserver/hudi-metaserver-server (apache#7674)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…apache#10220)

…e/hudi-metaserver/hudi-metaserver-server (apache#7673)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [HUDI-1517] create marker file for every log file (apache#4913) (apache#524)

---------

Co-authored-by: guanziyue <[email protected]>
Co-authored-by: Lokesh Jain <[email protected]>

Signed-off-by: studystill <[email protected]>

… path (apache#10836)

* get rid of collect in row writer clustering

* fix race condition

* add logging

---------

Co-authored-by: Jonathan Vexler <=>

Co-authored-by: Vova Kolmakov <[email protected]>

…rementalFetchInContinuousMode (apache#10867)

Co-authored-by: Vova Kolmakov <[email protected]>

…en building clustering plan (apache#10621)

* Get partitions from active timeline instead of listing when building clustering plan

* fix checkstyle

Co-authored-by: Vova Kolmakov <[email protected]>

…d in each range (apache#10869)

…og sync (apache#10460)

* Add parallel listing of existing partitions

* Improve with new approach

* Fix checkstyle

* Fix listing for empty list of commits

* Fix logic for HiveSyncTool

* Fix lint errors

* Fix IT

* Use custom thread names

* Address review comments

---------

Co-authored-by: vmakarevich <[email protected]>

apache#10870)

Co-authored-by: Vova Kolmakov <[email protected]>

…rtition or multi primary key tables creation (apache#10840)

Co-authored-by: 含风 <[email protected]>

…s tests (apache#10889)

Co-authored-by: Vova Kolmakov <[email protected]>

Co-authored-by: Vova Kolmakov <[email protected]>

…pache#10910)

…stBase` (apache#10912)

…ors and HoodieTables (apache#10908)

Co-authored-by: Vova Kolmakov <[email protected]>

…roduceDataStream (apache#10925)

…ing plan (apache#10923)

…eletes (apache#10913)

---------

Co-authored-by: Jonathan Vexler <=>

…enabled (apache#10928)

…ants to avoid duplicates (apache#8884)

…uld create default values for HoodieMetricsM3Config (apache#10936)

Co-authored-by: Krishen Bhan <“[email protected]”>

…ache#10866)

Co-authored-by: Vova Kolmakov <[email protected]>

…apache#10946)

Co-authored-by: Jonathan Vexler <=>

Co-authored-by: Vinaykumar Bhat <[email protected]>

…park row representation (apache#10778)

* make exceptions more specific

* use hudi avro exception

* Address review comments

* fix unnecessary changes

* add exception wrapping

* style

* address review comments

* remove . from config

* address review comments

* fix merge

* fix checkstyle

* Update hudi-common/src/main/java/org/apache/hudi/exception/HoodieRecordCreationException.java

Co-authored-by: Y Ethan Guo <[email protected]>

* Update hudi-common/src/main/java/org/apache/hudi/exception/HoodieAvroSchemaException.java

Co-authored-by: Y Ethan Guo <[email protected]>

* add javadoc to exception wrapper

---------

Co-authored-by: Jonathan Vexler <=>
Co-authored-by: Y Ethan Guo <[email protected]>

…10959)

…lidator with ignoreFailed mode (apache#10960)

* [HUDI-7571] Add api to get exception details in HoodieMetadataTableValidator with ignoreFailed mode

* Address comments

…e#10963)

…ache#10943)

…e#10967)

…e#10949)

…apache#10974)

…ey (apache#10947)

RecordLevelIndexSupport::filterQueryWithRecordKey() throws a NPE if the EqualTo
query predicate is not of the form `AttributeReference = Literal`. This is because
RecordLevelIndexSupport:::getAttributeLiteralTuple() returns null in such cases which
is then derefercend unconditionally.

This bug was rendering the functional index to not be used even when the query predicate
had spark functions on which functional index is built. Hence these column-stats based functional index
was not pruning files.

This PR makes the following minor changes.
1. Move some methods in RecordLevelIndexSupport into an object to make it static (to aid in unit testing)
2. Fix filterQueryWithRecordKey() by checking for null return values from the call to getAttributeLiteralTuple
3. Add unit tests in TestRecordLevelIndexSupport.scala

Co-authored-by: Vinaykumar Bhat <[email protected]>

…tor#doCommit (apache#10809)

…he#10986)

apache#10984)

…ound (apache#10987)

* [HUDI-7597] Add logs of Kafka offsets when the checkpoint is out of bound

* Adjust test

… for log file HoodieLogFile{xxx} (apache#10989)

* due to java.lang.IllegalStateException: Shutdown in progress, cause:
    when `org.apache.hudi.common.table.log.HoodieLogFormatWriter.close` tries to `removeShutdownHook`, hooks were already removed by JVM when triggered (hooks == null)

…der is closed (apache#10993)

…rics registry (apache#10635)

Currently HoodieMetadataMetrics stores metrics in memory and these metrics are not pushed by the metric reporters. The metric reporters are configured within Metrics instance. List of changes in the PR:

Metrics related classes have been moved from hudi-client-common to hudi-common.
HoodieMetadataMetrics now uses Metrics class so that all the reporters can be supported with it.
Some gaps in configs which are added in HoodieMetadataWriteUtils
Some metrics related apis and functionality has been moved to HoodieMetricsConfig. The HoodieWriteConfig APIs now delegate to HoodieMetricsConfig for the functionality.

…he#10999)

…#10479)

* fix all usages not in tests
* do pass through and fix
* fix test that didn't actually use a cluster commit
* make method private and fix naming
* revert write markers changes

---------

Co-authored-by: Jonathan Vexler <=>

* [HUDI-7601] Add heartbeat mechanism to refresh lock

…llow.multiple (apache#11013)