Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[auto-merge] branch-24.12 to branch-25.02 [skip ci] [bot] #11948

Merged
merged 3 commits into from
Jan 10, 2025
Merged

[auto-merge] branch-24.12 to branch-25.02 [skip ci] [bot] #11948

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
db484e1
Update rapids JNI dependency to 24.12.1 (#11943)
nvauto Jan 10, 2025
c3b373d
Update version to 24.12.1-SNAPSHOT [skip ci] (#11947)
nvauto Jan 10, 2025
6e3f6eb
Update latest changelog [skip ci] (#11950)
nvauto Jan 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Change log
Generated on 2024-12-16
Generated on 2025-01-10

## Release 24.12

Expand All @@ -24,7 +24,10 @@ Generated on 2024-12-16
### Bugs Fixed
|||
|:---|:---|
|[#11842](https://github.com/NVIDIA/spark-rapids/issues/11842)|[BUG] udf-examples-native case failed core dump|
|[#11718](https://github.com/NVIDIA/spark-rapids/issues/11718)|[BUG] update date/time APIs in CUDF java to avoid deprecated functions|
|[#10907](https://github.com/NVIDIA/spark-rapids/issues/10907)|from_json function parses a column containing an empty array, throws an exception.|
|[#11807](https://github.com/NVIDIA/spark-rapids/issues/11807)|[BUG] mismatched cpu and gpu result in test_lead_lag_for_structs_with_arrays intermittently|
|[#11793](https://github.com/NVIDIA/spark-rapids/issues/11793)|[BUG] "Time in Heuristic" should not include previous operator's compute time|
|[#11798](https://github.com/NVIDIA/spark-rapids/issues/11798)|[BUG] mismatch CPU and GPU result in test_months_between_first_day[DATAGEN_SEED=1733006411, TZ=Africa/Casablanca]|
|[#11790](https://github.com/NVIDIA/spark-rapids/issues/11790)|[BUG] test_hash_* failed "java.util.NoSuchElementException: head of empty list" or "Too many times of repartition, may hit a bug?"|
Expand Down Expand Up @@ -72,6 +75,10 @@ Generated on 2024-12-16
### PRs
|||
|:---|:---|
|[#11947](https://github.com/NVIDIA/spark-rapids/pull/11947)|Update version to 24.12.1-SNAPSHOT [skip ci]|
|[#11943](https://github.com/NVIDIA/spark-rapids/pull/11943)|Update rapids JNI dependency to 24.12.1|
|[#11944](https://github.com/NVIDIA/spark-rapids/pull/11944)|Update download page for 24.12.1 hot fix release [skip ci]|
|[#11876](https://github.com/NVIDIA/spark-rapids/pull/11876)|Update latest changelog [skip ci]|
|[#11874](https://github.com/NVIDIA/spark-rapids/pull/11874)|Remove 350db143 shim's build [skip ci]|
|[#11851](https://github.com/NVIDIA/spark-rapids/pull/11851)|Update latest changelog [skip ci]|
|[#11849](https://github.com/NVIDIA/spark-rapids/pull/11849)|Update rapids JNI and private dependency to 24.12.0|
Expand Down Expand Up @@ -236,8 +243,6 @@ Generated on 2024-12-16
|[#11183](https://github.com/NVIDIA/spark-rapids/issues/11183)|[BUG] Failed to split an empty string with error "ai.rapids.cudf.CudfException: parallel_for failed: cudaErrorInvalidDevice: invalid device ordinal"|
|[#11008](https://github.com/NVIDIA/spark-rapids/issues/11008)|Fix tests failures in ast_test.py|
|[#11265](https://github.com/NVIDIA/spark-rapids/issues/11265)|[BUG] segfaults seen in cuDF after prefetch calls intermittently|
|[#11025](https://github.com/NVIDIA/spark-rapids/issues/11025)|Fix tests failures in date_time_test.py|
|[#11065](https://github.com/NVIDIA/spark-rapids/issues/11065)|[BUG] Spark Connect Server (3.5.1) Can Not Running Correctly|

### PRs
|||
Expand Down
8 changes: 4 additions & 4 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,15 +127,15 @@ mvn -pl dist -PnoSnapshots package -DskipTests
Verify that shim-specific classes are hidden from a conventional classloader.

```bash
$ javap -cp dist/target/rapids-4-spark_2.12-24.12.0-SNAPSHOT-cuda11.jar com.nvidia.spark.rapids.shims.SparkShimImpl
$ javap -cp dist/target/rapids-4-spark_2.12-24.12.1-SNAPSHOT-cuda11.jar com.nvidia.spark.rapids.shims.SparkShimImpl
Error: class not found: com.nvidia.spark.rapids.shims.SparkShimImpl
```

However, its bytecode can be loaded if prefixed with `spark3XY` not contained in the package name

```bash
$ javap -cp dist/target/rapids-4-spark_2.12-24.12.0-SNAPSHOT-cuda11.jar spark320.com.nvidia.spark.rapids.shims.SparkShimImpl | head -2
Warning: File dist/target/rapids-4-spark_2.12-24.12.0-SNAPSHOT-cuda11.jar(/spark320/com/nvidia/spark/rapids/shims/SparkShimImpl.class) does not contain class spark320.com.nvidia.spark.rapids.shims.SparkShimImpl
$ javap -cp dist/target/rapids-4-spark_2.12-24.12.1-SNAPSHOT-cuda11.jar spark320.com.nvidia.spark.rapids.shims.SparkShimImpl | head -2
Warning: File dist/target/rapids-4-spark_2.12-24.12.1-SNAPSHOT-cuda11.jar(/spark320/com/nvidia/spark/rapids/shims/SparkShimImpl.class) does not contain class spark320.com.nvidia.spark.rapids.shims.SparkShimImpl
Compiled from "SparkShims.scala"
public final class com.nvidia.spark.rapids.shims.SparkShimImpl {
```
Expand Down Expand Up @@ -178,7 +178,7 @@ mvn package -pl dist -am -Dbuildver=340 -DallowConventionalDistJar=true
Verify `com.nvidia.spark.rapids.shims.SparkShimImpl` is conventionally loadable:

```bash
$ javap -cp dist/target/rapids-4-spark_2.12-24.12.0-SNAPSHOT-cuda11.jar com.nvidia.spark.rapids.shims.SparkShimImpl | head -2
$ javap -cp dist/target/rapids-4-spark_2.12-24.12.1-SNAPSHOT-cuda11.jar com.nvidia.spark.rapids.shims.SparkShimImpl | head -2
Compiled from "SparkShims.scala"
public final class com.nvidia.spark.rapids.shims.SparkShimImpl {
```
Expand Down
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ as a `provided` dependency.
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark_2.12</artifactId>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>
<scope>provided</scope>
</dependency>
```
6 changes: 3 additions & 3 deletions aggregator/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright (c) 2021-2024, NVIDIA CORPORATION.
Copyright (c) 2021-2025, NVIDIA CORPORATION.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Expand All @@ -22,13 +22,13 @@
<parent>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-jdk-profiles_2.12</artifactId>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>
<relativePath>../jdk-profiles/pom.xml</relativePath>
</parent>
<artifactId>rapids-4-spark-aggregator_2.12</artifactId>
<name>RAPIDS Accelerator for Apache Spark Aggregator</name>
<description>Creates an aggregated shaded package of the RAPIDS plugin for Apache Spark</description>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>

<properties>
<rapids.module>aggregator</rapids.module>
Expand Down
6 changes: 3 additions & 3 deletions api_validation/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright (c) 2020-2024, NVIDIA CORPORATION.
Copyright (c) 2020-2025, NVIDIA CORPORATION.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Expand All @@ -22,11 +22,11 @@
<parent>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-shim-deps-parent_2.12</artifactId>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>
<relativePath>../shim-deps/pom.xml</relativePath>
</parent>
<artifactId>rapids-4-spark-api-validation_2.12</artifactId>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>

<properties>
<rapids.module>api_validation</rapids.module>
Expand Down
6 changes: 3 additions & 3 deletions datagen/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,12 @@ Where `$SPARK_VERSION` is a compressed version number, like 330 for Spark 3.3.0.

After this the jar should be at
`target/datagen_2.12-$PLUGIN_VERSION-spark$SPARK_VERSION.jar`
for example a Spark 3.3.0 jar for the 24.12.0 release would be
`target/datagen_2.12-24.12.0-spark330.jar`
for example a Spark 3.3.0 jar for the 24.12.1 release would be
`target/datagen_2.12-24.12.1-spark330.jar`

To get a spark shell with this you can run
```shell
spark-shell --jars target/datagen_2.12-24.12.0-spark330.jar
spark-shell --jars target/datagen_2.12-24.12.1-spark330.jar
```

After that you should be good to go.
Expand Down
2 changes: 1 addition & 1 deletion datagen/ScaleTest.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ $SPARK_HOME/bin/spark-submit \
--conf spark.sql.parquet.datetimeRebaseModeInWrite=CORRECTED \
--class com.nvidia.rapids.tests.scaletest.ScaleTestDataGen \ # the main class
--jars $SPARK_HOME/examples/jars/scopt_2.12-3.7.1.jar \ # one dependency jar just shipped with Spark under $SPARK_HOME
./target/datagen_2.12-24.12.0-SNAPSHOT-spark332.jar \
./target/datagen_2.12-24.12.1-SNAPSHOT-spark332.jar \
1 \
10 \
parquet \
Expand Down
6 changes: 3 additions & 3 deletions datagen/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright (c) 2023-2024 NVIDIA CORPORATION.
Copyright (c) 2023-2025 NVIDIA CORPORATION.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Expand All @@ -21,13 +21,13 @@
<parent>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-shim-deps-parent_2.12</artifactId>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>
<relativePath>../shim-deps/pom.xml</relativePath>
</parent>
<artifactId>datagen_2.12</artifactId>
<name>Data Generator</name>
<description>Tools for generating large amounts of data</description>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>
<properties>
<rapids.module>datagen</rapids.module>
<target.classifier/>
Expand Down
6 changes: 3 additions & 3 deletions delta-lake/delta-20x/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright (c) 2023-2024 NVIDIA CORPORATION.
Copyright (c) 2023-2025 NVIDIA CORPORATION.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Expand All @@ -22,14 +22,14 @@
<parent>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-jdk-profiles_2.12</artifactId>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>
<relativePath>../../jdk-profiles/pom.xml</relativePath>
</parent>

<artifactId>rapids-4-spark-delta-20x_2.12</artifactId>
<name>RAPIDS Accelerator for Apache Spark Delta Lake 2.0.x Support</name>
<description>Delta Lake 2.0.x support for the RAPIDS Accelerator for Apache Spark</description>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>

<properties>
<rapids.module>../delta-lake/delta-20x</rapids.module>
Expand Down
6 changes: 3 additions & 3 deletions delta-lake/delta-21x/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright (c) 2023-2024 NVIDIA CORPORATION.
Copyright (c) 2023-2025 NVIDIA CORPORATION.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Expand All @@ -22,14 +22,14 @@
<parent>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-jdk-profiles_2.12</artifactId>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>
<relativePath>../../jdk-profiles/pom.xml</relativePath>
</parent>

<artifactId>rapids-4-spark-delta-21x_2.12</artifactId>
<name>RAPIDS Accelerator for Apache Spark Delta Lake 2.1.x Support</name>
<description>Delta Lake 2.1.x support for the RAPIDS Accelerator for Apache Spark</description>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>

<properties>
<rapids.module>../delta-lake/delta-21x</rapids.module>
Expand Down
6 changes: 3 additions & 3 deletions delta-lake/delta-22x/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright (c) 2023-2024 NVIDIA CORPORATION.
Copyright (c) 2023-2025 NVIDIA CORPORATION.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Expand All @@ -22,14 +22,14 @@
<parent>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-jdk-profiles_2.12</artifactId>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>
<relativePath>../../jdk-profiles/pom.xml</relativePath>
</parent>

<artifactId>rapids-4-spark-delta-22x_2.12</artifactId>
<name>RAPIDS Accelerator for Apache Spark Delta Lake 2.2.x Support</name>
<description>Delta Lake 2.2.x support for the RAPIDS Accelerator for Apache Spark</description>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>

<properties>
<rapids.module>../delta-lake/delta-22x</rapids.module>
Expand Down
6 changes: 3 additions & 3 deletions delta-lake/delta-23x/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright (c) 2023-2024 NVIDIA CORPORATION.
Copyright (c) 2023-2025 NVIDIA CORPORATION.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Expand All @@ -22,14 +22,14 @@
<parent>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-parent_2.12</artifactId>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

<artifactId>rapids-4-spark-delta-23x_2.12</artifactId>
<name>RAPIDS Accelerator for Apache Spark Delta Lake 2.3.x Support</name>
<description>Delta Lake 2.3.x support for the RAPIDS Accelerator for Apache Spark</description>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>

<properties>
<rapids.module>../delta-lake/delta-23x</rapids.module>
Expand Down
6 changes: 3 additions & 3 deletions delta-lake/delta-24x/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright (c) 2023-2024 NVIDIA CORPORATION.
Copyright (c) 2023-2025 NVIDIA CORPORATION.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Expand All @@ -22,14 +22,14 @@
<parent>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-jdk-profiles_2.12</artifactId>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>
<relativePath>../../jdk-profiles/pom.xml</relativePath>
</parent>

<artifactId>rapids-4-spark-delta-24x_2.12</artifactId>
<name>RAPIDS Accelerator for Apache Spark Delta Lake 2.4.x Support</name>
<description>Delta Lake 2.4.x support for the RAPIDS Accelerator for Apache Spark</description>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>

<properties>
<rapids.module>../delta-lake/delta-24x</rapids.module>
Expand Down
6 changes: 3 additions & 3 deletions delta-lake/delta-spark330db/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright (c) 2023-2024 NVIDIA CORPORATION.
Copyright (c) 2023-2025 NVIDIA CORPORATION.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Expand All @@ -22,14 +22,14 @@
<parent>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-shim-deps-parent_2.12</artifactId>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>
<relativePath>../../shim-deps/pom.xml</relativePath>
</parent>

<artifactId>rapids-4-spark-delta-spark330db_2.12</artifactId>
<name>RAPIDS Accelerator for Apache Spark Databricks 11.3 Delta Lake Support</name>
<description>Databricks 11.3 Delta Lake support for the RAPIDS Accelerator for Apache Spark</description>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>

<properties>
<rapids.module>../delta-lake/delta-spark330db</rapids.module>
Expand Down
6 changes: 3 additions & 3 deletions delta-lake/delta-spark332db/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright (c) 2023-2024 NVIDIA CORPORATION.
Copyright (c) 2023-2025 NVIDIA CORPORATION.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Expand All @@ -22,14 +22,14 @@
<parent>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-shim-deps-parent_2.12</artifactId>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>
<relativePath>../../shim-deps/pom.xml</relativePath>
</parent>

<artifactId>rapids-4-spark-delta-spark332db_2.12</artifactId>
<name>RAPIDS Accelerator for Apache Spark Databricks 12.2 Delta Lake Support</name>
<description>Databricks 12.2 Delta Lake support for the RAPIDS Accelerator for Apache Spark</description>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>

<properties>
<rapids.module>../delta-lake/delta-spark332db</rapids.module>
Expand Down
6 changes: 3 additions & 3 deletions delta-lake/delta-spark341db/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright (c) 2023-2024 NVIDIA CORPORATION.
Copyright (c) 2023-2025 NVIDIA CORPORATION.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Expand All @@ -22,14 +22,14 @@
<parent>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-shim-deps-parent_2.12</artifactId>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>
<relativePath>../../shim-deps/pom.xml</relativePath>
</parent>

<artifactId>rapids-4-spark-delta-spark341db_2.12</artifactId>
<name>RAPIDS Accelerator for Apache Spark Databricks 13.3 Delta Lake Support</name>
<description>Databricks 13.3 Delta Lake support for the RAPIDS Accelerator for Apache Spark</description>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>

<properties>
<rapids.compressed.artifact>false</rapids.compressed.artifact>
Expand Down
6 changes: 3 additions & 3 deletions delta-lake/delta-spark350db143/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright (c) 2023-2024 NVIDIA CORPORATION.
Copyright (c) 2023-2025 NVIDIA CORPORATION.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Expand All @@ -22,14 +22,14 @@
<parent>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-shim-deps-parent_2.12</artifactId>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>
<relativePath>../../shim-deps/pom.xml</relativePath>
</parent>

<artifactId>rapids-4-spark-delta-spark350db143_2.12</artifactId>
<name>RAPIDS Accelerator for Apache Spark Databricks 13.3 Delta Lake Support</name>
<description>Databricks 13.3 Delta Lake support for the RAPIDS Accelerator for Apache Spark</description>
<version>24.12.0-SNAPSHOT</version>
<version>24.12.1-SNAPSHOT</version>

<properties>
<rapids.compressed.artifact>false</rapids.compressed.artifact>
Expand Down
Loading
Loading