scala 2.13 dependency

set scala dependencies for 2.13

fix delta-core to delta-spark

fix for scala 2.13

[SPARK-45629][CORE][SQL][CONNECT][ML][STREAMING][BUILD][EXAMPLES] Fix `Implicit definition should have explicit type`

### What changes were proposed in this pull request?
This PR aims to fix `Implicit definition should have explicit type` in Scala 2.13.

This pr includes:
1. Declaration types for global variables of implicit
2. Add scala.annotation.warn

### Why are the changes needed?

- For implicit global variables without explicit type declaration, will get warnning :
warning: Implicit definition should have explicit type (inferred String) [quickfixable]
- No modifications are required for local variables.
Additionally, to handle cases involving reflection-related types like ClassTag in implicit variables, the [scala.annotation.warn](https://github.com/scala.annotation.warn) annotation is used to suppress the warning.

Furthermore, warnings generated in Spark will be treated as errors:

[error] ... Implicit definition should have explicit type (inferred org.json4s.DefaultFormats.type) [quickfixable]
...
[error]   implicit val formats = org.json4s.DefaultFormats

Jira link:
SPARK-45314: https://issues.apache.org/jira/browse/SPARK-45629

Related issue link about `implicit` :
scala/bug#5265

### Does this PR introduce _any_ user-facing change?

no

### How was this patch tested?

Most of the testing is completed through CI, and the example module is locally compiled and tested in IDEA Additionally, there are some writing changes that are verified through demo code

### Was this patch authored or co-authored using generative AI tooling?

no

Closes apache#43526 from laglangyue/SPARK-45629.

Lead-authored-by: tangjiafu <[email protected]>
Co-authored-by: tangjiafu <[email protected]>
Signed-off-by: yangjie01 <[email protected]>

# Conflicts:
#	connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala
#	core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala
#	pom.xml
#	project/SparkBuild.scala
#	sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
#	sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/OperatorStateMetadata.scala

Revert "[SPARK-45629][CORE][SQL][CONNECT][ML][STREAMING][BUILD][EXAMPLES] Fix `Implicit definition should have explicit type`"

This reverts commit bc67612.

Fix Implicit definition should have explicit type

fix install error

add args for scala 2.13

fix args

update pom java version

update pom jvm target version

fix implicit types

rollback scala 2.13.8

rollback

fmt

blank space

fix

fix import

fmt

fmt

not fail on violation

R/pkg/tests/fulltests/test_client.R

@@ -37,7 +37,7 @@ test_that("multiple packages don't produce a warning", {
 
 test_that("sparkJars sparkPackages as character vectors", {
   args <- generateSparkSubmitArgs("", "", c("one.jar", "two.jar", "three.jar"), "",
-                                  c("com.databricks:spark-avro_2.12:2.0.1"))
+                                  c("com.databricks:spark-avro_2.13:2.0.1"))
   expect_match(args, "--jars one.jar,two.jar,three.jar")
-  expect_match(args, "--packages com.databricks:spark-avro_2.12:2.0.1")
+  expect_match(args, "--packages com.databricks:spark-avro_2.13:2.0.1")
 })

assembly/pom.xml

@@ -20,12 +20,12 @@
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <groupId>org.apache.spark</groupId>
-    <artifactId>spark-parent_2.12</artifactId>
+    <artifactId>spark-parent_2.13</artifactId>
     <version>3.5.1</version>
     <relativePath>../pom.xml</relativePath>
   </parent>
 
-  <artifactId>spark-assembly_2.12</artifactId>
+  <artifactId>spark-assembly_2.13</artifactId>
   <name>Spark Project Assembly</name>
   <url>https://spark.apache.org/</url>
   <packaging>pom</packaging>

common/kvstore/pom.xml

@@ -21,12 +21,12 @@
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <groupId>org.apache.spark</groupId>
-    <artifactId>spark-parent_2.12</artifactId>
+    <artifactId>spark-parent_2.13</artifactId>
     <version>3.5.1</version>
     <relativePath>../../pom.xml</relativePath>
   </parent>
 
-  <artifactId>spark-kvstore_2.12</artifactId>
+  <artifactId>spark-kvstore_2.13</artifactId>
   <packaging>jar</packaging>
   <name>Spark Project Local DB</name>
   <url>https://spark.apache.org/</url>

common/network-common/pom.xml

@@ -21,12 +21,12 @@
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <groupId>org.apache.spark</groupId>
-    <artifactId>spark-parent_2.12</artifactId>
+    <artifactId>spark-parent_2.13</artifactId>
     <version>3.5.1</version>
     <relativePath>../../pom.xml</relativePath>
   </parent>
 
-  <artifactId>spark-network-common_2.12</artifactId>
+  <artifactId>spark-network-common_2.13</artifactId>
   <packaging>jar</packaging>
   <name>Spark Project Networking</name>
   <url>https://spark.apache.org/</url>

common/network-shuffle/pom.xml

@@ -21,12 +21,12 @@
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <groupId>org.apache.spark</groupId>
-    <artifactId>spark-parent_2.12</artifactId>
+    <artifactId>spark-parent_2.13</artifactId>
     <version>3.5.1</version>
     <relativePath>../../pom.xml</relativePath>
   </parent>
 
-  <artifactId>spark-network-shuffle_2.12</artifactId>
+  <artifactId>spark-network-shuffle_2.13</artifactId>
   <packaging>jar</packaging>
   <name>Spark Project Shuffle Streaming Service</name>
   <url>https://spark.apache.org/</url>

common/network-yarn/pom.xml

@@ -21,12 +21,12 @@
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <groupId>org.apache.spark</groupId>
-    <artifactId>spark-parent_2.12</artifactId>
+    <artifactId>spark-parent_2.13</artifactId>
     <version>3.5.1</version>
     <relativePath>../../pom.xml</relativePath>
   </parent>
 
-  <artifactId>spark-network-yarn_2.12</artifactId>
+  <artifactId>spark-network-yarn_2.13</artifactId>
   <packaging>jar</packaging>
   <name>Spark Project YARN Shuffle Service</name>
   <url>https://spark.apache.org/</url>

common/sketch/pom.xml

@@ -21,12 +21,12 @@
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <groupId>org.apache.spark</groupId>
-    <artifactId>spark-parent_2.12</artifactId>
+    <artifactId>spark-parent_2.13</artifactId>
     <version>3.5.1</version>
     <relativePath>../../pom.xml</relativePath>
   </parent>
 
-  <artifactId>spark-sketch_2.12</artifactId>
+  <artifactId>spark-sketch_2.13</artifactId>
   <packaging>jar</packaging>
   <name>Spark Project Sketch</name>
   <url>https://spark.apache.org/</url>

common/tags/pom.xml

@@ -21,12 +21,12 @@
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <groupId>org.apache.spark</groupId>
-    <artifactId>spark-parent_2.12</artifactId>
+    <artifactId>spark-parent_2.13</artifactId>
     <version>3.5.1</version>
     <relativePath>../../pom.xml</relativePath>
   </parent>
 
-  <artifactId>spark-tags_2.12</artifactId>
+  <artifactId>spark-tags_2.13</artifactId>
   <packaging>jar</packaging>
   <name>Spark Project Tags</name>
   <url>https://spark.apache.org/</url>

common/unsafe/pom.xml

@@ -21,12 +21,12 @@
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <groupId>org.apache.spark</groupId>
-    <artifactId>spark-parent_2.12</artifactId>
+    <artifactId>spark-parent_2.13</artifactId>
     <version>3.5.1</version>
     <relativePath>../../pom.xml</relativePath>
   </parent>
 
-  <artifactId>spark-unsafe_2.12</artifactId>
+  <artifactId>spark-unsafe_2.13</artifactId>
   <packaging>jar</packaging>
   <name>Spark Project Unsafe</name>
   <url>https://spark.apache.org/</url>

common/utils/pom.xml

@@ -21,12 +21,12 @@
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <groupId>org.apache.spark</groupId>
-    <artifactId>spark-parent_2.12</artifactId>
+    <artifactId>spark-parent_2.13</artifactId>
     <version>3.5.1</version>
     <relativePath>../../pom.xml</relativePath>
   </parent>
 
-  <artifactId>spark-common-utils_2.12</artifactId>
+  <artifactId>spark-common-utils_2.13</artifactId>
   <packaging>jar</packaging>
   <name>Spark Project Common Utils</name>
   <url>https://spark.apache.org/</url>

connector/avro/pom.xml

@@ -20,12 +20,12 @@
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <groupId>org.apache.spark</groupId>
-    <artifactId>spark-parent_2.12</artifactId>
+    <artifactId>spark-parent_2.13</artifactId>
     <version>3.5.1</version>
     <relativePath>../../pom.xml</relativePath>
   </parent>
 
-  <artifactId>spark-avro_2.12</artifactId>
+  <artifactId>spark-avro_2.13</artifactId>
   <properties>
     <sbt.project.name>avro</sbt.project.name>
   </properties>

connector/connect/client/jvm/pom.xml

@@ -21,12 +21,12 @@
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <groupId>org.apache.spark</groupId>
-    <artifactId>spark-parent_2.12</artifactId>
+    <artifactId>spark-parent_2.13</artifactId>
     <version>3.5.1</version>
     <relativePath>../../../../pom.xml</relativePath>
   </parent>
 
-  <artifactId>spark-connect-client-jvm_2.12</artifactId>
+  <artifactId>spark-connect-client-jvm_2.13</artifactId>
   <packaging>jar</packaging>
   <name>Spark Project Connect Client</name>
   <url>https://spark.apache.org/</url>

connector/connect/common/pom.xml

@@ -21,12 +21,12 @@
     <modelVersion>4.0.0</modelVersion>
     <parent>
         <groupId>org.apache.spark</groupId>
-        <artifactId>spark-parent_2.12</artifactId>
+        <artifactId>spark-parent_2.13</artifactId>
         <version>3.5.1</version>
         <relativePath>../../../pom.xml</relativePath>
     </parent>
 
-    <artifactId>spark-connect-common_2.12</artifactId>
+    <artifactId>spark-connect-common_2.13</artifactId>
     <packaging>jar</packaging>
     <name>Spark Project Connect Common</name>
     <url>https://spark.apache.org/</url>

connector/connect/server/pom.xml

@@ -21,12 +21,12 @@
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <groupId>org.apache.spark</groupId>
-    <artifactId>spark-parent_2.12</artifactId>
+    <artifactId>spark-parent_2.13</artifactId>
     <version>3.5.1</version>
     <relativePath>../../../pom.xml</relativePath>
   </parent>
 
-  <artifactId>spark-connect_2.12</artifactId>
+  <artifactId>spark-connect_2.13</artifactId>
   <packaging>jar</packaging>
   <name>Spark Project Connect Server</name>
   <url>https://spark.apache.org/</url>

connector/docker-integration-tests/pom.xml

@@ -21,12 +21,12 @@
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <groupId>org.apache.spark</groupId>
-    <artifactId>spark-parent_2.12</artifactId>
+    <artifactId>spark-parent_2.13</artifactId>
     <version>3.5.1</version>
     <relativePath>../../pom.xml</relativePath>
   </parent>
 
-  <artifactId>spark-docker-integration-tests_2.12</artifactId>
+  <artifactId>spark-docker-integration-tests_2.13</artifactId>
   <packaging>jar</packaging>
   <name>Spark Project Docker Integration Tests</name>
   <url>https://spark.apache.org/</url>

connector/kafka-0-10-assembly/pom.xml

@@ -20,12 +20,12 @@
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <groupId>org.apache.spark</groupId>
-    <artifactId>spark-parent_2.12</artifactId>
+    <artifactId>spark-parent_2.13</artifactId>
     <version>3.5.1</version>
     <relativePath>../../pom.xml</relativePath>
   </parent>
 
-  <artifactId>spark-streaming-kafka-0-10-assembly_2.12</artifactId>
+  <artifactId>spark-streaming-kafka-0-10-assembly_2.13</artifactId>
   <packaging>jar</packaging>
   <name>Spark Integration for Kafka 0.10 Assembly</name>
   <url>https://spark.apache.org/</url>

connector/kafka-0-10-sql/pom.xml

@@ -20,13 +20,13 @@
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <groupId>org.apache.spark</groupId>
-    <artifactId>spark-parent_2.12</artifactId>
+    <artifactId>spark-parent_2.13</artifactId>
     <version>3.5.1</version>
     <relativePath>../../pom.xml</relativePath>
   </parent>
 
   <groupId>org.apache.spark</groupId>
-  <artifactId>spark-sql-kafka-0-10_2.12</artifactId>
+  <artifactId>spark-sql-kafka-0-10_2.13</artifactId>
   <properties>
     <sbt.project.name>sql-kafka-0-10</sbt.project.name>
   </properties>

connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/JsonUtils.scala

@@ -21,14 +21,14 @@ import scala.collection.mutable.HashMap
 import scala.util.control.NonFatal
 
 import org.apache.kafka.common.TopicPartition
-import org.json4s.NoTypeHints
+import org.json4s.{Formats, NoTypeHints}
 import org.json4s.jackson.Serialization
 
 /**
  * Utilities for converting Kafka related objects to and from json.
  */
 private object JsonUtils {
-  private implicit val formats = Serialization.formats(NoTypeHints)
+  private implicit val formats: Formats = Serialization.formats(NoTypeHints)
 
   /**
    * Read TopicPartitions from json string
@@ -96,10 +96,8 @@ private object JsonUtils {
    */
   def partitionOffsets(partitionOffsets: Map[TopicPartition, Long]): String = {
     val result = new HashMap[String, HashMap[Int, Long]]()
-    implicit val order = new Ordering[TopicPartition] {
-      override def compare(x: TopicPartition, y: TopicPartition): Int = {
+    implicit val order: Ordering[TopicPartition] = (x: TopicPartition, y: TopicPartition) => {
         Ordering.Tuple2[String, Int].compare((x.topic, x.partition), (y.topic, y.partition))
-      }
     }
     val partitions = partitionOffsets.keySet.toSeq.sorted  // sort for more determinism
     partitions.foreach { tp =>

connector/kafka-0-10-token-provider/pom.xml

@@ -20,13 +20,13 @@
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <groupId>org.apache.spark</groupId>
-    <artifactId>spark-parent_2.12</artifactId>
+    <artifactId>spark-parent_2.13</artifactId>
     <version>3.5.1</version>
     <relativePath>../../pom.xml</relativePath>
   </parent>
 
   <groupId>org.apache.spark</groupId>
-  <artifactId>spark-token-provider-kafka-0-10_2.12</artifactId>
+  <artifactId>spark-token-provider-kafka-0-10_2.13</artifactId>
   <properties>
     <sbt.project.name>token-provider-kafka-0-10</sbt.project.name>
   </properties>

connector/kafka-0-10/pom.xml

@@ -20,12 +20,12 @@
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <groupId>org.apache.spark</groupId>
-    <artifactId>spark-parent_2.12</artifactId>
+    <artifactId>spark-parent_2.13</artifactId>
     <version>3.5.1</version>
     <relativePath>../../pom.xml</relativePath>
   </parent>
 
-  <artifactId>spark-streaming-kafka-0-10_2.12</artifactId>
+  <artifactId>spark-streaming-kafka-0-10_2.13</artifactId>
   <properties>
     <sbt.project.name>streaming-kafka-0-10</sbt.project.name>
   </properties>

connector/kinesis-asl-assembly/pom.xml

@@ -20,12 +20,12 @@
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <groupId>org.apache.spark</groupId>
-    <artifactId>spark-parent_2.12</artifactId>
+    <artifactId>spark-parent_2.13</artifactId>
     <version>3.5.1</version>
     <relativePath>../../pom.xml</relativePath>
   </parent>
 
-  <artifactId>spark-streaming-kinesis-asl-assembly_2.12</artifactId>
+  <artifactId>spark-streaming-kinesis-asl-assembly_2.13</artifactId>
   <packaging>jar</packaging>
   <name>Spark Project Kinesis Assembly</name>
   <url>https://spark.apache.org/</url>

connector/kinesis-asl/pom.xml

@@ -19,13 +19,13 @@
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <groupId>org.apache.spark</groupId>
-    <artifactId>spark-parent_2.12</artifactId>
+    <artifactId>spark-parent_2.13</artifactId>
     <version>3.5.1</version>
     <relativePath>../../pom.xml</relativePath>
   </parent>
 
   <!-- Kinesis integration is not included by default due to ASL-licensed code. -->
-  <artifactId>spark-streaming-kinesis-asl_2.12</artifactId>
+  <artifactId>spark-streaming-kinesis-asl_2.13</artifactId>
   <packaging>jar</packaging>
   <name>Spark Kinesis Integration</name>
 

connector/protobuf/pom.xml

@@ -20,12 +20,12 @@
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <groupId>org.apache.spark</groupId>
-    <artifactId>spark-parent_2.12</artifactId>
+    <artifactId>spark-parent_2.13</artifactId>
     <version>3.5.1</version>
     <relativePath>../../pom.xml</relativePath>
   </parent>
 
-  <artifactId>spark-protobuf_2.12</artifactId>
+  <artifactId>spark-protobuf_2.13</artifactId>
   <properties>
     <sbt.project.name>protobuf</sbt.project.name>
   </properties>

connector/spark-ganglia-lgpl/pom.xml

@@ -19,13 +19,13 @@
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <groupId>org.apache.spark</groupId>
-    <artifactId>spark-parent_2.12</artifactId>
+    <artifactId>spark-parent_2.13</artifactId>
     <version>3.5.1</version>
     <relativePath>../../pom.xml</relativePath>
   </parent>
 
   <!-- Ganglia integration is not included by default due to LGPL-licensed code -->
-  <artifactId>spark-ganglia-lgpl_2.12</artifactId>
+  <artifactId>spark-ganglia-lgpl_2.13</artifactId>
   <packaging>jar</packaging>
   <name>Spark Ganglia Integration</name>
 

core/pom.xml

@@ -20,12 +20,12 @@
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <groupId>org.apache.spark</groupId>
-    <artifactId>spark-parent_2.12</artifactId>
+    <artifactId>spark-parent_2.13</artifactId>
     <version>3.5.1</version>
     <relativePath>../pom.xml</relativePath>
   </parent>
 
-  <artifactId>spark-core_2.12</artifactId>
+  <artifactId>spark-core_2.13</artifactId>
   <packaging>jar</packaging>
   <name>Spark Project Core</name>
   <url>https://spark.apache.org/</url>