Skip to content

Commit

Permalink
[SPARK-49476][SQL] Fix nullability of base64 function
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

Fix the nullability of the `Base64` expression to be based on the child's nullability, and not always be nullable.

### Why are the changes needed?

apache#47303 had a side effect of changing the nullability by the switch to using `StaticInvoke`. This was also backported to Spark 3.5.2 and caused schema mismatch errors for stateful streams when we upgraded. This restores the previous behavior which is supported by StaticInvoke through the `returnNullable` argument. If the child is non-nullable, we know the result will be non-nullable.

### Does this PR introduce _any_ user-facing change?

Restores the nullability of the `Base64` expression to what is was in Spark 3.5.1 and earlier.

### How was this patch tested?

New UT

### Was this patch authored or co-authored using generative AI tooling?

No

Closes apache#47941 from Kimahriman/base64-nullability.

Lead-authored-by: Adam Binford <[email protected]>
Co-authored-by: Maxim Gekk <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
  • Loading branch information
Kimahriman and MaxGekk committed Sep 2, 2024
1 parent cef3c86 commit c274c5a
Show file tree
Hide file tree
Showing 2 changed files with 9 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -2714,7 +2714,8 @@ case class Base64(child: Expression, chunkBase64: Boolean)
dataType,
"encode",
Seq(child, Literal(chunkBase64, BooleanType)),
Seq(BinaryType, BooleanType))
Seq(BinaryType, BooleanType),
returnNullable = false)

override def toString: String = s"$prettyName($child)"

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -467,6 +467,13 @@ class StringExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
val b = $"b".binary.at(0)
val bytes = Array[Byte](1, 2, 3, 4)

assert(!Base64(Literal(bytes)).nullable)
assert(Base64(Literal.create(null, BinaryType)).nullable)
assert(Base64(Literal(bytes).castNullable()).nullable)
assert(!UnBase64(Literal("AQIDBA==")).nullable)
assert(UnBase64(Literal.create(null, StringType)).nullable)
assert(UnBase64(Literal("AQIDBA==").castNullable()).nullable)

checkEvaluation(Base64(Literal(bytes)), "AQIDBA==", create_row("abdef"))
checkEvaluation(Base64(UnBase64(Literal("AQIDBA=="))), "AQIDBA==", create_row("abdef"))
checkEvaluation(Base64(UnBase64(Literal(""))), "", create_row("abdef"))
Expand Down

0 comments on commit c274c5a

Please sign in to comment.