You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
UnaryPositive is no longer a UnaryExpression (but we have it as a GpuUnaryExpression). It is now a RuntimeReplaceable.
This change was introduced earlier in Spark 4 but it causes issues because UnaryPositive stopped being foldable. In the cast of lag (for window definitions), for example, we can have UnaryPositive in the lag definition as lag(x, +5), and it stopped working after the change to RuntimeReplaceable.
So I am not 100% sure how much of this we care about. I think we need to possibly move to RuntimeReplaceable for UnaryPositive (and possibly others?) for Spark 4. That said, filing this because I think we should investigate this a bit to see what needs to change/what is broken.
The text was updated successfully, but these errors were encountered:
RuntimeReplaceable gets replaced in logical plan translation and we never even see them by the time our code runs. At least that is the theory anyways. I would expect our code to still work so long as we can still compile. It just means that the replacement code will never run.
UnaryPositive
is no longer aUnaryExpression
(but we have it as aGpuUnaryExpression
). It is now aRuntimeReplaceable
.This change was introduced earlier in Spark 4 but it causes issues because
UnaryPositive
stopped being foldable. In the cast oflag
(for window definitions), for example, we can haveUnaryPositive
in the lag definition aslag(x, +5)
, and it stopped working after the change toRuntimeReplaceable
.With apache/spark@0fbd34c45b5#diff-68100e5e6079666747882f5070767601087afc8181f513d997f6e39a948eaa0dR1063,
RuntimeReplaceable
can now be evaluated.So I am not 100% sure how much of this we care about. I think we need to possibly move to
RuntimeReplaceable
forUnaryPositive
(and possibly others?) for Spark 4. That said, filing this because I think we should investigate this a bit to see what needs to change/what is broken.The text was updated successfully, but these errors were encountered: