Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG][AUDIT][SPARK-50377][SQL] Allow to evaluate foldable RuntimeReplaceable #11922

Open
abellina opened this issue Jan 6, 2025 · 1 comment
Labels
audit_4.0.0 Audit related tasks for 4.0.0 bug Something isn't working

Comments

@abellina
Copy link
Collaborator

abellina commented Jan 6, 2025

UnaryPositive is no longer a UnaryExpression (but we have it as a GpuUnaryExpression). It is now a RuntimeReplaceable.

This change was introduced earlier in Spark 4 but it causes issues because UnaryPositive stopped being foldable. In the cast of lag (for window definitions), for example, we can have UnaryPositive in the lag definition as lag(x, +5), and it stopped working after the change to RuntimeReplaceable.

With apache/spark@0fbd34c45b5#diff-68100e5e6079666747882f5070767601087afc8181f513d997f6e39a948eaa0dR1063, RuntimeReplaceable can now be evaluated.

So I am not 100% sure how much of this we care about. I think we need to possibly move to RuntimeReplaceable for UnaryPositive (and possibly others?) for Spark 4. That said, filing this because I think we should investigate this a bit to see what needs to change/what is broken.

@abellina abellina added ? - Needs Triage Need team to review and classify audit_4.0.0 Audit related tasks for 4.0.0 bug Something isn't working labels Jan 6, 2025
@revans2
Copy link
Collaborator

revans2 commented Jan 6, 2025

RuntimeReplaceable gets replaced in logical plan translation and we never even see them by the time our code runs. At least that is the theory anyways. I would expect our code to still work so long as we can still compile. It just means that the replacement code will never run.

@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Jan 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
audit_4.0.0 Audit related tasks for 4.0.0 bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants