Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Only recompute schema in TypeCoercion when necessary #10369

Closed
wants to merge 2 commits into from

Conversation

alamb
Copy link
Contributor

@alamb alamb commented May 3, 2024

Draft as it builds on #10356

Which issue does this PR close?

closes #10365

Rationale for this change

recomputing the Schema can be expensive, so only do it when necessary

What changes are included in this PR?

  1. Update TypeCoercionRewriter to track when the expression is actually rewritten (via Transformed::yes)
  2. Only recompute the schema when an expression was rewritten

This is a big change change as it needs to is properly track when expressions were changed which required updating the entire expression rewrite

Are these changes tested?

Existing CI

Are there any user-facing changes?

No functional changes

TODO performance tests

@alamb
Copy link
Contributor Author

alamb commented May 3, 2024

There are a few tests that don't pass for reason's I haven't debugged yet but it looks like recomputing schemas may be masking issues elsewhere. Once #10356 is merged I'll do another profiling run to figure out if recomputing schema shows up in the traces and will keep hacking on this if so

@alamb alamb force-pushed the alamb/type_coercion2 branch from 864eec2 to 8f45ee2 Compare May 3, 2024 16:16
@alamb
Copy link
Contributor Author

alamb commented May 17, 2024

Update here is I don't think I'll have a chance to work on this in the near term unless recomputing the schema shows up in planning benchmarks again.

I think given the level of effort involved, this approach is not likely to yield a large speedup (though I do think this PR's approach would be an improvement over the current code)

@alamb
Copy link
Contributor Author

alamb commented Jun 15, 2024

I realistically don't plan to spend any more time on this :(

@alamb alamb closed this Jun 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
logical-expr Logical plan and expressions optimizer Optimizer rules
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Onyl recompute schema in TypeCoercion when necessary
1 participant