Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add docs to update_coalesce_ctx_children. #14907

Merged
merged 5 commits into from
Feb 28, 2025

Conversation

wiedld
Copy link
Contributor

@wiedld wiedld commented Feb 27, 2025

Which issue does this PR close?

Rationale for this change

The initial PR to fix the above issue, also did a refactor and added docs.
I'm breaking up the initial PR, and this is only the refactor the docs portion.

What changes are included in this PR?

Refactor and add docs to more tightly define update_coalesce_ctx_children. Afterwards, I could remove a few of the conditionals in the other helper methods, since it's now handled with the proper and full labeling of the coalesce context (of which coalesce to remove).

Only docs.

Are these changes tested?

N/A

Are there any user-facing changes?

No

…text, selectively indicating when coalesce should be removed
@github-actions github-actions bot added the optimizer Optimizer rules label Feb 27, 2025
@wiedld wiedld changed the title refactor Refactor update_coalesce_ctx_children. Feb 27, 2025
@wiedld wiedld force-pushed the wiedld/refactor-coalesce-ctx branch from 8b1019a to 535e74f Compare February 27, 2025 04:10
@wiedld wiedld marked this pull request as ready for review February 27, 2025 04:15
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @wiedld -- I left some comments. I love the new comments. I am not sure about some of the refactoring

// since we are checking distribution requirements after the coalesce occurs
let parent = &coalesce_context.plan;

for child_context in coalesce_context.children.iter_mut() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this method used to check is_coalesce_partitions once before iterating through the children -- this PR now checks for each child.

As written this loop seems to update child_context.data multiple times (once in each loop body)

I think it is likely the same given the coalesce partitions has a single child, but I found the previous implementation easier to understand

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update_coalesce_ctx_children(&mut requirements);
let coalesce_can_be_removed = requirements.children.iter().any(|child| child.data);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this variable is called coalesce_can_be_removed but the comments above say that .data means:

The data
/// attribute stores whether the plan is a CoalescePartitionsExec or is
/// connected to a CoalescePartitionsExec via its children.

This seems inconsistent to me

Copy link
Contributor Author

@wiedld wiedld Feb 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I switched it to be docs only: #14907 (comment)
Then if I move forward with another refactoring attempt, I'll first make an issue to justify the refactor.

@berkaysynnada
Copy link
Contributor

If this is ready, I'd also like to take a look

@wiedld wiedld changed the title Refactor update_coalesce_ctx_children. Add docs to update_coalesce_ctx_children. Feb 27, 2025
@wiedld
Copy link
Contributor Author

wiedld commented Feb 27, 2025

@berkaysynnada and @alamb -- I just switched it to be only the docs in this PR.
And I'm about to push up pushed up a PR which is only the regression test for this issue.

That way we have some forward progress for now. And then we can have further discussions/decisions about what to do.

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's do this!

Copy link
Contributor

@berkaysynnada berkaysynnada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @wiedld to enhance documentation

@berkaysynnada berkaysynnada merged commit 101389d into apache:main Feb 28, 2025
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
optimizer Optimizer rules
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants