Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize filtered SortMergeJoin to avoid producing small/empty batches #14050

Open
comphead opened this issue Jan 8, 2025 · 0 comments
Open

Optimize filtered SortMergeJoin to avoid producing small/empty batches #14050

comphead opened this issue Jan 8, 2025 · 0 comments
Assignees
@comphead
Labels
enhancement New feature or request

Comments

@comphead
Copy link
Contributor

comphead commented Jan 8, 2025

Is your feature request related to a problem or challenge?

Related to #9846

In #9846 there is a couple of tasks to fix the correctness issues for SortMergeJoin with filter clause

As reported in apache/datafusion-comet#1211 (comment) the execution produces a lot of small/empty batches

Describe the solution you'd like

Concat small batches and produces batches equal or near equal to batch_size

Describe alternatives you've considered

No response

Additional context

No response
@comphead comphead added the enhancement New feature or request label Jan 8, 2025
@comphead comphead self-assigned this Jan 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@comphead comphead

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@comphead