-
Notifications
You must be signed in to change notification settings - Fork 67
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
When looking for source paths to parallelize, the optimizer.walkEntries examines only the first operator in a sequence. This prevents the optimizer from parallelizing a pool scan in the right input of a join expressed in subquery syntax, like from left | join (from right | ...) on left_key=right_key Change walkEntries to examine all operators in a sequence so almost all source paths are parallelized. (The exceptions are paths inside dag.Over and paths containing no operator after the source.)
- Loading branch information
Showing
2 changed files
with
37 additions
and
7 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
script: | | ||
export ZED_LAKE=test | ||
zed init -q | ||
zed create -q -orderby ts test | ||
# At the time of writing, the where operator is necessary because a pool scan | ||
# is parallelized only when followed by another operator. | ||
zc -C -P 2 "from test | join (from test | where true) on a=b" | sed -e 's/pool .*/.../' | ||
outputs: | ||
- name: stdout | ||
data: | | ||
lister ... | ||
| slicer | ||
| scatter ( | ||
=> | ||
seqscan ... | ||
=> | ||
seqscan ... | ||
) | ||
| merge ts:asc | ||
| fork ( | ||
=> | ||
pass | ||
=> | ||
lister ... | ||
| slicer | ||
| scatter ( | ||
=> | ||
seqscan ... | ||
=> | ||
seqscan ... | ||
) | ||
| merge ts:asc | ||
) | ||
| join on a=b |