collect_block in QuickwitCollector #4753
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
collect_block + using `first_vals` to batch fetch sort values
fulmicoton
reviewed
Mar 18, 2024
fulmicoton
reviewed
Mar 18, 2024
fulmicoton
reviewed
Mar 18, 2024
fulmicoton
reviewed
Mar 18, 2024
fulmicoton
reviewed
Mar 18, 2024
fulmicoton
reviewed
Mar 18, 2024
fulmicoton
reviewed
Mar 18, 2024
fulmicoton
reviewed
Mar 18, 2024
fulmicoton
reviewed
Mar 18, 2024
| let mut len = 0; | ||
| for &doc in docs { | ||
| filtered_docs_buffer[len] = doc; | ||
| len += if timestamp_filter.is_within_range(doc) { |
There was a problem hiding this comment.
Should we try to vectorize this too? (not in this PR of course)
There was a problem hiding this comment.
Fetching the values as block, yes definitely.
Filtering via SIMD maybe, the values are u64, so that would be just 4 integers per block on AVX2.
The fastest variant would probably be a pushdown to the codec where we could filter on the compressed data.
Although I'm not sure if the overhead is not too high with a block size of just 64 elements.
I'm also thinking that maybe we should use a branch here. Since the data we have is mostly ordered by timestamp the branch predictor should have a pretty easy job.
There was a problem hiding this comment.
Agreed on all points and doubts.
fulmicoton
reviewed
Mar 18, 2024
fulmicoton
requested changes
Mar 18, 2024
Co-authored-by: Paul Masurel <[email protected]>
fulmicoton
approved these changes
Mar 18, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
collect_block + using
first_valsto batch fetch sort values