-
Notifications
You must be signed in to change notification settings - Fork 338
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(inverted_index): introduce SstIndexCreator #3107
feat(inverted_index): introduce SstIndexCreator #3107
Conversation
Signed-off-by: Zhenchi <[email protected]>
Signed-off-by: Zhenchi <[email protected]>
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #3107 +/- ##
==========================================
- Coverage 85.59% 85.04% -0.55%
==========================================
Files 815 818 +3
Lines 133334 133862 +528
==========================================
- Hits 114123 113847 -276
- Misses 19211 20015 +804 |
184c3b9
to
9196123
Compare
Signed-off-by: Zhenchi <[email protected]>
9196123
to
171e2f6
Compare
Signed-off-by: Zhenchi <[email protected]>
Signed-off-by: Zhenchi <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
I hereby agree to the terms of the GreptimeDB CLA
What's changed and what's your intention?
Added
SstIndexCreator
, a crucial structure used by the mito engine for creating indexes, with methods:update
: Receives a batch and updates the index based on the batch.finish
: Completes index creation and writes the index file.abort
: Cancels index creation.Since external sorting is required during the index creation process, any interruption requires us to clean up the intermediate files generated by the external sorting, which is reflected in
do_cleanup
.Additional structures that serve
SstIndexCreator
were added:TempFileProvider
: ImplementsExternalTempFileProvider
, which is a dependency of external sorting.IntermediateLocation
: Responsible for providing paths for intermediate files used in external sorting.Statistics
: Responsible for recording statistical information during index creation and flushing it to metrics.IndexValuesCodec
: Responsible for decoding primary keys from the batch to values.Checklist
Refer to a related PR or issue link (optional)
#2705