Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(inverted_index): introduce SstIndexCreator #3107

Merged

Conversation

zhongzc
Copy link
Contributor

@zhongzc zhongzc commented Jan 5, 2024

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

Added SstIndexCreator, a crucial structure used by the mito engine for creating indexes, with methods:

  • update: Receives a batch and updates the index based on the batch.
  • finish: Completes index creation and writes the index file.
  • abort: Cancels index creation.

Since external sorting is required during the index creation process, any interruption requires us to clean up the intermediate files generated by the external sorting, which is reflected in do_cleanup.

Additional structures that serve SstIndexCreator were added:

  • TempFileProvider: Implements ExternalTempFileProvider, which is a dependency of external sorting.
  • IntermediateLocation: Responsible for providing paths for intermediate files used in external sorting.
  • Statistics: Responsible for recording statistical information during index creation and flushing it to metrics.
  • IndexValuesCodec: Responsible for decoding primary keys from the batch to values.

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR does not require documentation updates.

Refer to a related PR or issue link (optional)

#2705

Signed-off-by: Zhenchi <[email protected]>
Copy link

codecov bot commented Jan 5, 2024

Codecov Report

Attention: 136 lines in your changes are missing coverage. Please review.

Comparison is base (6e860bc) 85.59% compared to head (8767b0d) 85.04%.
Report is 6 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3107      +/-   ##
==========================================
- Coverage   85.59%   85.04%   -0.55%     
==========================================
  Files         815      818       +3     
  Lines      133334   133862     +528     
==========================================
- Hits       114123   113847     -276     
- Misses      19211    20015     +804     

@zhongzc zhongzc force-pushed the zhongzc/inverted-index-sst-creator branch from 184c3b9 to 9196123 Compare January 9, 2024 07:06
@zhongzc zhongzc force-pushed the zhongzc/inverted-index-sst-creator branch from 9196123 to 171e2f6 Compare January 9, 2024 07:07
src/mito2/src/sst/index/creator.rs Outdated Show resolved Hide resolved
src/mito2/src/sst/index/creator/temp_provider.rs Outdated Show resolved Hide resolved
src/mito2/src/sst/index/creator/statistics.rs Show resolved Hide resolved
Copy link
Contributor

@killme2008 killme2008 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhongzc zhongzc requested a review from waynexia January 9, 2024 09:13
Copy link
Member

@waynexia waynexia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhongzc zhongzc added this pull request to the merge queue Jan 9, 2024
Merged via the queue into GreptimeTeam:main with commit db98484 Jan 9, 2024
15 checks passed
@zhongzc zhongzc deleted the zhongzc/inverted-index-sst-creator branch January 9, 2024 09:34
@zhongzc zhongzc self-assigned this Jan 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs-not-required This change does not impact docs.
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

3 participants