feat: pre-download the ingested sst #4636
Conversation
…local cache to accelerate first query
MichaelScofield
requested review from
evenyag,
v0y4g3r and
waynexia
as code owners
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4636 +/- ##
==========================================
+ Coverage 84.89% 84.96% +0.06%
==========================================
Files 1101 1103 +2
Lines 197289 198042 +753
==========================================
+ Hits 167495 168264 +769
+ Misses 29794 29778 -16 |
| let remote_metadata = remote_store | ||
| .stat(remote_path) | ||
| .await | ||
| .context(error::OpenDalSnafu)?; | ||
| let reader = remote_store | ||
| .reader_with(remote_path) | ||
| .concurrent(DOWNLOAD_READER_CONCURRENCY) | ||
| .chunk(DOWNLOAD_READER_CHUNK_SIZE.as_bytes() as usize) | ||
| .await | ||
| .context(error::OpenDalSnafu)? | ||
| .into_futures_async_read(0..remote_metadata.content_length()) |
There was a problem hiding this comment.
We already stored the file size in the FileMeta so we don't need to call stat here. We can pass the file size (or the IndexValue) to the download method to avoid one stat call.
* refactor: pre-read the ingested sst file in object store to fill the local cache to accelerate first query * feat: pre-download the ingested SST from remote to accelerate following reads * resolve PR comments * resolve PR comments
I hereby agree to the terms of the GreptimeDB CLA.
Refer to a related PR or issue link (optional)
What's changed and what's your intention?
Pre-download the ingested SST files from remote object store to fill the local file cache to accelerate following reads.
Checklist