Optimize Index cache to avoid inverted Index file loading on cache miss

[WenyXu]

What type of enhancement is this?

Performance

What does the enhancement do?

In our current system, an index cache miss triggers the loading of the entire index file from s3, regardless of the actual amount of data required. This behavior causes significant inefficiencies in scenarios where only a small portion of the index file is needed for the query.

In some cases, only a small fraction of the index data is accessed. Despite this, the system downloads the entire 200 MiB index file. The download time dominates the query's total execution time, leading to degraded performance.

Implementation challenges

No response

[CookiePieWw]

Possible solution:

Currently the cache in mito can be divided into 2 kinds:

1. Cache for relatively small structs like metadata
   • SstMetaCache, VectorCache, metadata in InvertedIndexCache
2. Cache for some large content
   • PageCache, contents in InvertedIndexCache

For the first, we could keep the current cache management strategy, which means the caller is responsible for fetch values and put cache. We can provide a register method and a pair of getter/setter for extensity:

cache_manager.register_structured(TYPE, ...);
cache_manager.put(TYPE, key, value);
cache_manager.get(TYPE, key);

For the second, we could provide a page-based cache strategy. Here the upper caller simply gives offsets and size, and the cache manager is responsible for featching values and put cache. Similar to #5114

cache_manager.register_paged(TYPE, page_size, reader, ...);
cache_manager.read(TYPE, offset, size, ...);

The page-based cache fetches and caches remote files in fixed-size pages, and returns Vec<u8> as results.

Possible impl:

pub struct StructuredCache<K, V>;

pub struct PagedCache<K>;

pub trait PageReader;

pub trait PageKey {
    fn offset_to_keys(size, offset);
}

pub enum MitoCache;

pub struct CacheManager {
    cache: HashMap<String, MitoCache>;
}

[killme2008]

@WenyXu @CookiePieWw What's the progress of this issue?

[CookiePieWw]

The cache granularity of inverted index has been changed to a fixed size of page in #5114, and several optimizations have been introduced in #5145, #5146, #5147 and #5148.

I think we can close the issue for now? cc @WenyXu