Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement the core MetaCache and MetaCacheProvider #25543

Open
Tracked by #25539
hiltontj opened this issue Nov 13, 2024 · 0 comments · May be fixed by #25566
Open
Tracked by #25539

Implement the core MetaCache and MetaCacheProvider #25543

hiltontj opened this issue Nov 13, 2024 · 0 comments · May be fixed by #25566
Assignees
Labels

Comments

@hiltontj
Copy link
Contributor

hiltontj commented Nov 13, 2024

See the proposal for the metadata cache here.

An InfluxDB instance can have many metadata caches, i.e., the MetaCache type, configured across the various databases/tables. These will be managed by a MetaCacheProvider which will provide access to the underlying caches in various parts of the system.

MetaCache

The metadata cache is a hierarchical structure with a level in the hierarchy for each column in the cache. For this, we can use a BTree, since having the columns in order is a key aspect of the metadata cache.

Individual nodes in the tree hold a value and a last seen time that is used to age entries out of the cache on eviction. For multi-level hierarchies, branch nodes also hold a nested BTree for related values in the next column. For example:

Screenshot 2024-11-12 at 14 03 38

Here, the MetaCache was defined with the columns region, followed by host. The first level down from root holds values for the region column and the second for the host. If a user was to query for metadata values of host where region is equal to "us-east", then they would get the values "a" and "b".

Each cache has a maximum cardinality, and therefore needs to track the number of values it is holding. Pruning values when it exceeds this can be decoupled from insertion. There should be an API to prune a cache by removing its oldest entries.

Caches can also have a maximum age for entries, beyond which they are evicted regardless of the current cardinality of the cache. This max age is optional though.

MetaCacheProvider

There needs to be a type that holds all the metadata caches for the instance. This will provide an API for cache creation, deletion, insertion of values, and pruning.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant