feat: metadata cache provider and datafusion trait impls #25566
+1,762
−596
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hiltontj
force-pushed
the
hiltontj/meta-cache-write-path
branch
4 times, most recently
from
5d159c9
to
583f69e
Compare
hiltontj
commented
Nov 21, 2024
Comment on lines
+260
to
+294
| /// Compare the configuration of a given cache, producing a helpful error message if they differ | ||
| pub(crate) fn compare_config(&self, other: &Self) -> Result<(), anyhow::Error> { | ||
| if self.max_cardinality != other.max_cardinality { | ||
| bail!( | ||
| "incompatible `max_cardinality`, expected {}, got {}", | ||
| self.max_cardinality, | ||
| other.max_cardinality | ||
| ); | ||
| } | ||
| if self.max_age != other.max_age { | ||
| bail!( | ||
| "incompatible `max_age`, expected {}, got {}", | ||
| self.max_age.as_secs(), | ||
| other.max_age.as_secs() | ||
| ) | ||
| } | ||
| if self.column_ids != other.column_ids { | ||
| bail!( | ||
| "incompatible column id selection, expected {}, got {}", | ||
| self.column_ids | ||
| .iter() | ||
| .map(|id| id.to_string()) | ||
| .collect::<Vec<_>>() | ||
| .join(","), | ||
| other | ||
| .column_ids | ||
| .iter() | ||
| .map(|id| id.to_string()) | ||
| .collect::<Vec<_>>() | ||
| .join(","), | ||
| ) | ||
| } | ||
|
|
||
| Ok(()) | ||
| } |
There was a problem hiding this comment.
This is newly added to the core cache code, as it is used on cache creation in the MetaCacheProvider to give friendly errors if users attempt to overwrite an existing cache.
hiltontj
commented
Nov 21, 2024
Comment on lines
+468
to
+478
| /// A predicate that can be applied when gathering [`RecordBatch`]es from a [`MetaCache`] | ||
| /// | ||
| /// This is intended to be derived from a set of filter expressions in Datafusion by analyzing | ||
| /// them with a `LiteralGuarantee`. | ||
| /// | ||
| /// This uses a `BTreeSet` to store the values so that they are iterated over in sorted order. | ||
| #[derive(Debug, Clone, PartialEq, Eq)] | ||
| pub(crate) enum Predicate { | ||
| In(BTreeSet<Value>), | ||
| NotIn(BTreeSet<Value>), | ||
| } |
There was a problem hiding this comment.
This type was changed from the previous enum that also contained Eq and NotEq variants. Those were no longer needed with how predicates are pushed down from DataFusion (see the table_function.rs module)
hiltontj
force-pushed
the
hiltontj/meta-cache-write-path
branch
2 times, most recently
from
44d0ec2
to
9dc7cc8
Compare
hiltontj
requested review from
pauldix,
mgattozzi and
praveen-influx
and removed request for
pauldix and
mgattozzi
This adds the MetaDataCacheProvider for managing metadata caches in the influxdb3 instance. This includes APIs to create caches through the WAL as well as from a catalog on initialization, to write data into the managed caches, and to query data out of them. The query side is fairly involved, relying on Datafusion's TableFunctionImpl and TableProvider traits to make querying the cache using a user-defined table function (UDTF) possible. The predicate code was modified to only support two kinds of predicates: IN and NOT IN, which simplifies the code, and maps nicely with the DataFusion LiteralGuarantee which we leverage to derive the predicates from the incoming queries. A custom ExecutionPlan implementation was added specifically for the metadata cache that can report the predicates that are pushed down to the cache during query planning/execution. A big set of tests was added to to check that queries are working, and that predicates are being pushed down properly.
hiltontj
force-pushed
the
hiltontj/meta-cache-write-path
branch
from
9dc7cc8
to
088df39
Compare
hiltontj
commented
Nov 22, 2024
| "| us-west | d |", | ||
| "+---------+------+", | ||
| ], | ||
| }, |
There was a problem hiding this comment.
All of these test cases were removed because the Eq and NotEq variants were removed from the Predicate type.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #25543
Closes #25544
This adds the
MetaCacheProviderfor managing metadata caches in theinfluxdb3instance. This includes APIs to create caches through the WAL as well as from a catalog on initialization, to write data into the managed caches, and to query data out of them.The query side is fairly involved, relying on Datafusion's
TableFunctionImplandTableProvidertraits to make querying the cache using a user-defined table function (UDTF) possible.The predicate code was modified to only support two kinds of predicates: IN and NOT IN, which simplifies the code, and maps nicely with the DataFusion
LiteralGuaranteewhich we leverage to derive the predicates from the incoming queries.MetaCacheExec, a customExecutionPlanimplementation was added specifically for the metadata cache that can report the predicates that are pushed down to the cache during query planning/execution.A big set of tests was added to to check that queries are working, and that predicates are being pushed down properly.
Additional Notes
meta_cachemodule of theinfluxdb3_cachecrate around so that the primarymod.rsjust contains tests, and then has the following additional modules:cache.rs: (existing with some changes) core cache implementation - I left a couple comments on where the main changes were made to that codeprovider.rs: (new) contains code for the newMetaCacheProvidertable_function.rs: (new) contains code for DataFusion trait implementations