You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The last cache TableProvider has code to handle predicate pushdown here, however, there are no tests to verify that it is actually working as intended.
So, in light of #25550 and #25550, we should have tests in place to verify that the predicate pushdown works.
Proposed solution
#25566 used a custom ExecutionPlan to verify the predicate pushdown is working, so we may consider following that example, since the code in the metadata cache is very similar to the last cache.
The idea is to capture the predicates evaluated so they can be displayed in the EXPLAIN output of a query that contains predicates destined for the cache.
This has the added benefit of validating that the pushdown is working, but also gives us the means to diagnose the same issue should we suspect pushdown is not working in the wild.
Alternatives
We could write lower level tests that validate the conversion function directly, but this would not have the added benefit mentioned above of also providing external visibility via EXPLAIN.
Additional context
Here is the custom implementor of the ExecutionPlan trait used by the metadata cache:
/// For queries that do have predicates that get pushed down, the output will include them, e.g.:
///
/// ```text
/// MetaCacheExec: predicates=[[0 IN (us-east)], [1 IN (a,b)]] inner=MemoryExec: partitions=1, partition_sizes=[1]
/// ```
#[derive(Debug)]
structMetaCacheExec{
inner:MemoryExec,
predicates:Option<IndexMap<ColumnId,Predicate>>,
}
Should the pushdown not be working, we may want to adopt a similar strategy to the metadata cache to converting filter Exprs using DataFusion's LiteralGuarantee
The text was updated successfully, but these errors were encountered:
Problem
The last cache
TableProvider
has code to handle predicate pushdown here, however, there are no tests to verify that it is actually working as intended.So, in light of #25550 and #25550, we should have tests in place to verify that the predicate pushdown works.
Proposed solution
#25566 used a custom
ExecutionPlan
to verify the predicate pushdown is working, so we may consider following that example, since the code in the metadata cache is very similar to the last cache.The idea is to capture the predicates evaluated so they can be displayed in the
EXPLAIN
output of a query that contains predicates destined for the cache.This has the added benefit of validating that the pushdown is working, but also gives us the means to diagnose the same issue should we suspect pushdown is not working in the wild.
Alternatives
We could write lower level tests that validate the conversion function directly, but this would not have the added benefit mentioned above of also providing external visibility via
EXPLAIN
.Additional context
Here is the custom implementor of the
ExecutionPlan
trait used by the metadata cache:influxdb/influxdb3_cache/src/meta_cache/table_function.rs
Lines 251 to 276 in 20c74a7
Should the pushdown not be working, we may want to adopt a similar strategy to the metadata cache to converting filter
Expr
s using DataFusion'sLiteralGuarantee
The text was updated successfully, but these errors were encountered: