HybridParquetScan: Fix velox runtime error in hybrid scan when filter timestamp #12112
Conversation
Signed-off-by: Haoyang Li <[email protected]>
Signed-off-by: Haoyang Li <[email protected]>
There was a problem hiding this comment.
I see that we are missing a lot of tests around predicate push down for various data types that we say we support. Can you please file a follow on issue to test predicate push down for all data types that we support?
byte, short, int, long, float, double, date, decimal_32, decimal_64, string, struct, array, ...
I also see that we support a large number of expressions.
But I don't see hardly any test coverage for these expressions as a part of predicate push down.
I don't fee comfortable saying we support something with no tests for it. Could you file an issue to add in at least some basic tests for each of these expressions too? I am okay if it is the same issue as above.
| @@ -227,3 +227,19 @@ def udf_fallback(s): | |||
| assert_gpu_and_cpu_are_equal_collect( | |||
| lambda spark: spark.read.parquet(data_path).filter("ascii(a) >= 50 and udf_fallback(a) = 'udf_100'"), | |||
| conf=filter_split_conf) | |||
|
|
|||
| @allow_non_gpu(*non_utc_allow) | |||
| def test_hybrid_parquet_filter_pushdown_timestamp(spark_tmp_path): | |||
There was a problem hiding this comment.
Does this need to be marked as a hybrid test?
There was a problem hiding this comment.
nice catch, thanks
Signed-off-by: Haoyang Li <[email protected]>
|
build |
|
Follow-up issue filed #12124 |
|
build |
|
rekick, part of the ci hung due to HW issue |
…n filter timestamp (NVIDIA#12112)" This reverts commit c696b7e.
Fix #12111
Timestamp is not fully supported in Hybrid Filter, so conditions with Timestamp should not pushed to Hybrid Scan in filter splitter.
This pr fixes it.