-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: Implicitly plan UNNEST
as lateral
#13695
Conversation
UNNEST
as lateral
# Issue: https://github.com/apache/datafusion/issues/13659 | ||
# TODO: https://github.com/apache/datafusion/issues/10048 | ||
query error DataFusion error: This feature is not implemented: Physical plan does not support logical expression OuterReferenceColumn\(List\(Field \{ name: "item", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: \{\} \}\), Column \{ relation: Some\(Bare \{ table: "u" \}\), name: "column1" \}\) | ||
select * from unnest_table u, unnest(u.column1); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's better to add a logical plan test like
explain select * from unnest_table u, unnest(u.column1);
You can refer to how other LATERAL
tests do it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's good to keep this one and another planning one in the sqllogictest (for end-to-end SQL test).
Just like the following case:
datafusion/datafusion/sqllogictest/test_files/joins.slt
Lines 4051 to 4053 in d3cfc45
# Test CROSS JOIN LATERAL syntax (planning) | |
query TT | |
explain select t1_id, t1_name, i from join_t1 t1 cross join lateral (select * from unnest(generate_series(1, t1_int))) as series(i); |
and
datafusion/datafusion/sqllogictest/test_files/joins.slt
Lines 4068 to 4072 in d3cfc45
# Test CROSS JOIN LATERAL syntax (execution) | |
# TODO: https://github.com/apache/datafusion/issues/10048 | |
query error DataFusion error: This feature is not implemented: Physical plan does not support logical expression OuterReferenceColumn\(UInt32, Column \{ relation: Some\(Bare \{ table: "t1" \}\), name: "t1_int" \}\) | |
select t1_id, t1_name, i from join_t1 t1 cross join lateral (select * from unnest(generate_series(1, t1_int))) as series(i); | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you @rkrishn7
Can you please add some slt tests that actually run this kind of query to show them working and generating correct results?
Thank you 🙏
BTW, here are the instructions for sqllogictests: https://github.com/apache/datafusion/tree/main/datafusion/sqllogictest Ideally you should be able to extend one of the existing test files in https://github.com/apache/datafusion/tree/main/datafusion/sqllogictest/test_files |
4e8f62c
to
5ba8907
Compare
Added a few slt tests! Apologies for the delay! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @rkrishn7 it looks good to me.
Thanks @rkrishn7 and @goldmedal ! |
Which issue does this PR close?
Partial fix for #13659
Rationale for this change
Fixes support for implicitly planning table functions in the
FROM
clause as lateral.What changes are included in this PR?
This PR adds support for implicit lateral planning of
UNNEST
table factors.Are these changes tested?
Yes, sqllogictest added.
Are there any user-facing changes?
Potentially - because
UNNEST
is now implicitly lateral, like Postgres. However, since outer references still fail in the query planning/optimization phase, I'm not sure if documentation updates are required.