Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for BigQuery ANY TYPE data type #1602

Merged
merged 4 commits into from
Dec 15, 2024

Conversation

MartinSahlen
Copy link
Contributor

@MartinSahlen MartinSahlen commented Dec 13, 2024

As referenced in #1601 , here is a PR, with a test, that verifies correct parsing of BigQuery User-defined functions using templated SQL parameters (what a mouthful :D).

More docs on the actual feature/functionality in BQ can be found here: https://cloud.google.com/bigquery/docs/user-defined-functions

Thanks in advance for the consideration!

EDIT: This closes #1601

Copy link
Contributor

@iffyio iffyio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @MartinSahlen! The changes look reasonable to me, left a couple minor comments

@@ -8382,6 +8382,10 @@ impl<'a> Parser<'a> {
Ok(DataType::Tuple(field_defs))
}
Keyword::TRIGGER => Ok(DataType::Trigger),
Keyword::ANY => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Keyword::ANY => {
Keyword::ANY if self.peek_keyword(Keyword::TYPE) => {

Thinking we can add this guard to ensure that it doesn't conflict with a custom data type that happens to be called ANY?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes that is much more in line with the desired / intended behavior. As the current approach would result in a parsing error should we come across an ANY type I think.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried implementing this change but it seems to break. hmm

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As this is the first time I'm touching the codebase, I needed to understand the general mechanisms. Your change worked in conjunction with let _ = self.parse_keyword(Keyword::TYPE);, which makes intitutive sense after looking at the code in general.

Comment on lines 2100 to 2107
// Templated
concat!(
"CREATE OR REPLACE TEMPORARY FUNCTION ",
"my_function(param1 ANY TYPE) ",
"AS (",
"(SELECT 1)",
")",
),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the functionality itself is the data type, I'm thinking we can use a dedicated test for it? e.g fn test_parse_any_type()? Also can we add a test case to demonstrate the behavior in the other comment what happens in a sql like CREATE TABLE foo (x ANY);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, I was in fact doing that but reverted to inlining the test with other UDFs. Not being sure what was the preferred pattern but makes sense!

@iffyio iffyio changed the title Add bq templated udfs Add support for BigQuery ANY TYPE data type Dec 13, 2024
@MartinSahlen
Copy link
Contributor Author

Thanks @MartinSahlen! The changes look reasonable to me, left a couple minor comments

Thanks for the review and suggestions! I have now added the suggested changes and improvements, along with two seperate tests

  • test that the usage of "ANY TYPE" works in a function definition for BigQuery
  • test that the usage of the "ANY" type still works as a standalone type to support arbitrary or user-defined types without being restrictive. This also makes a lot of sense as the goal of this codebase is not to provide an opinionated or semantic validating parser.

Copy link
Contributor

@iffyio iffyio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks @MartinSahlen!
cc @alamb

@iffyio iffyio merged commit 7bc6ddb into apache:main Dec 15, 2024
8 checks passed
@MartinSahlen
Copy link
Contributor Author

LGTM! Thanks @MartinSahlen! cc @alamb

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fails to parse BigQuery UDF with "ANY TYPE" parameters
2 participants