-
Notifications
You must be signed in to change notification settings - Fork 380
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(bigquery): Fixes parsing of null type values returned from BQ server #14384
fix(bigquery): Fixes parsing of null type values returned from BQ server #14384
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please consider if absl::optional<std::string>
is a better representation. If you have a good reason to prefer a separate is_null
please let me know and I can review again.
@@ -316,6 +316,7 @@ bool operator==(Struct const& lhs, Struct const& rhs); | |||
|
|||
struct ColumnData { | |||
std::string value; | |||
bool is_null{false}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it would be more idiomatic to use absl::optional<std::string>
to represent this. At the very least it would avoid awkward situations like value == "foo"
and is_null == true
.
@@ -634,11 +634,11 @@ void to_json(nlohmann::json& j, ColumnData const& c) { | |||
j = nlohmann::json{{"v", c.value}}; | |||
} | |||
void from_json(nlohmann::json const& j, ColumnData& c) { | |||
SafeGetTo(c.value, j, "v"); | |||
SafeGetToWithNullable(c.value, c.is_null, j, "v"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If c.value
was absl::optional<std::string>
you could just use overloading on SafeGetTo
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So the main reason for the additional parameter is minimum disruption to the client code. Otherwise modifying std::string
to absl::optional will break the client code. In the past when this has happened we had to do the following for unblocking
-
Modify the client code to look at a non-stable version of google-cloud-cpp (i.e main branch)
-
Make the Client code changes
-
Wait till fix is available in google-cloud-cpp is stable release
-
Revert the client code to a stable-version of google-cloud-cpp.
I am mainly trying to avoid the above.
Reviewable status: 0 of 6 files reviewed, 2 unresolved discussions (waiting on @coryan)
google/cloud/bigquery/v2/minimal/internal/common_v2_resources.h
line 319 at r1 (raw file):
Previously, coryan (Carlos O'Ryan) wrote…
I think it would be more idiomatic to use
absl::optional<std::string>
to represent this. At the very least it would avoid awkward situations likevalue == "foo"
andis_null == true
.
Yeah...I thought about that but modifying the return value breaks our client code and its a lot of back and forth to get everything working as mentioned below
So the main reason for the additional parameter is minimum disruption to the client code. Otherwise modifying std::string
to absl::optional will break the client code. In the past when this has happened we had to do the following for unblocking
-
Modify the client code to look at a non-stable version of google-cloud-cpp (i.e main branch)
-
Make the Client code changes
-
Wait till fix is available in google-cloud-cpp is stable release
-
Revert the client code to a stable-version of google-cloud-cpp.
I am mainly trying to avoid the above.
google/cloud/bigquery/v2/minimal/internal/common_v2_resources.cc
line 637 at r1 (raw file):
Previously, coryan (Carlos O'Ryan) wrote…
If
c.value
wasabsl::optional<std::string>
you could just use overloading onSafeGetTo
?
Please see my comment above for why I am not modifying the return value.
If we go with the |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #14384 +/- ##
=======================================
Coverage 93.06% 93.06%
=======================================
Files 2191 2191
Lines 193219 193284 +65
=======================================
+ Hits 179814 179882 +68
+ Misses 13405 13402 -3 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Couple of options here:
-
I'll be OOO fairly soon so I didn't want the team to be blocked hence the above approach seemed to be the least disruptive. I can merge this code now and then come back and do the cleanup of using
absl::optional
. -
Someone from my team can patch this PR and do the
absl::optional
approach.
I am leaning towards (1) so as not to add additional load on the team and since this is an internal library. But I can understand as code owners if you think otherwise.
Let me know what you think.
Reviewable status: 0 of 6 files reviewed, 2 unresolved discussions (waiting on @coryan)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please file a bug for the cleanup and then we can merge.
Reviewed 6 of 6 files at r1, 1 of 1 files at r2, all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @jsrinnn)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good! Filed a tracking bug(#14387) and added a TODO in the code.
Reviewable status: 5 of 6 files reviewed, all discussions resolved (waiting on @coryan)
723947e
to
7ac1eb2
Compare
This PR includes the following changes for json parsing related to null values, based on what is returned from BQ Server API
Due to the above json
get_to
method crashes with a type exception . The fix checks for null value before callingget_to
in json_utils.hFrom the client perspective, we need to be able to distingush an empty string from a null value so that client api can work properly hence an additional
is_null
member has been added toColumnData
struct.Unit test changes for all of the above.
This change is