Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid encoding 0 values for proto3 fields #269

Open
mmsqe opened this issue Jul 31, 2019 · 8 comments
Open

Avoid encoding 0 values for proto3 fields #269

mmsqe opened this issue Jul 31, 2019 · 8 comments
Labels

Comments

@mmsqe
Copy link

mmsqe commented Jul 31, 2019

current behaviour:
when a enum field is set with zero-value, the value is encoded in the buffer

expected behaviour:
when a enum field is set with zero-value, the value is omitted in encoding

mmsqe pushed a commit to diginex/libra-core that referenced this issue Jul 31, 2019
mmsqe pushed a commit to diginex/libra-core that referenced this issue Aug 1, 2019
mmsqe pushed a commit to diginex/libra-core that referenced this issue Aug 2, 2019
@sigurdm
Copy link
Collaborator

sigurdm commented Sep 6, 2019

Is this relating to the binary encoding?
What does it mean to set it to zero-value? Are you using the 'clearX' method
Could you provide a small reproduction sample?

@mmsqe
Copy link
Author

mmsqe commented Sep 6, 2019

@sigmundch Not sure if it's protobuf standard since I need to add it mannualy after generating transaction.proto like below:

Screenshot 2019-09-06 at 6 36 25 PM

While Libra use rust-protobuf 2.7.0 will skip 0, no byte at all

if self.sequence_number != 0 {
    my_size += ::protobuf::rt::value_size(2, self.sequence_number, ::protobuf::wire_format::WireTypeVarint);
}

@sigurdm
Copy link
Collaborator

sigurdm commented Sep 6, 2019

Ah - I think I get it now.
I don't think this is about enum, but integer fields.
They should not be represented if they have the non-default value.

We could do that either when encoding or in the setter...
First we need to store in the metadata for each field has proto3 semantics.

Is this issue blocking you, or is it a nice-to-have?

@sigurdm sigurdm changed the title enum zero value for encoding in proto3 Avoid encoding 0 values for proto2 fields Sep 6, 2019
@mmsqe
Copy link
Author

mmsqe commented Sep 6, 2019

@sigmundch thanks for the explanation, no rush here

@sigurdm sigurdm changed the title Avoid encoding 0 values for proto2 fields Avoid encoding 0 values for proto3 fields Sep 9, 2019
@calmh
Copy link

calmh commented Dec 16, 2020

This is a somewhat blocking issue for me, for the following reason... Consider a trivial message with just an integer (or enum) field.

message Foo {
   int field = 1;
}

We can instantiate this with a zero field either by Foo() or Foo()..field = 0. These two don't compare equally using ==, which is also unexpected. But more importantly, an incoming message over the wire won't have the zero value present, meaning if we construct a Foo()..field = 0 and compare it to the corresponding message deserialized from the wire it won't be equal. If we do the same with any other integer value the are equal...

I would expect the default value handling to mean that messages are considered equal regardless of whether an int value is unset or set to zero.

Perhaps this is somewhat different than the issue subject here, but fixing this issue (not serializing zero fields) would highlight the problem further by making an object with a zero field not survive an encoding roundtrip and still compare equally. Currently it does, but only because zero fields are sent as explicit zeroes.

@osa1
Copy link
Member

osa1 commented May 11, 2022

This is a somewhat blocking issue for me, for the following reason... Consider a trivial message with just an integer (or enum) field.

message Foo {
int field = 1;
}

We can instantiate this with a zero field either by Foo() or Foo()..field = 0. These two don't compare equally using ==, which is also unexpected. But more importantly, an incoming message over the wire won't have the zero value present, meaning if we construct a Foo()..field = 0 and compare it to the corresponding message deserialized from the wire it won't be equal. If we do the same with any other integer value the are equal...

I would expect the default value handling to mean that messages are considered equal regardless of whether an int value is unset or set to zero.

Perhaps this is somewhat different than the issue subject here, but fixing this issue (not serializing zero fields) would highlight the problem further by making an object with a zero field not survive an encoding roundtrip and still compare equally. Currently it does, but only because zero fields are sent as explicit zeroes.

This is indeed a different issue. The issue in your case is this part:

These two don't compare equally using ==, which is also unexpected

I thought we had an issue for this already but I can't find it now. We should open an issue for this if we don't have one already.

As far as I can see proto specs do not say that default values for optional fields must be omitted. Only proto3 JSON spec says that (see #592, #585). So labeling this is "perf" instead of "bug".

@osa1 osa1 added the perf Related to runtime performance label May 11, 2022
@calmh
Copy link

calmh commented May 11, 2022

The proto3 language guide says about default values:

Also note that if a scalar message field is set to its default, the value will not be serialized on the wire.

It doesn't use the word "must" in capital letters or so, but I think it's pretty well established that this is how proto3 should work...

@osa1
Copy link
Member

osa1 commented May 11, 2022

Thanks for pointing that out, I had missed that. Marking this issue as bug.

@osa1 osa1 added bug and removed perf Related to runtime performance labels May 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants