feat!: add Edition 2023 Support

[mkruskal-google]

Protobuf.js was audited for the following features:

• Field presence - Full support of all 3 presence modes, although the behavior appears to be non-conformant in general. The feature values were hooked up to the existing behaviors, with new tests added for edition 2023.
• Packed encoding - Both packed and expanded encoding are supported and conformant. Some opportunistic cleanup was done, but was behavior-preserving.
• Delimited encoding - Proto2 groups are supported but mark a property on the message rather than the field. This information was moved to the field and the more general delimited logic was added (with tests).
• Enum type - All enums are open, this was left untouched. CLOSED enum_type will be ignored in edition 2023 (which is consistent with our long-term goal anyway).
• UTF8 validation - There doesn't seem to be any explicit validation of utf8, beyond what javascript may provide by default. This would treat proto2/proto3 and edition 2023 the same regardless of features.

The proto target CLI appeared to be broken in a number of edge cases related to editions features, so these were fixed and tested were added. It now supports proto2 and proto3 roundtrips, and conversions to proto2/proto3 (even from edition 2023) as long as they're valid. No extra validation layer was added, so impossible conversions will just produce invalid protos. A new target for edition 2023 wasn't added, but could be pretty easily if necessary in the future.

One major design flaw in protobuf.js, that's common in proto3-based code, is that the concept of files was largely dropped. The in-memory and JSON representation of descriptors puts all of the file metadata into the most-nested package namespace object, but this object may be shared by multiple files. This results in possible clobbering and merging, leading to malformed file metadata. In proto2/proto3, this largely doesn't matter outside of custom file options, but the effects can be seen in the hack that forces packed = false for proto2 descriptors to get the repeated field encoding correct. This problem was made worse in editions, where we store important metadata at the file level (edition + features). The solution implemented here collapses all of this information into the top-level objects instead of putting it on the namespace. This results in some minimal duplication, but has the ability to properly support proto2, proto3, and editions in addition to fixing some pre-existing bugs (demonstrated by some newly added tests).

Other changes:

• Group decoding will now look for the proper matching end-group tag instead of any end-group tag
• A bug in feature resolution for extensions was fixed
• A bug in how unresolved features showed up in the options struct was fixed
• Standard reflection helpers were added that are compatible with editions and proto2/proto3
• Proto2 and proto3 are now treated as special editions (with full feature resolution). This substantially increases test coverage of all of the new editions logic
• The JSON spec now fully supports proto2 descriptors, instead of the previous hack of forcing packed=false to make them behave properly

Migration Guide

• Code must always ensure .resolveAll() is called on dynamic built protos. This is needed when building from constructors directly or individual fromJSON/addJSON methods. Top-level builders like Root.fromJSON, Root.load, and protobuf.parse do not need to explicitly call this

[sofisl]

Mostly questions to determine whether something is breaking but seems mostly not.

[sofisl]

this seems like a breaking change - seems that syntax no longer determines nullability, just whether the field has presence & not required?

[mkruskal-google]

It's really just a refactor and simplification. All of the syntax logic has been encapsulated into feature helpers like hasPresence and required (see https://protobuf.dev/editions/implementation/#syntax_reflection), and feature resolution itself.

This should be logically equivalent for proto2/proto3, but is now compatible with editions also.

[sofisl]

this isn't equivalent to previous, seems like we're assigning a boolean vs. an object.

[mkruskal-google]

How so? this is just using similar short-circuit logic with && instead of ||. Basically, if there are features for the value already, merge them, otherwise assign null (i.e. copy parentFeaturesCopy)

[sofisl]

trust you, but we needed this to have feature-level enums

[mkruskal-google]

Yea I collapsed all protoFeatures directly into options in a cleanup commit to reduce the footprint of the PR. So the unresolved features get stored like regular options now, and resolved features get tracked separately

[sofisl]

@dcodeIO can you take a look at this PR? Debating whether this should be a breaking change or not - would love your input, thanks!