v1.10.0

This release contains some improvements to the protoparse package and new functionality in the protoprint package.

"github.com/jhump/protoreflect/desc/protoparse"

Changes/fixes:

• A couple of cases were identified where the protoc compiler would reject proto source, but the protoparse package would accept it:
  • The way protoc resolved relative names to fully-qualified names, for the request and response types in a method signature, differed slightly from protoparse. The protoparse package might accept a source program, but protoc might reject it due resolving a type name to something other than a type (such as a service or method name). This package now matches the behavior of protoc and will reject the same kinds of source.
  • The protoparse would accept a program with a range like 1to max, for example, whereas protoc requires a space between the 1 and the keyword to. This was due to a small difference in how tokens are lexed. The lexer in this package has been updated to now match the behavior of protoc and will now issue an error if there is no space between a numeric literal and subsequent identifier/keyword.
• The protoc compiler accepts '\v' (vertical tab) and '\f' (form feed) characters as whitespace, but protoparse would not. This is now fixed.
• The protoparse package could panic (with type assertion failures or with index-out-of-range failures) if the input contained a unicode code point in a particular range (well outside the range of allowed characters in a proto source file, other than inside string literals). This has been fixed.

"github.com/jhump/protoreflect/desc/protoprint"

Additions:

• A new CustomSortFunction field has been added to the Printer type. This can be used to define a custom order in which elements are printed to a file. This also includes a new interface type, protoprint.Element, which custom sort functions can use to inspect the elements for deciding how to order them.

v1.9.0

This release contains numerous improvements to the protoparse package, to more closely match protoc in terms of proto source files that are acceptable. It also contains some fixes in other packages.

"github.com/jhump/protoreflect/desc/builder"

Changes/fixes:

• When adding a message to another (to make a nested/enclosed type), the target enclosing message could be incorrectly detached from its parent element. This was the result of a typo in the implementation code and has been fixed.

"github.com/jhump/protoreflect/desc/protoparse"

Additions:

• The protoparse package now issues warnings when it detects that a source file has unused imports. This mirrors the warnings that protoc issues in the same cases. This feature requires the use of a WarningReporter with a parse operation. The concrete type of value provided to the warning reporter will be a protoparse.ErrorUnusedImport.

Changes/fixes:

• The protoc compiler was more strict than protoparse when it comes to resolving relative (vs. full qualified) names. This led to conditions where protoparse would accept a proto source file that protoc would reject. The issue is when the first component of an identifier could match multiple lexical scopes. In such a case protoc only matches the most enclosing scope. But protoparse would fallback to other enclosing scopes if the most enclosing scope could not be used to resolve a symbol. (Hard to describe succinctly, so see the example in this bug report.) This issue is now fixed and protoparse resolves names in the same manner as protoc.
• The protoc compiler uses "C++ enum scoping rules" for protobuf enums. This means that enum values are declared in the namespace of the enclosing enum (as siblings of the enum itself). But protoparse incorrectly treated the enum as the parent scope/namespace. This led to source files that protoparse would accept but that protoc would reject. This issue is now fixed.
• The use of custom options in oneof statements could incorrectly result in error messages about failing to resolve the custom option name, even if the source file and the option reference were valid. This has been fixed.

"github.com/jhump/protoreflect/dynamic/msgregistry"

Additions:

• A new error type, ErrUnexpectedType, was introduced. When a call to FindMessageTypeByUrl or FindEnumTypeByUrl fails because of a type mismatch (expecting a message, got an enum, or vice versa), this can now be determined programmatically by type-asserting the error to the new error type. This provides a proper/robust way to detect this kind of error (previously, callers would have to examine the error text, which is quite brittle).

v1.8.2

This release contains numerous improvements to the protoparse package, to more closely match protoc in terms of proto source files that are acceptable.

"github.com/jhump/protoreflect/desc/protoparse"

Changes/fixes:

• Extensions in a syntax = "proto3" source file were not allowed to have an optional keyword. However, as of the addition of "proto3 optional" support, this is now allowed by protoc. So protoparse now accepts such declarations, to match protoc functionality.
  • Extensions that have an explicit optional keyword are marked in the descriptor with the proto3_optional option. But, unlike normal fields with the proto3_optional option set, they are not (and, in fact, cannot be) included in implicit single-field oneofs.
• The official compiler, protoc, rejects proto source files for the following reasons. However the protoparse would accept such invalid source files. This has been remedied and protoparse also now rejects such programs:
  • An enum cannot contain a value named option or reserved. This is not an explicit check but is instead a limitation of how the protoc parser works: it assumes these keywords indicate options or reserved ranges, not the start of values with these names.
  • A message cannot begin a field declaration with the keyword reserved, for example in a proto3 file where a type (message or enum) named reserved is also defined. Similar to above, the protoc parser will never recognize such a statement as a field, but protoparse would.
  • A oneof cannot contain a field whose name matches a label keyword (optional, repeated, or required). Unlike the above two, this is not related to limits of the parser but is instead an explicit check to prevent common errors: since oneof blocks do not contain labels, a field thusly named is more likely to be a typo even if otherwise syntactically correct.
  • An enum can only allow aliases (via option allow_alias = true;) if it actually contains values that are aliases. Put another way: if there are no aliases, this option must not be set.
  • A message cannot use message-set wire format (via option message_set_wire_format = true;) if it has any normal fields. Message sets must have only extension fields. Similarly, a message cannot use message-set wire format if it has no extension ranges.
  • An extension for a message that uses message-set wire format must be a message type; scalar extensions are not allowed for messages that use message-set wire format.

v1.8.1

This release contains some small bug fixes to the protoparse package.

"github.com/jhump/protoreflect/desc/protoparse"

Changes/fixes:

• Source code info (including position information and comments) could be incorrectly generated for messages that contained a map field and a nested messages and/or group field after that map field. This manifested in incorrect position information and seemingly "lost" comments. This has been fixed.
• Source code info for the last element in a file could inadvertently be missing the element's trailing comment. This manifested in seemingly "lost" comments. This has been fixed.

v1.8.0

This release contains some additions to the protoparse package, including a new protoparse/ast sub-package. It also contains numerous bug fixes.

NOTE: The Go module for this repo now requires v1.4.2 (or higher) of the protobuf runtime packages in "github.com/golang/protobuf". While that is only a minor version bump from previous versions of this module (e.g. last release required at least v1.3.1), it is a non-trivial change. Version 1.4, while intended to be backwards compatible with v1.3, introduced a significant number of changes in order to be interoperable with the new API in "google.golang.org/protobuf".

"github.com/jhump/protoreflect/codec"

Changes/fixes:

• When serializing messages deterministically, if a message contained a map type that had messages as values, these nested value messages could be serialized non-deterministically. This is now fixed.

"github.com/jhump/protoreflect/desc/builder"

Changes/fixes:

• When a file builder included references to multiple other descriptors or builders with the same file name, it would produce an invalid descriptor that tried to import the same file name multiple times. Now, implicit dependencies are permitted to have duplicates as long as their descriptor protos are equal (according to protobuf message equality). Otherwise, when duplicates are detected, a better error message is returned.
• When a message included a mix of fields with explicit tags and fields without tags (i.e. tags that get auto-assigned when the message is built), an "index out of bounds" panic could occur. This has been corrected.

"github.com/jhump/protoreflect/desc/protoparse"

Additions:

• Adds a field named LookupImportProto to protoparse.Parser. This behaves similarly to the existing LookupImport field, except it can return a descriptor proto, instead of a fully-linked descriptor.
• Adds a new method named ParseToAST to protoparse.Parser. This is similar to the ParseFilesButDoNotLink method except that it returns an AST that represents the source file, instead of returning a descriptor proto.

Changes/fixes:

• Fixes a panic when parsing a file that does not contain legal protobuf source. In some cases, the file's AST could be left incomplete when parsing encountered syntax errors, which then caused a nil-dereference panic. This is corrected and a syntax error will be reported instead.
• Fixes a panic when a custom option value includes message literals for a map type but do not contain both key and value fields. Now these fields default to zero values for the respective type instead of causing a nil-dereference panic.
• Parsing a file that contained a BOM (byte order mark) would result in a syntax error. Source files are required to be UTF-8, but even the UTF-8 mark (0xEF 0xBB 0xBF) would be rejected. This has been fixed, and files with a UTF-8 BOM are accepted. Other BOM, for alternate UTF encodings, are not accepted.

"github.com/jhump/protoreflect/desc/protoparse/ast"

This is a brand new package which provides a model of the abstract syntax tree for protobuf source files. The AST includes much more information than descriptor protos, so they can be used for a greater variety of source tooling such as formatters and static analyzers.

The key abstraction is the ast.Node interface, which represents a node in the tree. A valid source file is an *ast.FileNode, which is the root of the tree. Many node types correspond to elements of descriptor protos, but the AST includes more information, including source information for everything in the file, including all comments (regardless of their location) and all punctuation.

"github.com/jhump/protoreflect/desc/protoprint"

Changes/fixes:

• When printing a file that used public or weak imports, the resulting output would be missing the "public" or "weak" keyword. This could cause subsequent parse/compile operations with these outputs to fail unexpectedly. This has been corrected.

"github.com/jhump/protoreflect/dynamic/msgregistry"

Changes/fixes:

• The Resolve method of the *MessageRegistry type would cause a nil-dereference panic if given an unknown message type. It now returns an error instead.

v1.7.1

This release contains a single bug-fix.

"github.com/jhump/protoreflect/dynamic"

• When converting a message with unrecognized fields to a dynamic message, if the original message contained unrecognized extensions, they would be duplicated in the resulting dynamic message.

  If the extensions are known to the dynamic message (via a dynamic.ExtensionRegistry), then non-repeated values can successfully be consolidated; but repeated extension fields will end up with duplicate entries. This only occurred when this module was used with v1.4+ of the "github.com/golang/protobuf" module. This has now been fixed.

  This could impact other usages of the dynamic package, such as "github.com/jhump/protoreflect/desc/protoprint", which converts a descriptor proto to a dynamic message, in order to interpret any unrecognized custom options. This could encounter the same issue described above, resulting in repeated custom options being duplicated in the printed output.

v1.7.0

This release contains some bug fixes and some new API. Most of the new API is to support the new proto3 optional feature introduced with protoc 3.12 (aka "field presence in proto3").

"github.com/jhump/protoreflect/desc"

Additions:

• The desc.FieldDescriptor and desc.OneOfDescriptor types have new methods related to "proto3 optional" fields: FieldDescriptor.IsProto3Optional, FieldDescriptor.HasPresence, and OneOfDescriptor.IsSynthetic.

"github.com/jhump/protoreflect/desc/builder"

Additions:

• The builder.FieldBuilder type has a new bool field Proto3Optional as well as related method SetProto3Optional, for working with and building "proto3 optional" fields.

"github.com/jhump/protoreflect/desc/protoparse"

Additions:

• Adds a new WarningReporter type and field of the same name on the Parser type. When the field is populated (e.g. non-nil), the value will be used to report warnings. Currently, the only warning supported is for files that do not have a syntax declaration. The reporter will receive a warning where the given error value is protoparse.ErrNoSyntax.

Changes/fixes:

• This package now accepts source files that use the optional field label for syntax = "proto3" files. Like protoc, this results in a synthetic single-field oneof that wraps the optional field.
• Fixes a panic that could occur when calling the GetPosition field of an ErrorWithPos value returned by this package. This occurred when there were I/O errors (or other errors returned by the parser's file accessor) when accessing the top-level filenames requested for the parse operation.
• When a custom error reporter is in use, the reporter can return nil to indicate that the operation should proceed. The spec states that even if the reporter returns nil and does not abort processing, the final result of the parse operation would be protoparse.ErrInvalidSource if any errors were reported. But for some errors (specifically: linking errors), errors could be reported but the final disposition would be successful. This has been remedied to correctly return an error if any errors are reported.

"github.com/jhump/protoreflect/desc/protoprint"

Changes/fixes:

• This package can now correctly print files that contain "proto3 optional" fields. Previously, it would explicitly print the synthetic one-of and not include the optional label; that is now fixed.

v1.6.1

This release contains many bug fixes.

"github.com/jhump/protoreflect/desc"

Changes/fixes:

• If a field descriptor is missing a json_name attribute, instead of using the proto field name, the library now falls back to computing a default JSON name in the same manner as protoc. This is necessary since descriptors that are generated into Java code by protoc, for example, do not have the attribute present unless the option was explicitly present in the source.

"github.com/jhump/protoreflect/desc/builder"

Changes/fixes:

• Previously, messages with "message set wire encoding" were subject to the same tag limitations as other messages. However, message set wire encoding should be allowed to have a wider range of valid tags (all the way up to 2^31-2).

"github.com/jhump/protoreflect/desc/protoparse"

Changes/fixes:

• This package now produces descriptors that are deterministic. Previously, custom options that were represented as unrecognized fields were serialized in non-deterministic order.
• When using the new v1.4 protobuf runtime, descriptors produced by this package mistakenly omitted custom options. This is now corrected, and descriptors should now be complete when used with any protobuf runtime versions.
• Previously, messages with "message set wire encoding" were subject to the same tag limitations as other messages. However, message set wire encoding should be allowed to have a wider range of valid tags (all the way up to 2^31-2).
• Fixes numerous small issues in the parser, including some that caused its behavior to diverge from protoc.
  • Options should support extensions of custom option messages, using syntax like so:

    option (custom.option).(some.extension) = "foo";
    

    However, protoparse would fail with a syntax error upon seeing the second open parenthesis. This is now fixed.
  • Related to the "message set wire encoding" fix above, if a message with message set encoding had an extension range that referenced the keyword max, the actual end tag encoded in the descriptor was wrong and did not represent the full allowed range for message set encoding. This is now fixed.
  • Numeric literals that had no floating point were not allowed to exceed the range of an int64. However, it should be possible to use larger values when providing values for uint64, float, and double types. This is now fixed.
  • Groups can be used inside of a oneof (with syntax = "proto2"). And groups can have options. However protoparse rejected definitions of groups with options that were inside of a oneof. This is now fixed.
  • A oneof is valid if it has only groups (with syntax = "proto2"). However, protoparse mistakenly rejected oneofs, stating they had to have at least one field. This is now fixed.
  • Validating reserved statements in messages and enums failed to identify duplicate reserved names in some cases. This is now fixed.

"github.com/jhump/protoreflect/dynamic"

Changes/fixes:

• When using the new v1.4 protobuf runtime, converting between generated and dynamic messages would mistakenly drop unrecognized fields and extensions. This is now corrected, and messages should preserve these fields when converting to and from dynamic messages to generated types.

v1.6.0

This release contains bug fixes and new features/APIs.

"github.com/jhump/protoreflect/codec"

Changes/fixes:

• The Buffer.EncodeFieldValue method will now use always packed encoding for repeated fields that are so configured. Previously, it only used packed encoding when the repeated field actually contained more than one value.
• Encoding messages should now be compatible with message types generated by v2 of the protobuf module.

"github.com/jhump/protoreflect/desc/protoparse"

Additions:

• The Parser struct has a new field named LookupImport. When set, if an imported file cannot be loaded via the parser's Accessor, this field will be invoked. Unlike Accessor, this field supplies an already parsed descriptor instead of protobuf source code to parse. This allows for parsed files' dependencies to provided in already-parsed descriptor form instead of having to re-parse said dependencies.

Changes/fixes:

• The ResolveFilenames function has been modified so it should work as expected/documented on Windows platforms, not just platforms that use forward slash (/) as a path separator.
• When parsing a file that contains an enum definition that has no values or a syntax error with one of the enum values, the parse operation would result in an "index out of range" panic. This has been fixed.
• Support was added for groups that have field options. This is a lesser-known feature of groups (which is a technically deprecated feature of the proto2 syntax). In addition to message options inside the group body (which this package already supported), they can also contain field-style options. For example:

  message Foo {
    optional group Bar = 1 [(field.option) = "baz1"] {
        option (msg.option) = "baz2";
    }
  }
  

  This package now provides the same level of support for this as does protoc. Earlier versions of this package would have reported a syntax error at the opening bracket ([) of the field options.

"github.com/jhump/protoreflect/desc/protoprint"

Changes/fixes:

• Support was added for groups that have field options. See the addition described above for the "protoparse" package for more details. Previously, a descriptor for a group that had both field and message options would result in only the message options being printed. Now the "protoprint" package also correctly prints field options for such descriptors.

"github.com/jhump/protoreflect/dynamic"

Changes/fixes:

• When marshaling a dynamic message to the text format, if the message contained a string field value that contained a double-quote character ("), the resulting text output would be invalid and unparseable. This has been fixed.
• The KnownTypeRegistry should now correctly identify well-known types when using generated types in v2 of the protobuf module.
• When marshaling a dynamic message that contained a map field with a nil value (e.g. the value type is a message and the value stored in the map is a nil pointer), the operation would panic. Now it serializes that map entry with no value (e.g. the synthetic message representing the map entry will be serialized with only a key and omit the value field). This is how the protobuf module's proto package handles this scenario in generated messages with map fields.

v1.5.0

This release contains bug fixes and new features/APIs.

This release also fixes the repo's go.mod file to be compatible with Go 1.13.

"github.com/jhump/protoreflect/codec"

Additions:

• This is a new package. It provides a Buffer type that provides API for interacting with the protobuf binary format. This makes it easy to write programs that can dynamically emit or consume a stream of protobuf-encoded data.

"github.com/jhump/protoreflect/desc"

Additions:

• Added a new CreateFileDescriptorsFromSet function. This is a convenience method around CreateFileDescriptors when the source of descriptors is a *FileDescriptorSet.

"github.com/jhump/protoreflect/desc/builder"

Changes/fixes:

• When custom options were used, BuilderOptions had to be used with a custom extension registry, or else those custom options would not make it into the built descriptor. Now, the custom option can be defined in the file being built or in any of its dependencies, and they will successfully be interpreted and retained. This means that BuilderOptions.Extensions should no longer be needed in most cases.
• Added new methods to FileBuilder: AddDependency and AddImportedDependency, which allow explicitly adding imports to the file. These can be used during building to resolve custom options.
• When using FromFile to convert and existing file descriptor into a builder and then building the result, the output descriptor would strip unused imports from the file. This is no longer the case, so that some imports that are only used to define custom options can be retained and custom options correctly interpreted.

"github.com/jhump/protoreflect/desc/protoparse"

Additions:

• A new ErrorReporter type and eponymous name on the Parser struct have been added. When the field is unset, the behavior matches previous versions. When set, the reporter will be called for each error encountered, and parsing may continue, allowing a single parse invocation to report many errors instead of failing after the first one.
• A new ErrorWithPos interface is provided to represent an error that includes source position. The type of errors returned by the parser were previously unspecified, but were often instances of the concrete exported type *ErrorWithSourcePos. Code that inspected errors and attempted type assertions to this type should instead type assert to this new interface as errors returned by the parser in the future may not be of this concrete type, but may still contain source position information.

Changes/fixes:

• When generating source code info for a descriptor, no entry was created for allow_alias options for enums. This is fixed.
• When generating source code info, an option with a boolean value could be missing its trailing comment. This is fixed.
• If multiple goroutines were invoking protoparse and the files being compiled used the package's copy of standard imports (such as google/protobuf/timestamp.proto et al), when the race detector was enabled, it was possible for it to be tripped. The race should be harmless in practice since the writes that trigger the race are actually no-ops, so it is mainly an issue when running tests with the race detector enabled. This has been fixed: concurrent invocations of protoparse no longer share a copy of these standard imports. Instead, they are cloned for each invocation to use safely.
• When generating source code info, the output is now significantly closer to that of protoc. Here are the most significant variances that were addressed:
  • The order of source code location entries in the resulting descriptor now matches protoc's order.
  • Updated the way source info was generated for group and map fields to address small discrepancies with protoc.
  • This package now correctly handles extension blocks and preserves comments for them, the same way that protoc generates source info for extension blocks.
  • Fixed generation of source locations for "intermediate" paths to match behavior of protoc. For example, protoc emits multiple entries for the path option (without an index), one for each actual option declared in the file. This package now does the same.
  • Updated computation of "column" in source locations to account for tab stabs. This mirrors protoc's logic, which assumes that tab stops are 8 characters apart.
• This package now accepts an empty file, just as protoc does. Instead of returning an error, it will return an empty descriptor.
• The UninterpretedOptions fields in options messages of descriptors produced by this package will now be nil when all options are interpreted. Previously, the field was set to an empty slice, which caused confusing/noisy output if the descriptor were then marshaled to JSON via the golang.org/protobuf/jsonpb package.

"github.com/jhump/protoreflect/desc/protoprint"

Changes/fixes:

• The default ordering of a descriptor, when no source info is available, has changed slightly. Previously, options would appear before package and import statements. Now options appear after package and import statements. If source info is available, elements will still retain their ordering per the source info.
• It was previously possible, when not explicitly sorting elements and when a file descriptor had source info present, for elements to be printed in non-deterministic order. In this mode, the printer uses relative order based on source info to sort elements. But, if source info had ambiguous entries (e.g. more than one entry that matched a printed element), the entry in source info that was used relied on map iteration order. This has been fixed and output is deterministic, even in cases such as this.
• Previously, trying to print a descriptor that had an extension whose type was a group (vs. a message) would result in a type conversion panic. This has been fixed.
• Previously, if a descriptor included source code info and it indicated that a set of extensions for the same extendee were defined in multiple extension blocks, this would not be preserved. Instead, all extensions for the same extendee were grouped into a single extension block when the descriptor was printed. Also, comments (leading and trailing) for the extension block were not included in the printed output. This has been corrected. Printing such a descriptor will group extensions correctly into blocks as well as include any comments.

"github.com/jhump/protoreflect/dynamic"

Changes/fixes:

• When unmarshaling a dynamic message from a slice of bytes, in the protobuf binary format, if the data contained an illegal size prefix for bytes, string, and nested message values, the operation could result in an "index out of bounds" panic. This has been fixed and an error will be correctly reported instead.