Options to produce a Descriptors.FileDescriptor out of a proto files with dependencies

[franz1981]

Hi @carlesarnal !

This question/discussion is related Hyperfoil/Hyperfoil#281, which requires to:

• parse a given proto file scanning its services information (including message types)
• allow to specify dependent proto files
• encode/decode dynamic messages based on the descriptions obtained from the proto(s) parsed

Our reference of user experience is analogous to https://ghz.sh/docs/example_config where the import paths are indeed a way to specify the additional proto files required to parse the main one ie "proto" in the mentioned documentation link.

By searching and modifying the apicurio-registry-protobuf-schema-utilities module, I have found 2 (not completed yet) ways to achieve what i need:

1. 2cf62b2 which provide a new method
   Descriptors.FileDescriptor toFileDescriptor(protoFileName, protoFileElement, dependencies)
   which requires specifying all the dependencies upfront (including well known types), although obtained without caring of parsing order (via ProtoParser.Companion.parse), but is not correctly populating the services in the parsed file descriptor (see testVertxGreeting failure). The nice thing is that DynamicSchema::init automatically resolve the provided dependencies.
2. 1216646 which is "fixing" an existing method ie loadSchema(packageName, fileName, schemaDefinition, deps) in order to allow deps maps to be placed in different folders than the main proto file loaded, see testProtoImportDifferentPackagesViaProtoFileToFileDescriptor. This method seems to work fine with services definitions, but requires a full-fat map of dependencies, each in the Descriptors.FileDescriptor form, which means with their dependencies fully resolved too, if any (assuming I would use the same util method of the "main" proto).
   If compared to 1. this approach requires to implement my own dependencies resolution mechanism and ensure all the required Descriptors.FileDescriptor to be parsed upfront in the right order.

My questions are:

• do we have already a method that does what I need, elsewhere?
• if not, which of the 2 approaches is the better/more correct/idiomatic one?

[carlesarnal]

Hey there!

Hi @carlesarnal !

This question/discussion is related Hyperfoil/Hyperfoil#281, which requires to:

• parse a given proto file scanning its services information (including message types)
• allow to specify dependent proto files
• encode/decode dynamic messages based on the descriptions obtained from the proto(s) parsed

Though one.

Our reference of user experience is analogous to https://ghz.sh/docs/example_config where the import paths are indeed a way to specify the additional proto files required to parse the main one ie "proto" in the mentioned documentation link.

By searching and modifying the apicurio-registry-protobuf-schema-utilities module, I have found 2 (not completed yet) ways to achieve what i need:

1. 2cf62b2 which provide a new method
   Descriptors.FileDescriptor toFileDescriptor(protoFileName, protoFileElement, dependencies)
   which requires specifying all the dependencies upfront (including well known types), although obtained without caring of parsing order (via ProtoParser.Companion.parse), but is not correctly populating the services in the parsed file descriptor (see testVertxGreeting failure). The nice thing is that DynamicSchema::init automatically resolve the provided dependencies.
2. 1216646 which is "fixing" an existing method ie loadSchema(packageName, fileName, schemaDefinition, deps) in order to allow deps maps to be placed in different folders than the main proto file loaded, see testProtoImportDifferentPackagesViaProtoFileToFileDescriptor. This method seems to work fine with services definitions, but requires a full-fat map of dependencies, each in the Descriptors.FileDescriptor form, which means with their dependencies fully resolved too, if any (assuming I would use the same util method of the "main" proto).
   If compared to 1. this approach requires to implement my own dependencies resolution mechanism and ensure all the required Descriptors.FileDescriptor to be parsed upfront in the right order.

My questions are:

• do we have already a method that does what I need, elsewhere?

Kind of, we do have a maven-plugin that does the dependency resolution, and the code can probably be extracted from there to the protobuf utilities library, for reference https://github.com/Apicurio/apicurio-registry/blob/main/utils/maven-plugin/src/main/java/io/apicurio/registry/maven/ProtobufDirectoryParser.java#L37.

• if not, which of the 2 approaches is the better/more correct/idiomatic one?

I believe 2 is the way to go here (that's why we have the resolution on the maven-plugin :)). In the end, if you have a proto definition with a dependency and you want to produce messages/requests in the end you're going to need the dependency definition as well.

That said, we recently discovered that we might have a bug resolving proto files with different packages, at least in runtime server (we're always talking about client-side operation from our perspective on this discussion), I'll keep you posted.

[franz1981]

Thanks for the response! I'll than stick with the second approach (and look at the plugin code) and prepare a PR to discuss the API/impl I'm using! many thanks!