Transport of stably serialized messages over gRPC

[cthulhu-rider]

Context

In current implementation we define type per message in NeoFS API in order to:

• do not tie the transport logic to gRPC lib
• implement stable serialization (Protocol Buffers with direct field order)

Given the need to serialize messages for signatures and the like, we can try to skip the extra conversion+serialization step (ToGRPCMessage methods) of the gRPC library utilities and pass the messages directly in binary form (our serialization follows Protocol Buffers).

Proposal

Research direct transmission of message types on the example of some request and pay attention on the optimality criterion:

1. memory overhead on extra conversion
2. network (and any other) overhead without an extra conversion
3. if the rejection of additional conversion will show a decent performance gain, how much will it be necessary to change and maintain the library API

It is worth mentioning that the results directly depend on the selected versions of the libraries: current and gRPC.

For experimentation, I propose to implement a test scenario based on a fake connection (net.Conn), which will allow you to calculate the amount of transmitted traffic (at least from the application side).

type fakeConn struct {
  net.Conn
  traffic int
}

func (x *fakeConn) Write(p []byte) (int, error) {
  n := len(p)
  x.traffic += n
  return n, nil
}

[fyrchik]

This will definitely lower the memory consumption.

The simplest way to gain control over what is transmitted over wire is to implement all methods from https://github.com/protocolbuffers/protobuf-go/blob/master/runtime/protoiface/methods.go#L17 . I am not sure that this methods are always called instead of the default implementation, though. This helps to transmit stably-serialized messages. However there is still a need for ProtoReflect method and there are some Super-tricky comments for the current implementation https://github.com/protocolbuffers/protobuf-go/blob/master/internal/impl/pointer_unsafe.go#L144 ( MessageStateOf is called from ProtoReflect).

As for having some interface for transmitting raw bytes, I am not sure this is possible at all, without reimplementing parts of gRPC from scratch.

Another approach is to implement stable marshalers directly on protobuf-generated structures. I have implemented simple protobuf plugin for generating them https://github.com/fyrchik/neofs-api-go/tree/marshal . Here we lose the benefit (arguable) of not tying the transport logic to gRPC, though I am not sure I understand what it means. However, it is easily extendable, fully-automatic and can help to enforce consistent naming.

The downside is that types are not so rich, but we don't have many places where types inferred from *.proto files are inconvenient (the only case that comes to mind is that we use uint32 for enums). protoc-gen-go implementation is not big https://github.com/protocolbuffers/protobuf-go/tree/fb30439f551a7e79e413e7b4f5f4dfb58e117d73/cmd/protoc-gen-go, so we can even fork it and add necessary code right there (stable marshaling, uint32 for enums, pointer-less slices) without an additional plugin.

[alexvanin]

Another approach is to implement stable marshalers directly on protobuf-generated structures. I have implemented simple protobuf plugin for generating them https://github.com/fyrchik/neofs-api-go/tree/marshal . Here we lose the benefit (arguable) of not tying the transport logic to gRPC, though I am not sure I understand what it means. However, it is easily extendable, fully-automatic and can help to enforce consistent naming.

That is very nice! I propose to test this approach in protobuf messages of control service in NeoFS Node. This service is not a part of the API so it can be a good place to test new structures generated by forked protoc-gen-go without affecting core protocol.