Internal: Unexpected EOF decoding stream

[danchia]

Bug Report

During routine operation, our service received "Internal: Unexpected EOF decoding stream" while using tonic to stream from an external API. We believe this is coming from tonic here in decode.rs

Version

tonic 0.7.1
tonic 0.8.0

Platform

amazon-eks-node-1.23-v20221112 on x86_64.
kernel: 5.4.219-126.411.amzn2

Description

It looks like the most likely reason this happen is the external API server hung up midway through the stream. In which case, this seems like a routine transport error which is usually reported by tonic as unknown: transport error rather than an internal error. An internal error makes it sound like there's a tonic bug.

Could you please check my understanding of the situation? If this is an error than can be routinely encountered, perhaps we should change the ErrorKind to something other than internal?

[LucioFranco]

@danchia can you provide the full error and relevant logs you have?

[LucioFranco]

I wonder if its related to #1183

[danchia]

Unfortunately, I don't have much more details to provide. The full error message I have available to me is "internal: Unexpected EOF decoding stream." and we don't have trace logging turned on in production.

[LucioFranco]

Yeah, without more logs its hard to say exactly what is going on. Did you have tls enabled? What tls backend are you using if so? The error messages from the transport are pretty bad tbh so I am not sure if its a bug or its just some weird interaction with a goaway and the tcp connection closing early.