You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
this is the recommended way to implement framing; there's just one issue right now:
we do not hard-code the number of chunks, neither in the file header nor in the graph header
to check if there are still chunks in the file, we peek ahead and read four bytes; the idea was that we will either hit EOF and are done, or we will find a varint which then tells us about the size of the chunks to decode
This approach has the following downsides
in case the file contains en empty chunk, the varint will be a single byte. If such a chunk is at the very end of the file, we will hit EOF
we need buffered IO (e.g. a file or BufferedReader) to support peeking without reading; it would be great if we could avoid that
We need to figure out how we can change our approach and iterate some more. This really only came up when implementing the Python lib. Maybe we just encode how many chunks there are in the graph header. Or we use a fixed size prefix instead of a varint.
The text was updated successfully, but these errors were encountered:
At the moment we implement framing by size-prefixing graph chunk messages with a varint.
this is the recommended way to implement framing; there's just one issue right now:
This approach has the following downsides
We need to figure out how we can change our approach and iterate some more. This really only came up when implementing the Python lib. Maybe we just encode how many chunks there are in the graph header. Or we use a fixed size prefix instead of a varint.
The text was updated successfully, but these errors were encountered: