Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stream ended prematurely when trying to readAllBytes #210

Open
yash29896 opened this issue May 10, 2023 · 0 comments
Open

Stream ended prematurely when trying to readAllBytes #210

yash29896 opened this issue May 10, 2023 · 0 comments

Comments

@yash29896
Copy link

yash29896 commented May 10, 2023

So I'm trying to switch our compressor to lz4-java from snappy and I was trying something like this for a POC

When I try a readAllBytes() on the LZ4FrameInputStream it throw the below exception an breaks.

ERROR

Exception in thread "main" java.io.IOException: Stream ended prematurely
	at net.jpountz.lz4.LZ4FrameInputStream.readInt(LZ4FrameInputStream.java:245)
	at net.jpountz.lz4.LZ4FrameInputStream.readBlock(LZ4FrameInputStream.java:259)
	at net.jpountz.lz4.LZ4FrameInputStream.read(LZ4FrameInputStream.java:353)
	at java.base/java.io.InputStream.readNBytes(InputStream.java:396)
	at java.base/java.io.InputStream.readAllBytes(InputStream.java:333)
	at com.yash.lz4.App.main(App.java:22)

Calling Function

    public static void main( String[] args ) throws IOException
    {
    	ByteHandler handler = new LZ4Compressor();
        byte[] b = {4, 34, 77, 24, 96, 112, 115, 6, 0, 0, -128, 3, 0, 1, 0, 0, 0};  // Test data from one of our unit tests
        InputStream inputStream = handler.deserialize(new ByteArrayInputStream(b));
        byte[] i = inputStream.readAllBytes();
    }

I've done some debugging and it seems like its not marking the frame as finished.

private void readBlock() throws IOException {
    // Reads Ok up to this point
    // Reads the last 6 bytes 3, 0, 1, 0, 0, 0  
    int blockSize = readInt(in);  
    final boolean compressed = (blockSize & LZ4FrameOutputStream.LZ4_FRAME_INCOMPRESSIBLE_MASK) == 0;
    blockSize &= ~LZ4FrameOutputStream.LZ4_FRAME_INCOMPRESSIBLE_MASK;  // blocksize = 6

    // Check for EndMark
    if (blockSize == 0) {           // Since this is 6 its doesnt enter the loop
......
      frameInfo.finish();
      return;
    }

Hence the next time it tries to read the stream which has already been completed it leaks into readBlock(); and breaks.

 public int read() throws IOException {
    while (!firstFrameHeaderRead || buffer.remaining() == 0) {
      if (!firstFrameHeaderRead || frameInfo.isFinished()) {
        if (firstFrameHeaderRead && readSingleFrame) {
          return -1;
        }
	if (!nextFrameInfo()) {
	  return -1;
	}
      }
      readBlock();
    }

This is our Compressor implementation

public final class LZ4Compressor implements ByteHandler {

	  
	  public OutputStream serialize(final OutputStream toSerialize) {
	    try {
	      return new LZ4FrameOutputStream(toSerialize);
	    } catch (IOException e) {
	      throw new UncheckedIOException(e);
	    }
	  }

	  public InputStream deserialize(final InputStream toDeserialize) {
	    try {
	      return new LZ4FrameInputStream(toDeserialize);
	    } catch (IOException e) {
	      throw new UncheckedIOException(e);
	    }
	  }

	  public ByteHandler getInstance() {
	    return new LZ4Compressor();
	  }
	}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant