block decompressor read more data than need

[scturtle]

nsz/nsz/BlockDecompressorReader.py

Line 34 in c7df044

self.CurrentBlock = ZstdDecompressor().decompress(self.nspf.read(decompressedBlockSize))

decompressedBlockSize should be self.CompressedBlockSizeList[blockID].

[nicoboss]

Thanks a lot for pointing this out. What an awesome find. This finally explains @gabest11's findings 4.5 years ago in #81. Since this behavior was reported I kept thinking about could possibly cause multiple reads on the same blocks during decompression without verification until you just found what is almost certainly the root cause of this. Fixing this will improve decompression speed by a lot.

[scturtle]

FYI, I have a more concise implementation to be used in IndependentNczDecompressor.py:

pos = sections[0].offset
section_id = 0
s = sections[section_id]

block_size = 1 << block_header.blockSizeExponent
for block_i in range(block_header.numberOfBlocks):
    data = nspf.read(block_header.compressedBlockSizeList[block_i])
    if len(data) < block_size:
        data = ZstdDecompressor().decompress(data)

    end = pos + len(data)
    while pos < end:
        if pos >= s.offset + s.size:
            section_id += 1
            s = sections[section_id]
        part_size = min(end, s.offset + s.size) - pos
        part, data = data[:part_size], data[part_size:]
        if s.cryptoType in (3, 4):
            crypto = AESCTR(s.cryptoKey, s.cryptoCounter, pos)
            part = crypto.encrypt(part)
        f.write(part)
        hash.update(part)
        pos += part_size

I have ported it to TinWoo Installer in this MR.

[nicoboss]

Awesome to see TinWoo getting support for block compression. This is the first time any open-source title installer implements block compression support as far I'm aware. There was a big discussion about it 2 years ago in #120 but you are the first one actually implementing it as far I'm aware.

Really cool how short you managed to make IndependentNczDecompressor. It would be cool if you create a merge request with booth the BlockDecompressorReader fix and your concise IndependentNczDecompressor implementation so you get propper credit for your contribution. I think we should leave the current IndependentNczDecompressor implementation and add your concise one under something like IndependentNczDecompressorConcise.py.