brotli.encode/brotli.decode, does not work with long strings

[jpolstre]

Hi, when the string is long, brotli.encode/decode cuts the string in half, I've tried with several long strings and the result is the same, with short strings it works fine.

code example (I'm running on Windows 11 x64 and es_compression 2.0.13):
use this string as an example (data variable): https://microsoftedge.github.io/Demos/json-dummy-data/256KB.json

import 'package:es_compression/brotli.dart';

 void main() async {

  WidgetsFlutterBinding.ensureInitialized();
  DynamicLibrary.open('assets/brotli/blobs/esbrotli-win64.dll');

   log('data--->${data.length}');//275353

  final encoded = brotli.encode(utf8.encode(data));
  final decoded = brotli.decode(encoded);

  log('data decoded--->${utf8.decode(decoded).length}');//130962

}

As you can see the data is halved in the brotli.encode/decode process.

Doctor summary (to see all details, run flutter doctor -v):
[√] Flutter (Channel stable, 3.24.3, on Microsoft Windows [Versi¢n 10.0.22631.4317], locale es-ES)
[√] Windows Version (Installed version of Windows is version 10 or higher)
[√] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
[√] Visual Studio - develop Windows apps (Visual Studio Community 2022 17.9.6)
[√] Android Studio (version 2024.1)
[√] IntelliJ IDEA Ultimate Edition (version 2024.2)
[√] VS Code (version 1.94.2)
[√] Connected device (2 available)
[√] Network resources
 

[sethloco]

Thank you for the great report. I will make some time this week to look into this and get a fix out.

[sethloco]

Sorry for the delay, I have not had time to fully address this yet.

However, I see that during encoding of 256KB.json:
BrotliCodec(inputBufferLength: 128288)
works but
BrotliCodec(inputBufferLength: 128287)
does not.

I will be investigating this more to understand where the difference is.

[TaranenkoDenis]

I did some testing on this issue with different file sizes. Changing the buffer size does not solve the problem in general, only in the specific case. The next logs were received with BrotliCodec(inputBufferLength: 128288).

File: .json
Original file size: 256575
Compressed file size: 8109
Decompressed file size: 256575
Are decompressed and original files equals? - true

File: .json
Original file size: 513148
Compressed file size: 8110
Decompressed file size: 256576
Are decompressed and original files equals? - false

File: .json
Original file size: 1026294
Compressed file size: 8110
Decompressed file size: 256576
Are decompressed and original files equals? - false

File: .json
Original file size: 5131466
Compressed file size: 8110
Decompressed file size: 256576
Are decompressed and original files equals? - false

After several experiments, I noticed some patterns:

1. When changing the parameter for large files, the compressed file size does not change much (you can see that the compressed file size stays around 8110).
2. The algorithm normally compresses/decompresses files approximately twice as large as inputBufferLength. Problems arise with larger files.