Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: handle cases where compressed representation is larger than input array #21

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

bjude
Copy link

@bjude bjude commented Aug 16, 2024

The fpzip algorithm can sometimes require a larger buffer than the input array size (pigeonhole principle). The extra 4 bytes in header_bytes was enough to mostly hide this issue, but we ran into a buffer overflow exception in production. Instead of always using the input array size + header size as the compression buffer size, try with this size and incrementally expand the buffer if it's not enough.

In the overwhelmingly common case (compressed size < input size), no extra work is done, only if the buffer is too small do we do any extra work. The values for MAX_ATTEMPTS and BUFFER_GROWTH_FACTOR are somewhat arbitrary, but 5 and 1.5 seem good enough. It would only be truly pathological inputs that would require more than this.

…t array

The fpzip algorithm can sometimes require a larger buffer than the input array size (pigeonhole principle). Instead of always using the input array size + header size as the compression buffer size, try with this size and incrementally expand the buffer if it's not enough.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant