Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Reader max window size and Writer window size #150

Open
wants to merge 4 commits into
base: 1.x
Choose a base branch
from

Conversation

cce
Copy link

@cce cce commented Mar 6, 2025

Adds NewWriterLevelDictWindowSize and NewReaderDictMaxWindowSize to limit memory usage during compression and decompression.

  • NewWriterLevelDictWindowSize added to set specific window size (in bytes, not as log base 2)
  • NewReaderDictMaxWindowSize added to set a maximum window size that this decompressor will allow, also in bytes.
  • Tests exercising the Writer can set window sizes, and that limiting them in the Reader works.

More documentation on window sizes are in RFC 9659 and in zstd.h:

/*! ZSTD_DCtx_setMaxWindowSize() :
 *  Refuses allocating internal buffers for frames requiring a window size larger than provided limit.
 *  This protects a decoder context from reserving too much memory for itself (potential attack scenario).
 *  This parameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode.
 *  By default, a decompression context accepts all window sizes <= (1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT)
 * @return : 0, or an error code (which can be tested using ZSTD_isError()).
 */
    ZSTD_c_windowLog=101,    /* Maximum allowed back-reference distance, expressed as power of 2.
                              * This will set a memory budget for streaming decompression,
                              * with larger values requiring more memory
                              * and typically compressing more.
                              * Must be clamped between ZSTD_WINDOWLOG_MIN and ZSTD_WINDOWLOG_MAX.
                              * Special: value 0 means "use default windowLog".

Resolves #123.

cce added 3 commits March 6, 2025 14:03
…mit memory

usage during compression and decompression. More details available in RFC 9659.
- NewWriterLevelDictWindowSize added to set specific window size (in bytes)
- NewReaderDictMaxWindowSize added to set a maximum window size (in bytes) that
  this decompressor will allow.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Setting decompression WindowSize?
1 participant