Allow non compressed WARC files to be used while recording
#915
Comments
Lisias
added a commit
to Lisias/pywb
that referenced
this issue
Aug 19, 2024
…owadays (and to think this started with Stacker on MS-DOS!), so it makes sense to use **uncompressed** WARC files on a BTRFS or NTFS with it activated. This commit deactivates the WARCIO gzip support when the filename **does not** ends with `.gz`, allowing the user to use these filesystems to reach the compression he wants without having to deal with uncompressing the WARC on use. for webrecorder#915
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Modern file systems now have compression schemes that surpasses userland compression schemes in convenience and sometimes even on efficiency, rendering the current use of
gzipinconvenient when such filesystems are used.As a use case,
BTRFSwithzstd:15gave me excellent results on compression, surpassing what gzip could do, with faster read access. At very least, I don't need to recompress theWARCballafter a recording session.Ideally it should be possible to allow the user to choose if they are want to use the
warciogzip support, or prefer to rely on the file system for such - preventing the user to uncompress the thing manually to take advantage of the file system compression.The text was updated successfully, but these errors were encountered: