Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Use sozipfile for zip creation #179

Closed
spwoodcock opened this issue Dec 13, 2023 · 1 comment
Closed

[FEATURE] Use sozipfile for zip creation #179

spwoodcock opened this issue Dec 13, 2023 · 1 comment

Comments

@spwoodcock
Copy link
Member

Is your feature request related to a problem? Please describe.

  • Currently files are zipped using Python zipfile module.
  • sozipfile is a new format that is integrated into GDAL.
  • It's use case is to add compression to formats that do not support it: flatgeobuf, geopackage, shapefile, etc.
  • There is a drop in replacement for the Python zipfile module:
import sozipfile.sozipfile as zipfile

Describe the solution you'd like

  • Use sozipfile instead of zipfile for .zip generation.
  • All sozipfile generated zips are fully backward compatible with standard zip readers.
  • sozipfile allows RANGE requests to be used on the file, so only the specific part that is required is extracted.
    • This required the vsicurl driver in GDAL, so for now is only.
    • A user could use QGIS to stream the data extract directly from the bucket.
  • I am uncertain about JS reader capability for sozip, but we could explore that in the future.

Describe alternatives you've considered

  • cotar is the same for .tar format, which I assume GZip compression should work for. But there is no Python or GDAL support.
@kshitijrajsharma
Copy link
Member

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants