Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add binary diff for mbtiles #1358

Merged
merged 12 commits into from
Jun 25, 2024
Merged

Add binary diff for mbtiles #1358

merged 12 commits into from
Jun 25, 2024

Conversation

nyurik
Copy link
Member

@nyurik nyurik commented Jun 1, 2024

  • mbtiles diff now has an additional --patch-type param with whole, bin-diff-raw and bin-diff-gz values:

    • whole stores different tiles as before - as whole tiles in the tiles table
    • bin-diff-raw computes binary difference between tiles, and stores them as brotli-encoded value in a bsdiffraw table, together with a xxh3_64 hash of the tile as it will be stored after patching
    • bin-diff-gz same as bin-diff-raw, but assumes the tiles are gzip-compressed, so it uncompresses them before comparing. The xxh3_64 stores the hash of the uncompressed tile. The data will be stored in the bsdiffrawgz table (identical structure with above)
  • mbtiles copy --apply-patch will automatically detect if bsdiffrawgz or bsdiffraw tables exist, and will use binary patching.

  • mbtiles apply-patch does not support binary patching yet

  • mbtiles copy --diff-with-file ... --patch-type ... is an alias to mbtiles diff --patch-type ...

@nyurik nyurik requested a review from sharkAndshark June 1, 2024 03:39
@nyurik nyurik force-pushed the bsdiffraw branch 2 times, most recently from c888a39 to 0183ff6 Compare June 14, 2024 02:49
@nyurik nyurik force-pushed the bsdiffraw branch 2 times, most recently from 3c855fe to 118273d Compare June 25, 2024 14:41
@nyurik nyurik marked this pull request as ready for review June 25, 2024 18:23
@nyurik nyurik changed the title Implement binary patch creation for mbtiles Add binary diff for mbtiles Jun 25, 2024
@nyurik nyurik enabled auto-merge (squash) June 25, 2024 19:23
@nyurik nyurik merged commit ccd550a into maplibre:main Jun 25, 2024
19 checks passed
@nyurik nyurik deleted the bsdiffraw branch June 25, 2024 19:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants