create tile-smush / speed up merging

[cldellow]

Today, mapt produces a bunch of thematic pmtiles/mbtiles files, and you're expected to eventually merge them into a single archive.

I had originally envisioned that for prod you might run mapt with a --single flag. This flag exists, but has some issues, like #5 and #6

So in https://github.com/hikeratlas/basemap, I'm currently using tile-join to join the mbtiles into a single archive.

This is a little unsatisfying. I'm using versatiles' Docker image. I think the Dockerfile is here, and it's based on the (abandoned?) https://github.com/mapbox/tippecanoe repo. It's slow! 2 min to join the nova-scotia layers.

If I build mapbox's tile-join directly and don't use Docker, it runs in 30 seconds. Huh, weird.

I also see that https://github.com/felt/tippecanoe exists, and has activity. If I build its tile-join directly and don't use Docker, it runs in 6 min 49 seconds.

I poked around a bit, it seems like both repos launch a new thread for every tile that is to be concatenated. That seems like madness, and might explain why Docker performs poorly - maybe there's syscall overhead.

No clue why felt's repo is so much slower. I do see that there is active development on tile-join there.

But, stepping back: tile-join has to handle quite a few scenarios. The mapt scenario is a little special: none of the tiles will have overlapping layers. In theory, a tool should be able to just slurp in all the tiles, decompress them, concatenate them, compress them, and spit them out again.

For Nova Scotia (a 300MB file), there is no world in which it should take 409 seconds on my 16-core machine (~0.04MB/core/sec). In fact, even 30 seconds (~0.6MB/core/sec) is suspicious.

I could create a tile-smush that is a bastardized fork of tilemaker specifically for this use case.

Or I could wait and see if protomaps/go-pmtiles#105 happens.

Opening this to capture the idea, will likely ignore it for a while so long as HikerAtlas is able to complete it's tile-join-and-convert-to-pmtiles step reliably within a GitHub Action's timeout window.

[cldellow]

On this run https://github.com/hikeratlas/basemap/actions/runs/11188387583/job/31107074184, tile-join took 2h11min for north-america.

The deadline is 6 hours, so the benefit of tile-smush would just be quality-of-life, not essential to using GH Actions.

[cldellow]

This ended up being not too bad: https://github.com/hikeratlas/tile-smush

The tile join step for NA in GitHub Actions decreased from 130 min to 25 min.

An example driver script is available at: https://github.com/hikeratlas/basemap/blob/main/bin/docker-tile-smush