You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
I'm currently looking into ways to build in a very constrained system, as a good measure, I want to be able to work with 4GB Ram and 30 GB disk max.
To achieve this goal I'm trying to implement an incremental build, for example for Germany. Its available either as a full dump, 3.9G or as 16 dumps, one for each state (See GeoFabrik page)
Describe the solution you'd like
I would like to see a functionality like tilemaker --merge. This way it should be possible to download each state separately, process and merge on the go, using a script to loop for example.
My expectation is that this would need less resources for lookup data structures etc for a single iteration, I don't mind if this would take considerably longer (5 times longer runtime would be acceptable, since in this scenario processing - CPU - is way cheaper than storage)
Describe alternatives you've considered
I've considered using tilemaker, but the memory management isn't very transparent, it's not even capable to give an estimation or hints how much memory operations might cost, it just fails. And using some sort of swap file don't really work either, it doesn't seem to use it efficiently.
Additional context
There has been significant ground work in #639, this added the --append flag, but also left out the mbtiles format for now.
The main issue here is certainly not, as a code comment suggests, the insertion, but merging of overlapping tiles.
Additionally it might be desirable to implement #536 to be able to merge with preexisting tiles in a directory structure.
And while we are at it, this can also be a possible way to implement #658.
The text was updated successfully, but these errors were encountered:
Are you trying to build Germany? Or Europe? The least I've seen work for the planet in one go is 16gb although you might be able to get away with even less. All of Europe might work with as little as 4gb.
I don't think it would be too hard to support an mbtiles merge mode, just want to see if we can support your use case with a single pass of planetiler and no merging.
My goal is to generate parts of Europe (Germany alone is working) within 6 hours on a very constrained system. this system is just a runtime Frankenstein of a GitHub Runner (Free Tier)...
Is your feature request related to a problem? Please describe.
I'm currently looking into ways to build in a very constrained system, as a good measure, I want to be able to work with 4GB Ram and 30 GB disk max.
To achieve this goal I'm trying to implement an incremental build, for example for Germany. Its available either as a full dump, 3.9G or as 16 dumps, one for each state (See GeoFabrik page)
Describe the solution you'd like
I would like to see a functionality like
tilemaker --merge
. This way it should be possible to download each state separately, process and merge on the go, using a script to loop for example.My expectation is that this would need less resources for lookup data structures etc for a single iteration, I don't mind if this would take considerably longer (5 times longer runtime would be acceptable, since in this scenario processing - CPU - is way cheaper than storage)
Describe alternatives you've considered
I've considered using
tilemaker
, but the memory management isn't very transparent, it's not even capable to give an estimation or hints how much memory operations might cost, it just fails. And using some sort of swap file don't really work either, it doesn't seem to use it efficiently.Additional context
There has been significant ground work in #639, this added the
--append
flag, but also left out thembtiles
format for now.The main issue here is certainly not, as a code comment suggests, the insertion, but merging of overlapping tiles.
Additionally it might be desirable to implement #536 to be able to merge with preexisting tiles in a directory structure.
And while we are at it, this can also be a possible way to implement #658.
The text was updated successfully, but these errors were encountered: