Releases: zerebubuth/planet-dump-ng
Add command line options for PBF & XML headers
Thanks to @ZKoshak , there are now options to control the "source" header in the PBF and "license", "copyright", "attribution" and "origin" attributes in the XML header. These default to the previous values, and can now be overridden using the --meta-file
or -M
command line arguments.
Update build for Debian
Explicit linking to pthread is now required for recent versions of Debian.
Try to avoid pblock overflow by tracking estimated size
Planet dump would fail intermittently complaining of block overflow (#25). The root cause of this was failing to encode relation 6677259 in the history PBF.
PBF files are sequences of individual blocks and every block must be smaller than 16MiB. To try and satisfy this, the PBF encoder checks the size of the encoded pblock on a regular basis, every N
elements. The assumption is that the average size of the element multiplied by N
would be smaller than 16MiB. Unfortunately with relation 6677259 that wasn't the case, as versions in its history have more than 25,000 members and were taking up about 100kiB each (so only 164 versions are needed to overflow, relation 6677259 has 440-ish).
This change implements an approximate size counter for relations, which should help to make sure that the pgroup is flushed when it gets too large, even if it hasn't reached N
elements yet. This hasn't seemed necessary yet for ways (they are limited to a fixed number of nodes each) or nodes, but extending to handle those - if necessary - should be straightforward.
Reduce relation size limits to try and stop exception during dump
This issue #25 keeps coming up, and this isn't a solution for it, but a work-around until the cause can be found and fixed.
Fixes for library updates
Fix pg_restore command line.
Seems I tested out the previous release on a machine with a truly ancient version of pg_restore
, so the behaviour of the -f
flag was old behaviour rather than new. Reverted to the previous version.
Better handling of exceptions in writer_thread
If an exception happened in writer_thread
then it was possible for the thread to exit, leaving the other threads waiting forever on a barrier. This change makes the thread with the exception continue the loop so that other threads can finish.
Changed tag ordering
Starting with version 1.2.0, planet-dump-ng
sorts tags by the UTF-8 encoded bytes of the keys. Most keys use only ASCII characters, so this corresponds to an alphabetic ordering for those keys. Other keys will depend on how the codepoint is expressed in UTF-8. Values do not contribute to ordering.
See openstreetmap/osmdbt#29 for more details.
Added concurrency limit for disk writing threads
Added a configurable concurrency limit for disk writing threads. This means that the amount of memory waiting to be written to disk shouldn't grow to the level where it uses up all available memory on the server.
Fixes for Ubuntu bionic
Two fixes:
- The behaviour of the standard C++ library has been fixed so that
std::vector<>::clear()
does not deallocate memory. It seems that previous versions of the library did deallocate, which is not standards conforming, but meant that a bug inplanet_dump_ng
went unnoticed and its memory usage exploded under the new version of the library. - The
pg_restore
tool from PostgreSQL now requires either a database name or file output argument. Previously the default was to output to stdout, but now this is a hard requirement.