Skip to content

Releases: zerebubuth/planet-dump-ng

Add command line options for PBF & XML headers

28 Oct 19:20
Compare
Choose a tag to compare

Thanks to @ZKoshak , there are now options to control the "source" header in the PBF and "license", "copyright", "attribution" and "origin" attributes in the XML header. These default to the previous values, and can now be overridden using the --meta-file or -M command line arguments.

Update build for Debian

24 Mar 23:55
ed5e41d
Compare
Choose a tag to compare

Explicit linking to pthread is now required for recent versions of Debian.

Try to avoid pblock overflow by tracking estimated size

05 Oct 16:35
Compare
Choose a tag to compare

Planet dump would fail intermittently complaining of block overflow (#25). The root cause of this was failing to encode relation 6677259 in the history PBF.

PBF files are sequences of individual blocks and every block must be smaller than 16MiB. To try and satisfy this, the PBF encoder checks the size of the encoded pblock on a regular basis, every N elements. The assumption is that the average size of the element multiplied by N would be smaller than 16MiB. Unfortunately with relation 6677259 that wasn't the case, as versions in its history have more than 25,000 members and were taking up about 100kiB each (so only 164 versions are needed to overflow, relation 6677259 has 440-ish).

This change implements an approximate size counter for relations, which should help to make sure that the pgroup is flushed when it gets too large, even if it hasn't reached N elements yet. This hasn't seemed necessary yet for ways (they are limited to a fixed number of nodes each) or nodes, but extending to handle those - if necessary - should be straightforward.

Reduce relation size limits to try and stop exception during dump

18 Sep 10:27
Compare
Choose a tag to compare

This issue #25 keeps coming up, and this isn't a solution for it, but a work-around until the cause can be found and fixed.

Fixes for library updates

30 Jun 12:55
Compare
Choose a tag to compare

PRs #28 and #29 fix issues with compiling planet-dump-ng with more up-to-date versions of Boost and protobuf.

Fix pg_restore command line.

11 Oct 20:32
Compare
Choose a tag to compare

Seems I tested out the previous release on a machine with a truly ancient version of pg_restore, so the behaviour of the -f flag was old behaviour rather than new. Reverted to the previous version.

Better handling of exceptions in writer_thread

11 Oct 16:51
Compare
Choose a tag to compare

If an exception happened in writer_thread then it was possible for the thread to exit, leaving the other threads waiting forever on a barrier. This change makes the thread with the exception continue the loop so that other threads can finish.

Changed tag ordering

24 Sep 12:26
Compare
Choose a tag to compare

Starting with version 1.2.0, planet-dump-ng sorts tags by the UTF-8 encoded bytes of the keys. Most keys use only ASCII characters, so this corresponds to an alphabetic ordering for those keys. Other keys will depend on how the codepoint is expressed in UTF-8. Values do not contribute to ordering.

See openstreetmap/osmdbt#29 for more details.

Added concurrency limit for disk writing threads

17 Jul 10:33
Compare
Choose a tag to compare

Added a configurable concurrency limit for disk writing threads. This means that the amount of memory waiting to be written to disk shouldn't grow to the level where it uses up all available memory on the server.

Fixes for Ubuntu bionic

16 Jul 15:46
Compare
Choose a tag to compare

Two fixes:

  1. The behaviour of the standard C++ library has been fixed so that std::vector<>::clear() does not deallocate memory. It seems that previous versions of the library did deallocate, which is not standards conforming, but meant that a bug in planet_dump_ng went unnoticed and its memory usage exploded under the new version of the library.
  2. The pg_restore tool from PostgreSQL now requires either a database name or file output argument. Previously the default was to output to stdout, but now this is a hard requirement.