OpenMapTiles 3.15.0 SNAPSHOT (#126)

* version bumped from 3.14.0 to 3.15.0-SNAPSHOT

* regenerate-openmaptiles.sh 07f243c5d9efa558fa539d7a31b2ae50507aaa9d (to match content of OMT PR 1457)

* SQL -> Java re-implementation of OMT PR 1457

* version bumped from 3.14.0 to 3.15.0-SNAPSHOT

* WaterName.areaToMinZoom(): improved handling of rounding and precission + added unit tests

* mvn spotless:apply

* water label min. zoom calculation simplified

* comment adjusted to be hopefully more useful

* mvn spotless:apply

* minzoom for CA_TRANSCANADA and US_INTERSTATE trunk now 4 (to match OMT PR 1440)

* minzoom for some other Canada trunks now 4 (to match OMT PR 1446)

* equals() simplified + clean-up of comments

* regenerate-openmaptiles.sh 5f7b2c11b3224759a21133381ca7d959a1f3cf51 (to match content of OMT PR 1465)

* GB road relations processing adjusted to match OMT PR 1465, e.g. handle also primary and secondary roads

* regenerate-openmaptiles.sh edb42f2db3c2b0ec37045367720eed84d7bbd71f (to match content of OMT PR 1466)

* IE road relations processing adjusted to match OMT PR 1466, e.g. handle IE roates in similar way as GB routes

* fixed handling of networkType for secondary GB routes

* clean-up: case statements simplified

* mvn spotless:apply

* clazz calculation moved up so that minzoom can be set to 3 for only lakes (to match OMT PR 1475)

* unit tests adjusted + extended to cover 'minzoom=3 fore lakes' change

* fixed minor typo from previous PR

* render POIs for large universities at low zoom (to match OMT PR 1479)

* clean-up, to make the diff/PR smaller

* regenerate-openmaptiles.sh 5e9b7c475d53a5bd5ea394da361594d3f4ce2d66 (to match content of OMT PR 1485)

* handle 'grade1' and 'tracktype' as per OMT PR 1485

* added implementation of agg_stop

It is based on OMT PR 1480 (which contains latest the fix) and the rest of older code
(which was not worling properly until the fix).

* clean-up: mvn spotless:apply

* Long ferries (as per OMT PR 1486)

* regenerate-openmaptiles.sh b3d67ed5b327c9059aeea0b3304772c6b4c8c7e9  (to match content of OMT PR 1489)

* Add aboriginal lands (as per OMT PR 1489)

* handle duplicate route relations (to match OMT PR 1501)

* regenerate-openmaptiles.sh master, to match several OMT PRs which adjusted only YML

* URLs in comments adjusted to match OMT PR 1560

* Convert separated addresses to dashed addresses

* add brunnel (and layer) attributes only for certain zoomlevels, depending on feature size (matching OMT PR 1579)

* unit test testInterstateMotorway(): brunnel tag for test line no longer available at Z8

* unit test testInterstateMotorway() clean-up: Z13 was tested twice

* minor clean-up: fixed unit test naming

* partial fix for differences in transportation_name layer

The difference is between OpenMapTiles/master (OMT) and
planetiler-openmaptiles/omt_3_15_0 (PT-OMT) (e.g. development versions).

The point is, that while PT-OMT was using limit of "8km" for Z9-Z11, OMT
is using limit "ST_Length(geometry) > 8000 / POWER(2, zoom_level - 9)
AND zoom_level BETWEEN 9 AND 11".

Some further differences still visible, hence further commits expected.

* further adjustments to better match what is done with ferries in OMT

... (as per OMT PR 1486)

But FERRY_MIN_PIXEL_SIZE is "too much" in the contexct of Planetiler,
since it is applied within tiles, hence causes gaps in lines if a line
"strikes a little" certain tile. Hence we will need to divert a little.

* ferry minLength tweak + clean-up

* mvn spotless:apply

* fixed minor typo

* minor reformatting

* ferry line length filter replaced with min. zoom calculation

hence the results are much closer to what OMT is doing for Z4-Z9

* testFerry() adjusted to match previous commit

ferry test polygon with area 1 now qualifies for min. zoom 5

* clea-up of unused stuff + mvn spotless:apply

* mvn spotless:apply

* added TODO node for follow-up pull-request/simplification

* clean-up: common getMinZoom() code moved to Utils

* minzoom clipping for brunnel was adjusted do Z9-Z12 -> test adjusted too

* clean-up

* use same tolerance for all transportation items, like OSM does

* clean-up, since ferry and non-ferry procesing is now same

* we need regenerate to work with master branch for now

* first sub-class search for agg_stop simplified a little

* contains() used instead of indexOf() for better readability

* numbers as list, not array, so that getFirst() and getLast() can be used

* better trimming and filtring of housenumbers

* adjusted handling of large house numbers

* several unit tests collapsed to one with @ParameterizedTest + @CsvSource

* AGG_STOP_SUBCLASS_ORDER simplified from Map to List

* fixed major omission from previous commit

* clamp() used to replace min()&max() combo

* agg_stop now implemented

* fixed typo in the error message

* prepare IE and GB boundary geometry outside of synchronized{}

* fixed typo in the error message

* mvn spotless:apply

* switch statements for IE and GB route networks simplified

* avoid RouteNetwork->String mapping, not needed for anyMatch()

* fix: attr. brunnel optional based on size on Z4-Z11, attr. layer optional between Z9-Z11

* tolerance change in transportation reverted, added note to README as per why

* fix: monzoom for sea&co. is Z0-Z14 based on area, for the rest it is Z3-Z14 again based on area

* clean-up: avoid doing area->side->area, do just area

* regenerate-openmaptiles.sh 6c31841f4674f15e15afde346a060cf7c22e6cdd (to match content of OMT PR 1591)

* relevant process() functions adjusted to match changes in transportation/mapping.yaml

* regenerate-openmaptiles.sh master, instead of 6c31841f4674f15e15afde346a060cf7c22e6cdd (to match content of OMT PR 1591, in a cleaner way)

* introduce duplicate housenumber filtering (matching OMT PR 1391)

* (less related) clean-up: use isEmpty() instead if size check

* testContainsHousenumber UT adjusted, since duplicate housenumber filtering is reducing amount of house numbers

* use combination of uic_ref, name, network and operator as key for agg_stop sets

If we rely on only on `uic_ref` we group together also stations which are
too far apart (even different cities). With this combo results seem OK,
e.g. all grouped stations are within around 950m (1000 pixels at Z14) of
each other (1000 being used in `PARTITION BY LabelGrid(...` in
`layers/poi/poi.sql` in OpenMapTiles).

* agg_stop comparison made more explicit, since we want to match same exact one

* mvn spotless:apply

* name now important for agg_stop processing, hence name:es (ab)used for unit tests

* agg_stop: simplified processing of nearest station

Results still same, only ordering is different:
- previously: agg_stop=1 first
- now: FIFO

* agg_stop: forther code simplification

* fixed major typo introduced in previous merge

* setMinPixelSize() + setMinZoom() used instead of areaToMinZoom()

* clean-up: unused stuff removed

* mvn spotless:apply

* setAttrWithMinSize() used instead of getBrunnelMinzoom()

getFerryMinzoom() kept since we'd like to replicate `sql_filter: ST_Length(...` from OMT

* getMinZoomForLength() no longer used, hence removed

* clean-up: LOG2 not used, hence removed

* added BY_TEMP_HAS_NAME comparator to avoid its repeated construction during run-time

* duplicate houcenumber processing simplified further

* clean-up: get(0) replaced with getFirst()

* clean-up: CPU-intensive prepare() moved out of synchronized block

* regenerate-openmaptiles.sh 3cf77e2a542d8a369bb08bf2538cdde0b3effb2b (to match content of OMT PR 1423)

* unit test adjusted for POI office class changes

* regenerate-openmaptiles.sh master (to match content of OMT PR 1544)

* added charging_station implementation matching OMT PR 1544

* use setMinPixelSizeBelowZoom() instead of uniAreaToMinZoom()

* use setMinPixelSizeBelowZoom() instead of getFerryMinzoom()

* fixed unit test, to match recent tweaks

README.md

@@ -26,11 +26,11 @@ available options.
 ## Differences from OpenMapTiles
 
 - Road name abbreviations are not implemented yet in the `transportation_name` layer
-- `agg_stop` tag not implemented yet in the `poi` layer
 - `brunnel` tag is excluded from `transportation_name` layer to avoid breaking apart long `transportation_name`
   lines, to revert this behavior set `--transportation-name-brunnel=true`
 - `rank` field on `mountain_peak` linestrings only has 3 levels (1: has wikipedia page and name, 2: has name, 3: no name
   or wikipedia page or name)
+- some line and polygon tolerances are different, can be tweaked with `--simplify-tolerance` parameter
 
 ## Customizing
 

pom.xml

@@ -22,7 +22,7 @@
 
   <groupId>org.openmaptiles</groupId>
   <artifactId>planetiler-openmaptiles</artifactId>
-  <version>3.14.0</version>
+  <version>3.15.0-SNAPSHOT</version>
 
   <name>OpenMapTiles Vector Tile Schema implementation for Planetiler tool</name>
 

scripts/regenerate-openmaptiles.sh

@@ -4,7 +4,8 @@ set -o errexit
 set -o pipefail
 set -o nounset
 
-TAG="${1:-"v3.14"}"
+# TODO: change to "v3.15" once that is released
+TAG="${1:-"master"}"
 echo "tag=${TAG}"
 
 BASE_URL="${2:-"https://raw.githubusercontent.com/openmaptiles/openmaptiles/"}"

src/main/java/org/openmaptiles/OpenMapTilesProfile.java

@@ -220,12 +220,12 @@ public interface NaturalEarthProcessor {
 
   /**
    * Layers should implement this interface to subscribe to elements from
-   * <a href="https://github.com/lukasmartinelli/osm-lakelines">OSM lake centerlines source</a>.
+   * <a href="https://github.com/openmaptiles/osm-lakelines">OSM lake centerlines source</a>.
    */
   public interface LakeCenterlineProcessor {
 
     /**
-     * Process an element from the <a href="https://github.com/lukasmartinelli/osm-lakelines">OSM lake centerlines
+     * Process an element from the <a href="https://github.com/openmaptiles/osm-lakelines">OSM lake centerlines
      * source</a>
      *
      * @see Profile#processFeature(SourceFeature, FeatureCollector)

src/main/java/org/openmaptiles/layers/Housenumber.java

@@ -43,9 +43,18 @@ OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 import com.onthegomap.planetiler.geo.GeometryException;
 import com.onthegomap.planetiler.stats.Stats;
 import com.onthegomap.planetiler.util.Translations;
+import java.util.Arrays;
+import java.util.Comparator;
 import java.util.List;
+import java.util.function.Predicate;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
 import org.openmaptiles.generated.OpenMapTilesSchema;
 import org.openmaptiles.generated.Tables;
+import org.openmaptiles.util.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /**
  * Defines the logic for generating map elements in the {@code housenumber} layer from source features.
@@ -59,19 +68,96 @@ public class Housenumber implements
   Tables.OsmHousenumberPoint.Handler,
   ForwardingProfile.FeaturePostProcessor {
 
-  public Housenumber(Translations translations, PlanetilerConfig config, Stats stats) {}
+  private static final Logger LOGGER = LoggerFactory.getLogger(Housenumber.class);
+  private static final String OSM_SEPARATOR = ";";
+  private static final String DISPLAY_SEPARATOR = "–";
+  private static final Pattern NO_CONVERSION_PATTERN = Pattern.compile("[^0-9;]");
+  private static final String TEMP_PARTITION = "_partition";
+  private static final String TEMP_HAS_NAME = "_has_name";
+  private static final Comparator<VectorTile.Feature> BY_TEMP_HAS_NAME = Comparator
+    .comparing(i -> (Boolean) i.attrs().get(TEMP_HAS_NAME), Boolean::compare);
+  private final Stats stats;
+
+  public Housenumber(Translations translations, PlanetilerConfig config, Stats stats) {
+    this.stats = stats;
+  }
+
+  private static String displayHousenumberNonumeric(List<String> numbers) {
+    return numbers.getFirst()
+      .concat(DISPLAY_SEPARATOR)
+      .concat(numbers.getLast());
+  }
+
+  protected static String displayHousenumber(String housenumber) {
+    if (!housenumber.contains(OSM_SEPARATOR)) {
+      return housenumber;
+    }
+
+    List<String> numbers = Arrays.stream(housenumber.split(OSM_SEPARATOR))
+      .map(String::trim)
+      .filter(Predicate.not(String::isEmpty))
+      .toList();
+    if (numbers.isEmpty()) {
+      // not much to do with strange/invalid entries like "3;" or ";" etc.
+      return housenumber;
+    }
+
+    Matcher matcher = NO_CONVERSION_PATTERN.matcher(housenumber);
+    if (matcher.find()) {
+      return displayHousenumberNonumeric(numbers);
+    }
+
+    // numeric display house number
+    var statistics = numbers.stream()
+      .collect(Collectors.summarizingLong(Long::parseUnsignedLong));
+    return String.valueOf(statistics.getMin())
+      .concat(DISPLAY_SEPARATOR)
+      .concat(String.valueOf(statistics.getMax()));
+  }
 
   @Override
   public void process(Tables.OsmHousenumberPoint element, FeatureCollector features) {
+    String housenumber;
+    try {
+      housenumber = displayHousenumber(element.housenumber());
+    } catch (NumberFormatException e) {
+      // should not be happening (thanks to NO_CONVERSION_PATTERN) but ...
+      stats.dataError("housenumber_range");
+      LOGGER.warn("Failed to convert housenumber range: {}", element.housenumber());
+      housenumber = element.housenumber();
+    }
+
+    String partition = Utils.coalesce(element.street(), "")
+      .concat(Utils.coalesce(element.blockNumber(), ""))
+      .concat(housenumber);
+    Boolean hasName = element.hasName() == null ? Boolean.FALSE : !element.hasName().isEmpty();
+
     features.centroidIfConvex(LAYER_NAME)
       .setBufferPixels(BUFFER_SIZE)
-      .setAttr(Fields.HOUSENUMBER, element.housenumber())
+      .setAttr(Fields.HOUSENUMBER, housenumber)
+      .setAttr(TEMP_PARTITION, partition)
+      .setAttr(TEMP_HAS_NAME, hasName)
       .setMinZoom(14);
   }
 
   @Override
   public List<VectorTile.Feature> postProcess(int zoom, List<VectorTile.Feature> list) throws GeometryException {
+    // remove duplicate house numbers, features without name tag are prioritized
+    var items = list.stream()
+      .collect(Collectors.groupingBy(f -> f.attrs().get(TEMP_PARTITION)))
+      .values().stream()
+      .flatMap(
+        g -> g.stream().min(BY_TEMP_HAS_NAME).stream()
+      )
+      .toList();
+
+    // remove temporary attributes
+    for (var item : items) {
+      item.attrs().remove(TEMP_HAS_NAME);
+      item.attrs().remove(TEMP_PARTITION);
+    }
+
     // reduces the size of some heavy z14 tiles with many repeated housenumber values by 60% or more
-    return FeatureMerge.mergeMultiPoint(list);
+    return FeatureMerge.mergeMultiPoint(items);
   }
 }

src/main/java/org/openmaptiles/layers/Park.java

@@ -91,15 +91,20 @@ public Park(Translations translations, PlanetilerConfig config, Stats stats) {
 
   @Override
   public void process(Tables.OsmParkPolygon element, FeatureCollector features) {
-    String protectionTitle = element.protectionTitle();
-    if (protectionTitle != null) {
-      protectionTitle = protectionTitle.replace(' ', '_').toLowerCase(Locale.ROOT);
+    String clazz;
+    if ("aboriginal_lands".equals(element.boundary())) {
+      clazz = "aboriginal_lands";
+    } else {
+      String protectionTitle = element.protectionTitle();
+      if (protectionTitle != null) {
+        protectionTitle = protectionTitle.replace(' ', '_').toLowerCase(Locale.ROOT);
+      }
+      clazz = coalesce(
+        nullIfEmpty(protectionTitle),
+        nullIfEmpty(element.boundary()),
+        nullIfEmpty(element.leisure())
+      );
     }
-    String clazz = coalesce(
-      nullIfEmpty(protectionTitle),
-      nullIfEmpty(element.boundary()),
-      nullIfEmpty(element.leisure())
-    );
 
     // park shape
     var outline = features.polygon(LAYER_NAME).setBufferPixels(BUFFER_SIZE)
@@ -138,7 +143,7 @@ private int getMinZoomForArea(double area) {
     // sql filter:    area > 70000*2^(20-zoom_level)
     // simplifies to: zoom_level > 20 - log(area / 70000) / log(2)
     int minzoom = (int) Math.floor(20 - Math.log(area / WORLD_AREA_FOR_70K_SQUARE_METERS) / LOG2);
-    minzoom = Math.min(14, Math.max(5, minzoom));
+    minzoom = Math.clamp(minzoom, 5, 14);
     return minzoom;
   }
 

src/main/java/org/openmaptiles/layers/Place.java

@@ -235,7 +235,7 @@ public void process(Tables.OsmCountryPoint element, FeatureCollector features) {
         rank = country.rank;
       }
 
-      rank = Math.max(1, Math.min(6, rank));
+      rank = Math.clamp(rank, 1, 6);
 
       features.point(LAYER_NAME).setBufferPixels(BUFFER_SIZE)
         .putAttrs(names)
@@ -261,7 +261,7 @@ public void process(Tables.OsmStatePoint element, FeatureCollector features) {
         if (nullOrEmpty(names.get(Fields.NAME_EN))) {
           names.put(Fields.NAME_EN, state.name);
         }
-        int rank = Math.min(6, Math.max(1, state.rank));
+        int rank = Math.clamp(state.rank, 1, 6);
 
         features.point(LAYER_NAME).setBufferPixels(BUFFER_SIZE)
           .putAttrs(names)