Missing Tiles at All Zoom Levels in Planetiler Output

[CrazyBug-11]

Description
I'm experiencing an issue where my Planetiler process does not generate a complete set of tiles for any zoom level. The process completes without errors, but the resulting MBTiles file lacks a significant number of tiles across all zoom levels. Below are the details of the setup and the code used.

Environment
Java Version: JDK 21.0.3.9-hotspot
Planetiler Version: 0.8-SNAPSHOT
Operating System: Windows
IDE: IntelliJ IDEA 2024.1.3

code
`@ParameterizedTest
@valuesource(strings = {
"--outputType=mbtiles --tile_weights=D:\Project\Java\server-code\src\main\resources\planetiler\tile_weights.tsv.gz --force",

})
void testPlanetilerRunnerParquet(String args) throws Exception {
Path mbtiles = Paths.get("E:\pbftest\planetiler\planetiler.mbtiles");

Planetiler.create(Arguments.fromArgs((args + " --tmpdir=E:server\\LineSpaceData\\pbftest\\planetiler").split("\\s+")))
  .setProfile(new Profile.NullProfile() {
    @Override
    public void processFeature(SourceFeature source, FeatureCollector features) {
      FeatureCollector.Feature feature;
      if (source.canBePolygon()) {
        feature = features.polygon("polygon_data").setZoomRange(0, 7);
      } else if (source.canBeLine()) {
        feature = features.line("line_data").setZoomRange(0, 14);
      } else if (source.isPoint()) {
        feature = features.point("point_data").setMaxZoom(14);
      } else {
        throw new RuntimeException("unkonw");
      }

      source.tags().forEach(feature::setAttr);
    }
  })
  .addParquetSource("parquet", List.of(Paths.get("E:\\LineSpaceData\\parquet\\AR.parquet")))
  .setOutput(mbtiles)
  .run();

}`

Log Output
0:00:00 INF - argument: stats=use in-memory stats
0:00:00 INF - Planetiler build git hash: be927fd
0:00:00 INF - Planetiler build version: 0.8-SNAPSHOT
0:00:00 INF - Planetiler build timestamp: 2024-07-01T07:12:12.525Z
0:00:01 INF - Building profile into file:///E:/测试数据/pbftest/planetiler测试/planetiler.mbtiles in these phases:
0:00:01 INF - parquet: Process features in E:\测试数据\parquet
0:00:01 INF - sort: Sort rendered features by tile ID
0:00:01 INF - archive: Encode each tile and write to TileArchiveConfig[format=MBTILES, scheme=FILE, uri=file:///E://pbftest/planetiler测试/planetiler.mbtiles, options={}]
0:00:01 INF - Using merge sort feature map, chunk size=1407mb max workers=16
0:00:01 INF [parquet] -
0:00:01 INF [parquet] - Starting...
0:00:02 INF [parquet:write] - madvise not available on this system to speed up temporary feature IO.
0:00:12 INF [parquet] - read: [ 1.3M 23% 137k/s ] blocks: [ 276 23% 27/s ] write: [ 0 0/s ] 1.4G
features: 15
cpus: 9.4 gc: 22% heap: 1G/4.2G direct: 29M postGC: 778M
-> (922/1.2k) -> process(39% 38% 43% 44% 41% 44% 41% 41% 43% 43% 38% 38% 40% 35% 38%) -> (1/66k) -> write( 1%)
0:00:22 INF [parquet] - read: [ 3.2M 53% 180k/s ] blocks: [ 643 53% 36/s ] write: [ 0 0/s ] 1.4G
features: 15
cpus: 9.9 gc: 26% heap: 1.3G/4.2G direct: 37M postGC: 1G
-> (556/1.2k) -> process(48% 46% 46% 49% 47% 49% 44% 48% 44% 47% 49% 54% 50% 48% 52%) -> (1/66k) -> write( 0%)
0:00:32 INF [parquet] - read: [ 4.9M 82% 174k/s ] blocks: [ 994 82% 34/s ] write: [ 0 0/s ] 1.4G
features: 15
cpus: 9.9 gc: 24% heap: 955M/4.2G direct: 30M postGC: 1G
-> (205/1.2k) -> process(46% 51% 50% 48% 50% 47% 52% 45% 47% 49% 47% 45% 46% 41% 47%) -> (1/66k) -> write( 1%)
0:00:37 INF [parquet] - read: [ 6M 100% 196k/s ] blocks: [ 1.2k 100% 40/s ] write: [ 1 <1/s ] 585
cpus: 9.8 gc: 26% heap: 1.6G/4.2G direct: 21M postGC: 1G
-> (0/1.2k) -> process( -% -% -% -% -% -% -% -% -% -% -% -% -% -% -%) -> (0/66k) -> write( -%)
0:00:37 INF [parquet] - Processed 6,066,913 parquet features (165k/s, 1,214 blocks, 1 files) in 37s cpu:5m50s gc:9s avg:9.6
0:00:37 INF [parquet] - Finished in 37s cpu:5m50s gc:9s avg:9.6
0:00:37 INF [parquet] - process 15x(45% 16s)
0:00:37 INF [parquet] - write 1x(0% 0.2s wait:35s)
0:00:37 INF [parquet] - Deleting node.db to make room for output file
0:00:37 INF [sort] -
0:00:37 INF [sort] - Starting...
0:00:37 INF [sort] - Grouped 2 chunks into 1
0:00:37 INF [sort] - chunks: [ 1 / 1 100% ] 585
cpus: 0 gc: 0% heap: 1.6G/4.2G direct: 21M postGC: 1G
-> (0/3) -> worker( -%)
0:00:37 INF [sort] - Finished in 0s cpu:0s avg:0
0:00:37 INF [sort] - worker 1x(0% 0s)
0:00:37 INF [sort] - read:0s write:0s sort:0s
0:00:37 INF [archive] -
0:00:37 INF [archive] - Starting...
0:00:38 INF [archive:write] - Starting z7
0:00:38 INF [archive:write] - Finished z7 in 0s cpu:0s avg:0
0:00:38 INF [archive] - features: [ 1 100% 4/s ] 585 tiles: [ 1 4/s ] 20k
cpus: 0.5 gc: 0% heap: 1.6G/4.2G direct: 21M postGC: 1G
read() -> (0/227) -> encode( -% -% -% -% -% -% -%) -> (0/213) -> write( -%)
last tile: 7/38/79 (z7 30%) https://onthegomap.github.io/planetiler-demo/#7.5/-39.90124/-71.71875
0:00:38 INF [archive] - Finished in 0.5s cpu:0.2s avg:0.4
0:00:38 INF [archive] - read 1x(0% 0s)
0:00:38 INF [archive] - encode 15x(0% 0s)
0:00:38 INF [archive] - write 1x(0% 0s)
0:00:38 INF [archive] - Finished in 38s cpu:5m51s gc:9s avg:9.3
0:00:38 INF [archive] - FINISHED!
0:00:38 INF [archive] -
0:00:38 INF [archive] - ----------------------------------------
0:00:38 INF [archive] - data errors:
0:00:38 INF [archive] - ----------------------------------------
0:00:38 INF [archive] - overall 38s cpu:5m51s gc:9s avg:9.3
0:00:38 INF [archive] - parquet 37s cpu:5m50s gc:9s avg:9.6
0:00:38 INF [archive] - process 15x(45% 16s)
0:00:38 INF [archive] - write 1x(0% 0.2s wait:35s)
0:00:38 INF [archive] - sort 0s cpu:0s avg:0
0:00:38 INF [archive] - archive 0.5s cpu:0.2s avg:0.4
0:00:38 INF [archive] - ----------------------------------------
0:00:38 INF [archive] - archive 20kB
0:00:38 INF [archive] - features 585B

Observations
The process completes without errors.
Features are processed from the Parquet file.
A significant number of tiles are missing at all zoom levels.
Only a few tiles with a limited number of features are generated.
Request
I would appreciate any insights or suggestions on why there is a significant number of missing tiles at all zoom levels and how to resolve this issue. Thank you!

You can use this description as your GitHub issue. Make sure to adjust any paths or details specific to your setup as needed.

[CrazyBug-11]

test data:
General Information
Name: AR.parquet
Size: 998.83 MB
Last Modified: July 1, 2024, Monday, China Standard Time, 7:03:01 PM
Provider: ogr
Information Provided by the Program
Storage Format: Parquet
Encoding: UTF-8
Geometry Type: Polygon (MultiPolygon)
Extent:
-73.27683209999999, -55.047093199999991
-53.637687700000007, -21.854216499999997
Total Number of Features: 6,066,913
Coordinate Reference System (CRS)
Name: EPSG:4326 - WGS 84
Units: Geographic coordinates (using longitude as the x-coordinate)
Type: Geographic (2D)

[msbarry]

Planetiler doesn't include empty tiles in the output. It only includes tiles with data in them. Could that explain what you're seeing?

[CrazyBug-11]

I've found that the Geometry obtained in this part of the code is empty during debugging. Is there any parameter that can be set to avoid this, ensuring at least one pixel point is retained?

By previewing the data through QGIS, here is the rendering at different zoom levels. Do you have any suggestions to ensure tile generation even at lower levels? Currently, there are no tiles generated for levels 1-6.

[CrazyBug-11]

Planetiler doesn't include empty tiles in the output. It only includes tiles with data in them. Could that explain what you're seeing?

I've found that the Geometry obtained in this part of the code is empty during debugging. Is there any parameter that can be set to avoid this, ensuring at least one pixel point is retained?

[msbarry]

Ahh so you're saying that the dataset is composed of buildings but at low zooms the buildings are too small and get removed?

[CrazyBug-11]

Yes, my apologies for not describing it clearly earlier. I have identified a few parameters:

arguments.getDouble("min_feature_size_at_max_zoom", "Default value for the minimum size in tile pixels of features to emit at the maximum zoom level to allow for overzooming", 256d / 4096), arguments.getDouble("min_feature_size", "Default value for the minimum size in tile pixels of features to emit below the maximum zoom level", 1), arguments.getDouble("simplify_tolerance_at_max_zoom", "Default value for the tile pixel tolerance to use when simplifying features at the maximum zoom level to allow for overzooming", 256d / 4096), arguments.getDouble("simplify_tolerance", "Default value for the tile pixel tolerance to use when simplifying features below the maximum zoom level", 0.1d),

By reducing these parameters, I noticed that more tiles are generated. Could you please confirm if my approach is correct? I apologize if this question seems basic.

[msbarry]

Yes, for this use-case that seems like it makes sense. There are a few other options as well:

1. You could call setMinPixelSize(0) on the features during process step, then use FeatureMerge.mergeOverlappingPolygons in postProcess similar to: https://github.com/onthegomap/planetiler-examples/blob/f49040cc7e1f067533f508716fee2ad1f83e5c91/overturelayers/Water.java#L63
2. @bdon was looking into turning polygons into points when they collapse to be too small. Brandon, what did you end up landing on there?

[bdon]

I didn't get far on that. The approach would be to somehow deterministically sample the points at low zooms, but that may require a first pass over the data to determine what % to sample. Open to ideas on how to do this efficiently using the profiles APIs.

[CrazyBug-11]

Yes, for this use-case that seems like it makes sense. There are a few other options as well:

1. You could call setMinPixelSize(0) on the features during process step, then use FeatureMerge.mergeOverlappingPolygons in postProcess similar to: https://github.com/onthegomap/planetiler-examples/blob/f49040cc7e1f067533f508716fee2ad1f83e5c91/overturelayers/Water.java#L63
2. @bdon was looking into turning polygons into points when they collapse to be too small. Brandon, what did you end up landing on there?

Thank you very much for your patient response. By adjusting the parameters, I was able to generate more tiles.

[CrazyBug-11]

hello，@msbarry
Currently, I am testing some large datasets, and the generated tiles, especially at lower zoom levels, often result in large blank areas due to the small size of features. This gives the impression that there are no features in these areas, which is misleading. Is there a way to ensure that some features are retained in these regions to indicate that there is data, rather than leaving them completely blank? Could you please provide some suggestions on how I should address this issue?

[msbarry]

Are you using setMinPixelSize(0)? If so and you are still seeing blank areas then you'll need to switch to the approach of turning them into points and sampling below a certain zoom level.

[CrazyBug-11]

Yes, I am currently using setMinPixelSize(0), but unfortunately, there are still some blank areas. According to my understanding, setMinPixelSize(0) should represent the raw data, so theoretically, there shouldn't be any blank areas. Additionally, could you please suggest how I should switch to the approach of turning them into points and sampling below a certain zoom level?

[msbarry]

Vector tile shapes get snapped to integer coordinates on a 4096x4096 grid, so even with min size set to 0 shapes can collapse down to less than a single pixel and will get dropped.

With no changes to planetiler though, you could look at which zoom features start to disappear and do:

int minPolygonZoom = ...;
var polgons = features.polygon("buildings").setMinZoom(minPolygonZoom);
var points = features.centroid("buildings").setMaxZoom(minPolygonZoom - 1);

On the planetiler side we could change that behavior so when min size=0 and a feature collapses to less than a pixel, it gets replaced with the smallest triangle or square that covers the original geometry. Or we could add an opt-in feature where those polygons collapse down to a single point instead like feature.setCollapseToPoints(true) or something. Sampling could happen independently. @bdon how do either of those options sound from overture buildings perspective?

[CrazyBug-11]

In planetiler I can't find the logic that features smaller than a single pixel will be discarded. Can you tell me if this logic is in planetiler or in other code?

[msbarry]

The first thing to try would be changing calls to GeometryPredicision.reduce in GeoUtils.snapAndFixPolygon to:

GeometryPrecisionReducer reducer = new GeometryPrecisionReducer(tilePrecision);
reducer.setRemoveCollapsedComponents(false);
return reducer.reduce(geom);

If that doesn't work, it may need custom handling.

[bdon]

for Overture Buildings it makes sense to collapse down to a square "dust" polygon that covers the original geometry. Other we would need a second style rule just for the generalized points.