Configurable tile systems for projections other than web mercator

[burleight]

This is first attempt at addressing #343

Adds a tile_systems section to the yml config:

tile_systems:
  nztm:
    srid: 2193
    bounds: [1000000, 4700000, 2600000, 6300000]
  wgs84:
    srid: 4326
    bounds: [-180, -90, 180, 90]

Then, the tile URL becomes /{source}:{tile_system}/{z}/{x}/{y}

Issues:

• This in effect leads to each table source being duplicated in the catalog. E.g. table1, table1:nztm, table1:wgs84 etc
• The URL for a multiple source request becomes /{source}:{tile_system},{source2}:{tile_system}/{z}/{x}/{y}.
• Nothing is done to prevent multiple source requests using different tile systems
• Needs test coverage
• Configuration documentation needs updating

For the OpenLayer users out there. the trick to using this is to giving VectorTile source a custom tile grid:

new ol.source.VectorTile({
      format: new ol.format.MVT(),
      projection: 'EPSG:2193',
      tileGrid: new ol.tilegrid.TileGrid({
          extent: [1000000, 4700000, 2600000, 6300000], // square of width 1600000
          resolutions: (new Array(10)).fill(0).map((e, i) => 1600000 / (256 * Math.pow(2, i)))
      }),
      url: `http://0.0.0.0:3000/${layer}/{z}/{x}/{y}`
  })

[codecov-commenter]

Codecov Report

Patch coverage: 26.82% and project coverage change: -0.35 ⚠️

Comparison is base (3719f9c) 59.17% compared to head (79125ae) 58.83%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #611      +/-   ##
==========================================
- Coverage   59.17%   58.83%   -0.35%     
==========================================
  Files          30       31       +1     
  Lines        2756     2806      +50     
==========================================
+ Hits         1631     1651      +20     
- Misses       1125     1155      +30     

Impacted Files	Coverage Δ
src/lib.rs	33.51% <ø> (ø)
src/pg/table_source.rs	39.21% <0.00%> (-3.34%)	⬇️
src/pg/configurator.rs	40.07% <5.26%> (-2.33%)	⬇️
src/pg/config.rs	82.69% <22.22%> (-4.05%)	⬇️
src/config.rs	63.52% <50.00%> (+3.30%)	⬆️
src/tilesystems/config.rs	94.44% <94.44%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[nyurik]

Thanks @burleight for working on this! How common would you think it would be for the users to want all projections for all table sources? What do you think as the most common use cases? We need to address two basic cases: table sources when auto-configuring PG connection, and a case when the user wants to pre-configure a specific source with a given projection.

I think for auto-config, we should add something to the postgres.auto_publish.tables, e.g. the config you gave as an example. The id_format may also need to be expanded to support it, e.g. table.{proj}.{schema}.{table}.{column}, but it wouldn't be set to that by default (to avoid empty proj var.

The individual per-table configurations may be better off to support only a single projection using postgres.tables.table_source_id where these params might be placed

[burleight]

How common would you think it would be for the users to want all projections for all table sources?

This is our use case, we have a lot of data we display in EPSG:2193, and we'd like to experiment with using martin to serve it.

There are global coordinate systems, such as EPSG:4326 and EPSG:3857, as well as local coordinate systems, such as EPSG:2193, which is the current New Zealand mainland coordinate system. Global to global, global to local and local to global transformations almost always make sense, but local to local systems don't always apply. I.e. it wouldn't make sense to project from the New Zealand coordinate system into the Swiss coordinate system.

So, I may help if we had a way to filter our transformations that don't make sense. We could configure which transformations to a tile system are allowed:

tile_systems:
  swiss:
    srid: 2056
    bounds: ...
  nztm:
    srid: 2193
    bounds: [1000000, 4700000, 2600000, 6300000]
    restrict_to_srids: [4326, 3857, 27200] # 27200 is an older New Zealand local projection

If restrict_to_srids is specified, then the tile_system is only auto published for tables/geometry columns with those srids. Implicitly, always allow identity transformations 2193 -> 2193

I think for auto-config, we should add something to the postgres.auto_publish.tables

How about keeping tile_systems global, then reference tile system ids in postgres.auto_publish.tables and postgres.tables

So, by default, if someone specifies the tile systems global, all of them are used by auto_publish but, this could be overriden in postgres.auto_publish.tables.tile_systems like this:

postgres:
  auto_publish:
    tables:
      # only auto publish the default Web Mercator system
      tile_systems: ['default'] 
      # or allow the default and some custom systems
      tile_systems: ['default', 'nztm'] 
      # only auto publish a custom system
      tile_systems: ['nztm']
  # individual table sources may override which tile systems apply
  tables:
    table_source_id:
      schema: public
      table: my_swiss_data
      tile_systems: ['default', 'swiss']

[nyurik]

Thanks, but do you think that for each table source you would want to publish all possible projections? Or should the projection simply be a setting on the source itself? At the end of the day, there will still be an internal source_id => handler mapping, and that source_id could be anything, including foo:bar syntax (only simple letters/digits are allowed at the moment though, but that can change). So in theory src:swiss and src:nztm is no different than src_swiss and src_nztm - you just have to specify the same config twice. I just feel weird about having a global switch that enables reprojection on every source... plus we could treat this as a "style" rather than projection --- basically it could be a bunch of settings including projection that re-configure a source.

I propose we break this problem into parts:

• support configuration for just one pre-configured source: currently we already have the srid and bounds params on a table source, so need to figure out if any extra settings are needed, implement the needed query modification, merge, ship.
• decide if/how we could optimize multiple pre-configured sources, e.g. introduce a way to share a bunch of settings across multiple sources.
• decide if/how we should support auto-discovery
• decide if we want to have martin-level (as oppose to just postgres-tables level) re-projection support

[burleight]

Our use case is, we have a bunch of datasets, most of them NZTM. We'd be setting up a vector tile server that publishes these datasets. For the apps that we write, we'd be picking the best projection for the job. So, if we're making something which only applies to the NZ mainland, we'd tile the data with NZTM2000Quad, in other cases we'd use web mercator.

should the projection simply be a setting on the source itself?

Should martin lift these variants into source setting. Instead of source_id => handler, would it be better to have a source_id => variant_id => handler mapping? Tile variants could be accessed with query params - {source_id}/{z}/{x}/{y}?tileSystem={tileSystemId}. Also, with query params, people could write postgresql functions that support multiple tile schemes and a setting in postgres.functions markup, which tile systems a function supports.

support configuration for just one pre-configured source: currently we already have the srid and bounds params on a table source, so need to figure out if any extra settings are needed

table.srid defines the input projection and table.bounds restricts the output extent, they're both orthogonal to the tiling method chosen, so the table source configuration would need an extra setting for specifying how the source should be tiled.

decide if we want to have martin-level (as oppose to just postgres-tables level) re-projection support

I still think the definition of a tile system should sit outside of the postgres context. I'd also like to have a route, similar to catalog, which exposes the tile system metadata

[nyurik]

• So technically you can already solve the whole thing without a single Martin change -- you can simply create a pg function that accepts table name and projection params, and just return the resulting tile. Not ideal, and the autogenerated tilejson might be messed up, so you may have to have some static hosting of it, but could work.
• tile system - does it include re-projection and/or other things? Per your code, you do transform into a non-3857, and use projection bounds for the tile envelope. Guess its simple enough. Possibly anything else? What type of a catalog-like output do you see?
• the issue with moving tile system outside of postgres is that we would need to support it in pre-tiled cases. Martin supports pg functions, pmtiles and mbtiles - all of which return web mercator x,y,z -> tile_blob tiles, and those tiles could be vector or raster tiles.
• Query params like ?projection=foo or tilesystem=foo could work, but it goes so much against the whole CDN/caching concept, plus it limits the whole thing to pg tables only, and wouldn't work well with anything else (mixing multiple sources becomes a mess). Plus it codifies params structure for tiles - not ideal either.
• The issue still not answered - can we start small - first the minimal one where a user can specify geometry ST_transform srid and the output envelope bounds as additional params to the pre-configured TABLE settings? No CLI changes, only yaml for now. We could call it tile_srid and tile_bounds or some other name.

[nyurik]

should the projection simply be a setting on the source itself?

Should martin lift these variants into source setting. Instead of source_id => handler, would it be better to have a source_id => variant_id => handler mapping? Tile variants could be accessed with query params - {source_id}/{z}/{x}/{y}?tileSystem={tileSystemId}. Also, with query params, people could write postgresql functions that support multiple tile schemes and a setting in postgres.functions markup, which tile systems a function supports.

No, i think we should not expose this as a parameter at all, at least not in the first iteration. Instead, given a table of "my_geometries", you can configure two sources (in yaml): my_3857_geometries and my_2193_geometries, and the client decides dynamically which one to use (you can put srid at the end for simplicity of coding - i wanted to highlight the fact that srid part of the source id is not handled in any special way by martin itself)

so once it is possible to configure two separate data sources, each with its own geometry transofrmation function and its own tiling schema, we could proceed to other optimizations - like making it possible to auto-generate such configurations, etc

[raboczi]

Is there anything that should be done to anticipate the needs of the proposed Adaptive Composite Map Projection support maplibre/maplibre#190? Perhaps particular SRIDs are only required at certain zoom levels, for instance.

[nyurik]

Is there anything that should be done to anticipate the needs of the proposed Adaptive Composite Map Projection support maplibre/maplibre#190? Perhaps particular SRIDs are only required at certain zoom levels, for instance.

That's a good question... I am not an expert on projection and globe re-projection, so can't tell for certain - but my understanding was that it is possible to construct a globe view from web mercator ... some-magically-how... One concern though is that web mercator hides the poles, so it would have to fake them by simply showing them as snow

[nyurik]

hi @burleight, do you need help with this PR? Any blockers? Thx!

[burleight]

hi @burleight, do you need help with this PR? Any blockers? Thx!

I believe I got to the point where the code was working but the configuration was a bit verbose/ugly and there wasn't much test coverage of the new changes.

Thank you.

[gdnwr]

When can it be used?

[CommanderStorm]

When can it be used?

As mentioned above, this PR is blocked on testing. If you want to contribute in this area, this would be appreciated 🙏.
You can be the one who pushes this over the finish line..

As for timelines: sorry, we don't have any. This is a volunteer project and things get done when they get done.