The OpenMapTiles toolbox for generating TM2Source projects, imposm3 mappings and SQL instructions from OpenMapTiles layers. We encourage other people to use this for their vector tile projects as well since this approach works well for us.
Check out the OpenMapTiles project for a real world example.
You need either just Docker or Python 3 installed on your system. If running without Docker, some of the scripts require graphviz and sqlite3 dependencies.
The easiest is to use docker directly to run this command. You do not need to clone openmaptiles-tools locally, just clone the openmaptiles repo and run from its root.
Note: container scripts can only access files from the given directory and below, e.g. in this example - ${PWD} - current dir.
docker run -it --rm -u $(id -u ${USER}):$(id -g ${USER}) \
-v "${PWD}:/tileset" \
openmaptiles/openmaptiles-tools \
<script-name> <script-parameters>Where the <script-name> could be any of the scripts in the bin/ directory, e.g. generate-imposm3 openmaptiles.yaml.
# Some tool require these packages. On Debian/Ubuntu you can install them with
sudo apt install graphviz sqlite3
# install the package directly from git
python3 -m pip install git+https://github.com/openmaptiles/openmaptiles-tools
# Run the script you want, e.g. from the openmaptiles dir:
generate-imposm3 openmaptiles.yaml
# If the script doesn't run, make sure your PATH includes default PIP bin directory.
# On Debian/Ubuntu that would be ~/.local/bin/ (under your home dir).
# Otherwise just run it with ~/.local/bin/generate-imposm3 ...Make sure you have all dependencies from the Usage section above. You should have the latest Python (3.6+)
# Get OpenMapTiles layer data
git clone https://github.com/openmaptiles/openmaptiles.git
# Get the tools repo
git clone https://github.com/openmaptiles/openmaptiles-tools.git
# Run scripts from the root of the tools repository
cd openmaptiles-tools
# Install required dependencies (might need to run with sudo
python3.6 -m pip install -r requirements.txt
# The PYTHONPATH=$PWD allows script to find its modules located in the current dir.
PYTHONPATH=$PWD python3 bin/generate-imposm3 ../openmaptiles/openmaptiles.yamlUse make test to run all of the tests locally. The Makefile will build a docker image with all the code, run all tests, and compare the build result with the files in the testdata/expected dir.
Run make rebuild-expected after you modify the output produced by the generation scripts. This will re-create the expected test results to match the actual ones, and make sure the changes are what you want.
You define a self contained Layer together with SQL files and layer and data source definitions (like an imposm3 mapping file) that you can then reference in a Tileset where you mix and match with other layers.
Take a look or copy a standard layer like building to get started with your own layer. A layer consists out of a Layer definition written in YAML format.
There you specify the layer properties like id, buffer_size and possible Markdown documentation (description and fields).
You can also reference SQL files in schema for writing the necessary queries for your layer or create generalized tables.
We encourage you to have a function per layer which takes the bounding box and zoom level. This makes it easy
to test and reuse.
If your data is based of OSM you can also directly reference a imposm3 mapping file to choose the OSM data you need.
layer:
id: "building"
description: Buildings from OpenStreetMap
buffer_size: 4
datasource:
query: (SELECT geometry FROM layer_building(!bbox!, z(!scale_denominator!))) AS t
fields:
render_height: An approximated height from levels and height of building.
class:
description: Defines a subclass of a building (one of the known values).
# Values can be either a list of strings, or a dictionary
# Dictionary defines mapping of OSM values to the OMT field value
values:
school:
subclass: ['school','kindergarten']
alcohol_place:
shop: ['bar']
subclass: ['alcohol','beverages','wine%']
schema:
- ./building.sql
datasources:
- type: imposm3
mapping_file: ./mapping.yamlFor the well known values (enums), the fields section can also contain the mapping of the input (OSM) values.
The above example has two output fields - render_height and class. The class field could be one of the predefined
values. An object would have class=school if the OSM object has subclass either school or kindergarten.
An object would have class=alcohol_place if it either has shop=bar or subclass having one of the 3 values.
If a layer SQL files contains %%FIELD_MAPPING: class%%, generate-sql script will replace it
SELECT CASE
%%FIELD_MAPPING: class%%
ELSE NULL
END, ...into
SELECT CASE
WHEN "subclass" IN ('school', 'kindergarten') THEN 'school'
WHEN "shop"='bar'
OR "subclass" IN ('alcohol','beverages')
OR "subclass" LIKE 'wine%'
THEN 'alcohol_place'
ELSE NULL
END, ...A Tileset defines which layer will be in your vector tile set (layers)
and metadata used for generating a TM2Source project to actually generate the vector tiles.
tileset:
layers:
- layers/building/building.yaml
- layers/housenumber/housenumber.yaml
- layers/poi/poi.yaml
name: Street Level
description: A tileset showing street level info like building, housenumbers and POIs.
attribution: "OpenStreetMap contributors"
maxzoom: 14
minzoom: 13
center: [-12.2168, 28.6135, 4]
bounds: [-180.0,-85.0511,180.0,85.0511]
pixel_scale: 256
defaults:
srs: +proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0.0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs +over
datasource:
srid: 900913Use test-perf to evaluate tile generation performance against a PostgreSQL database.
This utility can test individual layers, several layers at once, as well as the whole tile.
It has several pre-defined testing areas to provide cross-test consistency.
Results are printed as histogram graphs, showing tile size distribution.
The utility can compare current run with a previously saved one, highlighting large changes.
If multiple zoom levels are tested, another histogram shows per-zoom size distribution.
Run with --help to see all options.
test-perf <tileset> ...
Just like postserve below, test-perf requires PostgreSQL connection.
Postserve is an OpenMapTiles map vector tile test server that dynamically generates metadata and tiles directly from PostgreSQL database based on the tileset file definition.
postserve <tileset> ...
Use postserve <tileset> to start serving. Use --help to get the list of Postgres connection parameters.
If you have a full planet database, you may want to use MIN_ZOOM=6 postserve ... to avoid accidental slow low-zoom
tile generation.
- clone openmaptiles repo (
openmaptiles-toolsrepo is not needed with docker) - get a PostgreSQL server running with the openmaptiles-imported OSM data, e.g. by following quickstart guide.
- run
docker pull openmaptiles/openmaptiles-toolsto download the latest tools version - from inside the openmaptiles repo dir, run this command.
(This assumes PostgreSQL is on the localhost:5432, but if it runs inside docker, you may want to change
--net=hostto--net=openmaptiles_postgres_connto match the openmaptiles quickstart, and also expose port 8090 to the host with-p 8090:8090)
docker run -it --rm -u $(id -u ${USER}):$(id -g ${USER}) \
-v "${PWD}:/tileset" --net=host \
openmaptiles/openmaptiles-tools \
postserve openmaptiles.yaml
You can view tiles with any MVT-supporting viewer, such as:
- Maputnik editor (online) -- change the data source to
http://localhost:8090 - QGIS desktop -- add
Vector Tiles Readerplugin, and add a vector tile server connection with TileJSON URL set tohttp://localhost:8090.
Uses tileset definition to create a PostgreSQL
prepared or
create function SQL code
to generate an entire vector tile in the Mapbox Vector Tile format with a single getTile(z,x,y) query
using PostGIS MVT support.
Use --help to get all parameters.
NOTE: Current openmaptiles/postgis image (v2.9 and before) has incorrect support for the ST_AsMVT(). Until Postgis is updated, please use sophox/postgis docker image (based on the latest mdillon/postgis:11 base image). Another known bug is PostgreSQL JIT could make tile generation horribly slow in PG11+, and may need to be disabled.
generate-sqltomvt <tileset>
Takes a tileset definition an generates an imposm3 mapping file for importing OSM data.
generate-imposm3 <tileset>
Assembles all SQL referenced in the layer definitions into an SQL script that can be executed with psql.
If --dir option is given, generates .sql files that can be executed in parallel.
generate-sql <tileset>
generate-sql <tileset> --dir <outputdir>
Takes a tileset definition and generates Markdown documentation.
generate-doc <tileset>
dependency: graphviz
Takes a source code from the imposm3 mapping file and the SQL postprocessing code,
and parsing for the etldoc: graphviz based comments, and generate an svg file.
The .dot and the .svg filename prefix is etl_
generate-etlgraph <tileset> <target-directory>
generate-etlgraph layers/landcover/landcover.yaml ./build/devdoc
generate-etlgraph layers/railway/railway.yaml ./build/etlgraph
example:
input command: generate-etlgraph layers/landcover/landcover.yaml
output fies:
layers/landcover/etl_landcover.dotlayers/landcover/etl_landcover.svg
example:
generate-sqlquery layers/landcover/landcover.yaml 14
dependency: sqlite3
example:
generate_metadata ./data/tiles.mbtiles
Takes a tileset definition and generates a TM2Source YAML project file. You need to provide PostgreSQL database connection settings before generating the project.
generate-tm2source <tileset> --host="localhost" --port=5432 --database="osm" --user="osm" --password="osm"
The import-sql script can execute a single SQL file in Postgres when the file is given as the first parameter.
If ran without any arguments, import-sql executes all of the following:
- SQL files from
$OMT_UTIL_DIR- by default contains the sql/language.sql script. - SQL files from
$VT_UTIL_DIR- by default contains Mapbox's postgis-vt-util.sql helper functions. - SQL files from
$SQL_DIR- defaults to/sql-- this volume is empty initially, but should contain build results of running other generation scripts. If this directory containsparallel/subdirectory,import-sqlwill assume the parallel/*.sql files are safe to execute in parallel, up toMAX_PARALLEL_PSQLat a time (defaults to 5). The script will also executerun_first.sqlbefore, andrun_last.sqlafter the files inparallel/dir (if they exist).
Generating and importing SQL could be done in a single step with &&, e.g.
generate-sqltomvt openmaptiles.yaml > "$SQL_DIR/mvt.sql" && import-sqlOptionally you may pass extra arguments to psql by using PSQL_OPTIONS environment variable. For example PSQL_OPTIONS=-a makes psql echo all commands read from a file into stdout.
PSQL_OPTIONS allows multiple arguments as well, and understands quotes, e.g. you can pass a whole query as a single argument surrounded by quotes -- PSQL_OPTIONS="-a -c 'SELECT ...'"
Most PostgreSQL-related images support standard PostgreSQL environment variables like PGUSER, PGPASSWORD, PGHOST, PGDATABASE, and optionally PGPORT.
The only exception is the postgis image, which uses different variables (POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_HOST, POSTGRES_DB, POSTGRES_PORT) during database creation due to how the official Docker postgres image has been set up.
Materialized views can be refreshed in parallel using refresh-views command. This could be especially useful if the CREATE MATERIALIZED VIEW statements had WITH NO DATA clause.