Skip to content

Commit

Permalink
refactored the development docker-compose.yml into a separate file
Browse files Browse the repository at this point in the history
  • Loading branch information
CommanderStorm committed Mar 18, 2024
1 parent 369866b commit 71da442
Show file tree
Hide file tree
Showing 4 changed files with 179 additions and 30 deletions.
25 changes: 13 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@
[![API Uptime over 30 days](https://nav-monitoring.mm.rbg.tum.de/api/badge/2/uptime/720?label=API%20Uptime/30&labelSuffix=d)](https://uptime.nav.tum.sexy/status/navigatum)
[![CDN Uptime over 30 days](https://nav-monitoring.mm.rbg.tum.de/api/badge/1/uptime/720?label=CDN%20Uptime/30&labelSuffix=d)](https://uptime.nav.tum.sexy/status/navigatum)

NavigaTUM is a tool developed by students for students, to help you get around at [TUM](https://tum.de). Feel free to contribute, we are open to new people 😄.
NavigaTUM is a tool developed by students for students, to help you get around at [TUM](https://tum.de).
Feel free to contribute, we are open to new people 😄.

Features:

Expand All @@ -30,7 +31,9 @@ You can consume our API Documentation in two ways:
- We also describe our API in an [OpenAPI 3.0](https://de.wikipedia.org/wiki/OpenAPI) compliant file.
You can find it [here](./openapi.yaml).
Using this Specification you can generate your own client to access the API in the language of your choice.
To do this head over to the [Swagger Editor](https://editor.swagger.io/?url=https://raw.githubusercontent.com/TUM-Dev/navigatum/main/openapi.yaml) or other similar [OpenAPI tools](https://openapi.tools/).
To do this head over to
the [Swagger Editor](https://editor.swagger.io/?url=https://raw.githubusercontent.com/TUM-Dev/navigatum/main/openapi.yaml)
or other similar [OpenAPI tools](https://openapi.tools/).

Note: The API is still under development, and we are open to Issues, Feature Requests or Pull Requests.

Expand All @@ -55,12 +58,6 @@ git clone https://github.com/TUM-Dev/Navigatum.git
cd Navigatum
```

> [!NOTE]
> You can skip all the parts if you run `docker compose up --build` in the root of the repository
> and then open [http://localhost:3000](http://localhost:3000) in your browser.
> This will run the server, the data processing and the webclient in docker containers.
> Initial setup will take a while, as it needs to download maps data for the entire planet (~90GB).
### Data Processing

In case you do not want to work on the data processing, you can instead
Expand All @@ -75,7 +72,9 @@ Else you can follow the steps in the [data documentation](data/README.md).

### Server

If you want to work on the webclient only (and not server or data), you don't need to set up the server. You can instead either use the public API (see the [webclient documentation](webclient/README.md#Testing)) or use our ready-made docker images to run the server locally:
If you want to work on the webclient only (and not server or data), you don't need to set up the server.
You can instead either use the public API (see the [webclient documentation](webclient/README.md#Testing)) or use our
ready-made docker images to run the server locally:

> [!NOTE]
> Because we have a tileserver, running the following command is required once.
Expand All @@ -85,21 +84,23 @@ If you want to work on the webclient only (and not server or data), you don't ne
> ```
```bash
docker compose up --build
docker compose -f docker-compose.dev.yml up --build
```
Else you can follow the steps in the [server documentation](server/README.md).

### Webclient

Follow the steps in the [webclient documentation](webclient/README.md).
If you want to only run the webclient locally, you can skip the "Data" and "Server" steps above and use docker (as seen above) or you can [edit the webclient configuration](webclient/README.md#testing) to point to production.
If you want to only run the webclient locally, you can skip the "Data" and "Server" steps above and use docker (as seen
above) or you can [edit the webclient configuration](webclient/README.md#testing) to point to production.

### Formatting

We have multiple programming languages in this repository, and we use different tools to format them.

since we use [pre-commit](https://pre-commit.com/) to format our code, you can install it in an virtual environment with:
since we use [pre-commit](https://pre-commit.com/) to format our code, you can install it in an virtual environment
with:

```bash
python3 -m venv venv
Expand Down
146 changes: 146 additions & 0 deletions docker-compose.dev.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
services:
webclient:
restart: unless-stopped
build: ./webclient
ports:
- "3000:3000"
environment:
TILESERVER_URL: http://tileserver:8080
CDN_URL: http://data:3002
MAIN_API_URL: http://main-api:3003
FEEDBACK_API_URL: http://feedback-api:3004
depends_on:
tileserver:
condition: service_healthy
data:
condition: service_healthy
main-api:
condition: service_healthy
feedback-api:
condition: service_healthy
# maps
tileserver-init-sprites:
image: alpine:latest
command: sh -c "mkdir -p /data/sprites/ && rm -f /data/sprites/* && wget -P /data/sprites ${TILE_SPRITES_URL}/osm-liberty.json ${TILE_SPRITES_URL}/[email protected] ${TILE_SPRITES_URL}/osm-liberty.png ${TILE_SPRITES_URL}/[email protected]"
volumes:
- tileserver-src:/data
tileserver-init-config:
image: alpine:latest
command: sh -c "apk update --no-progress --quiet && apk add rsync --no-progress --quiet && rsync /to_data/ /data/"
volumes:
- tileserver-src:/data
- ./deployment/k3s/files/:/to_data/:ro
depends_on:
tileserver-init-sprites:
condition: service_completed_successfully
tileserver-srv-src: # needed for simpler initialisation of dev environments
image: nginx:1.25
restart: on-failure
volumes:
- tileserver-src:/usr/share/nginx/html/maps/vol:ro
depends_on:
tileserver-init-config:
condition: service_completed_successfully
tileserver-init-sprites:
condition: service_completed_successfully
healthcheck:
test: service nginx status || exit 1
retries: 2
interval: 10s
start_period: 10s
tileserver:
image: maptiler/tileserver-gl:latest
restart: unless-stopped
command: /usr/src/app/docker-entrypoint.sh --public_url=https://nav.tum.de/maps/
tmpfs:
- /tmp
volumes:
- tileserver-src:/data/:ro
read_only: true
ports:
- "8080:8080"
depends_on:
tileserver-init-config:
condition: service_completed_successfully
tileserver-init-sprites:
condition: service_completed_successfully
# cdn
data:
restart: unless-stopped
build: ./data
ports:
- "3002:3002"
# server
main-api:
restart: unless-stopped
build: ./server
command: /bin/navigatum-main-api
ports:
- "3003:3003"
volumes:
- type: tmpfs
target: /home/navigatum/.cache
user: 1000:3000
environment:
MIELI_URL: http://meilisearch:7700
TILESERVER_URL: http://tileserver:8080
CDN_URL: http://data:3002/cdn
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_DB: ${POSTGRES_DB}
POSTGRES_URL: db:5432
depends_on:
meilisearch:
condition: service_healthy
db:
condition: service_healthy
healthcheck:
test: wget -q --spider http://localhost:3003/api/status
retries: 5
interval: 10s
start_period: 60s
meilisearch:
image: getmeili/meilisearch:v1.7.1
restart: unless-stopped
ports:
- "7700:7700"
healthcheck:
test: wget -q --spider http://localhost:7700/health
retries: 5
interval: 10s
start_period: 10s
feedback-api:
restart: unless-stopped
build: ./server
command: /bin/navigatum-feedback
ports:
- "3004:3004"
healthcheck:
test: wget -q --spider http://localhost:3004/api/feedback/status
retries: 5
start_period: 10s
db:
image: postgres:16
restart: unless-stopped
environment:
PGDATA: /var/lib/postgresql/data/pgdata
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_DB: ${POSTGRES_DB}
volumes:
- postgres-data:/var/lib/postgresql/data
ports:
- "5432:5432"
healthcheck:
test: [ "CMD", "pg_isready", "-U", "${POSTGRES_USER}" ]
retries: 5
interval: 10s
start_period: 10s

volumes:
db-data:
driver: local
tileserver-src:
driver: local
postgres-data:
driver: local
20 changes: 8 additions & 12 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
services:
webclient:
image: ghcr.io/tum-dev/navigatum-webclient:main
restart: always
build: ./webclient
restart: unless-stopped
labels:
- "traefik.enable=true"
- "traefik.http.routers.navigatum-webclient.entrypoints=webs"
Expand Down Expand Up @@ -42,7 +41,7 @@ services:
condition: service_completed_successfully
tileserver-srv-src: # needed for simpler initialisation of dev environments
image: nginx:1.25
restart: on-failure
restart: unless-stopped
labels:
- "traefik.enable=true"
- "traefik.http.routers.navigatum-tileserver-srv.entrypoints=webs"
Expand All @@ -63,7 +62,7 @@ services:
start_period: 10s
tileserver:
image: maptiler/tileserver-gl:latest
restart: always
restart: unless-stopped
labels:
- "traefik.enable=true"
- "traefik.http.routers.navigatum-tileserver.entrypoints=webs"
Expand All @@ -88,27 +87,25 @@ services:
# cdn
data:
image: ghcr.io/tum-dev/navigatum-data:main
restart: always
restart: unless-stopped
labels:
- "traefik.enable=true"
- "traefik.http.routers.navigatum-data.entrypoints=webs"
- "traefik.http.routers.navigatum-data.tls.certresolver=leacme"
- "traefik.http.routers.navigatum-data.rule=Host(`nav.tum.de`) && PathPrefix(`/cdn/`)"
- "traefik.http.services.navigatum-data.loadbalancer.server.port=3002"
build: ./data
expose:
- "3002"
# server
main-api:
image: ghcr.io/tum-dev/navigatum-server:main
restart: always
restart: unless-stopped
labels:
- "traefik.enable=true"
- "traefik.http.routers.navigatum-main-api.entrypoints=webs"
- "traefik.http.routers.navigatum-main-api.tls.certresolver=leacme"
- "traefik.http.routers.navigatum-main-api.rule=Host(`nav.tum.de`) && PathPrefix(`/api/`)"
- "traefik.http.services.navigatum-main-api.loadbalancer.server.port=3003"
build: ./server
command: /bin/navigatum-main-api
expose:
- "3003"
Expand Down Expand Up @@ -136,7 +133,7 @@ services:
start_period: 60s
meilisearch:
image: getmeili/meilisearch:v1.7.1
restart: always
restart: unless-stopped
expose:
- "7700"
healthcheck:
Expand All @@ -146,14 +143,13 @@ services:
start_period: 10s
feedback-api:
image: ghcr.io/tum-dev/navigatum-server:main
restart: always
restart: unless-stopped
labels:
- "traefik.enable=true"
- "traefik.http.routers.navigatum-feedback-api.entrypoints=webs"
- "traefik.http.routers.navigatum-feedback-api.tls.certresolver=leacme"
- "traefik.http.routers.navigatum-feedback-api.rule=Host(`nav.tum.de`) && PathPrefix(`/api/feedback`)"
- "traefik.http.services.navigatum-feedback-api.loadbalancer.server.port=3004"
build: ./server
command: /bin/navigatum-feedback
expose:
- "3004"
Expand All @@ -180,7 +176,7 @@ services:
start_period: 10s
redirect:
image: traefik/whoami
restart: always
restart: unless-stopped
labels:
- "traefik.enable=true"
- "traefik.http.routers.navigatum-redirect-unsecure.entrypoints=web"
Expand Down
18 changes: 12 additions & 6 deletions map/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,21 +13,27 @@ A tileserver takes these two components and produces a variety of formats (png,

Sadly tilesets are really large (`germany` is ~10GB, `planet` ~90GB).
Because of limited badwith and storage space we can't provide a tileset for everyone.
You can generate your own tileset from [OpenStreetMap Data](https://osmdata.openstreetmap.de/) via [planettiler](https://github.com/onthegomap/planetiler) or other equivalent tools.
You can generate your own tileset from [OpenStreetMap Data](https://osmdata.openstreetmap.de/)
via [planettiler](https://github.com/onthegomap/planetiler) or other equivalent tools.

From our experience the best way to generate a tileset is to use [planettiler](https://github.com/onthegomap/planetiler), as their perofrmance is by far (other competitors are not even close by our tests) the best, and they can work in resourece constreained environments.
From our experience the best way to generate a tileset is to
use [planettiler](https://github.com/onthegomap/planetiler), as their perofrmance is by far (other competitors are not
even close by our tests) the best, and they can work in resourece constreained environments.

From a resource perspective, you need about 2x the size of the tileset as free space on your disk and above 10GB in free RAM.
From a resource perspective, you need about 2x the size of the tileset as free space on your disk and above 10GB in free
RAM.
If you really need a tileset and can't meet these requirements, shoot us a message, and we'll see what we can do.
Generating `europe` takes 3h10m on a modern laptop with 32GB RAM and an SSD. The following commands are optimised for this.
Generating `europe` takes 3h10m on a modern laptop with 32GB RAM and an SSD. The following commands are optimised for
this.

From the root of the repository, run:

```bash
docker run -it -e JAVA_TOOL_OPTIONS="-Xmx10g" -v "$(pwd)/map":/data ghcr.io/onthegomap/planetiler:latest --download --area=bavaria --languages=de,en --Xmx10g --storage=mmap
```

For `planet`, you might want to increase the `--Xmx` parameter to 20GB. For 128GB of RAM or more you will want to use `--storage=ram` instead of `--storage=mmap`.
For `planet`, you might want to increase the `--Xmx` parameter to 20GB. For 128GB of RAM or more you will want to
use `--storage=ram` instead of `--storage=mmap`.

### Serve the tileset

Expand All @@ -38,7 +44,7 @@ This may be one optimisation point in the future.
From the root of the repository, run:

```bash
docker compose up --build
docker compose -f docker-compose.dev.yml up --build
```

### Edit the style
Expand Down

0 comments on commit 71da442

Please sign in to comment.