Skip to content

Commit

Permalink
Merge branch 'main' into chore/data-update
Browse files Browse the repository at this point in the history
  • Loading branch information
CommanderStorm authored Nov 8, 2024
2 parents 86465f6 + 760400f commit 595d9bd
Show file tree
Hide file tree
Showing 11 changed files with 46 additions and 50 deletions.
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,13 +64,13 @@ cd Navigatum
### Data Processing

In case you do not want to work on the data processing, you can instead
download the latest compiled files via running the server.
download the latest compiled files by running the server.

Else you can follow the steps in the [data documentation](data/README.md).
Otherwise, you can follow the steps in the [data documentation](data/README.md).

### Server

If you want to work on the webclient only (and not server or data), you don't need to set up the server.
If you want to work only on the webclient (and not server or data), you don't need to set up the server.
You can instead either use the public API (see the [webclient documentation](webclient/README.md#Testing)) or use our
ready-made docker images to run the server locally:

Expand All @@ -90,7 +90,7 @@ docker compose -f docker-compose.local.yml up --build
> these can be resolved via upstream language improvements such as
> [polonius](https://blog.rust-lang.org/inside-rust/2023/10/06/polonius-update.html), [cranelift](https://github.com/rust-lang/rustc_codegen_cranelift), [paralell-frontend](https://blog.rust-lang.org/2023/11/09/parallel-rustc.html),....
Else you can follow the steps in the [server documentation](server/README.md).
Otherwise, you can follow the steps in the [server documentation](server/README.md).

### Webclient

Expand Down
4 changes: 2 additions & 2 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Reporting a Vulnerability

Please report any and all security vulnerabilitys you found, or you think you found at navigatum (at) tum.de .
Please report any and all security vulnerabilities you found, or you think you found at navigatum (at) tum.de .
We will diagnose the issue internally and propose a fix.
The timeline of a fix depends on how severe the problem is and what impacts it has.
As a reward for reporting such vulnerabilitys you can get exclusive stickers at the MPI student council office or other small things.
As a reward for reporting such vulnerabilities you can get exclusive stickers at the MPI student council office or other small things.
13 changes: 7 additions & 6 deletions data/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ This folder contains:
- Custom data inserted into the datasets
- Custom patches applied on the source data

The code to retrieve external data, as well as externally retrieved data is located under `external`.
The code to retrieve external data as well as externally retrieved data is located under `external`.

> [!WARNING]
> A lot of this code is more a work-in-progress than finished.
Expand All @@ -22,12 +22,12 @@ The code to retrieve external data, as well as externally retrieved data is loca
### Prerequisites

For getting started, there are some system dependencies which you will need.
Please follow the [system dependencies docs](/resources/documentation/Dependencys.md) before trying to run this part of
Please follow the [system dependencies docs](/resources/documentation/Dependencies.md) before trying to run this part of
our project.

### Dependencies

Since data needs some python dependency's, you will need to install them first.
Since data needs some python dependencies, you will need to install them first.
We recommend doing this in a virtual environment.

From the root of the project, run:
Expand Down Expand Up @@ -123,10 +123,11 @@ data

## Compilation process

The data compilation is made of indiviual processing steps, where each step adds new or modifies the current data. The
basic structure of the data however stays the same from the beginning on and is specified in `data-format_*.yaml`.
The data compilation is made of individual processing steps, where each step adds new or modifies the current data.
The final structure of the data, is specified in `data-format_*.yaml`.
Some work is underway to ensure that this format is actually being followed via simplifying the data backend and migrating the database server from managing a json blob to "real" tables. This is not done yet.

- **Step 00**: The first step reads the base root node, areas, buildings etc. from the
- **Step 00**: The first step reads the base root node, areas, buildings, etc. from the
`sources/00_areatree` file and creates an object collection (python dictionary)
with the data format as mentioned above.
- **Steps 01-29**: Within these steps, new rooms or POIs might be added, however no
Expand Down
12 changes: 6 additions & 6 deletions deployment/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Deployment

This Project details how to deploy NavigaTUM
This Project details how to deploy NavigaTUM.
The documentation for the specific sub-services can be found in the respective directories:

- [Data](../data/README.md)
Expand Down Expand Up @@ -30,7 +30,7 @@ The current physical infrastructure is as follows:
On each of the servers we run a docker cluster configured via ansible.
The deployment script is located at `TUM-Dev/d9s`.
We intentionally are not running kubernetes, as we never need to scale above 0.1 CPUs in active deployment.
We are confindent that even if we were to add more universitys, more load would not exceed the capacity of one server.
We are confindent that even if we were to add more universities, more load would not exceed the capacity of one server.

On every of these nodes we run the following supporting services:

Expand All @@ -41,10 +41,10 @@ On every of these nodes we run the following supporting services:

We run the following components at these paths:

- `/` [webclient](../webclient) written in [`Vue`]/[`nuxt-4`](https://nuxt.dev)
- `/api/` [server](../server) written in [`Rust`](https://nuxt.dev)/[`actix-web`](https://nuxt.dev)
- `/tiles/` [`marvin`] tiles from OSM
- `/map/` [`tileserver-gl`] rendering tiles from OSM
- `/` [webclient](../webclient) written in [`Vue`](https://vuejs.org/)/[`nuxt-4`](https://nuxt.dev)
- `/api/` [server](../server) written in [`Rust`](https://www.rust-lang.org/)/[`actix-web`](https://actix.rs/)
- `/tiles/` [`martin`](https://martin.maplibre.org/) tiles from OSM
- `/map/` [`tileserver-gl`](https://github.com/maptiler/tileserver-gl) rendering tiles from OSM
slated to be removed in favor of `/tiles/`
- `/cdn` nginx server serving the static files we need
slated to be folded into the `server`-startup
Expand Down
12 changes: 6 additions & 6 deletions map/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,14 @@ This folder contains the static maps tileserver and vector tiles server for Navi

## Getting started

As a basis of generating images it is important to have a tileset (`output.mbtiles`) and a stile.
As a basis of generating images it is important to have a tileset (`output.mbtiles`) and a style.
The style is a JSON file that defines how the map should look like.
The tileset is a sqlite database that contains the map data.
A tileserver takes these two components and produces a variety of formats ([MVT](https://github.com/mapbox/vector-tile-spec), png, webp, json, etc.) for the frontend.

### Edit the style

You cannot currently not preview the style in our tileserver martin (see [martin#1120](https://github.com/maplibre/martin/issues/1120)).
You cannot currently preview the style in our tileserver martin (see [martin#1120](https://github.com/maplibre/martin/issues/1120)).
Therefore, for editing the style we use [Maputnik](https://github.com/maputnik/editor).
It is a web-based editor for Maplibre styles.
You can use it to edit the style and see the changes live.
Expand All @@ -25,7 +25,7 @@ To run maputnik, you can either

Our style can be found here and can either be "Load[ed] from Url" or uploaded into maputnik manually:
```
https://github.com/TUM-Dev/NavigaTUM/raw/refs/heads/main/map/styles/navigatum-basemap.json
https://raw.githubusercontent.com/TUM-Dev/NavigaTUM/refs/heads/main/map/styles/navigatum-basemap.json
```

| Step 1 | Step 2 |
Expand All @@ -35,13 +35,13 @@ https://github.com/TUM-Dev/NavigaTUM/raw/refs/heads/main/map/styles/navigatum-ba
### generate your own tileset

Sadly tilesets are really large (`germany` is ~10GB, `planet` ~90GB).
Because of limited badwith and storage space we can't provide a tileset for everyone.
Because of limited bandwith and storage space we can't provide a tileset for everyone.
You can generate your own tileset from [OpenStreetMap Data](https://osmdata.openstreetmap.de/)
via [planettiler](https://github.com/onthegomap/planetiler) or other equivalent tools.

From our experience the best way to generate a tileset is to
use [planettiler](https://github.com/onthegomap/planetiler), as their perofrmance is by far (other competitors are not
even close by our tests) the best, and they can work in resourece constreained environments.
use [planettiler](https://github.com/onthegomap/planetiler), as their performance is by far (other competitors are not
even close by our tests) the best, and they can work in resource constrained environments.

From a resource perspective, you need about 2x the size of the tileset as free space on your disk and above 10GB in free
RAM.
Expand Down
2 changes: 1 addition & 1 deletion map/download.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ rm -rf ./gtfs_feeds/*
mkdir --parents gtfs_feeds
mkdir --parents data/transit_tiles

echo -- download dependencys --
echo -- download dependencies --
if command -v apk > /dev/null 2>&1
then
apk --update add --quiet wget
Expand Down
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
# Dependencys
# Dependencies

Our project has a few system-level-dependencies, which are generally usefully and a few which are only used for some
Our project has a few system-level-dependencies, which are generally useful and a few which are only used for some
parts of the project.
If you get stuck or have any questions, feel free to contact us. We are happy to help.

## General

### OS

We recommend using a Linux based OS, as we have not tested the project on Windows or Mac.
We recommend using a Linux-based OS, as we have not tested the project on Windows or Mac.
("There be dragons", but we will try to improve this part if you show us where we fail)
If you are using Windows, use [WSL](https://docs.microsoft.com/en-us/windows/wsl/install-win10) to run Linux on Windows.
If you are using Windows, use [WSL](https://learn.microsoft.com/en-us/windows/wsl/install) to run Linux on Windows.

Please make sure that your OS is up-to-date, before we start. (Trust me, this has fucked over multiple people...)
On Ubuntu this is as easy as running `sudo apt update && sudo apt upgrade`.
Expand Down Expand Up @@ -66,6 +66,7 @@ To install Rust, you can use [rustup](https://rustup.rs/).

The server uses OpenSSL to verify TLS certificates.

On Debian-based systems like Ubuntu, you can install it with:
```bash
sudo apt-get install build-essential pkg-config openssl libssl-dev
```
Expand All @@ -75,9 +76,9 @@ sudo apt-get install build-essential pkg-config openssl libssl-dev
#### NodeJS

We use NodeJS for the webclient.
Setting NodeJS up is a bit more complicated than setting up python/rust, but it is still pretty easy.
Setting up NodeJS is a bit more complicated than setting up python/rust, but it is still pretty easy.

- On linux, you can get it through your favorite package manager.
- On Linux, you can get it through your favorite package manager.
You normally should need to install `nodejs` and `pnpm`.
- On WSL, use [this guide](https://learn.microsoft.com/en-us/windows/dev-environment/javascript/nodejs-on-wsl)
and [this guide](https://pnpm.io/installation)
6 changes: 3 additions & 3 deletions server/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion server/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ ARG BUILDKIT_SBOM_SCAN_CONTEXT=true

FROM rust:1.82-alpine AS compiler

# to ache the build this line inludes all the dependencys all servers need
# to ache the build this line inludes all the dependencies all servers need
# this is not an issue since we copy the generated binary to a more minimal envornment
# Descriptions:
# - musl-dev is needed for musl to compile the binary
Expand Down
6 changes: 3 additions & 3 deletions server/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ This folder contains the main backend server for NavigaTUM.

### Prerequisites

For getting started, there are some system dependencys which you will need.
Please follow the [system dependencys docs](/resources/documentation/Dependencys.md) before trying to run this part of
For getting started, there are some system dependencies which you will need.
Please follow the [system dependencies docs](/resources/documentation/Dependencies.md) before trying to run this part of
our project.

### Additional dependency
Expand All @@ -21,7 +21,7 @@ endpoint is a non-essential part of the project.

#### How to Set up the Databases

At the beginning of the main api we set up both meilisearch and the database.
At the beginning of the main API we set up both meilisearch and the database.
This will ensure that the sqlite database and meilisearch index is created.

This requires meilisearch to be online.
Expand Down
18 changes: 6 additions & 12 deletions webclient/README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# Webclient

This folder contains the JavaScript based webclient for NavigaTUM.
This folder contains the JavaScript-based webclient for NavigaTUM.

## Getting started

### Prerequisites

For getting started, there are some system dependencies which you will need.
Please follow the [system dependencies docs](/resources/documentation/Dependencys.md) before trying to run this part of
Please follow the [system dependencies docs](/resources/documentation/Dependencies.md) before trying to run this part of
our project.

### Recommended IDE Setup
Expand All @@ -17,15 +17,9 @@ Most modern IDEs (like the PyCharm+RustRover+WebStorm combination) should work a

## Dependencies

### Prerequisites

For getting started, there are some system dependencys which you will need.
Please follow the [system dependencys docs](/resources/documentation/Dependencys.md) before trying to run this part of
our project.
### Installing Dependencies

### Installing Dependency's

```bash
```sh
pnpm install
```

Expand All @@ -36,11 +30,11 @@ By default, the webclient will connect to the server on `https://nav.tum.de`.
If you want to connect to a local version instead, change the environemnt
variable `NUXT_PUBLIC_{API,CDN,FEEDBACK,MAPS}_URL` to the appropriate value.

To get a local server running, please:
To get a local server running, you can do so:

- either via following the [guide to local development](../server/README.md), or
- via [docker](https://docs.docker.com/) by commenting out the webclient from the docker-compose-file and running
```bash
```sh
docker compose -f docker-compose.local.yml up --build
```

Expand Down

0 comments on commit 595d9bd

Please sign in to comment.