Skip to content

Commit

Permalink
docs: update project structure in docs (#1404)
Browse files Browse the repository at this point in the history 
* docs: update project structure in docs

* docs: mention domain and liquibase

* docs: added c4

* docs: updated toc

* docs: pr feedback

---------

Co-authored-by: Yalz <[email protected]>
  • Loading branch information
@Yalz
Yalz and Yalz authored Oct 25, 2024
1 parent 52f932d commit fffc2c9
Show file tree
Hide file tree
Showing 5 changed files with 163 additions and 182 deletions.
225 changes: 48 additions & 177 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,56 +11,69 @@ open data.
## Table Of Contents

- [LDES Server](#ldes-server)
* [Table Of Contents](#table-of-contents)
* [Set-up of the LDES Server](#set-up-of-the-ldes-server)
* [How To Run](#how-to-run)
+ [Locally](#locally)
- [Maven](#maven)
- [Profiles](#profiles)
+ [Docker Setup](#docker-setup)
- [Docker-compose](#docker-compose)
- [The Config Files](#the-config-files)
- [The docker compose File](#the-docker-compose-file)
- [Starting the Dockerized Application](#starting-the-dockerized-application)
* [Developer Information](#developer-information)
+ [How To Build](#how-to-build)
+ [How To Test and View Coverage](#how-to-test-and-view-coverage)
- [Table Of Contents](#table-of-contents)
- [Structure of the application](#structure-of-the-application)
- [How To Run](#how-to-run)
- [Maven](#maven)
- [Developer Information](#developer-information)
- [How To Build](#how-to-build)
- [How To Test and View Coverage](#how-to-test-and-view-coverage)
- [Unit and Integration Tests](#unit-and-integration-tests)
- [Only Unit Tests](#only-unit-tests)
- [Only Integration Tests](#only-integration-tests)
- [Auto-Configurable Modules](#auto-configurable-modules)
+ [Tracing and Metrics](#tracing-and-metrics)
- [Tracing and Metrics](#tracing-and-metrics)
- [Local Tracing and Metrics](#local-tracing-and-metrics)
- [Pyroscope instrumentation](#pyroscope-instrumentation)
- [Using Docker](#using-docker)
+ [Health and Info](#health-and-info)
- [Health and Info](#health-and-info)
- [Local Health and Info](#local-health-and-info)
- [Docker](#docker)
+ [Logging](#logging)
- [Exposing of details](#exposing-of-details)
- [Logging](#logging)
- [Logging configuration](#logging-configuration)

## Set-up of the LDES Server
## Structure of the application

The current implementation consists of the following modules:
LDES Server Dependency Graph With All Profiles Activated:
![LDES Server Dependency Graph With All Profiles Activated](./content/ldes-server-graph.png)
Above is a dependency graph of the LDES server with all the profiles activated.
The graph is color-coded as follows:
- **Orange**: Main functionalities
- **Blue**: Interfaces
- **Lavender**: Plugin fragmentations

- `ldes-server-application` which starts the spring boot application
- `ldes-server-domain` which contains the domain logic of the ldes server
- `ldes-server-infra-mongo` which allows to store ldes members and fragments in a mongoDB
- `ldes-server-port-ingest` which allows to ingest ldes members
- `ldes-server-port-fetch-rest` which allows to retrieve fragments via HTTP
- `ldes-fragmentisers` which support different types of fragmentations
The `ldes-server-domain` is the core domain module of the LDES server and is always loaded in.
To keep the graph clean, it is not shown.
Next to that, the `postgres-liquibase` module will always be loaded in to assure correct database creation.

The modules `ldes-server-infra-mongo` is built so
that it can be replaced by other implementations without the need for code changes in ldes-server-domain.

## How To Run
The LDES server is built using the Spring Boot framework and is structured as a multi-module Maven project.
Each maven profile represents a different functionality of the LDES server that can be toggled.
The default exported image contains all the profiles, but a custom image can be created with only the needed
dependencies.

| Profile | Dependencies | Description |
|----------------------------|--------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|
| `fragmentation` | `postgres-pagination-repository` | Allows basic fragmentation (pagination) |
| `maintenance` | `postgres-maintenance-repository` | Allows the LDES server to perform maintenance operations on its Event Streams and Members: compaction, retention, deletion. |
| **Interfaces** | | |
| `http-admin` | `ldes-server-admin-rest`,`postgres-admin-repository`* | Gives access to REST API to create and manage Event Streams and Views. |
| `http-ingest` | `ldes-server-ingest-rest`,`postgres-ingest-repository` | Gives access to REST API to ingest members into the LDES. |
| `http-fetch` | `ldes-server-fetch-rest`,`postgres-fetch-repository` | Gives access to REST API to fetch Event Streams, its Views and pages. |
| **Plugin Fragmentations** | | |
| `fragmentation-timebased` | `ldes-server-fragmentation-timebased-hierarchical` | Allows fragmentation in based on a timebased property. |
| `fragmentation-geospatial` | `ldes-server-fragmentation-geospatial` | Allows fragmentation in based on a geospatial property. |
| `fragmentation-reference` | `ldes-server-fragmentation-reference` | Allows fragmentation in based on a textual property. |


We'll show you how to set up your own LDES server both locally and via Docker using a mongo db for storage and a rest
endpoint for ingestion and fetch.
Afterwards, you can change storage, ingestion and fetch options by plugging in other components.
*: The `postgres-admin-repository`, as shown by the dependency graph, will be loaded in by the other above-mentioned functionality profiles.
But when used separately, it needs to be loaded in manually.

### Locally
## How To Run

To run the LDES server, we refer to the versioned documentation available [here](https://informatievlaanderen.github.io/VSDS-LDESServer4J/).

#### Maven
### Maven

To locally run the LDES server in Maven, move to the `ldes-server-application` directory and run the Spring Boot
application.
Expand All @@ -80,139 +93,7 @@ cd ldes-server-application
mvn spring-boot:run -P{profiles (comma separated with no spaces) }
```

for example:

```mvn
mvn spring-boot:run -P{fragmentation-pagination,http-fetch,http-ingest,queue-none,storage-mongo}
```

To enrich the server, certain Maven profiles can be activated:

#### Profiles

| Profile Group | Profile Name | Description | Parameters | Further Info |
|------------------------------------------|--------------------------------------|---------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------|
| **HTTP Endpoints (Fetch/Ingestion)** | http-ingest | Enables a HTTP endpoint to insert LDES members. | [HTTP configuration](#example-http-ingest-fetch-configuration) | Endpoint:<br><br>- URL: /{ldes.collection-name}<br>- Request type: POST<br>- Accept: "application/n-quads", "application/n-triples" |
| **HTTP Endpoints (Fetch/Ingestion)** | http-fetch | Enables a HTTP endpoint to retrieve LDES fragments | [Example Views Configuration](#example-views-configuration) | Endpoint:<br>- URL: /{views.name}<br><br>- Request type: GET<br>- Accept: "application/n-quads", "application/ld+json" |
| **Storage** | storage-mongo | Allows the LDES server to read and write from a mongo database. | [Mongo configuration](#example-mongo-configuration) | |
| **Timebased Fragmentation[DEPRECATED]** | fragmentation-timebased | Supports timebased fragmentation. | [Timebased fragmentation configuration](#example-timebased-fragmentation) | |
| **Geospatial Fragmentation** | fragmentation-geospatial | Supports geospatial fragmentation. | [Geospatial fragmentation configuration](#example-geospatial-fragmentation) | |
| **Pagination Fragmentation** | fragmentation-pagination | Supports pagination. | [Pagination configuration](#example-pagination) | The pagenumbers start with pagenumber 1 |
| **Hierarchical Timebased Fragmentation** | fragmentation-timebased-hierarchical | Supports hierarchical timebased fragmentation. | [Timebased fragmentation configuration](#example-hierarchical-timebased-fragmentation) | |
| **Ldes-queues** | queue-none | Members are fragmented immediately. | N/A activating the profile is enough | |
| **Ldes-queues** | queue-in-memory | Members are queued in memory before fragmentation. | N/A activating the profile is enough | |
| **HTTP Endpoints (Admin)** | http-admin | Enables HTTP endpoints. These will be used later to configure different streams | | |
| **Instrumentation** | instrumentation | Enables pyroscope to collect data for instrumentation. | [Pyroscope configuration](#Pyroscope instrumentation) | |

The main functionalities of the server are ingesting and fetching, these profiles depend on other supporting profiles to
function properly:

- http-ingest: requires at least one queue, one fragmentation and one storage profile.
- http-fetch: requires at least one storage profile

#### Config

To run the server, some properties must be configured. The minimal config can be
found [here](ldes-server-application/examples/minimal-config-application.yml)

##### Overview server config

```yaml
ldes-server:
host-name: "http://localhost:8080"
use-relative-url: true
```
| Property | Description | Required | Default | Example | Supported values |
|:---------------------------|:--------------------------------------------------------------------------------------------------------------------|:---------|:---------------------|:----------------------|:--------------------------|
| host-name | The host name of the server, used as a prefix for resources hosted on the server. | Yes | N/A | http://localhost:8080 | HTTP and HTTPS urls |
| use-relative-url | Determines if the resources hosted on the server are constructed with a relative URI | No | false | true, false | true, false |
| max-jsonld-cache-capacity | A cache is used when fetching json-ld contexts. The number of cached contexts can be configured with this property. | No | 100 | 50 | Integer |
| admin.port | Determines the port on which the admin api will be available | No | value of server.port | 8080 | any available port number |
| fetch.port | Determines the port on which the fetch api will be available | No | value of server.port | 8080 | any available port number |
| ingest.port | Determines the port on which the ingest api will be available | No | value of server.port | 8080 | any available port number |
##### Example Serving Static Content
```yaml
spring:
mvc:
static-path-pattern: { pattern used in url for static content, e.g. /content/** for serving on http://localhost:8080/content/ }
web:
resources:
static-locations: { source folder of static content, e.g. file:/opt/files }
```
##### Example Serving DCAT Metadata
Supported file formats: .ttl, .rdf, .nq and .jsonld
Templates for configuring the DCAT metadata can be found [here](templates/dcat)
A detailed explanation on how to manage and retrieve the DCAT metadata can be
found [here](ldes-server-admin/README.md#dcat-endpoints).
##### Relative urls
To enable relative urls on the server, set the following property:
```yaml
ldes-server:
use-relative-url: true
```
When fetching any page using relative urls, any resource hosted on the server will have a URI relative to the requested
page.
> **Note**: When using relative urls, GET requests can not be performed with N-quads or triples as accept-type.
> This is because these types don't support resolving the relative URI's
##### Port bindings
To change the ports of the diffrent API's, use the following config:
```yaml
ldes-server:
admin:
port: 8080
fetch:
port: 8081
ingest:
port: 8082
```
Any combination of these ports can be the same of completely omitted.
When no port number is given, the default server port will be used.
### Docker Setup
#### Docker-compose
There are 2 files where you can configure the dockerized application:
- [The config files](#the-config-files)
- [The docker compose file](#the-docker-compose-file)
#### The Config Files
Runtime settings can be defined in the configuration files. Use [config.env](docker-compose/config.env) for a public
setup and be sure not to commit this file, as it contains secrets. For a local setup,
use [config.local.env](docker-compose/config.local.env).
#### The docker compose File
Change the `env_file` to `config.env` or `config.local.env` in [compose.yml](docker-compose.yml) according to
your needs.

#### Starting the Dockerized Application

Run the following commands to start the containers:

```bash
COMPOSE_DOCKER_CLI_BUILD=1 docker-compose build
docker-compose up
```

> **Note**: Using Docker Desktop might fail because of incorrect environment variable interpretation.
>
> ```unexpected character "-" in variable name near ...```
The needed profiles can be found in the Structure of the application section.

## Developer Information

Expand Down Expand Up @@ -256,16 +137,6 @@ For running the integration tests of the project, execute the following command
mvn clean verify -Dunittestskip=true
```

#### Auto-Configurable Modules
- ldes-server-infra-mongo
- ldes-server-port-ingest
- ldes-server-port-publication-rest
- ldes-fragmentisers-timebased-hierarchical
- ldes-fragmentisers-geospatial
- ldes-fragmentisers-pagination
- ldes-fragmentisers-reference
### Tracing and Metrics

Additionally, it is possible to keep track of metrics and tracings of the LDES Server.
Expand Down
Loading

0 comments on commit fffc2c9

Please sign in to comment.