Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(docs): Update READMEs with some development tips and a DAG #1817

Merged
merged 5 commits into from
May 28, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,16 @@ Loculus is a software package to power microbial genomial databases.

### [Visit the Loculus documentation website](https://loculus-project.github.io/loculus/)

## Development

Additional documentation for development is available in each folder's README. This file contains a high-level overview of the project and shared development information that is best kept in one place.

If you would like to develop with a full local loculus instance you need to first:

1. Deploy a local kubernetes instance: [kubernetes](/kubernetes/README.md)
2. Deploy the backend: [backend](/backend/README.md)
3. Deploy the frontend/website: [website](/website/README.md)

## Architecture

- Backend code is in `backend`, see [`backend/README.md`](/backend/README.md)
Expand All @@ -27,7 +35,6 @@ While the documentation is still a work in progress, a look at the [`.github/wor
- [`website.yml`](/.github/workflows/website.yml) runs the website tests and builds the website docker image
- [`e2e-k3d.yml`](/.github/workflows/e2e-k3d.yml) runs the end-to-end tests


## Authorization

### User management
Expand Down
26 changes: 17 additions & 9 deletions backend/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ All commands mentioned in this section are run from the `backend` directory unle

### Prerequisites

* Java 21 installed on your system
* A running PostgreSQL database (e.g. via a local [Kubernetes deployment](../kubernetes/README.md))])
- Java 21 installed on your system
- A running PostgreSQL database (e.g. via a local [Kubernetes deployment](../kubernetes/README.md))])

### Starting the backend

Expand All @@ -24,18 +24,24 @@ The backend is configured via
[Spring properties](https://docs.spring.io/spring-boot/docs/current/reference/html/features.html#features.external-config)
that need to be passed on startup, e.g. via command line argument.
You need to set:
* the database URL, username and password:

- the database URL, username and password:

```
--spring.datasource.url=jdbc:postgresql://localhost:5432/loculus
--spring.datasource.username=postgres
--spring.datasource.password=unsecure
```
* the path to the config file (use `../generate_local_test_config.sh` to generate this file):

- the path to the config file (use `../generate_local_test_config.sh` to generate this file):

```
--loculus.config.path=../website/tests/config/backend_config.json
```
* the url to fetch the public key for JWT verification

- the url to fetch the public key for JWT verification
(corresponding to the `jwks_uri` value in the `/.well-known/openid-configuration` endpoint of the Keycloak server):

```
--spring.security.oauth2.resourceserver.jwt.jwk-set-uri=http://localhost:8083/realms/loculus/protocol/openid-connect/certs
```
Expand All @@ -44,13 +50,15 @@ We use Flyway, so that the service can provision an empty/existing DB without an

The service listens, by default, to **port 8079**: <http://localhost:8079/swagger-ui/index.html>.

Note: When using a postgresSQL development platform (e.g. pgAdmin) the hostname is 127.0.0.1 and not localhost - this is defined in the `deploy.py` file.

### Operating the backend behind a proxy

When running the backend behind a proxy, the proxy needs to set X-Forwarded headers:

* X-Forwarded-For
* X-Forwarded-Proto
* X-Forwarded-Prefix
- X-Forwarded-For
- X-Forwarded-Proto
- X-Forwarded-Prefix

## Development

Expand Down Expand Up @@ -98,7 +106,7 @@ The backend provides a Swagger UI at <http://localhost:8079/swagger-ui/index.htm
We use Swagger to document the API and to provide a playground for testing the API.
Especially for manual testing during the development, this is very useful.

OpenAPI does not deal well with NDJSON.
OpenAPI does not deal well with NDJSON.
Since the API has endpoints that deal with NDJSON, the documentation of those endpoints is to be understood as
"the provided schema is a valid JSON schema for each line of the NDJSON file".

Expand Down
8 changes: 8 additions & 0 deletions ingest/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,14 @@

The following is a rough overview of the pipeline's steps.

The snakemake DAG can be visualized using [graphviz](https://en.wikipedia.org/wiki/Graphviz):

```bash
snakemake --dag | dot -Tpng > dag.png
```

![snakemake DAG](dag.png)
anna-parker marked this conversation as resolved.
Show resolved Hide resolved

### Download data from NCBI virus

Using NCBI `datasets` CLI, download all sequences and corresponding NCBI curated metadata for a configurable taxon. The taxon is specified using the NCBI Taxonomy ID, and includes all child taxa, i.e. dowloading sequences for the Ebola virus taxon ID includes all sequences for more specific Ebola virus (sub)species taxon ids.
Expand Down
Binary file added ingest/dag.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions kubernetes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ helm install loculus kubernetes/loculus -f my-values.yaml

Install [k3d](https://k3d.io/v5.6.0/) and [helm](https://helm.sh/).

We deploy to kubernetes via the `../deploy.py` script. It requires you to have `pyyaml` and `requests` installed.

### Setup for local development

Expand Down
Loading