Update READMEs with some development tips.

 Add an ingest DAG (I always use this for development so I feel it could be useful for others).

README.md

@@ -4,8 +4,16 @@ Loculus is a software package to power microbial genomial databases.
 
 ### [Visit the Loculus documentation website](https://loculus-project.github.io/loculus/)
 
+## Development
+
 Additional documentation for development is available in each folder's README. This file contains a high-level overview of the project and shared development information that is best kept in one place.
 
+If you would like to develop loculus locally you need to first:
+
+1. Deploy a local kubernetes instance: [kubernetes](/kubernetes/README.md)
+2. Deploy the backend: [backend](/backend/README.md)
+3. Deploy the frontend/website: [website](/website/README.md)
+
 ## Architecture
 
 - Backend code is in `backend`, see [`backend/README.md`](/backend/README.md)
@@ -27,7 +35,6 @@ While the documentation is still a work in progress, a look at the [`.github/wor
 - [`website.yml`](/.github/workflows/website.yml) runs the website tests and builds the website docker image
 - [`e2e-k3d.yml`](/.github/workflows/e2e-k3d.yml) runs the end-to-end tests
 
-
 ## Authorization
 
 ### User management

backend/README.md

@@ -6,8 +6,8 @@ All commands mentioned in this section are run from the `backend` directory unle
 
 ### Prerequisites
 
-* Java 21 installed on your system
-* A running PostgreSQL database (e.g. via a local [Kubernetes deployment](../kubernetes/README.md))])
+- Java 21 installed on your system
+- A running PostgreSQL database (e.g. via a local [Kubernetes deployment](../kubernetes/README.md))])
 
 ### Starting the backend
 
@@ -24,18 +24,24 @@ The backend is configured via
 [Spring properties](https://docs.spring.io/spring-boot/docs/current/reference/html/features.html#features.external-config)
 that need to be passed on startup, e.g. via command line argument.
 You need to set:
-* the database URL, username and password:
+
+- the database URL, username and password:
+
 ```
 --spring.datasource.url=jdbc:postgresql://localhost:5432/loculus
 --spring.datasource.username=postgres
 --spring.datasource.password=unsecure
 ```
-* the path to the config file (use `../generate_local_test_config.sh` to generate this file):
+
+- the path to the config file (use `../generate_local_test_config.sh` to generate this file):
+
 ```
 --loculus.config.path=../website/tests/config/backend_config.json
 ```
-* the url to fetch the public key for JWT verification 
+
+- the url to fetch the public key for JWT verification
   (corresponding to the `jwks_uri` value in the `/.well-known/openid-configuration` endpoint of the Keycloak server):
+
 ```
 --spring.security.oauth2.resourceserver.jwt.jwk-set-uri=http://localhost:8083/realms/loculus/protocol/openid-connect/certs
 ```
@@ -44,13 +50,15 @@ We use Flyway, so that the service can provision an empty/existing DB without an
 
 The service listens, by default, to **port 8079**: <http://localhost:8079/swagger-ui/index.html>.
 
+Note: When using a postgresSQL development platform (e.g. pgAdmin) the hostname is 127.0.0.1 and not localhost - this is defined in the `deploy.py` file.
+
 ### Operating the backend behind a proxy
 
 When running the backend behind a proxy, the proxy needs to set X-Forwarded headers:
 
-* X-Forwarded-For
-* X-Forwarded-Proto
-* X-Forwarded-Prefix
+- X-Forwarded-For
+- X-Forwarded-Proto
+- X-Forwarded-Prefix
 
 ## Development
 
@@ -98,7 +106,7 @@ The backend provides a Swagger UI at <http://localhost:8079/swagger-ui/index.htm
 We use Swagger to document the API and to provide a playground for testing the API.
 Especially for manual testing during the development, this is very useful.
 
-OpenAPI does not deal well with NDJSON. 
+OpenAPI does not deal well with NDJSON.
 Since the API has endpoints that deal with NDJSON, the documentation of those endpoints is to be understood as
 "the provided schema is a valid JSON schema for each line of the NDJSON file".
 

ingest/README.md

@@ -4,6 +4,14 @@
 
 The following is a rough overview of the pipeline's steps.
 
+The snakemake DAG can be visualized using [graphviz](https://en.wikipedia.org/wiki/Graphviz):
+
+```bash
+snakemake --dag  | dot -Tpng > dag.png
+```
+
+![snakemake DAG](dag.png)
+
 ### Download data from NCBI virus
 
 Using NCBI `datasets` CLI, download all sequences and corresponding NCBI curated metadata for a configurable taxon. The taxon is specified using the NCBI Taxonomy ID, and includes all child taxa, i.e. dowloading sequences for the Ebola virus taxon ID includes all sequences for more specific Ebola virus (sub)species taxon ids.

kubernetes/README.md

@@ -32,6 +32,7 @@ helm install loculus kubernetes/loculus -f my-values.yaml
 
 Install [k3d](https://k3d.io/v5.6.0/) and [helm](https://helm.sh/).
 
+We deploy kubernetes via the `../deploy.py` script, this requires you to have `pyyaml` and `requests` installed. We recommend following the steps in [ingest](/ingest/README.md) to activate the loculus-ingest mamba environment which includes these packages, but the packages can also be installed individually.
 
 ### Setup for local development