diff --git a/README.md b/README.md index 59a8223b..1dc0e4d1 100644 --- a/README.md +++ b/README.md @@ -2,17 +2,29 @@ [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/bzkf/diz-in-a-box/badge)](https://api.securityscorecards.dev/projects/github.com/bzkf/diz-in-a-box) -DIZ in a box. +This software is used to transform oncological basic data set (oBDS) XML files from tumor documentation systems to HL7® FHIR® and to a tabular format. + +## Modular Pipeline +![Figure Modular Pipeline](img/fig1.png) + +[publication to be cited here] + ## Installation -### Prerequisites +### Installation: Docker Compose Setup +Please follow along here: [docker-compose/README.md](docker-compose/README.md) + + +### Installation: kubernetes setup + +#### Prerequisites - [helm](https://github.com/helm/helm) -### Steps +#### Steps -#### Install K3S Cluster +##### Install K3S Cluster ```sh curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.26.1+k3s1 sh - @@ -74,7 +86,7 @@ chmod +x ./dist/air-gapped/bin/import-images-into-k3s.sh IMAGE_FOLDER=./dist/air-gapped/images ./dist/air-gapped/bin/import-images-into-k3s.sh ``` -#### Install Strimzi Operator and Kafka +##### Install Strimzi Operator and Kafka @@ -91,7 +103,7 @@ kubectl wait kafkabridge/bzkf-dizbox-bridge --for=condition=Ready --timeout=300s kubectl get all -A ``` -#### Install DIZ-in-a-box +##### Install DIZ-in-a-box ```sh helm upgrade --install --wait --timeout=10m --version=2.2.2 diz-in-a-box oci://ghcr.io/bzkf/diz-in-a-box/charts/diz-in-a-box diff --git a/docker-compose/README.md b/docker-compose/README.md index 4614f961..6d7659a0 100644 --- a/docker-compose/README.md +++ b/docker-compose/README.md @@ -1,39 +1,61 @@ -# obds-to-fhir Docker Compose Version +# diz-in-a-box Docker Compose Version -## Run only the obds-to-fhir job +![Figure Modular Pipeline](../img/fig1.png) -```sh -docker compose -f compose.obds-to-fhir.yaml up -``` +## Installation + +### 1. Prepare Data Import +#### a) ONKOSTAR data base connector +Configure kafka-connect in [compose.full-yaml](compose.full.yaml) and [docker-compose/kafka-connect-passwords.properties](kafka-connect-passwords.properties). +The oBDS single report XML-files will be loaded into the Kafka cluster. +For more information about kafka-connect, refer to [7. Enable Kafka Connect and the connector.](#7-enable-kafka-konnect-and-the-connector) +#### b) Folder import +Copy your oBDS collection report XML-files to [docker-compose/input-obds-reports](). -## Run while also starting a Kafka cluster and Kafka connect + +### 2. Start the Kafka Cluster ```sh -docker compose -f compose.obds-to-fhir.yaml -f compose.full.yaml up +docker compose -f compose.full.yaml up ``` -Open to view the cluster's topics. +Open to view the cluster's topics and the progress of your pipeline. + -## Load sample data from a oBDS Sammelmeldung into the Kafka cluster +### 3. Load data +#### a) ONKOSTAR data base connector +The oBDS single report XML-files from ONKOSTAR data base will be loaded into the Kafka cluster with step 2. + + +#### b) Folder import +Decompose oBDS collection report XML-files from [docker-compose/input-obds-reports](docker-compose/input-obds-reports) into single XML reports and load them into the Kafka cluster. ```sh docker compose -f compose.decompose-xmls.yaml up ``` -## Convert the FHIR resources to a CSV dataset +### 4. Transform oBDS XML-data to FHIR + +```sh +docker compose -f compose.obds-to-fhir.yaml up +``` +We currently use the FHIR profiles defined under https://simplifier.net/oncology. + + +### 5. Convert the FHIR resources to a CSV dataset ```sh sudo chown -R 1001:1001 ./opal-output/ docker compose -f compose.obds-fhir-to-opal.yaml up ``` -## Start the entire stack +### 6. Start the entire stack ```sh docker compose -f compose.obds-to-fhir.yaml -f compose.full.yaml -f compose.decompose-xmls.yaml -f compose.obds-fhir-to-opal.yaml up ``` -## Enable Kafka Connect and the connector +### 7. Enable Kafka Connect and the connector Make sure to have access to Onkostar tables `lkr_meldung`, `lkr_meldung_export` and `erkrankung`. @@ -60,7 +82,7 @@ curl -X POST \ http://localhost:8083/connectors ``` -## Run with enabled pseudonymization +### 8. Run with enabled pseudonymization > **Warning** > Requires gPAS to be set-up and the [anonymization.yaml](anonymization.yaml) to be configured @@ -69,8 +91,29 @@ curl -X POST \ docker compose -f compose.obds-to-fhir.yaml -f compose.full.yaml -f compose.pseudonymization.yaml up ``` -## Run with enabled pseudonymization and sending resources to a FHIR server +### 9. Run with enabled pseudonymization and sending resources to a FHIR server ```sh docker compose -f compose.obds-to-fhir.yaml -f compose.full.yaml -f compose.fhir-server.yaml -f compose.pseudonymization.yaml up ``` + + +### 10. Air-gapped installation + +In case of absence of Internet connectivity, container images cannot be pulled from the registry. Instead, download the air-gapped installer and move it to the deployment machine: + + + +```sh +curl -L -O https://github.com/bzkf/diz-in-a-box/releases/download/v2.2.2/air-gapped-installer.tgz +``` + + + +Run the following steps on the deployment machine. + +Extract the archive: + +```sh +tar xvzf ./air-gapped-installer.tgz +``` diff --git a/img/fig1.png b/img/fig1.png new file mode 100644 index 00000000..e2dc5455 Binary files /dev/null and b/img/fig1.png differ