From 473be230b5d0e1729dfc2c57bac7a95e51821423 Mon Sep 17 00:00:00 2001 From: nanjiangshu Date: Wed, 4 Dec 2024 18:50:29 +0000 Subject: [PATCH] Add instructions to run the SDA stack in README --- GETTINGSTARTED.md | 93 ------------------------------------ README.md | 119 ++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 116 insertions(+), 96 deletions(-) delete mode 100644 GETTINGSTARTED.md diff --git a/GETTINGSTARTED.md b/GETTINGSTARTED.md deleted file mode 100644 index cb86ff5de..000000000 --- a/GETTINGSTARTED.md +++ /dev/null @@ -1,93 +0,0 @@ -## Getting Started developing components of the SDA stack - -Should one wish to engage in the development of the SDA stack itself, the prerequisite is the installation of [Go](https://www.golang.org/) on the respective machine. -The recommended version can be checked by running: - -```sh -$ make go-version-check -... -``` - -In preparation for local development, it is essential to verify the proper installation of Go, including the establishment of a [GOPATH](https://golang.org/doc/code.html#GOPATH). Confirm that $GOPATH/bin is included in the system's path, as certain distributions may package outdated versions of build tools. Subsequently, proceed to clone the repository. SDA employs [Go Modules](https://github.com/golang/go/wiki/Modules), and it is advisable to perform the cloning operation outside the GOPATH. Following this, obtain any necessary build tools by initializing the environment through bootstrapping: - -```sh -$ make bootstrap -... -``` - -### Makefile options - -The Makefile is primarily designed to be an aid during development work. - -#### Building the containers - -To build all containers for the SDA stack: - -```sh -$ make build-all -... -``` - -To build the container for a specific component replace `all` with the folder name: - -```sh -$ make build- -... -``` - -#### Running the integration tests - -This will build the container and run the integration test for the PostgreSQL container. The same test will run on every PR in github: - -```sh -$ make integrationtest-postgres -... -``` - -This will build the RabbitMQ and SDA containers and run the integration test for the RabbitMQ container. The same test will run on every PR in github: - -```sh -$ make integrationtest-rabbitmq -... -``` - -This will build all containers and run the integration tests for the SDA stack. The same test will run on every PR in github: - -```sh -$ make integrationtest-sda -... -``` - -#### Linting the GO code - -To run golangci-lint for all go components: - -```sh -$ make lint-all -... -``` - -To run golangci-lint for a specific component replace `all` with the folder name (`sda`, `sda-auth`, `sda-download`): - -```sh -$ make lint- -... -``` - -#### Running the static code tests - -For the go code this means running `go test -count=1 ./...` in the target folder. For the *sftp-inbox* this calls `mvn test -B` inside a maven container. - -To run the static code tests for all components: - -```sh -$ make test-all -... -``` - -To run the static code tests for a specific component replace `all` with the folder name (`sda`, `sda-auth`, `sda-download`, `sda-sftp-inbox`): - -```sh -$ make test- -... -``` diff --git a/README.md b/README.md index fcd1f9ff2..7d3132c46 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,119 @@ # Sensitive Data Archive -`SDA` contains all components of [NeIC Sensitive Data Archive](https://neic-sda.readthedocs.io/en/latest/) It can be used as part of a [Federated EGA](https://ega-archive.org/federated) or as a isolated Sensitive Data Archive. +The `SDA` contains all components of [NeIC Sensitive Data Archive](https://neic-sda.readthedocs.io/en/latest/). It can be used as part of a [Federated EGA](https://ega-archive.org/federated) or as a standalone Sensitive Data Archive. -For more information about the different components see the readme files in the respecive folders. -For more information on how to start developing read [Getting Started developing components of the SDA stack](/GETTINGSTARTED.md). \ No newline at end of file +For more information about the different components, please refer to the README files in their respective folders. + +## How to run the SDA stack +The following instructions outline the steps to set up and run the `SDA` services for development and testing using Docker. These steps are based on the provided [Makefile](./Makefile) commands. + +### Prerequisites +Ensure you have the following installed on your system: + +- [`Go`](https://www.golang.org/): The required version is specified in the `sda` Dockerfile. Verify using + ```sh + $ make go-version-check + ``` + +- Docker: Version 24 or higher. Verify using + ```sh + $ make docker-version-check + ``` +- Docker Compose: Version 2 or higher. For Linux, ensure the [Compose plugin](https://docs.docker.com/compose/install/linux/) is installed. + +In preparation for local development, it is essential to verify that `$GOPATH/bin` is part of the system PATH, as certain distributions may package outdated versions of build tools. SDA uses [Go Modules](https://github.com/golang/go/wiki/Modules), and it is advisable to clone the repository outside the `GOPATH`. After cloning, initialize the environment and obtain necessary build tools using the bootstrap command: + +```sh +$ make bootstrap +``` + +### Build Docker images + +Build the required Docker images for all SDA services: + +```sh +$ make build-all +``` + +You can also build images for individual services by replacing `all` with the folder name (`postgresql`, `rabbitmq`, `sda`, `sda-download`, `sda-sftp-inbox`, `sda-admin`), for example + +```sh +$ make build-sda +``` + +### Running the services + +#### Start services with Docker Compose +The following command will build all required images, bring up all services using the Docker Compose file [sda-s3-integration.yml](.github/integration/sda-s3-integration.yml) (configured for S3 as the storage method) and run the integration test: + +```sh +$ make integrationtest-sda-s3-run +``` + +#### Shut down all services and clean up resources +The following command will shut down all services and clean up all related resources: + +```sh +$ make integrationtest-sda-s3-down +``` + +For the setup with POSIX as the storage method, use +`make integrationtest-sda-posix-run` and `make integrationtest-sda-posix-down` to start and shut down services. For the setup including the [`sync`](https://github.com/neicnordic/sda-sync) service, use `make integrationtest-sda-sync-run` and `make integrationtest-sda-sync-down` to start and shut down services. + +#### Running the integration tests +This will build all required images, bring up the services, run the integration test, and then shut down services and clean up resources. The same test runs on every pull request (PR) in GitHub. + +- Integration test for the database: + ```sh + make integrationtest-postgres + ``` +- Integration test for RabbitMQ: + ```sh + make integrationtest-rabbitmq + ``` +- Integration test for all SDA setups (including S3, POSIX and sync): + ```sh + make integrationtest-sda + ``` +- Integration test for SDA using POSIX as the storage method: + ```sh + make integrationtest-sda-posix + ``` +- Integration test for SDA using S3 as the storage method: + ```sh + make integrationtest-sda-s3 + ``` +- Integration test for SDA including the sync service: + ```sh + make integrationtest-sda-sync + ``` + +### Linting the Go code + +To run `golangci-lint` for all Go components: + +```sh +$ make lint-all +``` + +To run `golangci-lint` for a specific component, replace `all` with the folder name (`sda`, `sda-auth`, `sda-download`), for example: + +```sh +$ make lint-sda +``` + +### Running the static code tests + +For Go code, this means running `go test -count=1 ./...` in the target folder. For the *sftp-inbox* this calls `mvn test -B` inside a Maven container. + +To run the static code tests for all components: + +```sh +$ make test-all +``` + +To run the static code tests for a specific component, replace `all` with the folder name (`sda`, `sda-admin`, `sda-download`, `sda-sftp-inbox`), for example: + +```sh +$ make test-sda +``` \ No newline at end of file