From 17628c80559ef14bd3851e2c256f21e97c830557 Mon Sep 17 00:00:00 2001 From: Ben Keller Date: Tue, 2 Jun 2020 15:33:11 -0700 Subject: [PATCH] Update developer documentation --- docs/development/index.md | 323 ++++++++++++++++++++++++++------------ 1 file changed, 224 insertions(+), 99 deletions(-) diff --git a/docs/development/index.md b/docs/development/index.md index 8308428a1..f3bc46ee1 100644 --- a/docs/development/index.md +++ b/docs/development/index.md @@ -2,6 +2,8 @@ These guidelines are intended for those working directly on [Aquarium](aquarium.bio). +Everyone else should visit the installation page on [Aquarium.bio](aquarium.bio). + --- ## Getting Started @@ -32,7 +34,7 @@ The following commands will allow you to run Aquarium in Rails development mode In development mode, Aquarium is available at `localhost:3000`. -3. **Stop** the Aquarium services, type `ctrl-c` followed by +3. **Stop** the Aquarium services, by typing `ctrl-c` followed by ```bash docker-compose down -v @@ -44,42 +46,44 @@ The following commands will allow you to run Aquarium in Rails development mode To run commands within the running Aquarium container, precede each command with - ```bash - docker-compose exec app - ``` +```bash +docker-compose exec app +``` For instance, you can run the Rails console with the command - ```bash - docker-compose exec app rails c - ``` +```bash +docker-compose exec app rails c +``` And, you can also run an interactive shell within the Aquarium container with the command - ```bash - docker-compose exec app /bin/sh - ``` +```bash +docker-compose exec app /bin/sh +``` which allows you to work within the container. For commands that don't require that Aquarium is running, using a command starting with - ```bash - docker-compose run --rm app - ``` +```bash +docker-compose run --rm app +``` -will create a temporary container. +will create a temporary container. This can be useful for running single commands such as - ```bash - docker-compose run --rm app rspec - ``` +```bash +docker-compose run --rm app rspec +``` which runs RSpec on the tests in the `spec` directory. +More details on the docker-compose commands can be found [here](https://docs.docker.com/compose/reference/overview/). + ## Switching databases -The configuration for Docker uses the MySQL Docker image, which is capable of automatically importing a database dump the first time it is started. +The development configuration uses the MySQL Docker image, which is capable of automatically importing a database dump the first time it is started. You can switch the database, but doing so will destroy any changes you have made to the current database. If you want to save these changes, you will have to create a database dump. @@ -106,7 +110,7 @@ cp production_dump.sql docker/mysql_init/dump.sql ``` Note: It may be necessary to run migrations on database dump from a prior version of Aquarium. -See the Docker installation instructions at [aquarium.bio](aquarium.bio) for details. +See the migration instructions below. ## Restoring the default database dump @@ -122,6 +126,14 @@ Before restarting Aquarium, remove the MySQL files with rm -rf docker/db/* ``` +## Migrating the Database + +```bash +docker-compose up -d +docker-compose exec app RAILS_ENV=production rake db:migrate +docker-compose down -v +``` + ## Testing Aquarium ### Running Tests @@ -130,28 +142,37 @@ rm -rf docker/db/* docker-compose run --rm app rspec ``` +Some of the tests do intentionally raise exceptions, so do not be concerned if these failures seem to be missed. + +Test coverage is captured by simplecov in the file `coverage/index.html`. + ### Adding Tests +Add additional RSpec tests in the `spec` directory. +New tests can use FactoryBot factories for several of the models that are located in the `spec/factories` directory. + ## Editing Aquarium ### Documenting changes +Use the CHANGE_LOG file to document changes that Aquarium users may need to know about. + ### Formatting Aquarium code The Aquarium repository is setup to use [RuboCop](https://rubocop.readthedocs.io). -When you make changes to Aquarium code, run the command +When you make changes to Aquarium code, run the command - ```bash - docker-compose run --rm app rubocop -x - ``` +```bash +docker-compose run --rm app rubocop -x +``` to fix layout issues. Then run the command - ```bash - docker-compose run --rm app rubocop -x - ``` +```bash +docker-compose run --rm app rubocop +``` to see if you have introduced any other issues. This will check for several potential issues that occur in Rails apps. @@ -159,9 +180,8 @@ This will check for several potential issues that occur in Rails apps. You should fix any issues, but be certain to test them. RuboCop can do other auto-corrections, but don't use that feature unless your tests ensure that the behavior is not changed. -Because RuboCop may change, it may be necessary to make changes to the `.rubocop.yml` file in the repository directory. +Because RuboCop periodically changes, it can be necessary to make changes to the `.rubocop.yml` file in the repository directory. When the Ruby version is changed the target version in this file should also be changed. -Otherwise, you probably wont need to change this file. ### Fixing Style TODOs @@ -171,13 +191,13 @@ However, it also identifies issues that we should try to eliminate. The process of doing this is to pick one issue, fix it, test the fix, and then update the todo file. When fixing issues make sure that there is a test that will exercise the fix, and be extra careful when applying auto-correct. -Some fixes may not be possible without affecting the Krill library for protocols, which could break protocols that are in use. +_Note_: Some fixes may not be possible without affecting the Krill library for protocols, which could break protocols that are in use. This command will regenerate the `.rubocop_todo.yml` file - ```bash - docker-compose run -rm app rubocop --auto-gen-config - ``` +```bash +docker-compose run -rm app rubocop --auto-gen-config +``` ### Documenting Aquarium Ruby Code @@ -294,7 +314,7 @@ Keep it up-to-date if you change something that affects Aquarium development. If there are any failures, fix them and start over. -5. Fix any layout problems +5. Fix any layout problems ```bash docker-compose run --rm app rubocop -x @@ -320,7 +340,7 @@ Keep it up-to-date if you change something that affects Aquarium development. 10. Update the version number in `package.json` and `config/initializers/version.rb` to the new version number. -11. Update API documentation by running +11. Update API documentation by running ```bash docker-compose run --rm app yard @@ -332,7 +352,7 @@ Keep it up-to-date if you change something that affects Aquarium development. git log v$OLDVERSION.. ``` -11. Ensure all changes have been committed and pushed. +13. Ensure all changes have been committed and pushed. ```bash git status && git log --branches --not --remotes @@ -340,118 +360,223 @@ Keep it up-to-date if you change something that affects Aquarium development. Commit and push any changes found. -12. Create a tag for the new version: +14. Create a tag for the new version: ```bash git tag -a v$NEWVERSION -m "Aquarium version $NEWVERSION" git push --tags ``` -13. [Create a release on github](https://help.github.com/articles/creating-releases/). +15. [Create a release on github](https://help.github.com/articles/creating-releases/). Visit the [Aquarium releases page](https://github.com/klavinslab/aquarium/releases). + - Click "Tags". - Click "add release notes" for the new tag, use the change log as the release notes. - Click "publish release". -14. (Update zenodo entry) +16. (Update zenodo entry) -15. Push image to Docker Hub +17. Push image to Docker Hub ```bash bash ./aquarium.sh build docker push aquariumbio/aquarium:v$NEWVERSION ``` -## Aquarium Internals - ## Aquarium Configuration -Aquarium is configured to run within Docker using the following files: +### Docker image + +Files: ```bash aquarium -|-- docker -| |-- db # directory to store database files -| |-- mysql_init # database dump to initialize database -| |-- s3 # directory for minio files -| |-- aquarium-entrypoint.sh # entrypoint for running Aquarium -| |-- krill-entrypoint.sh # entrypoint for running Krill -| |-- nginx.development.conf # nginx configuration for development server -| `-- nginx.production.conf # nginx configuration for production server -|-- docker-compose.override.yml # development compose file -|-- docker-compose.production.yml # production compose file -|-- docker-compose.windows.yml # windows compose file +|-- Dockerfile # defines the image for Aquarium +`-- entrypoint.sh # entrypoint for Docker image +``` + +The Dockerfile defines the images: + +- aquarium-development -- image for running Aquarium in development mode +- aquarium-builder -- temporary image for production builds +- aquarium -- image for running Aquarium in production model + +This image is used for both Aquarium and Krill services. + +The entrypoint script determines how the image starts up. + +### Parameters + +Files: + +```bash +aquarium +|-- .env # docker-compose environment file (see setup.sh) |-- docker-compose.yml # base compose file -`-- Dockerfile # defines the image for Aquarium +`-- setup.sh ``` -### Images +Aquarium is parameterized to use environment variables to configure it to use with different services. +These are set in the `docker-compose.yml` file using values from the `.env` file. +The script `setup.sh` updates the values in the .env file, creating missing values as needed. +(This script also ensures that a database dump.sql file exists.) + +The full set of environment variables is identified below. +Values need to be given unless the variable has a default. + +**Database**: +The database is configured to use MySQL by default with the hostname configured for [local deployment](http://klavinslab.org/aquarium-local/). -The `Dockerfile` configures the `basebuilder` Aquarium image that contains the configuration needed by development or production environments. +| Variable | Description | Default | +| ----------- | ----------------------------------- | ------------ | +| DB_NAME | the name of the database | `production` | +| DB_USER | the database user | `aquarium` | +| DB_PASSWORD | the password of the user | – | +| DB_ADAPTER | the database adapter name | `mysql2` | +| DB_HOST | the network address of the database | `db` | +| DB_PORT | the network port of the database | `3306` | -The `basebuilder` image is based on the Ruby Alpine linux Docker image that includes Rails. -In addition the image includes: +**Email**: +To use the AWS SES set the `EMAIL_SERVICE` to `AWS` along with -1. Development tools needed to configure and run Aquarium. -2. Javascript components used by Aquarium webpages. -3. Gems used by Aquarium. -4. The `aquarium` application (minus the files in `.dockerignore`). -5. The entrypoint scripts for running Aquarium and Krill from Docker. +| Variable | Description | Default | +| --------------------- | ---------------------------------- | ------- | +| AWS_REGION | the region for the AWS server | – | +| AWS_ACCESS_KEY_ID | the access key id for your account | – | +| AWS_SECRET_ACCESS_KEY | the access key for your account | – | -Note that this base configuration includes the configuration files in `config`, though the docker-compose configurations override these. +**Krill**: +Set the environment variable `KRILL_HOST` -copies rails configuration files from the `docker/aquarium` directory into the correct place in the image; and adds the `docker/aquarium-entrypoint.sh` and `docker/krill-entrypoint.sh` scripts for starting the Aquarium services. -The configuration also ensures that the `docker/db` and `docker/s3` directories needed for the database and [minio](https://minio.io) S3 server are created on the host as required by the compose files. -The `devbuilder` and `prodbuilder` configurations build an image with environment specific files. +| Variable | Description | Default | +| ---------- | ----------------------------------- | ------- | +| KRILL_HOST | the hostname for the krill server | `krill` | +| KRILL_PORT | the port served by the krill server | 3500 | -### Compose files +**S3**: +Aquarium is configured to use either AWS S3 or minio, and is set to use minio by default with the hostname configured for [local deployment](http://klavinslab.org/aquarium-local/). -The docker-compose files are defined to allow Aquarium to be run locally in production or development modes and on Windows. +To use minio set the following variables -Specifically, the files are meant to be combined as follows: +| Variable | Description | Default | +| -------------------- | --------------------------------------- | ---------------- | +| S3_PROTOCOL | network protocol for S3 service | `http` | +| S3_HOST | network address of the S3 service | `localhost:9000` | +| S3_REGION | name of S3 region | `us-west-1` | +| S3_BUCKET_NAME | name of S3 bucket | `development` | +| S3_ACCESS_KEY_ID | the access key id for the minio service | – | +| S3_SECRET_ACCESS_KEY | the access key for the minio service | – | -- `docker-compose.yml` and `docker-compose.override.yml` runs production, -- `docker-compose.yml` and `docker-compose.dev.yml` runs development, and -- adding `docker-compose.windows.yml` allows MySQL to run on Windows. +For the local deployment, the minio service is named `s3`, but it is necessary to redirect `localhost:9000` in order to use the minio docker image. -The order of the files is significant since the later files add or replace definitions from the base file. +To use AWS S3 set the variable `S3_SERVICE` to `AWS` along with the following variables -Note that the command for the first combination +| Variable | Description | Default | +| -------------------- | ---------------------------------- | ------- | +| S3_REGION | name of S3 region | – | +| S3_BUCKET_NAME | name of S3 bucket | – | +| S3_ACCESS_KEY_ID | the access key id for your account | – | +| S3_SECRET_ACCESS_KEY | the access key for your account | – | + +**Timezone**: +Set the variable `TZ` to the desired timezone for your instance. +This should match the timezone for your database. + +### Instance configuration + +Files: ```bash -docker-compose -f docker-compose.yml -f docker-compose.override.yml up +aquarium +|-- biofab-eula.yml # example of end user license agreement for a lab +`-- instance.yml # (optional) specifies instance ``` -is equivalent to the simpler command +Some configuration can be done using a couple of YAML files. -```bash -docker-compose up +The first is the file `instance.yml` with keys for the values you want to set. +For instance, to change the name of the instance to `Wonder Lab` use the file + +```yaml +default: &default + instance_name: Wonder Lab + +production: + <<: *default + +development: + <<: *default ``` -The compose files are designed so that running in production is the default configuration to support users who are doing protocol development on a local instance. -To support this, the `docker-compose.yml` file contains the common configuration for both environments, including identifying the entrypoints for Aquarium and Krill, mounting host directories used by Aquarium, MySQL and minio for persistent storage. -The file `docker-compose.override.yml` is run by default and adds production environment files to the base configuration, while `docker-compose.dev.yml` adds the configurations for the development environment. +And, then map the Aquarium path `/aquarium/config/instance.yml` to this file. +For instance, in the docker-compose.yml file, add the following line to the `volumes` for +the aquarium service: -Much of the key differences between environments are handled by mounting different files with the different configurations. -For instance, the Rails configuration files are replaced by files from `docker/aquarium` that are mounted over the counterparts in `config`. -In the case of configuring MySQL and nginx, the mount points for the configuration files are particular to the image used for these services, some of which are not terribly well documented. -Using volumes in this way is convenient, but can also be the source of great mystery when a mounted volume overrides the files in the image. -Tweak the volumes with care. +```yaml +- ./instance.yml:/aquarium/config/instance.yml +``` -### Database +The following values can be set using this file or environment variables: + +| Config key | Environment Variable | Default | +| ----------------- | -------------------- | ------------------------------------- | +| lab_name | LAB_NAME | `Your Lab` | +| lab_email_address | LAB_EMAIL_ADDRESS | – | +| logo_path | LOGO_PATH | `aquarium-logo.png` | +| image_uri | IMAGE_URI | _S3_PROTOCOL_`://`_S3_HOST_`/images/` | + +In addition to the instance details, the user agreement for the lab can be set by creating a YAML file containing the agreement. +The YAML must include the keys `title` with the title of the agreement, `updated` with the date last updated, and `clauses`, which is a list of pairs of `title` and `text` pairs. +The default can be found in the file `config/user_agreement.yml`: + +```yaml +title: End User Agreement +updated: 21 January 2020 +clauses: + - title: No User Agreement + text: This instance of Aquarium has no user agreement. If you manage this instance, you may want to add one. + - title: Agreement file format + text: | + A user agreement is given as a YAML file with values for 'title' and 'updated' and a list 'clauses'. + The first two values are strings, and the value for updated is the date. + Each clause has a 'title' and 'text' both of which are strings. + The title of each clause should be preceded by a hyphen ('-'). + - title: Find out more + title: More detail on configuration can be found at the site aquarium.bio. -The `docker/mysql_init` directory contains the database dump that is used to initialize the database when it is run the first time. -The MySQL service is configured to use the `docker/db` directory to store its files, and removing the contents of this directory (`rm -rf docker/db/*`) will cause the database to initialize the next time the service is started. +``` -### Local S3 server +To use a different file, say the `biofab-eula.yml` file, add the following line to the `volumes` for +the aquarium service: -A [minio](minio.io) service `s3` is used to manage files uploaded to Aquarium in the local configuration. -The `s3` service is configured so that these files are stored in `docker/s3`. -The current configuration does not allow the pre-signed URLs returned by Aquarium to be used because the host is configured to be the hostname of the service, which is only accessible from within Docker. -However, the files are also be accessible through the minio webclient at `localhost:9000`. +```yaml +- ./biofab-eula.yml:/aquarium/config/user_agreement.yml +``` + +### Execution environment + +Files: + +```bash +aquarium +|-- aquarium.sh # script to run Aquarium in production mode +|-- develop-compose.sh # script to run Aquarium in development mode (for compatibility) +|-- docker +| |-- db # directory to store database files +| |-- mysql_init # database dump to initialize database +| |-- s3 # directory for minio files +| |-- nginx.development.conf # nginx configuration for development server +| `-- nginx.production.conf # nginx configuration for production server +|-- docker-compose.override.yml # development compose file +|-- docker-compose.production.yml # production compose file +|-- docker-compose.windows.yml # windows compose file +|-- docker-compose.selenium.yml # adds selenium service +`-- docker-compose.yml # base compose file +``` -### Local web server +The variants of `docker-compose.yml` files determine how the services used by Aquarium are configured. +Within the aquarium repository these are set to run Aquarium using MySQL for the database, minio for S3, and nginx as the reverse proxy. +The compose files mount relevant files in the `docker` sub-directory, which is where the database and S3 files are stored. +The S3 data files depend on the `S3_SECRET_ACCESS_KEY` and so have to be removed when this value is changed, so be careful if data needs to be kept. -Access to Aquarium and the S3 webclient is handled by nginx. -For development, all requests to port 3000 are forwarded to Aquarium, while for production, static files are served by nginx and other requests are handled by puma via a socket. -See the nginx configuration files in the `docker` directory. +The scripts `aquarium.sh` and `develop-compose.sh` are convenience scripts for running the `docker-compose` commands for Aquarium in production and development modes.