diff --git a/.github/workflows/linkspector.yml b/.github/workflows/linkspector.yml new file mode 100644 index 000000000..ad3412d6c --- /dev/null +++ b/.github/workflows/linkspector.yml @@ -0,0 +1,18 @@ +# Linkspector: Verify links in MD files +--- +name: Linkspector +on: [pull_request] +permissions: + pull-requests: write +jobs: + check-links: + name: runner / linkspector + runs-on: ubuntu-22.04 + steps: + - uses: actions/checkout@v4 + - name: Run linkspector + uses: umbrelladocs/action-linkspector@v1 + with: + github_token: ${{ secrets.github_token }} + reporter: github-pr-review + filter_mode: file diff --git a/README.md b/README.md index f5be819d3..a529e46ad 100644 --- a/README.md +++ b/README.md @@ -177,43 +177,7 @@ All have the password: `password` If you modify the database models (e.g. tables or indexes), you must create a migration for the changes. We use `Alembic` (via `flask-migrate`) which compares our database models with the running database to generate a suggested migration. -The new migration can be autogenerated in two main ways: - -1. Make sure the development setup is running (`docker-compose up`) while doing the changes. - -2. Reset the container using `docker-compose down` and then remove `flask init-db $$DB_TYPE &&` in `docker-compose.yml`. It will prevent the population of the database, allowing you to install the old schema and build your migration on top of it by running `docker-compose up` - -To create the migration, run the command `flask db migrate -m ` in the running backend: - -```bash -docker exec dds_backend flask db migrate -m -``` - -This will create a migration in the folder `migrations/versions`. Confirm that the changes in the file match the changes you did, otherwise change the `upgrade` and `downgrade` functions as needed. Keep an eye out for changes to the `apscheduler` tables and indexes, and make sure they are not included in the migration. Once the migration looks ok, test it by running `flask db upgrade` in the backend: - -```bash -docker exec dds_backend flask db upgrade -``` - -Finally, confirm that the database looks correct after running the migration and commit the migration file to git. Note that you need to run `black` on the generated migration file. - -> ↩️ If you want to start over, restore the content of `migrations/versions` (remove new files, run `git restore` on the folder) and start from autogeneration method 2. - -### Database issues while running `docker-compose up` - -If you run into issues with complaints about the db while running `docker-compose up` you can try to reset the containers by running `docker-compose down` before trying again. If you still have issues, try cleaning up containers and volumes manually. - -> ⚠️ These commands will remove _all_ containers and volumes! -> If you are working on other projects please be more selective. - -```bash -docker rm $(docker ps -a -q) -f -docker volume prune -``` - -Then run `docker-compose up` as normal. The images will be rebuilt from scratch before launch. - -If there are still issues, try deleting the `pycache` folders and repeat the above steps. +For instructions on how to do this, see [the README in the migrations directory](./migrations/README.md). ## Run tests @@ -257,7 +221,7 @@ Equally, if you want to tear down you need to run pytest _twice_ without it, as ## Production Instance -The production version of the backend image is published at the [GitHub Container Registry (GHCR)](ghcr.io/scilifelabdatacentre/dds-backend). It can also be built by running: +The production version of the backend image is published at the [GitHub Container Registry (GHCR, ghcr.io/scilifelabdatacentre/dds-backend)](https://github.com/scilifelabdatacentre/dds_web/pkgs/container/dds-backend). It can also be built by running: ```bash docker build --target production -f Dockerfiles/backend.Dockerfile . diff --git a/SPRINTLOG.md b/SPRINTLOG.md index 75562b52b..4c3ad9df1 100644 --- a/SPRINTLOG.md +++ b/SPRINTLOG.md @@ -239,8 +239,8 @@ _Nothing merged during this sprint_ # 2023-04-14 - 2023-04-28 -- Documentation: Minor update of Technical Overview ((#1411)[https://github.com/ScilifelabDataCentre/dds_web/pull/1411]) -- Documentation: Account roles and their permissions ((#1412)[https://github.com/ScilifelabDataCentre/dds_web/pull/1412]) +- Documentation: Minor update of Technical Overview ([#1411](https://github.com/ScilifelabDataCentre/dds_web/pull/1411)) +- Documentation: Account roles and their permissions ([#1412](https://github.com/ScilifelabDataCentre/dds_web/pull/1412)) # 2023-05-26 - 2023-06-09 @@ -472,3 +472,4 @@ _Nothing merged during this sprint_ # 2024-12-16 - 2024-12-20 - New version: 2.9.0 ([#1584](https://github.com/ScilifelabDataCentre/dds_web/pull/1584)) +- Instructions regarding database migrations moved to migrations directory, and Linkspector action added to scan for incorrect links in MD ([#1576](https://github.com/ScilifelabDataCentre/dds_web/pull/1576)) diff --git a/migrations/README b/migrations/README deleted file mode 100644 index 0e0484415..000000000 --- a/migrations/README +++ /dev/null @@ -1 +0,0 @@ -Single-database configuration for Flask. diff --git a/migrations/README.md b/migrations/README.md new file mode 100644 index 000000000..243eacc91 --- /dev/null +++ b/migrations/README.md @@ -0,0 +1,110 @@ +# Making database changes - Adding a new migration + +If you modify the database models (e.g. tables or indexes), you must create a migration for the changes. We use `Alembic` (via `flask-migrate`) which compares our database models with the running database to generate a suggested migration. + +This document explains how to generate new database migration. + +## Steps for Generating Migrations + +### 1. Start the Containers + +Before making any changes to the database models, ensure the containers are running locally. + +```bash +docker compose up # See root README for information regarding the different profiles +``` + +### 2. Modify the Database Models + +Edit the `models.py` file to reflect the changes you need. This might include: + +- Adding new tables or columns +- Modifying existing fields +- Deleting tables or columns + +### 3. Create a new migration + +To create the migration, pick **one of the following options**. This will create a migration in the folder `migrations/versions`. + +#### Option A + +```bash +# 1. Enter the container +docker exec -it dds_backend /bin/sh + +# 2. Generate the migration +flask db migrate -m +``` + +#### Option B + +```bash +# **Outside** of the container, run the following command +# This will generate the migration +docker exec dds_backend flask db migrate -m +``` + +### 4. Review the Generated Migration File + +Now, a new migration will have been generated in the `migrations/versions` directory. **Review this file carefully** to ensure it correctly represents your intended changes. If not, change the `upgrade` and `downgrade` functions as needed. + +> **NB!** Keep an eye out for changes to the `apscheduler` tables and indexes, and **make sure they are not included in the migration**. +> Apscheduler is no longer used for the cronjobs, but are listed in the requirements still. This note will be removed once we have removed `apscheduler` all together. + +For reference on available operations and customization options, check the [Alembic Documentation](https://alembic.sqlalchemy.org/en/latest/ops.html). + +### 5. Test the migration + +Once the migration looks ok, test it using one of the following options. Confirm that the database looks correct. + +#### Option A + +```bash +# 1. Enter the container +docker exec -it dds_backend /bin/sh + +# 2. Run the database upgrade +flask db upgrade +``` + +#### Option B + +```bash +# **Outside** of the container, run the following command +# This will run the migration and upgrade the database +docker exec dds_backend flask db upgrade +``` + +### 6. Commit the Migration File + +Finally, commit the migration file to git. + +## How to start over + +If you want to start over, e.g. if something went wrong when following the steps outlined above, follow the steps below. + +1. Restore the content of `migrations/versions`: + + a) Remove the new migrations file(s) + b) Run `git restore` on the `migrations/versions` folder + +2. Reset the container: `docker compose down` +3. Remove `flask init-db $$DB_TYPE &&` in the `docker-compose.yml` file. This will prevent the population of the database and instead allow the install of the old schema. +4. Start the container again: `docker-compose up` +5. Follow the [Steps for Generating Migrations](#steps-for-generating-migrations) section + +## Database issues while running `docker-compose up` + +If you run into issues with complaints about the db while running `docker compose up` you can try to reset the containers by running `docker compose down` before trying again. If you still have issues, try cleaning up containers and volumes manually. + +> ⚠️ These commands will remove _all_ containers and volumes! +> If you are working on other projects please be more selective. + +```bash +docker rm $(docker ps -a -q) -f +docker volume prune +``` + +Then run `docker compose up` as normal. The images will be rebuilt from scratch before launch. + +If there are still issues, try deleting the `pycache` folders and repeat the above steps.