Merge pull request #1576 from ScilifelabDataCentre/migrations-readme

New README in migrations directory

.github/workflows/linkspector.yml

@@ -0,0 +1,18 @@
+# Linkspector: Verify links in MD files
+---
+name: Linkspector
+on: [pull_request]
+permissions:
+  pull-requests: write
+jobs:
+  check-links:
+    name: runner / linkspector
+    runs-on: ubuntu-22.04
+    steps:
+      - uses: actions/checkout@v4
+      - name: Run linkspector
+        uses: umbrelladocs/action-linkspector@v1
+        with:
+          github_token: ${{ secrets.github_token }}
+          reporter: github-pr-review
+          filter_mode: file

README.md

@@ -177,43 +177,7 @@ All have the password: `password`
 
 If you modify the database models (e.g. tables or indexes), you must create a migration for the changes. We use `Alembic` (via `flask-migrate`) which compares our database models with the running database to generate a suggested migration.
 
-The new migration can be autogenerated in two main ways:
-
-1. Make sure the development setup is running (`docker-compose up`) while doing the changes.
-
-2. Reset the container using `docker-compose down` and then remove `flask init-db $$DB_TYPE &&` in `docker-compose.yml`. It will prevent the population of the database, allowing you to install the old schema and build your migration on top of it by running `docker-compose up`
-
-To create the migration, run the command `flask db migrate -m <migration name>` in the running backend:
-
-```bash
-docker exec dds_backend flask db migrate -m <migration name>
-```
-
-This will create a migration in the folder `migrations/versions`. Confirm that the changes in the file match the changes you did, otherwise change the `upgrade` and `downgrade` functions as needed. Keep an eye out for changes to the `apscheduler` tables and indexes, and make sure they are not included in the migration. Once the migration looks ok, test it by running `flask db upgrade` in the backend:
-
-```bash
-docker exec dds_backend flask db upgrade
-```
-
-Finally, confirm that the database looks correct after running the migration and commit the migration file to git. Note that you need to run `black` on the generated migration file.
-
-> ↩️ If you want to start over, restore the content of `migrations/versions` (remove new files, run `git restore` on the folder) and start from autogeneration method 2.
-
-### Database issues while running `docker-compose up`
-
-If you run into issues with complaints about the db while running `docker-compose up` you can try to reset the containers by running `docker-compose down` before trying again. If you still have issues, try cleaning up containers and volumes manually.
-
-> ⚠️ These commands will remove _all_ containers and volumes!
-> If you are working on other projects please be more selective.
-
-```bash
-docker rm $(docker ps -a -q) -f
-docker volume prune
-```
-
-Then run `docker-compose up` as normal. The images will be rebuilt from scratch before launch.
-
-If there are still issues, try deleting the `pycache` folders and repeat the above steps.
+For instructions on how to do this, see [the README in the migrations directory](./migrations/README.md).
 
 ## Run tests
 
@@ -257,7 +221,7 @@ Equally, if you want to tear down you need to run pytest _twice_ without it, as
 
 ## Production Instance
 
-The production version of the backend image is published at the [GitHub Container Registry (GHCR)](ghcr.io/scilifelabdatacentre/dds-backend). It can also be built by running:
+The production version of the backend image is published at the [GitHub Container Registry (GHCR, ghcr.io/scilifelabdatacentre/dds-backend)](https://github.com/scilifelabdatacentre/dds_web/pkgs/container/dds-backend). It can also be built by running:
 
 ```bash
 docker build --target production -f Dockerfiles/backend.Dockerfile .

SPRINTLOG.md

@@ -239,8 +239,8 @@ _Nothing merged during this sprint_
 
 # 2023-04-14 - 2023-04-28
 
-- Documentation: Minor update of Technical Overview ((#1411)[https://github.com/ScilifelabDataCentre/dds_web/pull/1411])
-- Documentation: Account roles and their permissions ((#1412)[https://github.com/ScilifelabDataCentre/dds_web/pull/1412])
+- Documentation: Minor update of Technical Overview ([#1411](https://github.com/ScilifelabDataCentre/dds_web/pull/1411))
+- Documentation: Account roles and their permissions ([#1412](https://github.com/ScilifelabDataCentre/dds_web/pull/1412))
 
 # 2023-05-26 - 2023-06-09
 
@@ -472,3 +472,4 @@ _Nothing merged during this sprint_
 # 2024-12-16 - 2024-12-20
 
 - New version: 2.9.0 ([#1584](https://github.com/ScilifelabDataCentre/dds_web/pull/1584))
+- Instructions regarding database migrations moved to migrations directory, and Linkspector action added to scan for incorrect links in MD ([#1576](https://github.com/ScilifelabDataCentre/dds_web/pull/1576))

migrations/README.md

@@ -0,0 +1,110 @@
+# Making database changes - Adding a new migration
+
+If you modify the database models (e.g. tables or indexes), you must create a migration for the changes. We use `Alembic` (via `flask-migrate`) which compares our database models with the running database to generate a suggested migration.
+
+This document explains how to generate new database migration.
+
+## Steps for Generating Migrations
+
+### 1. Start the Containers
+
+Before making any changes to the database models, ensure the containers are running locally.
+
+```bash
+docker compose up # See root README for information regarding the different profiles
+```
+
+### 2. Modify the Database Models
+
+Edit the `models.py` file to reflect the changes you need. This might include:
+
+- Adding new tables or columns
+- Modifying existing fields
+- Deleting tables or columns
+
+### 3. Create a new migration
+
+To create the migration, pick **one of the following options**. This will create a migration in the folder `migrations/versions`.
+
+#### Option A
+
+```bash
+# 1. Enter the container
+docker exec -it dds_backend /bin/sh
+
+# 2. Generate the migration
+flask db migrate -m <migration name>
+```
+
+#### Option B
+
+```bash
+# **Outside** of the container, run the following command
+# This will generate the migration
+docker exec dds_backend flask db migrate -m <migration name>
+```
+
+### 4. Review the Generated Migration File
+
+Now, a new migration will have been generated in the `migrations/versions` directory. **Review this file carefully** to ensure it correctly represents your intended changes. If not, change the `upgrade` and `downgrade` functions as needed.
+
+> **NB!** Keep an eye out for changes to the `apscheduler` tables and indexes, and **make sure they are not included in the migration**.
+> Apscheduler is no longer used for the cronjobs, but are listed in the requirements still. This note will be removed once we have removed `apscheduler` all together.
+
+For reference on available operations and customization options, check the [Alembic Documentation](https://alembic.sqlalchemy.org/en/latest/ops.html).
+
+### 5. Test the migration
+
+Once the migration looks ok, test it using one of the following options. Confirm that the database looks correct.
+
+#### Option A
+
+```bash
+# 1. Enter the container
+docker exec -it dds_backend /bin/sh
+
+# 2. Run the database upgrade
+flask db upgrade
+```
+
+#### Option B
+
+```bash
+# **Outside** of the container, run the following command
+# This will run the migration and upgrade the database
+docker exec dds_backend flask db upgrade
+```
+
+### 6. Commit the Migration File
+
+Finally, commit the migration file to git.
+
+## How to start over
+
+If you want to start over, e.g. if something went wrong when following the steps outlined above, follow the steps below.
+
+1. Restore the content of `migrations/versions`:
+
+   a) Remove the new migrations file(s)
+   b) Run `git restore` on the `migrations/versions` folder
+
+2. Reset the container: `docker compose down`
+3. Remove `flask init-db $$DB_TYPE &&` in the `docker-compose.yml` file. This will prevent the population of the database and instead allow the install of the old schema.
+4. Start the container again: `docker-compose up`
+5. Follow the [Steps for Generating Migrations](#steps-for-generating-migrations) section
+
+## Database issues while running `docker-compose up`
+
+If you run into issues with complaints about the db while running `docker compose up` you can try to reset the containers by running `docker compose down` before trying again. If you still have issues, try cleaning up containers and volumes manually.
+
+> ⚠️ These commands will remove _all_ containers and volumes!
+> If you are working on other projects please be more selective.
+
+```bash
+docker rm $(docker ps -a -q) -f
+docker volume prune
+```
+
+Then run `docker compose up` as normal. The images will be rebuilt from scratch before launch.
+
+If there are still issues, try deleting the `pycache` folders and repeat the above steps.