Merge pull request #1150 from ScilifelabDataCentre/dev

Before release 1.0.1

.github/workflows/dockerhub.yml

@@ -1,4 +1,5 @@
-name: Publish Docker image
+---
+name: Publish Docker Image
 on:
   push:
     branches:
@@ -19,12 +20,24 @@ jobs:
         with:
           username: ${{ secrets.DOCKERHUB_USERNAME }}
           password: ${{ secrets.DOCKERHUB_TOKEN }}
-      - id: docker-tag
-        uses: yuya-takeyama/docker-tag-from-github-ref-action@v1
-      - name: Push backend to Docker Hub
+      - name: Log in to Github Container Repository
+        uses: docker/login-action@v1
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      - name: Docker metadata
+        id: meta
+        uses: docker/metadata-action@v3
+        with:
+          images: |
+            scilifelabdatacentre/dds-backend
+            ghcr.io/scilifelabdatacentre/dds-backend
+      - name: Publish image
         uses: docker/build-push-action@v2
         with:
           file: Dockerfiles/backend.Dockerfile
           context: .
           push: true
-          tags: scilifelabdatacentre/dds-backend:${{ steps.docker-tag.outputs.tag }}
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}

CHANGELOG.md

@@ -69,3 +69,15 @@ Please add a _short_ line describing the PR you make, if the PR implements a spe
 - Change from apscheduler to flask-apscheduler - solves the app context issue ([#1109](https://github.com/ScilifelabDataCentre/dds_web/pull/1109))
 - Send an email to all Unit Admins when a Unit Admin has reset their password ([#1110](https://github.com/ScilifelabDataCentre/dds_web/pull/1110)).
 - Patch: Add check for unanswered invite when creating project and adding user who is already invited ([#1117](https://github.com/ScilifelabDataCentre/dds_web/pull/1117))
+- Cronjob: Scheduled task for changing project status from Available to Expired ([#1116](https://github.com/ScilifelabDataCentre/dds_web/pull/1116))
+- Cronjob: Scheduled task for changing project status from Expired to Archived ([#1115](https://github.com/ScilifelabDataCentre/dds_web/pull/1115))
+- Add a Flask command for finding and deleting "lost files" (files that exist only in db or s3) ([#1124](https://github.com/ScilifelabDataCentre/dds_web/pull/1124))
+
+## Sprint (2022-04-06 - 2022-04-20)
+
+- New endpoint for adding a message of the day to the database ([#1136](https://github.com/ScilifelabDataCentre/dds_web/pull/1136))
+- Patch: Custom error for PI email validation ([#1146](https://github.com/ScilifelabDataCentre/dds_web/pull/1146))
+- New Data Delivery System logo ([#1148](https://github.com/ScilifelabDataCentre/dds_web/pull/1148))
+- Cronjob: Scheduled task for deleting unanswered invites after a week ([#1147](https://github.com/ScilifelabDataCentre/dds_web/pull/1147))
+- Checkbox in registration form and policy to agree to ([#1151](https://github.com/ScilifelabDataCentre/dds_web/pull/1151))
+- Patch: Add checks for valid public_id when creating new unit to avoid bucket name errors ([#1154](https://github.com/ScilifelabDataCentre/dds_web/pull/1154))

CONTRIBUTING.md

@@ -0,0 +1,5 @@
+**In progress**
+
+1. Steps for creating good issues or pull requests.
+2. Links to external documentation, mailing lists, or a code of conduct.
+3. Community and behavioral expectations.

README.md

@@ -20,7 +20,7 @@ The Delivery Portal consists of two components:
 
 The backend interface is built using [Flask](https://flask.palletsprojects.com/en/2.0.x/).
 
-See the [ADR] for information on the design decisions.
+See the [ADR](https://github.com/ScilifelabDataCentre/dds_web/wiki/Architecture-Decision-Record,-ADR) for information on the design decisions.
 
 ---
 

SECURITY.md

@@ -0,0 +1,16 @@
+# Security Policy
+
+## Supported Versions
+
+Use this section to tell people about which versions of your project are
+currently being supported with security updates.
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 1.0.0   | :white_check_mark: |
+
+## Reporting security issues
+
+Please send us your report privately via: **email comming**
+
+DO NOT file a public issue.

dds_web/__init__.py

@@ -9,6 +9,7 @@
 import datetime
 import pathlib
 import sys
+import re
 
 # Installed
 import click
@@ -257,6 +258,7 @@ def load_user(user_id):
         app.cli.add_command(fill_db_wrapper)
         app.cli.add_command(create_new_unit)
         app.cli.add_command(update_uploaded_file_with_log)
+        app.cli.add_command(lost_files_s3_db)
 
         with app.app_context():  # Everything in here has access to sessions
             from dds_web.database import models
@@ -367,9 +369,31 @@ def create_new_unit(
     days_in_available,
     days_in_expired,
 ):
-    """Create a new unit."""
+    """Create a new unit.
+
+    Rules for bucket names, which are affected by the public_id at the moment:
+    https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html
+    """
     from dds_web.database import models
 
+    error_message = ""
+    if len(public_id) > 50:
+        error_message = "The 'public_id' can be a maximum of 50 characters"
+    elif re.findall(r"[^a-zA-Z0-9.-]", public_id):
+        error_message = (
+            "The 'public_id' can only contain letters, numbers, dots (.) and hyphens (-)."
+        )
+    elif public_id[0] in [".", "-"]:
+        error_message = "The 'public_id' must begin with a letter or number."
+    elif public_id.count(".") > 2:
+        error_message = "The 'public_id' should not contain more than two dots."
+    elif public_id.startswith("xn--"):
+        error_message = "The 'public_id' cannot begin with the 'xn--' prefix."
+
+    if error_message:
+        flask.current_app.logger.error(error_message)
+        return
+
     new_unit = models.Unit(
         name=name,
         public_id=public_id,
@@ -456,3 +480,105 @@ def update_uploaded_file_with_log(project, path_to_log_file):
 
         flask.current_app.logger.info(f"Files added: {files_added}")
         flask.current_app.logger.info(f"Errors while adding files: {errors}")
+
+
+@click.command("lost-files")
+@click.argument("action_type", type=click.Choice(["find", "list", "delete"]))
+@flask.cli.with_appcontext
+def lost_files_s3_db(action_type: str):
+    """
+    Identify (and optionally delete) files that are present in S3 or in the db, but not both.
+
+    Args:
+        action_type (str): "find", "list", or "delete"
+    """
+    from dds_web.database import models
+    import boto3
+
+    for unit in models.Unit.query:
+        session = boto3.session.Session()
+
+        resource = session.resource(
+            service_name="s3",
+            endpoint_url=unit.safespring_endpoint,
+            aws_access_key_id=unit.safespring_access,
+            aws_secret_access_key=unit.safespring_secret,
+        )
+
+        db_count = 0
+        s3_count = 0
+        for project in unit.projects:
+            try:
+                s3_filenames = set(
+                    entry.key for entry in resource.Bucket(project.bucket).objects.all()
+                )
+            except resource.meta.client.exceptions.NoSuchBucket:
+                flask.current_app.logger.warning("Missing bucket %s", project.bucket)
+                continue
+
+            try:
+                db_filenames = set(entry.name_in_bucket for entry in project.files)
+            except sqlalchemy.exc.OperationalError:
+                flask.current_app.logger.critical("Unable to connect to db")
+
+            diff_db = db_filenames.difference(s3_filenames)
+            diff_s3 = s3_filenames.difference(db_filenames)
+            if action_type == "list":
+                for file_entry in diff_db:
+                    flask.current_app.logger.info(
+                        "Entry %s (%s, %s) not found in S3", file_entry, project, unit
+                    )
+                for file_entry in diff_s3:
+                    flask.current_app.logger.info(
+                        "Entry %s (%s, %s) not found in database", file_entry, project, unit
+                    )
+            elif action_type == "delete":
+                # s3 can only delete 1000 objects per request
+                batch_size = 1000
+                s3_to_delete = list(diff_s3)
+                for i in range(0, len(s3_to_delete), batch_size):
+                    resource.meta.client.delete_objects(
+                        Bucket=project.bucket,
+                        Delete={
+                            "Objects": [
+                                {"Key": entry} for entry in s3_to_delete[i : i + batch_size]
+                            ]
+                        },
+                    )
+
+                db_entries = models.File.query.filter(
+                    sqlalchemy.and_(
+                        models.File.name_in_bucket.in_(diff_db),
+                        models.File.project_id == project.id,
+                    )
+                )
+                for db_entry in db_entries:
+                    try:
+                        for db_entry_version in db_entry.versions:
+                            if db_entry_version.time_deleted is None:
+                                db_entry_version.time_deleted = datetime.datetime.utcnow()
+                        db.session.delete(db_entry)
+                        db.session.commit()
+                    except (sqlalchemy.exc.SQLAlchemyError, sqlalchemy.exc.OperationalError):
+                        db.session.rollback()
+                        flask.current_app.logger.critical("Unable to delete the database entries")
+                        sys.exit(1)
+
+            # update the counters at the end of the loop to have accurate numbers for delete
+            s3_count += len(diff_s3)
+            db_count += len(diff_db)
+
+    if s3_count or db_count:
+        action_word = "Found" if action_type in ("find", "list") else "Deleted"
+        flask.current_app.logger.info(
+            "%s %d entries for lost files (%d in db, %d in s3)",
+            action_word,
+            s3_count + db_count,
+            db_count,
+            s3_count,
+        )
+        if action_type in ("find", "list"):
+            sys.exit(1)
+
+    else:
+        flask.current_app.logger.info("Found no lost files")

dds_web/api/__init__.py

@@ -73,6 +73,7 @@ def output_json(data, code, headers=None):
 # Units #################################################################################### Units #
 
 api.add_resource(unit.AllUnits, "/unit/info/all", endpoint="all_units")
+api.add_resource(unit.MOTD, "/unit/motd", endpoint="motd")
 
 # Invoicing ############################################################################ Invoicing #
 api.add_resource(user.ShowUsage, "/usage", endpoint="usage")

dds_web/api/project.py

@@ -289,7 +289,9 @@ def delete_project(self, project: models.Project, current_time: datetime.datetim
         except (TypeError, DatabaseError, DeletionError, BucketNotFoundError) as err:
             flask.current_app.logger.exception(err)
             db.session.rollback()
-            raise DeletionError(message="Server Error: Status was not updated") from err
+            raise DeletionError(
+                project=project.public_id, message="Server Error: Status was not updated"
+            ) from err
 
         delete_message = (
             f"\nAll files in project '{project.public_id}' deleted and project info cleared"
@@ -328,7 +330,9 @@ def archive_project(
         except (TypeError, DatabaseError, DeletionError, BucketNotFoundError) as err:
             flask.current_app.logger.exception(err)
             db.session.rollback()
-            raise DeletionError(message="Server Error: Status was not updated") from err
+            raise DeletionError(
+                project=project.public_id, message="Server Error: Status was not updated"
+            ) from err
 
         return (
             models.ProjectStatuses(
@@ -604,8 +608,6 @@ def post(self):
             return {"warning": warning_message}
 
         # Add a new project to db
-        import pymysql
-
         try:
             new_project = project_schemas.CreateProjectSchema().load(p_info)
             db.session.add(new_project)
@@ -670,7 +672,7 @@ def post(self):
                         flask.current_app.logger.error(err)
                         addition_status = "Unexpected database error."
                     else:
-                        addition_status = f"Error for {user.get('email')}: {err}"
+                        addition_status = f"Error for '{user.get('email')}': {err}"
                     user_addition_statuses.append(addition_status)
                     continue
 
@@ -700,7 +702,7 @@ def post(self):
                             role=user.get("role"),
                         )
                     except DatabaseError as err:
-                        addition_status = f"Error for {user['email']}: {err.description}"
+                        addition_status = f"Error for '{user['email']}': {err.description}"
                     else:
                         addition_status = add_user_result["message"]
                     user_addition_statuses.append(addition_status)

dds_web/api/schemas/project_schemas.py

@@ -88,7 +88,7 @@ class Meta:
     pi = marshmallow.fields.String(
         required=True,
         allow_none=False,
-        validate=marshmallow.validate.Email(),
+        validate=marshmallow.validate.Email(error="The PI email is invalid."),
         error_messages={
             "required": {"message": "A principal investigator is required."},
             "null": {"message": "A principal investigator is required."},

dds_web/api/unit.py

@@ -8,12 +8,15 @@
 
 # Installed
 import flask_restful
+import flask
 import structlog
 
 # Own modules
-from dds_web import auth
+from dds_web import auth, db
 from dds_web.database import models
-from dds_web.api.dds_decorators import logging_bind_request, handle_db_error
+from dds_web.api.dds_decorators import json_required, logging_bind_request, handle_db_error
+from dds_web import utils
+import dds_web.errors as ddserr
 
 
 # initiate bound logger
@@ -59,3 +62,25 @@ def get(self):
                 "Contact Email",
             ],
         }
+
+
+class MOTD(flask_restful.Resource):
+    """Add a new MOTD message."""
+
+    @auth.login_required(role=["Super Admin"])
+    @logging_bind_request
+    @json_required
+    @handle_db_error
+    def post(self):
+        """Add a MOTD."""
+
+        curr_date = utils.current_time()
+        json_input = flask.request.json
+        motd = json_input.get("message")
+        if not motd:
+            raise ddserr.DDSArgumentError(message="No MOTD specified.")
+
+        flask.current_app.logger.debug(motd)
+        new_motd = models.MOTD(message=motd, date_created=curr_date)
+        db.session.add(new_motd)
+        db.session.commit()

dds_web/api/user.py

@@ -216,21 +216,24 @@ def invite_user(email, new_user_role, project=None, unit=None):
             if "Unit" in auth.current_user().role:
                 # Give new unit user access to all projects of the unit
                 auth.current_user().unit.invites.append(new_invite)
-                for unit_project in auth.current_user().unit.projects:
-                    if unit_project.is_active:
-                        try:
-                            share_project_private_key(
-                                from_user=auth.current_user(),
-                                to_another=new_invite,
-                                from_user_token=dds_web.security.auth.obtain_current_encrypted_token(),
-                                project=unit_project,
-                            )
-                        except ddserr.KeyNotFoundError as keyerr:
-                            projects_not_shared[
-                                unit_project.public_id
-                            ] = "You do not have access to the project(s)"
-                        else:
-                            goahead = True
+                if auth.current_user().unit.projects:
+                    for unit_project in auth.current_user().unit.projects:
+                        if unit_project.is_active:
+                            try:
+                                share_project_private_key(
+                                    from_user=auth.current_user(),
+                                    to_another=new_invite,
+                                    from_user_token=dds_web.security.auth.obtain_current_encrypted_token(),
+                                    project=unit_project,
+                                )
+                            except ddserr.KeyNotFoundError as keyerr:
+                                projects_not_shared[
+                                    unit_project.public_id
+                                ] = "You do not have access to the project(s)"
+                            else:
+                                goahead = True
+                else:
+                    goahead = True
 
                 if not project:  # specified project is disregarded for unituser invites
                     msg = f"{str(new_invite)} was successful."
@@ -1101,4 +1104,4 @@ def get(self):
             for user in unit_row.users
         ]
 
-        return {"users": unit_users, "keys": keys, "unit": unit_row.name}
+        return {"users": unit_users, "keys": keys, "unit": unit_row.name, "empty": not unit_users}