Merge pull request #1306 from ScilifelabDataCentre/dev

New release v2.2.0 - Wed 2022-10-26, 9 AM

.github/pull_request_template.md

@@ -17,12 +17,13 @@
 - [ ] Bug fix (non-breaking)
 - [ ] New feature (non-breaking)
 - [ ] Breaking change (breaking, will cause existing functionality to not work as expected)
+- [ ] Tests (only)
 
 # Checklist:
 
 ## General
 
-- [ ] [Changelog](../CHANGELOG.md): New row added
+- [ ] [Changelog](../CHANGELOG.md): New row added. Not needed when PR includes _only_ tests.
 - [ ] Database schema has changed
   - [ ] A new migration is included in the PR
   - [ ] The change does not require a migration
@@ -35,10 +36,13 @@
 
 - [ ] Blocking PRs have been merged
 - [ ] Rebase / update of branch done
-- [ ] Product Owner / Scrum Master
-  - [ ] The [version](../dds_web/version.py) is updated (PR to `master` branch)
-  - [ ] I am bumping the major version (e.g. 1.x.x to 2.x.x)
-    - [ ] I have made the corresponding changes to the CLI version
+- [ ] PR to `master` branch (Product Owner / Scrum Master)
+  - [ ] The [version](../dds_web/version.py) is updated
+    - [ ] I am bumping the major version (e.g. 1.x.x to 2.x.x)
+      - [ ] I have made the corresponding changes to the CLI version
+  - [ ] Backward compatible
+    - [ ] Yes: The code works together with `dds_cli/master` branch
+    - [ ] No: The code **does not** entirely / at all work together with the `dds_cli/master` branch. _Please add detailed and clear information about the broken features_
 
 ## Checks
 

.github/workflows/codeql-analysis.yml

@@ -28,7 +28,9 @@ jobs:
       actions: read
       contents: read
       security-events: write
-
+    concurrency:
+      group: ${{ github.ref }}-codeql
+      cancel-in-progress: true
     strategy:
       fail-fast: false
       matrix:

.github/workflows/docker-compose-tests.yml

@@ -8,6 +8,9 @@ on:
 
 jobs:
   pytest:
+    concurrency:
+      group: ${{ github.ref }}-pytest
+      cancel-in-progress: true
     runs-on: ubuntu-latest
 
     steps:

.github/workflows/publish_and_trivyscan.yml

@@ -13,6 +13,13 @@ jobs:
     if: github.repository == 'ScilifelabDataCentre/dds_web'
     name: Push Docker image to Docker Hub
     runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      packages: write
+      security-events: write
+    concurrency:
+      group: ${{ github.ref }}-docker-trivy
+      cancel-in-progress: true
     steps:
       - name: Check out the repo
         uses: actions/checkout@v2
@@ -61,5 +68,6 @@ jobs:
           file: Dockerfiles/backend.Dockerfile
           context: .
           push: true
+          build-args: version=${{ github.ref_name }}
           tags: ${{ steps.meta.outputs.tags }}
           labels: ${{ steps.meta.outputs.labels }}

.github/workflows/trivy.yml

@@ -20,7 +20,7 @@ jobs:
       - name: Run Trivy vulnerability scanner
         uses: aquasecurity/[email protected]
         with:
-          image-ref: "ghcr.io/${{ env.REPOSITORY_OWNER }}/dds-backend:dev"
+          image-ref: "ghcr.io/${{ env.REPOSITORY_OWNER }}/dds-backend:latest"
           format: "sarif"
           output: "trivy-results.sarif"
           severity: "CRITICAL,HIGH"

CHANGELOG.md

@@ -1,6 +1,6 @@
 # Data Delivery System Web / API: Changelog
 
-Please add a _short_ line describing the PR you make, if the PR implements a specific feature or functionality, or refactor. Not needed if you add very small and unnoticable changes.
+Please add a _short_ line describing the PR you make, if the PR implements a specific feature or functionality, or refactor. Not needed if you add very small and unnoticable changes. Not needed when PR includes _only_ tests for already existing feature.
 
 ## Sprint (2022-02-09 - 2022-02-23)
 
@@ -151,3 +151,16 @@ Please add a _short_ line describing the PR you make, if the PR implements a spe
 - New table: `Maintenance`, for keeping track of DDS maintenance mode ([#1284](https://github.com/ScilifelabDataCentre/dds_web/pull/1284))
 - New endpoint: SetMaintenance - set maintenance mode to on or off ([#1286](https://github.com/ScilifelabDataCentre/dds_web/pull/1286))
 - New endpoint: AnyProjectsBusy - check if any projects are busy in DDS ([#1288](https://github.com/ScilifelabDataCentre/dds_web/pull/1288))
+
+## Sprint (2022-09-30 - 2022-10-14)
+
+- Bug fix: Fix the Invite.projects database model ([#1290](https://github.com/ScilifelabDataCentre/dds_web/pull/1290))
+- New endpoint: ListInvites - list invites ([#1294](https://github.com/ScilifelabDataCentre/dds_web/pull/1294))
+
+## Sprint (2022-10-14 - 2022-10-28)
+
+- Limit projects listing to active projects only; a `--show-all` flag can be used for listing all projects, active and inactive ([#1302](https://github.com/ScilifelabDataCentre/dds_web/pull/1302))
+- Return name of project creator from UserProjects ([#1303](https://github.com/ScilifelabDataCentre/dds_web/pull/1303))
+- Add version to the footer of the web pages ([#1304](https://github.com/ScilifelabDataCentre/dds_web/pull/1304))
+- Add link to the dds instance to the end of all emails ([#1305](https://github.com/ScilifelabDataCentre/dds_web/pull/1305))
+- Troubleshooting steps added to web page ([#1309](https://github.com/ScilifelabDataCentre/dds_web/pull/1309))

Dockerfiles/backend.Dockerfile

@@ -19,6 +19,10 @@ RUN apk add jpeg-dev zlib-dev libjpeg
 RUN apk add tzdata
 ENV TZ="UCT"
 
+# Extract version from Github during build
+ARG version
+ENV DDS_VERSION=$version
+
 # Copy the content to a code folder in container
 COPY ./requirements.txt /code/requirements.txt
 

dds_web/__init__.py

@@ -36,7 +36,6 @@
 
 from dds_web.scheduled_tasks import scheduler
 
-
 ####################################################################################################
 # GLOBAL VARIABLES ############################################################## GLOBAL VARIABLES #
 ####################################################################################################
@@ -190,11 +189,11 @@ def create_app(testing=False, database_uri=None):
         @app.before_request
         def prepare():
             """Populate flask globals for template rendering"""
-            from dds_web.utils import verify_cli_version
-            from dds_web.utils import get_active_motds
+            from dds_web.utils import verify_cli_version, get_active_motds, block_if_maintenance
 
             # Verify cli version compatible
             if "api/v1" in flask.request.path:
+                block_if_maintenance()
                 verify_cli_version(version_cli=flask.request.headers.get("X-Cli-Version"))
 
             # Get message of the day
@@ -272,6 +271,11 @@ def load_user(user_id):
         app.cli.add_command(update_uploaded_file_with_log)
         app.cli.add_command(lost_files_s3_db)
 
+        # Make version available inside jinja templates:
+        @app.template_filter("dds_version")
+        def dds_version_filter(_):
+            return os.environ.get("DDS_VERSION", "Unknown")
+
         with app.app_context():  # Everything in here has access to sessions
             from dds_web.database import models
 
@@ -312,21 +316,6 @@ def load_user(user_id):
 def fill_db_wrapper(db_type):
     from dds_web.database import models
 
-    maintenance_active: bool = db_type == "production"
-    flask.current_app.logger.info(
-        f"Setting maintenance as {'active' if maintenance_active else 'inactive'}..."
-    )
-
-    old_maintenance: models.Maintenance = models.Maintenance.query.all()
-    if len(old_maintenance) > 1:
-        for row in old_maintenance[1::]:
-            db.session.delete(row)
-        old_maintenance[0].active = maintenance_active
-    else:
-        new_maintenance: models.Maintenance = models.Maintenance(active=maintenance_active)
-        db.session.add(new_maintenance)
-    db.session.commit()
-
     if db_type == "production":
         username = flask.current_app.config["SUPERADMIN_USERNAME"]
         password = flask.current_app.config["SUPERADMIN_PASSWORD"]

dds_web/api/__init__.py

@@ -76,6 +76,7 @@ def output_json(data, code, headers=None):
     user.RequestTOTPActivation, "/user/totp/activate", endpoint="request_totp_activation"
 )
 api.add_resource(user.Users, "/users", endpoint="users")
+api.add_resource(user.InvitedUsers, "/user/invites", endpoint="list_invites")
 
 # Super Admins ###################################################################### Super Admins #
 

dds_web/api/project.py

@@ -472,14 +472,27 @@ def format_project_dict(self, current_user):
             "Unit Personnel",
         ]
 
+        # Get info for projects
+        get_all = flask.request.json.get("show_all", False) if flask.request.json else False
+        all_filters = (
+            [] if get_all else [models.Project.is_active == True]
+        )  # Default is to only get active projects
+        all_filters.append(
+            models.Project.public_id.in_([x.public_id for x in current_user.projects])
+        )
+
+        # Apply the filters
+        user_projects = models.Project.query.filter(sqlalchemy.and_(*all_filters)).all()
+
         # Get info for all projects
-        for p in current_user.projects:
+        for p in user_projects:
             project_info = {
                 "Project ID": p.public_id,
                 "Title": p.title,
                 "PI": p.pi,
                 "Status": p.current_status,
                 "Last updated": p.date_updated if p.date_updated else p.date_created,
+                "Created by": p.creator.name,
             }
 
             # Get proj size and update total size

dds_web/api/user.py

@@ -1254,3 +1254,99 @@ def get_users(unit: models.Unit = None):
             "keys": keys,
             "empty": not users_to_return,
         }
+
+
+class InvitedUsers(flask_restful.Resource):
+    """Provide a list of invited users."""
+
+    @auth.login_required
+    def get(self):
+        current_user = auth.current_user()
+
+        # columns to return, map to displayed column names
+        key_map = {
+            "email": "Email",
+            "role": "Role",
+            "created_at": "Created",
+            "projects": "Projects",
+        }
+        key_order = [key_map["email"], key_map["role"], key_map["projects"], key_map["created_at"]]
+        if current_user.role == "Super Admin":
+            key_map["unit"] = "Unit"
+            key_order.insert(1, "Unit")
+
+        def row_to_dict(entry) -> dict:
+            """Convert a db row to a dict, extracting only wanted columns."""
+            hit = {}
+            for key in key_map.keys():
+                hit[key_map[key]] = getattr(entry, key) or ""
+            # represent projects with public_id
+            hit["Projects"] = [project.public_id for project in hit["Projects"]]
+            # represent unit with name
+            if hit.get("Unit"):
+                hit["Unit"] = hit["Unit"].name
+            return hit
+
+        if current_user.role == "Super Admin":
+            # superadmin can see all invites
+            raw_invites = models.Invite.query.all()
+            hits = []
+            for inv in raw_invites:
+                entry = row_to_dict(inv)
+                if inv.role == "Super Admin":
+                    entry["Projects"] = "----"
+                hits.append(entry)
+
+        elif current_user.role in ("Unit Admin", "Unit Personnel"):
+            # unit users can see all invites to the unit and any projects it owns
+            # start by getting all invites, then filter by project and unit
+            unit = current_user.unit
+            unit_projects = set(proj.id for proj in unit.projects)
+            unit_projects_pubid = set(proj.public_id for proj in unit.projects)
+            raw_invites = models.Invite.query.all()
+            hits = []
+            for inv in raw_invites:
+                if inv.role == "Researcher" and set(proj.id for proj in inv.projects).intersection(
+                    unit_projects
+                ):
+                    entry = row_to_dict(inv)
+                    # do not list projects the unit does not own
+                    entry["Projects"] = [
+                        project for project in entry["Projects"] if project in unit_projects_pubid
+                    ]
+                    hits.append(entry)
+                elif inv.role in ("Unit Admin", "Unit Personnel") and inv.unit == unit:
+                    hits.append(row_to_dict(inv))
+
+        elif current_user.role == "Researcher":
+            # researchers can see invitations to projects where they are the owner
+            project_connections = (
+                models.ProjectUsers.query.filter_by(user_id=current_user.username)
+                .filter_by(owner=1)
+                .all()
+            )
+
+            if not project_connections:
+                raise ddserr.RoleException(
+                    message="You need to have the role 'Project Owner' in at least one project to be able to list any invites."
+                )
+            # use set intersection to find overlaps between projects for invite and current_user
+            user_projects = set(entry.project.id for entry in project_connections)
+            user_projects_pubid = set(entry.project.public_id for entry in project_connections)
+            raw_invites = models.Invite.query.all()
+            hits = []
+            for inv in raw_invites:
+                if inv.role == "Researcher" and set(proj.id for proj in inv.projects).intersection(
+                    user_projects
+                ):
+                    entry = row_to_dict(inv)
+                    # do not list projects the current user does not own
+                    entry["Projects"] = [
+                        project for project in entry["Projects"] if project in user_projects_pubid
+                    ]
+                    hits.append(entry)
+        else:
+            # in case further roles are defined in the future
+            raw_hits = []
+
+        return {"invites": hits, "keys": key_order}

dds_web/database/models.py

@@ -835,8 +835,11 @@ class Invite(db.Model):
     @property
     def projects(self):
         """Return list of project items."""
-
-        return [proj.project for proj in self.project_associations]
+        if self.project_invite_keys:
+            projects = [proj.project for proj in self.project_invite_keys]
+        else:
+            projects = []
+        return projects
 
     def __str__(self):
         """Called by str(), creates representation of object"""

dds_web/development/db_init.py

@@ -28,6 +28,9 @@
 def fill_db():
     """Fills the database with initial entries used for development."""
 
+    maintenance_row = models.Maintenance.query.first()
+    maintenance_row.active = False
+
     # Foreign key/relationship updates:
     # The model with the row db.relationship should append the row of the model with foreign key
 

dds_web/errors.py

@@ -423,3 +423,12 @@ def __init__(
     ):
         super().__init__(message)
         general_logger.warning(message)
+
+
+class MaintenanceOngoingException(LoggedHTTPException):
+
+    code = http.HTTPStatus.SERVICE_UNAVAILABLE
+
+    def __init__(self, message="Maintenance of DDS is ongoing."):
+        """Inform that maintenance is ongoing."""
+        super().__init__(message)

dds_web/security/auth.py

@@ -77,6 +77,8 @@ def get_user_roles_common(user):
     has been specified, the user role is returned as Project Owner. Otherwise, it is Researcher.
 
     For all other users, return the value of the role set in the database table.
+
+    Not run if the endpoint accepts all roles.
     """
     if flask.request.path in "/api/v1/proj/create" and user.role not in [
         "Unit Admin",