Skip to content

Commit

Permalink
Merge pull request #1374 from ScilifelabDataCentre/dev-fix
Browse files Browse the repository at this point in the history
New release PR because of required master- and dev fix
  • Loading branch information
i-oden authored Jan 31, 2023
2 parents faf8bb1 + 3705574 commit f5b6ffd
Show file tree
Hide file tree
Showing 33 changed files with 1,871 additions and 1,226 deletions.
171 changes: 115 additions & 56 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -1,67 +1,126 @@
> **Before submitting the PR, please go through the sections below and fill in what you can. If there are any items that are irrelevant for the current PR, remove the row. If a relevant option is missing, please add it as an item and add a PR comment informing that the new option should be included into this template.**
<!--
> **Before _submitting_ PR:**
>
> - Fill in and tick fields
> - _Remove all rows_ that are not relevant for the current PR
> - Revelant option missing? Add it as an item and add a PR comment informing that the new option should be included into this template.
>
> **Before _merging_ PR:**
>
> _Tick all relevant items._
-->

> **All _relevant_ items should be ticked before the PR is merged**
## **1. This PR contains the following changes...**

# Description
_Add a summary of the changes and the related issue._

- [ ] Summary of the changes and the related issue:
- [ ] Motivation and context regarding why the change is needed:
- [ ] List / description of any dependencies or other changes required for this change:
- Fixes an issue in GitHub / Jira:
- [ ] Yes: _[link to GitHub issue / Jira task ID]_
- [ ] No
## **2. The following additional changes are required for this to work**

## Type of change
_Add information on additional changes required for the PR changes to work, both locally and in the deployments._

> E.g. Does the deployment setup need anything for this to work?
## **3. The PR fixes the following GitHub issue / Jira task**

<!-- Comment out the item which does not apply here.-->

- [ ] GitHub issue (link):
- [ ] Jira task (ID, `DDS-xxxx`):
- [ ] The PR does not fix a specific GitHub issue or Jira task

## **4. What _type of change(s)_ does the PR contain?**

<!--
- "Breaking": The change will cause existing functionality to not work as expected.
- Workflow: E.g. a new github action or changes to this PR template. Anything that alters our or the codes workflow.
-->

- [ ] Bug fix
- [ ] Breaking: _Describe_
- [ ] Non-breaking
- [ ] Documentation
- [ ] New feature
- [ ] Breaking: _Describe_
- [ ] Breaking: _Please describe the reason for the break and how we can fix it._
- [ ] Non-breaking
- [ ] Database change
- [ ] Migration _included in PR_
- [ ] Migration _not needed_
- [ ] Bug fix
- [ ] Breaking: _Please describe the reason for the break and how we can fix it._
- [ ] Non-breaking
- [ ] Security Alert fix
- [ ] Documentation
- [ ] Tests **(only)**
- [ ] Workflow

_"Breaking": The change will cause existing functionality to not work as expected._

# Checklist:

## General

- [ ] [Changelog](../CHANGELOG.md): New row added. Not needed when PR includes _only_ tests.
- [ ] Database schema has changed
- [ ] A new migration is included in the PR
- [ ] The change does not require a migration
- [ ] Code change
- [ ] Self-review of code done
- [ ] Comments added, particularly in hard-to-understand areas
- Documentation update
- [ ] Done
- [ ] Not needed

## Repository / Releases

- [ ] Blocking PRs have been merged
- [ ] Rebase / update of branch done
- [ ] PR to `master` branch (Product Owner / Scrum Master)
- [ ] The [version](../dds_web/version.py) is updated
- [ ] I am bumping the major version (e.g. 1.x.x to 2.x.x)
- [ ] I have made the corresponding changes to the CLI version
- Backward compatible
- [ ] Yes: The code works together with `dds_cli/master` branch
- [ ] No: The code **does not** entirely / at all work together with the `dds_cli/master` branch. _Please add detailed and clear information about the broken features_

## Checks

- [ ] CodeQL passes
- [ ] Formatting: Black & Prettier checks pass
- Tests
- [ ] I have added tests for the new code
- [ ] The tests pass
- Trivy / Snyk:
- [ ] There are no new security alerts
- [ ] This PR fixes new security alerts
- [ ] Security alerts have been dismissed
- [ ] PR will be merged with new security alerts; This is why: _Please add a short description here_
## **5. Checklist**

<!-- Comment out the items which do not apply here.-->

### **Always**

<!-- Always go through the following items. -->

- [Changelog](../CHANGELOG.md)
- [ ] Added
- [ ] Not needed (E.g. PR contains _only_ tests)
- Rebase / Update / Merge _from_ base branch (the branch from which the current is forked)
- [ ] Done
- [ ] Not needed
- Blocking PRs
- [ ] Merged
- [ ] No blocking PRs
- PR to `master` branch
- [ ] Yes: Go to the section [PR to master](#pr-to-master)
- [ ] No

### If PR consists of **code change(s)**

<!-- If the PR contains code changes, the following need to be checked.-->

- Self review
- [ ] Done
- Comments, docstrings, etc
- [ ] Added / Updated
- Documentation
- [ ] Updated
- [ ] Update not needed

### If PR is to **master**

<!-- Is your PR to the master branch? The following items need to be checked off. -->

- [ ] I have followed steps 1-5 in [the release instructions](../doc/procedures/new_release.md)
- [ ] I am bumping the major version (e.g. 1.x.x to 2.x.x)
- [ ] I have made the corresponding changes to the CLI version

**Is this version _backward compatible?_**

- [ ] Yes: The code works together with `dds_cli/master` branch
- [ ] No: The code **does not** entirely / at all work together with the `dds_cli/master` branch. _Please add detailed and clear information about the broken features_

## **6. Actions / Scans**

<!-- Go through all checkboxes. All actions must pass before merging is allowed.-->

- **Black**: Python code formatter. Does not execute. Only tests.
Run `black .` locally to execute formatting.
- [ ] Passed
- **Prettier**: General code formatter. Our use case: MD and yaml mainly.
Run `npx prettier --write .` locally to execute formatting.
- [ ] Passed
- **Tests**: Pytest to verify that functionality works as expected.
- [ ] New tests added
- [ ] No new tests
- [ ] Passed
- **CodeQL**: Scan for security vulnerabilities, bugs, errors
- [ ] New alerts: _Go through them and either fix, dismiss och ignore. Add reasoning in items below._
- [ ] Alerts fixed: _What?_
- [ ] Alerts ignored / dismissed: _Why?_
- [ ] Passed
- **Trivy**: Security scanner
- [ ] New alerts: _Go through them and either fix, dismiss och ignore. Add reasoning in items below._
- [ ] Alerts fixed: _What?_
- [ ] Alerts ignored / dismissed: _Why?_
- [ ] Passed
- **Snyk**: Security scanner
- [ ] New alerts: _Go through them and either fix, dismiss och ignore. Add reasoning in items below._
- [ ] Alerts fixed: _What?_
- [ ] Alerts ignored / dismissed: _Why?_
- [ ] Passed
18 changes: 14 additions & 4 deletions .github/workflows/codeql-analysis.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
# CodeQL analysis:
# CodeQL is the analysis engine used by developers to automate security checks,
# and by security researchers to perform variant analysis.
# In CodeQL, code is treated like data. Security vulnerabilities, bugs,
# and other errors are modeled as queries that can be executed against databases
# extracted from code. You can run the standard CodeQL queries, written by GitHub
# researchers and community contributors, or write your own to use in custom analyses.
# Queries that find potential bugs highlight the result directly in the source file.
# https://codeql.github.com/docs/codeql-overview/about-codeql/
# ----------------------------------------------------------------------------
# For most projects, this workflow file will not need changing; you simply need
# to commit it to your repository.
#
Expand Down Expand Up @@ -41,11 +51,11 @@ jobs:

steps:
- name: Checkout repository
uses: actions/checkout@v2
uses: actions/checkout@v3

# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v1
uses: github/codeql-action/init@v2
with:
languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
Expand All @@ -56,7 +66,7 @@ jobs:
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below)
- name: Autobuild
uses: github/codeql-action/autobuild@v1
uses: github/codeql-action/autobuild@v2

# ℹ️ Command-line programs to run using the OS shell.
# 📚 https://git.io/JvXDl
Expand All @@ -70,4 +80,4 @@ jobs:
# make release

- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v1
uses: github/codeql-action/analyze@v2
4 changes: 4 additions & 0 deletions .github/workflows/docker-compose-tests.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
# Pytest
# Runs all test in the dds_web by executing the docker compose yml files for testing.
# The actual pytest command is not in this file.

name: Tests

on:
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/prettier.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
# Prettier
# Prettier is a code formatter. We mostly use it for the markdown files.
# https://prettier.io/

name: Lint with Prettier
on: [push, pull_request]

Expand Down
25 changes: 16 additions & 9 deletions .github/workflows/publish_and_trivyscan.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,18 @@
---
name: Publish Docker Image and run Trivy Security Scan
# GHCR:
# GitHub Container Repository
# Images from ghcr are used when deploying prod and dev.
#
# Trivy
# Trivy is a comprehensive and versatile security scanner. Trivy has scanners that look for
# security issues, and targets where it can find those issues.
#
# This action publishes a docker image and then runs the trivy security scan on that image.
# Potential security issues will be uploaded to the security tab in the repository.
#
# https://github.com/aquasecurity/trivy
# ---------------------------------------
# Publish Docker Image to GHCR and run Trivy Security Scan
name: GHCR and Trivy Scan
on:
pull_request:
push:
Expand All @@ -11,7 +24,7 @@ on:
jobs:
push_to_registry:
if: github.repository == 'ScilifelabDataCentre/dds_web'
name: Push Docker image to Docker Hub
name: Push image
runs-on: ubuntu-latest
permissions:
contents: read
Expand All @@ -23,11 +36,6 @@ jobs:
steps:
- name: Check out the repo
uses: actions/checkout@v3
- name: Log in to Docker Hub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Log in to Github Container Repository
uses: docker/login-action@v2
with:
Expand All @@ -39,7 +47,6 @@ jobs:
uses: docker/metadata-action@v4
with:
images: |
scilifelabdatacentre/dds-backend
ghcr.io/scilifelabdatacentre/dds-backend
- name: Ensure lowercase name
run: echo IMAGE_REPOSITORY=$(echo ${{ github.repository }} | tr '[:upper:]' '[:lower:]') >> $GITHUB_ENV
Expand Down
5 changes: 5 additions & 0 deletions .github/workflows/python-black.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# Lint Python - Black
# This action lints python using black - a python code formatter.
# https://github.com/psf/black
# This does not format the code, this only detects and informs on issues.
# To format with black, run `black .` locally in the repository.
name: Lint Python
on:
push:
Expand Down
45 changes: 45 additions & 0 deletions .github/workflows/snyk-scan.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Snyk IaC:
# A sample workflow which checks out your Infrastructure as Code Configuration files,
# such as Kubernetes, Helm & Terraform and scans them for any security issues.
# The results are then uploaded to GitHub Security Code Scanning
#
# For more examples, including how to limit scans to only high-severity issues
# and fail PR checks, see https://github.com/snyk/actions/
# -----------------------------------------------------------
# This workflow uses actions that are not certified by GitHub.
# They are provided by a third-party and are governed by
# separate terms of service, privacy policy, and support
# documentation.
# ------------------------------------------------------------
name: Snyk IaC

on:
push:
branches: ["dev", master]
pull_request:
# The branches below must be a subset of the branches above
branches: ["dev"]
schedule:
- cron: "0 7,13 * * *"

jobs:
snyk:
permissions:
contents: read
security-events: write
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@master
- name: Run Snyk to check for vulnerabilities
uses: snyk/actions/python@master
continue-on-error: true # To make sure that SARIF upload gets called
env:
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
with:
command: code test
args: --sarif-file-output=snyk.sarif
- name: Upload result to GitHub Code Scanning
uses: github/codeql-action/upload-sarif@v2
with:
sarif_file: snyk.sarif
category: snyk
5 changes: 5 additions & 0 deletions .github/workflows/trivy.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# Trivy:
# Trivy (pronunciation) is a comprehensive and versatile security scanner.
# Trivy has scanners that look for security issues, and targets where it can find those issues.
# https://github.com/aquasecurity/trivy
# ---------------------------------
name: trivy
on:
schedule:
Expand Down
16 changes: 15 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,10 +176,24 @@ Please add a _short_ line describing the PR you make, if the PR implements a spe
- Changed support email ([#1324](https://github.com/ScilifelabDataCentre/dds_web/pull/1324))
- Allow Super Admin login during maintenance ([#1333](https://github.com/ScilifelabDataCentre/dds_web/pull/1333))

## Sprint (2022-12-09 - 2022-12-23)
## Sprint (2022-12-09 - 2023-01-09) - Longer sprint due to Christmas

- Dependency: Bump `certifi` due to CVE-2022-23491 ([#1337](https://github.com/ScilifelabDataCentre/dds_web/pull/1337))
- Dependency: Bump `jwcrypto` due to CVE-2022-3102 ([#1339](https://github.com/ScilifelabDataCentre/dds_web/pull/1339))
- Cronjob: Get number of units and users for reporting ([#1324](https://github.com/ScilifelabDataCentre/dds_web/pull/1335))
- Add ability to change project information via ProjectInfo endpoint ([#1331](https://github.com/ScilifelabDataCentre/dds_web/pull/1331))
- Fix the reporting file path ([1345](https://github.com/ScilifelabDataCentre/dds_web/pull/1345))

## Sprint (2023-01-09 - 2023-01-20)

- Refactoring: Move flask commands to own module `commands.py` ([#1351](https://github.com/ScilifelabDataCentre/dds_web/pull/1351))
- Workflow: Scan with Snyk on PR and schedule ([#1349](https://github.com/ScilifelabDataCentre/dds_web/pull/1349))
- Flask command (cronjob): Monitor unit usage and warn if above level ([#1350](https://github.com/ScilifelabDataCentre/dds_web/pull/1350))

## Sprint (2023-01-20 - 2023-02-03)

- Workflow: Do not publish to DockerHub anymore ([#1357](https://github.com/ScilifelabDataCentre/dds_web/pull/1357))
- Refactoring: move cronjobs previously handled by APScheduler to flask commands ([#1355](https://github.com/ScilifelabDataCentre/dds_web/pull/1355))
- Bug: Fix type issue in 0c9c237cced5 (latest) migration ([#1360](https://github.com/ScilifelabDataCentre/dds_web/pull/1360))
- Database: New `Reporting` table for saving unit / user stats every month ([#1363](https://github.com/ScilifelabDataCentre/dds_web/pull/1363))
- Version bump: 2.2.6 ([#1375](https://github.com/ScilifelabDataCentre/dds_web/pull/1375))
2 changes: 1 addition & 1 deletion Dockerfiles/backend.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ RUN apk add mariadb-client
###################
## BUILD FRONTEND
###################
FROM node:16 as nodebuilder
FROM node:18 as nodebuilder
COPY ./dds_web/static /build
WORKDIR /build
RUN npm install -g npm@latest --quiet
Expand Down
2 changes: 1 addition & 1 deletion Dockerfiles/nodebuilder.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM node:16
FROM node:18
RUN mkdir /build
WORKDIR /build
RUN npm install -g npm@latest --quiet
Expand Down
Loading

0 comments on commit f5b6ffd

Please sign in to comment.