Merge pull request #1493 from ScilifelabDataCentre/dev

New release 2023-11-22

.github/pull_request_template.md

@@ -28,6 +28,8 @@ What _type of change(s)_ does the PR contain?
 - [ ] Database change: _Remember the to include a new migration version, **or** explain here why it's not needed._ <!-- Should be checked when you've changed something in `models.py`. For a guide on how to add the a new migration version, look at the "Database changes" section in the README.md. -->
 - [ ] Bug fix <!-- Should be checked when a bug is fixed in existing functionality. If the bug fix also is a breaking change (see above), add info about that beside this check box. -->
 - [ ] Security Alert fix <!-- Should be checked if the PR attempts to solve a security vulnerability, e.g. reported by the "Security" tab in the repo. -->
+  - [ ] Package update <!-- Should be checked if the Security alert fix consists of updating a package / dependency version -->
+    - [ ] Major version update <!-- Should be checked if the package / dependency version update is a major upgrade, e.g. 1.0.0 to 2.0.0 -->
 - [ ] Documentation <!-- Should be checked if the PR adds or updates documentation such as e.g. Technical Overview or a architecture decision (dds_web/doc/architecture/decisions.) -->
 - [ ] Workflow <!-- Should be checked if the PR includes a change in e.g. the github actions files (dds_web/.github/*) or another type of workflow change. Anything that alters our or the codes workflow. -->
 - [ ] Tests **only** <!-- Should only be checked if the PR only contains tests, none of the other types of changes listed above. -->

CHANGELOG.rst

@@ -1,6 +1,20 @@
 Changelog
 ==========
 
+.. _2.6.0:
+
+2.6.0 - 2023-11-22
+~~~~~~~~~~~~~~~~~~~~~~~
+
+- New endpoint `AddFailedFiles` to allow a retry of saving files to the database after issues during upload.
+- Cronjobs:
+    - Updated command: `quarterly-usage` changed to `monthly-usage` and refactored to catch errors and send emails.
+    - New command `send-usage` to collect usage rows from the `Usage` table and send csv files to support email.
+- Dependencies: 
+    - `Pillow` from `9.3.0` to `10.1.0`
+    - `urllib3` from `1.26.8` to `1.26.18`
+    - `postcss` (npm) from `8.4.28` to `8.4.31`
+
 .. _2.5.2:
 
 2.5.2 - 2023-10-25

SPRINTLOG.md

@@ -315,7 +315,18 @@ _Nothing merged in CLI during this sprint_
 - Add flag --verify-checksum to the comand in email template ([#1478])(https://github.com/ScilifelabDataCentre/dds_web/pull/1478)
 - Improved email layout; Highlighted information and commands when project is released ([#1479])(https://github.com/ScilifelabDataCentre/dds_web/pull/1479)
 
-# 2023-10-16 - 2023-10-27
+# 2023-10-16 - 2023-11-03 (Longer sprint due to OKR prep and höstlov)
 
 - Added new API endpoint ProjectStatus.patch to extend the deadline ([#1480])(https://github.com/ScilifelabDataCentre/dds_web/pull/1480)
 - New version: 2.5.2 ([#1482](https://github.com/ScilifelabDataCentre/dds_web/pull/1482))
+- New endpoint `AddFailedFiles` for adding failed files to database ([#1472](https://github.com/ScilifelabDataCentre/dds_web/pull/1472))
+- Change the generate usage command to monthly instead of quartely, and add the command to send a usage report specifying the number of months ([#1476](https://github.com/ScilifelabDataCentre/dds_web/pull/1476))
+- New ADR record regarding OKR 2024 ([#1483](https://github.com/ScilifelabDataCentre/dds_web/pull/1483))
+
+# 2023-11-6 - 2023-11-17
+
+- Updated Pillow package version to address vulnerabities ([#1486](https://github.com/ScilifelabDataCentre/dds_web/pull/1486))
+- Updated urllib3 package version to address vulnerabities ([#1487](https://github.com/ScilifelabDataCentre/dds_web/pull/1487))
+- Updated PostCss Node package to address vulnerabities ([#1489](https://github.com/ScilifelabDataCentre/dds_web/pull/1489))
+- Updated Several node libraries to address vulnerabities ([#1492](https://github.com/ScilifelabDataCentre/dds_web/pull/1492))
+- New version: 2.6.0 ([#1494](https://github.com/ScilifelabDataCentre/dds_web/pull/1494))

dds_web/__init__.py

@@ -274,7 +274,8 @@ def load_user(user_id):
             set_available_to_expired,
             set_expired_to_archived,
             delete_invites,
-            quarterly_usage,
+            monthly_usage,
+            send_usage,
             collect_stats,
             monitor_usage,
             update_unit,
@@ -291,7 +292,8 @@ def load_user(user_id):
         app.cli.add_command(set_available_to_expired)
         app.cli.add_command(set_expired_to_archived)
         app.cli.add_command(delete_invites)
-        app.cli.add_command(quarterly_usage)
+        app.cli.add_command(monthly_usage)
+        app.cli.add_command(send_usage)
         app.cli.add_command(collect_stats)
         app.cli.add_command(monitor_usage)
 

dds_web/api/__init__.py

@@ -50,6 +50,7 @@ def output_json(data, code, headers=None):
 api.add_resource(files.FileInfo, "/file/info", endpoint="file_info")
 api.add_resource(files.FileInfoAll, "/file/all/info", endpoint="all_file_info")
 api.add_resource(files.UpdateFile, "/file/update", endpoint="update_file")
+api.add_resource(files.AddFailedFiles, "/file/failed/add", endpoint="add_failed_files")
 
 # Projects ############################################################################## Projects #
 api.add_resource(project.UserProjects, "/proj/list", endpoint="list_projects")

dds_web/api/files.py

@@ -740,3 +740,22 @@ def put(self):
             db.session.commit()
 
         return {"message": "File info updated."}
+
+
+class AddFailedFiles(flask_restful.Resource):
+    """Get files from log file and save to database."""
+
+    @auth.login_required(role=["Unit Admin", "Unit Personnel"])
+    @json_required
+    @handle_validation_errors
+    def put(self):
+        """Run flask command with failed_delivery_log."""
+
+        # Verify project ID and access
+        project = project_schemas.ProjectRequiredSchema().load(flask.request.args)
+
+        # Get the request json and pass it to add_uploaded_files_to_db
+        request_json = flask.request.get_json(silent=True)
+
+        files_added, errors = dds_web.utils.add_uploaded_files_to_db(project, request_json)
+        return {"files_added": [file.name for file in files_added], "message": errors}