Merge pull request #64 from IQSS/develop

Update

doc/release-notes/7492_muting_notifications.md

@@ -0,0 +1,9 @@
+This release has a new feature that allows users to control which notifications they receive. How to enable this feature and which options can be set by the admins is described in the [Letting users manage receiving notifications section of the Admin Guide](https://guides.dataverse.org/en/latest/admin/user-administration.html#letting-users-manage-receiving-notifications). See also #7492.
+
+In addition, the existing API endpoint for listing notifications has been enhanced to show the subject, text, and timestamp of notifications. See also #8487.
+
+## New DB Settings
+
+- :ShowMuteOptions
+- :AlwaysMuted
+- :NeverMuted

doc/release-notes/8608-bagit-support-validate-checksums.md

@@ -0,0 +1,10 @@
+## BagIt Support - Automatic checksum validation on zip file upload
+The BagIt file handler detects and transforms zip files with a BagIt package format into Dataverse DataFiles. The system validates the checksums of the files in the package payload as described in the first manifest file with a hash algorithm that we support. Take a look at `BagChecksumType class <https://github.com/IQSS/dataverse/tree/develop/src/main/java/edu/harvard/iq/dataverse/util/bagit/BagChecksumType.java>`_ for the list of the currently supported hash algorithms.
+
+The handler will not allow packages with checksum errors. The first 5 errors will be displayed to the user. This is configurable though database settings.
+
+The checksum validation uses a thread pool to improve performance. This thread pool can be adjusted to your Dataverse installation requirements.
+
+The BagIt file handler is disabled by default. Use the ``:BagItHandlerEnabled`` database settings to enable it: ``curl -X PUT -d 'true' http://localhost:8080/api/admin/settings/:BagItHandlerEnabled``
+
+For more configuration settings see the user guide: https://guides.dataverse.org/en/latest/installation/config.html#bagit-file-handler

doc/sphinx-guides/source/admin/user-administration.rst

@@ -75,3 +75,50 @@ Using the API token 7ae33670-be21-491d-a244-008149856437 as an example:
 ``delete from apitoken where tokenstring = '7ae33670-be21-491d-a244-008149856437';``
 
 You should expect the output ``DELETE 1`` after issuing the command above.
+
+.. _mute-notifications:
+
+Letting Users Manage Notifications
+-----------------------------------
+
+See :ref:`account-notifications` in the User Guide for how notifications are described to end users.
+
+You can let users manage which notification types they wish to receive by setting :ref:`:ShowMuteOptions` to "true":
+
+``curl -X PUT -d 'true' http://localhost:8080/api/admin/settings/:ShowMuteOptions``
+
+This enables additional settings for each user in the notifications tab of their account page. The users can select which in-app notifications and/or e-mails they wish to receive out of the following list:
+
+* ``APIGENERATED`` API token is generated
+* ``ASSIGNROLE`` Role is assigned
+* ``CHECKSUMFAIL`` Checksum validation failed
+* ``CHECKSUMIMPORT`` Dataset had file checksums added via a batch job
+* ``CONFIRMEMAIL`` Email Verification
+* ``CREATEACC`` Account is created
+* ``CREATEDS`` Your dataset is created
+* ``CREATEDV`` Dataverse collection is created
+* ``DATASETCREATED`` Dataset was created by user
+* ``FILESYSTEMIMPORT`` Dataset has been successfully uploaded and verified
+* ``GRANTFILEACCESS`` Access to file is granted
+* ``INGESTCOMPLETEDWITHERRORS`` Ingest completed with errors
+* ``INGESTCOMPLETED`` Ingest is completed
+* ``PUBLISHEDDS`` Dataset is published
+* ``PUBLISHFAILED_PIDREG`` Publish has failed
+* ``REJECTFILEACCESS`` Access to file is rejected
+* ``REQUESTFILEACCESS`` Access to file is requested
+* ``RETURNEDDS`` Returned from review
+* ``REVOKEROLE`` Role is revoked
+* ``STATUSUPDATED`` Status of dataset has been updated
+* ``SUBMITTEDDS`` Submitted for review
+* ``WORKFLOW_FAILURE`` External workflow run has failed
+* ``WORKFLOW_SUCCESS`` External workflow run has succeeded
+
+After enabling this feature, all notifications are enabled by default, until this is changed by the user.
+
+You can shorten this list by configuring some notification types (e.g., ``ASSIGNROLE`` and ``REVOKEROLE``) to be always muted for everyone and not manageable by users (not visible in the user interface) with the :ref:`:AlwaysMuted` setting:
+
+``curl -X PUT -d 'ASSIGNROLE,REVOKEROLE' http://localhost:8080/api/admin/settings/:AlwaysMuted``
+
+Finally, you can set some notifications (e.g., ``REQUESTFILEACCESS``, ``GRANTFILEACCESS`` and ``REJECTFILEACCESS``) as always enabled for everyone and not manageable by users (grayed out in the user interface) with the :ref:`:NeverMuted` setting:
+
+``curl -X PUT -d 'REQUESTFILEACCESS,GRANTFILEACCESS,REJECTFILEACCESS' http://localhost:8080/api/admin/settings/:NeverMuted``

doc/sphinx-guides/source/api/native-api.rst

@@ -2836,13 +2836,90 @@ Show Info About Single Metadata Block
 Notifications
 -------------
 
+See :ref:`account-notifications` in the User Guide for an overview. For a list of all the notification types mentioned below (e.g. ASSIGNROLE), see :ref:`mute-notifications` in the Admin Guide.
+
 Get All Notifications by User
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Each user can get a dump of their notifications by passing in their API token::
+Each user can get a dump of their notifications by passing in their API token:
+
+.. code-block:: bash
+
+  curl -H "X-Dataverse-key:$API_TOKEN" $SERVER_URL/api/notifications/all
+
+Delete Notification by User
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each user can delete notifications by passing in their API token and specifying notification ID (e.g., 555):
+
+.. code-block:: bash
+
+  export NOTIFICATION_ID=555
+
+  curl -H X-Dataverse-key:$API_TOKEN -X DELETE "$SERVER_URL/api/notifications/$NOTIFICATION_ID"
+
+Get All Muted In-app Notifications by User
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each user can get a list of their muted in-app notification types by passing in their API token:
+
+.. code-block:: bash
+
+  curl -H X-Dataverse-key:$API_TOKEN -X GET "$SERVER_URL/api/notifications/mutedNotifications"
+
+Mute In-app Notification by User
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each user can mute in-app notifications by passing in their API token and specifying notification type to be muted (e.g., ASSIGNROLE):
+
+.. code-block:: bash
+
+  export NOTIFICATION_TYPE=ASSIGNROLE
+
+  curl -H X-Dataverse-key:$API_TOKEN -X PUT "$SERVER_URL/api/notifications/mutedNotifications/$NOTIFICATION_TYPE"
+
+Unmute In-app Notification by User
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each user can unmute in-app notifications by passing in their API token and specifying notification type to be unmuted (e.g., ASSIGNROLE):
+
+.. code-block:: bash
+
+  export NOTIFICATION_TYPE=ASSIGNROLE
+
+  curl -H X-Dataverse-key:$API_TOKEN -X DELETE "$SERVER_URL/api/notifications/mutedNotifications/$NOTIFICATION_TYPE"
+
+Get All Muted Email Notifications by User
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each user can get a list of their muted email notification types by passing in their API token:
+
+.. code-block:: bash
+
+  curl -H X-Dataverse-key:$API_TOKEN -X GET "$SERVER_URL/api/notifications/mutedEmails"
+
+Mute Email Notification by User
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each user can mute email notifications by passing in their API token and specifying notification type to be muted (e.g., ASSIGNROLE):
+
+.. code-block:: bash
+
+  export NOTIFICATION_TYPE=ASSIGNROLE
+
+  curl -H X-Dataverse-key:$API_TOKEN -X PUT "$SERVER_URL/api/notifications/mutedEmails/$NOTIFICATION_TYPE"
+
+Unmute Email Notification by User
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each user can unmute email notifications by passing in their API token and specifying notification type to be unmuted (e.g., ASSIGNROLE):
+
+.. code-block:: bash
+
+  export NOTIFICATION_TYPE=ASSIGNROLE
+
+  curl -H X-Dataverse-key:$API_TOKEN -X DELETE "$SERVER_URL/api/notifications/mutedEmails/$NOTIFICATION_TYPE"
 
-    curl -H "X-Dataverse-key:$API_TOKEN" $SERVER_URL/api/notifications/all
-    
 .. _User Information:
 
 User Information

doc/sphinx-guides/source/developers/sql-upgrade-scripts.rst

@@ -26,7 +26,7 @@ How to Create a SQL Upgrade Script
 
 We assume you have already read the :doc:`version-control` section and have been keeping your feature branch up to date with the "develop" branch.
 
-Create a new file called something like ``V4.11.0.1__5565-sanitize-directory-labels.sql`` in the ``src/main/resources/db/migration`` directory. Use a version like "4.11.0.1" in the example above where the previously released version was 4.11, ensuring that the version number is unique. Note that this is not the version that you expect the code changes to be included in (4.12 in this example). For the "description" you should the name of your branch, which should include the GitHub issue you are working on, as in the example above. To read more about Flyway file naming conventions, see https://flywaydb.org/documentation/migrations#naming
+Create a new file called something like ``V4.11.0.1__5565-sanitize-directory-labels.sql`` in the ``src/main/resources/db/migration`` directory. Use a version like "4.11.0.1" in the example above where the previously released version was 4.11, ensuring that the version number is unique. Note that this is not the version that you expect the code changes to be included in (4.12 in this example). When the previously released version is a patch version (e.g. 5.10.1), use "5.10.1.1" for the first SQL script version (rather than "5.10.1.0.1"). For the "description" you should the name of your branch, which should include the GitHub issue you are working on, as in the example above. To read more about Flyway file naming conventions, see https://flywaydb.org/documentation/migrations#naming
 
 The SQL migration script you wrote will be part of the war file and executed when the war file is deployed. To see a history of Flyway database migrations that have been applied, look at the ``flyway_schema_history`` table.
 

doc/sphinx-guides/source/installation/config.rst

@@ -698,6 +698,10 @@ The image below indicates that the page layout consists of three main blocks: a
 
 |dvPageBlocks|
 
+.. To edit, use dvBrandingCustBlocks.drawio with https://app.diagrams.net
+.. |dvPageBlocks| image:: ./img/dvBrandingCustBlocks.png
+   :class: img-responsive
+
 Installation Name/Brand Name
 ++++++++++++++++++++++++++++
 
@@ -1038,6 +1042,22 @@ Disabling Custom Dataset Terms
 
 See :ref:`:AllowCustomTermsOfUse` for how to disable the "Custom Dataset Terms" option.
 
+.. _BagIt File Handler:
+
+BagIt File Handler
+------------------
+
+BagIt file handler detects and transforms zip files with a BagIt package format into Dataverse DataFiles. The system validates the checksums of the files in the package payload as described in the first manifest file with a hash algorithm that we support. Take a look at `BagChecksumType class <https://github.com/IQSS/dataverse/tree/develop/src/main/java/edu/harvard/iq/dataverse/util/bagit/BagChecksumType.java>`_ for the list of the currently supported hash algorithms.
+
+The checksum validation uses a thread pool to improve performance. This thread pool can be adjusted to your Dataverse installation requirements.
+
+BagIt file handler configuration settings:
+
+- :ref:`:BagItHandlerEnabled`
+- :ref:`:BagValidatorJobPoolSize`
+- :ref:`:BagValidatorMaxErrors`
+- :ref:`:BagValidatorJobWaitInterval`
+
 .. _BagIt Export:
 
 BagIt Export
@@ -2536,6 +2556,49 @@ To enable redirects to the zipper on a different server:
 
 ``curl -X PUT -d 'https://zipper.example.edu/cgi-bin/zipdownload' http://localhost:8080/api/admin/settings/:CustomZipDownloadServiceUrl``
 
+:CreateDataFilesMaxErrorsToDisplay
+++++++++++++++++++++++++++++++++++
+
+Number of errors to display to the user when creating DataFiles from a file upload. It defaults to 5 errors.
+
+``curl -X PUT -d '1' http://localhost:8080/api/admin/settings/:CreateDataFilesMaxErrorsToDisplay``
+
+.. _:BagItHandlerEnabled:
+
+:BagItHandlerEnabled
++++++++++++++++++++++
+
+Part of the database settings to configure the BagIt file handler. Enables the BagIt file handler. By default, the handler is disabled.
+
+``curl -X PUT -d 'true' http://localhost:8080/api/admin/settings/:BagItHandlerEnabled``
+
+.. _:BagValidatorJobPoolSize:
+
+:BagValidatorJobPoolSize
+++++++++++++++++++++++++
+
+Part of the database settings to configure the BagIt file handler. The number of threads the checksum validation class uses to validate a single zip file. Defaults to 4 threads
+
+``curl -X PUT -d '10' http://localhost:8080/api/admin/settings/:BagValidatorJobPoolSize``
+
+.. _:BagValidatorMaxErrors:
+
+:BagValidatorMaxErrors
+++++++++++++++++++++++
+
+Part of the database settings to configure the BagIt file handler. The maximum number of errors allowed before the validation job aborts execution. This is to avoid processing the whole BagIt package. Defaults to 5 errors.
+
+``curl -X PUT -d '2' http://localhost:8080/api/admin/settings/:BagValidatorMaxErrors``
+
+.. _:BagValidatorJobWaitInterval:
+
+:BagValidatorJobWaitInterval
+++++++++++++++++++++++++++++
+
+Part of the database settings to configure the BagIt file handler. This is the period in seconds to check for the number of errors during validation. Defaults to 10.
+
+``curl -X PUT -d '60' http://localhost:8080/api/admin/settings/:BagValidatorJobWaitInterval``
+
 :ArchiverClassName
 ++++++++++++++++++
 
@@ -2760,6 +2823,29 @@ To remove the override and go back to the default list:
 
 ``curl -X PUT -d '' http://localhost:8080/api/admin/settings/:FileCategories``
 
-.. To edit, use dvBrandingCustBlocks.drawio with https://app.diagrams.net
-.. |dvPageBlocks| image:: ./img/dvBrandingCustBlocks.png
-   :class: img-responsive
+.. _:ShowMuteOptions:
+
+:ShowMuteOptions
+++++++++++++++++
+
+Allows users to mute notifications by showing additional configuration options in the Notifications tab of the account page (see :ref:`account-notifications` in the User Guide). By default, this setting is "false" and users cannot mute any notifications (this feature is not shown in the user interface).
+
+For configuration details, see :ref:`mute-notifications`.
+
+.. _:AlwaysMuted:
+
+:AlwaysMuted
+++++++++++++
+
+Overrides the default empty list of always muted notifications. Always muted notifications cannot be unmuted by the users. Always muted notifications are not shown in the notification settings for the users.
+
+For configuration details, see :ref:`mute-notifications`.
+
+.. _:NeverMuted:
+
+:NeverMuted
++++++++++++
+
+Overrides the default empty list of never muted notifications. Never muted notifications cannot be muted by the users. Always muted notifications are grayed out and are not adjustable by the user.
+
+For configuration details, see :ref:`mute-notifications`.

doc/sphinx-guides/source/user/account.rst

@@ -155,19 +155,27 @@ The My Data section of your account page displays a listing of all the Dataverse
 
 You can use the Add Data button to create a new Dataverse collection or dataset. By default, the new Dataverse collection or dataset will be created in the root Dataverse collection, but from the create form you can use the Host Dataverse collection dropdown menu to choose a different Dataverse collection, for which you have the proper access privileges. However, you will not be able to change this selection after you create your Dataverse collection or dataset.
 
+.. _account-notifications:
+
 Notifications
 -------------
 
-Notifications appear in the notifications tab on your account page and are also displayed as a number next to your account name.
+Notifications appear in the notifications tab on your account page and are also displayed as a number next to your account name. You also receive notifications via email.
+
+If your admin has enabled the option to change the notification settings, you will find an overview of the notification and email settings in the notifications tab. There, you can select which notifications and/or emails you wish to receive. If certain notification or email options are greyed out, you can’t change the setting for this notification because the admin has set these as never to be muted by the user. You control the in-app and the email notifications separately in the two lists.
 
-You will receive a notification when:
+You will typically receive a notification or email when:
 
 - You've created your account.
 - You've created a Dataverse collection or added a dataset.
-- Another Dataverse installation user has requested access to restricted files in a dataset that you published. (If you submitted your dataset for review and it was published by a curator, the curators of the Dataverse collection that contains your dataset will get a notification about requests to access your restricted files.)
+- Another Dataverse installation user has requested access to restricted files in a dataset that you published. (If you submitted your dataset for review, and it was published by a curator, the curators of the Dataverse collection that contains your dataset will get a notification about requests to access your restricted files.)
 - A file in one of your datasets has finished the ingest process.
 
-Notifications will only be emailed one time even if you haven't read the notification on the Dataverse installation.
+There are other notification types that you can receive, e.g., notification on granted roles, API key generation, etc. These types of notifications are less common and are not described here. Some other notifications are limited to specific roles. For example, if the installation has a curation workflow, reviewers get notified when a new dataset is submitted for review.
+
+Notifications will only be emailed once, even if you haven't read the in-app notification.
+
+It's possible to manage notifications via API. See :ref:`notifications` in the API Guide.
 
 API Token
 ---------

src/main/java/edu/harvard/iq/dataverse/DatasetPage.java

@@ -1660,15 +1660,7 @@ public void setSelectedTemplate(Template selectedTemplate) {
     public void updateSelectedTemplate(ValueChangeEvent event) {
 
         selectedTemplate = (Template) event.getNewValue();
-        if (selectedTemplate != null) {
-            //then create new working version from the selected template
-            workingVersion.updateDefaultValuesFromTemplate(selectedTemplate);
-            updateDatasetFieldInputLevels();
-        } else {
-            workingVersion.initDefaultValues(licenseServiceBean.getDefault());
-            updateDatasetFieldInputLevels();
-        }
-        resetVersionUI();
+
     }
 
     /*
@@ -1762,6 +1754,21 @@ public void handleChange() {
 
     public void handleChangeButton() {
 
+        if (selectedTemplate != null) {
+            //then create new working version from the selected template
+            workingVersion.updateDefaultValuesFromTemplate(selectedTemplate);
+            updateDatasetFieldInputLevels();
+        } else {
+            workingVersion.initDefaultValues(licenseServiceBean.getDefault());
+            updateDatasetFieldInputLevels();
+        }
+        /*
+        Issue 8646: necessary for the access popup which is shared by the dataset page and the file page
+        */
+        setFileAccessRequest(workingVersion.getTermsOfUseAndAccess().isFileAccessRequest());
+        setTermsOfAccess(workingVersion.getTermsOfUseAndAccess().getTermsOfAccess());
+
+        resetVersionUI();
     }
 
     public boolean isShapefileType(FileMetadata fm) {