Merge remote-tracking branch 'IQSS/develop' into

TDL/7493-improve_BagGnerator_failure_handling

.github/workflows/guides_build_sphinx.yml

@@ -2,7 +2,9 @@ name: "Guides Build Status"
 on: 
   pull_request:
     paths:
-      - 'doc/sphinx-guides/**/*.rst'
+      - "doc/sphinx-guides/**/*.rst"
+      - "doc/sphinx-guides/**/requirements.txt"
+      - "doc/sphinx-guides/**/conf.py"
 
 jobs:
   docs:

conf/solr/8.11.1/schema.xml

@@ -315,7 +315,7 @@
     <field name="grantNumber" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="grantNumberAgency" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="grantNumberValue" type="text_en" multiValued="true" stored="true" indexed="true"/>
-    <field name="journalArticleType" type="text_en" multiValued="false" stored="true" indexed="true"/>
+    <field name="journalArticleType" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="journalIssue" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="journalPubDate" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="journalVolume" type="text_en" multiValued="true" stored="true" indexed="true"/>

doc/release-notes/5663-shib-confirm-email.md

@@ -0,0 +1,7 @@
+For Shib users we now set the emailconfirmed timestamp on login. (The guides say we do this already but are wrong. It was only being set on account creation.)
+
+For Shib users, I also prevent "check for your welcome email to verify your address" from being shown in the in-app welcome/new account notification.
+
+I put in a check to make sure Shib users never get a "verify your email address" email notification.
+
+Finally, I removed the hasNoStaleVerificationTokens check from the hasVerifiedEmail method. We've never worried about if there are stale verification tokens in the database or not and this check was preventing "Verified" from being shown, even when the user has a timestamp (the timestamp being the way we know if an email is verified or not).

doc/release-notes/7492_muting_notifications.md

@@ -0,0 +1,9 @@
+This release has a new feature that allows users to control which notifications they receive. How to enable this feature and which options can be set by the admins is described in the [Letting users manage receiving notifications section of the Admin Guide](https://guides.dataverse.org/en/latest/admin/user-administration.html#letting-users-manage-receiving-notifications). See also #7492.
+
+In addition, the existing API endpoint for listing notifications has been enhanced to show the subject, text, and timestamp of notifications. See also #8487.
+
+## New DB Settings
+
+- :ShowMuteOptions
+- :AlwaysMuted
+- :NeverMuted

doc/release-notes/8191-require-toa-or-request-access.md

@@ -0,0 +1,17 @@
+## Release Highlights
+
+### Terms of Access or Request Access Required for Restricted Files
+
+Beginning in this release, Restricted Files must have either Terms of Access or Request Access enabled. This change is to ensure that for each file in a Dataverse installation there is a clear path to get to the data, either through requesting access to the data or to provide context about why requesting access is not enabled.
+
+In the "Notes for Dataverse Installation Administrators" section, we have provided a query to help proactively identify datasets that need to be updated.
+
+## Notes for Dataverse Installation Administrators
+
+### Identifying Datasets Requiring Terms of Access or Request Access Changes
+
+In support of the change to require either Terms of Access or Request Access for all Restricted Files, we have provided a query to identify datasets in your installation where at least one file has neither Terms of Access or Request Access enabled:
+
+https://github.com/IQSS/dataverse/raw/develop/scripts/issues/8191/
+
+This will allow you to reach out to those dataset owners as appropriate.

doc/release-notes/8227-verify-email.md

@@ -0,0 +1 @@
+The "Verify Email" button has been changed to "Send Verification Email" and rather than sometimes showing a popup now always sends a fresh verification email (and invalidates previous verification emails).

doc/release-notes/8295-support-postgresql-14.md

@@ -0,0 +1,8 @@
+This release upgrades the bundled PostgreSQL JDBC driver to support major version 14.
+
+Note that the newer PostgreSQL driver required a Flyway version bump, which entails positive and negative consequences:
+
+- The newer version of Flyway supports PostgreSQL 14 and includes a number of security fixes.
+- As of version 8.0 the Flyway Community Edition dropped support for PostgreSQL 9.6 and older.
+
+Upgrade instructions may be found under “PostgreSQL Update” in the 5.10 release notes: https://github.com/IQSS/dataverse/releases/tag/v5.10

doc/release-notes/8380-counter-processor-update.md

@@ -0,0 +1 @@
+This release includes support for counter-processor-0.1.04 for processing Make Data Count metrics. If you are running Make Data Counts support, you should reinstall/reconfigure counter-processor as described in the latest Guides. (For existing installations, note that counter-processor-0.1.04 requires a newer version of python so you will need to follow the full counter-processor install. Also note that if you configure the new version the same way, it will reprocess the days in the current month when it is first run. This is normal and will not affect the metrics in Dataverse.)

doc/release-notes/8456-upgrade-primefaces.md

@@ -0,0 +1,4 @@
+### Primefaces 11
+
+Primefaces, the open source UI framework upon which the Dataverse front end is built, has been updated to the most recent version. This provides security updates and bug fixes and will also allow Dataverse developers to take advantage of new features and enhancements.
+

doc/release-notes/8533_semantic-api-updates.md

@@ -0,0 +1,14 @@
+
+## Notes for Developers and Integrators
+
+This release include an update to the experimental semantic API and the underlying assignment of URIs to metadatablock termshat are not explicitly mapped to terms in community vocabularies. The change affects the output of the OAI_ORE metadata export, the OAI_ORE file in archival bags, and the input/out allowed for those terms in the semantic api. For those updating integrating code or existing files intended for input into this release of Dataverse: URIs of the form:
+  https://dataverse.org/schema/<block name>/<parentField name>#<childField title>, and
+  https://dataverse.org/schema/<block name>/<Field title>
+  are both replaced with URIs of the form:
+  https://dataverse.org/schema/<block name>/<Field name>
+
+## Additional Release Steps
+
+Upgrade should include re-export of metadata files (only the OAI_ORE is affected).
+
+For this PR and other changes coming from DataCommons, it will also be advisable for people archiving Bags to re-archive. More detail on the overall set if changes in those tbd PRs.

doc/release-notes/8561-license-info-in-submit-for-review-pop-up.md

@@ -0,0 +1 @@
+Information about the license selection (or custom terms) is now available in the confirmation popup when contributors click "Submit for Review". Previously, this was only available in the confirmation popup for the "Publish" button, which contributors do not see.

doc/release-notes/8595-cvv-field-solr-update.md

@@ -0,0 +1,3 @@
+Controlled vocabulary fields that do not allow multiple entries are not indexed properly in Dataverse instances configured to support multiple languages. This release fixes the schema.xml file for the one field affected in the standard metadata blocks (journalArticleType) and updates the api/admin/index/solr/schema to provide the correct information for use with the update-fields.sh script described in the [Metadata Customization section of the Admin Guide](https://guides.dataverse.org/en/latest/admin/metadatacustomization.html#updating-the-solr-schema).
+
+The release should include updating the schema.xml file for solr or running the update-fields.sh script and reindexing (whatever standard instructions we give for schema changes.)

doc/release-notes/8600-duplicate-templates.md

@@ -0,0 +1,9 @@
+###Duplicate Templates in Database
+Prior to this release making a copy of a Dataset Template was creating two copies, one of which is visible in the dataverse, the other not being assigned 
+a dataverse was invisible to the user (https://github.com/IQSS/dataverse/issues/8600). This release fixes the issue. 
+
+If you would like to remove these orphan templates you may run the follwing script:
+
+https://github.com/IQSS/dataverse/raw/develop/scripts/issues/8600/delete_orphan_templates_8600.sh
+
+Also, admin APIs for finding and deleting templates have been added.

doc/release-notes/8608-bagit-support-validate-checksums.md

@@ -0,0 +1,10 @@
+## BagIt Support - Automatic checksum validation on zip file upload
+The BagIt file handler detects and transforms zip files with a BagIt package format into Dataverse DataFiles. The system validates the checksums of the files in the package payload as described in the first manifest file with a hash algorithm that we support. Take a look at `BagChecksumType class <https://github.com/IQSS/dataverse/tree/develop/src/main/java/edu/harvard/iq/dataverse/util/bagit/BagChecksumType.java>`_ for the list of the currently supported hash algorithms.
+
+The handler will not allow packages with checksum errors. The first 5 errors will be displayed to the user. This is configurable though database settings.
+
+The checksum validation uses a thread pool to improve performance. This thread pool can be adjusted to your Dataverse installation requirements.
+
+The BagIt file handler is disabled by default. Use the ``:BagItHandlerEnabled`` database settings to enable it: ``curl -X PUT -d 'true' http://localhost:8080/api/admin/settings/:BagItHandlerEnabled``
+
+For more configuration settings see the user guide: https://guides.dataverse.org/en/latest/installation/config.html#bagit-file-handler

doc/release-notes/8663-update-create-ds-doc.md

@@ -0,0 +1,5 @@
+## Release Highlights
+
+### Documentation of Create Dataset API has been Updated
+
+Previous versions of the guides omitted the content header in the example API call for creating a datset. Due to a code change introduced in Dataverse version 5.6 calls without the content header will fail.

doc/release-notes/ds54-csp.md

@@ -0,0 +1 @@
+As always, widgets can be embedded in the `<iframe>` HTML tag, but the HTTP header "Content-Security-Policy" is now being sent on non-widget pages to prevent them from being embedded.

doc/sphinx-guides/requirements.txt

@@ -1,2 +1,7 @@
 # current version as of this writing
 Sphinx==3.5.4
+# Necessary workaround for ReadTheDocs for Sphinx 3.x - unnecessary as of Sphinx 4.5+
+Jinja2>=3.0.2,<3.1
+
+# Sphinx - Additional modules
+sphinx-icon==0.1.2

doc/sphinx-guides/source/_static/api/dataset-create.jsonld

@@ -2,14 +2,14 @@
   "http://purl.org/dc/terms/title": "Darwin's Finches",
   "http://purl.org/dc/terms/subject": "Medicine, Health and Life Sciences",
   "http://purl.org/dc/terms/creator": {
-      "https://dataverse.org/schema/citation/author#Name": "Finch, Fiona",
-      "https://dataverse.org/schema/citation/author#Affiliation": "Birds Inc."
+      "https://dataverse.org/schema/citation/authorName": "Finch, Fiona",
+      "https://dataverse.org/schema/citation/authorAffiliation": "Birds Inc."
   },
-  "https://dataverse.org/schema/citation/Contact": {
-    "https://dataverse.org/schema/citation/datasetContact#E-mail": "[email protected]",
-    "https://dataverse.org/schema/citation/datasetContact#Name": "Finch, Fiona"
+  "https://dataverse.org/schema/citation/datasetContact": {
+    "https://dataverse.org/schema/citation/datasetContactEmail": "[email protected]",
+    "https://dataverse.org/schema/citation/datasetContactName": "Finch, Fiona"
   },
-  "https://dataverse.org/schema/citation/Description": {
-    "https://dataverse.org/schema/citation/dsDescription#Text": "Darwin's finches (also known as the Galápagos finches) are a group of about fifteen species of passerine birds."
+  "https://dataverse.org/schema/citation/dsDescription": {
+    "https://dataverse.org/schema/citation/dsDescriptionValue": "Darwin's finches (also known as the Galápagos finches) are a group of about fifteen species of passerine birds."
   }
 }

doc/sphinx-guides/source/_static/api/dataset-migrate.jsonld

@@ -1,25 +1,25 @@
 {
-"citation:Depositor": "Admin, Dataverse",
-"Title": "Test Dataset",
-"Subject": "Computer and Information Science",
-"Creator": {
-  "author:Name": "Admin, Dataverse",
-  "author:Affiliation": "GDCC"
+"citation:depositor": "Admin, Dataverse",
+"title": "Test Dataset",
+"subject": "Computer and Information Science",
+"author": {
+  "citation:authorName": "Admin, Dataverse",
+  "citation:authorAffiliation": "GDCC"
 },
-"Deposit Date": "2020-10-08",
-"citation:Distributor": {
-  "distributor:Name": "Demo Dataverse Repository",
-  "distributor:Affiliation": "Dataverse Community",
-  "distributor:Abbreviation": "GDCC",
-  "distributor:URL": "https://dataverse.org/global-dataverse-community-consortium"
+"dateOfDeposit": "2020-10-08",
+"citation:distributor": {
+  "citation:distributorName": "Demo Dataverse Repository",
+  "citation:distributorAffiliation": "Dataverse Community",
+  "citation:distributorAbbreviation": "GDCC",
+  "citation:distributorURL": "https://dataverse.org/global-dataverse-community-consortium"
 },
-"citation:Contact": {
-"datasetContact:Name": "Admin, Dataverse",
-"datasetContact:Affiliation": "GDCC",
-"datasetContact:E-mail": "[email protected]"
+"citation:datasetContact": {
+"citation:datasetContactName": "Admin, Dataverse",
+"citation:datasetContactAffiliation": "GDCC",
+"citation:datasetContactEmail": "[email protected]"
 },
-"citation:Description": {
-  "dsDescription:Text": "A short description"
+"citation:dsDescription": {
+  "citation:dsDescriptionValue": "A short description"
 },
 "@id": "doi:10.33564/FK27U7YBV",
 "schema:version": "1.0",
@@ -29,15 +29,11 @@
 "dvcore:fileRequestAccess": false
 },
 "@context": {
-  "Creator": "http://purl.org/dc/terms/creator",
-  "Deposit Date": "http://purl.org/dc/terms/dateSubmitted",
-  "Subject": "http://purl.org/dc/terms/subject",
-  "Title": "http://purl.org/dc/terms/title",
-  "author": "https://dataverse.org/schema/citation/author#",
+  "author": "http://purl.org/dc/terms/creator",
+  "dateOfDeposit": "http://purl.org/dc/terms/dateSubmitted",
+  "subject": "http://purl.org/dc/terms/subject",
+  "title": "http://purl.org/dc/terms/title",
   "citation": "https://dataverse.org/schema/citation/",
-  "datasetContact": "https://dataverse.org/schema/citation/datasetContact#",
-  "distributor": "https://dataverse.org/schema/citation/distributor#",
-  "dsDescription": "https://dataverse.org/schema/citation/dsDescription#",
   "dvcore": "https://dataverse.org/schema/core#",
   "schema": "http://schema.org/"
 }}

doc/sphinx-guides/source/_static/docsdataverse_org.css

@@ -60,7 +60,9 @@ ul.navbar-nav li.dropdown.open > span.dropdown-toggle {background: #e7e7e7;}
 }
 
 a.headerlink {
-	display: none;
+    font-size: 60%;
+    vertical-align: middle;
+    color: #d14e6f;
 }
 
 #sidebar.bs-sidenav {