Merge pull request #62 from IQSS/develop

Update

.github/workflows/guides_build_sphinx.yml

@@ -2,7 +2,9 @@ name: "Guides Build Status"
 on: 
   pull_request:
     paths:
-      - 'doc/sphinx-guides/**/*.rst'
+      - "doc/sphinx-guides/**/*.rst"
+      - "doc/sphinx-guides/**/requirements.txt"
+      - "doc/sphinx-guides/**/conf.py"
 
 jobs:
   docs:

conf/docker-aio/1prep.sh

@@ -12,10 +12,10 @@ cd ../../
 cp -r scripts conf/docker-aio/testdata/
 cp doc/sphinx-guides/source/_static/util/createsequence.sql conf/docker-aio/testdata/doc/sphinx-guides/source/_static/util/
 
-wget -q https://downloads.apache.org/maven/maven-3/3.8.4/binaries/apache-maven-3.8.4-bin.tar.gz
-tar xfz apache-maven-3.8.4-bin.tar.gz
+wget -q https://downloads.apache.org/maven/maven-3/3.8.5/binaries/apache-maven-3.8.5-bin.tar.gz
+tar xfz apache-maven-3.8.5-bin.tar.gz
 mkdir maven
-mv apache-maven-3.8.4/* maven/
+mv apache-maven-3.8.5/* maven/
 echo "export JAVA_HOME=/usr/lib/jvm/jre-openjdk" > maven/maven.sh
 echo "export M2_HOME=../maven" >> maven/maven.sh
 echo "export MAVEN_HOME=../maven" >> maven/maven.sh

conf/solr/8.11.1/schema.xml

@@ -315,7 +315,7 @@
     <field name="grantNumber" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="grantNumberAgency" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="grantNumberValue" type="text_en" multiValued="true" stored="true" indexed="true"/>
-    <field name="journalArticleType" type="text_en" multiValued="false" stored="true" indexed="true"/>
+    <field name="journalArticleType" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="journalIssue" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="journalPubDate" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="journalVolume" type="text_en" multiValued="true" stored="true" indexed="true"/>

doc/release-notes/5.10.1-release-notes.md

@@ -0,0 +1,96 @@
+# Dataverse Software 5.10.1
+
+This release brings new features, enhancements, and bug fixes to the Dataverse Software. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project.
+
+## Release Highlights
+
+### Bug Fix for Request Access
+
+Dataverse Software 5.10 contains a bug where the "Request Access" button doesn't work from the file listing on the dataset page if the dataset contains custom terms. This has been fixed in PR #8555.
+
+### Bug Fix for Searching and Selecting Controlled Vocabulary Values
+
+Dataverse Software 5.10 contains a bug where the search option is no longer present when selecting from more than ten controlled vocabulary values. This has been fixed in PR #8521.
+
+## Major Use Cases and Infrastructure Enhancements
+
+Changes and fixes in this release include:
+
+- Users can use the "Request Access" button when the dataset has custom terms. (Issue #8553, PR #8555)
+- Users can search when selecting from more than ten controlled vocabulary values. (Issue #8519, PR #8521)
+- The default file categories ("Documentation", "Data", and "Code") can be redefined through the `:FileCategories` database setting. (Issue #8461, PR #8478)
+- Documentation on troubleshooting Excel ingest errors was improved. (PR #8541)
+- Internationalized controlled vocabulary values can now be searched. (Issue #8286, PR #8435)
+- Curation labels can be internationalized. (Issue #8381, PR #8466)
+- "NONE" is no longer accepted as a license using the SWORD API (since 5.10). See "Backward Incompatibilities" below for details. (Issue #8551, PR #8558).
+
+## Notes for Dataverse Installation Administrators
+
+### PostgreSQL Version 10+ Required Soon
+
+Because 5.10.1 is a bug fix release, an upgrade to PostgreSQL is not required. However, this upgrade is still coming in the next non-bug fix release. For details, please see the release notes for 5.10: https://github.com/IQSS/dataverse/releases/tag/v5.10
+
+### Payara Upgrade
+
+You may notice that the Payara version used in the install scripts has been updated from 5.2021.5 to 5.2021.6. This was to address a bug where it was not possible to easily update the logging level. For existing installations, this release does not require upgrading Payara and a Payara upgrade is not part of the Upgrade Instructions below. For more information, see PR #8508.
+
+## New JVM Options and DB Settings
+
+The following DB settings have been added:
+
+- `:FileCategories` - The default list of the pre-defined file categories ("Documentation", "Data" and "Code") can now be redefined with a comma-separated list (e.g. `'Docs,Data,Code,Workflow'`).
+
+See the [Database Settings](https://guides.dataverse.org/en/5.10.1/installation/config.html#database-settings) section of the Guides for more information.
+
+## Notes for Developers and Integrators
+
+In the "Backward Incompatibilities" section below, note changes in the API regarding licenses and the SWORD API.
+
+## Backward Incompatibilities
+
+As of Dataverse 5.10, "NONE" is no longer supported as a valid license when creating a dataset using the SWORD API. The API Guide has been updated to reflect this. Additionally, if you specify an invalid license, a list of available licenses will be returned in the response.
+
+## Complete List of Changes
+
+For the complete list of code changes in this release, see the [5.10.1 Milestone](https://github.com/IQSS/dataverse/milestone/102?closed=1) in Github.
+
+For help with upgrading, installing, or general questions please post to the [Dataverse Community Google Group](https://groups.google.com/forum/#!forum/dataverse-community) or email [email protected].
+
+## Installation
+
+If this is a new installation, please see our [Installation Guide](https://guides.dataverse.org/en/5.10.1/installation/). Please also contact us to get added to the [Dataverse Project Map](https://guides.dataverse.org/en/5.10.1/installation/config.html#putting-your-dataverse-installation-on-the-map-at-dataverse-org) if you have not done so already.
+
+## Upgrade Instructions
+
+0\. These instructions assume that you've already successfully upgraded from Dataverse Software 4.x to Dataverse Software 5 following the instructions in the [Dataverse Software 5 Release Notes](https://github.com/IQSS/dataverse/releases/tag/v5.0). After upgrading from the 4.x series to 5.0, you should progress through the other 5.x releases before attempting the upgrade to 5.10.1.
+
+If you are running Payara as a non-root user (and you should be!), **remember not to execute the commands below as root**. Use `sudo` to change to that user first. For example, `sudo -i -u dataverse` if `dataverse` is your dedicated application user.  
+
+In the following commands we assume that Payara 5 is installed in `/usr/local/payara5`. If not, adjust as needed.
+
+`export PAYARA=/usr/local/payara5`
+
+(or `setenv PAYARA /usr/local/payara5` if you are using a `csh`-like shell)
+
+1\. Undeploy the previous version.
+
+- `$PAYARA/bin/asadmin list-applications`
+- `$PAYARA/bin/asadmin undeploy dataverse<-version>`
+
+2\. Stop Payara and remove the generated directory
+
+- `service payara stop`
+- `rm -rf $PAYARA/glassfish/domains/domain1/generated`
+
+3\. Start Payara
+
+- `service payara start`
+
+4\. Deploy this version.
+
+- `$PAYARA/bin/asadmin deploy dataverse-5.10.1.war`
+
+5\. Restart payara
+
+- `service payara stop`
+- `service payara start`

doc/release-notes/5663-shib-confirm-email.md

@@ -0,0 +1,7 @@
+For Shib users we now set the emailconfirmed timestamp on login. (The guides say we do this already but are wrong. It was only being set on account creation.)
+
+For Shib users, I also prevent "check for your welcome email to verify your address" from being shown in the in-app welcome/new account notification.
+
+I put in a check to make sure Shib users never get a "verify your email address" email notification.
+
+Finally, I removed the hasNoStaleVerificationTokens check from the hasVerifiedEmail method. We've never worried about if there are stale verification tokens in the database or not and this check was preventing "Verified" from being shown, even when the user has a timestamp (the timestamp being the way we know if an email is verified or not).

doc/release-notes/8191-require-toa-or-request-access.md

@@ -0,0 +1,17 @@
+## Release Highlights
+
+### Terms of Access or Request Access Required for Restricted Files
+
+Beginning in this release, Restricted Files must have either Terms of Access or Request Access enabled. This change is to ensure that for each file in a Dataverse installation there is a clear path to get to the data, either through requesting access to the data or to provide context about why requesting access is not enabled.
+
+In the "Notes for Dataverse Installation Administrators" section, we have provided a query to help proactively identify datasets that need to be updated.
+
+## Notes for Dataverse Installation Administrators
+
+### Identifying Datasets Requiring Terms of Access or Request Access Changes
+
+In support of the change to require either Terms of Access or Request Access for all Restricted Files, we have provided a query to identify datasets in your installation where at least one file has neither Terms of Access or Request Access enabled:
+
+https://github.com/IQSS/dataverse/raw/develop/scripts/issues/8191/
+
+This will allow you to reach out to those dataset owners as appropriate.

doc/release-notes/8227-verify-email.md

@@ -0,0 +1 @@
+The "Verify Email" button has been changed to "Send Verification Email" and rather than sometimes showing a popup now always sends a fresh verification email (and invalidates previous verification emails).

doc/release-notes/8295-support-postgresql-14.md

@@ -0,0 +1,8 @@
+This release upgrades the bundled PostgreSQL JDBC driver to support major version 14.
+
+Note that the newer PostgreSQL driver required a Flyway version bump, which entails positive and negative consequences:
+
+- The newer version of Flyway supports PostgreSQL 14 and includes a number of security fixes.
+- As of version 8.0 the Flyway Community Edition dropped support for PostgreSQL 9.6 and older.
+
+Upgrade instructions may be found under “PostgreSQL Update” in the 5.10 release notes: https://github.com/IQSS/dataverse/releases/tag/v5.10

doc/release-notes/8380-counter-processor-update.md

@@ -0,0 +1 @@
+This release includes support for counter-processor-0.1.04 for processing Make Data Count metrics. If you are running Make Data Counts support, you should reinstall/reconfigure counter-processor as described in the latest Guides. (For existing installations, note that counter-processor-0.1.04 requires a newer version of python so you will need to follow the full counter-processor install. Also note that if you configure the new version the same way, it will reprocess the days in the current month when it is first run. This is normal and will not affect the metrics in Dataverse.)

doc/release-notes/8456-upgrade-primefaces.md

@@ -0,0 +1,4 @@
+### Primefaces 11
+
+Primefaces, the open source UI framework upon which the Dataverse front end is built, has been updated to the most recent version. This provides security updates and bug fixes and will also allow Dataverse developers to take advantage of new features and enhancements.
+

doc/release-notes/8525-ingest-optional-skip.md

@@ -0,0 +1 @@
+Tabular ingest can be skipped via API.

doc/release-notes/8533_semantic-api-updates.md

@@ -0,0 +1,14 @@
+
+## Notes for Developers and Integrators
+
+This release include an update to the experimental semantic API and the underlying assignment of URIs to metadatablock termshat are not explicitly mapped to terms in community vocabularies. The change affects the output of the OAI_ORE metadata export, the OAI_ORE file in archival bags, and the input/out allowed for those terms in the semantic api. For those updating integrating code or existing files intended for input into this release of Dataverse: URIs of the form:
+  https://dataverse.org/schema/<block name>/<parentField name>#<childField title>, and
+  https://dataverse.org/schema/<block name>/<Field title>
+  are both replaced with URIs of the form:
+  https://dataverse.org/schema/<block name>/<Field name>
+
+## Additional Release Steps
+
+Upgrade should include re-export of metadata files (only the OAI_ORE is affected).
+
+For this PR and other changes coming from DataCommons, it will also be advisable for people archiving Bags to re-archive. More detail on the overall set if changes in those tbd PRs.

doc/release-notes/8561-license-info-in-submit-for-review-pop-up.md

@@ -0,0 +1 @@
+Information about the license selection (or custom terms) is now available in the confirmation popup when contributors click "Submit for Review". Previously, this was only available in the confirmation popup for the "Publish" button, which contributors do not see.

doc/release-notes/8595-cvv-field-solr-update.md

@@ -0,0 +1,3 @@
+Controlled vocabulary fields that do not allow multiple entries are not indexed properly in Dataverse instances configured to support multiple languages. This release fixes the schema.xml file for the one field affected in the standard metadata blocks (journalArticleType) and updates the api/admin/index/solr/schema to provide the correct information for use with the update-fields.sh script described in the [Metadata Customization section of the Admin Guide](https://guides.dataverse.org/en/latest/admin/metadatacustomization.html#updating-the-solr-schema).
+
+The release should include updating the schema.xml file for solr or running the update-fields.sh script and reindexing (whatever standard instructions we give for schema changes.)

doc/release-notes/8600-duplicate-templates.md

@@ -0,0 +1,9 @@
+###Duplicate Templates in Database
+Prior to this release making a copy of a Dataset Template was creating two copies, one of which is visible in the dataverse, the other not being assigned 
+a dataverse was invisible to the user (https://github.com/IQSS/dataverse/issues/8600). This release fixes the issue. 
+
+If you would like to remove these orphan templates you may run the follwing script:
+
+https://github.com/IQSS/dataverse/raw/develop/scripts/issues/8600/delete_orphan_templates_8600.sh
+
+Also, admin APIs for finding and deleting templates have been added.

doc/release-notes/8663-update-create-ds-doc.md

@@ -0,0 +1,5 @@
+## Release Highlights
+
+### Documentation of Create Dataset API has been Updated
+
+Previous versions of the guides omitted the content header in the example API call for creating a datset. Due to a code change introduced in Dataverse version 5.6 calls without the content header will fail.

doc/release-notes/ds54-csp.md

@@ -0,0 +1 @@
+As always, widgets can be embedded in the `<iframe>` HTML tag, but the HTTP header "Content-Security-Policy" is now being sent on non-widget pages to prevent them from being embedded.

doc/sphinx-guides/requirements.txt

@@ -1,2 +1,7 @@
 # current version as of this writing
 Sphinx==3.5.4
+# Necessary workaround for ReadTheDocs for Sphinx 3.x - unnecessary as of Sphinx 4.5+
+Jinja2>=3.0.2,<3.1
+
+# Sphinx - Additional modules
+sphinx-icon==0.1.2

doc/sphinx-guides/source/_static/api/dataset-create.jsonld

@@ -2,14 +2,14 @@
   "http://purl.org/dc/terms/title": "Darwin's Finches",
   "http://purl.org/dc/terms/subject": "Medicine, Health and Life Sciences",
   "http://purl.org/dc/terms/creator": {
-      "https://dataverse.org/schema/citation/author#Name": "Finch, Fiona",
-      "https://dataverse.org/schema/citation/author#Affiliation": "Birds Inc."
+      "https://dataverse.org/schema/citation/authorName": "Finch, Fiona",
+      "https://dataverse.org/schema/citation/authorAffiliation": "Birds Inc."
   },
-  "https://dataverse.org/schema/citation/Contact": {
-    "https://dataverse.org/schema/citation/datasetContact#E-mail": "[email protected]",
-    "https://dataverse.org/schema/citation/datasetContact#Name": "Finch, Fiona"
+  "https://dataverse.org/schema/citation/datasetContact": {
+    "https://dataverse.org/schema/citation/datasetContactEmail": "[email protected]",
+    "https://dataverse.org/schema/citation/datasetContactName": "Finch, Fiona"
   },
-  "https://dataverse.org/schema/citation/Description": {
-    "https://dataverse.org/schema/citation/dsDescription#Text": "Darwin's finches (also known as the Galápagos finches) are a group of about fifteen species of passerine birds."
+  "https://dataverse.org/schema/citation/dsDescription": {
+    "https://dataverse.org/schema/citation/dsDescriptionValue": "Darwin's finches (also known as the Galápagos finches) are a group of about fifteen species of passerine birds."
   }
 }

doc/sphinx-guides/source/_static/api/dataset-create_en.jsonld

@@ -0,0 +1,16 @@
+{
+  "http://purl.org/dc/terms/title": "Darwin's Finches",
+  "http://purl.org/dc/terms/subject": "Medicine, Health and Life Sciences",
+  "http://schema.org/inLanguage":"en",
+  "http://purl.org/dc/terms/creator": {
+      "https://dataverse.org/schema/citation/author#Name": "Finch, Fiona",
+      "https://dataverse.org/schema/citation/author#Affiliation": "Birds Inc."
+  },
+  "https://dataverse.org/schema/citation/Contact": {
+    "https://dataverse.org/schema/citation/datasetContact#E-mail": "[email protected]",
+    "https://dataverse.org/schema/citation/datasetContact#Name": "Finch, Fiona"
+  },
+  "https://dataverse.org/schema/citation/Description": {
+    "https://dataverse.org/schema/citation/dsDescription#Text": "Darwin's finches (also known as the Galápagos finches) are a group of about fifteen species of passerine birds."
+  }
+}