From b87fdf732e4837ea137804eea4fe6af7da707515 Mon Sep 17 00:00:00 2001 From: Stephen Kraffmiller Date: Mon, 2 Dec 2024 13:44:39 -0500 Subject: [PATCH 01/34] #10952 create doc/outline --- doc/release-notes/6.5-release-notes.md | 192 +++++++++++++++++++++++++ 1 file changed, 192 insertions(+) create mode 100644 doc/release-notes/6.5-release-notes.md diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md new file mode 100644 index 00000000000..ccc8a161ae5 --- /dev/null +++ b/doc/release-notes/6.5-release-notes.md @@ -0,0 +1,192 @@ +# Dataverse 6.5 + +Please note: To read these instructions in full, please go to https://github.com/IQSS/dataverse/releases/tag/v6.5 rather than the list of releases, which will cut them off. + +This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. + +## Release Highlights + +New features in Dataverse 6.5: + +- and more! Please see below. + + +## Features Added + + + + +## Bugs Fixed + + +## API Updates + + + +## Settings Added + + + +## Backward Incompatible Changes + + + +## Complete List of Changes + +For the complete list of code changes in this release, see the [6.5 milestone](https://github.com/IQSS/dataverse/issues?q=milestone%3A6.5+is%3Aclosed) in GitHub. + +## Getting Help + +For help with upgrading, installing, or general questions please post to the [Dataverse Community Google Group](https://groups.google.com/g/dataverse-community) or email support@dataverse.org. + + +## Installation + +If this is a new installation, please follow our [Installation Guide](https://guides.dataverse.org/en/latest/installation/). Please don't be shy about [asking for help](https://guides.dataverse.org/en/latest/installation/intro.html#getting-help) if you need it! + +Once you are in production, we would be delighted to update our [map of Dataverse installations](https://dataverse.org/installations) around the world to include yours! Please [create an issue](https://github.com/IQSS/dataverse-installations/issues) or email us at support@dataverse.org to join the club! + +You are also very welcome to join the [Global Dataverse Community Consortium](https://www.gdcc.io/) (GDCC). + + +## Upgrade Instructions + +Upgrading requires a maintenance window and downtime. Please plan accordingly, create backups of your database, etc. + +These instructions assume that you've already upgraded through all the 5.x releases and are now running Dataverse 6.4. + +0\. These instructions assume that you are upgrading from the immediate previous version. If you are running an earlier version, the only supported way to upgrade is to progress through the upgrades to all the releases in between before attempting the upgrade to this version. + +If you are running Payara as a non-root user (and you should be!), **remember not to execute the commands below as root**. Use `sudo` to change to that user first. For example, `sudo -i -u dataverse` if `dataverse` is your dedicated application user. + +In the following commands, we assume that Payara 6 is installed in `/usr/local/payara6`. If not, adjust as needed. + +```shell +export PAYARA=/usr/local/payara6` +``` + +(or `setenv PAYARA /usr/local/payara6` if you are using a `csh`-like shell) + +1\. Undeploy the previous version + +```shell +$PAYARA/bin/asadmin undeploy dataverse-6.4 +``` + +2\. Stop and start Payara + +```shell +service payara stop +sudo service payara start +``` + +3\. Deploy this version + +```shell +$PAYARA/bin/asadmin deploy dataverse-6.5.war +``` + +Note: if you have any trouble deploying, stop Payara, remove the following directories, start Payara, and try to deploy again. + +```shell +service payara stop +rm -rf $PAYARA/glassfish/domains/domain1/generated +rm -rf $PAYARA/glassfish/domains/domain1/osgi-cache +rm -rf $PAYARA/glassfish/domains/domain1/lib/databases +``` + +4\. For installations with internationalization: + +Please remember to update translations via [Dataverse language packs](https://github.com/GlobalDataverseCommunityConsortium/dataverse-language-packs). + +5\. Restart Payara + +```shell +service payara stop +service payara start +``` + +6\. Update metadata blocks + +These changes reflect incremental improvements made to the handling of core metadata fields. + +```shell +wget https://raw.githubusercontent.com/IQSS/dataverse/v6.4/scripts/api/data/metadatablocks/citation.tsv + +curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file citation.tsv +``` + +7\. Update Solr schema.xml file. Start with the standard v6.4 schema.xml, then, if your installation uses any custom or experimental metadata blocks, update it to include the extra fields (step 7a). + +Stop Solr (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/6.4/installation/prerequisites.html#solr-init-script)). + +```shell +service solr stop +``` + +Replace schema.xml + +```shell +wget https://raw.githubusercontent.com/IQSS/dataverse/v6.4/conf/solr/schema.xml +cp schema.xml /usr/local/solr/solr-9.4.1/server/solr/collection1/conf +``` + +Start Solr (but if you use any custom metadata blocks, perform the next step, 7a first). + +```shell +service solr start +``` + +7a\. For installations with custom or experimental metadata blocks: + +Before starting Solr, update the schema to include all the extra metadata fields that your installation uses. We do this by collecting the output of the Dataverse schema API and feeding it to the `update-fields.sh` script that we supply, as in the example below (modify the command lines as needed to reflect the names of the directories, if different): + +```shell + wget https://raw.githubusercontent.com/IQSS/dataverse/v6.4/conf/solr/update-fields.sh + chmod +x update-fields.sh + curl "http://localhost:8080/api/admin/index/solr/schema" | ./update-fields.sh /usr/local/solr/solr-9.4.1/server/solr/collection1/conf/schema.xml +``` + +Now start Solr. + +8\. Reindex Solr + +Below is the simplest way to reindex Solr: + +```shell +curl http://localhost:8080/api/admin/index +``` + +The API above rebuilds the existing index "in place". If you want to be absolutely sure that your index is up-to-date and consistent, you may consider wiping it clean and reindexing everything from scratch (see [the guides](https://guides.dataverse.org/en/latest/admin/solr-search-index.html)). Just note that, depending on the size of your database, a full reindex may take a while and the users will be seeing incomplete search results during that window. + +9\. Run reExportAll to update dataset metadata exports + +This step is necessary because of changes described above for the `Datacite` and `oai_dc` export formats. + +Below is the simple way to reexport all dataset metadata. For more advanced usage, please see [the guides](http://guides.dataverse.org/en/6.4/admin/metadataexport.html#batch-exports-through-the-api). + +```shell +curl http://localhost:8080/api/admin/metadata/reExportAll +``` + +10\. Pushing updated metadata to DataCite + +(If you don't use DataCite, you can skip this.) + +Above you updated the citation metadata block and Solr with the new "relationType" field. With these two changes, the "Relation Type" fields will be available and creation/publication of datasets will result in the expanded XML being sent to DataCite. You've also already run "reExportAll" to update the `Datacite` metadata export format. + +Entries at DataCite for published datasets can be updated by a superuser using an API call (newly [documented](https://guides.dataverse.org/en/6.4/admin/dataverses-datasets.html#update-metadata-for-all-published-datasets-at-the-pid-provider)): + +`curl -X POST -H 'X-Dataverse-key:' http://localhost:8080/api/datasets/modifyRegistrationPIDMetadataAll` + +This will loop through all published datasets (and released files with PIDs). As long as the loop completes, the call will return a 200/OK response. Any PIDs for which the update fails can be found using the following command: + +`grep 'Failure for id' server.log` + +Failures may occur if PIDs were never registered, or if they were never made findable. Any such cases can be fixed manually in DataCite Fabrica or using the [Reserve a PID](https://guides.dataverse.org/en/6.4/api/native-api.html#reserve-a-pid) API call and the newly documented `/api/datasets//modifyRegistration` call respectively. See https://guides.dataverse.org/en/6.4/admin/dataverses-datasets.html#send-dataset-metadata-to-pid-provider. Please reach out with any questions. + +PIDs can also be updated by a superuser on a per-dataset basis using + +`curl -X POST -H 'X-Dataverse-key:' http://localhost:8080/api/datasets//modifyRegistrationMetadata` + +### Additional Upgrade Steps From eae7478e6d0e8cdf55d16e9a9abfe5e23645eb78 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 3 Dec 2024 15:34:01 -0500 Subject: [PATCH 02/34] start adding features, bug fixes, etc to notes #10952 --- .../11018-update-dataverse-endpoint-update.md | 8 ---- .../11049-oai-identifiers-as-pids.md | 5 -- .../220-harvard-edu-audit-files.md | 16 ------- doc/release-notes/6.5-release-notes.md | 47 +++++++++++++++++++ .../7239-mydata-results-by-username.md | 3 -- doc/release-notes/8184-rename-private-url.md | 11 ----- .../8941-adding-fileCount-in-solr.md | 15 ------ ...s-labels-not-translated-in-result-block.md | 7 --- ...50-5-improve-list-linked-dataverses-API.md | 5 -- doc/release-notes/expose-export-formats.md | 2 - 10 files changed, 47 insertions(+), 72 deletions(-) delete mode 100644 doc/release-notes/11018-update-dataverse-endpoint-update.md delete mode 100644 doc/release-notes/11049-oai-identifiers-as-pids.md delete mode 100644 doc/release-notes/220-harvard-edu-audit-files.md delete mode 100644 doc/release-notes/7239-mydata-results-by-username.md delete mode 100644 doc/release-notes/8184-rename-private-url.md delete mode 100644 doc/release-notes/8941-adding-fileCount-in-solr.md delete mode 100644 doc/release-notes/9408-fix-facets-labels-not-translated-in-result-block.md delete mode 100644 doc/release-notes/9650-5-improve-list-linked-dataverses-API.md delete mode 100644 doc/release-notes/expose-export-formats.md diff --git a/doc/release-notes/11018-update-dataverse-endpoint-update.md b/doc/release-notes/11018-update-dataverse-endpoint-update.md deleted file mode 100644 index c2d9cf64af3..00000000000 --- a/doc/release-notes/11018-update-dataverse-endpoint-update.md +++ /dev/null @@ -1,8 +0,0 @@ -The updateDataverse API endpoint has been updated to support an "inherit from parent" configuration for metadata blocks, facets, and input levels. - -When it comes to omitting any of these fields in the request JSON: - -- Omitting ``facetIds`` or ``metadataBlockNames`` causes the Dataverse collection to inherit the corresponding configuration from its parent. -- Omitting ``inputLevels`` removes any existing custom input levels in the Dataverse collection. - -Previously, not setting these fields meant keeping the existing ones in the Dataverse. diff --git a/doc/release-notes/11049-oai-identifiers-as-pids.md b/doc/release-notes/11049-oai-identifiers-as-pids.md deleted file mode 100644 index 8b53a461a70..00000000000 --- a/doc/release-notes/11049-oai-identifiers-as-pids.md +++ /dev/null @@ -1,5 +0,0 @@ -## When harvesting, Dataverse can now use the identifier from the OAI-PMH record header as the persistent id for the harvested dataset. - -This will allow harvesting from sources that do not include a persistent id in their oai_dc metadata records, but use valid dois or handles as the OAI-PMH record header identifiers. - -It is also possible to optionally configure a harvesting client to use this OAI-PMH identifier as the **preferred** choice for the persistent id. See the [Harvesting Clients API](https://guides.dataverse.org/en/6.5/api/native-api.html#create-a-harvesting-client) section of the Guides, #11049 and #10982 for more information. \ No newline at end of file diff --git a/doc/release-notes/220-harvard-edu-audit-files.md b/doc/release-notes/220-harvard-edu-audit-files.md deleted file mode 100644 index fc857e3a02b..00000000000 --- a/doc/release-notes/220-harvard-edu-audit-files.md +++ /dev/null @@ -1,16 +0,0 @@ -### New API to Audit Datafiles across the database - -This is a superuser only API endpoint to audit Datasets with DataFiles where the physical files are missing or the file metadata is missing. -The Datasets scanned can be limited by optional firstId and lastId query parameters, or a given CSV list of Dataset Identifiers. -Once the audit report is generated, a superuser can either delete the missing file(s) from the Dataset or contact the author to re-upload the missing file(s). - -The JSON response includes: -- List of files in each DataFile where the file exists in the database but the physical file is not in the file store. -- List of DataFiles where the FileMetadata is missing. -- Other failures found when trying to process the Datasets - -curl -H "X-Dataverse-key:$API_TOKEN" "http://localhost:8080/api/admin/datafiles/auditFiles" -curl -H "X-Dataverse-key:$API_TOKEN" "http://localhost:8080/api/admin/datafiles/auditFiles?firstId=0&lastId=1000" -curl -H "X-Dataverse-key:$API_TOKEN" "http://localhost:8080/api/admin/datafiles/auditFiles?datasetIdentifierList=doi:10.5072/FK2/RVNT9Q,doi:10.5072/FK2/RVNT9Q" - -For more information, see [the docs](https://dataverse-guide--11016.org.readthedocs.build/en/11016/api/native-api.html#datafile-audit), #11016, and [#220](https://github.com/IQSS/dataverse.harvard.edu/issues/220) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index ccc8a161ae5..7a0d4f81da8 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -13,15 +13,57 @@ New features in Dataverse 6.5: ## Features Added +### Private URL Renamed to Preview URL +With this release the name of the URL that may be used by dataset administrators to share a draft version of a dataset has been changed from Private URL to Preview URL. +Also, additional information about the creation of Preview URLs has been added to the popup accessed via edit menu of the Dataset Page. + +Any Private URLs created in previous versions of Dataverse will continue to work. + +The old "privateUrl" API endpoints for the creation and deletion of Preview (formerly Private) URLs have been deprecated. They will continue to work but please switch to the "previewUrl" equivalents that have been [documented](https://dataverse-guide--10961.org.readthedocs.build/en/10961/api/native-api.html#create-a-preview-url-for-a-dataset) in the API Guide. + +See also #8184, #8185, #10950, and #10961. + +### Harvested Dataset PID from Record Header + +When harvesting, Dataverse can now use the identifier from the OAI-PMH record header as the persistent id for the harvested dataset. + +This will allow harvesting from sources that do not include a persistent id in their oai_dc metadata records, but use valid DOIs or handles as the OAI-PMH record header identifiers. + +It is also possible to optionally configure a harvesting client to use this OAI-PMH identifier as the **preferred** choice for the persistent id. See the [Harvesting Clients API](https://guides.dataverse.org/en/6.5/api/native-api.html#create-a-harvesting-client) section of the Guides, #11049 and #10982 for more information. ## Bugs Fixed +### My Data Filter by Username Feature Fixed + +The superuser-only feature of filtering by a username on the My Data page was not working. The "Results for Username" field now returns data for the desired user. See also #7239 and #10980. + +### Facets Filter Labels Now Translated Above Search Results + +On the main page, it's possible to filter results using search facets. If internationalization (i18n) has been activated in the Dataverse installation, allowing pages to be displayed in several languages, the facets are translated in the filter column. However, they weren't being translated above the search results, remaining in the default language, English. + +This version of Dataverse fix this, and includes internationalization in the facets visible in the search results section. For more information, see #9408 and #10158. ## API Updates +### fileCount Added to Search API + +A new search field called `fileCount` can be searched to discover the number of files per dataset. See also #8941 and #10598. + +### List Dataset Metadata Exporters + +A list of available dataset metadata exporters can now be retrieved programmatically via API. See [the docs](https://dataverse-guide--10739.org.readthedocs.build/en/10739/api/native-api.html#get-export-formats) and #10739. + +### Audit Data Files +A superuser-only API endpoint has been added to audit datasets with data files where the physical files are missing or the file metadata is missing. See [the docs](https://dataverse-guide--11016.org.readthedocs.build/en/11016/api/native-api.html#datafile-audit), #11016, and [#220](https://github.com/IQSS/dataverse.harvard.edu/issues/220). + +### Update Collection API Inheritance + +The update collection (dataverse) API endpoint has been updated to support an "inherit from parent" configuration for metadata blocks, facets, and input levels. + +Previously, not setting these fields meant using a copy of the settings from the parent collection, which could get out of sync.. See also [the docs](https://preview.guides.gdcc.io/en/develop/api/native-api.html#update-a-dataverse-collection), #11018, and #11026. ## Settings Added @@ -29,6 +71,11 @@ New features in Dataverse 6.5: ## Backward Incompatible Changes +Generally speaking, see the [API Changelog](https://preview.guides.gdcc.io/en/develop/api/changelog.html) for a list of backward-incompatible changes. + +### List Collections Linked to a Dataset + +The API endpoint that returns a list of collections that a dataset has been linked to has been improved to provide a more structured JSON response. See [the docs](https://dataverse-guide--9665.org.readthedocs.build/en/9665/admin/dataverses-datasets.html#list-collections-that-are-linked-from-a-dataset), #9650, and #9665. ## Complete List of Changes diff --git a/doc/release-notes/7239-mydata-results-by-username.md b/doc/release-notes/7239-mydata-results-by-username.md deleted file mode 100644 index fa1ce56d89e..00000000000 --- a/doc/release-notes/7239-mydata-results-by-username.md +++ /dev/null @@ -1,3 +0,0 @@ -## Fix My Data filter results by username for Administrators - -The filtering for the username on the MyData page was not working. This is only available for superusers. This fixes the "Results for Username" field to return the data for the desired user. See also #7239 and #10980. diff --git a/doc/release-notes/8184-rename-private-url.md b/doc/release-notes/8184-rename-private-url.md deleted file mode 100644 index 7acb03fd735..00000000000 --- a/doc/release-notes/8184-rename-private-url.md +++ /dev/null @@ -1,11 +0,0 @@ -###Private URL renamed Preview URL - -With this release the name of the URL that may be used by dataset administrators to share a draft version of a dataset has been changed from Private URL to Preview URL. - -Also, additional information about the creation of Preview URLs has been added to the popup accessed via edit menu of the Dataset Page. - -Any Private URLs created in previous versions of Dataverse will continue to work. - -The old "privateUrl" API endpoints for the creation and deletion of Preview (formerly Private) URLs have been deprecated. They will continue to work but please switch to the "previewUrl" equivalents that have been [documented](https://dataverse-guide--10961.org.readthedocs.build/en/10961/api/native-api.html#create-a-preview-url-for-a-dataset) in the API Guide. - -See also #8184, #8185, #10950, and #10961. diff --git a/doc/release-notes/8941-adding-fileCount-in-solr.md b/doc/release-notes/8941-adding-fileCount-in-solr.md deleted file mode 100644 index 164b91e6123..00000000000 --- a/doc/release-notes/8941-adding-fileCount-in-solr.md +++ /dev/null @@ -1,15 +0,0 @@ -## Release Highlights - -### Adding fileCount as SOLR field - -A new search field called `fileCount` can be searched to discover the number of files per dataset. (#10598) - -## Upgrade Instructions - -1. Update your Solr `schema.xml` to include the new field. -For details, please see https://guides.dataverse.org/en/latest/admin/metadatacustomization.html#updating-the-solr-schema - -2. Reindex Solr. -Once the schema.xml is updated, Solr must be restarted and a reindex initiated. -For details, see https://guides.dataverse.org/en/latest/admin/solr-search-index.html but here is the reindex command: -`curl http://localhost:8080/api/admin/index` diff --git a/doc/release-notes/9408-fix-facets-labels-not-translated-in-result-block.md b/doc/release-notes/9408-fix-facets-labels-not-translated-in-result-block.md deleted file mode 100644 index 344859e2dbd..00000000000 --- a/doc/release-notes/9408-fix-facets-labels-not-translated-in-result-block.md +++ /dev/null @@ -1,7 +0,0 @@ -## Fix facets filter labels not translated in result block - -On the main page, it's possible to filter results using search facets. If internationalization (i18n) has been activated in the Dataverse installation, allowing pages to be displayed in several languages, the facets are translated in the filter column. However, they aren't translated in the search results and remain in the default language, English. - -This version of Dataverse fix this, and includes internationalization in the facets visible in the search results section. - -For more information, see issue [#9408](https://github.com/IQSS/dataverse/issues/9408) and pull request [#10158](https://github.com/IQSS/dataverse/pull/10158) diff --git a/doc/release-notes/9650-5-improve-list-linked-dataverses-API.md b/doc/release-notes/9650-5-improve-list-linked-dataverses-API.md deleted file mode 100644 index 8c79955891b..00000000000 --- a/doc/release-notes/9650-5-improve-list-linked-dataverses-API.md +++ /dev/null @@ -1,5 +0,0 @@ -The following API have been added: - -/api/datasets/{datasetId}/links - -It lists the linked dataverses to a dataset. It can be executed only by administrators. \ No newline at end of file diff --git a/doc/release-notes/expose-export-formats.md b/doc/release-notes/expose-export-formats.md deleted file mode 100644 index a21906d7bbb..00000000000 --- a/doc/release-notes/expose-export-formats.md +++ /dev/null @@ -1,2 +0,0 @@ -# New API method for listing the available exporters -Found at `/api/info/exportFormats`, produces an object with available format names as keys, and as values an object with various info about the exporter. See also #10739. \ No newline at end of file From e520a9a746aacd01e0bc3d9f633cdf98cfbe14ac Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 3 Dec 2024 15:54:18 -0500 Subject: [PATCH 03/34] add more to notes #10952 --- ...pearing-in-search-results-for-anon-user.md | 11 --------- ...0969-order-subfields-version-difference.md | 2 -- .../10977-globus-filesize-lookup.md | 6 ----- .../11012-get-dataverse-api-ext.md | 1 - doc/release-notes/6.5-release-notes.md | 24 ++++++++++++++++++- 5 files changed, 23 insertions(+), 21 deletions(-) delete mode 100644 doc/release-notes/10947-unpublished-files-appearing-in-search-results-for-anon-user.md delete mode 100644 doc/release-notes/10969-order-subfields-version-difference.md delete mode 100644 doc/release-notes/10977-globus-filesize-lookup.md delete mode 100644 doc/release-notes/11012-get-dataverse-api-ext.md diff --git a/doc/release-notes/10947-unpublished-files-appearing-in-search-results-for-anon-user.md b/doc/release-notes/10947-unpublished-files-appearing-in-search-results-for-anon-user.md deleted file mode 100644 index 66ea04b124f..00000000000 --- a/doc/release-notes/10947-unpublished-files-appearing-in-search-results-for-anon-user.md +++ /dev/null @@ -1,11 +0,0 @@ -## Unpublished file bug fix - -A bug fix was made that gets the major version of a Dataset when all major versions were deaccessioned. This fixes the incorrect showing of the files as "Unpublished" in the search list even when they are published. -This fix affects the indexing, meaning these datasets must be re-indexed once Dataverse is updated. This can be manually done by calling the index API for each affected Dataset. - -Example: -```shell -curl http://localhost:8080/api/admin/index/dataset?persistentId=doi:10.7910/DVN/6X4ZZL -``` - -See also #10947 and #10974. diff --git a/doc/release-notes/10969-order-subfields-version-difference.md b/doc/release-notes/10969-order-subfields-version-difference.md deleted file mode 100644 index 3f245ebe069..00000000000 --- a/doc/release-notes/10969-order-subfields-version-difference.md +++ /dev/null @@ -1,2 +0,0 @@ -Bug Fix: -In order to facilitate the comparison between the draft version and the published version of a dataset, a sort on subfields has been added (#10969) \ No newline at end of file diff --git a/doc/release-notes/10977-globus-filesize-lookup.md b/doc/release-notes/10977-globus-filesize-lookup.md deleted file mode 100644 index 49fd10d9ffe..00000000000 --- a/doc/release-notes/10977-globus-filesize-lookup.md +++ /dev/null @@ -1,6 +0,0 @@ -## A new Globus optimization setting - -An optimization has been added for the Globus upload workflow, with a corresponding new database setting: `:GlobusBatchLookupSize` - - -See the [Database Settings](https://guides.dataverse.org/en/6.5/installation/config.html#GlobusBatchLookupSize) section of the Guides for more information. \ No newline at end of file diff --git a/doc/release-notes/11012-get-dataverse-api-ext.md b/doc/release-notes/11012-get-dataverse-api-ext.md deleted file mode 100644 index 641aa373174..00000000000 --- a/doc/release-notes/11012-get-dataverse-api-ext.md +++ /dev/null @@ -1 +0,0 @@ -The JSON payload of the getDataverse endpoint has been extended to include properties isMetadataBlockRoot and isFacetRoot. diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index 7a0d4f81da8..30bd0f68732 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -25,6 +25,12 @@ The old "privateUrl" API endpoints for the creation and deletion of Preview (for See also #8184, #8185, #10950, and #10961. +### GlobusBatchLookupSize + +An optimization has been added for the Globus upload workflow, with a corresponding new database setting: `:GlobusBatchLookupSize` + +See the [Database Settings](https://guides.dataverse.org/en/6.5/installation/config.html#GlobusBatchLookupSize) section of the Guides, #10977, and #11040 for more information. + ### Harvested Dataset PID from Record Header When harvesting, Dataverse can now use the identifier from the OAI-PMH record header as the persistent id for the harvested dataset. @@ -39,12 +45,24 @@ It is also possible to optionally configure a harvesting client to use this OAI- The superuser-only feature of filtering by a username on the My Data page was not working. The "Results for Username" field now returns data for the desired user. See also #7239 and #10980. +### Version Differences Details Sorting Added + +In order to facilitate the comparison between the draft version and the published version of a dataset, a sort on subfields has been added. See #10969. + ### Facets Filter Labels Now Translated Above Search Results On the main page, it's possible to filter results using search facets. If internationalization (i18n) has been activated in the Dataverse installation, allowing pages to be displayed in several languages, the facets are translated in the filter column. However, they weren't being translated above the search results, remaining in the default language, English. This version of Dataverse fix this, and includes internationalization in the facets visible in the search results section. For more information, see #9408 and #10158. +### Unpublished File Bug Fix Related to Deaccessioning + +A bug fix was made that gets the major version of a Dataset when all major versions were deaccessioned. This fixes the incorrect showing of the files as "Unpublished" in the search list even when they are published. This fix affects the indexing, meaning these datasets must be re-indexed once Dataverse is updated. See also #10947 and #10974. + +### Globus "missing properties" Logging Fixed + +In previous releases, logging would show Globus-related strings were missing from properties files. This has been fixed. See #11030 for details. + ## API Updates ### fileCount Added to Search API @@ -65,9 +83,13 @@ The update collection (dataverse) API endpoint has been updated to support an "i Previously, not setting these fields meant using a copy of the settings from the parent collection, which could get out of sync.. See also [the docs](https://preview.guides.gdcc.io/en/develop/api/native-api.html#update-a-dataverse-collection), #11018, and #11026. -## Settings Added +### isMetadataBlockRoot and isFacetRoot + +The JSON payload of the "get collection" endpoint has been extended to include properties isMetadataBlockRoot and isFacetRoot. See also [the docs](https://preview.guides.gdcc.io/en/develop/api/native-api.html#view-a-dataverse-collection), #11012, and #11013. +## Settings Added +- :GlobusBatchLookupSize ## Backward Incompatible Changes From 3e69dd1ccfb4f97e0937cf86cb4acfea89429d82 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 4 Dec 2024 14:48:11 -0500 Subject: [PATCH 04/34] more features, etc #10952 --- ...17-guestbook-question-size-limit-raised.md | 1 - .../10914-users-token-api-credentials.md | 3 --- .../10919-minor-DataCiteXML-bugfix.md | 1 - doc/release-notes/10939-i18n-docker.md | 5 ---- doc/release-notes/6.5-release-notes.md | 24 +++++++++++++++++++ 5 files changed, 24 insertions(+), 10 deletions(-) delete mode 100644 doc/release-notes/10117-guestbook-question-size-limit-raised.md delete mode 100644 doc/release-notes/10914-users-token-api-credentials.md delete mode 100644 doc/release-notes/10919-minor-DataCiteXML-bugfix.md delete mode 100644 doc/release-notes/10939-i18n-docker.md diff --git a/doc/release-notes/10117-guestbook-question-size-limit-raised.md b/doc/release-notes/10117-guestbook-question-size-limit-raised.md deleted file mode 100644 index ab5e84d78fe..00000000000 --- a/doc/release-notes/10117-guestbook-question-size-limit-raised.md +++ /dev/null @@ -1 +0,0 @@ -Custom questions in Guestbooks can now be more than 255 characters and the bug causing a silent failure when questions were longer than this limit has been fixed. \ No newline at end of file diff --git a/doc/release-notes/10914-users-token-api-credentials.md b/doc/release-notes/10914-users-token-api-credentials.md deleted file mode 100644 index 888214481f6..00000000000 --- a/doc/release-notes/10914-users-token-api-credentials.md +++ /dev/null @@ -1,3 +0,0 @@ -Extended the users/token GET endpoint to support any auth mechanism for retrieving the token information. - -Previously, this endpoint only accepted an API token to retrieve its information. Now, it accepts any authentication mechanism and returns the associated API token information. diff --git a/doc/release-notes/10919-minor-DataCiteXML-bugfix.md b/doc/release-notes/10919-minor-DataCiteXML-bugfix.md deleted file mode 100644 index 4fa0c1142b1..00000000000 --- a/doc/release-notes/10919-minor-DataCiteXML-bugfix.md +++ /dev/null @@ -1 +0,0 @@ -A minor bug fix was made to avoid sending a useless ", null" in the DataCiteXML sent to DataCite and in the DataCite export when a dataset has a metadata entry for "Software Name" and no entry for "Software Version". The bug fix will update datasets upon publication. Anyone with existing published datasets with this problem can be fixed by [pushing updated metadata to DataCite for affected datasets](https://guides.dataverse.org/en/6.4/admin/dataverses-datasets.html#update-metadata-for-a-published-dataset-at-the-pid-provider) and [re-exporting the dataset metadata](https://guides.dataverse.org/en/6.4/admin/metadataexport.html#batch-exports-through-the-api) or by following steps 9 and 10 in the v6.4 release notes to update and re-export all datasets. diff --git a/doc/release-notes/10939-i18n-docker.md b/doc/release-notes/10939-i18n-docker.md deleted file mode 100644 index d9887b684db..00000000000 --- a/doc/release-notes/10939-i18n-docker.md +++ /dev/null @@ -1,5 +0,0 @@ -## Multiple Language in Docker - -Configuration and documentation has been added to explain how to set up multiple languages (e.g. English and French) in the tutorial for setting up Dataverse in Docker. - -See also #10939 diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index 30bd0f68732..2d6345941bb 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -13,6 +13,10 @@ New features in Dataverse 6.5: ## Features Added +### Longer Custom Questions in Guestbooks + +Custom questions in Guestbooks can now be more than 255 characters and the bug causing a silent failure when questions were longer than this limit has been fixed. See also #9492, #10117, #10118. + ### Private URL Renamed to Preview URL With this release the name of the URL that may be used by dataset administrators to share a draft version of a dataset has been changed from Private URL to Preview URL. @@ -25,6 +29,12 @@ The old "privateUrl" API endpoints for the creation and deletion of Preview (for See also #8184, #8185, #10950, and #10961. +## Multiple Language in Docker + +Configuration and documentation has been added to explain how to set up multiple languages (e.g. English and French) in the tutorial for setting up Dataverse in Docker. + +See also [the docs](https://dataverse-guide--10940.org.readthedocs.build/en/10940/container/running/demo.html#multiple-languages), #10939, and #10940. + ### GlobusBatchLookupSize An optimization has been added for the Globus upload workflow, with a corresponding new database setting: `:GlobusBatchLookupSize` @@ -59,12 +69,20 @@ This version of Dataverse fix this, and includes internationalization in the fac A bug fix was made that gets the major version of a Dataset when all major versions were deaccessioned. This fixes the incorrect showing of the files as "Unpublished" in the search list even when they are published. This fix affects the indexing, meaning these datasets must be re-indexed once Dataverse is updated. See also #10947 and #10974. +### Minor DataCiteXML Fix + +A minor bug fix was made to avoid sending a useless ", null" in the DataCiteXML sent to DataCite and in the DataCite export when a dataset has a metadata entry for "Software Name" and no entry for "Software Version". The bug fix will update datasets upon publication. Anyone with existing published datasets with this problem can be fixed by [pushing updated metadata to DataCite for affected datasets](https://guides.dataverse.org/en/6.5/admin/dataverses-datasets.html#update-metadata-for-a-published-dataset-at-the-pid-provider) and [re-exporting the dataset metadata](https://guides.dataverse.org/en/6.5/admin/metadataexport.html#batch-exports-through-the-api) or by steps 9 and 10 in the [v6.4 release notes](https://github.com/IQSS/dataverse/releases/tag/v6.4) to update and re-export all datasets. See also #10919. + ### Globus "missing properties" Logging Fixed In previous releases, logging would show Globus-related strings were missing from properties files. This has been fixed. See #11030 for details. ## API Updates +### Editing Collections + +A new endpoint (`PUT /api/dataverses/`) for updating an existing collection (dataverse) has been added. It uses the same JSON structure as the one used for collection creation. See also [the docs](https://preview.guides.gdcc.io/en/develop/api/native-api.html#update-a-dataverse-collection), #10904, and #10925. + ### fileCount Added to Search API A new search field called `fileCount` can be searched to discover the number of files per dataset. See also #8941 and #10598. @@ -87,6 +105,12 @@ Previously, not setting these fields meant using a copy of the settings from the The JSON payload of the "get collection" endpoint has been extended to include properties isMetadataBlockRoot and isFacetRoot. See also [the docs](https://preview.guides.gdcc.io/en/develop/api/native-api.html#view-a-dataverse-collection), #11012, and #11013. +### Get API Token Supports Any Auth + +The `/api/users/token` endpoint has been extended to support any auth mechanism for retrieving the token information. + +Previously, this endpoint only accepted an API token to retrieve its information. Now, it accepts any authentication mechanism and returns the associated API token information. See #10914 and #10924. + ## Settings Added - :GlobusBatchLookupSize From 65fbd27290be8c563f39fd63c727f6c43717e97d Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 4 Dec 2024 14:55:20 -0500 Subject: [PATCH 05/34] more content #10952 --- doc/release-notes/10901deaccessioned file edit fix.md | 1 - doc/release-notes/10904-edit-dataverse-collection-endpoint.md | 1 - doc/release-notes/6.5-release-notes.md | 4 ++++ 3 files changed, 4 insertions(+), 2 deletions(-) delete mode 100644 doc/release-notes/10901deaccessioned file edit fix.md delete mode 100644 doc/release-notes/10904-edit-dataverse-collection-endpoint.md diff --git a/doc/release-notes/10901deaccessioned file edit fix.md b/doc/release-notes/10901deaccessioned file edit fix.md deleted file mode 100644 index db12b1fc978..00000000000 --- a/doc/release-notes/10901deaccessioned file edit fix.md +++ /dev/null @@ -1 +0,0 @@ -When a dataset was deaccessioned and was the only previous version it will cause an error when trying to update the files. \ No newline at end of file diff --git a/doc/release-notes/10904-edit-dataverse-collection-endpoint.md b/doc/release-notes/10904-edit-dataverse-collection-endpoint.md deleted file mode 100644 index b9256941eea..00000000000 --- a/doc/release-notes/10904-edit-dataverse-collection-endpoint.md +++ /dev/null @@ -1 +0,0 @@ -Adds a new endpoint (`PUT /api/dataverses/`) for updating an existing Dataverse collection using a JSON file following the same structure as the one used in the API for the creation. diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index 2d6345941bb..9500c90f53a 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -51,6 +51,10 @@ It is also possible to optionally configure a harvesting client to use this OAI- ## Bugs Fixed +### Updating Files Now Possible When Latest and Only Dataset Version is Deaccessioned + +When a dataset was deaccessioned and was the only previous version it would cause an error when trying to update the files. This has been fixed. See #9351 and #10901. + ### My Data Filter by Username Feature Fixed The superuser-only feature of filtering by a username on the My Data page was not working. The "Results for Username" field now returns data for the desired user. See also #7239 and #10980. From aa56e627fcd72edb4e0f93f416ed4aa7a032b64a Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 4 Dec 2024 15:55:04 -0500 Subject: [PATCH 06/34] more content #10952 --- ...-add-api-for-comparing-dataset-versions.md | 21 ------------------- doc/release-notes/10889_bump_PG17_FlyWay10.md | 7 ------- doc/release-notes/6.5-release-notes.md | 20 +++++++++++++++++- 3 files changed, 19 insertions(+), 29 deletions(-) delete mode 100644 doc/release-notes/10888-add-api-for-comparing-dataset-versions.md delete mode 100644 doc/release-notes/10889_bump_PG17_FlyWay10.md diff --git a/doc/release-notes/10888-add-api-for-comparing-dataset-versions.md b/doc/release-notes/10888-add-api-for-comparing-dataset-versions.md deleted file mode 100644 index b82441ee11a..00000000000 --- a/doc/release-notes/10888-add-api-for-comparing-dataset-versions.md +++ /dev/null @@ -1,21 +0,0 @@ -The following API have been added: - -/api/datasets/{persistentId}/versions/{versionId0}/compare/{versionId1} - -This API lists the changes between 2 dataset versions. The Json response shows the changes per field within the Metadata block and the Terms Of Access. Also listed are the files that have been added or removed. Files that have been modified will also display the new file data plus the fields that have been modified. -When compare includes an unpublished/draft version the api token must be associated with a user having view unpublished privileges -An error will be returned if VERSION0 was not created before VERSION1 - -Example of Metadata Block field change: -```json -{ - "blockName": "Life Sciences Metadata", - "changed": [ - { - "fieldName": "Design Type", - "oldValue": "", - "newValue": "Parallel Group Design; Nested Case Control Design" - } - ] -} -``` diff --git a/doc/release-notes/10889_bump_PG17_FlyWay10.md b/doc/release-notes/10889_bump_PG17_FlyWay10.md deleted file mode 100644 index 932c06fbc3d..00000000000 --- a/doc/release-notes/10889_bump_PG17_FlyWay10.md +++ /dev/null @@ -1,7 +0,0 @@ -This release bumps both the Postgres JDBC driver and Flyway versions. This should better support Postgres version 17, and as of version 10 Flyway no longer requires a paid subscription to support older versions of Postgres. - -While we don't encourage the use of older Postgres versions, this flexibility may benefit some of our long-standing installations in their upgrade paths. Postgres 13 remains the version used with automated testing. - -As part of this update, the containerized development environment now uses Postgres 17 instead of 16. Developers must delete their data (`rm -rf docker-dev-volumes`) and start with an empty database. They can rerun the quickstart in the dev guide. - -The Docker compose file used for [evaluations or demos](https://dataverse-guide--10912.org.readthedocs.build/en/10912/container/running/demo.html) has been upgraded from Postgres 13 to 17. diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index 9500c90f53a..16bd925eac8 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -29,7 +29,21 @@ The old "privateUrl" API endpoints for the creation and deletion of Preview (for See also #8184, #8185, #10950, and #10961. -## Multiple Language in Docker +### PostgreSQL and Flyway Updates + +This release bumps the version of PostgreSQL and Flyway used in containers as well as the PostgreSQL JDBC driver used all installations, including classic (non-Docker) installations. PostgreSQL and its driver have been bumped to version 17. Flyway has been bumped to version 10. + +PostgreSQL 13 remains the version used with automated testing, leading us to continue to [recommend](https://guides.dataverse.org/en/6.5/installation/prerequisites.html#postgresql) that version for classic installations. + +As of Flyway 10, supporting older versions of PostgreSQL no longer requires a paid subscription. While we don't encourage the use of older PostgreSQL versions, this flexibility may benefit some of our long-standing installations in their upgrade paths. + +As part of this update, the containerized development environment now uses Postgres 17 instead of 16. Developers must delete their data (`rm -rf docker-dev-volumes`) and start with an empty database, as [explained](https://groups.google.com/g/dataverse-dev/c/ffoNj5UXyzU/m/nE5oGY_sAQAJ) on the dev mailing list. They can rerun the quickstart in the dev guide. + +The Docker compose file used for [evaluations or demos](https://guides.dataverse.org/en/6.4/container/running/demo.html) has been upgraded from Postgres 13 to 17. + +See also #10889 and #10912. + +## Multiple Languages in Docker Configuration and documentation has been added to explain how to set up multiple languages (e.g. English and French) in the tutorial for setting up Dataverse in Docker. @@ -95,6 +109,10 @@ A new search field called `fileCount` can be searched to discover the number of A list of available dataset metadata exporters can now be retrieved programmatically via API. See [the docs](https://dataverse-guide--10739.org.readthedocs.build/en/10739/api/native-api.html#get-export-formats) and #10739. +### Comparing Dataset Versions + +An API has been added to compare dataset versions. See [the docs](https://preview.guides.gdcc.io/en/develop/api/native-api.html#compare-versions-of-a-dataset), #10888, and #10945. + ### Audit Data Files A superuser-only API endpoint has been added to audit datasets with data files where the physical files are missing or the file metadata is missing. See [the docs](https://dataverse-guide--11016.org.readthedocs.build/en/11016/api/native-api.html#datafile-audit), #11016, and [#220](https://github.com/IQSS/dataverse.harvard.edu/issues/220). From 8f8bce84ec9b9705d4250897739f96e8537a64c5 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 5 Dec 2024 13:04:48 -0500 Subject: [PATCH 07/34] more content #10952 --- .../10772-fix-importDDI-otherId.md | 2 -- .../10793-optimisticlockexception handling.md | 2 -- .../10814-Differencing improvement.md | 3 -- ...837-exclude-others-ns-harvesting-oai-dc.md | 3 -- ...d-expiration-date-to-recreate-token-api.md | 1 - ...date-to-conditions-to-display-image_url.md | 8 ----- doc/release-notes/6.5-release-notes.md | 36 +++++++++++++++++-- 7 files changed, 33 insertions(+), 22 deletions(-) delete mode 100644 doc/release-notes/10772-fix-importDDI-otherId.md delete mode 100644 doc/release-notes/10793-optimisticlockexception handling.md delete mode 100644 doc/release-notes/10814-Differencing improvement.md delete mode 100644 doc/release-notes/10837-exclude-others-ns-harvesting-oai-dc.md delete mode 100644 doc/release-notes/10857-add-expiration-date-to-recreate-token-api.md delete mode 100644 doc/release-notes/10886-update-to-conditions-to-display-image_url.md diff --git a/doc/release-notes/10772-fix-importDDI-otherId.md b/doc/release-notes/10772-fix-importDDI-otherId.md deleted file mode 100644 index d5a9018b2b2..00000000000 --- a/doc/release-notes/10772-fix-importDDI-otherId.md +++ /dev/null @@ -1,2 +0,0 @@ -Bug Fix : -This PR fixes the `edu.harvard.iq.dataverse.util.json.JsonParseException: incorrect multiple for field otherId` error when DDI harvested data contains multiple ortherId. \ No newline at end of file diff --git a/doc/release-notes/10793-optimisticlockexception handling.md b/doc/release-notes/10793-optimisticlockexception handling.md deleted file mode 100644 index 3312063be8f..00000000000 --- a/doc/release-notes/10793-optimisticlockexception handling.md +++ /dev/null @@ -1,2 +0,0 @@ -Improvements have been made in handling the errors when a dataset has been edited in one window and an attempt is made to -edit/publish it in another. diff --git a/doc/release-notes/10814-Differencing improvement.md b/doc/release-notes/10814-Differencing improvement.md deleted file mode 100644 index 49bbdae3e1b..00000000000 --- a/doc/release-notes/10814-Differencing improvement.md +++ /dev/null @@ -1,3 +0,0 @@ -### More Scalable Dataset Version Differencing - -Differencing between dataset versions, which is done during dataset edit operations and to populate the dataset page versions table has been made signficantly more scalable. diff --git a/doc/release-notes/10837-exclude-others-ns-harvesting-oai-dc.md b/doc/release-notes/10837-exclude-others-ns-harvesting-oai-dc.md deleted file mode 100644 index c1826bfaed5..00000000000 --- a/doc/release-notes/10837-exclude-others-ns-harvesting-oai-dc.md +++ /dev/null @@ -1,3 +0,0 @@ -Some repository extend the "oai_dc" metadata prefix with specific namespaces. In this case, harvesting of these datasets is not possible, as an XML parsing error is raised. - -The PR [#10837](https://github.com/IQSS/dataverse/pull/10837) allows the harvesting of these datasets by excluding tags with namespaces that are not "dc:", and harvest only metadata with the "dc" namespace. diff --git a/doc/release-notes/10857-add-expiration-date-to-recreate-token-api.md b/doc/release-notes/10857-add-expiration-date-to-recreate-token-api.md deleted file mode 100644 index b450867c630..00000000000 --- a/doc/release-notes/10857-add-expiration-date-to-recreate-token-api.md +++ /dev/null @@ -1 +0,0 @@ -An optional query parameter called 'returnExpiration' has been added to the 'users/token/recreate' endpoint, which, if set to true, returns the expiration time in the response message. diff --git a/doc/release-notes/10886-update-to-conditions-to-display-image_url.md b/doc/release-notes/10886-update-to-conditions-to-display-image_url.md deleted file mode 100644 index 6dfe8eb9f2d..00000000000 --- a/doc/release-notes/10886-update-to-conditions-to-display-image_url.md +++ /dev/null @@ -1,8 +0,0 @@ -Search API (/api/search) responses for Datafiles include image_url for the thumbnail if each of the following are true: -1. The DataFile is not Harvested -2. A Thumbnail is available for the Datafile -3. If the Datafile is Restricted then the caller must have Download File Permission for the Datafile -4. The Datafile is NOT actively embargoed -5. The Datafile's retention period has NOT expired - -See also #10875 and #10886. diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index 16bd925eac8..b66ae2f1b21 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -29,6 +29,10 @@ The old "privateUrl" API endpoints for the creation and deletion of Preview (for See also #8184, #8185, #10950, and #10961. +### More Scalable Dataset Version Differencing + +Differencing between dataset versions, which is done during dataset edit operations and to populate the dataset page versions table has been made signficantly more scalable. See #10814 and #10818. + ### PostgreSQL and Flyway Updates This release bumps the version of PostgreSQL and Flyway used in containers as well as the PostgreSQL JDBC driver used all installations, including classic (non-Docker) installations. PostgreSQL and its driver have been bumped to version 17. Flyway has been bumped to version 10. @@ -55,6 +59,12 @@ An optimization has been added for the Globus upload workflow, with a correspond See the [Database Settings](https://guides.dataverse.org/en/6.5/installation/config.html#GlobusBatchLookupSize) section of the Guides, #10977, and #11040 for more information. +### Harvesting "oai_dc" Metadata Prefix When Extended With Specific Namespaces + +Some repository extend the "oai_dc" metadata prefix with specific namespaces. In this case, harvesting of these datasets was not possible because an XML parsing error was raised. + +Harvesting of these datasets has been fixed by excluding tags with namespaces that are not "dc:" and harvest only metadata with the "dc" namespace. See #10837. + ### Harvested Dataset PID from Record Header When harvesting, Dataverse can now use the identifier from the OAI-PMH record header as the persistent id for the harvested dataset. @@ -63,6 +73,10 @@ This will allow harvesting from sources that do not include a persistent id in t It is also possible to optionally configure a harvesting client to use this OAI-PMH identifier as the **preferred** choice for the persistent id. See the [Harvesting Clients API](https://guides.dataverse.org/en/6.5/api/native-api.html#create-a-harvesting-client) section of the Guides, #11049 and #10982 for more information. +### Harvested Datasets Can Have Multiple "otherId" Values + +When harvesting using the DDI format, datasets can now have multiple "otherId" values. See #10772. + ## Bugs Fixed ### Updating Files Now Possible When Latest and Only Dataset Version is Deaccessioned @@ -73,6 +87,10 @@ When a dataset was deaccessioned and was the only previous version it would caus The superuser-only feature of filtering by a username on the My Data page was not working. The "Results for Username" field now returns data for the desired user. See also #7239 and #10980. +### Better Handling of Parallel Edit/Publish Errors + +Improvements have been made in handling the errors when a dataset has been edited in one browser window and an attempt is made to edit/publish it in another. (This practice is discouraged.) See #10793 and #10794. + ### Version Differences Details Sorting Added In order to facilitate the comparison between the draft version and the published version of a dataset, a sort on subfields has been added. See #10969. @@ -127,11 +145,23 @@ Previously, not setting these fields meant using a copy of the settings from the The JSON payload of the "get collection" endpoint has been extended to include properties isMetadataBlockRoot and isFacetRoot. See also [the docs](https://preview.guides.gdcc.io/en/develop/api/native-api.html#view-a-dataverse-collection), #11012, and #11013. -### Get API Token Supports Any Auth +### Image URLs from the Search API + +As of 6.4 (thanks to #10855) `image_url` is being returned from the Search API. The logic has been updated to only show the image if each of the following are true: + +1. The DataFile is not Harvested +2. A Thumbnail is available for the Datafile +3. If the Datafile is Restricted then the caller must have Download File Permission for the Datafile +4. The Datafile is NOT actively embargoed +5. The Datafile's retention period has NOT expired + +See also #10875 and #10886. + +## API Tokens -The `/api/users/token` endpoint has been extended to support any auth mechanism for retrieving the token information. +An optional query parameter called "returnExpiration" has been added to the "/api/users/token/recreate" endpoint, which, if set to true, returns the expiration time in the response message. See [the docs](https://preview.guides.gdcc.io/en/develop/api/native-api.html#recreate-a-token), #10857 and #10858. -Previously, this endpoint only accepted an API token to retrieve its information. Now, it accepts any authentication mechanism and returns the associated API token information. See #10914 and #10924. +The `/api/users/token` endpoint has been extended to support any auth mechanism for retrieving the token information. Previously, this endpoint only accepted an API token to retrieve its information. Now, it accepts any authentication mechanism and returns the associated API token information. See #10914 and #10924. ## Settings Added From c331d895573af5edcbafb94afeeef8dd9c265456 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 5 Dec 2024 13:19:44 -0500 Subject: [PATCH 08/34] added last of the snippets (for now) #10952 --- doc/release-notes/10379-MetricsBugsFixes.md | 10 ------ .../10661-guestbook-email-bug-fix.md | 4 --- .../10688_whitespace_trimming.md | 6 ---- .../10697-improve-permission-indexing.md | 7 ---- ...C Citation and DOI parsing improvements.md | 3 -- ...0742-newest-oldest-sort-order-backwards.md | 3 -- doc/release-notes/6.5-release-notes.md | 35 +++++++++++++++++++ 7 files changed, 35 insertions(+), 33 deletions(-) delete mode 100644 doc/release-notes/10379-MetricsBugsFixes.md delete mode 100644 doc/release-notes/10661-guestbook-email-bug-fix.md delete mode 100644 doc/release-notes/10688_whitespace_trimming.md delete mode 100644 doc/release-notes/10697-improve-permission-indexing.md delete mode 100644 doc/release-notes/10708 - MDC Citation and DOI parsing improvements.md delete mode 100644 doc/release-notes/10742-newest-oldest-sort-order-backwards.md diff --git a/doc/release-notes/10379-MetricsBugsFixes.md b/doc/release-notes/10379-MetricsBugsFixes.md deleted file mode 100644 index 0ebc6d99f0b..00000000000 --- a/doc/release-notes/10379-MetricsBugsFixes.md +++ /dev/null @@ -1,10 +0,0 @@ - -### Metrics API Bug fixes - -Two bugs in the Metrics API have been fixed: - -- The /datasets and /datasets/byMonth endpoints could report incorrect values if/when they have been called using the dataLocation parameter (which allows getting metrics for local, remote (harvested), or all datasets) as the metrics cache was not storing different values for these cases. - -- Metrics endpoints who's calculation relied on finding the latest published datasetversion were incorrect if/when the minor version number was > 9. - -When deploying the new release, the [/api/admin/clearMetricsCache](https://guides.dataverse.org/en/latest/api/native-api.html#metrics) API should be called to remove old cached values that may be incorrect. \ No newline at end of file diff --git a/doc/release-notes/10661-guestbook-email-bug-fix.md b/doc/release-notes/10661-guestbook-email-bug-fix.md deleted file mode 100644 index 05e70c9762a..00000000000 --- a/doc/release-notes/10661-guestbook-email-bug-fix.md +++ /dev/null @@ -1,4 +0,0 @@ - -### Guestbook Email Validation Bug fix - -Guestbook UI Form: Email address is now checked for valid email format diff --git a/doc/release-notes/10688_whitespace_trimming.md b/doc/release-notes/10688_whitespace_trimming.md deleted file mode 100644 index 52904c00fbf..00000000000 --- a/doc/release-notes/10688_whitespace_trimming.md +++ /dev/null @@ -1,6 +0,0 @@ -### Added whitespace trimming to uploaded custom metadata TSV files - -When loading custom metadata blocks using the `api/admin/datasetfield/load` API, whitespace can be introduced into field names. -This change trims whitespace at the beginning and end of all values read into the API before persisting them. - -For more information, see #10688. diff --git a/doc/release-notes/10697-improve-permission-indexing.md b/doc/release-notes/10697-improve-permission-indexing.md deleted file mode 100644 index b232b1c4d3c..00000000000 --- a/doc/release-notes/10697-improve-permission-indexing.md +++ /dev/null @@ -1,7 +0,0 @@ -### Reindexing after a role assignment is less memory intensive - -Adding/removing a user from a role on a collection, particularly the root collection, could lead to a significant increase in memory use resulting in Dataverse itself failing with an out-of-memory condition. Such changes now consume much less memory. - -If you have experienced out-of-memory failures in Dataverse in the past that could have been caused by this problem, you may wish to run a [reindex in place](https://guides.dataverse.org/en/latest/admin/solr-search-index.html#reindex-in-place) to update any out-of-date information. - -For more information, see #10697 and #10698. diff --git a/doc/release-notes/10708 - MDC Citation and DOI parsing improvements.md b/doc/release-notes/10708 - MDC Citation and DOI parsing improvements.md deleted file mode 100644 index 86c1bb14d32..00000000000 --- a/doc/release-notes/10708 - MDC Citation and DOI parsing improvements.md +++ /dev/null @@ -1,3 +0,0 @@ -MDC Citation retrieval with the PID settings has been fixed. -PID parsing in Dataverse is now case insensitive, improving interaction with services that may change the case of PIDs. -Warnings related to managed/excluded PID lists for PID providers have been reduced diff --git a/doc/release-notes/10742-newest-oldest-sort-order-backwards.md b/doc/release-notes/10742-newest-oldest-sort-order-backwards.md deleted file mode 100644 index 0afaf45449d..00000000000 --- a/doc/release-notes/10742-newest-oldest-sort-order-backwards.md +++ /dev/null @@ -1,3 +0,0 @@ -## Minor bug fix to UI to fix the order of the files on the Dataset Files page when ordering by Date - -A fix was made to the ui to fix the ordering 'Newest' and 'Oldest' which were reversed diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index b66ae2f1b21..47034806d90 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -33,6 +33,14 @@ See also #8184, #8185, #10950, and #10961. Differencing between dataset versions, which is done during dataset edit operations and to populate the dataset page versions table has been made signficantly more scalable. See #10814 and #10818. +### Reindexing After a Role Assignment is Less Memory Intensive + +Adding or removing a user from a role on a collection, particularly the root collection, could lead to a significant increase in memory use, resulting in Dataverse itself failing with an out-of-memory condition. Such changes now consume much less memory. + +If you have experienced out-of-memory failures in Dataverse in the past that could have been caused by this problem, you may wish to run a [reindex in place](https://guides.dataverse.org/en/latest/admin/solr-search-index.html#reindex-in-place) to update any out-of-date information. + +For more information, see #10697 and #10698. + ### PostgreSQL and Flyway Updates This release bumps the version of PostgreSQL and Flyway used in containers as well as the PostgreSQL JDBC driver used all installations, including classic (non-Docker) installations. PostgreSQL and its driver have been bumped to version 17. Flyway has been bumped to version 10. @@ -79,6 +87,14 @@ When harvesting using the DDI format, datasets can now have multiple "otherId" v ## Bugs Fixed +### Sort Order for Files + +"Newest" and "Oldest" were reversed when sorting files on the dataset landing page. This has been fixed. See #10742 and #11000. + +### Guestbook Email Validation + +In the Guestbook UI form, the email address is now checked for validity. See #10661 and #11022. + ### Updating Files Now Possible When Latest and Only Dataset Version is Deaccessioned When a dataset was deaccessioned and was the only previous version it would cause an error when trying to update the files. This has been fixed. See #9351 and #10901. @@ -109,6 +125,10 @@ A bug fix was made that gets the major version of a Dataset when all major versi A minor bug fix was made to avoid sending a useless ", null" in the DataCiteXML sent to DataCite and in the DataCite export when a dataset has a metadata entry for "Software Name" and no entry for "Software Version". The bug fix will update datasets upon publication. Anyone with existing published datasets with this problem can be fixed by [pushing updated metadata to DataCite for affected datasets](https://guides.dataverse.org/en/6.5/admin/dataverses-datasets.html#update-metadata-for-a-published-dataset-at-the-pid-provider) and [re-exporting the dataset metadata](https://guides.dataverse.org/en/6.5/admin/metadataexport.html#batch-exports-through-the-api) or by steps 9 and 10 in the [v6.4 release notes](https://github.com/IQSS/dataverse/releases/tag/v6.4) to update and re-export all datasets. See also #10919. +### PIDs and Make Data Count Citation Retrieval + +Make Data Count (MDC) citation retrieval with the PID settings has been fixed. PID parsing in Dataverse is now case insensitive, improving interaction with services that may change the case of PIDs. Warnings related to managed/excluded PID lists for PID providers have been reduced. See #10708. + ### Globus "missing properties" Logging Fixed In previous releases, logging would show Globus-related strings were missing from properties files. This has been fixed. See #11030 for details. @@ -145,6 +165,10 @@ Previously, not setting these fields meant using a copy of the settings from the The JSON payload of the "get collection" endpoint has been extended to include properties isMetadataBlockRoot and isFacetRoot. See also [the docs](https://preview.guides.gdcc.io/en/develop/api/native-api.html#view-a-dataverse-collection), #11012, and #11013. +### Whitespace Trimming When Loading Metadata Block TSV Files + +When loading custom metadata blocks using the `api/admin/datasetfield/load` API, whitespace can be introduced into field names. Whitespace is now trimmed from the beginning and end of all values read into the API before persisting them. See #10688 and #10696. + ### Image URLs from the Search API As of 6.4 (thanks to #10855) `image_url` is being returned from the Search API. The logic has been updated to only show the image if each of the following are true: @@ -157,6 +181,17 @@ As of 6.4 (thanks to #10855) `image_url` is being returned from the Search API. See also #10875 and #10886. +### Metrics API Bug Fixes + +Two bugs in the Metrics API have been fixed: + +- The /datasets and /datasets/byMonth endpoints could report incorrect values if/when they have been called using the dataLocation parameter (which allows getting metrics for local, remote (harvested), or all datasets) as the metrics cache was not storing different values for these cases. + +- Metrics endpoints who's calculation relied on finding the latest published datasetversion were incorrect if/when the minor version number was > 9. + +When deploying the new release, the [/api/admin/clearMetricsCache](https://guides.dataverse.org/en/latest/api/native-api.html#metrics) API should be called to remove old cached values that may be incorrect. +See #10379 and #10865. + ## API Tokens An optional query parameter called "returnExpiration" has been added to the "/api/users/token/recreate" endpoint, which, if set to true, returns the expiration time in the response message. See [the docs](https://preview.guides.gdcc.io/en/develop/api/native-api.html#recreate-a-token), #10857 and #10858. From cabf738f0d94495c7156dcc3a48feefd51f72a17 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 5 Dec 2024 15:23:10 -0500 Subject: [PATCH 09/34] fix URLs #10952 --- doc/release-notes/6.5-release-notes.md | 30 +++++++++++++------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index 47034806d90..c12da62009a 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -25,7 +25,7 @@ Also, additional information about the creation of Preview URLs has been added t Any Private URLs created in previous versions of Dataverse will continue to work. -The old "privateUrl" API endpoints for the creation and deletion of Preview (formerly Private) URLs have been deprecated. They will continue to work but please switch to the "previewUrl" equivalents that have been [documented](https://dataverse-guide--10961.org.readthedocs.build/en/10961/api/native-api.html#create-a-preview-url-for-a-dataset) in the API Guide. +The old "privateUrl" API endpoints for the creation and deletion of Preview (formerly Private) URLs have been deprecated. They will continue to work but please switch to the "previewUrl" equivalents that have been [documented](https://guides.dataverse.org/en/6.5/api/native-api.html#create-a-preview-url-for-a-dataset) in the API Guide. See also #8184, #8185, #10950, and #10961. @@ -59,7 +59,7 @@ See also #10889 and #10912. Configuration and documentation has been added to explain how to set up multiple languages (e.g. English and French) in the tutorial for setting up Dataverse in Docker. -See also [the docs](https://dataverse-guide--10940.org.readthedocs.build/en/10940/container/running/demo.html#multiple-languages), #10939, and #10940. +See also [the docs](https://guides.dataverse.org/en/6.5/container/running/demo.html#multiple-languages), #10939, and #10940. ### GlobusBatchLookupSize @@ -73,11 +73,11 @@ Some repository extend the "oai_dc" metadata prefix with specific namespaces. In Harvesting of these datasets has been fixed by excluding tags with namespaces that are not "dc:" and harvest only metadata with the "dc" namespace. See #10837. -### Harvested Dataset PID from Record Header +### Harvested Dataset PID from Record Header When harvesting, Dataverse can now use the identifier from the OAI-PMH record header as the persistent id for the harvested dataset. -This will allow harvesting from sources that do not include a persistent id in their oai_dc metadata records, but use valid DOIs or handles as the OAI-PMH record header identifiers. +This will allow harvesting from sources that do not include a persistent id in their oai_dc metadata records, but use valid DOIs or handles as the OAI-PMH record header identifiers. It is also possible to optionally configure a harvesting client to use this OAI-PMH identifier as the **preferred** choice for the persistent id. See the [Harvesting Clients API](https://guides.dataverse.org/en/6.5/api/native-api.html#create-a-harvesting-client) section of the Guides, #11049 and #10982 for more information. @@ -137,7 +137,7 @@ In previous releases, logging would show Globus-related strings were missing fro ### Editing Collections -A new endpoint (`PUT /api/dataverses/`) for updating an existing collection (dataverse) has been added. It uses the same JSON structure as the one used for collection creation. See also [the docs](https://preview.guides.gdcc.io/en/develop/api/native-api.html#update-a-dataverse-collection), #10904, and #10925. +A new endpoint (`PUT /api/dataverses/`) for updating an existing collection (dataverse) has been added. It uses the same JSON structure as the one used for collection creation. See also [the docs](https://guides.dataverse.org/en/6.5/api/native-api.html#update-a-dataverse-collection), #10904, and #10925. ### fileCount Added to Search API @@ -145,25 +145,25 @@ A new search field called `fileCount` can be searched to discover the number of ### List Dataset Metadata Exporters -A list of available dataset metadata exporters can now be retrieved programmatically via API. See [the docs](https://dataverse-guide--10739.org.readthedocs.build/en/10739/api/native-api.html#get-export-formats) and #10739. +A list of available dataset metadata exporters can now be retrieved programmatically via API. See [the docs](https://guides.dataverse.org/en/6.5/api/native-api.html#get-export-formats) and #10739. ### Comparing Dataset Versions -An API has been added to compare dataset versions. See [the docs](https://preview.guides.gdcc.io/en/develop/api/native-api.html#compare-versions-of-a-dataset), #10888, and #10945. +An API has been added to compare dataset versions. See [the docs](https://guides.dataverse.org/en/6.5/api/native-api.html#compare-versions-of-a-dataset), #10888, and #10945. ### Audit Data Files -A superuser-only API endpoint has been added to audit datasets with data files where the physical files are missing or the file metadata is missing. See [the docs](https://dataverse-guide--11016.org.readthedocs.build/en/11016/api/native-api.html#datafile-audit), #11016, and [#220](https://github.com/IQSS/dataverse.harvard.edu/issues/220). +A superuser-only API endpoint has been added to audit datasets with data files where the physical files are missing or the file metadata is missing. See [the docs](https://guides.dataverse.org/en/6.5/api/native-api.html#datafile-audit), #11016, and [#220](https://github.com/IQSS/dataverse.harvard.edu/issues/220). ### Update Collection API Inheritance The update collection (dataverse) API endpoint has been updated to support an "inherit from parent" configuration for metadata blocks, facets, and input levels. -Previously, not setting these fields meant using a copy of the settings from the parent collection, which could get out of sync.. See also [the docs](https://preview.guides.gdcc.io/en/develop/api/native-api.html#update-a-dataverse-collection), #11018, and #11026. +Previously, not setting these fields meant using a copy of the settings from the parent collection, which could get out of sync.. See also [the docs](https://guides.dataverse.org/en/6.5/api/native-api.html#update-a-dataverse-collection), #11018, and #11026. ### isMetadataBlockRoot and isFacetRoot -The JSON payload of the "get collection" endpoint has been extended to include properties isMetadataBlockRoot and isFacetRoot. See also [the docs](https://preview.guides.gdcc.io/en/develop/api/native-api.html#view-a-dataverse-collection), #11012, and #11013. +The JSON payload of the "get collection" endpoint has been extended to include properties isMetadataBlockRoot and isFacetRoot. See also [the docs](https://guides.dataverse.org/en/6.5/api/native-api.html#view-a-dataverse-collection), #11012, and #11013. ### Whitespace Trimming When Loading Metadata Block TSV Files @@ -185,16 +185,16 @@ See also #10875 and #10886. Two bugs in the Metrics API have been fixed: -- The /datasets and /datasets/byMonth endpoints could report incorrect values if/when they have been called using the dataLocation parameter (which allows getting metrics for local, remote (harvested), or all datasets) as the metrics cache was not storing different values for these cases. +- The /datasets and /datasets/byMonth endpoints could report incorrect values if/when they have been called using the dataLocation parameter (which allows getting metrics for local, remote (harvested), or all datasets) as the metrics cache was not storing different values for these cases. -- Metrics endpoints who's calculation relied on finding the latest published datasetversion were incorrect if/when the minor version number was > 9. +- Metrics endpoints who's calculation relied on finding the latest published datasetversion were incorrect if/when the minor version number was > 9. When deploying the new release, the [/api/admin/clearMetricsCache](https://guides.dataverse.org/en/latest/api/native-api.html#metrics) API should be called to remove old cached values that may be incorrect. See #10379 and #10865. ## API Tokens -An optional query parameter called "returnExpiration" has been added to the "/api/users/token/recreate" endpoint, which, if set to true, returns the expiration time in the response message. See [the docs](https://preview.guides.gdcc.io/en/develop/api/native-api.html#recreate-a-token), #10857 and #10858. +An optional query parameter called "returnExpiration" has been added to the "/api/users/token/recreate" endpoint, which, if set to true, returns the expiration time in the response message. See [the docs](https://guides.dataverse.org/en/6.5/api/native-api.html#recreate-a-token), #10857 and #10858. The `/api/users/token` endpoint has been extended to support any auth mechanism for retrieving the token information. Previously, this endpoint only accepted an API token to retrieve its information. Now, it accepts any authentication mechanism and returns the associated API token information. See #10914 and #10924. @@ -204,11 +204,11 @@ The `/api/users/token` endpoint has been extended to support any auth mechanism ## Backward Incompatible Changes -Generally speaking, see the [API Changelog](https://preview.guides.gdcc.io/en/develop/api/changelog.html) for a list of backward-incompatible changes. +Generally speaking, see the [API Changelog](https://guides.dataverse.org/en/6.5/api/changelog.html) for a list of backward-incompatible changes. ### List Collections Linked to a Dataset -The API endpoint that returns a list of collections that a dataset has been linked to has been improved to provide a more structured JSON response. See [the docs](https://dataverse-guide--9665.org.readthedocs.build/en/9665/admin/dataverses-datasets.html#list-collections-that-are-linked-from-a-dataset), #9650, and #9665. +The API endpoint that returns a list of collections that a dataset has been linked to has been improved to provide a more structured JSON response. See [the docs](https://guides.dataverse.org/en/6.5/admin/dataverses-datasets.html#list-collections-that-are-linked-from-a-dataset), #9650, and #9665. ## Complete List of Changes From 8d1b4a18dc2a40b29d48b797387cf8d4f1d5d07e Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 5 Dec 2024 15:40:06 -0500 Subject: [PATCH 10/34] add highlights and rework content a bit #10952 --- doc/release-notes/6.5-release-notes.md | 37 +++++++++++++++----------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index c12da62009a..c2d21062914 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -6,18 +6,19 @@ This release brings new features, enhancements, and bug fixes to Dataverse. Than ## Release Highlights -New features in Dataverse 6.5: - +Highlights in Dataverse 6.5 include: + +- new API endpoints, including editing of collections, Search API file counts, listing of exporters, comparing dataset versions, and auditing data files +- UX improvements, especially Preview URLs +- increased harvesting flexibility +- performance gains +- a security vulnerability addressed +- many bugs fixes - and more! Please see below. - ## Features Added -### Longer Custom Questions in Guestbooks - -Custom questions in Guestbooks can now be more than 255 characters and the bug causing a silent failure when questions were longer than this limit has been fixed. See also #9492, #10117, #10118. - -### Private URL Renamed to Preview URL +### Private URL Renamed to Preview URL and Improved With this release the name of the URL that may be used by dataset administrators to share a draft version of a dataset has been changed from Private URL to Preview URL. @@ -41,6 +42,10 @@ If you have experienced out-of-memory failures in Dataverse in the past that cou For more information, see #10697 and #10698. +### Longer Custom Questions in Guestbooks + +Custom questions in Guestbooks can now be more than 255 characters and the bug causing a silent failure when questions were longer than this limit has been fixed. See also #9492, #10117, #10118. + ### PostgreSQL and Flyway Updates This release bumps the version of PostgreSQL and Flyway used in containers as well as the PostgreSQL JDBC driver used all installations, including classic (non-Docker) installations. PostgreSQL and its driver have been bumped to version 17. Flyway has been bumped to version 10. @@ -61,12 +66,6 @@ Configuration and documentation has been added to explain how to set up multiple See also [the docs](https://guides.dataverse.org/en/6.5/container/running/demo.html#multiple-languages), #10939, and #10940. -### GlobusBatchLookupSize - -An optimization has been added for the Globus upload workflow, with a corresponding new database setting: `:GlobusBatchLookupSize` - -See the [Database Settings](https://guides.dataverse.org/en/6.5/installation/config.html#GlobusBatchLookupSize) section of the Guides, #10977, and #11040 for more information. - ### Harvesting "oai_dc" Metadata Prefix When Extended With Specific Namespaces Some repository extend the "oai_dc" metadata prefix with specific namespaces. In this case, harvesting of these datasets was not possible because an XML parsing error was raised. @@ -85,6 +84,12 @@ It is also possible to optionally configure a harvesting client to use this OAI- When harvesting using the DDI format, datasets can now have multiple "otherId" values. See #10772. +### GlobusBatchLookupSize + +An optimization has been added for the Globus upload workflow, with a corresponding new database setting: `:GlobusBatchLookupSize` + +See the [Database Settings](https://guides.dataverse.org/en/6.5/installation/config.html#GlobusBatchLookupSize) section of the Guides, #10977, and #11040 for more information. + ## Bugs Fixed ### Sort Order for Files @@ -99,7 +104,7 @@ In the Guestbook UI form, the email address is now checked for validity. See #10 When a dataset was deaccessioned and was the only previous version it would cause an error when trying to update the files. This has been fixed. See #9351 and #10901. -### My Data Filter by Username Feature Fixed +### My Data Filter by Username Feature Restored The superuser-only feature of filtering by a username on the My Data page was not working. The "Results for Username" field now returns data for the desired user. See also #7239 and #10980. @@ -121,7 +126,7 @@ This version of Dataverse fix this, and includes internationalization in the fac A bug fix was made that gets the major version of a Dataset when all major versions were deaccessioned. This fixes the incorrect showing of the files as "Unpublished" in the search list even when they are published. This fix affects the indexing, meaning these datasets must be re-indexed once Dataverse is updated. See also #10947 and #10974. -### Minor DataCiteXML Fix +### Minor DataCiteXML Fix (Useless Null) A minor bug fix was made to avoid sending a useless ", null" in the DataCiteXML sent to DataCite and in the DataCite export when a dataset has a metadata entry for "Software Name" and no entry for "Software Version". The bug fix will update datasets upon publication. Anyone with existing published datasets with this problem can be fixed by [pushing updated metadata to DataCite for affected datasets](https://guides.dataverse.org/en/6.5/admin/dataverses-datasets.html#update-metadata-for-a-published-dataset-at-the-pid-provider) and [re-exporting the dataset metadata](https://guides.dataverse.org/en/6.5/admin/metadataexport.html#batch-exports-through-the-api) or by steps 9 and 10 in the [v6.4 release notes](https://github.com/IQSS/dataverse/releases/tag/v6.4) to update and re-export all datasets. See also #10919. From 7503ea36833834890ba9e8edd99f037610e775ba Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 5 Dec 2024 15:48:50 -0500 Subject: [PATCH 11/34] add Relation Type bug (no release note snippet) #10952 #10926 --- doc/release-notes/6.5-release-notes.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index c2d21062914..3d18d1ae197 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -92,6 +92,10 @@ See the [Database Settings](https://guides.dataverse.org/en/6.5/installation/con ## Bugs Fixed +### Relation Type (Related Publication) and DataCite + +The subfield "Relation Type" was added to the field "Related Publication" in Dataverse 6.4 (#10632) but couldn't be used without workarounds described in an [announcement](https://groups.google.com/g/dataverse-community/c/zlRGJtu3x4g/m/GtVZ26uaBQAJ) about the problem. The bug has been fixed and workarounds and are longer required. See also #10926. + ### Sort Order for Files "Newest" and "Oldest" were reversed when sorting files on the dataset landing page. This has been fixed. See #10742 and #11000. From 77834a24a64ad8e29c377cb836a0d3128a9af846 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 5 Dec 2024 15:50:57 -0500 Subject: [PATCH 12/34] demo docker feature, fix heading #10952 --- doc/release-notes/6.5-release-notes.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index 3d18d1ae197..30a2fad5504 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -60,12 +60,6 @@ The Docker compose file used for [evaluations or demos](https://guides.dataverse See also #10889 and #10912. -## Multiple Languages in Docker - -Configuration and documentation has been added to explain how to set up multiple languages (e.g. English and French) in the tutorial for setting up Dataverse in Docker. - -See also [the docs](https://guides.dataverse.org/en/6.5/container/running/demo.html#multiple-languages), #10939, and #10940. - ### Harvesting "oai_dc" Metadata Prefix When Extended With Specific Namespaces Some repository extend the "oai_dc" metadata prefix with specific namespaces. In this case, harvesting of these datasets was not possible because an XML parsing error was raised. @@ -84,6 +78,12 @@ It is also possible to optionally configure a harvesting client to use this OAI- When harvesting using the DDI format, datasets can now have multiple "otherId" values. See #10772. +### Multiple Languages in Docker + +Configuration and documentation has been added to explain how to set up multiple languages (e.g. English and French) in the tutorial for setting up Dataverse in Docker. + +See also [the docs](https://guides.dataverse.org/en/6.5/container/running/demo.html#multiple-languages), #10939, and #10940. + ### GlobusBatchLookupSize An optimization has been added for the Globus upload workflow, with a corresponding new database setting: `:GlobusBatchLookupSize` From 8ce290902a1d4a3ea2bb54f7f2474b5102f0724a Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 5 Dec 2024 16:02:06 -0500 Subject: [PATCH 13/34] tweaks #10952 --- doc/release-notes/6.5-release-notes.md | 24 +++++------------------- 1 file changed, 5 insertions(+), 19 deletions(-) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index 30a2fad5504..45b53c673b7 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -132,7 +132,7 @@ A bug fix was made that gets the major version of a Dataset when all major versi ### Minor DataCiteXML Fix (Useless Null) -A minor bug fix was made to avoid sending a useless ", null" in the DataCiteXML sent to DataCite and in the DataCite export when a dataset has a metadata entry for "Software Name" and no entry for "Software Version". The bug fix will update datasets upon publication. Anyone with existing published datasets with this problem can be fixed by [pushing updated metadata to DataCite for affected datasets](https://guides.dataverse.org/en/6.5/admin/dataverses-datasets.html#update-metadata-for-a-published-dataset-at-the-pid-provider) and [re-exporting the dataset metadata](https://guides.dataverse.org/en/6.5/admin/metadataexport.html#batch-exports-through-the-api) or by steps 9 and 10 in the [v6.4 release notes](https://github.com/IQSS/dataverse/releases/tag/v6.4) to update and re-export all datasets. See also #10919. +A minor bug fix was made to avoid sending a useless ", null" in the DataCiteXML sent to DataCite and in the DataCite export when a dataset has a metadata entry for "Software Name" and no entry for "Software Version". The bug fix will update datasets upon publication. Anyone with existing published datasets with this problem can be fixed by [pushing updated metadata to DataCite for affected datasets](https://guides.dataverse.org/en/6.5/admin/dataverses-datasets.html#update-metadata-for-a-published-dataset-at-the-pid-provider) and [re-exporting the dataset metadata](https://guides.dataverse.org/en/6.5/admin/metadataexport.html#batch-exports-through-the-api). See "Pushing updated metadata to DataCite" in the upgrade instructions below. See also #10919. ### PIDs and Make Data Count Citation Retrieval @@ -201,7 +201,7 @@ Two bugs in the Metrics API have been fixed: When deploying the new release, the [/api/admin/clearMetricsCache](https://guides.dataverse.org/en/latest/api/native-api.html#metrics) API should be called to remove old cached values that may be incorrect. See #10379 and #10865. -## API Tokens +### API Tokens An optional query parameter called "returnExpiration" has been added to the "/api/users/token/recreate" endpoint, which, if set to true, returns the expiration time in the response message. See [the docs](https://guides.dataverse.org/en/6.5/api/native-api.html#recreate-a-token), #10857 and #10858. @@ -219,7 +219,6 @@ Generally speaking, see the [API Changelog](https://guides.dataverse.org/en/6.5/ The API endpoint that returns a list of collections that a dataset has been linked to has been improved to provide a more structured JSON response. See [the docs](https://guides.dataverse.org/en/6.5/admin/dataverses-datasets.html#list-collections-that-are-linked-from-a-dataset), #9650, and #9665. - ## Complete List of Changes For the complete list of code changes in this release, see the [6.5 milestone](https://github.com/IQSS/dataverse/issues?q=milestone%3A6.5+is%3Aclosed) in GitHub. @@ -228,7 +227,6 @@ For the complete list of code changes in this release, see the [6.5 milestone](h For help with upgrading, installing, or general questions please post to the [Dataverse Community Google Group](https://groups.google.com/g/dataverse-community) or email support@dataverse.org. - ## Installation If this is a new installation, please follow our [Installation Guide](https://guides.dataverse.org/en/latest/installation/). Please don't be shy about [asking for help](https://guides.dataverse.org/en/latest/installation/intro.html#getting-help) if you need it! @@ -297,13 +295,7 @@ service payara start 6\. Update metadata blocks -These changes reflect incremental improvements made to the handling of core metadata fields. - -```shell -wget https://raw.githubusercontent.com/IQSS/dataverse/v6.4/scripts/api/data/metadatablocks/citation.tsv - -curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file citation.tsv -``` +No changes required for this release. 7\. Update Solr schema.xml file. Start with the standard v6.4 schema.xml, then, if your installation uses any custom or experimental metadata blocks, update it to include the extra fields (step 7a). @@ -350,8 +342,6 @@ The API above rebuilds the existing index "in place". If you want to be absolute 9\. Run reExportAll to update dataset metadata exports -This step is necessary because of changes described above for the `Datacite` and `oai_dc` export formats. - Below is the simple way to reexport all dataset metadata. For more advanced usage, please see [the guides](http://guides.dataverse.org/en/6.4/admin/metadataexport.html#batch-exports-through-the-api). ```shell @@ -360,11 +350,9 @@ curl http://localhost:8080/api/admin/metadata/reExportAll 10\. Pushing updated metadata to DataCite -(If you don't use DataCite, you can skip this.) +(If you don't use DataCite, you can skip this. Also, if you aren't affected by the "useless null" bug described above, you can skip this.) -Above you updated the citation metadata block and Solr with the new "relationType" field. With these two changes, the "Relation Type" fields will be available and creation/publication of datasets will result in the expanded XML being sent to DataCite. You've also already run "reExportAll" to update the `Datacite` metadata export format. - -Entries at DataCite for published datasets can be updated by a superuser using an API call (newly [documented](https://guides.dataverse.org/en/6.4/admin/dataverses-datasets.html#update-metadata-for-all-published-datasets-at-the-pid-provider)): +Entries at DataCite for published datasets can be updated by a superuser using an API call (newly [documented](https://guides.dataverse.org/en/6.5/admin/dataverses-datasets.html#update-metadata-for-all-published-datasets-at-the-pid-provider)): `curl -X POST -H 'X-Dataverse-key:' http://localhost:8080/api/datasets/modifyRegistrationPIDMetadataAll` @@ -377,5 +365,3 @@ Failures may occur if PIDs were never registered, or if they were never made fin PIDs can also be updated by a superuser on a per-dataset basis using `curl -X POST -H 'X-Dataverse-key:' http://localhost:8080/api/datasets//modifyRegistrationMetadata` - -### Additional Upgrade Steps From 7885fa26db09a06d3c4e49e59a266d2b8ed3ec9b Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 5 Dec 2024 16:13:42 -0500 Subject: [PATCH 14/34] typo Co-authored-by: Omer Fahim --- doc/release-notes/6.5-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index 45b53c673b7..60d26b58c70 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -13,7 +13,7 @@ Highlights in Dataverse 6.5 include: - increased harvesting flexibility - performance gains - a security vulnerability addressed -- many bugs fixes +- many bug fixes - and more! Please see below. ## Features Added From 48d77d33ab6dada93495e445d7ac1d197759b06f Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 5 Dec 2024 16:14:01 -0500 Subject: [PATCH 15/34] spelling Co-authored-by: Omer Fahim --- doc/release-notes/6.5-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index 60d26b58c70..4d900b12485 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -32,7 +32,7 @@ See also #8184, #8185, #10950, and #10961. ### More Scalable Dataset Version Differencing -Differencing between dataset versions, which is done during dataset edit operations and to populate the dataset page versions table has been made signficantly more scalable. See #10814 and #10818. +Differencing between dataset versions, which is done during dataset edit operations and to populate the dataset page versions table has been made significantly more scalable. See #10814 and #10818. ### Reindexing After a Role Assignment is Less Memory Intensive From 5e21b80dcc9c140da8afed4de8ac50ac8f878fab Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 6 Dec 2024 12:35:53 -0500 Subject: [PATCH 16/34] remove trailing ` --- doc/release-notes/6.5-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index 4d900b12485..7488013b1dd 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -249,7 +249,7 @@ If you are running Payara as a non-root user (and you should be!), **remember no In the following commands, we assume that Payara 6 is installed in `/usr/local/payara6`. If not, adjust as needed. ```shell -export PAYARA=/usr/local/payara6` +export PAYARA=/usr/local/payara6 ``` (or `setenv PAYARA /usr/local/payara6` if you are using a `csh`-like shell) From 6cf718e24819040c030391c99dcec12597393494 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 6 Dec 2024 12:55:25 -0500 Subject: [PATCH 17/34] tweak deploy step, remove mdb update and schema.xml steps (not needed) #10952 --- doc/release-notes/6.5-release-notes.md | 61 +++++++------------------- 1 file changed, 15 insertions(+), 46 deletions(-) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index 7488013b1dd..3d05f9e9edb 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -254,20 +254,26 @@ export PAYARA=/usr/local/payara6 (or `setenv PAYARA /usr/local/payara6` if you are using a `csh`-like shell) -1\. Undeploy the previous version +1\. List deployed applications + +```shell +$PAYARA/bin/asadmin list-applications +``` + +2\. Undeploy the previous version (should match "list-applications" above) ```shell $PAYARA/bin/asadmin undeploy dataverse-6.4 ``` -2\. Stop and start Payara +3\. Stop and start Payara ```shell -service payara stop +sudo service payara stop sudo service payara start ``` -3\. Deploy this version +4\. Deploy this version ```shell $PAYARA/bin/asadmin deploy dataverse-6.5.war @@ -282,55 +288,18 @@ rm -rf $PAYARA/glassfish/domains/domain1/osgi-cache rm -rf $PAYARA/glassfish/domains/domain1/lib/databases ``` -4\. For installations with internationalization: +5\. For installations with internationalization: Please remember to update translations via [Dataverse language packs](https://github.com/GlobalDataverseCommunityConsortium/dataverse-language-packs). -5\. Restart Payara +6\. Restart Payara ```shell service payara stop service payara start ``` -6\. Update metadata blocks - -No changes required for this release. - -7\. Update Solr schema.xml file. Start with the standard v6.4 schema.xml, then, if your installation uses any custom or experimental metadata blocks, update it to include the extra fields (step 7a). - -Stop Solr (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/6.4/installation/prerequisites.html#solr-init-script)). - -```shell -service solr stop -``` - -Replace schema.xml - -```shell -wget https://raw.githubusercontent.com/IQSS/dataverse/v6.4/conf/solr/schema.xml -cp schema.xml /usr/local/solr/solr-9.4.1/server/solr/collection1/conf -``` - -Start Solr (but if you use any custom metadata blocks, perform the next step, 7a first). - -```shell -service solr start -``` - -7a\. For installations with custom or experimental metadata blocks: - -Before starting Solr, update the schema to include all the extra metadata fields that your installation uses. We do this by collecting the output of the Dataverse schema API and feeding it to the `update-fields.sh` script that we supply, as in the example below (modify the command lines as needed to reflect the names of the directories, if different): - -```shell - wget https://raw.githubusercontent.com/IQSS/dataverse/v6.4/conf/solr/update-fields.sh - chmod +x update-fields.sh - curl "http://localhost:8080/api/admin/index/solr/schema" | ./update-fields.sh /usr/local/solr/solr-9.4.1/server/solr/collection1/conf/schema.xml -``` - -Now start Solr. - -8\. Reindex Solr +7\. Reindex Solr Below is the simplest way to reindex Solr: @@ -340,7 +309,7 @@ curl http://localhost:8080/api/admin/index The API above rebuilds the existing index "in place". If you want to be absolutely sure that your index is up-to-date and consistent, you may consider wiping it clean and reindexing everything from scratch (see [the guides](https://guides.dataverse.org/en/latest/admin/solr-search-index.html)). Just note that, depending on the size of your database, a full reindex may take a while and the users will be seeing incomplete search results during that window. -9\. Run reExportAll to update dataset metadata exports +8\. Run reExportAll to update dataset metadata exports Below is the simple way to reexport all dataset metadata. For more advanced usage, please see [the guides](http://guides.dataverse.org/en/6.4/admin/metadataexport.html#batch-exports-through-the-api). @@ -348,7 +317,7 @@ Below is the simple way to reexport all dataset metadata. For more advanced usag curl http://localhost:8080/api/admin/metadata/reExportAll ``` -10\. Pushing updated metadata to DataCite +9\. Pushing updated metadata to DataCite (If you don't use DataCite, you can skip this. Also, if you aren't affected by the "useless null" bug described above, you can skip this.) From a66c0857e4b1f016666faa71811406600e348d20 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 6 Dec 2024 12:56:42 -0500 Subject: [PATCH 18/34] more sudo #10952 --- doc/release-notes/6.5-release-notes.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index 3d05f9e9edb..e21fd0ce0fd 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -295,8 +295,8 @@ Please remember to update translations via [Dataverse language packs](https://gi 6\. Restart Payara ```shell -service payara stop -service payara start +sudo service payara stop +sudo service payara start ``` 7\. Reindex Solr From 0a82885f1825824122883b2bcd25d397a938d3da Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 6 Dec 2024 13:13:24 -0500 Subject: [PATCH 19/34] whoops, we do need to update schema.xml #10952 --- doc/release-notes/6.5-release-notes.md | 51 +++++++++++++++++++++++--- 1 file changed, 46 insertions(+), 5 deletions(-) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index e21fd0ce0fd..e58a6fcce4b 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -150,7 +150,7 @@ A new endpoint (`PUT /api/dataverses/`) for updating an existing col ### fileCount Added to Search API -A new search field called `fileCount` can be searched to discover the number of files per dataset. See also #8941 and #10598. +A new search field called `fileCount` can be searched to discover the number of files per dataset. The upgrade instructions below explain how to update your Solr `schema.xml` file to add the new field. See also #8941 and #10598. ### List Dataset Metadata Exporters @@ -282,7 +282,7 @@ $PAYARA/bin/asadmin deploy dataverse-6.5.war Note: if you have any trouble deploying, stop Payara, remove the following directories, start Payara, and try to deploy again. ```shell -service payara stop +sudo service payara stop rm -rf $PAYARA/glassfish/domains/domain1/generated rm -rf $PAYARA/glassfish/domains/domain1/osgi-cache rm -rf $PAYARA/glassfish/domains/domain1/lib/databases @@ -298,8 +298,49 @@ Please remember to update translations via [Dataverse language packs](https://gi sudo service payara stop sudo service payara start ``` +7\. Update Solr schema.xml file. Start with the standard v6.5 schema.xml, then, if your installation uses any custom or experimental metadata blocks, update it to include the extra fields (step 7a). + +Stop Solr (usually `sudo service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/6.5/installation/prerequisites.html#solr-init-script)). + +```shell +sudo service solr stop +``` + +Replace schema.xml + +```shell +wget https://raw.githubusercontent.com/IQSS/dataverse/v6.5/conf/solr/schema.xml +cp schema.xml /usr/local/solr/solr-9.4.1/server/solr/collection1/conf +``` + +Start Solr (but if you use any custom metadata blocks, perform the next step, 7a first). + +```shell +sudo service solr start +``` + +7a\. For installations with custom or experimental metadata blocks: + +Before starting Solr, update the `schema.xml` file to include all the extra metadata fields that your installation uses. + +We do this by collecting the output of Dataverse's Solr schema API endpoint (`/api/admin/index/solr/schema`) and piping it to the `update-fields.sh` script which updates the `schema.xml` file supplied as an argument. + +The example below assumes the default installation location of Solr, but you can modify the commands as needed. + +```shell +wget https://raw.githubusercontent.com/IQSS/dataverse/v6.5/conf/solr/update-fields.sh +chmod +x update-fields.sh +curl "http://localhost:8080/api/admin/index/solr/schema" | sudo ./update-fields.sh /usr/local/solr/solr-9.4.1/server/solr/collection1/conf/schema.xml +``` + +Now start Solr. + + +```shell +sudo service solr start +``` -7\. Reindex Solr +8\. Reindex Solr Below is the simplest way to reindex Solr: @@ -309,7 +350,7 @@ curl http://localhost:8080/api/admin/index The API above rebuilds the existing index "in place". If you want to be absolutely sure that your index is up-to-date and consistent, you may consider wiping it clean and reindexing everything from scratch (see [the guides](https://guides.dataverse.org/en/latest/admin/solr-search-index.html)). Just note that, depending on the size of your database, a full reindex may take a while and the users will be seeing incomplete search results during that window. -8\. Run reExportAll to update dataset metadata exports +9\. Run reExportAll to update dataset metadata exports Below is the simple way to reexport all dataset metadata. For more advanced usage, please see [the guides](http://guides.dataverse.org/en/6.4/admin/metadataexport.html#batch-exports-through-the-api). @@ -317,7 +358,7 @@ Below is the simple way to reexport all dataset metadata. For more advanced usag curl http://localhost:8080/api/admin/metadata/reExportAll ``` -9\. Pushing updated metadata to DataCite +10\. Pushing updated metadata to DataCite (If you don't use DataCite, you can skip this. Also, if you aren't affected by the "useless null" bug described above, you can skip this.) From 83c22b212543e37e675c25c782b242518bb0a584 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 6 Dec 2024 13:17:04 -0500 Subject: [PATCH 20/34] yet more sudo and call out non-root user #10952 --- doc/release-notes/6.5-release-notes.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index e58a6fcce4b..b288794b9e6 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -300,6 +300,8 @@ sudo service payara start ``` 7\. Update Solr schema.xml file. Start with the standard v6.5 schema.xml, then, if your installation uses any custom or experimental metadata blocks, update it to include the extra fields (step 7a). +Run the commands below as a non-root user. + Stop Solr (usually `sudo service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/6.5/installation/prerequisites.html#solr-init-script)). ```shell @@ -310,7 +312,7 @@ Replace schema.xml ```shell wget https://raw.githubusercontent.com/IQSS/dataverse/v6.5/conf/solr/schema.xml -cp schema.xml /usr/local/solr/solr-9.4.1/server/solr/collection1/conf +sudo cp schema.xml /usr/local/solr/solr-9.4.1/server/solr/collection1/conf ``` Start Solr (but if you use any custom metadata blocks, perform the next step, 7a first). From c1df8c5108dc73dc28b03b8e1fafd7e3eefaa39e Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 6 Dec 2024 14:10:49 -0500 Subject: [PATCH 21/34] tweaks #10952 --- doc/release-notes/6.5-release-notes.md | 44 ++++++++++++-------------- 1 file changed, 20 insertions(+), 24 deletions(-) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index b288794b9e6..62ff9ba6e93 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -1,18 +1,18 @@ # Dataverse 6.5 -Please note: To read these instructions in full, please go to https://github.com/IQSS/dataverse/releases/tag/v6.5 rather than the list of releases, which will cut them off. +Please note: To read these instructions in full, please go to https://github.com/IQSS/dataverse/releases/tag/v6.5 rather than the [list of releases](https://github.com/IQSS/dataverse/releases), which will cut them off. -This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. +This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project! ## Release Highlights -Highlights in Dataverse 6.5 include: +Highlights for Dataverse 6.5 include: - new API endpoints, including editing of collections, Search API file counts, listing of exporters, comparing dataset versions, and auditing data files - UX improvements, especially Preview URLs - increased harvesting flexibility - performance gains -- a security vulnerability addressed +- a [security vulnerability](https://github.com/IQSS/dataverse-security/issues/98) addressed - many bug fixes - and more! Please see below. @@ -20,7 +20,7 @@ Highlights in Dataverse 6.5 include: ### Private URL Renamed to Preview URL and Improved -With this release the name of the URL that may be used by dataset administrators to share a draft version of a dataset has been changed from Private URL to Preview URL. +The name of the URL that may be used by dataset administrators to share a draft version of a dataset has been changed from Private URL to Preview URL. Also, additional information about the creation of Preview URLs has been added to the popup accessed via edit menu of the Dataset Page. @@ -30,17 +30,17 @@ The old "privateUrl" API endpoints for the creation and deletion of Preview (for See also #8184, #8185, #10950, and #10961. -### More Scalable Dataset Version Differencing +### Showing Differences Between Dataset Versions is More Scalable -Differencing between dataset versions, which is done during dataset edit operations and to populate the dataset page versions table has been made significantly more scalable. See #10814 and #10818. +Showing differences between dataset versions, which is done during dataset edit operations and to populate the dataset page versions table, has been made significantly more scalable. See #10814 and #10818. -### Reindexing After a Role Assignment is Less Memory Intensive +### Version Differences Details Sorting Added -Adding or removing a user from a role on a collection, particularly the root collection, could lead to a significant increase in memory use, resulting in Dataverse itself failing with an out-of-memory condition. Such changes now consume much less memory. +In order to facilitate the comparison between the draft version and the published version of a dataset, a sort on subfields has been added. See #10969. -If you have experienced out-of-memory failures in Dataverse in the past that could have been caused by this problem, you may wish to run a [reindex in place](https://guides.dataverse.org/en/latest/admin/solr-search-index.html#reindex-in-place) to update any out-of-date information. +### Reindexing After a Role Assignment is Less Memory Intensive -For more information, see #10697 and #10698. +Adding or removing a user from a role on a collection, particularly the root collection, could lead to a significant increase in memory use, resulting in Dataverse itself failing with an out-of-memory condition. Such changes now consume much less memory. A Solr reindexing step is included in the upgrade instructions below. See also #10697 and #10698. ### Longer Custom Questions in Guestbooks @@ -54,7 +54,7 @@ PostgreSQL 13 remains the version used with automated testing, leading us to con As of Flyway 10, supporting older versions of PostgreSQL no longer requires a paid subscription. While we don't encourage the use of older PostgreSQL versions, this flexibility may benefit some of our long-standing installations in their upgrade paths. -As part of this update, the containerized development environment now uses Postgres 17 instead of 16. Developers must delete their data (`rm -rf docker-dev-volumes`) and start with an empty database, as [explained](https://groups.google.com/g/dataverse-dev/c/ffoNj5UXyzU/m/nE5oGY_sAQAJ) on the dev mailing list. They can rerun the quickstart in the dev guide. +As part of this update, the containerized development environment now uses Postgres 17 instead of 16. Developers must delete their data (`rm -rf docker-dev-volumes`) and start with an empty database (rerun the [quickstart](https://guides.dataverse.org/en/6.5/developers/dev-environment.html#quickstart) in the dev guide), as [explained](https://groups.google.com/g/dataverse-dev/c/ffoNj5UXyzU/m/nE5oGY_sAQAJ) on the dev mailing list. The Docker compose file used for [evaluations or demos](https://guides.dataverse.org/en/6.4/container/running/demo.html) has been upgraded from Postgres 13 to 17. @@ -62,9 +62,9 @@ See also #10889 and #10912. ### Harvesting "oai_dc" Metadata Prefix When Extended With Specific Namespaces -Some repository extend the "oai_dc" metadata prefix with specific namespaces. In this case, harvesting of these datasets was not possible because an XML parsing error was raised. +Some data repositories extend the "oai_dc" metadata prefix with specific namespaces. In this case, harvesting of these datasets into Dataverse was not possible because an XML parsing error was raised. -Harvesting of these datasets has been fixed by excluding tags with namespaces that are not "dc:" and harvest only metadata with the "dc" namespace. See #10837. +Harvesting of these datasets has been fixed by excluding tags with namespaces that are not "dc:". That is, only harvesting metadata with the "dc" namespace. See #10837. ### Harvested Dataset PID from Record Header @@ -80,21 +80,21 @@ When harvesting using the DDI format, datasets can now have multiple "otherId" v ### Multiple Languages in Docker -Configuration and documentation has been added to explain how to set up multiple languages (e.g. English and French) in the tutorial for setting up Dataverse in Docker. +Documentation has been added to explain how to set up multiple languages (e.g. English and French) in the tutorial for setting up Dataverse in Docker. -See also [the docs](https://guides.dataverse.org/en/6.5/container/running/demo.html#multiple-languages), #10939, and #10940. +See [the tutorial](https://guides.dataverse.org/en/6.5/container/running/demo.html#multiple-languages), #10939, and #10940. ### GlobusBatchLookupSize An optimization has been added for the Globus upload workflow, with a corresponding new database setting: `:GlobusBatchLookupSize` -See the [Database Settings](https://guides.dataverse.org/en/6.5/installation/config.html#GlobusBatchLookupSize) section of the Guides, #10977, and #11040 for more information. +See the [Database Settings](https://guides.dataverse.org/en/6.5/installation/config.html#GlobusBatchLookupSize) section of the guides, #10977, and #11040 for more information. ## Bugs Fixed ### Relation Type (Related Publication) and DataCite -The subfield "Relation Type" was added to the field "Related Publication" in Dataverse 6.4 (#10632) but couldn't be used without workarounds described in an [announcement](https://groups.google.com/g/dataverse-community/c/zlRGJtu3x4g/m/GtVZ26uaBQAJ) about the problem. The bug has been fixed and workarounds and are longer required. See also #10926. +The subfield "Relation Type" was added to the field "Related Publication" in Dataverse 6.4 (#10632) but couldn't be used without workarounds described in an [announcement](https://groups.google.com/g/dataverse-community/c/zlRGJtu3x4g/m/GtVZ26uaBQAJ) about the problem. The bug has been fixed and workarounds and are longer required. See #10926 and the announcement above. ### Sort Order for Files @@ -110,15 +110,11 @@ When a dataset was deaccessioned and was the only previous version it would caus ### My Data Filter by Username Feature Restored -The superuser-only feature of filtering by a username on the My Data page was not working. The "Results for Username" field now returns data for the desired user. See also #7239 and #10980. +The superuser-only feature of filtering by a username on the My Data page was not working. This "Results for Username" field now returns data for the desired user. See also #7239 and #10980. ### Better Handling of Parallel Edit/Publish Errors -Improvements have been made in handling the errors when a dataset has been edited in one browser window and an attempt is made to edit/publish it in another. (This practice is discouraged.) See #10793 and #10794. - -### Version Differences Details Sorting Added - -In order to facilitate the comparison between the draft version and the published version of a dataset, a sort on subfields has been added. See #10969. +Improvements have been made in handling the errors when a dataset has been edited in one browser window and an attempt is made to edit or publish it in another. (This practice is discouraged, by the way.) See #10793 and #10794. ### Facets Filter Labels Now Translated Above Search Results From deb2d8821bfba71e9836f03a6f7c127e2024f28c Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 6 Dec 2024 15:15:18 -0500 Subject: [PATCH 22/34] many more tweaks #10952 --- doc/release-notes/6.5-release-notes.md | 55 +++++++++++++++----------- 1 file changed, 31 insertions(+), 24 deletions(-) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index 62ff9ba6e93..afaa3a6021e 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -110,7 +110,7 @@ When a dataset was deaccessioned and was the only previous version it would caus ### My Data Filter by Username Feature Restored -The superuser-only feature of filtering by a username on the My Data page was not working. This "Results for Username" field now returns data for the desired user. See also #7239 and #10980. +The superuser-only feature of filtering by a username on the My Data page was not working. Entering a username in the "Results for Username" field now returns data for the desired user. See also #7239 and #10980. ### Better Handling of Parallel Edit/Publish Errors @@ -118,13 +118,11 @@ Improvements have been made in handling the errors when a dataset has been edite ### Facets Filter Labels Now Translated Above Search Results -On the main page, it's possible to filter results using search facets. If internationalization (i18n) has been activated in the Dataverse installation, allowing pages to be displayed in several languages, the facets are translated in the filter column. However, they weren't being translated above the search results, remaining in the default language, English. - -This version of Dataverse fix this, and includes internationalization in the facets visible in the search results section. For more information, see #9408 and #10158. +On the main page, it's possible to filter results using search facets. If internationalization (i18n) has been enabled in the Dataverse installation, allowing pages to be displayed in several languages, the facets were correctly translated in the filter column at the left. However, they were not being translated above the search results, remaining in the default language, English. This has been fixed. See #9408 and #10158. ### Unpublished File Bug Fix Related to Deaccessioning -A bug fix was made that gets the major version of a Dataset when all major versions were deaccessioned. This fixes the incorrect showing of the files as "Unpublished" in the search list even when they are published. This fix affects the indexing, meaning these datasets must be re-indexed once Dataverse is updated. See also #10947 and #10974. +A bug fix was made related to retrieval of the major version of a Dataset when all major versions were deaccessioned. This fixes the incorrect showing of the files as "Unpublished" in the search list even when they are published. In the upgrade instructions below, there is a step to reindex Solr. See also #10947 and #10974. ### Minor DataCiteXML Fix (Useless Null) @@ -136,7 +134,7 @@ Make Data Count (MDC) citation retrieval with the PID settings has been fixed. P ### Globus "missing properties" Logging Fixed -In previous releases, logging would show Globus-related strings were missing from properties files. This has been fixed. See #11030 for details. +In previous releases, logging would show Globus-related strings were missing from properties files. This has been fixed. See #11030. ## API Updates @@ -146,7 +144,7 @@ A new endpoint (`PUT /api/dataverses/`) for updating an existing col ### fileCount Added to Search API -A new search field called `fileCount` can be searched to discover the number of files per dataset. The upgrade instructions below explain how to update your Solr `schema.xml` file to add the new field. See also #8941 and #10598. +A new search field called `fileCount` can be searched to discover the number of files per dataset. The upgrade instructions below explain how to update your Solr `schema.xml` file to add the new field and reindex Solr. See also #8941 and #10598. ### List Dataset Metadata Exporters @@ -164,7 +162,7 @@ A superuser-only API endpoint has been added to audit datasets with data files w The update collection (dataverse) API endpoint has been updated to support an "inherit from parent" configuration for metadata blocks, facets, and input levels. -Previously, not setting these fields meant using a copy of the settings from the parent collection, which could get out of sync.. See also [the docs](https://guides.dataverse.org/en/6.5/api/native-api.html#update-a-dataverse-collection), #11018, and #11026. +Previously, not setting these fields meant using a copy of the settings from the parent collection, which could get out of sync. See also [the docs](https://guides.dataverse.org/en/6.5/api/native-api.html#update-a-dataverse-collection), #11018, and #11026. ### isMetadataBlockRoot and isFacetRoot @@ -172,17 +170,17 @@ The JSON payload of the "get collection" endpoint has been extended to include p ### Whitespace Trimming When Loading Metadata Block TSV Files -When loading custom metadata blocks using the `api/admin/datasetfield/load` API, whitespace can be introduced into field names. Whitespace is now trimmed from the beginning and end of all values read into the API before persisting them. See #10688 and #10696. +When loading custom metadata blocks using the `api/admin/datasetfield/load` API endpoint, whitespace can be introduced into field names. Whitespace is now trimmed from the beginning and end of all values read into the API before persisting them. See #10688 and #10696. ### Image URLs from the Search API -As of 6.4 (thanks to #10855) `image_url` is being returned from the Search API. The logic has been updated to only show the image if each of the following are true: +As of 6.4 (#10855) `image_url` is being returned from the Search API. The logic has been updated to only show the image if each of the following are true: -1. The DataFile is not Harvested -2. A Thumbnail is available for the Datafile -3. If the Datafile is Restricted then the caller must have Download File Permission for the Datafile -4. The Datafile is NOT actively embargoed -5. The Datafile's retention period has NOT expired +1. The data file is not harvested +2. A thumbnail is available for the data file +3. If the data file is restricted, then the caller must have DownloadFile permission for the data file +4. The data file is NOT actively embargoed +5. The data file's retention period has NOT expired See also #10875 and #10886. @@ -190,26 +188,27 @@ See also #10875 and #10886. Two bugs in the Metrics API have been fixed: -- The /datasets and /datasets/byMonth endpoints could report incorrect values if/when they have been called using the dataLocation parameter (which allows getting metrics for local, remote (harvested), or all datasets) as the metrics cache was not storing different values for these cases. +- The /datasets and /datasets/byMonth endpoints could report incorrect values if or when they have been called using the "dataLocation" parameter (which allows getting metrics for local, remote (harvested), or all datasets) as the metrics cache was not storing different values for these cases. + +- Metrics endpoints whose calculation relied on finding the latest published dataset version were incorrect if/when the minor version number was > 9. -- Metrics endpoints who's calculation relied on finding the latest published datasetversion were incorrect if/when the minor version number was > 9. +The upgrade instructions below include a step for clearing the metrics cache. -When deploying the new release, the [/api/admin/clearMetricsCache](https://guides.dataverse.org/en/latest/api/native-api.html#metrics) API should be called to remove old cached values that may be incorrect. -See #10379 and #10865. +See also #10379 and #10865. ### API Tokens -An optional query parameter called "returnExpiration" has been added to the "/api/users/token/recreate" endpoint, which, if set to true, returns the expiration time in the response message. See [the docs](https://guides.dataverse.org/en/6.5/api/native-api.html#recreate-a-token), #10857 and #10858. +An optional query parameter called "returnExpiration" has been added to the `/api/users/token/recreate` endpoint, which, if set to true, returns the expiration time in the response. See [the docs](https://guides.dataverse.org/en/6.5/api/native-api.html#recreate-a-token), #10857 and #10858. -The `/api/users/token` endpoint has been extended to support any auth mechanism for retrieving the token information. Previously, this endpoint only accepted an API token to retrieve its information. Now, it accepts any authentication mechanism and returns the associated API token information. See #10914 and #10924. +The `/api/users/token` endpoint has been extended to support any auth mechanism for retrieving the token information. Previously this endpoint only accepted an API token to retrieve its information. Now it accepts any authentication mechanism and returns the associated API token information. See #10914 and #10924. ## Settings Added -- :GlobusBatchLookupSize +- `:GlobusBatchLookupSize` ## Backward Incompatible Changes -Generally speaking, see the [API Changelog](https://guides.dataverse.org/en/6.5/api/changelog.html) for a list of backward-incompatible changes. +Generally speaking, see the [API Changelog](https://guides.dataverse.org/en/latest/api/changelog.html) for a list of backward-incompatible API changes. ### List Collections Linked to a Dataset @@ -356,7 +355,15 @@ Below is the simple way to reexport all dataset metadata. For more advanced usag curl http://localhost:8080/api/admin/metadata/reExportAll ``` -10\. Pushing updated metadata to DataCite +10\. Clear metrics cache + +Run the [clearMetricsCache](https://guides.dataverse.org/en/6.5/api/native-api.html#metrics) API endpoint to remove old cached values that may be incorrect. + +```shell +curl -X DELETE http://localhost:8080/api/admin/clearMetricsCache +``` + +11\. Pushing updated metadata to DataCite (If you don't use DataCite, you can skip this. Also, if you aren't affected by the "useless null" bug described above, you can skip this.) From 7611a212e25e82b68176a422c1f2a5925bde3dad Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 6 Dec 2024 15:21:34 -0500 Subject: [PATCH 23/34] update how we use sudo #10952 --- doc/release-notes/6.5-release-notes.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index afaa3a6021e..fbe37f89baa 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -239,9 +239,9 @@ These instructions assume that you've already upgraded through all the 5.x relea 0\. These instructions assume that you are upgrading from the immediate previous version. If you are running an earlier version, the only supported way to upgrade is to progress through the upgrades to all the releases in between before attempting the upgrade to this version. -If you are running Payara as a non-root user (and you should be!), **remember not to execute the commands below as root**. Use `sudo` to change to that user first. For example, `sudo -i -u dataverse` if `dataverse` is your dedicated application user. +If you are running Payara as a non-root user (and you should be!), **remember not to execute the commands below as root**. By default, Payara runs as the `dataverse` user. In the commands below, we use sudo to run the commands as a non-root user. -In the following commands, we assume that Payara 6 is installed in `/usr/local/payara6`. If not, adjust as needed. +Also, we assume that Payara 6 is installed in `/usr/local/payara6`. If not, adjust as needed. ```shell export PAYARA=/usr/local/payara6 From dd64ebbc55cac820b2bc7258982f9f8bee981c9b Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 9 Dec 2024 09:33:03 -0500 Subject: [PATCH 24/34] ever more sudo #10952 --- doc/release-notes/6.5-release-notes.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index fbe37f89baa..25caa64b145 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -278,9 +278,9 @@ Note: if you have any trouble deploying, stop Payara, remove the following direc ```shell sudo service payara stop -rm -rf $PAYARA/glassfish/domains/domain1/generated -rm -rf $PAYARA/glassfish/domains/domain1/osgi-cache -rm -rf $PAYARA/glassfish/domains/domain1/lib/databases +sudo rm -rf $PAYARA/glassfish/domains/domain1/generated +sudo rm -rf $PAYARA/glassfish/domains/domain1/osgi-cache +sudo rm -rf $PAYARA/glassfish/domains/domain1/lib/databases ``` 5\. For installations with internationalization: From 98d68c95fa0fd303f2343bb45932682d587fd384 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 9 Dec 2024 10:19:13 -0500 Subject: [PATCH 25/34] explain how to download the war file before deploying #10952 --- doc/release-notes/6.5-release-notes.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index 25caa64b145..ad5d3575dd5 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -268,9 +268,10 @@ sudo service payara stop sudo service payara start ``` -4\. Deploy this version +4\. Download and deploy this version ```shell +wget https://github.com/IQSS/dataverse/releases/download/v6.5/dataverse-6.5.war $PAYARA/bin/asadmin deploy dataverse-6.5.war ``` From 48246a635753914e9bf1c3fcdb1560f8f15b7b26 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 9 Dec 2024 10:20:52 -0500 Subject: [PATCH 26/34] note that Solr dir can differ #10952 --- doc/release-notes/6.5-release-notes.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index ad5d3575dd5..019d437ed06 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -306,6 +306,8 @@ sudo service solr stop Replace schema.xml +Please note that the path to Solr may differ than the example below. + ```shell wget https://raw.githubusercontent.com/IQSS/dataverse/v6.5/conf/solr/schema.xml sudo cp schema.xml /usr/local/solr/solr-9.4.1/server/solr/collection1/conf From ab7c90e607b072af6e962f8fc0ec652a76d44b53 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 10 Dec 2024 10:59:01 -0500 Subject: [PATCH 27/34] typo Co-authored-by: Omer Fahim --- doc/release-notes/6.5-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index 019d437ed06..56521c45d71 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -94,7 +94,7 @@ See the [Database Settings](https://guides.dataverse.org/en/6.5/installation/con ### Relation Type (Related Publication) and DataCite -The subfield "Relation Type" was added to the field "Related Publication" in Dataverse 6.4 (#10632) but couldn't be used without workarounds described in an [announcement](https://groups.google.com/g/dataverse-community/c/zlRGJtu3x4g/m/GtVZ26uaBQAJ) about the problem. The bug has been fixed and workarounds and are longer required. See #10926 and the announcement above. +The subfield "Relation Type" was added to the field "Related Publication" in Dataverse 6.4 (#10632) but couldn't be used without workarounds described in an [announcement](https://groups.google.com/g/dataverse-community/c/zlRGJtu3x4g/m/GtVZ26uaBQAJ) about the problem. The bug has been fixed and workarounds are no longer required. See #10926 and the announcement above. ### Sort Order for Files From 63ff790d4b3c319bb674fba8201eb8e3c8686fc9 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 10 Dec 2024 11:01:51 -0500 Subject: [PATCH 28/34] typo Co-authored-by: Omer Fahim --- doc/release-notes/6.5-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index 56521c45d71..7bb9cbb1e73 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -106,7 +106,7 @@ In the Guestbook UI form, the email address is now checked for validity. See #10 ### Updating Files Now Possible When Latest and Only Dataset Version is Deaccessioned -When a dataset was deaccessioned and was the only previous version it would cause an error when trying to update the files. This has been fixed. See #9351 and #10901. +When a dataset was deaccessioned, and was the only previous version, it would cause an error when trying to update the files. This has been fixed. See #9351 and #10901. ### My Data Filter by Username Feature Restored From 452cca4f15805063b769003620762a44cdc8260b Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 10 Dec 2024 14:49:00 -0500 Subject: [PATCH 29/34] word choice Co-authored-by: Omer Fahim --- doc/release-notes/6.5-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index 7bb9cbb1e73..a2cac8ed800 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -306,7 +306,7 @@ sudo service solr stop Replace schema.xml -Please note that the path to Solr may differ than the example below. +Please note that the path to Solr may differ from the example below. ```shell wget https://raw.githubusercontent.com/IQSS/dataverse/v6.5/conf/solr/schema.xml From ce6a99a49f54caaf54bce633dd08869365c87e8c Mon Sep 17 00:00:00 2001 From: Stephen Kraffmiller Date: Wed, 11 Dec 2024 13:38:31 -0500 Subject: [PATCH 30/34] #10952 add not about breadcrumbs in Anon Preview URL --- doc/release-notes/6.5-release-notes.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index a2cac8ed800..c866d52e42c 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -24,6 +24,8 @@ The name of the URL that may be used by dataset administrators to share a draft Also, additional information about the creation of Preview URLs has been added to the popup accessed via edit menu of the Dataset Page. +Users of the Anonymous Preview URL will no longer be able to see the name of the Dataverse that this dataset is in but will be able to see the name of the repository. + Any Private URLs created in previous versions of Dataverse will continue to work. The old "privateUrl" API endpoints for the creation and deletion of Preview (formerly Private) URLs have been deprecated. They will continue to work but please switch to the "previewUrl" equivalents that have been [documented](https://guides.dataverse.org/en/6.5/api/native-api.html#create-a-preview-url-for-a-dataset) in the API Guide. From 929128bb344df97678c49e3efa31d22c5074e646 Mon Sep 17 00:00:00 2001 From: Stephen Kraffmiller Date: Wed, 11 Dec 2024 13:40:19 -0500 Subject: [PATCH 31/34] #10952 reword --- doc/release-notes/6.5-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index c866d52e42c..17abec2907a 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -24,7 +24,7 @@ The name of the URL that may be used by dataset administrators to share a draft Also, additional information about the creation of Preview URLs has been added to the popup accessed via edit menu of the Dataset Page. -Users of the Anonymous Preview URL will no longer be able to see the name of the Dataverse that this dataset is in but will be able to see the name of the repository. +Users of the Anonymous Preview URL will no longer be able to see the name of the Dataverse that the dataset is in but will be able to see the name of the repository. Any Private URLs created in previous versions of Dataverse will continue to work. From a0508d1989c94e87e7fd7ddfaf7853cc69936224 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 11 Dec 2024 13:59:56 -0500 Subject: [PATCH 32/34] add link to #11085 --- doc/release-notes/6.5-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index 17abec2907a..a45efb59556 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -30,7 +30,7 @@ Any Private URLs created in previous versions of Dataverse will continue to work The old "privateUrl" API endpoints for the creation and deletion of Preview (formerly Private) URLs have been deprecated. They will continue to work but please switch to the "previewUrl" equivalents that have been [documented](https://guides.dataverse.org/en/6.5/api/native-api.html#create-a-preview-url-for-a-dataset) in the API Guide. -See also #8184, #8185, #10950, and #10961. +See also #8184, #8185, #10950, #10961, and #11085. ### Showing Differences Between Dataset Versions is More Scalable From fe13a437c9c990086945c48a21dd32ce7e80ee6f Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 11 Dec 2024 15:13:43 -0500 Subject: [PATCH 33/34] add cvoc overview fix #10952 --- doc/release-notes/6.5-release-notes.md | 4 ++++ doc/release-notes/display_overview_fix.md | 1 - 2 files changed, 4 insertions(+), 1 deletion(-) delete mode 100644 doc/release-notes/display_overview_fix.md diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index a45efb59556..8cccc7114cf 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -134,6 +134,10 @@ A minor bug fix was made to avoid sending a useless ", null" in the DataCiteXML Make Data Count (MDC) citation retrieval with the PID settings has been fixed. PID parsing in Dataverse is now case insensitive, improving interaction with services that may change the case of PIDs. Warnings related to managed/excluded PID lists for PID providers have been reduced. See #10708. +### Quirk in Overview Display When Using External Controlled Variables + +This bugfix corrects an issue when there are duplicated entries on the metadata page. It is fixed by correcting an IF-clause in metadataFragment.xhtml. See #11005 and #11034. + ### Globus "missing properties" Logging Fixed In previous releases, logging would show Globus-related strings were missing from properties files. This has been fixed. See #11030. diff --git a/doc/release-notes/display_overview_fix.md b/doc/release-notes/display_overview_fix.md deleted file mode 100644 index 73a01435caf..00000000000 --- a/doc/release-notes/display_overview_fix.md +++ /dev/null @@ -1 +0,0 @@ -This bugfix corrects an issue when there are duplicated entries on the metadata page. It is fixed by correcting an IF-clause in metadataFragment.xhtml. \ No newline at end of file From 2de57e13135a8512218a651552e924cce512382a Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 11 Dec 2024 15:16:26 -0500 Subject: [PATCH 34/34] remove "in place" from Solr reindex step #10952 --- doc/release-notes/6.5-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.5-release-notes.md b/doc/release-notes/6.5-release-notes.md index 8cccc7114cf..2e27f4419bd 100644 --- a/doc/release-notes/6.5-release-notes.md +++ b/doc/release-notes/6.5-release-notes.md @@ -354,7 +354,7 @@ Below is the simplest way to reindex Solr: curl http://localhost:8080/api/admin/index ``` -The API above rebuilds the existing index "in place". If you want to be absolutely sure that your index is up-to-date and consistent, you may consider wiping it clean and reindexing everything from scratch (see [the guides](https://guides.dataverse.org/en/latest/admin/solr-search-index.html)). Just note that, depending on the size of your database, a full reindex may take a while and the users will be seeing incomplete search results during that window. +The API above rebuilds the existing index. If you want to be absolutely sure that your index is up-to-date and consistent, you may consider wiping it clean and reindexing everything from scratch (see [the guides](https://guides.dataverse.org/en/latest/admin/solr-search-index.html)). Just note that, depending on the size of your database, a full reindex may take a while and the users will be seeing incomplete search results during that window. 9\. Run reExportAll to update dataset metadata exports