Merge branch 'IQSS:develop' into 9836_always_add_validity_field_to_so…

…lr_doc

.gitattributes

@@ -0,0 +1,4 @@
+# https://www.git-scm.com/docs/gitattributes
+
+# This set mandatory LF line endings for .sh files preventing from windows users to having to change the value of their git config --global core.autocrlf to 'false' or 'input'
+*.sh text eol=lf

.github/ISSUE_TEMPLATE/bug_report.md

@@ -3,7 +3,7 @@ name: Bug report
 about: Did you encounter something unexpected or incorrect in the Dataverse software?
   We'd like to hear about it!
 title: ''
-labels: ''
+labels: 'Type: Bug'
 assignees: ''
 
 ---

.github/ISSUE_TEMPLATE/feature_request.md

@@ -2,7 +2,7 @@
 name: Feature request
 about: Suggest an idea or new feature for the Dataverse software!
 title: 'Feature Request/Idea:'
-labels: ''
+labels: 'Type: Feature'
 assignees: ''
 
 ---

README.md

@@ -1,7 +1,7 @@
 Dataverse®  
 ===============
 
-Dataverse is an [open source][] software platform for sharing, finding, citing, and preserving research data (developed by the [Data Science and Products team](http://www.iq.harvard.edu/people/people/data-science-products) at the [Institute for Quantitative Social Science](http://iq.harvard.edu/) and the [Dataverse community][]).
+Dataverse is an [open source][] software platform for sharing, finding, citing, and preserving research data (developed by the [Dataverse team](https://dataverse.org/about) at the [Institute for Quantitative Social Science](https://iq.harvard.edu/) and the [Dataverse community][]).
 
 [dataverse.org][] is our home on the web and shows a map of Dataverse installations around the world, a list of [features][], [integrations][] that have been made possible through [REST APIs][], our development [roadmap][], and more.
 
@@ -15,7 +15,7 @@ We love contributors! Please see our [Contributing Guide][] for ways you can hel
 
 Dataverse is a trademark of President and Fellows of Harvard College and is registered in the United States.
 
-[![Dataverse Project logo](src/main/webapp/resources/images/dataverseproject_logo.jpg?raw=true "Dataverse Project")](http://dataverse.org)
+[![Dataverse Project logo](src/main/webapp/resources/images/dataverseproject_logo.jpg "Dataverse Project")](http://dataverse.org)
 
 [![API Test Status](https://jenkins.dataverse.org/buildStatus/icon?job=IQSS-dataverse-develop&subject=API%20Test%20Status)](https://jenkins.dataverse.org/job/IQSS-dataverse-develop/)
 [![API Test Coverage](https://img.shields.io/jenkins/coverage/jacoco?jobUrl=https%3A%2F%2Fjenkins.dataverse.org%2Fjob%2FIQSS-dataverse-develop&label=API%20Test%20Coverage)](https://jenkins.dataverse.org/job/IQSS-dataverse-develop/ws/target/coverage-it/index.html)
@@ -26,15 +26,15 @@ Dataverse is a trademark of President and Fellows of Harvard College and is regi
 [dataverse.org]: https://dataverse.org
 [demo.dataverse.org]: https://demo.dataverse.org
 [Dataverse community]: https://dataverse.org/developers
-[Installation Guide]: http://guides.dataverse.org/en/latest/installation/index.html
+[Installation Guide]: https://guides.dataverse.org/en/latest/installation/index.html
 [latest release]: https://github.com/IQSS/dataverse/releases
 [features]: https://dataverse.org/software-features
 [roadmap]: https://www.iq.harvard.edu/roadmap-dataverse-project
 [integrations]: https://dataverse.org/integrations
-[REST APIs]: http://guides.dataverse.org/en/latest/api/index.html
+[REST APIs]: https://guides.dataverse.org/en/latest/api/index.html
 [Contributing Guide]: CONTRIBUTING.md
 [mailing list]: https://groups.google.com/group/dataverse-community
 [community call]: https://dataverse.org/community-calls
-[chat.dataverse.org]: http://chat.dataverse.org
+[chat.dataverse.org]: https://chat.dataverse.org
 [Dataverse Community Meeting]: https://dataverse.org/events
 [open source]: LICENSE.md

doc/release-notes/10001-datasets-files-api-user-permissions.md

@@ -0,0 +1,13 @@
+- New query parameter `includeDeaccessioned` added to the getVersion endpoint (/api/datasets/{id}/versions/{versionId}) to consider deaccessioned versions when searching for versions.
+
+
+- New endpoint to get user permissions on a dataset (/api/datasets/{id}/userPermissions). In particular, the user permissions that this API call checks, returned as booleans, are the following:
+
+  - Can view the unpublished dataset
+  - Can edit the dataset
+  - Can publish the dataset
+  - Can manage the dataset permissions
+  - Can delete the dataset draft
+
+
+- New permission check "canManageFilePermissions" added to the existing endpoint for getting user permissions on a file (/api/access/datafile/{id}/userPermissions).

doc/release-notes/6.0-release-notes.md

@@ -166,6 +166,22 @@ If you are running Payara as a non-root user (and you should be!), **remember no
 
    Your `:MDCLogPath` database setting might be pointing to a Payara 5 directory such as `/usr/local/payara5/glassfish/domains/domain1/logs`. If so, edit this to be Payara 6. You'll probably want to copy your logs over as well.
 
+1. If you've enabled access logging or any other site-specific configuration, be sure to preserve them. For instance, the default domain.xml includes
+
+   ```
+        <http-service>
+        <access-log></access-log>
+   ```
+
+   but you may wish to include
+
+   ```
+        <http-service access-logging-enabled="true">
+        <access-log format="%client.name% %datetime% %request% %status% %response.length% %header.user-agent% %header.referer% %cookie.JSESSIONID% %header.x-forwarded-for%"></access-log>
+   ```
+
+   Be sure to keep a previous copy of your domain.xml for reference.
+
 1. Update systemd unit file (or other init system) from `/usr/local/payara5` to `/usr/local/payara6`, if applicable.
 
 1. Start Payara.

doc/release-notes/9590-intellij-redeploy.md

@@ -0,0 +1,3 @@
+Developers can enjoy a dramatically faster feedback loop when iterating on code if they are using IntelliJ IDEA Ultimate (free educational licenses are available) and the Payara Platform Tools plugin.
+
+For details, see http://preview.guides.gdcc.io/en/develop/container/dev-usage.html#intellij-idea-ultimate-and-payara-platform-tools

doc/release-notes/9763-versions-api-improvements.md

@@ -0,0 +1,8 @@
+# Improvements in the /versions API
+
+- optional pagination has been added to `/api/datasets/{id}/versions` that may be useful in datasets with a large number of versions;
+- a new flag `includeFiles` is added to both `/api/datasets/{id}/versions` and `/api/datasets/{id}/versions/{vid}` (true by default), providing an option to drop the file information from the output;
+- when files are requested to be included, some database lookup optimizations have been added to improve the performance on datasets with large numbers of files. 
+
+This is reflected in the [Dataset Versions API](https://guides.dataverse.org/en/9763-lookup-optimizations/api/native-api.html#dataset-versions-api) section of the Guide.
+

doc/release-notes/9995-files-api-downloadsize-with-criteria-and-deaccessioned-support.md

@@ -0,0 +1,12 @@
+Extended the getDownloadSize endpoint (/api/datasets/{id}/versions/{versionId}/downloadsize), including the following new features:
+
+- The endpoint now accepts a new boolean optional query parameter "includeDeaccessioned", which, if enabled, causes the endpoint to consider deaccessioned dataset versions when searching for versions to obtain the file total download size.
+
+
+- The endpoint now supports filtering by criteria. In particular, it accepts the following optional criteria query parameters:
+
+  - contentType
+  - accessStatus
+  - categoryName
+  - tabularTagName
+  - searchText

doc/sphinx-guides/source/_static/installation/files/etc/init.d/solr

@@ -7,7 +7,7 @@
 
 SOLR_DIR="/usr/local/solr/solr-9.3.0"
 SOLR_COMMAND="bin/solr"
-SOLR_ARGS="-m 1g -j jetty.host=127.0.0.1"
+SOLR_ARGS="-m 1g"
 SOLR_USER=solr
 
 case $1 in

doc/sphinx-guides/source/_static/installation/files/etc/systemd/solr.service

@@ -6,7 +6,7 @@ After = syslog.target network.target remote-fs.target nss-lookup.target
 User = solr
 Type = forking
 WorkingDirectory = /usr/local/solr/solr-9.3.0
-ExecStart = /usr/local/solr/solr-9.3.0/bin/solr start -m 1g -j "jetty.host=127.0.0.1"
+ExecStart = /usr/local/solr/solr-9.3.0/bin/solr start -m 1g
 ExecStop = /usr/local/solr/solr-9.3.0/bin/solr stop
 LimitNOFILE=65000
 LimitNPROC=65000

doc/sphinx-guides/source/_static/navbarscroll.js

@@ -1,6 +1,6 @@
 /*  
     Use to fix hidden section headers behind the navbar when using links with targets
-    See: http://stackoverflow.com/questions/10732690/offsetting-an-html-anchor-to-adjust-for-fixed-header    
+    See: https://stackoverflow.com/questions/10732690/offsetting-an-html-anchor-to-adjust-for-fixed-header    
 */
 $jqTheme(document).ready(function() {
   $jqTheme('a[href*="#"]:not([href="#"])').on('click', function() {

doc/sphinx-guides/source/_templates/navbar.html

@@ -15,7 +15,7 @@
                 <span class="icon-bar"></span>
                 <span class="icon-bar"></span>
             </button>
-            <a class="navbar-brand" href="http://dataverse.org" id="dataverse-org-homepage-url">Dataverse Project</a>
+            <a class="navbar-brand" href="https://dataverse.org" id="dataverse-org-homepage-url">Dataverse Project</a>
         </div>
         <!-- Collect the nav links, forms, and other content for toggling -->
 
@@ -24,15 +24,15 @@
                 <li class="dropdown">
                     <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">About <span class="caret"></span></a>
                     <ul class="dropdown-menu">
-                        <li><a target="_blank" href="http://dataverse.org/about">About the Project</a></li>
-                        <li><a target="_blank" href="http://dataverse.org/add-data">Add Data</a></li>
-                        <li><a target="_blank" href="http://dataverse.org/blog">Blog</a></li>
-                        <li><a target="_blank" href="http://dataverse.org/presentations">Presentations</a></li>
-                        <li><a target="_blank" href="http://dataverse.org/publications">Publications</a></li>
+                        <li><a target="_blank" href="https://dataverse.org/about">About the Project</a></li>
+                        <li><a target="_blank" href="https://dataverse.org/add-data">Add Data</a></li>
+                        <li><a target="_blank" href="https://dataverse.org/blog">Blog</a></li>
+                        <li><a target="_blank" href="https://dataverse.org/presentations">Presentations</a></li>
+                        <li><a target="_blank" href="https://dataverse.org/publications">Publications</a></li>
                     </ul>
                 </li>
                 <li>
-                    <a target="_blank" href="http://dataverse.org/">
+                    <a target="_blank" href="https://dataverse.org/">
                         Community
                     </a>
                 </li>
@@ -49,18 +49,18 @@
                 <li>
                     <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Software <span class="caret"></span></a>
                     <ul class="dropdown-menu">
-                        <li><a target="_blank" href="http://dataverse.org/software-features">Features</a></li>
-                        <li><a target="_blank" href="http://github.com/IQSS/dataverse">Source Code</a></li>
-                        <li><a target="_blank" href="http://guides.dataverse.org/en/latest/user/index.html">User Guide</a></li>
-                        <li><a target="_blank" href="http://guides.dataverse.org/en/latest/installation/index.html">Installation Guide</a></li>
-                        <li><a target="_blank" href="http://guides.dataverse.org/en/latest/api/index.html">API Guide</a></li>
-                        <li><a target="_blank" href="http://guides.dataverse.org/en/latest/developers/index.html">Developer Guide</a></li>
-                        <li><a target="_blank" href="http://guides.dataverse.org/en/latest/style/index.html">Style Guide</a></li>
-                        <li><a target="_blank" href="http://guides.dataverse.org/en/latest/admin/index.html">Admin Guide</a></li>
+                        <li><a target="_blank" href="https://dataverse.org/software-features">Features</a></li>
+                        <li><a target="_blank" href="https://github.com/IQSS/dataverse">Source Code</a></li>
+                        <li><a target="_blank" href="https://guides.dataverse.org/en/latest/user/index.html">User Guide</a></li>
+                        <li><a target="_blank" href="https://guides.dataverse.org/en/latest/installation/index.html">Installation Guide</a></li>
+                        <li><a target="_blank" href="https://guides.dataverse.org/en/latest/api/index.html">API Guide</a></li>
+                        <li><a target="_blank" href="https://guides.dataverse.org/en/latest/developers/index.html">Developer Guide</a></li>
+                        <li><a target="_blank" href="https://guides.dataverse.org/en/latest/style/index.html">Style Guide</a></li>
+                        <li><a target="_blank" href="https://guides.dataverse.org/en/latest/admin/index.html">Admin Guide</a></li>
                     </ul>
                 </li>
                 <li>
-                    <a target="_blank" href="http://dataverse.org/">
+                    <a target="_blank" href="https://dataverse.org/">
                         Contact
                     </a>
                 </li>

doc/sphinx-guides/source/admin/monitoring.rst

@@ -14,7 +14,7 @@ In production you'll want to monitor the usual suspects such as CPU, memory, fre
 Munin
 +++++
 
-http://munin-monitoring.org says, "A default installation provides a lot of graphs with almost no work." From RHEL or CentOS 7, you can try the following steps.
+https://munin-monitoring.org says, "A default installation provides a lot of graphs with almost no work." From RHEL or CentOS 7, you can try the following steps.
 
 Enable the EPEL yum repo (if you haven't already):
 

doc/sphinx-guides/source/api/client-libraries.rst

@@ -24,7 +24,7 @@ Java
 
 https://github.com/IQSS/dataverse-client-java is the official Java library for Dataverse APIs.
 
-`Richard Adams <http://www.researchspace.com/electronic-lab-notebook/about_us_team.html>`_ from `ResearchSpace <http://www.researchspace.com>`_ created and maintains this library.
+`Richard Adams <https://www.researchspace.com/electronic-lab-notebook/about_us_team.html>`_ from `ResearchSpace <https://www.researchspace.com>`_ created and maintains this library.
 
 Javascript
 ----------
@@ -54,7 +54,7 @@ There are multiple Python modules for interacting with Dataverse APIs.
 
 `python-dvuploader <https://github.com/gdcc/python-dvuploader>`_ implements Jim Myers' excellent `dv-uploader <https://github.com/GlobalDataverseCommunityConsortium/dataverse-uploader>`_ as a Python module. It offers parallel direct uploads to Dataverse backend storage, streams files directly instead of buffering them in memory, and supports multi-part uploads, chunking data accordingly.
 
-`pyDataverse <https://github.com/gdcc/pyDataverse>`_ primarily allows developers to manage Dataverse collections, datasets and datafiles. Its intention is to help with data migrations and DevOps activities such as testing and configuration management. The module is developed by `Stefan Kasberger <http://stefankasberger.at>`_ from `AUSSDA - The Austrian Social Science Data Archive <https://aussda.at>`_.  
+`pyDataverse <https://github.com/gdcc/pyDataverse>`_ primarily allows developers to manage Dataverse collections, datasets and datafiles. Its intention is to help with data migrations and DevOps activities such as testing and configuration management. The module is developed by `Stefan Kasberger <https://stefankasberger.at>`_ from `AUSSDA - The Austrian Social Science Data Archive <https://aussda.at>`_.  
 
 `UBC's Dataverse Utilities <https://ubc-library-rc.github.io/dataverse_utils/>`_ are a set of Python console utilities which allow one to upload datasets from a tab-separated-value spreadsheet, bulk release multiple datasets, bulk delete unpublished datasets, quickly duplicate records. replace licenses, and more. For additional information see their `PyPi page <https://pypi.org/project/dataverse-utils/>`_.
 
@@ -68,8 +68,7 @@ R
 https://github.com/IQSS/dataverse-client-r is the official R package for Dataverse APIs. The latest release can be installed from `CRAN <https://cran.r-project.org/package=dataverse>`_. 
 The R client can search and download datasets. It is useful when automatically (instead of manually) downloading data files as part of a script. For bulk edit and upload operations, we currently recommend pyDataverse.
 
-The package is currently maintained by  `Shiro Kuriwaki <https://github.com/kuriwaki>`_. It was originally created by `Thomas Leeper <http://thomasleeper.com>`_ and then formerly maintained by `Will Beasley <https://github.com/wibeasley>`_.
-
+The package is currently maintained by  `Shiro Kuriwaki <https://github.com/kuriwaki>`_. It was originally created by `Thomas Leeper <https://thomasleeper.com>`_ and then formerly maintained by `Will Beasley <https://github.com/wibeasley>`_.
 
 Ruby
 ----

doc/sphinx-guides/source/api/dataaccess.rst

@@ -426,6 +426,7 @@ This method returns the permissions that the calling user has on a particular fi
 In particular, the user permissions that this method checks, returned as booleans, are the following:
 
 * Can download the file
+* Can manage the file permissions
 * Can edit the file owner dataset
 
 A curl example using an ``id``::

doc/sphinx-guides/source/api/external-tools.rst

@@ -11,7 +11,7 @@ Introduction
 
 External tools are additional applications the user can access or open from your Dataverse installation to preview, explore, and manipulate data files and datasets. The term "external" is used to indicate that the tool is not part of the main Dataverse Software.
 
-Once you have created the external tool itself (which is most of the work!), you need to teach a Dataverse installation how to construct URLs that your tool needs to operate. For example, if you've deployed your tool to fabulousfiletool.com your tool might want the ID of a file and the siteUrl of the Dataverse installation like this: https://fabulousfiletool.com?fileId=42&siteUrl=http://demo.dataverse.org
+Once you have created the external tool itself (which is most of the work!), you need to teach a Dataverse installation how to construct URLs that your tool needs to operate. For example, if you've deployed your tool to fabulousfiletool.com your tool might want the ID of a file and the siteUrl of the Dataverse installation like this: https://fabulousfiletool.com?fileId=42&siteUrl=https://demo.dataverse.org
 
 In short, you will be creating a manifest in JSON format that describes not only how to construct URLs for your tool, but also what types of files your tool operates on, where it should appear in the Dataverse installation web interfaces, etc. 
 
@@ -102,7 +102,7 @@ Terminology
 
     httpMethod                   Either ``GET`` or ``POST``.
 
-    queryParameters              **Key/value combinations** that can be appended to the toolUrl. For example, once substitution takes place (described below) the user may be redirected to ``https://fabulousfiletool.com?fileId=42&siteUrl=http://demo.dataverse.org``.
+    queryParameters              **Key/value combinations** that can be appended to the toolUrl. For example, once substitution takes place (described below) the user may be redirected to ``https://fabulousfiletool.com?fileId=42&siteUrl=https://demo.dataverse.org``.
 
     query parameter keys         An **arbitrary string** to associate with a value that is populated with a reserved word (described below). As the author of the tool, you have control over what "key" you would like to be passed to your tool. For example, if you want to have your tool receive and operate on the query parameter "dataverseFileId=42" instead of just "fileId=42", that's fine.
 

doc/sphinx-guides/source/api/getting-started.rst

@@ -9,7 +9,7 @@ If you are a researcher or curator who wants to automate parts of your workflow,
 Servers You Can Test With
 -------------------------
 
-Rather than using a production Dataverse installation, API users are welcome to use http://demo.dataverse.org for testing. You can email [email protected] if you have any trouble with this server.  
+Rather than using a production Dataverse installation, API users are welcome to use https://demo.dataverse.org for testing. You can email [email protected] if you have any trouble with this server.  
 
 If you would rather have full control over your own test server, deployments to AWS, Docker, and more are covered in the :doc:`/developers/index` and the :doc:`/installation/index`.