From b607feb0c249c4c6b7cea85c12eab48719b109c8 Mon Sep 17 00:00:00 2001 From: Henning Timm Date: Tue, 18 Oct 2022 11:28:26 +0200 Subject: [PATCH 1/8] Removed link to resolved issue --- doc/sphinx-guides/source/admin/solr-search-index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/sphinx-guides/source/admin/solr-search-index.rst b/doc/sphinx-guides/source/admin/solr-search-index.rst index 5685672eceb..41b9c7b6a8f 100644 --- a/doc/sphinx-guides/source/admin/solr-search-index.rst +++ b/doc/sphinx-guides/source/admin/solr-search-index.rst @@ -36,7 +36,7 @@ Please note that the moment you issue this command, it will appear to end users Start Async Reindex ~~~~~~~~~~~~~~~~~~~ -Please note that this operation may take hours depending on the amount of data in your system. This known issue is being tracked at https://github.com/IQSS/dataverse/issues/50 +Please note that this operation may take hours depending on the amount of data in your system. ``curl http://localhost:8080/api/admin/index`` From 0a11e454a89912704389b694e45ba5a3fcf10171 Mon Sep 17 00:00:00 2001 From: Henning Timm Date: Tue, 18 Oct 2022 11:28:48 +0200 Subject: [PATCH 2/8] Made Solr casing consistent --- doc/sphinx-guides/source/admin/solr-search-index.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/sphinx-guides/source/admin/solr-search-index.rst b/doc/sphinx-guides/source/admin/solr-search-index.rst index 41b9c7b6a8f..ef661c14ef9 100644 --- a/doc/sphinx-guides/source/admin/solr-search-index.rst +++ b/doc/sphinx-guides/source/admin/solr-search-index.rst @@ -60,7 +60,7 @@ If indexing stops, this command should pick up where it left off based on which Manual Reindexing ----------------- -If you have made manual changes to a dataset in the database or wish to reindex a dataset that solr didn't want to index properly, it is possible to manually reindex Dataverse collections and datasets. +If you have made manual changes to a dataset in the database or wish to reindex a dataset that Solr didn't want to index properly, it is possible to manually reindex Dataverse collections and datasets. Reindexing Dataverse Collections ++++++++++++++++++++++++++++++++ @@ -89,7 +89,7 @@ To re-index a dataset by its database ID: Manually Querying Solr ---------------------- -If you suspect something isn't indexed properly in solr, you may bypass the Dataverse installation's web interface and query the command line directly to verify what solr returns: +If you suspect something isn't indexed properly in Solr, you may bypass the Dataverse installation's web interface and query the command line directly to verify what Solr returns: ``curl "http://localhost:8983/solr/collection1/select?q=dsPersistentId:doi:10.15139/S3/HFV0AO"`` From 6da6fd06e668e2edef4bca4d7369b4528cf1d92e Mon Sep 17 00:00:00 2001 From: Henning Timm Date: Tue, 18 Oct 2022 11:29:16 +0200 Subject: [PATCH 3/8] Minor punctuation fix --- doc/sphinx-guides/source/admin/solr-search-index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/sphinx-guides/source/admin/solr-search-index.rst b/doc/sphinx-guides/source/admin/solr-search-index.rst index ef661c14ef9..faf1a578387 100644 --- a/doc/sphinx-guides/source/admin/solr-search-index.rst +++ b/doc/sphinx-guides/source/admin/solr-search-index.rst @@ -22,7 +22,7 @@ Get a list of all database objects that are missing in Solr, and Solr documents ``curl http://localhost:8080/api/admin/index/status`` -Remove all Solr documents that are orphaned (ie not associated with objects in the database): +Remove all Solr documents that are orphaned (i.e. not associated with objects in the database): ``curl http://localhost:8080/api/admin/index/clear-orphans`` From e3f3fb4e7209d9917c7484a570bb6eac8916a777 Mon Sep 17 00:00:00 2001 From: Henning Timm Date: Tue, 18 Oct 2022 11:29:38 +0200 Subject: [PATCH 4/8] Highlighted search string --- doc/sphinx-guides/source/admin/solr-search-index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/sphinx-guides/source/admin/solr-search-index.rst b/doc/sphinx-guides/source/admin/solr-search-index.rst index faf1a578387..bcf55480625 100644 --- a/doc/sphinx-guides/source/admin/solr-search-index.rst +++ b/doc/sphinx-guides/source/admin/solr-search-index.rst @@ -69,7 +69,7 @@ Dataverse collections must be referenced by database object ID. If you have dire ``select id from dataverse where alias='dataversealias';`` -should work, or you may click the Dataverse Software's "Edit" menu and look for dataverseId= in the URLs produced by the drop-down. Then, to re-index: +should work, or you may click the Dataverse Software's "Edit" menu and look for *dataverseId=* in the URLs produced by the drop-down. Then, to re-index: ``curl http://localhost:8080/api/admin/index/dataverses/135`` From 9cac9fd19ac347a1f7bf97d86369521e63368762 Mon Sep 17 00:00:00 2001 From: Henning Timm Date: Tue, 18 Oct 2022 11:41:25 +0200 Subject: [PATCH 5/8] Made casing of Solr consistent across all documents --- doc/sphinx-guides/source/admin/harvestserver.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/sphinx-guides/source/admin/harvestserver.rst b/doc/sphinx-guides/source/admin/harvestserver.rst index 88004d9dc5f..6f4f23fc587 100644 --- a/doc/sphinx-guides/source/admin/harvestserver.rst +++ b/doc/sphinx-guides/source/admin/harvestserver.rst @@ -115,10 +115,10 @@ Some useful examples of search queries to define OAI sets: ``keywordValue:censorship`` -Important: New SOLR schema required! +Important: New Solr schema required! ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -In order to be able to define OAI sets, your SOLR server must be upgraded with the search schema that came with release 4.5 (or later), and all your local datasets must be re-indexed, once the new schema is installed. +In order to be able to define OAI sets, your Solr server must be upgraded with the search schema that came with release 4.5 (or later), and all your local datasets must be re-indexed, once the new schema is installed. OAI Set updates --------------- From 25421a6df52381aec04658070110264b7d2d0a7d Mon Sep 17 00:00:00 2001 From: Henning Timm Date: Tue, 18 Oct 2022 21:17:44 +0200 Subject: [PATCH 6/8] Extend information on indexing times --- doc/sphinx-guides/source/admin/solr-search-index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/sphinx-guides/source/admin/solr-search-index.rst b/doc/sphinx-guides/source/admin/solr-search-index.rst index bcf55480625..0d3db3eeac6 100644 --- a/doc/sphinx-guides/source/admin/solr-search-index.rst +++ b/doc/sphinx-guides/source/admin/solr-search-index.rst @@ -36,7 +36,7 @@ Please note that the moment you issue this command, it will appear to end users Start Async Reindex ~~~~~~~~~~~~~~~~~~~ -Please note that this operation may take hours depending on the amount of data in your system. +Please note that this operation may take hours depending on the amount of data in your system and whether or not you installation is using full-text indexing. More information on this, as well as some reference times, can be found at https://github.com/IQSS/dataverse/issues/50. ``curl http://localhost:8080/api/admin/index`` From 3ba789d4caeae8b6c0f34c5780fc1e05abea3525 Mon Sep 17 00:00:00 2001 From: Henning Timm Date: Tue, 18 Oct 2022 21:27:40 +0200 Subject: [PATCH 7/8] Fixed minor typo --- doc/sphinx-guides/source/admin/solr-search-index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/sphinx-guides/source/admin/solr-search-index.rst b/doc/sphinx-guides/source/admin/solr-search-index.rst index 0d3db3eeac6..769c1ee5a0d 100644 --- a/doc/sphinx-guides/source/admin/solr-search-index.rst +++ b/doc/sphinx-guides/source/admin/solr-search-index.rst @@ -1,7 +1,7 @@ Solr Search Index ================= -A Dataverse installation requires Solr to be operational at all times. If you stop Solr, you should see a error about this on the root Dataverse installation page, which is powered by the search index Solr provides. You can set up Solr by following the steps in our Installation Guide's :doc:`/installation/prerequisites` and :doc:`/installation/config` sections explaining how to configure it. This section you're reading now is about the care and feeding of the search index. PostgreSQL is the "source of truth" and the Dataverse installation will copy data from PostgreSQL into Solr. For this reason, the search index can be rebuilt at any time. Depending on the amount of data you have, this can be a slow process. You are encouraged to experiment with production data to get a sense of how long a full reindexing will take. +A Dataverse installation requires Solr to be operational at all times. If you stop Solr, you should see an error about this on the root Dataverse installation page, which is powered by the search index Solr provides. You can set up Solr by following the steps in our Installation Guide's :doc:`/installation/prerequisites` and :doc:`/installation/config` sections explaining how to configure it. This section you're reading now is about the care and feeding of the search index. PostgreSQL is the "source of truth" and the Dataverse installation will copy data from PostgreSQL into Solr. For this reason, the search index can be rebuilt at any time. Depending on the amount of data you have, this can be a slow process. You are encouraged to experiment with production data to get a sense of how long a full reindexing will take. .. contents:: Contents: :local: From 912bd89732f7f101a21c522781b5cce30e41bfc8 Mon Sep 17 00:00:00 2001 From: Henning Timm Date: Tue, 18 Oct 2022 21:28:48 +0200 Subject: [PATCH 8/8] Extended description of full vs in-place index --- doc/sphinx-guides/source/admin/solr-search-index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/sphinx-guides/source/admin/solr-search-index.rst b/doc/sphinx-guides/source/admin/solr-search-index.rst index 769c1ee5a0d..e6f7b588ede 100644 --- a/doc/sphinx-guides/source/admin/solr-search-index.rst +++ b/doc/sphinx-guides/source/admin/solr-search-index.rst @@ -9,7 +9,7 @@ A Dataverse installation requires Solr to be operational at all times. If you st Full Reindex ------------- -There are two ways to perform a full reindex of the Dataverse installation search index. Starting with a "clear" ensures a completely clean index but involves downtime. Reindexing in place doesn't involve downtime but does not ensure a completely clean index. +There are two ways to perform a full reindex of the Dataverse installation search index. Starting with a "clear" ensures a completely clean index but involves downtime. Reindexing in place doesn't involve downtime but does not ensure a completely clean index (e.g. stale entries from destroyed datasets can remain in the index). Clear and Reindex +++++++++++++++++