creating-editing-and-removing-specialist-document-types-and-finders.md

Creating, editing and removing specialist document types and finders

To create or edit a new specialist document you will have to make changes to this application, publishing-api and search-api. You will not have to make any changes to frontend applications.

IMPORTANT

Recent changes to Specialist Publisher now allow users to self-serve. These requests will be coming through Zendesk, as a code diff for the specialist-publisher repo. Follow the steps below to validate the thoroughness of the changes and open the remaining PRs against publishing-api and search-api, if needed.

Creating a specialist document type

1. Add a schema to Publishing API

See example PR here.

1. Add the format to allowed document types list.
2. Add any new field definitions to this file.
3. Run bundle exec rake build_schemas to regenerate schemas.

When the PR is reviewed and its tests passing, it can be merged and deployed at this point.

2. Create a new specialist document format in Specialist Publisher

Create the schema

See CMA cases.

New formats should be added with target_stack: "draft" so that departments can preview the finder before you publish it.

You'll need to generate your own UUIDs for the content_id (of the finder), and the signup_content_id fields:

$ irb
irb(main):001:0> require "securerandom"
=> true
irb(main):002:0> SecureRandom.uuid
=> "5087e8b6-ee54-40f9-b592-8c2813c7037d"

For a breakdown of email subscription options see Configure the email sign up page.

Create the model

Include any required validations, tests and factories.

See CMA cases.

Create the view template

Create a view file in the views folder. See CMA cases example.

Whilst a handful of legacy finders still use a custom view, all new finders should be referencing the shared template, which determines which form fields to display based on the finder schema.

NB: The select type of an input (one/multiple) is now configured under the specialist_publisher_properties in the schema.

3. Configure Search API

Search API needs copies of the schema very similar to the one in Specialist Publisher. See:

• CMA case schema (example)
• field definitions

You'll also need to add your document format to:

• the main ES types list govuk.json
• migrated_formats.yaml
• mapped_document_types.yaml

Finally, you'll need to add your custom fields to:

• elasticsearch_presenter.rb
• specialist_presenter.rb

4. Configure the email sign up page

The email sign up page is rendered by Finder Frontend using the schema configuration in specialist publisher.

The finder default is to have no email subscription. Email subscriptions can be set as:

1. Subscribe to all fields

• Configure signup_content_id - a new UUID for the email signup page.
• Whilst the above step is enough to configure email subscription, it does not offer the user any filtering options. You can additionally allow the user to preserve their facet selection when navigating to the email subscription page:
  • In the email_filter_options hash, set email_filter_by to all_selected_facets. This will pick up all the facets that have allowed_values and filterable: true. See example.
  • Edit email-alert-api by adding all filterable facets to valid_tags.rb.
• You may exclude some of the facets by additionally setting all_selected_facets_except_for - see example.
• Set subscription_list_title_prefix (optional).

2. Subscribe to specific fields(set)

• Configure the signup_content_id and, optionally, the subscription_list_title_prefix, as in the previous step.
• Set email_filter_by to a specific facet - see example.
• Edit email-alert-api by adding the new filters to valid_tags.rb.

Breakdown of the email options:

• subscription_list_title_prefix - typically set in most finders. It defines the beginning of the email title. For example, CMA cases have this value set to CMA cases, meaning an email title would read as "CMA cases with digital markets unit". When omitted, the email title will only reference subscribed facets.
• email_alert_topic_name_overrides - changes the display label of individual facets options in the email title the user receives upon subscribing.
• downcase_email_alert_topic_names - downcases the display label of individual facet options in the email title the user receives upon subscribing.
• pre_checked_email_alert_checkboxes - takes an array of facets, which will appear pre-checked on the email signup page.

3. Signup link

If the subscription is managed by an external service, it should be set via a signup_link. See example.

5. Deploy a finder to the draft stack (for previewing)

To deploy a new finder for previewing:

1. Ensure the finder target_stack is set to draft
2. Deploy the finder to draft stack
3. Wait for department's feedback and approval and agree on a release date

NB: Depending on the finder requirements, you may choose to allow the users to publish documents in preview mode, which would enable them to test the full finder filtering functionality. Changes to the schema requested after documents have been published, could require running a reindex, and there is a risk of loss of data. In order to prevent users from publishing, we could give only basic Signon permissions whilst in preview mode. Signon access to the Specialist Publisher app, only gives the user writer access (they may create, edit, and update, but not publish or unpublish).

6. Publish a finder to the live stack

To release the finder to the live stack:

1. Open a new PR changing the target_stack of the finder from draft to live in Specialist Publisher json schema config
2. Deploy the finder to the live stack

7. Deploy a finder

1. Merge and deploy Publishing API, Specialist Publisher and Search API, if changed.
   • Ensure you deploy Publishing API first, to avoid schema validation errors.
2. Also deploy Email Alert API if you have made changes to it.
3. Depending on the changes you've made, you might need to update mappings in Search API.
   • You will likely need to run rake SEARCH_INDEX=govuk 'search:update_schema'. For further details, see this section on reindexing.
4. Publish the finder by running the rake task publishing_api:publish_finders or publishing_api:publish_finder[your_format_name_based_on_the_schema_file] against the specialist publisher app (rake tasks here).

8. Permissions

Specialist Publisher grants access to the publishing interface for a document type to the following Signon users:

1. Users that belong to the owner organisation - have default "write" permissions, allowing them to view and draft new documents.
2. Users that belong to the owner organisation AND have editor permission in Signon - are considered "departmental editors" and can publish, unpublish and discard documents.
3. Users that have the permission <your_new_document_type>_editor in Signon are granted "departmental editor" access regardless of their organisation. This is sometimes required for cross-departmental documents. These special permissions need to be created manually in Signon. You do not need to create this permission unless cross-departmental access has been explicitly requested.

You'll need to manually grant users access to the Specialist Publisher app in Signon, and the editor permission (or custom cross-organisation permissions from step 3 above) if appropriate.

Editing a specialist document type

We often receive requests to add new fields to a specialist document or to add new values to existing fields.

Adding a new field to an existing specialist document

1. In publishing-api:

   • Add the new field in the specialist_document schema. See example commit.
   • Run bundle exec rake build_schemas to regenerate schemas after adding the new value(s).

2. In specialist publisher:

   • Add the new field to the relevant model, including any required validations, tests and factories.
   • Add fields to relevant schema files.
   • For legacy finders that don't use the shared view, you may need to update the view.

See this commit for an example.

3. In search-api, add the new field in the following places (see this commit for an example):

   • the relevant schema in the elasticsearch_types directory.
   • the elasticsearch_presenter.
   • the specialist_presenter.
   • the field_definitions file.

4. Republish the finder

Removing a field from an existing specialist document

NB: In order to remove a field, ensure no document are tagged with that field, or that the finder's owners are aware of the data loss implications.

1. In publishing-api:

   • Remove the field from the specialist_document schema.
   • Run bundle exec rake build_schemas to regenerate schemas after removing the new value(s).

   See removal of key_reference in example commit.

2. In specialist publisher:

   • Remove field from model.
   • Remove field usage from view (if this is a legacy finder that isn't referring to the shared view).
   • Remove field from schema files.
   • Remove any other usages, such as from tests and factories.

   See removal of key_reference in example commit.

3. In search-api, remove the field from:

   • The relevant schema in the elasticsearch_types directory.
   • The elasticsearch_presenter.
   • The specialist_presenter.
   • The field_definitions file.

   See removal of key_reference in example commit.

4. Republish the finder

Adding values for existing fields on a specialist document

Specific values for fields of type array are now defined only in the specialist_publisher app. To add a value:

1. In specialist publisher, add the new values to the relevant file in the schema directory. See this commit for an example.
2. Deploy Specialist Publisher.
3. Publish the finder by running the rake task publishing_api:publish_finders or publishing_api:publish_finder[your_format_name_based_on_the_schema_file] against the specialist publisher app (rake tasks here).

Removing values for existing fields on a specialist document

NB: In order to remove a field value, ensure no document are tagged with that value, or that the finder's owners are aware of the data loss implications.

Specific values for fields of type array are now defined only in the specialist_publisher app. To remove a value:

1. In specialist publisher, remove the value entry from the allowed_values array of the relevant field, from the corresponding json file in the schema directory.
2. Deploy Specialist Publisher.
3. Publish the finder by running the rake task publishing_api:publish_finders or publishing_api:publish_finder[your_format_name_based_on_the_schema_file] against the specialist publisher app (rake tasks here).

Editing a specialist finder

The schema files that define a specialist document, are also used to configure that document's specialist finder. See this commit for an example.

Unpublishing a specialist finder

Important: In order to remove a finder, you should ensure there is a clear business requirement in doing so, and that all associated documents have been correctly dealt with - unpublished, redirected, migrated or removed. If the code is not removed from all relevant repositories, there is a chance the finder will be inadvertently republished, for example by republishing all finders. You must therefore make sure the code is removed in order to keep the finder unpublished.

The following steps are required to remove a finder:

1. Unpublish the finder in all environments using the provided rake task:

   rake unpublish:redirect_finder["uk_market_conformity_assessment_bodies","https://redirection_link.gov.uk"]
   

2. Remove usages from publishing-api:

   • Remove the format from allowed document types list.
   • Remove the field definitions from this file.
   • If present, remove the example from this directory.
   • Run bundle exec rake build_schemas to regenerate schemas.

   See example commit.

3. Remove usages from specialist publisher. See example commit.

4. Remove usages from search-api. See example commit.

5. Remove any usages from finder-frontend, if applicable.

6. Deploy all changes.

Reindexing breakdown

Regardless on the changes you're trying to make, you can implement them in such a way that a full reindex is not necessary.

Here's a breakdown of the different scenarios:

1. Add new, change or delete facet options

No mapping update or reindex necessary, because Elasticsearch does not have an allowlist of options. Any value works, as long as it is the correct type.

2. Add new facet

Mapping update (via update_schema rake task) necessary, but full reindex not required. Once the mapping has been updated, you can start adding data to both existing and new documents for the new field.

Run: rake SEARCH_INDEX=govuk 'search:update_schema' in a Search API console, to update the mapping.

3. Change name of facet

Just change the facet label in Specialist Publisher.

There's no actual need to change the field key. If you nonetheless want to change the key as well:

• declare a new field by following steps similar to adding a new field
• remove the old field config from Specialist Publisher
• run update_schema as in 2) above
• republish the finder
• retag and republish all the content from Specialist Publisher with the new field.

The Elasticsearch mapping for the old field would still exist, but it wouldn't be doing any harm.

4. Change type of facet

To avoid a reindex, you can follow the same steps as for a field key change at 3) above.

5. Delete facet

Remove config from Specialist Publisher and republish all content. The Elasticsearch mapping for the field would still exist, but it wouldn't be doing any harm.

If you still want to reindex, follow the instructions here. A full reindex takes around 30-45 minutes on Production, or 3-4 hours on Integration.