docs: How to debug failed submissions (#4018)

theopensystemslab · Dec 2, 2024 · 5392673 · 5392673
1 parent 928dee0
commit 5392673
Show file tree

Hide file tree

Showing 21 changed files with 106 additions and 5 deletions.
diff --git a/...how-to/how-to-setup-aws-s3-submissions.md → ...to/aws/how-to-setup-aws-s3-submissions.md b/...how-to/how-to-setup-aws-s3-submissions.md → ...to/aws/how-to-setup-aws-s3-submissions.md
diff --git a/...ow-to/how-to-setup-aws-sso-credentials.md → ...o/aws/how-to-setup-aws-sso-credentials.md b/...ow-to/how-to-setup-aws-sso-credentials.md → ...o/aws/how-to-setup-aws-sso-credentials.md
diff --git a/doc/how-to/how-to-debug-failed-submissions.md b/doc/how-to/how-to-debug-failed-submissions.md
@@ -0,0 +1,99 @@
+# How to debug failed submissions
+
+This guide will explain how to troubleshoot failed application submissions. 
+
+If a live application submission fails it is important to  investigate this urgently, and aim for same-day re-submission if possible. Luckily this doesn't happen very often! 
+
+More information on how to re-submit applications can be found in [./how-to-resubmit-an-application.md](./how-to-resubmit-an-application.md).
+
+## Finding out that a submission has failed
+
+### Via `#planx-errors`
+
+Normally we find out that a submission has failed through a notification in the `#planx-errors` Slack channel, and these are usually easy to spot as they are longer than the normal errors. 
+
+They also start with `Error: Sending to <backend> failed`.
+
+E.g.
+
+```json
+> New production error in planx (error severity)
+>
+Error: Sending to BOPS v2 failed (southwark - a-5e5510n-1d-a-535510n1d): Error: Invalid DigitalPlanning payload for session a-5e5510n-1d-a-535510n1d. Errors: [   {     "instancePath": "/data/applicant/ownership/owners/0",     "schemaPath": "#/required",     "keyword": "required",     "params": {       "missingProperty": "noticeGiven"     },     "message": "must have required property 'noticeGiven'"   },   {     "instancePath": "/data/applicant/ownership/owners/0",     "schemaPath": "#/required",     "keyword": "required",     "params": {       "missingProperty": "noNoticeReason"     },     "message": "must have required property 'noNoticeReason'"   },   {     "instancePath": "/data/applicant/ownership/ow… 
+> Show more
+> Occurred in file:///api/dist/modules/send/bops/bops.js:98
+
+```
+
+### Via `#planx-notifications`
+
+There are normally three notifications per Lawful Development Certificate (LDC) submission in the `#planx-notifications` Slack channel, so a tell-tale sign of an error is if there are fewer messages. 
+
+For example, along with the production error above, for this submission there were only these two messages (the BOPS submission was missing):
+
+```
+6:02PM New GOV Pay payment  cj14av1r8hd8vdqp5e2tn5gto1 with status success [southwark]
+6:03PM New Uniform submission a-5e5510n-1d-a-535510n1d [Southwark]
+```
+
+When we successfully re-submitted the application, we got this third notification:
+
+```
+11:06AM New BOPS submission 24-00242-LDCP [https://southwark.bops.services/api/v2/planning_applications]
+```
+
+### Feedback from users
+
+Ideally we never want to get in the situation where an end-user notices the error before we do. But we may be informed about a failed submission via Slack channels shared with Local Planning Authorities (LPAs), such as the Open Digital Planning (ODP) Slack workspace.
+
+## Investigating the error
+
+We have several admin API endpoints that exist to help debug submission issues. These can be viewed in the [Swagger docs](https://api.editor.planx.uk/docs/#/admin).
+
+If it is not obvious from the error notification, you can follow these steps to investigate further:
+
+#### Inspect the whole error message
+
+1. Get the `sessionId` from the error notification.
+2. In a browser, hit `api.editor.planx.uk/admin/session/{sessions-id}/digital-planning-application`. This should output the full error message.
+
+#### Inspect the erroring payload
+3. To get the schema payload that caused this error, add a `skipValidation` query param to the `digital-planning-application` URL, i.e.`?skipValidation=true`
+
+#### Compare the payload with the schema
+1. There are useful online tools which can help find the validation issue, such as `json-schema.app`. 
+2. Load the current schema URL (e.g. `https://theopensystemslab.github.io/digital-planning-data-schemas/v0.7.1/schemas/application.json` ), and you can compare the erroring payload directly with the schema via the window on the right in the same tool. 
+
+> [!TIP]
+> The schema URL can be found at the bottom of the payload:
+>
+> `{ ..."schema":  "https://theopensystemslab.github.io/digital-planning-data-schemas/v0.7.1/schemas/application.json" }`
+
+#### Read the code
+1. The flow passport is mapped to the schema in this file: https://github.com/theopensystemslab/planx-core/blob/005ef823e68eaf9f7d7d5a2c793a6a729e0b6475/src/export/digitalPlanning/model.ts
+
+#### Inspect the Hasura record for the submission
+1. Use the admin `/summary` endpoint. This will output the hasura record for the submission, including the entire passport.
+
+#### Go through the flow that the submission came from 
+1. Trace the journey that the user took based on their passport variables, and find out if/where there is a mismatch between content and schema.
+
+> [!TIP]
+> It might be useful to toggle the data fields for the flow to help match the passport answers with the questions.
+
+> [!WARNING]
+> Note that the published flow that the applicant used might be different from the current version of the editor flow!
+
+## Fixing the payload
+
+If the solution involves fixing the submission payload before resubmitting, you can do this as follows: 
+
+1. Go to the production hasura console and filter the `lowcal_sessions` table with the `session_id`. 
+2. Make a copy of the contents of the `data` field (it is a good idea to store this temporarily in your code editor in case multiple changes are needed).
+3. Fix the payload (e.g. this might mean inserting a new key-value pair in the JSON) and validate the payload with a tool like [jsonlint.com](jsonlint.com).
+4. Replace the data field in Hasura with the corrected and validated payload.
+5. Save the data, then rerun the admin endpoints from earlier (without the `skipValidation` param) to check that there are no longer any errors.
+6. If all is good, resubmit the application - follow the instructions in [./how-to-resubmit-an-application.md](./how-to-resubmit-an-application.md).
+
+> [!CAUTION]
+> Production data should only be edited when pairing with another OSL developer!
diff --git a/doc/how-to/how-to-resubmit-an-application.md b/doc/how-to/how-to-resubmit-an-application.md
@@ -27,13 +27,15 @@ In the near future, this process should be replaced by a re-submission API which
    * `bops_application.response.message` should be changed from `"Application created"` to `"FAILED"`
 
 6. In Insomnia (or HTTP client of choice), prepare to re-submit the payload - 
-   * API Endpoint - `https://api.editor.planx.uk/${destination}/${localAuthority}`
-      * For example, https://api.editor.planx.uk/uniform/aylesbury-vale
+   * API Endpoint - `https://api.editor.planx.uk/create-send-events/${sessionId}`
+      * For example, https://api.editor.planx.uk/create-send-events/a535510n-1d-a535510n-1d
    * HTTP Method - POST
    * Headers - `{ "Authorization": ${HASURA_PLANX_API_KEY} }`
      * This secret can be found via Pulumi or your `.env` file
+
+This can also be done directly from the [Swagger docs](https://api.editor.planx.uk/docs/).
 
-> ⚠️ *Uniform instances do not neatly map to local authorities. Please take care to ensure that the value for `localAuthority` is taken from `uniform_applications.destination` when re-submitting to Uniform*
+> ⚠️ *Uniform instances do not neatly map to local authorities. Please take care to ensure that the value for `localAuthority` in the payload is taken from `uniform_applications.destination` when re-submitting to Uniform*
 
 7. Prepare the payload copied from Step 3. Ensure the structure is correct, see example - 
 
@@ -47,9 +49,9 @@ In the near future, this process should be replaced by a re-submission API which
 
 8. Send! ✉️
 
-9. Check response message from BoPS / Uniform to confirm success or failure. Rectify any issues if possible (e.g. malfored payload)
+9. Check response message from BoPS / Uniform to confirm success or failure (e.g. you can check the submissions log in the PlanX Editor). Rectify any issues if possible (e.g. malformed payload)
 
-10. Notify partners on `#planx-notifications` channel of the re-submissions
+10. Notify partners on `#planx-notifications` channel of the re-submissions. It is useful to link missing submissions to their counterpart payment logs, if applicable (reply in thread).
 
 ## Process if the application has been paid for, but there are no submission attempts recorded
 

diff --git a/...how-to/how-to-add-allow-list-variables.md → ...tabase/how-to-add-allow-list-variables.md b/...how-to/how-to-add-allow-list-variables.md → ...tabase/how-to-add-allow-list-variables.md
diff --git a/...w-to/how-to-grant-metabase-permissions.md → ...base/how-to-grant-metabase-permissions.md b/...w-to/how-to-grant-metabase-permissions.md → ...base/how-to-grant-metabase-permissions.md
diff --git a/...tup-metabase-for-a-new-team-or-service.md → ...tup-metabase-for-a-new-team-or-service.md b/...tup-metabase-for-a-new-team-or-service.md → ...tup-metabase-for-a-new-team-or-service.md
diff --git a/...etup-metabase/default_values_filtered.png → ...etup-metabase/default_values_filtered.png b/...etup-metabase/default_values_filtered.png → ...etup-metabase/default_values_filtered.png
diff --git a/.../setup-metabase/duplicate_a_dashboard.png → .../setup-metabase/duplicate_a_dashboard.png b/.../setup-metabase/duplicate_a_dashboard.png → .../setup-metabase/duplicate_a_dashboard.png
diff --git a/...es/setup-metabase/edit_flow_in_hasura.png → ...es/setup-metabase/edit_flow_in_hasura.png b/...es/setup-metabase/edit_flow_in_hasura.png → ...es/setup-metabase/edit_flow_in_hasura.png
diff --git a/.../images/setup-metabase/enable_sharing.png → .../images/setup-metabase/enable_sharing.png b/.../images/setup-metabase/enable_sharing.png → .../images/setup-metabase/enable_sharing.png
diff --git a/.../images/setup-metabase/new_collection.png → .../images/setup-metabase/new_collection.png b/.../images/setup-metabase/new_collection.png → .../images/setup-metabase/new_collection.png
diff --git a/...tup-metabase/only_duplicate_dashboard.png → ...tup-metabase/only_duplicate_dashboard.png b/...tup-metabase/only_duplicate_dashboard.png → ...tup-metabase/only_duplicate_dashboard.png
diff --git a/...p-metabase/service_slug_default_value.png → ...p-metabase/service_slug_default_value.png b/...p-metabase/service_slug_default_value.png → ...p-metabase/service_slug_default_value.png
diff --git a/...images/setup-metabase/share_with_team.png → ...images/setup-metabase/share_with_team.png b/...images/setup-metabase/share_with_team.png → ...images/setup-metabase/share_with_team.png
diff --git a/...etup-metabase/team_slug_default_value.png → ...etup-metabase/team_slug_default_value.png b/...etup-metabase/team_slug_default_value.png → ...etup-metabase/team_slug_default_value.png
diff --git a/...ow-to/images/setup-metabase/templates.png → ...abase/images/setup-metabase/templates.png b/...ow-to/images/setup-metabase/templates.png → ...abase/images/setup-metabase/templates.png
diff --git a/.../setup-metabase/update_analytics_link.png → .../setup-metabase/update_analytics_link.png b/.../setup-metabase/update_analytics_link.png → .../setup-metabase/update_analytics_link.png
diff --git a/doc/how-to/how-to-add-a-secret.md → doc/how-to/secrets/how-to-add-a-secret.md b/doc/how-to/how-to-add-a-secret.md → doc/how-to/secrets/how-to-add-a-secret.md
diff --git a/doc/how-to/how-to-add-a-team-secret.md → ...ow-to/secrets/how-to-add-a-team-secret.md b/doc/how-to/how-to-add-a-team-secret.md → ...ow-to/secrets/how-to-add-a-team-secret.md
diff --git a/doc/how-to/how-to-generate-a-secret.md → ...ow-to/secrets/how-to-generate-a-secret.md b/doc/how-to/how-to-generate-a-secret.md → ...ow-to/secrets/how-to-generate-a-secret.md