feat: generate JSON payloads for discretionary services

[jessicamcinchak]

Changes:

• Generates & includes a Digital Planning Application JSON for every "Send to email" or "Upload to S3" submission, skipping validation if not a supported application type
  • For example, this should be sufficient for Barnet to begin testing "Report a Planning Breach" with Power Automate
• Does not change anything about current BOPS or Uniform payloads
  • BOPS currently rejects invalid payloads, they may accept discretionary services in future
  • Uniform zip can't contain JSON files at all currently
• Moved pay helpers out of send module into pay!

Open questions:

• What's a "generic" or "discretionary" payload to begin with? I've decided it's simply the ODP Schema shape, but without any validation for a couple of reasons:
  • We have existing methods to generate this
  • data already has defined representations for some of our most common component types (FindProperty, ContactInput) & responses is already generic enough to support a service that literally sets zero data fields
  • We can focus our dev time & energy on maintaining a single set of passport mappings going forward, benefiting both validated statutory service and discretionary service payloads
  • I imagine a slightly more complex future (but we don't have to get there yet) that is maybe like:
    • Applications types within "DigitalPlanningApplication" schema definition
    • Application type(s) within "DigitalPlanningPreApplication" schema definition
    • Then all (invalid) discretionary things
• What's the best way to consume ODP Schema type definitions in planx-new? See this thread with initial ideas: https://opensystemslab.slack.com/archives/C01E3AC0C03/p1716279436365409
  • Is this implementation in or out of scope of this initial piece of work?

Testing:

• Sent through a Report a Breach via Barnet successfully end-to-end with Kev ✔️ https://3184.planx.pizza/barnet/report-a-planning-breach-template-v1-test/settings/submissions
  • This highlights where we should improve our schema passport mappings (eg default to undefined rather than 0/false more often), but that's something we can address as a follow-up in planx-core https://api.3184.planx.pizza/admin/session/d7896b95-ca38-4a82-ad5c-4815c07953eb/digital-planning-application?skipValidation=true

[jessicamcinchak]

Implement a standard JSON payload shape that will work for any Planx service

[github-actions]

Removed vultr server and associated DNS entries

[DafyddLlyr]

Apologies for the slow turnaround time here!

Approved - very happy for this to progress as is - pragmatic and lightweight solution which is great ✅

Two non-blocking comments which may be worth a discussion -

Skipping all validation
What we're doing by skipping all validation is potentially sending anything or nothing across, and we can't describe the object to recipients easily (e.g. by sharing the schema). I think currently{} and {"hello": "world"} could be posted here. This is obviously massively unlikely / effectively impossible as it's going through our current transformer which we know works very well and is validated.

Longer term, once we have multiple schemas in the schema repo, or other utils functions, I think we could validate a discretionary payload against the schema derived from on optional schema. Something like -

import type { PartialDeep } from "type-fest";


type Discretionary = PartialDeep<Schema>

This should mean all fields are optional, but still have to conform to expectations where present - sort of a best of both worlds?

This means we could also share something (a json schema doc or a single simple payload) with consumers to say "you're going to get one of these objects". As I've said above, this isn't super pressing as we're using our current payload generation - this means failures are unlikely and we already have the existing schema to share 👍

Missing passport
Currently we're just sending a subset of the ODP schema with this approach. If a discretionary service is mostly concerned with another domain area, there's potentially a lot of data not being passed on (outside the QuestionAndResponses[] structure).

I think it's worth considering adding the user's passport in a discretionary service payload which gives Editors and consumers full power to ask questions, and then build integrations based on these responses.

The passport could be a separate JSON file, or even better just attached to the payload -

type Discretionary = Schema & { passport?: Record<string, string, string[], boolean, number, object>

This would obviously be untyped, unstructured, and unvalidated beyond the simple example above, but would allow Editors to set a random data field, and allow these fields to be consumed by integrations.

[DafyddLlyr]

👍

[jessicamcinchak]

@DafyddLlyr - thanks for taking a look at this one! re your two points:

1. You're right that PartialDeep<DigitalPlanningApplication> would be a nicer type here - but after looking at this closer I think this return type should be implemented in the planx-core method responsible for actually generating & returning the invalidated payload (eg here), rather than as part of this change? I can pick this up in a future PR.

2. I've explicitly avoided returning/appending the passport for a couple reasons:

• Current discretionary services are typically not setting any data fields or logic (therefore existing responses captures everything already) besides default ones (eg _address, site boundary, etc).
• The raw passport would add a lot of noise and I think we'd want to filter out any _-prefixed variables like _constraints etc
• Including the raw passport feels like a revert back to our planx_debug_data days with BOPS and I think has potential to build a reliance that we do not want
• I sort of think the whole point of the ODP Schema is to show value of structured data when services have validation rules, and it makes sense that is going to be better for statutory services than discretionary ones here !