Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add ODP Schema validation example for FileTypes to publish checks #3373

Merged
merged 7 commits into from
Jul 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 26 additions & 6 deletions api.planx.uk/modules/flows/docs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,25 @@ components:
type: array
items:
type: string
ValidationCheck:
type: object
properties:
title:
type: string
example: File types
required: true
status:
type: string
enum:
- Pass
- Fail
- Warn
- Not applicable
example: Pass
message:
type: string
example: Your flow has valid file types
required: true
responses:
CopyFlow:
content:
Expand Down Expand Up @@ -124,19 +143,20 @@ components:
properties:
message:
type: string
required: false
description:
type: string
required: false
required: true
alteredNodes:
oneOf:
- type: array
items:
$ref: "#/components/schemas/Node"
- type: "null"
updatedFlow:
$ref: "#/components/schemas/FlowData"
validationChecks:
required: false
oneOf:
- type: array
items:
$ref: "#/components/schemas/ValidationCheck"
- type: "null"
FlattenData:
content:
application/json:
Expand Down
89 changes: 79 additions & 10 deletions api.planx.uk/modules/flows/validate/service.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { getValidSchemaValues } from "@opensystemslab/planx-core";
import {
ComponentType,
Edges,
Expand All @@ -6,6 +7,7 @@
} from "@opensystemslab/planx-core/types";
import * as jsondiffpatch from "jsondiffpatch";
import intersection from "lodash/intersection";
import countBy from "lodash/countBy";

import { dataMerged, getMostRecentPublishedFlow } from "../../../helpers";
import {
Expand All @@ -23,7 +25,7 @@

type ValidationResponse = {
title: string;
status: "Pass" | "Fail" | "Not applicable";
status: "Pass" | "Fail" | "Warn" | "Not applicable";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was expecting to see this change also picked up in docs.yaml, but it looks like we've missed adding in the validationChecks property.

Happy for this to be picked up here, or in a follow up PR 👍

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for spotting this, Swagger docs now updated ✅

Screenshot from 2024-07-08 12-22-07

message: string;
};

Expand Down Expand Up @@ -55,16 +57,20 @@
const validationChecks = [];
const sections = validateSections(flattenedFlow);
const inviteToPay = validateInviteToPay(flattenedFlow);
validationChecks.push(sections, inviteToPay);
const fileTypes = validateFileTypes(flattenedFlow);
validationChecks.push(sections, inviteToPay, fileTypes);

// Sort validation checks by status: Fail, Pass, Not applicable
const applicableChecks = validationChecks
.filter((v) => v.status !== "Not applicable")
.sort((a, b) => a.status.localeCompare(b.status));
// Arrange list of validation checks in order of status: Fail, Warn, Pass, Not applicable
const failingChecks = validationChecks.filter((v) => v.status == "Fail");
const warningChecks = validationChecks.filter((v) => v.status === "Warn");
const passingChecks = validationChecks.filter((v) => v.status === "Pass");
const notApplicableChecks = validationChecks.filter(
(v) => v.status === "Not applicable",
);
const sortedValidationChecks = applicableChecks.concat(notApplicableChecks);
const sortedValidationChecks = failingChecks
.concat(warningChecks)
.concat(passingChecks)
.concat(notApplicableChecks);
Copy link
Member Author

@jessicamcinchak jessicamcinchak Jul 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyone have a more clever suggestion for how to sort in a custom order by "status" ? This is explicit and easy-to-read, but a bit lengthy and I feel like there's probably some cool one-liner I'm missing?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I can think of a few ways!

Map.groupBy is pretty new but seems to have decent coverage https://caniuse.com/?search=map.groupby

I think it would then just be Map.groupBy(passingChecks, (check) => check.status)).

If we hit browser compatibility issues here we could import groupBy from lodash, or use a reduce().

Docs: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map/groupBy

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah just realised we're in the API here on Node 18 - I don't think Map.groupBy() would be available.

Lodash or reduce() would be the way to go here 👍


return {
alteredNodes,
Expand Down Expand Up @@ -193,9 +199,8 @@
};

const inviteToPayEnabled = (flowGraph: FlowGraph): boolean => {
const payNodes = Object.entries(flowGraph).filter(
(entry): entry is [string, Node] =>
isComponentType(entry, ComponentType.Pay),
const payNodes = Object.entries(flowGraph).filter((entry) =>
isComponentType(entry, ComponentType.Pay),
);
const payNodeStatuses = payNodes.map(
([_nodeId, node]) => node?.data?.allowInviteToPay,
Expand All @@ -206,4 +211,68 @@
);
};

const validateFileTypes = (flowGraph: FlowGraph): ValidationResponse => {
// Get all passport variables set by FileUpload and/or FileUploadAndLabel
const allFileFns = [
...getFileUploadNodeFns(flowGraph),
...getFileUploadAndLabelNodeFns(flowGraph),
];
if (allFileFns.length < 1) {
return {
title: "File types",
status: "Not applicable",
message: "Your flow is not using FileUpload or UploadAndLabel",
};
}

// Get all file types supported by current release of ODP Schema & compare
const validFileTypes = getValidSchemaValues("FileType");
const invalidFileFns: string[] = [];
allFileFns.forEach((fn) => {
if (!validFileTypes?.includes(fn)) {
invalidFileFns.push(fn);
}
});
if (invalidFileFns.length > 0) {
// Get unique fns with count of occurances
const countInvalidFileFns = countBy(invalidFileFns);
const summarisedInvalidFileFns: string[] = [];
Object.entries(countInvalidFileFns).map(([k, v]: [string, number]) => {
summarisedInvalidFileFns.push(`${k} (${v})`);
});
return {
title: "File types",
status: "Warn",
message: `Your FileUpload or UploadAndLabel are setting data fields that are not supported by the current release of the ODP Schema: ${summarisedInvalidFileFns.join(", ")}`,
};
}

return {
title: "File types",
status: "Pass",
message:
"Files collected via FileUpload or UploadAndLabel are all supported by the ODP Schema",
};
};

const getFileUploadNodeFns = (flowGraph: FlowGraph): string[] => {
const fileUploadNodes = Object.entries(flowGraph).filter((entry) =>
isComponentType(entry, ComponentType.FileUpload),
);
return fileUploadNodes.map(([_nodeId, node]) => node.data?.fn as string);
};

const getFileUploadAndLabelNodeFns = (flowGraph: FlowGraph): string[] => {
// Exclude Upload & Label nodes used in "info-only" mode with a hidden dropzone
const uploadAndLabelNodes = Object.entries(flowGraph).filter(
(entry) =>
isComponentType(entry, ComponentType.FileUploadAndLabel) &&
entry[1].data?.hideDropZone !== true,
);
const uploadAndLabelFileTypes = uploadAndLabelNodes
.map(([_nodeId, node]: [string, Node]) => node.data?.fileTypes)
.flat();
return uploadAndLabelFileTypes?.map((file: any) => file?.fn as string);

Check warning on line 275 in api.planx.uk/modules/flows/validate/service.ts

View workflow job for this annotation

GitHub Actions / Run API Tests

Unexpected any. Specify a different type
jessicamcinchak marked this conversation as resolved.
Show resolved Hide resolved
};

export { validateAndDiffFlow };
Loading
Loading