Create a new priority scoring consumable resource #354

apmasell · 2024-02-13T18:54:21Z

Adds a new consumable resource that can collect data from other systems and then score the data to decide whether a job should launch.

Jira ticket:

Includes a change file
Updates developer documentation

grapigeau · 2024-02-26T14:40:58Z

vidarr-core/src/main/java/ca/on/oicr/gsi/vidarr/core/PriorityConsumableResource.java

Is this PR assuming the removal of the ca.on.oicr.gsi.vidarr.server.PriorityByWorkflow class? That seems logical to me and if not there will be a conflict for key priority.

Oh, it seems they do. Up to you. I can rename either of them.

I don't have a strong preference for renaming either one or the other. Whichever seems more logical :)

Okay, I've talked to Alexis and we're going with delete.

grapigeau

This will also have to be removed from Main: private final PriorityByWorkflow priorityPerWorkflow = new PriorityByWorkflow();
*Meant to comment above. Re: removing the PriorityByWorkflow

apmasell · 2024-02-26T16:37:40Z

Okay. I think the deletion is complete now.

avarsava · 2024-02-26T18:08:24Z

plugin-guide.md

@@ -211,6 +211,25 @@ to filter on runs. A filter could query Pinery and get all the external
 identifiers associated with that run and then construct a query based on those
 to match workflow runs that use any of those identifiers.

+# Priority Consumable Resource Inputs, Formulas, and Scorers
+The priority consumable resource takes plugins for the inputs, formulas, and
+scorers. This follows the same pattern as the other plugins: an implementation


This seems to be the first time 'formula' and 'scorer' appears in this document, so I'm not sure what these are, or what other plugins use them?

avarsava · 2024-02-26T18:21:27Z

plugin-guide.md

@@ -273,6 +292,36 @@ active.
 }
 ```

+### Priority Consumable Resource
+The priority consumable resource resource operates by computing a number, a


"resource resource"

avarsava · 2024-02-26T18:21:56Z

plugin-guide.md

+based on that number.
+
+The resource first takes data from the submission request and then
+implmentations of `PriorityInput` consume this data and produce a numeric


implmentations -> implementations

avarsava · 2024-02-26T18:25:19Z

plugin-guide.md

 This can be limited to a particular input type format.

+## Remote
+Allows input to be requested from a remote service via a JSON payload.


Is this meant to be a subsection of 'Input Provisioner'?

avarsava · 2024-02-26T18:25:53Z

plugin-guide.md

+The `"schema"` property defines a type, including an object types, that will be
+required on submission. The data provided by the submission will be send via
+`POST` request as the body to the URL provided. The endpoint must respond with
+an integer for the priority or null to use the default priority. The result


Is this feature specifically for the priority consumable resource?

avarsava · 2024-02-26T18:27:26Z

plugin-guide.md

+```
+
+### JSON Dictionary
+Takes input as a string and lookups up the value of that in a dictionary. If


"lookups up"

avarsava · 2024-02-26T18:32:12Z

plugin-guide.md

+3. All the records that were returned by the query are scanned for a matching
+   label set.
+4. If a matching label set is found, the last recorded value will be used,
+   regardless of when Prometheus obeserved it.


obeserved -> observed

avarsava · 2024-02-26T18:37:07Z

plugin-guide.md

+Priority scorers provided in Víðarr core.
+
+### All Of
+Checks serveral priority scorers and allows permits the workflow run to proceed


serveral -> several

avarsava · 2024-02-26T18:37:27Z

plugin-guide.md

+
+### All Of
+Checks serveral priority scorers and allows permits the workflow run to proceed
+if all scorers allow it proceed.


allow it to proceed?

avarsava · 2024-02-26T18:37:58Z

plugin-guide.md

+among the top 20 workflow run in their respective workflow type.
+
+### Any Of
+Checks serveral priority scorers and allows permits the workflow run to proceed


serveral -> several

avarsava · 2024-02-26T18:40:33Z

plugin-guide.md

+
+### Any Of
+Checks serveral priority scorers and allows permits the workflow run to proceed
+if any scorer would allow it proceed.


allow it to proceed

avarsava · 2024-02-26T18:44:40Z

plugin-guide.md

+
+```
+{
+  "type": "subtract",


The others are 'product' and 'sum', should this one be 'difference' for consistency?

apmasell · 2024-02-29T13:10:05Z

Fixed all the concerns.

avarsava · 2024-03-01T13:25:22Z

plugin-guide.md

+### Remote Input
+Takes an arbitrary JSON value and sends it to remote HTTP endpoint for
+evaluation. That endpoint must return a single number. The result will be
+cached. The `"schema"` is a stanadard Víðarr type that should be requested from


this one appears to be outstanding

avarsava · 2024-03-01T13:25:46Z

plugin-guide.md

+```
+
+The `"schema"` property defines a type, including an object types, that will be
+required on submission. The data provided by the submission will be send via


will be send -> sent

I'm not sure what's outstanding here. There's only one Remote Input now.

avarsava · 2024-03-01T13:32:21Z

plugin-guide.md

+
+```
+{
+  "type": "oneOf",


why is everything like-this except that this one is likeThis ?

Historical reasons. I'm not sure if I should change them all to kebab case.

avarsava · 2024-03-01T13:53:36Z

vidarr-core/src/main/java/ca/on/oicr/gsi/vidarr/core/PriorityConsumableResource.java

+  }
+
+  @Override
+  public int priority() {


This priority is different than the job prioritys that the ConsumableResource manages, right?

Adds a new consumable resource that can collect data from other systems and then score the data to decide whether a job should launch.

avarsava requested review from avarsava and grapigeau February 14, 2024 14:13

grapigeau reviewed Feb 26, 2024

View reviewed changes

apmasell force-pushed the priority_lookup branch from a5f40bb to c9cf166 Compare February 26, 2024 16:37

avarsava reviewed Feb 26, 2024

View reviewed changes

avarsava reviewed Mar 1, 2024

View reviewed changes

avarsava approved these changes Mar 1, 2024

View reviewed changes

Create a new priority scoring consumable resource

92ed389

Adds a new consumable resource that can collect data from other systems and then score the data to decide whether a job should launch.

grapigeau approved these changes Mar 1, 2024

View reviewed changes

apmasell force-pushed the priority_lookup branch from 19be94e to 92ed389 Compare March 1, 2024 15:33

avarsava merged commit af69310 into oicr-gsi:master Mar 6, 2024
1 check passed

apmasell deleted the priority_lookup branch March 6, 2024 13:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create a new priority scoring consumable resource #354

Create a new priority scoring consumable resource #354

apmasell commented Feb 13, 2024 •

edited by Christopher-Ho

Loading

grapigeau Feb 26, 2024

apmasell Feb 26, 2024

grapigeau Feb 26, 2024

apmasell Feb 26, 2024

grapigeau left a comment •

edited

Loading

apmasell commented Feb 26, 2024

avarsava Feb 26, 2024

avarsava Feb 26, 2024

avarsava Feb 26, 2024

avarsava Feb 26, 2024

avarsava Feb 26, 2024

avarsava Feb 26, 2024

avarsava Feb 26, 2024

avarsava Feb 26, 2024

avarsava Feb 26, 2024

avarsava Feb 26, 2024

avarsava Feb 26, 2024

avarsava Feb 26, 2024

apmasell commented Feb 29, 2024

avarsava Mar 1, 2024

avarsava Mar 1, 2024

apmasell Mar 1, 2024

avarsava Mar 1, 2024

apmasell Mar 1, 2024

avarsava Mar 1, 2024

apmasell Mar 1, 2024

Create a new priority scoring consumable resource #354

Create a new priority scoring consumable resource #354

Conversation

apmasell commented Feb 13, 2024 • edited by Christopher-Ho Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grapigeau left a comment • edited Loading

Choose a reason for hiding this comment

apmasell commented Feb 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apmasell commented Feb 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apmasell commented Feb 13, 2024 •

edited by Christopher-Ho

Loading

grapigeau left a comment •

edited

Loading