Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(server): add schemas and structs #1036

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

feat(server): add schemas and structs #1036

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3fc6bf4
feat(server): schemas adr protobuf
coltmcnealy-lh Sep 27, 2024
3027205
now it compiles
coltmcnealy-lh Sep 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
175 changes: 147 additions & 28 deletions docs/docs/08-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -759,10 +759,11 @@ Declares a Variable.

| Field | Label | Type | Description |
| ----- | ---- | ---- | ----------- |
| `type` | | [VariableType](#variabletype) | The Type of the variable. |
| `primitive` | oneof `type`| [PrimitiveType](#primitivetype) | The primitive type of the variable. |
| `schema` | oneof `type`| [SchemaId](#schemaid) | The ID of the schema that the variable belongs to. |
| `name` | | string | The name of the variable. |
| `default_value` | optional| [VariableValue](#variablevalue) | Optional default value if the variable isn't set; for example, in a ThreadRun if you start a ThreadRun or WfRun without passing a variable in, then this is used. |
| `masked_value` | | bool | If true, the variable value will show as a masked string. |
| `is_masked` | | bool | If true, the variable value will show as a masked string. |
<!-- end Fields -->
<!-- end HasFields -->

Expand Down Expand Up @@ -1137,6 +1138,20 @@ ID for a ScheduledWfRun



### Message `SchemaId` {#schemaid}

ID for a Schema


| Field | Label | Type | Description |
| ----- | ---- | ---- | ----------- |
| `name` | | string | The name of the `Schema`. |
| `version` | | int32 | The version of the `Schema`. |
<!-- end Fields -->
<!-- end HasFields -->



### Message `TaskDefId` {#taskdefid}

ID for a TaskDef.
Expand Down Expand Up @@ -1863,7 +1878,8 @@ EXPERIMENTAL: Creates a WorkflowEventDef
| Field | Label | Type | Description |
| ----- | ---- | ---- | ----------- |
| `name` | | string | |
| `type` | | [VariableType](#variabletype) | |
| `primitive_type` | oneof `type`| [PrimitiveType](#primitivetype) | |
| `schema` | oneof `type`| [SchemaId](#schemaid) | |
<!-- end Fields -->
<!-- end HasFields -->

Expand Down Expand Up @@ -2976,7 +2992,7 @@ A UserTaskField is a specific field of data to be entered into a UserTaskRun.
| Field | Label | Type | Description |
| ----- | ---- | ---- | ----------- |
| `name` | | string | The name of the field. When a UserTaskRun is completed, the NodeOutput is a single-level JSON_OBJ. Each key is the name of the field. Must be unique. |
| `type` | | [VariableType](#variabletype) | The type of the output. Must be a primitive type (STR, BOOL, INT, DOUBLE). |
| `type` | | [PrimitiveType](#primitivetype) | The type of the output. Must be a primitive type (STR, BOOL, INT, DOUBLE). |
| `description` | optional| string | Optional description which can be displayed by the User Task UI application. Does not affect WfRun execution. |
| `display_name` | | string | The name to be displayed by the User Task UI application. Does not affect WfRun execution. |
| `required` | | bool | Whether this field is required for UserTaskRun completion. |
Expand Down Expand Up @@ -3049,6 +3065,108 @@ SDK, which allows the Task Method to determine where the TaskRun comes from.



### Message `InlineSchema` {#inlineschema}

An `InlineSchema` is the actual representation of the Schema.


| Field | Label | Type | Description |
| ----- | ---- | ---- | ----------- |
| `fields` | repeated| [SchemaField](#schemafield) | The fields in this schema. |
<!-- end Fields -->
<!-- end HasFields -->



### Message `InlineStruct` {#inlinestruct}

An `InlineStruct` is a pre-validated set of fields that are part of a `Struct`.


| Field | Label | Type | Description |
| ----- | ---- | ---- | ----------- |
| `fields` | repeated| [StructField](#structfield) | The fields in the inline struct. |
<!-- end Fields -->
<!-- end HasFields -->



### Message `Schema` {#schema}

A `Schema` is a versioned metadata object (tenant-scoped) inside LittleHorse
that defines the structure and content of a variable value. It allows strong typing.


| Field | Label | Type | Description |
| ----- | ---- | ---- | ----------- |
| `id` | | [SchemaId](#schemaid) | The id of the `Schema`. This includes the version. |
| `description` | optional| string | Optionally description of the schema. |
| `schema` | | [InlineSchema](#inlineschema) | The `InlineSchema` defines the actual structure of any `Struct` using this `Schema`. |
<!-- end Fields -->
<!-- end HasFields -->



### Message `SchemaField` {#schemafield}

A `SchemaField` defines a field inside a `Schema`.


| Field | Label | Type | Description |
| ----- | ---- | ---- | ----------- |
| `name` | | string | The name of the field. |
| `optional` | | bool | Whether the field is optional. |
| `primitive` | oneof `field_type`| [PrimitiveType](#primitivetype) | Specifies that the field is a primitive `VariableValue` of the specified type. |
| `schema_id` | oneof `field_type`| [SchemaId](#schemaid) | Specifies that the field is of a specific `Schema`. |
| `inline_schema` | oneof `field_type`| [InlineSchema](#inlineschema) | Specifies that field must be a `Struct` conforming to the accompanying schema. |
<!-- end Fields -->
<!-- end HasFields -->



### Message `Struct` {#struct}

A Struct is a strongly-typed structure containing fields. The Struct is defined
according to the `Schema` object.


| Field | Label | Type | Description |
| ----- | ---- | ---- | ----------- |
| `schema_id` | | [SchemaId](#schemaid) | The id of the schema. |
| `struct` | | [InlineStruct](#inlinestruct) | The content of the Struct |
<!-- end Fields -->
<!-- end HasFields -->



### Message `StructField` {#structfield}

A StructField represents the value for a single field in a struct.


| Field | Label | Type | Description |
| ----- | ---- | ---- | ----------- |
| `primitive` | oneof `struct_value`| [VariableValue](#variablevalue) | The `value` of the field is an untyped primitive `VariableValue`. |
| `struct` | oneof `struct_value`| [InlineStruct](#inlinestruct) | The `value` of the field is a complex `Struct`. |
| `list` | oneof `struct_value`| [StructField.FieldList](#structfieldfieldlist) | The `value` of the field is a list of fields. |
<!-- end Fields -->
<!-- end HasFields -->



### Message `StructField.FieldList` {#structfieldfieldlist}

A FieldList is a sub-structure of a `Struct`


| Field | Label | Type | Description |
| ----- | ---- | ---- | ----------- |
| `fields` | repeated| [StructField](#structfield) | |
<!-- end Fields -->
<!-- end HasFields -->



### Message `Variable` {#variable}

A Variable is an instance of a variable assigned to a WfRun.
Expand Down Expand Up @@ -3082,6 +3200,7 @@ from a TaskRun, as the value of a WfRun's Variable, etc.
| `str` | oneof `value`| string | A string. |
| `int` | oneof `value`| int64 | The `INT` variable type is stored as a 64-bit integer. The `INT` can be cast to a `DOUBLE`. |
| `bytes` | oneof `value`| bytes | An arbitrary String of bytes. |
| `struct` | oneof `value`| [Struct](#struct) | A strongly-typed struct |
<!-- end Fields -->
<!-- end HasFields -->

Expand Down Expand Up @@ -3412,7 +3531,7 @@ JSON Path.
| Field | Label | Type | Description |
| ----- | ---- | ---- | ----------- |
| `field_path` | | string | Denotes the path in JSONPath format (according to the Java Jayway library) that has a field we should index. |
| `field_type` | | [VariableType](#variabletype) | Is the type of the field we are indexing. |
| `field_type` | | [PrimitiveType](#primitivetype) | Is the type of the field we are indexing. |
<!-- end Fields -->
<!-- end HasFields -->

Expand Down Expand Up @@ -3838,14 +3957,14 @@ world.

### Message `WorkflowEventDef` {#workfloweventdef}


A `WorkflowEventDef` defines the blueprint for a `WorkflowEvent`.


| Field | Label | Type | Description |
| ----- | ---- | ---- | ----------- |
| `id` | | [WorkflowEventDefId](#workfloweventdefid) | |
| `created_at` | | google.protobuf.Timestamp | |
| `type` | | [VariableType](#variabletype) | |
| `id` | | [WorkflowEventDefId](#workfloweventdefid) | The `id` of the `WorkflowEventDef`. |
| `created_at` | | google.protobuf.Timestamp | The timestamp at which the `WorkflowEventDef` was created. |
| `type` | | [PrimitiveType](#primitivetype) | The type of the content of the `WorkflowEvent`.<br/><br/>TODO: Support schemas |
<!-- end Fields -->
<!-- end HasFields -->

Expand Down Expand Up @@ -3949,25 +4068,7 @@ Metrics



### Enum TaskStatus {#taskstatus}
Status of a TaskRun.

| Name | Number | Description |
| ---- | ------ | ----------- |
| TASK_SCHEDULED | 0 | Scheduled in the Task Queue but not yet picked up by a Task Worker. |
| TASK_RUNNING | 1 | Picked up by a Task Worker, but not yet reported or timed out. |
| TASK_SUCCESS | 2 | Successfully completed. |
| TASK_FAILED | 3 | Task Worker reported a technical failure while attempting to execute the TaskRun |
| TASK_TIMEOUT | 4 | Task Worker did not report a result in time. |
| TASK_OUTPUT_SERIALIZING_ERROR | 5 | Task Worker reported that it was unable to serialize the output of the TaskRun. |
| TASK_INPUT_VAR_SUB_ERROR | 6 | Task Worker was unable to deserialize the input variables into appropriate language-specific objects to pass into the Task Function |
| TASK_EXCEPTION | 8 | Task Function business logic determined that there was a business exception. |
| TASK_PENDING | 9 | Refers to a TaskAttempt that is not yet scheduled. This happens when using retries with an ExponentialBackoffRetryPolicy: the TaskAttempt isn't supposed to be scheduled until it "matures", but it does already exist. |




### Enum VariableType {#variabletype}
### Enum PrimitiveType {#primitivetype}
Type of a Varaible in LittleHorse. Corresponds to the possible value type's of a
VariableValue.

Expand All @@ -3984,6 +4085,24 @@ The `JSON_OBJ` variable allows you to store complex objects in the JSON format.
| BYTES | 6 | The `BYTES` variable type allows you to store an arbitrary byte string. |




### Enum TaskStatus {#taskstatus}
Status of a TaskRun.

| Name | Number | Description |
| ---- | ------ | ----------- |
| TASK_SCHEDULED | 0 | Scheduled in the Task Queue but not yet picked up by a Task Worker. |
| TASK_RUNNING | 1 | Picked up by a Task Worker, but not yet reported or timed out. |
| TASK_SUCCESS | 2 | Successfully completed. |
| TASK_FAILED | 3 | Task Worker reported a technical failure while attempting to execute the TaskRun |
| TASK_TIMEOUT | 4 | Task Worker did not report a result in time. |
| TASK_OUTPUT_SERIALIZING_ERROR | 5 | Task Worker reported that it was unable to serialize the output of the TaskRun. |
| TASK_INPUT_VAR_SUB_ERROR | 6 | Task Worker was unable to deserialize the input variables into appropriate language-specific objects to pass into the Task Function |
| TASK_EXCEPTION | 8 | Task Function business logic determined that there was a business exception. |
| TASK_PENDING | 9 | Refers to a TaskAttempt that is not yet scheduled. This happens when using retries with an ExponentialBackoffRetryPolicy: the TaskAttempt isn't supposed to be scheduled until it "matures", but it does already exist. |


<!-- end Enums -->


Expand Down
2 changes: 1 addition & 1 deletion schemas/littlehorse/common_enums.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ enum MetricsWindowLength {

// Type of a Varaible in LittleHorse. Corresponds to the possible value type's of a
// VariableValue.
enum VariableType {
enum PrimitiveType {
// An object represented as a json string. <br/>
//
// The `JSON_OBJ` variable allows you to store complex objects in the JSON format.
Expand Down
10 changes: 8 additions & 2 deletions schemas/littlehorse/common_wfspec.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,13 @@ message VariableMutation {
// Declares a Variable.
message VariableDef {
// The Type of the variable.
VariableType type = 1;
oneof type {
// The primitive type of the variable.
PrimitiveType primitive = 1;

// The ID of the schema that the variable belongs to.
SchemaId schema = 5;
}

// The name of the variable.
string name = 2;
Expand All @@ -119,7 +125,7 @@ message VariableDef {
optional VariableValue default_value = 3;

// If true, the variable value will show as a masked string.
bool masked_value = 4;
bool is_masked = 4;
}

// Operator for comparing two values to create a boolean expression.
Expand Down
9 changes: 9 additions & 0 deletions schemas/littlehorse/object_id.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,15 @@ message WorkflowEventDefId {
string name = 1;
}

// ID for a Schema
message SchemaId {
// The name of the `Schema`.
string name = 1;

// The version of the `Schema`.
int32 version = 2;
}

// ID for a TaskWorkerGroup.
message TaskWorkerGroupId {
// TaskWorkerGroups are uniquely identified by their TaskDefId.
Expand Down
5 changes: 4 additions & 1 deletion schemas/littlehorse/service.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -338,7 +338,10 @@ message PutTaskDefRequest {
// EXPERIMENTAL: Creates a WorkflowEventDef
message PutWorkflowEventDefRequest {
string name = 1;
VariableType type = 2;
oneof type {
PrimitiveType primitive_type = 2;
SchemaId schema = 3;
}
}

// Creates a UserTaskDef.
Expand Down
2 changes: 1 addition & 1 deletion schemas/littlehorse/user_tasks.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ message UserTaskField {
string name = 1;

// The type of the output. Must be a primitive type (STR, BOOL, INT, DOUBLE).
VariableType type = 2;
PrimitiveType type = 2;

// Optional description which can be displayed by the User Task UI application.
// Does not affect WfRun execution.
Expand Down
Loading
Loading