Skip to content

Commit

Permalink
Added complete descriptions. Diagrams missing. WIP.
Browse files Browse the repository at this point in the history
  • Loading branch information
@f-zimmer
f-zimmer committed Jul 3, 2024
1 parent 16519d1 commit 098998a
Show file tree
Hide file tree
Showing 8 changed files with 59 additions and 67 deletions.
Binary file modified docs/developer/data-plane-signaling/data-plane-signaling-state-machine.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions docs/developer/data-plane-signaling/data-plane-signaling-state-machine.puml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,10 @@ received -[dashed]-> started_push
started_push -[dashed]-> failed
started_push -[dashed]-> completed

failed --> failed
failed -[dashed]-> failed
failed --> notified : DP

completed --> completed
completed -[dashed]-> completed
completed --> notified : DP

notified --> [*]
Expand Down
122 changes: 57 additions & 65 deletions docs/developer/data-plane-signaling/data-plane-signaling.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,29 +39,19 @@ The data plane access token may be renewable based on the capabilities of its as

### The Signaling Protocol

All requests must support idempotent behavior. Data planes must therefore perform request de-duplication. After a data plane commits a request, it will return an ack to the control plane, which will transition the `TransferProcess` to its next state (e.g., `STARTED`, `SUSPENDED`, `TERMINATED`). If a successful ack is not received, the control plane will resend the request during a subsequent tick period.

##### 1. `START`

During the transfer process `STARTING` phase, a data plane will be selected by the default push and pull `DataFlowControllers`, which will then send a `DataFlowStartMessage` (renamed from `DataFlowRequest`) to the data plane.

The control plane (`DataFlowController`) will record which data plane was selected for the transfer process so that it can properly route subsequent, start, stop, and terminate requests.

For client pull transfers, the data plane will return a `DataAddress` and an access token.

If the data flow was previously `SUSPENDED`, the data plane may elect to return the same `DataAddress` or create a new one.
All requests must support idempotent behavior. Data planes must therefore perform request de-duplication. After a data plane commits a request, it will return an ack to the control plane, which will transition the `TransferProcess` to its next state. If a successful ack is not received, the control plane will resend the request during a subsequent tick period.

### States

The Data Plane Signaling protocol is represented by the following states:

- **STARTED:** Transits to started, in case of a PUSH transfer, if a START message was received and the transfer could be initiated. In case of PULL, if the EDR token was issued and provided to CP successfully.
- **RECEIVED:** Transits to received, in case of PUSH and a START message was received. Yet the transfer did not start so far, or failed to start and it will be retried.
- **COMPLETED:** In case of PUSH transfer, if the transfer completed successfully. Proceeds to try to notify the CP about outcome.
- **NOTIFIED:** Transits to notified, if the DP successfully sent an update about whether it completed or failed the transfer.
- **FAILED:** Transits into failed state if: 1. In case of PUSH transfer, doing the transfer (even exhausted retries) failed or it could not be initiated. or 2. if notifying the CP about a
- **SUSPENDED:** In case the CP sent a Suspend Message.
- **TERMINATED:** In case the CP sent a Terminate Message.
- **RECEIVED:** The Data Plane received a [DataFlowStartMessage](#DataFlowStartMessage) with a `FlowType` of `PUSH`. It acknowledges receiving the message by responding with a [DataFlowResponseMessage](#DataFlowResponseMessage).
- **STARTED:** In case of a `PULL` transfer, the EDR token was issued and provided to the Control Plane successfully using a [DataFlowResponseMessage](#DataFlowResponseMessage). In case of a `PUSH` transfer, the transfer process was initiated successfully.
- **COMPLETED:** A `PUSH` transfer was completed successfully. The Data Plane tries to notify the Control Plane about the outcome.
- **FAILED:** The Data Plane failed to complete the transfer process and continuously tries to notify the Control Plane about the issue.
- **NOTIFIED:** The Data Plane successfully notified the Control Plane about the outcome of the transfer, either being a success or failure.
- **SUSPENDED:** The Control Plane sent a [DataFlowSuspendMessage](#DataFlowSuspendMessage) to the Data Plane, thus suspending the transfer process.
- **TERMINATED:** The Control Plane sent a [DataFlowTerminateMessage](#DataFlowTerminateMessage) to the Data Plane, thus terminating the transfer process.

##### State Machine

Expand All @@ -71,46 +61,48 @@ The Data Plane Signaling state machine can be seen in the following diagram:

### Message Types

lorem ipsum
The Data Plane Signaling supports the following message types, in order to allow for the Control Plane to fulfill a `TransferProcess` using a suitable Data Plane. A Data Plane gets selected based on their capabilities by a Data Plane Framework Selector. Furthermore the Control Plane (`DataFlowController`) will record which Data Plane was selected for the transfer process so that it can properly route subsequent, start, stop, and terminate requests.

#### DataFlowStartMessage

| Message Type | Start |
|---------------------|--------------------------------------------------------------------|
| **Sent by** | Control Plane |
| **Resulting state** | `RECEIVED` or `STARTED` dependent on the `FlowType` (Push or Pull) |
| **Response** | DataFlowResponseMessage |
| **Schema** | JSON Schema |
| **Example** | [DataFlowStartMessage](./examples/DataFlowStartMessage.json) |
| **Diagram(s)** | |
| Message Type | Start |
|---------------------|--------------------------------------------------------------------------|
| **Sent by** | Control Plane |
| **Resulting state** | `RECEIVED` or `STARTED` dependent on the `FlowType` (Push or Pull) |
| **Response** | [DataFlowResponseMessage](#DataFlowResponseMessage) |
| **Schema** | JSON Schema |
| **Example** | [DataFlowStartMessage](message-types/examples/DataFlowStartMessage.json) |
| **Diagram(s)** | |

Initiates the data flow by signaling the Data Plane to start the transfer process. Dependent on the `FlowType` the following things will happen:
- `PUSH`: Data Plane acknowledges the request using a [DataFlowResponseMessage](#DataFlowResponseMessage) and tries to start the transfer process. Will notify Control Plane about the respective outcome after either success or failure.
- `PULL`: Data Plane returns callback address to pull the data from and an access token using a [DataFlowResponseMessage](#DataFlowResponseMessage).

Initiates the data flow by signaling the data plane to start the transfer process. Dependent on the `FlowType` the following things will happen:
- `PUSH`: Data plane initiates transfer process and tries to complete it. Will notify Control Plane about the respective outcome after completion or failure.
- `PULL`: Data plane returns callback address to pull the data from and an access token using a `DataFlowResponseMessage`.
Furthermore, if the data flow was previously `SUSPENDED`, the data plane may elect to return the prior used `DataAddress` or create a new one.

#### DataFlowSuspendMessage

| Message Type | Suspend |
|---------------------|------------------------------------------------------------------|
| **Sent by** | Control Plane |
| **Resulting state** | `SUSPENDED` |
| **Response** | - |
| **Schema** | JSON Schema |
| **Example** | [DataFlowSuspendMessage](./examples/DataFlowSuspendMessage.json) |
| **Diagram(s)** | |
| Message Type | Suspend |
|---------------------|------------------------------------------------------------------------------|
| **Sent by** | Control Plane |
| **Resulting state** | `SUSPENDED` |
| **Response** | - |
| **Schema** | JSON Schema |
| **Example** | [DataFlowSuspendMessage](message-types/examples/DataFlowSuspendMessage.json) |
| **Diagram(s)** | |

Leads to the suspension of the data flow, thus stopping it and invalidating the associated access token. If the data flow was previously `SUSPENDED`, the data plane may elect to return the same `DataAddress` or create a new one.
Leads to the suspension of the data flow, thus stopping it and invalidating the associated access token.

#### DataFlowTerminateMessage

| Message Type | Terminate |
|---------------------|----------------------------------------------------------------------|
| **Sent by** | Control Plane |
| **Resulting state** | `TERMINATED` |
| **Response** | - |
| **Schema** | JSON Schema |
| **Example** | [DataFlowTerminateMessage](./examples/DataFlowTerminateMessage.json) |
| **Diagram(s)** | |
| Message Type | Terminate |
|---------------------|----------------------------------------------------------------------------------|
| **Sent by** | Control Plane |
| **Resulting state** | `TERMINATED` |
| **Response** | - |
| **Schema** | JSON Schema |
| **Example** | [DataFlowTerminateMessage](message-types/examples/DataFlowTerminateMessage.json) |
| **Diagram(s)** | |

Leads to the termination of the data flow, thus stopping it and invalidating the associated access token.

Expand All @@ -120,39 +112,39 @@ The response messages are either sent as a direct response to an incoming messag

#### DataFlowResponseMessage

| Message Type | Response |
|---------------------|--------------------------------------------------------------------|
| **Sent by** | Data Plane |
| **Resulting state** | `STARTED` |
| **Schema** | JSON Schema |
| **Example** | [DataFlowResponseMessage](./examples/DataFlowResponseMessage.json) |
| **Diagram(s)** | |
| Message Type | Response |
|---------------------|--------------------------------------------------------------------------------|
| **Sent by** | Data Plane |
| **Resulting state** | - |
| **Schema** | JSON Schema |
| **Example** | [DataFlowResponseMessage](message-types/examples/DataFlowResponseMessage.json) |
| **Diagram(s)** | |

This message is a direct response message to a DataFlowStartMessage and acknowledges receiving during a `PULL` transfer.
This message is a direct response message to a [DataFlowStartMessage](#DataFlowStartMessage) and acknowledges receiving it. Will be used to provide access token and callback address in case of a `PULL` transfer.

#### TransferProcessCompleteRequest

| Message Type | Response |
| Message Type | Notification |
|---------------------|--------------------------|
| **Sent by** | Data Plane |
| **Resulting state** | `NOTIFIED` |
| **Schema** | *None due to empty body* |
| **Example** | - |
| **Diagram(s)** | - |

This message is not a direct response to an incoming message. It rather gets sent by the Data Plane if the already started transfer process was successfully completed. Successfully sending this message leads to the transfer state machine to proceed to the state `NOTIFIED`.
This message is not a direct response to an incoming message. It rather gets sent by the Data Plane if the already started transfer process was successfully completed. Successfully sending this message leads to the transfer state machine to proceed to the state `NOTIFIED` in case of a `PUSH` transfer.

#### TransferProcessFailRequest

| Message Type | Response |
|---------------------|--------------------------------------------------------------------------|
| **Sent by** | Data Plane |
| **Resulting state** | `NOTIFIED` |
| **Schema** | JSON Schema |
| **Example** | [TransferProcessFailRequest](./examples/TransferProcessFailRequest.json) |
| **Diagram(s)** | |
| Message Type | Notification |
|---------------------|--------------------------------------------------------------------------------------|
| **Sent by** | Data Plane |
| **Resulting state** | `NOTIFIED` |
| **Schema** | JSON Schema |
| **Example** | [TransferProcessFailRequest](message-types/examples/TransferProcessFailRequest.json) |
| **Diagram(s)** | |

This message is not a direct response to an incoming message. It rather gets sent by the Data Plane if the already started transfer process encountered an error, thus failed to be fulfilled. Successfully sending this message leads to the transfer state machine to proceed to the state `NOTIFIED`.
This message is not a direct response to an incoming message. It rather gets sent by the Data Plane if the already started transfer process encountered an error, thus failed to be fulfilled. Successfully sending this message leads to the transfer state machine to proceed to the state `NOTIFIED` in case of a `PUSH` transfer.

## II. Control Plane Refactoring

Expand Down
0 ...ing/examples/DataFlowResponseMessage.json → ...pes/examples/DataFlowResponseMessage.json
View file
File renamed without changes.
0 ...naling/examples/DataFlowStartMessage.json → ...-types/examples/DataFlowStartMessage.json
View file
File renamed without changes.
0 ...ling/examples/DataFlowSuspendMessage.json → ...ypes/examples/DataFlowSuspendMessage.json
View file
File renamed without changes.
0 ...ng/examples/DataFlowTerminateMessage.json → ...es/examples/DataFlowTerminateMessage.json
View file
File renamed without changes.
0 .../examples/TransferProcessFailRequest.json → .../examples/TransferProcessFailRequest.json
View file
File renamed without changes.

0 comments on commit 098998a

Please sign in to comment.