diff --git a/ee/ingester/Earthfile b/ee/ingester/Earthfile index 2406b9353a..12495930ee 100644 --- a/ee/ingester/Earthfile +++ b/ee/ingester/Earthfile @@ -140,4 +140,11 @@ generate-sdk: -p packageName=ingesterclient RUN rm -rf ./client/test RUN ls -lisah - SAVE ARTIFACT ./client AS LOCAL pkg/client \ No newline at end of file + SAVE ARTIFACT ./client AS LOCAL pkg/client + +openapi-doc: + FROM node:22-alpine + RUN npm install -g widdershins + COPY api/openapi.yaml . + RUN widdershins openapi.yaml --search false --language_tabs 'http:HTTP' -o openapi.md --omitHeader --summary --code --httpsnippet + SAVE ARTIFACT openapi.md AS LOCAL api/openapi.md \ No newline at end of file diff --git a/ee/ingester/README.md b/ee/ingester/README.md index bd30f8dc86..3de37ed8b6 100644 --- a/ee/ingester/README.md +++ b/ee/ingester/README.md @@ -1,5 +1,31 @@ # Ingester +This component is dedicated to data ingestion on Formance modules. +It allows to create connectors (elasticsearch, clickhouse, ...), and create pipelines, plugging modules (ledger, payments...) on them. + +# Modules requirements + +The components must follow some guidelines to be able to be integrated with the ingester. +Each must : +* have a /logs endpoint returning past logs in a normalized format + > **_NOTE:_** The endpoint must also support the cursors. +* push logs in a topic named `.` on the stack broker + +The log is a json object containing following properties: + +| Name | Description | Type | Required | +|-|-----------------------------------|-|-| +| id | The log id | string or bigint | Yes | +| date | The log date | string / rfc3339 | Yes | +| type | The log type. Defined by modules. | string | Yes | +| payload | The log payload. | object | Yes | +| shard | The log shard. Can be used to split data of the modules using modules business rules | Yes | + + +> **_NOTE:_** The `shard` property allow to split logs of a modules. +> The format of the field is defined by the modules and can be empty if not required. +> However, it must not be greater than 256 characters. + ## Pipeline state diagram ```mermaid @@ -10,4 +36,8 @@ stateDiagram-v2 if_state --> FetchData: if state == INIT FetchData --> ProcessEvents: All data initialized.\nSwitch to state READY ProcessEvents --> FetchData: Reset triggered.\nSwitch to state INIT -``` \ No newline at end of file +``` + +## API Spec + +See [Spec](./api/openapi.md) \ No newline at end of file diff --git a/ee/ingester/api/openapi.md b/ee/ingester/api/openapi.md new file mode 100644 index 0000000000..db21a8e21d --- /dev/null +++ b/ee/ingester/api/openapi.md @@ -0,0 +1,962 @@ + + +

Formance Simple ingester Service API v0.1.0

+ +> Scroll down for code samples, example requests and responses. Select a language for code samples from the tabs above or the mobile navigation menu. + +Base URLs: + +* http://localhost:8080 + +

Connectors

+ +## List connectors + + + +`GET /connectors` + +> Example responses + +> 200 Response + +```json +{ + "cursor": { + "pageSize": 15, + "hasMore": false, + "previous": "YXVsdCBhbmQgYSBtYXhpbXVtIG1heF9yZXN1bHRzLol=", + "next": "", + "data": [ + { + "id": "string", + "createdAt": "2019-08-24T14:15:22Z" + } + ] + } +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|200|[OK](https://tools.ietf.org/html/rfc7231#section-6.3.1)|Connectors list|Inline| +|default|Default|General error|[Error](#schemaerror)| + +

Response Schema

+ +Status Code **200** + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|» cursor|any|false|none|none| + +*allOf* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|[Cursor](#schemacursor)|false|none|none| +|»»» pageSize|integer(int64)|true|none|none| +|»»» hasMore|boolean|true|none|none| +|»»» previous|string|false|none|none| +|»»» next|string|false|none|none| + +*and* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|object|false|none|none| +|»»» data|[allOf]|false|none|none| + +*allOf* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»»»» *anonymous*|object|false|none|none| +|»»»»» driver|string|true|none|none| +|»»»»» config|object|true|none|none| + +*and* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»»»» *anonymous*|object|false|none|none| +|»»»»» id|string|true|none|none| +|»»»»» createdAt|string(date-time)|true|none|none| + + + +## Create connector + + + +`POST /connectors` + +> Body parameter + +```json +{ + "driver": "string", + "config": {} +} +``` + +

Parameters

+ +|Name|In|Type|Required|Description| +|---|---|---|---|---| +|body|body|[ConnectorConfiguration](#schemaconnectorconfiguration)|false|none| + +> Example responses + +> 201 Response + +```json +{ + "data": { + "id": "string", + "createdAt": "2019-08-24T14:15:22Z" + } +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|201|[Created](https://tools.ietf.org/html/rfc7231#section-6.3.2)|Created pipeline|Inline| +|default|Default|General error|[Error](#schemaerror)| + +

Response Schema

+ +Status Code **201** + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|» data|any|true|none|none| + +*allOf* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|object|false|none|none| +|»»» driver|string|true|none|none| +|»»» config|object|true|none|none| + +*and* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|object|false|none|none| +|»»» id|string|true|none|none| +|»»» createdAt|string(date-time)|true|none|none| + + + +## Get connector state + + + +`GET /connectors/{connectorID}` + +

Parameters

+ +|Name|In|Type|Required|Description| +|---|---|---|---|---| +|connectorID|path|string|true|The connector id| + +> Example responses + +> 200 Response + +```json +{ + "data": { + "id": "string", + "createdAt": "2019-08-24T14:15:22Z" + } +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|200|[OK](https://tools.ietf.org/html/rfc7231#section-6.3.1)|Connector information|Inline| +|default|Default|General error|[Error](#schemaerror)| + +

Response Schema

+ +Status Code **200** + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|» data|any|true|none|none| + +*allOf* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|object|false|none|none| +|»»» driver|string|true|none|none| +|»»» config|object|true|none|none| + +*and* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|object|false|none|none| +|»»» id|string|true|none|none| +|»»» createdAt|string(date-time)|true|none|none| + + + +## Delete connector + + + +`DELETE /connectors/{connectorID}` + +

Parameters

+ +|Name|In|Type|Required|Description| +|---|---|---|---|---| +|connectorID|path|string|true|The connector id| + +> Example responses + +> default Response + +```json +{ + "errorCode": "string", + "errorMessage": "string" +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|204|[No Content](https://tools.ietf.org/html/rfc7231#section-6.3.5)|Connector deleted|None| +|default|Default|General error|[Error](#schemaerror)| + + + +

Pipelines

+ +## List pipelines + + + +`GET /pipelines` + +> Example responses + +> 200 Response + +```json +{ + "cursor": { + "pageSize": 15, + "hasMore": false, + "previous": "YXVsdCBhbmQgYSBtYXhpbXVtIG1heF9yZXN1bHRzLol=", + "next": "", + "data": [ + { + "id": "string", + "state": { + "label": "STOP", + "cursor": "string", + "previousState": {} + }, + "createdAt": "2019-08-24T14:15:22Z" + } + ] + } +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|200|[OK](https://tools.ietf.org/html/rfc7231#section-6.3.1)|Pipelines list|Inline| +|default|Default|General error|[Error](#schemaerror)| + +

Response Schema

+ +Status Code **200** + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|» cursor|any|false|none|none| + +*allOf* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|[Cursor](#schemacursor)|false|none|none| +|»»» pageSize|integer(int64)|true|none|none| +|»»» hasMore|boolean|true|none|none| +|»»» previous|string|false|none|none| +|»»» next|string|false|none|none| + +*and* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|object|false|none|none| +|»»» data|[allOf]|false|none|none| + +*allOf* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»»»» *anonymous*|object|false|none|none| +|»»»»» module|string|true|none|none| +|»»»»» connectorID|string|true|none|none| + +*and* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»»»» *anonymous*|object|false|none|none| +|»»»»» id|string|true|none|none| +|»»»»» state|[State](#schemastate)|true|none|none| +|»»»»»» label|string|true|none|none| +|»»»»»» cursor|string|false|none|none| +|»»»»»» previousState|[State](#schemastate)|false|none|none| +|»»»»» createdAt|string(date-time)|true|none|none| + +#### Enumerated Values + +|Property|Value| +|---|---| +|label|STOP| +|label|PAUSE| +|label|INIT| +|label|READY| + + + +## Create pipeline + + + +`POST /pipelines` + +> Body parameter + +```json +{ + "module": "string", + "connectorID": "string" +} +``` + +

Parameters

+ +|Name|In|Type|Required|Description| +|---|---|---|---|---| +|body|body|[CreatePipelineRequest](#schemacreatepipelinerequest)|false|none| + +> Example responses + +> 201 Response + +```json +{ + "data": { + "id": "string", + "state": { + "label": "STOP", + "cursor": "string", + "previousState": {} + }, + "createdAt": "2019-08-24T14:15:22Z" + } +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|201|[Created](https://tools.ietf.org/html/rfc7231#section-6.3.2)|Created ipeline|Inline| +|default|Default|General error|[Error](#schemaerror)| + +

Response Schema

+ +Status Code **201** + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|» data|any|true|none|none| + +*allOf* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|object|false|none|none| +|»»» module|string|true|none|none| +|»»» connectorID|string|true|none|none| + +*and* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|object|false|none|none| +|»»» id|string|true|none|none| +|»»» state|[State](#schemastate)|true|none|none| +|»»»» label|string|true|none|none| +|»»»» cursor|string|false|none|none| +|»»»» previousState|[State](#schemastate)|false|none|none| +|»»» createdAt|string(date-time)|true|none|none| + +#### Enumerated Values + +|Property|Value| +|---|---| +|label|STOP| +|label|PAUSE| +|label|INIT| +|label|READY| + + + +## Get pipeline state + + + +`GET /pipelines/{pipelineID}` + +

Parameters

+ +|Name|In|Type|Required|Description| +|---|---|---|---|---| +|pipelineID|path|string|true|The pipeline id| + +> Example responses + +> 200 Response + +```json +{ + "data": { + "id": "string", + "state": { + "label": "STOP", + "cursor": "string", + "previousState": {} + }, + "createdAt": "2019-08-24T14:15:22Z" + } +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|200|[OK](https://tools.ietf.org/html/rfc7231#section-6.3.1)|Pipeline information|Inline| +|default|Default|General error|[Error](#schemaerror)| + +

Response Schema

+ +Status Code **200** + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|» data|any|true|none|none| + +*allOf* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|object|false|none|none| +|»»» module|string|true|none|none| +|»»» connectorID|string|true|none|none| + +*and* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|object|false|none|none| +|»»» id|string|true|none|none| +|»»» state|[State](#schemastate)|true|none|none| +|»»»» label|string|true|none|none| +|»»»» cursor|string|false|none|none| +|»»»» previousState|[State](#schemastate)|false|none|none| +|»»» createdAt|string(date-time)|true|none|none| + +#### Enumerated Values + +|Property|Value| +|---|---| +|label|STOP| +|label|PAUSE| +|label|INIT| +|label|READY| + + + +## Delete pipeline + + + +`DELETE /pipelines/{pipelineID}` + +

Parameters

+ +|Name|In|Type|Required|Description| +|---|---|---|---|---| +|pipelineID|path|string|true|The pipeline id| + +> Example responses + +> default Response + +```json +{ + "errorCode": "string", + "errorMessage": "string" +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|204|[No Content](https://tools.ietf.org/html/rfc7231#section-6.3.5)|Pipeline deleted|None| +|default|Default|General error|[Error](#schemaerror)| + + + +## Reset pipeline + + + +`POST /pipelines/{pipelineID}/reset` + +

Parameters

+ +|Name|In|Type|Required|Description| +|---|---|---|---|---| +|pipelineID|path|string|true|The pipeline id| + +> Example responses + +> default Response + +```json +{ + "errorCode": "string", + "errorMessage": "string" +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|202|[Accepted](https://tools.ietf.org/html/rfc7231#section-6.3.3)|Pipeline reset|None| +|default|Default|General error|[Error](#schemaerror)| + + + +## Pause pipeline + + + +`POST /pipelines/{pipelineID}/pause` + +

Parameters

+ +|Name|In|Type|Required|Description| +|---|---|---|---|---| +|pipelineID|path|string|true|The pipeline id| + +> Example responses + +> default Response + +```json +{ + "errorCode": "string", + "errorMessage": "string" +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|202|[Accepted](https://tools.ietf.org/html/rfc7231#section-6.3.3)|Pipeline pause|None| +|default|Default|General error|[Error](#schemaerror)| + + + +## Resume pipeline + + + +`POST /pipelines/{pipelineID}/resume` + +

Parameters

+ +|Name|In|Type|Required|Description| +|---|---|---|---|---| +|pipelineID|path|string|true|The pipeline id| + +> Example responses + +> default Response + +```json +{ + "errorCode": "string", + "errorMessage": "string" +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|202|[Accepted](https://tools.ietf.org/html/rfc7231#section-6.3.3)|Pipeline resumed|None| +|default|Default|General error|[Error](#schemaerror)| + + + +## Start pipeline + + + +`POST /pipelines/{pipelineID}/start` + +

Parameters

+ +|Name|In|Type|Required|Description| +|---|---|---|---|---| +|pipelineID|path|string|true|The pipeline id| + +> Example responses + +> default Response + +```json +{ + "errorCode": "string", + "errorMessage": "string" +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|202|[Accepted](https://tools.ietf.org/html/rfc7231#section-6.3.3)|Pipeline started|None| +|default|Default|General error|[Error](#schemaerror)| + + + +## Stop pipeline + + + +`POST /pipelines/{pipelineID}/stop` + +

Parameters

+ +|Name|In|Type|Required|Description| +|---|---|---|---|---| +|pipelineID|path|string|true|The pipeline id| + +> Example responses + +> default Response + +```json +{ + "errorCode": "string", + "errorMessage": "string" +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|202|[Accepted](https://tools.ietf.org/html/rfc7231#section-6.3.3)|Pipeline stopped|None| +|default|Default|General error|[Error](#schemaerror)| + + + +# Schemas + +

CreatePipelineRequest

+ + + + + + +```json +{ + "module": "string", + "connectorID": "string" +} + +``` + +### Properties + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|module|string|true|none|none| +|connectorID|string|true|none|none| + +

CreateConnectorRequest

+ + + + + + +```json +{ + "driver": "string", + "config": {} +} + +``` + +### Properties + +*None* + +

PipelineConfiguration

+ + + + + + +```json +{ + "module": "string", + "connectorID": "string" +} + +``` + +### Properties + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|module|string|true|none|none| +|connectorID|string|true|none|none| + +

ConnectorConfiguration

+ + + + + + +```json +{ + "driver": "string", + "config": {} +} + +``` + +### Properties + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|driver|string|true|none|none| +|config|object|true|none|none| + +

Connector

+ + + + + + +```json +{ + "id": "string", + "createdAt": "2019-08-24T14:15:22Z" +} + +``` + +### Properties + +allOf + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|*anonymous*|[ConnectorConfiguration](#schemaconnectorconfiguration)|false|none|none| + +and + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|*anonymous*|object|false|none|none| +|» id|string|true|none|none| +|» createdAt|string(date-time)|true|none|none| + +

Pipeline

+ + + + + + +```json +{ + "id": "string", + "state": { + "label": "STOP", + "cursor": "string", + "previousState": {} + }, + "createdAt": "2019-08-24T14:15:22Z" +} + +``` + +### Properties + +allOf + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|*anonymous*|[PipelineConfiguration](#schemapipelineconfiguration)|false|none|none| + +and + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|*anonymous*|object|false|none|none| +|» id|string|true|none|none| +|» state|[State](#schemastate)|true|none|none| +|» createdAt|string(date-time)|true|none|none| + +

State

+ + + + + + +```json +{ + "label": "STOP", + "cursor": "string", + "previousState": { + "label": "STOP", + "cursor": "string", + "previousState": {} + } +} + +``` + +### Properties + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|label|string|true|none|none| +|cursor|string|false|none|none| +|previousState|[State](#schemastate)|false|none|none| + +#### Enumerated Values + +|Property|Value| +|---|---| +|label|STOP| +|label|PAUSE| +|label|INIT| +|label|READY| + +

Error

+ + + + + + +```json +{ + "errorCode": "string", + "errorMessage": "string" +} + +``` + +### Properties + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|errorCode|string|true|none|none| +|errorMessage|string|true|none|none| + +

Cursor

+ + + + + + +```json +{ + "pageSize": 15, + "hasMore": false, + "previous": "YXVsdCBhbmQgYSBtYXhpbXVtIG1heF9yZXN1bHRzLol=", + "next": "" +} + +``` + +### Properties + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|pageSize|integer(int64)|true|none|none| +|hasMore|boolean|true|none|none| +|previous|string|false|none|none| +|next|string|false|none|none| + diff --git a/ee/ingester/api/openapi.yaml b/ee/ingester/api/openapi.yaml index a399b61bf4..5789e7f3d0 100644 --- a/ee/ingester/api/openapi.yaml +++ b/ee/ingester/api/openapi.yaml @@ -242,7 +242,7 @@ components: - data CreateConnectorResponse: - description: Created ipeline + description: Created pipeline content: application/json: schema: diff --git a/ee/ingester/generated.md b/ee/ingester/generated.md new file mode 100644 index 0000000000..697138822e --- /dev/null +++ b/ee/ingester/generated.md @@ -0,0 +1,1121 @@ +--- +title: Formance Simple ingester Service API v0.1.0 +language_tabs: + - http: HTTP +language_clients: + - http: "" +toc_footers: [] +includes: [] +search: false +highlight_theme: darkula +headingLevel: 2 + +--- + + + +

Formance Simple ingester Service API v0.1.0

+ +> Scroll down for code samples, example requests and responses. Select a language for code samples from the tabs above or the mobile navigation menu. + +Base URLs: + +* http://localhost:8080 + +

Connectors

+ +## listConnectors + + + +> Code samples + +```http +GET http://localhost:8080/connectors HTTP/1.1 +Host: localhost:8080 +Accept: application/json + +``` + +`GET /connectors` + +*List connectors* + +> Example responses + +> 200 Response + +```json +{ + "cursor": { + "pageSize": 15, + "hasMore": false, + "previous": "YXVsdCBhbmQgYSBtYXhpbXVtIG1heF9yZXN1bHRzLol=", + "next": "", + "data": [ + { + "id": "string", + "createdAt": "2019-08-24T14:15:22Z" + } + ] + } +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|200|[OK](https://tools.ietf.org/html/rfc7231#section-6.3.1)|Connectors list|Inline| +|default|Default|General error|[Error](#schemaerror)| + +

Response Schema

+ +Status Code **200** + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|» cursor|any|false|none|none| + +*allOf* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|[Cursor](#schemacursor)|false|none|none| +|»»» pageSize|integer(int64)|true|none|none| +|»»» hasMore|boolean|true|none|none| +|»»» previous|string|false|none|none| +|»»» next|string|false|none|none| + +*and* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|object|false|none|none| +|»»» data|[allOf]|false|none|none| + +*allOf* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»»»» *anonymous*|object|false|none|none| +|»»»»» driver|string|true|none|none| +|»»»»» config|object|true|none|none| + +*and* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»»»» *anonymous*|object|false|none|none| +|»»»»» id|string|true|none|none| +|»»»»» createdAt|string(date-time)|true|none|none| + + + +## createConnector + + + +> Code samples + +```http +POST http://localhost:8080/connectors HTTP/1.1 +Host: localhost:8080 +Content-Type: application/json +Accept: application/json + +``` + +`POST /connectors` + +*Create connector* + +> Body parameter + +```json +{ + "driver": "string", + "config": {} +} +``` + +

Parameters

+ +|Name|In|Type|Required|Description| +|---|---|---|---|---| +|body|body|[ConnectorConfiguration](#schemaconnectorconfiguration)|false|none| + +> Example responses + +> 201 Response + +```json +{ + "data": { + "id": "string", + "createdAt": "2019-08-24T14:15:22Z" + } +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|201|[Created](https://tools.ietf.org/html/rfc7231#section-6.3.2)|Created pipeline|Inline| +|default|Default|General error|[Error](#schemaerror)| + +

Response Schema

+ +Status Code **201** + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|» data|any|true|none|none| + +*allOf* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|object|false|none|none| +|»»» driver|string|true|none|none| +|»»» config|object|true|none|none| + +*and* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|object|false|none|none| +|»»» id|string|true|none|none| +|»»» createdAt|string(date-time)|true|none|none| + + + +## getConnectorState + + + +> Code samples + +```http +GET http://localhost:8080/connectors/{connectorID} HTTP/1.1 +Host: localhost:8080 +Accept: application/json + +``` + +`GET /connectors/{connectorID}` + +*Get connector state* + +

Parameters

+ +|Name|In|Type|Required|Description| +|---|---|---|---|---| +|connectorID|path|string|true|The connector id| + +> Example responses + +> 200 Response + +```json +{ + "data": { + "id": "string", + "createdAt": "2019-08-24T14:15:22Z" + } +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|200|[OK](https://tools.ietf.org/html/rfc7231#section-6.3.1)|Connector information|Inline| +|default|Default|General error|[Error](#schemaerror)| + +

Response Schema

+ +Status Code **200** + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|» data|any|true|none|none| + +*allOf* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|object|false|none|none| +|»»» driver|string|true|none|none| +|»»» config|object|true|none|none| + +*and* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|object|false|none|none| +|»»» id|string|true|none|none| +|»»» createdAt|string(date-time)|true|none|none| + + + +## deleteConnector + + + +> Code samples + +```http +DELETE http://localhost:8080/connectors/{connectorID} HTTP/1.1 +Host: localhost:8080 +Accept: application/json + +``` + +`DELETE /connectors/{connectorID}` + +*Delete connector* + +

Parameters

+ +|Name|In|Type|Required|Description| +|---|---|---|---|---| +|connectorID|path|string|true|The connector id| + +> Example responses + +> default Response + +```json +{ + "errorCode": "string", + "errorMessage": "string" +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|204|[No Content](https://tools.ietf.org/html/rfc7231#section-6.3.5)|Connector deleted|None| +|default|Default|General error|[Error](#schemaerror)| + + + +

Pipelines

+ +## listPipelines + + + +> Code samples + +```http +GET http://localhost:8080/pipelines HTTP/1.1 +Host: localhost:8080 +Accept: application/json + +``` + +`GET /pipelines` + +*List pipelines* + +> Example responses + +> 200 Response + +```json +{ + "cursor": { + "pageSize": 15, + "hasMore": false, + "previous": "YXVsdCBhbmQgYSBtYXhpbXVtIG1heF9yZXN1bHRzLol=", + "next": "", + "data": [ + { + "id": "string", + "state": { + "label": "STOP", + "cursor": "string", + "previousState": {} + }, + "createdAt": "2019-08-24T14:15:22Z" + } + ] + } +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|200|[OK](https://tools.ietf.org/html/rfc7231#section-6.3.1)|Pipelines list|Inline| +|default|Default|General error|[Error](#schemaerror)| + +

Response Schema

+ +Status Code **200** + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|» cursor|any|false|none|none| + +*allOf* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|[Cursor](#schemacursor)|false|none|none| +|»»» pageSize|integer(int64)|true|none|none| +|»»» hasMore|boolean|true|none|none| +|»»» previous|string|false|none|none| +|»»» next|string|false|none|none| + +*and* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|object|false|none|none| +|»»» data|[allOf]|false|none|none| + +*allOf* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»»»» *anonymous*|object|false|none|none| +|»»»»» module|string|true|none|none| +|»»»»» connectorID|string|true|none|none| + +*and* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»»»» *anonymous*|object|false|none|none| +|»»»»» id|string|true|none|none| +|»»»»» state|[State](#schemastate)|true|none|none| +|»»»»»» label|string|true|none|none| +|»»»»»» cursor|string|false|none|none| +|»»»»»» previousState|[State](#schemastate)|false|none|none| +|»»»»» createdAt|string(date-time)|true|none|none| + +#### Enumerated Values + +|Property|Value| +|---|---| +|label|STOP| +|label|PAUSE| +|label|INIT| +|label|READY| + + + +## createPipeline + + + +> Code samples + +```http +POST http://localhost:8080/pipelines HTTP/1.1 +Host: localhost:8080 +Content-Type: application/json +Accept: application/json + +``` + +`POST /pipelines` + +*Create pipeline* + +> Body parameter + +```json +{ + "module": "string", + "connectorID": "string" +} +``` + +

Parameters

+ +|Name|In|Type|Required|Description| +|---|---|---|---|---| +|body|body|[CreatePipelineRequest](#schemacreatepipelinerequest)|false|none| + +> Example responses + +> 201 Response + +```json +{ + "data": { + "id": "string", + "state": { + "label": "STOP", + "cursor": "string", + "previousState": {} + }, + "createdAt": "2019-08-24T14:15:22Z" + } +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|201|[Created](https://tools.ietf.org/html/rfc7231#section-6.3.2)|Created ipeline|Inline| +|default|Default|General error|[Error](#schemaerror)| + +

Response Schema

+ +Status Code **201** + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|» data|any|true|none|none| + +*allOf* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|object|false|none|none| +|»»» module|string|true|none|none| +|»»» connectorID|string|true|none|none| + +*and* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|object|false|none|none| +|»»» id|string|true|none|none| +|»»» state|[State](#schemastate)|true|none|none| +|»»»» label|string|true|none|none| +|»»»» cursor|string|false|none|none| +|»»»» previousState|[State](#schemastate)|false|none|none| +|»»» createdAt|string(date-time)|true|none|none| + +#### Enumerated Values + +|Property|Value| +|---|---| +|label|STOP| +|label|PAUSE| +|label|INIT| +|label|READY| + + + +## getPipelineState + + + +> Code samples + +```http +GET http://localhost:8080/pipelines/{pipelineID} HTTP/1.1 +Host: localhost:8080 +Accept: application/json + +``` + +`GET /pipelines/{pipelineID}` + +*Get pipeline state* + +

Parameters

+ +|Name|In|Type|Required|Description| +|---|---|---|---|---| +|pipelineID|path|string|true|The pipeline id| + +> Example responses + +> 200 Response + +```json +{ + "data": { + "id": "string", + "state": { + "label": "STOP", + "cursor": "string", + "previousState": {} + }, + "createdAt": "2019-08-24T14:15:22Z" + } +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|200|[OK](https://tools.ietf.org/html/rfc7231#section-6.3.1)|Pipeline information|Inline| +|default|Default|General error|[Error](#schemaerror)| + +

Response Schema

+ +Status Code **200** + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|» data|any|true|none|none| + +*allOf* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|object|false|none|none| +|»»» module|string|true|none|none| +|»»» connectorID|string|true|none|none| + +*and* + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|»» *anonymous*|object|false|none|none| +|»»» id|string|true|none|none| +|»»» state|[State](#schemastate)|true|none|none| +|»»»» label|string|true|none|none| +|»»»» cursor|string|false|none|none| +|»»»» previousState|[State](#schemastate)|false|none|none| +|»»» createdAt|string(date-time)|true|none|none| + +#### Enumerated Values + +|Property|Value| +|---|---| +|label|STOP| +|label|PAUSE| +|label|INIT| +|label|READY| + + + +## deletePipeline + + + +> Code samples + +```http +DELETE http://localhost:8080/pipelines/{pipelineID} HTTP/1.1 +Host: localhost:8080 +Accept: application/json + +``` + +`DELETE /pipelines/{pipelineID}` + +*Delete pipeline* + +

Parameters

+ +|Name|In|Type|Required|Description| +|---|---|---|---|---| +|pipelineID|path|string|true|The pipeline id| + +> Example responses + +> default Response + +```json +{ + "errorCode": "string", + "errorMessage": "string" +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|204|[No Content](https://tools.ietf.org/html/rfc7231#section-6.3.5)|Pipeline deleted|None| +|default|Default|General error|[Error](#schemaerror)| + + + +## resetPipeline + + + +> Code samples + +```http +POST http://localhost:8080/pipelines/{pipelineID}/reset HTTP/1.1 +Host: localhost:8080 +Accept: application/json + +``` + +`POST /pipelines/{pipelineID}/reset` + +*Reset pipeline* + +

Parameters

+ +|Name|In|Type|Required|Description| +|---|---|---|---|---| +|pipelineID|path|string|true|The pipeline id| + +> Example responses + +> default Response + +```json +{ + "errorCode": "string", + "errorMessage": "string" +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|202|[Accepted](https://tools.ietf.org/html/rfc7231#section-6.3.3)|Pipeline reset|None| +|default|Default|General error|[Error](#schemaerror)| + + + +## pausePipeline + + + +> Code samples + +```http +POST http://localhost:8080/pipelines/{pipelineID}/pause HTTP/1.1 +Host: localhost:8080 +Accept: application/json + +``` + +`POST /pipelines/{pipelineID}/pause` + +*Pause pipeline* + +

Parameters

+ +|Name|In|Type|Required|Description| +|---|---|---|---|---| +|pipelineID|path|string|true|The pipeline id| + +> Example responses + +> default Response + +```json +{ + "errorCode": "string", + "errorMessage": "string" +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|202|[Accepted](https://tools.ietf.org/html/rfc7231#section-6.3.3)|Pipeline pause|None| +|default|Default|General error|[Error](#schemaerror)| + + + +## resumePipeline + + + +> Code samples + +```http +POST http://localhost:8080/pipelines/{pipelineID}/resume HTTP/1.1 +Host: localhost:8080 +Accept: application/json + +``` + +`POST /pipelines/{pipelineID}/resume` + +*Resume pipeline* + +

Parameters

+ +|Name|In|Type|Required|Description| +|---|---|---|---|---| +|pipelineID|path|string|true|The pipeline id| + +> Example responses + +> default Response + +```json +{ + "errorCode": "string", + "errorMessage": "string" +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|202|[Accepted](https://tools.ietf.org/html/rfc7231#section-6.3.3)|Pipeline resumed|None| +|default|Default|General error|[Error](#schemaerror)| + + + +## startPipeline + + + +> Code samples + +```http +POST http://localhost:8080/pipelines/{pipelineID}/start HTTP/1.1 +Host: localhost:8080 +Accept: application/json + +``` + +`POST /pipelines/{pipelineID}/start` + +*Start pipeline* + +

Parameters

+ +|Name|In|Type|Required|Description| +|---|---|---|---|---| +|pipelineID|path|string|true|The pipeline id| + +> Example responses + +> default Response + +```json +{ + "errorCode": "string", + "errorMessage": "string" +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|202|[Accepted](https://tools.ietf.org/html/rfc7231#section-6.3.3)|Pipeline started|None| +|default|Default|General error|[Error](#schemaerror)| + + + +## stopPipeline + + + +> Code samples + +```http +POST http://localhost:8080/pipelines/{pipelineID}/stop HTTP/1.1 +Host: localhost:8080 +Accept: application/json + +``` + +`POST /pipelines/{pipelineID}/stop` + +*Stop pipeline* + +

Parameters

+ +|Name|In|Type|Required|Description| +|---|---|---|---|---| +|pipelineID|path|string|true|The pipeline id| + +> Example responses + +> default Response + +```json +{ + "errorCode": "string", + "errorMessage": "string" +} +``` + +

Responses

+ +|Status|Meaning|Description|Schema| +|---|---|---|---| +|202|[Accepted](https://tools.ietf.org/html/rfc7231#section-6.3.3)|Pipeline stopped|None| +|default|Default|General error|[Error](#schemaerror)| + + + +# Schemas + +

CreatePipelineRequest

+ + + + + + +```json +{ + "module": "string", + "connectorID": "string" +} + +``` + +### Properties + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|module|string|true|none|none| +|connectorID|string|true|none|none| + +

CreateConnectorRequest

+ + + + + + +```json +{ + "driver": "string", + "config": {} +} + +``` + +### Properties + +*None* + +

PipelineConfiguration

+ + + + + + +```json +{ + "module": "string", + "connectorID": "string" +} + +``` + +### Properties + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|module|string|true|none|none| +|connectorID|string|true|none|none| + +

ConnectorConfiguration

+ + + + + + +```json +{ + "driver": "string", + "config": {} +} + +``` + +### Properties + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|driver|string|true|none|none| +|config|object|true|none|none| + +

Connector

+ + + + + + +```json +{ + "id": "string", + "createdAt": "2019-08-24T14:15:22Z" +} + +``` + +### Properties + +allOf + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|*anonymous*|[ConnectorConfiguration](#schemaconnectorconfiguration)|false|none|none| + +and + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|*anonymous*|object|false|none|none| +|» id|string|true|none|none| +|» createdAt|string(date-time)|true|none|none| + +

Pipeline

+ + + + + + +```json +{ + "id": "string", + "state": { + "label": "STOP", + "cursor": "string", + "previousState": {} + }, + "createdAt": "2019-08-24T14:15:22Z" +} + +``` + +### Properties + +allOf + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|*anonymous*|[PipelineConfiguration](#schemapipelineconfiguration)|false|none|none| + +and + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|*anonymous*|object|false|none|none| +|» id|string|true|none|none| +|» state|[State](#schemastate)|true|none|none| +|» createdAt|string(date-time)|true|none|none| + +

State

+ + + + + + +```json +{ + "label": "STOP", + "cursor": "string", + "previousState": { + "label": "STOP", + "cursor": "string", + "previousState": {} + } +} + +``` + +### Properties + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|label|string|true|none|none| +|cursor|string|false|none|none| +|previousState|[State](#schemastate)|false|none|none| + +#### Enumerated Values + +|Property|Value| +|---|---| +|label|STOP| +|label|PAUSE| +|label|INIT| +|label|READY| + +

Error

+ + + + + + +```json +{ + "errorCode": "string", + "errorMessage": "string" +} + +``` + +### Properties + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|errorCode|string|true|none|none| +|errorMessage|string|true|none|none| + +

Cursor

+ + + + + + +```json +{ + "pageSize": 15, + "hasMore": false, + "previous": "YXVsdCBhbmQgYSBtYXhpbXVtIG1heF9yZXN1bHRzLol=", + "next": "" +} + +``` + +### Properties + +|Name|Type|Required|Restrictions|Description| +|---|---|---|---|---| +|pageSize|integer(int64)|true|none|none| +|hasMore|boolean|true|none|none| +|previous|string|false|none|none| +|next|string|false|none|none| +