Skip to content

Commit

Permalink
feat: add some documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
gfyrag committed Aug 29, 2024
1 parent fa56ec0 commit 38e9fcf
Show file tree
Hide file tree
Showing 5 changed files with 2,123 additions and 3 deletions.
9 changes: 8 additions & 1 deletion ee/ingester/Earthfile
Original file line number Diff line number Diff line change
Expand Up @@ -140,4 +140,11 @@ generate-sdk:
-p packageName=ingesterclient
RUN rm -rf ./client/test
RUN ls -lisah
SAVE ARTIFACT ./client AS LOCAL pkg/client
SAVE ARTIFACT ./client AS LOCAL pkg/client

openapi-doc:
FROM node:22-alpine
RUN npm install -g widdershins
COPY api/openapi.yaml .
RUN widdershins openapi.yaml --search false --language_tabs 'http:HTTP' -o openapi.md --omitHeader --summary --code --httpsnippet
SAVE ARTIFACT openapi.md AS LOCAL api/openapi.md
32 changes: 31 additions & 1 deletion ee/ingester/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,31 @@
# Ingester

This component is dedicated to data ingestion on Formance modules.
It allows to create connectors (elasticsearch, clickhouse, ...), and create pipelines, plugging modules (ledger, payments...) on them.

# Modules requirements

The components must follow some guidelines to be able to be integrated with the ingester.
Each must :
* have a /logs endpoint returning past logs in a normalized format
> **_NOTE:_** The endpoint must also support the cursors.
* push logs in a topic named `<stack>.<module name>` on the stack broker

The log is a json object containing following properties:

| Name | Description | Type | Required |
|-|-----------------------------------|-|-|
| id | The log id | string or bigint | Yes |
| date | The log date | string / rfc3339 | Yes |
| type | The log type. Defined by modules. | string | Yes |
| payload | The log payload. | object | Yes |
| shard | The log shard. Can be used to split data of the modules using modules business rules | Yes |


> **_NOTE:_** The `shard` property allow to split logs of a modules.
> The format of the field is defined by the modules and can be empty if not required.
> However, it must not be greater than 256 characters.
## Pipeline state diagram

```mermaid
Expand All @@ -10,4 +36,8 @@ stateDiagram-v2
if_state --> FetchData: if state == INIT
FetchData --> ProcessEvents: All data initialized.\nSwitch to state READY
ProcessEvents --> FetchData: Reset triggered.\nSwitch to state INIT
```
```

## API Spec

See [Spec](./api/openapi.md)
Loading

0 comments on commit 38e9fcf

Please sign in to comment.