-
Notifications
You must be signed in to change notification settings - Fork 40
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
5 changed files
with
32 additions
and
26 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,11 +1,15 @@ | ||
# Arrangement | ||
|
||
Arrangement stores the state in the dataflow's process. Streams of update flows are sent to an arrangement, and the arrangement stores them for further querying and updating. | ||
Arrangement stores the state in the dataflow's process. It stores the streams of update flows for further querying and updating. | ||
|
||
The arrangement essentially stores key-value pairs with timestamps to mark their change time. | ||
|
||
Internally, the arrangement receives tuples like | ||
`((Key Row, Value Row), timestamp, diff)` and stores them in memory. One can query key-value pairs at a certain time using the `get(now: Timestamp, key: Row)` method, and retrieve the value for the given key at the specified time `now`. | ||
The arrangement also assumes that everything older than a certain time (also known as the low watermark) has already been ingested and does not keep a history for them. | ||
`((Key Row, Value Row), timestamp, diff)` and stores them in memory. One can query key-value pairs at a certain time using the `get(now: Timestamp, key: Row)` method. | ||
The arrangement also assumes that everything older than a certain time (also known as the low watermark) has already been ingested to the sink tables and does not keep a history for them. | ||
|
||
NOTE: The arrangement allows for the removal of keys by setting the `diff` to -1 in incoming tuples. Moreover, if a row has been previously added to the arrangement and the same key is inserted with a different value, the original value is overwritten with the new value. | ||
:::tip NOTE | ||
|
||
The arrangement allows for the removal of keys by setting the `diff` to -1 in incoming tuples. Moreover, if a row has been previously added to the arrangement and the same key is inserted with a different value, the original value is overwritten with the new value. | ||
|
||
::: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,11 +1,13 @@ | ||
# Arrangement | ||
|
||
Arrangement 存储数据流进程中的状态。更新流被发送到 Arrangement,Arrangement 会存储这些更新流,以便进一步查询和更新。 | ||
Arrangement 存储数据流进程中的状态,存储 flow 的更新流(stream)以供进一步查询和更新。 | ||
|
||
Arrangement 主要存储带有时间戳的键值对,以标记其更改时间。 | ||
Arrangement 本质上存储的是带有时间戳的键值对。 | ||
在内部,Arrangement 接收类似 `((Key Row, Value Row), timestamp, diff)` 的 tuple,并将其存储在内存中。 | ||
你可以使用 `get(now: Timestamp, key: Row)` 查询某个时间的键值对。 | ||
Arrangement 假定早于某个时间(也称为 Low Watermark)的所有内容都已被写入到 sink 表中,不会为其保留历史记录。 | ||
|
||
在内部,Arrangement 接收的元组包括 | ||
`((Key Row, Value Row), timestamp, diff)` 这样的元组并将其存储在内存中。人们可以使用 `get(now: Timestamp, key: Row)` 方法查询某个时间的键值对,并检索指定时间 `now` 的给定键值。 | ||
该安排还假定,所有早于一定时间(也称为 Low Watermark)的内容都已被摄取,因此不会为它们保留历史记录。 | ||
|
||
注意:Arrangement 允许通过将传入元组中的 `diff` 设置为 -1 来删除键。此外,如果之前已向 Arrangement 添加了一行,而插入的相同键值不同,则会用新值覆盖原值。 | ||
:::tip 注意 | ||
Arrangement 允许通过将传入 tuple 的 `diff` 设置为 -1 来删除键。 | ||
此外,如果已将行数据添加到 Arrangement 并且使用不同的值插入相同的键,则原始值将被新值覆盖。 | ||
::: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,17 +1,16 @@ | ||
# 概览 | ||
# 概述 | ||
|
||
## 简介 | ||
|
||
`Flownode` 为数据库提供了一种简单的流处理(称为 `flow`)能力。 | ||
`Flownode` 管理 `flow`,这些 `flow` 是从 `source` 接收数据并将数据发送到 `sink` 的任务。 | ||
|
||
Flownode “为数据库提供了一种简单的流式处理能力(称为 `flow`)。 | ||
Flownode “管理 `flow`,`flow` 是主动从作为数据源的表接收数据并将计算结果发送到结果表的任务。 | ||
|
||
在当前版本中,`Flownode` 仅支持 Standalone 模式。未来,我们将支持分布式模式。 | ||
在当前版本中,`Flownode` 仅在单机模式中支持,未来将支持分布式模式。 | ||
|
||
## 组件 | ||
|
||
一个 `Flownode` 包含流的流处理过程所需的所有组件。在此,我们列出了其中的重要部分: | ||
`Flownode` 包含了 flow 流式处理的所有组件,以下是关键部分: | ||
|
||
- `FlownodeManager`,用于接收从 ”前端 "转发的插入信息,并将结果发送回流的汇表。 | ||
- 一定数量的 `FlowWorker` 实例,每个实例在单独的线程中运行。目前,Standalone 模式下只有一个 `FlowWorker`,但将来可能会改变。 | ||
- `flow` 是一个主动从作为数据源的表接收数据,并向结果表发送数据的任务。它由 `FlownodeManager` 管理,并由 `FlowWorker` 运行。 | ||
- `FlownodeManager`:用于接收从 `Frontend` 转发的插入数据并将结果发送回 flow 的 sink 表。 | ||
- 一定数量的 `FlowWorker` 实例,每个实例在单独的线程中运行。当前在单机模式中只有一个 flow worker,但这可能会在未来发生变化。 | ||
- `Flow` 是一个主动从 `source` 接收数据并将数据发送到 `sink` 的任务。由 `FlownodeManager` 管理并由 `FlowWorker` 运行。 |