Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: add doc for loki ingester protocol #1329

Merged
merged 10 commits into from
Nov 27, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .typo.toml
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
[files]
extend-exclude = ["*/**/df-functions.md"]
extend-exclude = ["*/**/df-functions.md", "pnpm-lock.yaml"]
2 changes: 1 addition & 1 deletion docs/db-cloud-shared/clients/otlp-logs-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ GreptimeDB is an observability backend to consume OpenTelemetry Logs natively vi

To send OpenTelemetry Logs to GreptimeDB through OpenTelemetry SDK libraries, use the following information:

* URL: `https://<host>/v1/otlp/v1/logs`
* URL: `http{s}://<host>/v1/otlp/v1/logs`
* Headers:
* `X-Greptime-DB-Name`: `<dbname>`
* `Authorization`: `Basic` authentication, which is a Base64 encoded string of `<username>:<password>`. For more information, please refer to [Authentication](https://docs.greptime.com/user-guide/deployments/authentication/static/) and [HTTP API](https://docs.greptime.com/user-guide/protocols/http#authentication).
Expand Down
2 changes: 1 addition & 1 deletion docs/db-cloud-shared/clients/otlp-metrics-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ GreptimeDB is an observability backend to consume OpenTelemetry Metrics natively

To send OpenTelemetry Metrics to GreptimeDB through OpenTelemetry SDK libraries, use the following information:

* URL: `https://<host>/v1/otlp/v1/metrics`
* URL: `http{s}://<host>/v1/otlp/v1/metrics`
* Headers:
* `X-Greptime-DB-Name`: `<dbname>`
* `Authorization`: `Basic` authentication, which is a Base64 encoded string of `<username>:<password>`. For more information, please refer to [Authentication](https://docs.greptime.com/user-guide/deployments/authentication/static/) and [HTTP API](https://docs.greptime.com/user-guide/protocols/http#authentication)
Expand Down
123 changes: 123 additions & 0 deletions docs/user-guide/ingest-data/for-observerbility/loki.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
# Loki

## Usage

### API

To send Logs to GreptimeDB through Raw HTTP API, use the following information:

* URL: `http{s}://<host>/v1/loki/api/v1/push`
* Headers:
* `X-Greptime-DB-Name`: `<dbname>`
* `Authorization`: `Basic` authentication, which is a Base64 encoded string of `<username>:<password>`. For more information, please refer to [Authentication](https://docs.greptime.com/user-guide/deployments/authentication/static/) and [HTTP API](https://docs.greptime.com/user-guide/protocols/http#authentication).
* `X-Greptime-Log-Table-Name`: `<table_name>` (optional) - The table name to store the logs. If not provided, the default table name is `loki_logs`.

The request uses binary protobuf to encode the payload, The defined schema is the same as the [logproto.proto](https://github.com/grafana/loki/blob/main/pkg/logproto/logproto.proto).

### Example Code

paomian marked this conversation as resolved.
Show resolved Hide resolved
[Grafana Alloy](https://grafana.com/docs/alloy/latest/) is a vendor-neutral distribution of the OpenTelemetry (OTel) Collector. Alloy uniquely combines the very best OSS observability signals in the community.

It suplies a Loki exporter that can be used to send logs to GreptimeDB. Here is an example configuration:

```hcl
loki.source.file "greptime" {
targets = [
{__path__ = "/tmp/foo.txt"},
]
forward_to = [loki.write.greptime_loki.receiver]
}
loki.write "greptime_loki" {
endpoint {
url = "${GREPTIME_SCHEME:=http}://${GREPTIME_HOST:=greptimedb}:${GREPTIME_PORT:=4000}/v1/loki/api/v1/push"
headers = {
"x-greptime-db-name" = "${GREPTIME_DB:=public}",
"x-greptime-log-table-name" = "${GREPTIME_LOG_TABLE_NAME:=loki_demo_logs}",
}
}
external_labels = {
"job" = "greptime"
"from" = "alloy"
}
}
```

We listen to the file `/tmp/foo.txt` and send the logs to GreptimeDB. The logs are stored in the table `loki_demo_logs` with the external labels `job` and `from`.

For more information, please refer to the [Grafana Alloy loki.write documentation](https://grafana.com/docs/alloy/latest/reference/components/loki/loki.write/).

You can run the following command to check the data in the table:

```sql
SELECT * FROM loki_demo_logs;
+----------------------------+------------------------+--------------+-------+----------+
| greptime_timestamp | line | filename | from | job |
+----------------------------+------------------------+--------------+-------+----------+
| 2024-11-25 11:02:31.256251 | Greptime is very cool! | /tmp/foo.txt | alloy | greptime |
+----------------------------+------------------------+--------------+-------+----------+
1 row in set (0.01 sec)
```

## Data Model

The Loki logs data model is mapped to the GreptimeDB data model according to the following rules:

Default table schema without external labels:

```sql
DESC loki_demo_logs;
+--------------------+---------------------+------+------+---------+---------------+
| Column | Type | Key | Null | Default | Semantic Type |
+--------------------+---------------------+------+------+---------+---------------+
| greptime_timestamp | TimestampNanosecond | PRI | NO | | TIMESTAMP |
| line | String | | YES | | FIELD |
+--------------------+---------------------+------+------+---------+---------------+
5 rows in set (0.00 sec)
```

- greptime_timestamp: The timestamp of the log.
- line: The log message.

if you specify the external labels, we will add them as tags to the table schema. like `job` and `from` in the above example.
You can't specify tags manually, all labels are treated as tags and string type.
Please do not attempt to pre-create the table using SQL to specify tag columns, as this will cause a type mismatch and write failure.

### Example

The following is an example of the table schema:

```sql
DESC loki_demo_logs;
+--------------------+---------------------+------+------+---------+---------------+
| Column | Type | Key | Null | Default | Semantic Type |
+--------------------+---------------------+------+------+---------+---------------+
| greptime_timestamp | TimestampNanosecond | PRI | NO | | TIMESTAMP |
| line | String | | YES | | FIELD |
| filename | String | PRI | YES | | TAG |
| from | String | PRI | YES | | TAG |
| job | String | PRI | YES | | TAG |
+--------------------+---------------------+------+------+---------+---------------+
5 rows in set (0.00 sec)
```

```sql
SHOW CREATE TABLE loki_demo_logs\G
*************************** 1. row ***************************
Table: loki_demo_logs
Create Table: CREATE TABLE IF NOT EXISTS `loki_demo_logs` (
`greptime_timestamp` TIMESTAMP(9) NOT NULL,
`line` STRING NULL,
`filename` STRING NULL,
`from` STRING NULL,
`job` STRING NULL,
TIME INDEX (`greptime_timestamp`),
PRIMARY KEY (`filename`, `from`, `job`)
)

ENGINE=mito
WITH(
append_mode = 'true'
)
1 row in set (0.00 sec)
```
Original file line number Diff line number Diff line change
Expand Up @@ -137,8 +137,8 @@ otelcol.exporter.otlphttp "greptimedb_logs" {
endpoint = "${GREPTIME_SCHEME:=http}://${GREPTIME_HOST:=greptimedb}:${GREPTIME_PORT:=4000}/v1/otlp/"
headers = {
"X-Greptime-DB-Name" = "${GREPTIME_DB:=public}",
"x-greptime-log-table-name" = "demo_logs",
"x-greptime-log-extract-keys" = "filename,log.file.name,loki.attribute.labels",
"X-Greptime-Log-Table-Name" = "demo_logs",
"X-Greptime-Gog-Extract-Keys" = "filename,log.file.name,loki.attribute.labels",
}
auth = otelcol.auth.basic.credentials.handler
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,4 @@ GreptimeDB integrates seamlessly with leading observability tools to provide a c
- [Vector](vector.md): Use GreptimeDB as a sink for Vector, ideal for complex data pipelines and diverse data sources.
- [OpenTelemetry](opentelemetry.md): Collect and export telemetry data to GreptimeDB for detailed observability insights.
- [InfluxDB Line Protocol](influxdb-line-protocol.md): A widely-used protocol for time-series data, facilitating migration from InfluxDB to GreptimeDB. The Telegraf integration method is also introduced in this document.
- [Loki](loki.md): A widely-used log write protocol, facilitating migration from Loki to GreptimeDB. The Alloy integration method is also introduced in this document.
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
# Loki

## 使用方法

### API

要通过原始 HTTP API 将日志发送到 GreptimeDB,请使用以下信息:

* URL: `http{s}://<host>/v1/loki/api/v1/push`
* Headers:
* `X-Greptime-DB-Name`: `<dbname>`
* `Authorization`: `Basic` 认证,这是一个 Base64 编码的 `<username>:<password>` 字符串。更多信息,请参考 [认证](https://docs.greptime.com/user-guide/deployments/authentication/static/) 和 [HTTP API](https://docs.greptime.com/user-guide/protocols/http#authentication)。
* `X-Greptime-Log-Table-Name`: `<table_name>`(可选)- 存储日志的表名。如果未提供,默认表名为 `loki_logs`。

请求使用二进制 protobuf 编码负载,定义的格式与 [logproto.proto](https://github.com/grafana/loki/blob/main/pkg/logproto/logproto.proto) 相同。

### 示例代码

[Grafana Alloy](https://grafana.com/docs/alloy/latest/) 是一个供应商中立的 OpenTelemetry (OTel) Collector 发行版。Alloy 独特地结合了社区中最好的开源可观测性信号。

它提供了一个 Loki 导出器,可以用来将日志发送到 GreptimeDB。以下是一个配置示例:

```hcl
loki.source.file "greptime" {
targets = [
{__path__ = "/tmp/foo.txt"},
]
forward_to = [loki.write.greptime_loki.receiver]
}

loki.write "greptime_loki" {
endpoint {
url = "${GREPTIME_SCHEME:=http}://${GREPTIME_HOST:=greptimedb}:${GREPTIME_PORT:=4000}/v1/loki/api/v1/push"
headers = {
"x-greptime-db-name" = "${GREPTIME_DB:=public}",
"x-greptime-log-table-name" = "${GREPTIME_LOG_TABLE_NAME:=loki_demo_logs}",
}
}
external_labels = {
"job" = "greptime"
"from" = "alloy"
}
}
```

我们监听文件 `/tmp/foo.txt` 并将日志发送到 GreptimeDB。日志存储在表 `loki_demo_logs` 中,并带有 label `job` 和 `from`。

更多信息,请参考 [Grafana Alloy loki.write 文档](https://grafana.com/docs/alloy/latest/reference/components/loki/loki.write/)。

你可以运行以下命令来检查表中的数据:

```sql
SELECT * FROM loki_demo_logs;
+----------------------------+------------------------+--------------+-------+----------+
| greptime_timestamp | line | filename | from | job |
+----------------------------+------------------------+--------------+-------+----------+
| 2024-11-25 11:02:31.256251 | Greptime is very cool! | /tmp/foo.txt | alloy | greptime |
+----------------------------+------------------------+--------------+-------+----------+
1 row in set (0.01 sec)
```

## 数据模型

Loki 日志数据模型根据以下规则映射到 GreptimeDB 数据模型:

没有 label 的默认表结构:

```sql
DESC loki_demo_logs;
+--------------------+---------------------+------+------+---------+---------------+
| Column | Type | Key | Null | Default | Semantic Type |
+--------------------+---------------------+------+------+---------+---------------+
| greptime_timestamp | TimestampNanosecond | PRI | NO | | TIMESTAMP |
| line | String | | YES | | FIELD |
+--------------------+---------------------+------+------+---------+---------------+
5 rows in set (0.00 sec)
```

- greptime_timestamp: 日志的时间戳。
- line: 日志消息。

如果你指定了外部 label,我们会将它们添加为表结构中的 tag。例如上面的 `job` 和 `from`。
在这种写入方式下不能手动指定,所有 label 都被视为 tag 并且类型为字符串。请不要尝试使用 SQL 提前创建表来指定 tag 列,这会导致类型不匹配而写入失败。

### 示例

以下是表结构的示例:

```sql
DESC loki_demo_logs;
+--------------------+---------------------+------+------+---------+---------------+
| Column | Type | Key | Null | Default | Semantic Type |
+--------------------+---------------------+------+------+---------+---------------+
| greptime_timestamp | TimestampNanosecond | PRI | NO | | TIMESTAMP |
| line | String | | YES | | FIELD |
| filename | String | PRI | YES | | TAG |
| from | String | PRI | YES | | TAG |
| job | String | PRI | YES | | TAG |
+--------------------+---------------------+------+------+---------+---------------+
5 rows in set (0.00 sec)
```

```sql
SHOW CREATE TABLE loki_demo_logs\G
*************************** 1. row ***************************
Table: loki_demo_logs
Create Table: CREATE TABLE IF NOT EXISTS `loki_demo_logs` (
`greptime_timestamp` TIMESTAMP(9) NOT NULL,
`line` STRING NULL,
`filename` STRING NULL,
`from` STRING NULL,
`job` STRING NULL,
TIME INDEX (`greptime_timestamp`),
PRIMARY KEY (`filename`, `from`, `job`)
)

ENGINE=mito
WITH(
append_mode = 'true'
)
1 row in set (0.00 sec)
```
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,5 @@ GreptimeDB 与领先的可观测性工具无缝集成,为你提供系统健康
- [Vector](vector.md):将 GreptimeDB 用作 Vector 的接收端,适用于复杂的数据流水线和多样化的数据源。
- [OpenTelemetry](opentelemetry.md):将 telemetry 数据收集并导出到 GreptimeDB,以获取详细的可观测性洞察。
- [InfluxDB Line Protocol](influxdb-line-protocol.md):一种广泛使用的时间序列数据协议,便于从 InfluxDB 迁移到 GreptimeDB。该文档同样介绍了 Telegraf 的集成方式。
- [Loki](loki.md):一种广泛使用的日志写入协议,便于从 Loki 迁移到 GreptimeDB。本文档还介绍了 Alloy 集成方法。

Loading