Skip to content

Commit

Permalink
Merge branch 'main' into feat/otelcol-filelogreceiver
Browse files Browse the repository at this point in the history
  • Loading branch information
dehaansa committed Feb 15, 2025
2 parents c2b3c30 + 60596bb commit bef8864
Show file tree
Hide file tree
Showing 113 changed files with 5,129 additions and 701 deletions.
50 changes: 33 additions & 17 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,16 @@ internal API changes are not present.
Main (unreleased)
-----------------

### Breaking changes

- (_Experimental_) In `prometheus.write.queue` changed `parallelism` from attribute to a block to allow for dynamic scaling. (@mattdurham)

### Features

- - Add `otelcol.receiver.filelog` component to read otel log entries from files (@dehaansa)
- (_Public preview_) Add `otelcol.receiver.filelog` component to read otel log entries from files (@dehaansa)

- (_Public preview_) Add a `otelcol.processor.cumulativetodelta` component to convert metrics from
cumulative temporality to delta. (@madaraszg-tulip)

- (_Experimental_) Add a `stage.windowsevent` block in the `loki.process` component. This aims to replace the existing `stage.eventlogmessage`. (@wildum)

Expand All @@ -31,24 +38,29 @@ Main (unreleased)
- Add json format support for log export via faro receiver (@ravishankar15)

- (_Experimental_) Various changes to the experimental component `database_observability.mysql`:
- Always log `instance` label key (@cristiangreco)
- Improve parsing of truncated queries (@cristiangreco)
- Capture schema name for query samples (@cristiangreco)
- Fix handling of view table types when detecting schema (@matthewnolf)
- Fix error handling during result set iteration (@cristiangreco)
- Better support for table name parsing (@cristiangreco)
- Better error handling for components (@cristiangreco)
- Add namespace to `connection_info` metric (@cristiangreco)
- Added table columns parsing (@cristiagreco)
- Add enable/disable collector configurability to `database_observability.mysql`. This removes the `query_samples_enabled` argument, now configurable via enable/disable collector. (@fridgepoet)
- Refactor cache config in schema_table collector (@cristiangreco)
- Use labels for some indexed logs elements (@cristiangreco)

- Reduce CPU usage of `loki.source.windowsevent` by up to 85% by updating the bookmark file every 10 seconds instead of after every event and by
- `connection_info`: add namespace to the metric (@cristiangreco)
- `query_sample`: better support for table name parsing (@cristiangreco)
- `query_sample`: capture schema name for query samples (@cristiangreco)
- `query_sample`: fix error handling during result set iteration (@cristiangreco)
- `query_sample`: improve parsing of truncated queries (@cristiangreco)
- `schema_table`: add table columns parsing (@cristiagreco)
- `schema_table`: correctly quote schema and table name in SHOW CREATE (@cristiangreco)
- `schema_table`: fix handling of view table types when detecting schema (@matthewnolf)
- `schema_table`: refactor cache config in schema_table collector (@cristiangreco)
- Component: add enable/disable collector configurability to `database_observability.mysql`. This removes the `query_samples_enabled` argument, now configurable via enable/disable collector. (@fridgepoet)
- Component: always log `instance` label key (@cristiangreco)
- Component: better error handling for collectors (@cristiangreco)
- Component: use labels for some indexed logs elements (@cristiangreco)

- Reduce CPU usage of `loki.source.windowsevent` by up to 85% by updating the bookmark file every 10 seconds instead of after every event and by
optimizing the retrieval of the process name. (@wildum)

- Ensure consistent service_name label handling in `pyroscope.receive_http` to match Pyroscope's behavior. (@marcsanmi)

- Improved memory and CPU performance of Prometheus pipelines by changing the underlying implementation of targets (@thampiotr)

- Add `config_merge_strategy` in `prometheus.exporter.snmp` to optionally merge custom snmp config with embedded config instead of replacing. Useful for providing SNMP auths. (@v-zhuravlev)

### Bugfixes

- Fix log rotation for Windows in `loki.source.file` by refactoring the component to use the runner pkg. This should also reduce CPU consumption when tailing a lot of files in a dynamic environment. (@wildum)
Expand All @@ -63,20 +75,24 @@ Main (unreleased)

- Fixed an issue where `loki.process` would sometimes output live debugging entries out-of-order (@thampiotr)

- Fixed a bug where components could be evaluated concurrently without the full context during a config reload (@wildum)

### Other changes

- Upgrading to Prometheus v2.54.1. (@ptodev)
- `discovery.docker` has a new `match_first_network` attribute for matching the first network
- `discovery.docker` has a new `match_first_network` attribute for matching the first network
if the container has multiple networks defined, thus avoiding collecting duplicate targets.
- `discovery.ec2`, `discovery.kubernetes`, `discovery.openstack`, and `discovery.ovhcloud`
add extra `__meta_` labels.
- `prometheus.remote_write` supports Azure OAuth and Azure SDK authentication.
- `discovery.linode` has a new `region` attribute, as well as extra `__meta_` labels.
- A new `scrape_native_histograms` argument for `prometheus.scrape`.
This is enabled by default and can be used to explicitly disable native histogram support.
In previous versions of Alloy, native histogram support has also been enabled by default
In previous versions of Alloy, native histogram support has also been enabled by default
as long as `scrape_protocols` starts with `PrometheusProto`.

- Change the stability of the `remotecfg` feature from "public preview" to "generally available". (@erikbaranowski)

v1.6.1
-----------------

Expand Down
5 changes: 3 additions & 2 deletions docs/sources/introduction/supported-platforms.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,9 @@ The following operating systems and hardware architecture are supported.

## Linux

* Minimum version: kernel 2.6.32 or later
* Minimum version: kernel 4.x or later
* Architectures: AMD64, ARM64
* Within the Linux distribution lifecycle

## Windows

Expand All @@ -27,5 +28,5 @@ The following operating systems and hardware architecture are supported.

## FreeBSD

* Minimum version: FreeBSD 10 or later
* Within the FreeBSD lifecycle
* Architectures: AMD64
2 changes: 2 additions & 0 deletions docs/sources/reference/compatibility/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -310,6 +310,7 @@ The following components, grouped by namespace, _export_ OpenTelemetry `otelcol.
- [otelcol.exporter.syslog](../components/otelcol/otelcol.exporter.syslog)
- [otelcol.processor.attributes](../components/otelcol/otelcol.processor.attributes)
- [otelcol.processor.batch](../components/otelcol/otelcol.processor.batch)
- [otelcol.processor.cumulativetodelta](../components/otelcol/otelcol.processor.cumulativetodelta)
- [otelcol.processor.deltatocumulative](../components/otelcol/otelcol.processor.deltatocumulative)
- [otelcol.processor.discovery](../components/otelcol/otelcol.processor.discovery)
- [otelcol.processor.filter](../components/otelcol/otelcol.processor.filter)
Expand Down Expand Up @@ -348,6 +349,7 @@ The following components, grouped by namespace, _consume_ OpenTelemetry `otelcol
- [otelcol.connector.spanmetrics](../components/otelcol/otelcol.connector.spanmetrics)
- [otelcol.processor.attributes](../components/otelcol/otelcol.processor.attributes)
- [otelcol.processor.batch](../components/otelcol/otelcol.processor.batch)
- [otelcol.processor.cumulativetodelta](../components/otelcol/otelcol.processor.cumulativetodelta)
- [otelcol.processor.deltatocumulative](../components/otelcol/otelcol.processor.deltatocumulative)
- [otelcol.processor.discovery](../components/otelcol/otelcol.processor.discovery)
- [otelcol.processor.filter](../components/otelcol/otelcol.processor.filter)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ authentication > kerberos | [kerberos][] | Authenticates against
metadata | [metadata][] | Configures how to retrieve metadata from Kafka brokers. | no
metadata > retry | [retry][] | Configures how to retry metadata retrieval. | no
retry_on_failure | [retry_on_failure][] | Configures retry mechanism for failed requests. | no
queue | [queue][] | Configures batching of data before sending. | no
sending_queue | [sending_queue][] | Configures batching of data before sending. | no
producer | [producer][] | Kafka producer configuration, | no
debug_metrics | [debug_metrics][] | Configures the metrics which this component generates to monitor its state. | no

Expand All @@ -103,7 +103,7 @@ For example, `authentication > tls` refers to a `tls` block defined inside an `a
[metadata]: #metadata-block
[retry]: #retry-block
[retry_on_failure]: #retry_on_failure-block
[queue]: #queue-block
[sending_queue]: #sending_queue-block
[producer]: #producer-block
[debug_metrics]: #debug_metrics-block

Expand Down Expand Up @@ -149,9 +149,9 @@ The `retry_on_failure` block configures how failed requests to Kafka are retried

{{< docs/shared lookup="reference/components/otelcol-retry-block.md" source="alloy" version="<ALLOY_VERSION>" >}}

### queue block
### sending_queue block

The `queue` block configures an in-memory buffer of batches before data is sent to the gRPC server.
The `sending_queue` block configures an in-memory buffer of batches before data is sent to the gRPC server.

{{< docs/shared lookup="reference/components/otelcol-queue-block.md" source="alloy" version="<ALLOY_VERSION>" >}}

Expand Down Expand Up @@ -243,4 +243,4 @@ Connecting some components may not be sensible or components may require further
Refer to the linked documentation for more details.
{{< /admonition >}}

<!-- END GENERATED COMPATIBLE COMPONENTS -->
<!-- END GENERATED COMPATIBLE COMPONENTS -->
Original file line number Diff line number Diff line change
@@ -0,0 +1,169 @@
---
canonical: https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.processor.cumulativetodelta/
aliases:
- ../otelcol.processor.cumulativetodelta/ # /docs/alloy/latest/reference/otelcol.processor.cumulativetodelta/
description: Learn about otelcol.processor.cumulativetodelta
labels:
stage: public-preview
title: otelcol.processor.cumulativetodelta
---

# `otelcol.processor.cumulativetodelta`

{{< docs/shared lookup="stability/public_preview.md" source="alloy" version="<ALLOY_VERSION>" >}}

`otelcol.processor.cumulativetodelta` accepts metrics from other `otelcol` components and converts metrics with the cumulative temporality to delta.

{{< admonition type="note" >}}
`otelcol.processor.cumulativetodelta` is a wrapper over the upstream OpenTelemetry Collector `cumulativetodelta` processor.
Bug reports or feature requests will be redirected to the upstream repository, if necessary.
{{< /admonition >}}

You can specify multiple `otelcol.processor.cumulativetodelta` components by giving them different labels.

## Usage

```alloy
otelcol.processor.cumulativetodelta "<LABEL>" {
output {
metrics = [...]
}
}
```

## Arguments

You can use the following arguments with `otelcol.processor.cumulativetodelta`:

| Name | Type | Description | Default | Required |
| --------------- | ---------- | ---------------------------------------------------------------------- | -------- | -------- |
| `initial_value` | `string` | Handling of the first observed point for a given metric identity. | `"auto"` | no |
| `max_staleness` | `duration` | The total time a state entry will live past the time it was last seen. | `"0"` | no |

`otelcol.processor.cumulativetodelta` tracks incoming metric streams.
Sum and exponential histogram metrics with delta temporality are tracked and converted into cumulative temporality.

If a new sample hasn't been received since the duration specified by `max_staleness`, tracked streams are considered stale and dropped.
When set to `"0"`, the state is retained indefinitely.

The `initial_value` sets the handling of the first observed point for a given metric identity.
When the collector (re)starts, there's no record of how much of a given cumulative counter has already been converted to delta values.

* `"auto"` (default): Send the observed value if the start time is set AND the start time happens after the component started AND the start time is different from the timestamp.
This is suitable for gateway deployments. This heuristic is like `drop`, but it keeps values for newly started counters which could not have had previous observed values.
* `"keep"`: Send the observed value as the delta value. This is suitable for when the incoming metrics haven't been observed before. For example, when you are running the collector as a sidecar, the collector lifecycle is tied to the metric source.
* `"drop"`: Keep the observed value but don't send it. This is suitable for gateway deployments. It guarantees that all delta counts it produces haven't been observed before, but drops the values between the first two observations.

## Blocks

You can use the following blocks with `otelcol.processor.cumulativetodelta`:

| Block | Description | Required |
| -------------------------------- | -------------------------------------------------------------------------- | -------- |
| [`output`][output] | Configures where to send received telemetry data. | yes |
| [`debug_metrics`][debug_metrics] | Configures the metrics that this component generates to monitor its state. | no |
| [`exclude`][exclude] | Configures which metrics to not convert to delta. | no |
| [`include`][include] | Configures which metrics to convert to delta. | no |

If metric matches both `include` and `exclude`, exclude takes preference.
If neither `include` nor `exclude` are supplied, no filtering is applied.

[include]: #include
[exclude]: #exclude
[output]: #output
[debug_metrics]: #debug_metrics

### `output`

<span class="badge docs-labels__stage docs-labels__item">Required</span>

{{< docs/shared lookup="reference/components/output-block.md" source="alloy" version="<ALLOY_VERSION>" >}}

### `include`

The `include` block configures which metrics to convert to delta.

The following attributes are supported:

| Name | Type | Description | Default | Required |
| ------------ | -------------- | --------------------------------------- | ------- | -------- |
| `metrics` | `list(string)` | Names or patterns to convert to delta | | no |
| `match_type` | `string` | Match type to use, `strict` or `regexp` | | no |

If one of `metrics` or `match_type` is supplied, the other must be supplied too.

### `exclude`

The `exclude` block configures which metrics not to convert to delta.
`exclude` takes precedence over `include`

The following attributes are supported:

| Name | Type | Description | Default | Required |
| ------------ | -------------- | --------------------------------------- | ------- | -------- |
| `metrics` | `list(string)` | Names or patterns to convert to delta | | no |
| `match_type` | `string` | Match type to use, `strict` or `regexp` | | no |

If one of `metrics` or `match_type` is supplied, the other must be supplied too.

### `debug_metrics`

{{< docs/shared lookup="reference/components/otelcol-debug-metrics-block.md" source="alloy" version="<ALLOY_VERSION>" >}}

## Exported fields

The following fields are exported and can be referenced by other components:

| Name | Type | Description |
| ------- | ------------------ | ---------------------------------------------------------------- |
| `input` | `otelcol.Consumer` | A value that other components can use to send telemetry data to. |

`input` accepts `otelcol.Consumer` data for metrics.

## Component health

`otelcol.processor.cumulativetodelta` is only reported as unhealthy if given an invalid configuration.

## Debug information

`otelcol.processor.cumulativetodelta` doesn't expose any component-specific debug information.

## Example


This example converts cumulative temporality metrics to delta before sending it to [`otelcol.exporter.otlp`][otelcol.exporter.otlp] for further processing.

```alloy
otelcol.processor.cumulativetodelta "default" {
output {
metrics = [otelcol.exporter.otlp.production.input]
}
}
otelcol.exporter.otlp "production" {
client {
endpoint = sys.env("OTLP_SERVER_ENDPOINT")
}
}
```

[otelcol.exporter.otlp]: ../otelcol.exporter.otlp/

<!-- START GENERATED COMPATIBLE COMPONENTS -->

## Compatible components

`otelcol.processor.cumulativetodelta` can accept arguments from the following components:

- Components that export [OpenTelemetry `otelcol.Consumer`](../../../compatibility/#opentelemetry-otelcolconsumer-exporters)

`otelcol.processor.cumulativetodelta` has exports that can be consumed by the following components:

- Components that consume [OpenTelemetry `otelcol.Consumer`](../../../compatibility/#opentelemetry-otelcolconsumer-consumers)

{{< admonition type="note" >}}
Connecting some components may not be sensible or components may require further configuration to make the connection work correctly.
Refer to the linked documentation for more details.
{{< /admonition >}}

<!-- END GENERATED COMPATIBLE COMPONENTS -->
Loading

0 comments on commit bef8864

Please sign in to comment.