diff --git a/pkg/ottl/LANGUAGE.md b/pkg/ottl/LANGUAGE.md index c5e09d58ff7b..ee4545228a74 100644 --- a/pkg/ottl/LANGUAGE.md +++ b/pkg/ottl/LANGUAGE.md @@ -1,6 +1,6 @@ ## Grammar -The OTTL grammar includes function invocations, Values and Boolean Expressions. These parts all fit into a Statement, which is the basis of execution in the OTTL. +OTTL grammar includes function invocations, Values and Boolean Expressions. These parts all fit into a Statement, which is the basis of execution in OTTL. ### Design principles @@ -18,9 +18,9 @@ An Editor is made up of 2 parts: - a string identifier. The string identifier must start with a lowercase letter. - zero or more Values (comma separated) surrounded by parentheses (`()`). -**The OTTL has no built-in Editors.** +**OTTL has no built-in Editors.** Users must supply a map between string identifiers and Editor implementations. -The OTTL will use this map to determine which implementation to call when executing a Statement. +OTTL will use this map to determine which implementation to call when executing a Statement. See [ottlfuncs](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/ottlfuncs#editors) for pre-made, usable Editors. ### Converters @@ -32,9 +32,9 @@ Converters are made up of 3 parts: - zero or more Values (comma separated) surrounded by parentheses (`()`). - a combination of zero or more a string key (`["key"]`) or int key (`[0]`) -**The OTTL has no built-in Converters.** +**OTTL has no built-in Converters.** Users must include Converters in the same map that Editors are supplied. -The OTTL will use this map and reflection to generate Converters that can then be invoked by the user. +OTTL will use this map and reflection to generate Converters that can then be invoked by the user. See [ottlfuncs](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/ottlfuncs#converters) for pre-made, usable Converters. When keys are supplied the value returned by the Converter will be indexed by the keys in order. @@ -89,7 +89,7 @@ For slice parameters, the following types are supported: - `string` - `float64` - `int64` -- `uint8`. Byte slice literals are parsed as byte slices by the OTTL. +- `uint8`. Byte slice literals are parsed as byte slices by OTTL. - `Getter` To make a parameter optional, use the `Optional` type, which takes a type argument for the underlying @@ -118,7 +118,7 @@ Values are passed as function parameters or are used in a Boolean Expression. Va ### Paths -A Path Value is a reference to a telemetry field. Paths are made up of lowercase identifiers, dots (`.`), and square brackets combined with a string key (`["key"]`) or int key (`[0]`). **The interpretation of a Path is NOT implemented by the OTTL.** Instead, the user must provide a `PathExpressionParser` that the OTTL can use to interpret paths. As a result, how the Path parts are used is up to the user. However, it is recommended that the parts be used like so: +A Path Value is a reference to a telemetry field. Paths are made up of lowercase identifiers, dots (`.`), and square brackets combined with a string key (`["key"]`) or int key (`[0]`). **The interpretation of a Path is NOT implemented by OTTL.** Instead, the user must provide a `PathExpressionParser` that OTTL can use to interpret paths. As a result, how the Path parts are used is up to the user. However, it is recommended that the parts be used like so: - Identifiers are used to map to a telemetry field. - Dots (`.`) are used to separate nested fields. @@ -171,8 +171,8 @@ Example Map Values: Literals are literal interpretations of the Value into a Go value. Accepted literals are: - Strings. Strings are represented as literals by surrounding the string in double quotes (`""`). -- Ints. Ints are represented by any digit, optionally prepended by plus (`+`) or minus (`-`). Internally the OTTL represents all ints as `int64` -- Floats. Floats are represented by digits separated by a dot (`.`), optionally prepended by plus (`+`) or minus (`-`). The leading digit is optional. Internally the OTTL represents all Floats as `float64`. +- Ints. Ints are represented by any digit, optionally prepended by plus (`+`) or minus (`-`). Internally OTTL represents all ints as `int64` +- Floats. Floats are represented by digits separated by a dot (`.`), optionally prepended by plus (`+`) or minus (`-`). The leading digit is optional. Internally OTTL represents all Floats as `float64`. - Bools. Bools are represented by the exact strings `true` and `false`. - Nil. Nil is represented by the exact string `nil`. - Byte slices. Byte slices are represented via a hex string prefaced with `0x` @@ -187,7 +187,7 @@ Example Literals ### Enums -Enums are uppercase identifiers that get interpreted during parsing and converted to an `int64`. **The interpretation of an Enum is NOT implemented by the OTTL.** Instead, the user must provide a `EnumParser` that the OTTL can use to interpret the Enum. The `EnumParser` returns an `int64` instead of a function, which means that the Enum's numeric value is retrieved during parsing instead of during execution. +Enums are uppercase identifiers that get interpreted during parsing and converted to an `int64`. **The interpretation of an Enum is NOT implemented by OTTL.** Instead, the user must provide a `EnumParser` that OTTL can use to interpret the Enum. The `EnumParser` returns an `int64` instead of a function, which means that the Enum's numeric value is retrieved during parsing instead of during execution. Within the grammar Enums are always used as `int64`. As a result, the Enum's symbol can be used as if it is an Int value. @@ -292,7 +292,7 @@ Examples: ## Accessing signal telemetry -Access to signal telemetry is provided to OTTL functions through a `TransformContext` that is created by the user and passed during statement evaluation. To allow functions to operate on the `TransformContext`, the OTTL provides `Getter`, `Setter`, and `GetSetter` interfaces. +Access to signal telemetry is provided to OTTL functions through a `TransformContext` that is created by the user and passed during statement evaluation. To allow functions to operate on the `TransformContext`, OTTL provides `Getter`, `Setter`, and `GetSetter` interfaces. ### Getters and Setters @@ -305,6 +305,6 @@ Getters allow for reading the following types of data. See the respective sectio It is possible to update the Value in a telemetry field using a Setter. For read and write access, the `GetSetter` interface extends both interfaces. -## Logging inside a OTTL function +## Logging inside an OTTL function -To emit logs inside a OTTL function, add a parameter of type [`component.TelemetrySettings`](https://pkg.go.dev/go.opentelemetry.io/collector/component#TelemetrySettings) to the function signature. The OTTL will then inject the TelemetrySettings that were passed to `NewParser` into the function. TelemetrySettings can be used to emit logs. +To emit logs inside an OTTL function, add a parameter of type [`component.TelemetrySettings`](https://pkg.go.dev/go.opentelemetry.io/collector/component#TelemetrySettings) to the function signature. OTTL will then inject the TelemetrySettings that were passed to `NewParser` into the function. TelemetrySettings can be used to emit logs. diff --git a/pkg/ottl/README.md b/pkg/ottl/README.md index 50ec9d9449ea..b7d7a125f45f 100644 --- a/pkg/ottl/README.md +++ b/pkg/ottl/README.md @@ -9,12 +9,12 @@ [alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha -The OpenTelemetry Transformation Language is a language for transforming open telemetry data based on the [OpenTelemetry Collector Processing Exploration](https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/rfcs/processing.md). +The OpenTelemetry Transformation Language (OTTL) is a small, domain-specific programming language intended to process data with OpenTelemetry-native concepts and constructs. -This package reads in OTTL statements and converts them to invokable functions/booleans based on the OTTL's grammar. +This package implements everything necessary to use OTTL in a Collector component or in another user-facing system. - [Getting Started](#getting-started) -- [Examples](#examples) +- [Where to use OTTL](#where-to-use-ottl) - [Troubleshooting](#troubleshooting) - [Resources](#resources) @@ -24,117 +24,26 @@ If you're looking to write OTTL statements for a component's configuration check See [OTTL Functions](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/ottlfuncs#ottl-functions) for a list of functions available for use in the OTTL statements of most components. -OTTL Contexts define how you access the fields on a piece of telemetry. See the table to find the exact list of available fields: +OTTL Contexts define how you access the fields on a given telemetry item. See the table to find the exact list of available fields: | Telemetry | OTTL Context | |-------------------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| `Resource` | [Resource](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottlresource/README.md) | -| `Instrumentation Scope` | [Instrumentation Scope](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottlscope/README.md) | -| `Span` | [Span](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottlspan/README.md) | -| `Span Event` | [SpanEvent](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottlspanevent/README.md) | -| `Metric` | [Metric](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottlmetric/README.md) | -| `Datapoint` | [DataPoint](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottldatapoint/README.md) | -| `Log` | [Log](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottllog/README.md) | +| `Resource` | [Resource](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/contexts/ottlresource/README.md) | +| `Instrumentation Scope` | [Instrumentation Scope](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/contexts/ottlscope/README.md) | +| `Span` | [Span](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/contexts/ottlspan/README.md) | +| `Span Event` | [SpanEvent](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/contexts/ottlspanevent/README.md) | +| `Metric` | [Metric](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/contexts/ottlmetric/README.md) | +| `Datapoint` | [DataPoint](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/contexts/ottldatapoint/README.md) | +| `Log` | [Log](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/contexts/ottllog/README.md) | -### Component Creators +To understand what OTTL offers as a language, check out [OTTL's grammar doc](./LANGUAGE.md). -If you're looking to use OTTL in your component, check out [the OTTL grammar](./LANGUAGE.md). +## Where to use OTTL -## Examples - -These examples contain a SQL-like declarative language. Applied statements interact with only one signal, but statements can be declared across multiple signals. Functions used in examples are indicative of what could be useful. - -### Remove a forbidden attribute - -``` -traces: - delete(attributes["http.request.header.authorization"]) -metrics: - delete(attributes["http.request.header.authorization"]) -logs: - delete(attributes["http.request.header.authorization"]) -``` - -### Remove all attributes except for some - -``` -traces: - keep_keys(attributes, ["http.method", "http.status_code"]) -metrics: - keep_keys(attributes, ["http.method", "http.status_code"]) -logs: - keep_keys(attributes, ["http.method", "http.status_code"]) -``` - -### Reduce cardinality of an attribute - -``` -traces: - replace_match(attributes["http.target"], "/user/*/list/*", "/user/{userId}/list/{listId}") -``` - -### Reduce cardinality of a span name - -``` -traces: - replace_match(name, "GET /user/*/list/*", "GET /user/{userId}/list/{listId}") -``` - -### Reduce cardinality of any matching attribute - -``` -traces: - replace_all_matches(attributes, "/user/*/list/*", "/user/{userId}/list/{listId}") -``` - -### Decrease the size of the telemetry payload - -``` -traces: - delete(resource.attributes["process.command_line"]) -metrics: - delete(resource.attributes["process.command_line"]) -logs: - delete(resource.attributes["process.command_line"]) -``` - -### Attach information from resource into telemetry - -``` -metrics: - set(attributes["k8s_pod"], resource.attributes["k8s.pod.name"]) -``` - -### Decorate error spans with additional information - -``` -traces: - set(attributes["whose_fault"], "theirs") where attributes["http.status"] == 400 or attributes["http.status"] == 404 - set(attributes["whose_fault"], "ours") where attributes["http.status"] == 500 -``` - -### Update a spans ID - -``` -logs: - set(span_id, SpanID(0x0000000000000000)) -traces: - set(span_id, SpanID(0x0000000000000000)) -``` - -### Convert metric name to snake case - -``` -metrics: - set(metric.name, ConvertCase(metric.name, "snake")) -``` - -### Check if an attribute exists - -``` -traces: - set(attributes["test-passed"], true) where attributes["target-attribute"] != nil -``` +- To modify your data as it passes through a pipeline, use the [transform processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/transformprocessor/README.md). +- To remove data from your pipeline, use the [filter processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/README.md). +- To select spans to be sampled, use the [tail sampling processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md). +- To route data between pipelines, use the [routing connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/connector/routingconnector/README.md). ## Troubleshooting @@ -160,6 +69,7 @@ service: ## Resources These are previous conference presentations given about OTTL: + - [OTTL Me Why Transforming Telemetry in the OpenTelemetry Collector Just Got Better](https://youtu.be/uVs0oUV72CE) - [Managing Observability Data at the Edge with the OpenTelemetry Collector and OTTL](https://youtu.be/GO0ulYLxy_8) - [The OTTL Cookbook: A Collection of Solutions to Common Problems](https://www.youtube.com/watch?v=UGTU0-KT_60) \ No newline at end of file diff --git a/pkg/ottl/ottlfuncs/README.md b/pkg/ottl/ottlfuncs/README.md index bf0ea3458897..890ceef7d6fe 100644 --- a/pkg/ottl/ottlfuncs/README.md +++ b/pkg/ottl/ottlfuncs/README.md @@ -39,9 +39,9 @@ Editors are what OTTL uses to transform telemetry. Editors: -- Are allowed to transform telemetry. When a Function is invoked the expectation is that the underlying telemetry is modified in some way. -- May have side effects. Some Functions may generate telemetry and add it to the telemetry payload to be processed in this batch. -- May return values. Although not common and not required, Functions may return values. +- Are allowed to transform telemetry. When an Editor is invoked the expectation is that the underlying telemetry is modified in some way. +- May have side effects. Some Editors may generate telemetry and add it to the telemetry payload to be processed in this batch. +- May return values. Although not common and not required, Editors may return values. Available Editors: @@ -69,9 +69,9 @@ The `append` function appends single or multiple string values to `target`. Resulting field is always of type `pcommon.Slice` and will not convert the types of existing or new items in the slice. This means that it is possible to create a slice whose elements have different types. Be careful when using `append` to set attribute values, as this will produce values that are not possible to create through OpenTelemetry APIs [according to](https://opentelemetry.io/docs/specs/otel/common/#attribute) the OpenTelemetry specification. - - `append(attributes["tags"], "prod")` - - `append(attributes["tags"], values = ["staging", "staging:east"])` - - `append(attributes["tags_copy"], attributes["tags"])` +- `append(attributes["tags"], "prod")` +- `append(attributes["tags"], values = ["staging", "staging:east"])` +- `append(attributes["tags_copy"], attributes["tags"])` ### delete_key