Ensuring Conformance of Application OTLP Streams to Semantic Conventions via Registry Validation

[lquerel]

Problem

An application’s adherence to semantic conventions is currently only ensured by the goodwill of the developers and reviewers of the application. It is possible, and even highly likely, to see divergences and improper uses emerge during the lifecycle of the application and/or the semantic conventions. This lack of consistency between the semantic conventions used by an application and the instrumentation integrated into it is unsatisfactory.

Solutions

• Use a type-safe client SDK generated from the semantic conventions. This approach is not the subject of this issue.
• Use Weaver to verify the conformity of the OTLP stream emitted by the application (during the build pipeline or emitted by tests) against the registry associated with the application. This solution is described in more detail below.

Command: registry live-check usage

Usage: weaver registry live-check [OPTIONS]

Options:
      --debug...
          Enable debugging information.

  -r, --registry <REGISTRY>
          Local folder, Git repository URL, or Git archive URL of the semantic convention registry. For Git URLs, a sub-folder can be specified using the `[sub-folder]` syntax after the URL.

          [default: https://github.com/open-telemetry/semantic-conventions.git[model]]

      --quiet
          Enable quiet mode (minimal output).

  -s, --follow-symlinks
          Boolean flag to specify whether to follow symlinks when loading the registry. Default is false.

      --future
          Enable the most recent validation rules for the semconv registry. It is recommended to enable this flag when checking a new registry. Note: `semantic_conventions` main branch should always enable this flag.

  -p, --otlp-grpc-port <OTLP_GRPC_PORT>
          Port used by the gRPC OTLP listener.

          [default: 4317]

  -a, --admin-port <ADMIN_PORT>
          Port used by the HTTP admin port (endpoints: /stop).

          [default: 4320]

  -t, --inactivity-timeout <INACTIVITY_TIMEOUT>
          Maximum inactivity time in seconds before stopping the listener.

          [default: 10]

      --diagnostic-format <DIAGNOSTIC_FORMAT>
          Format used to render diagnostic messages. Predefined formats are: ansi, json, gh_workflow_command.

          [default: ansi]

      --diagnostic-template <DIAGNOSTIC_TEMPLATE>
          Path to the directory where the diagnostic templates are located.

          [default: diagnostic_templates]

  -h, --help
          Print help (see a summary with '-h').

A skeleton of this command has been defined in #548. This skeleton includes loading the resolved version of the registry as well as an iterator for incoming OTLP messages. At a minimum, this command should be able to generate a report containing:

• A list of attributes defined in the registry and observed in the stream.
• A list of attributes defined in the registry but not observed in the stream.
• A list of attributes observed in the stream but not defined in the registry.

Other elements that could be added to this report include:

• Identification of attributes in the stream that do not follow naming rules.
• Identification of attributes that are very similar to an attribute in the registry.
• Application of these checks to signals, not just attributes.
• ...