Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Document the new 9.0 tracing features #377

Merged
merged 2 commits into from
Nov 16, 2024
Merged

Document the new 9.0 tracing features #377

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1b9e749
Document the new 9.0 tracing features
roji Nov 9, 2024
de2b9c5
Redo to document new builder pattern, plus expanded AsyncLocal example
roji Nov 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 69 additions & 1 deletion conceptual/Npgsql/diagnostics/tracing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
# Tracing with OpenTelemetry (experimental)

> [!NOTE]
> Support for tracing via OpenTelemetry has been introduced in Npgsql 6.0.
>
> [The OpenTelemetry specifications for database tracing](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/database.md) are currently experimental, so Npgsql's support may change in upcoming releases.

[OpenTelemetry](https://opentelemetry.io/) is a widely-adopted framework for distributed observability across many languages and components; its tracing standards allow applications and libraries to emit information on activities and events, which can be exported by the application, stored and analyzed. Activities typically have start and end times, and can encompass other activities recursively; this allows you to analyze e.g. exactly how much time was spent in the database when handling a certain HTTP call.

## Basic usage

To make Npgsql emit tracing data, reference the [Npgsql.OpenTelemetry](https://www.nuget.org/packages/Npgsql.OpenTelemetry) NuGet package from your application, and set up tracing as follows:

```csharp
Expand All @@ -29,3 +30,70 @@ For example, Zipkin visualizes traces in the following way:
![Zipkin UI Sample](/img/zipkin.png)

In this trace, the Npgsql query (to database testdb) took around 800ms, and was nested inside the application's `work1` activity, which also had another unrelated `subtask1`. This allows understanding the relationships between the different activities, and where time is being spent.

## Configuration options

> [!NOTE]
>
> This feature was introduced in Npgsql 9.0

Once you've enabled Npgsql tracing as above, you can tweak its configuration via the <xref:Npgsql.NpgsqlDataSourceBuilder.ConfigureTracingOptions*?displayProperty=nameWithType> API:

```csharp
dataSourceBuilder.ConfigureTracing(o => o
// Set the command SQL as the span name
.ConfigureCommandSpanNameProvider(cmd => cmd.CommandText)
// Filter out COMMIT commands
.ConfigureCommandFilter(cmd => !cmd.CommandText.StartsWith("COMMIT", StringComparison.OrdinalIgnoreCase)));
```

This allows you to:

* Specify a filter which determines which commands get traced
* Set the the tracing span name (e.g. use the command's SQL as the span name)
* Add arbitrary tags to the tracing span, based on the command
* Disable the time-to-first-read event that's emitted in spans

## Using `AsyncLocal` to pass arbitrary information to your callbacks

The callbacks available via <xref:Npgsql.NpgsqlDataSourceBuilder.ConfigureTracingOptions*?displayProperty=nameWithType> only accept the <xref:Npgsql.NpgsqlCommand> or <xref:Npgsql.NpgsqlBatch> as their parameters; this makes it difficult to e.g. assign arbitrary names to your commands, so that show up as the span names in your tracing monitor. You can use .NET [`AsyncLocal`](https://learn.microsoft.com/dotnet/api/system.threading.asynclocal-1) to flow arbitrary information from the command call site (where you execute the command) to your tracing callbacks to achieve this.

For example, the following adds an `ExecuteReaderWithSpanNameAsync` extension method to <xref:Npgsql.NpgsqlCommand>:

```c#
internal static class DbCommandExtensions
{
internal static readonly AsyncLocal<string?> CommandName = new();

public static async Task<NpgsqlDataReader> ExecuteReaderWithSpanNameAsync(this NpgsqlCommand command, string spanName)
{
var previousValue = CommandName.Value;
CommandName.Value = "FetchAllUsers";

try
{
return await command.ExecuteReaderAsync();
}
finally
{
CommandName.Value = previousValue;
}
}
}
```

You can now configure your data source to use this span name in commands:

```c#
dataSourceBuilder.ConfigureTracing(o =>
o.ConfigureCommandSpanNameProvider(_ =>
DbCommandExtensions.CommandName.Value));
```

At this point, you can execute commands as follows, and see the provided value appearing in your tracing:

```c#
await using var reader = await command.ExecuteReaderWithSpanNameAsync("FetchAllUsers");
```

We'll likely work on future improvements to streamline this and make the above unnecessary.
42 changes: 23 additions & 19 deletions conceptual/Npgsql/release-notes/9.0.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,48 +5,52 @@ Npgsql version 9.0 will be released together with .NET 9 and will be available o
> [!NOTE]
> We're considering to start dropping support for synchronous API (`NpgsqlConnection.Open`, `NpgsqlCommand.ExecuteNonQuery`, etc) starting with Npgsql 10.0. The current plan is to deprecate the API by throwing a runtime exception by default (with a switch to re-enable synchronous I/O) for Npgsql 10.0, while completely removing it for Npgsql 11.0. This is in line with ASP.NET Core and .NET runtime in general, which are moving in the direction of async I/O only (for example, `System.IO.Pipelines` doesn't have synchronous I/O). If you have any questions or want to share you experience/issues with async I/O, please feel free to post in the [issue](https://github.com/npgsql/npgsql/issues/5865).

## Add support for interval's infinity values via NodaTime's Period.MinValue/Period.MaxValue
## Tracing improvements

PostgreSQL 17 added support for infinity values with `interval` type. In turn, Npgsql 9.0 adds native support to read and write them via NodaTime's `Period.MinValue` and `Period.MaxValue`. Note that while using this feature with previous versions of PostgreSQL, instead of infinity values you'll get the minimum and maximum values for `interval` type due to the way infinity values are implemented by PostgreSQL. See [this](https://github.com/npgsql/npgsql/issues/5696) issue for more info.
Several quality-of-life improvements have been implemented for Npgsql's OpenTelemetry tracing support. You can now do the following via the new <xref:Npgsql.NpgsqlDataSourceBuilder.ConfigureTracingOptions*?displayProperty=nameWithType> API:

## Add support for cidr <-> IPNetwork mapping
* Specify a filter which determines which commands get traced
* Set the the tracing span name (e.g. use the command's SQL as the span name)
* Add arbitrary tags to the tracing span, based on the command
* Disable the time-to-first-read event that's emitted in spans

.NET 8 added a new type [IPNetwork](https://learn.microsoft.com/en-us/dotnet/api/system.net.ipnetwork?view=net-8.0) which represents an IP network with an [IPAddress](https://learn.microsoft.com/en-us/dotnet/api/system.net.ipaddress?view=net-8.0) containing the network prefix and an `int` defining the prefix length. This type seems to be a perfect fit for PostgreSQL's `cidr` type, which is why we added support to read and write it. The default when reading a `cidr` is still `NpgsqlCidr` in 9.0, though this will likely change in Npgsql 10.0. See [this](https://github.com/npgsql/npgsql/issues/5821) issue for more info.
See the [tracing documentation](/doc/diagnostics/tracing.html) for more information.

## Add support for direct SSL
## Mapping improvements

PostgreSQL 17 added support for direct SSL. Direct SSL allows clients to skip sending an SSL support request, which saves a roundtrip while opening a physical connection. This behavior is disabled by default (as it's not supported with previous versions of PostgreSQL), but you can enable it via `SslNegotiation` property in connection string or environment variable `PGSSLNEGOTIATION`. See [this](https://github.com/npgsql/npgsql/issues/5677) issue for more info.
### Add support for cidr <-> IPNetwork mapping

## Add support to modify SslClientAuthenticationOptions
.NET 8 added a new type [IPNetwork](https://learn.microsoft.com/en-us/dotnet/api/system.net.ipnetwork?view=net-8.0) which represents an IP network with an [IPAddress](https://learn.microsoft.com/en-us/dotnet/api/system.net.ipaddress?view=net-8.0) containing the network prefix and an `int` defining the prefix length. This type seems to be a perfect fit for PostgreSQL's `cidr` type, which is why we added support to read and write it. The default when reading a `cidr` is still `NpgsqlCidr` in 9.0, though this will likely change in Npgsql 10.0. See [this issue](https://github.com/npgsql/npgsql/issues/5821) for more info.

Npgsql 9.0 has a new callback `NpgsqlDataSourceBuilder.UseSslClientAuthenticationOptionsCallback` which is called while connecting to PostgreSQL via `SslStream`. This allows users to modify [SslClientAuthenticationOptions](https://learn.microsoft.com/en-us/dotnet/api/system.net.security.sslclientauthenticationoptions?view=net-8.0), for example changing the supported TLS ciphers. See [this](https://github.com/npgsql/npgsql/issues/5478) issue for more info.
### Add support for interval's infinity values via NodaTime's Period.MinValue/Period.MaxValue

## Add support to modify NegotiateAuthenticationClientOptions
PostgreSQL 17 added support for infinity values with `interval` type. In turn, Npgsql 9.0 adds native support to read and write them via NodaTime's `Period.MinValue` and `Period.MaxValue`. Note that while using this feature with previous versions of PostgreSQL, instead of infinity values you'll get the minimum and maximum values for `interval` type due to the way infinity values are implemented by PostgreSQL. See [this issue](https://github.com/npgsql/npgsql/issues/5696) for more info.

Npgsql 9.0 has a new callback `NpgsqlDataSourceBuilder.NegotiateOptionsCallback` which is called while performing GSSAPI authentication (such as Kerberos). This allows users to modify [NegotiateAuthenticationClientOptions](https://learn.microsoft.com/en-us/dotnet/api/system.net.security.negotiateauthenticationclientoptions?view=net-8.0), for example changing `Credential` property to implement password-based Kerberos authentication. See [this](https://github.com/npgsql/npgsql/issues/5181) issue for more info.
## Security-related improvements

## Add support for parallel in-progress transactions with logical streaming replication protocol V4
* PostgreSQL 17 added support for **direct SSL**. Direct SSL allows clients to skip sending an SSL support request, which saves a roundtrip while opening a physical connection. This behavior is disabled by default (as it's not supported with previous versions of PostgreSQL), but you can enable it via the `SslNegotiation` property in connection string or environment variable `PGSSLNEGOTIATION`. See [this issue](https://github.com/npgsql/npgsql/issues/5677) for more info.
* The new <xref:Npgsql.NpgsqlDataSourceBuilder.UseSslClientAuthenticationOptionsCallback*?displayProperty=nameWithType> callback is called while connecting to PostgreSQL via `SslStream`, and allows users to modify [SslClientAuthenticationOptions](https://learn.microsoft.com/en-us/dotnet/api/system.net.security.sslclientauthenticationoptions?view=net-8.0), e.g. to change the supported TLS ciphers. See [this issue](https://github.com/npgsql/npgsql/issues/5478) for more info.
* The new <xref:Npgsql.NpgsqlDataSourceBuilder.UseNegotiateOptionsCallback*?displayProperty=nameWithType> callback is called while performing GSSAPI authentication (such as Kerberos), and allows users to modify [NegotiateAuthenticationClientOptions](https://learn.microsoft.com/en-us/dotnet/api/system.net.security.negotiateauthenticationclientoptions?view=net-8.0), e.g. to change the `Credential` property to implement password-based Kerberos authentication. See [this](https://github.com/npgsql/npgsql/issues/5181) issue for more info.

This change allows clients to handle multiple in-progress transactions in parallel instead of sequentially. See [this](https://github.com/npgsql/npgsql/issues/5760) issue for more info.
## Replication improvements

## Add GetFieldName method to ReplicationValue class

This change allows clients to receive the name of changed column while reading rows from replication stream. See [this](https://github.com/npgsql/npgsql/issues/5718) issue for more info.
* Parallel in-progress transactions with logical streaming replication protocol V4 allow clients to handle multiple in-progress transactions in parallel instead of sequentially. See [this issue](https://github.com/npgsql/npgsql/issues/5760) for more info.
* <xref:Npgsql.Replication.PgOutput.ReplicationValue.GetFieldName?displayProperty=nameWithType> `ReplicationValue.GetFieldName` allows clients to receive the name of changed column while reading rows from replication stream. See [this issue](https://github.com/npgsql/npgsql/issues/5718) for more info.

## Breaking changes

### .NET Standard 2.0 (and .NET Framework) is not supported

Npgsql 9.0 drops support for .NET Standard 2.0, and in turn .NET Framework. Npgsql is a constantly evolving driver, which makes it problematic to add new features which use APIs that do not exist on older versions of .NET. This doesn't mean you can't use Npgsql with .NET Standard 2.0: we're still committed to supporting Npgsql 8.0 (which does support .NET Standard 2.0). See [this](https://github.com/npgsql/npgsql/issues/5296) issue for more info.
Npgsql 9.0 drops support for .NET Standard 2.0, and in turn .NET Framework. Npgsql is a constantly evolving driver, which makes it problematic to add new features which use APIs that do not exist on older versions of .NET. This doesn't mean you can't use Npgsql with .NET Standard 2.0: we're still committed to supporting Npgsql 8.0 (which does support .NET Standard 2.0). See [this issue](https://github.com/npgsql/npgsql/issues/5296) for more info.

### Change some PgOutputReplicationOptions properties to support logical streaming replication protocol V4

The main changes are in the `PgOutputReplicationOptions` class, where `ProtocolVersion` and `StreamingMode` properties were changed to an enum. See [this](https://github.com/npgsql/npgsql/issues/5760) issue for more info.
The main changes are in the `PgOutputReplicationOptions` class, where `ProtocolVersion` and `StreamingMode` properties were changed to an enum. See [this issue](https://github.com/npgsql/npgsql/issues/5760) for more info.

### Multiple ssl related callbacks on NpgsqlDataSourceBuilder are deprecated in favor of UseSslClientAuthenticationOptionsCallback

With the new `UseSslClientAuthenticationOptionsCallback` callback, users have much more control over the way Npgsql connects to PostgreSQL via `SslStream`. This makes other callbacks, like `UseUserCertificateValidationCallback` and `UseClientCertificate` less useful, which is why we're obsoleting them. See [this](https://github.com/npgsql/npgsql/issues/5478) issue for more info.
With the new `UseSslClientAuthenticationOptionsCallback` callback, users have much more control over the way Npgsql connects to PostgreSQL via `SslStream`. This makes other callbacks, like `UseUserCertificateValidationCallback` and `UseClientCertificate` less useful, which is why we're obsoleting them. See [this issue](https://github.com/npgsql/npgsql/issues/5478) for more info.

### The default value of ConnectionLifetime property in connection string is set to 1 hour

Previously, the default value of this property was set to 0, which made connections last indefinitely. The old behavior was problematic because each physical connection on PostgreSQL's side holds on certain caches, which can only grow over time. See [this](https://github.com/npgsql/npgsql/pull/5662) pull request for more info.
Previously, the default value of this property was set to 0, which made connections last indefinitely. The old behavior was problematic because each physical connection on PostgreSQL's side holds on certain caches, which can only grow over time. See [this pull request](https://github.com/npgsql/npgsql/pull/5662) for more info.