Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stats/opentelemetry: separate out interceptors for tracing and metrics #8063

Open
wants to merge 57 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
7ddbe46
replace dial with newclient
janardhankrishna-sai Dec 23, 2024
0486355
Merge branch 'master' of https://github.com/janardhanvissa/grpc-go
janardhankrishna-sai Jan 1, 2025
53fa9cd
Merge branch 'grpc:master' into master
janardhanvissa Jan 8, 2025
4435b8a
Merge branch 'grpc:master' into master
janardhanvissa Jan 13, 2025
a413555
Merge branch 'grpc:master' into master
janardhanvissa Jan 20, 2025
4e203c3
Merge branch 'grpc:master' into master
janardhanvissa Jan 30, 2025
e9ad552
Merge branch 'grpc:master' into master
janardhanvissa Jan 30, 2025
71804b4
refactor tracing and metrics interceptors separately
janardhankrishna-sai Feb 3, 2025
4b3bd26
adding opentelemetry tracing for client-server
janardhankrishna-sai Jan 30, 2025
69df069
fixing vet issues
janardhankrishna-sai Feb 3, 2025
770f430
reverting newclient changes and fixing vet issues
janardhankrishna-sai Feb 3, 2025
b97a3ca
reverting otel trace for client/server changes
janardhankrishna-sai Feb 3, 2025
c89f3c9
fixing vet issue
janardhankrishna-sai Feb 3, 2025
3f07e48
renaming receiver name
janardhankrishna-sai Feb 5, 2025
5e8a4a5
unused param fix
janardhankrishna-sai Feb 5, 2025
76e422a
fixing vet issue
janardhankrishna-sai Feb 5, 2025
8fa0b03
refactor client interceptor separately for traces and metrics
janardhankrishna-sai Feb 12, 2025
1f41a49
moving tracing code to client_tracing.go file
janardhankrishna-sai Feb 12, 2025
d74c61d
revert previous commit
janardhankrishna-sai Feb 12, 2025
170eef6
adding separate interceptors for traces and metrics of server
janardhankrishna-sai Feb 17, 2025
7f5f539
separating HandleRPC interceptors of traces and metrics
janardhankrishna-sai Feb 17, 2025
50999a0
updating client and server metrics
janardhankrishna-sai Feb 18, 2025
68b8966
removing metrics code from tracingstatshandler and unused parameters
janardhankrishna-sai Feb 19, 2025
8b56f0f
addressing review comments
janardhankrishna-sai Feb 24, 2025
ac6080f
addressing review comments
janardhankrishna-sai Feb 25, 2025
ed5506c
Merge branch 'master' into refactor-tracing-metrics
janardhanvissa Feb 25, 2025
be72377
updating go.sum
janardhankrishna-sai Feb 25, 2025
a5ed115
fixing linter issue
janardhankrishna-sai Feb 25, 2025
f243b43
addressing review comments for client
janardhankrishna-sai Mar 7, 2025
efb738f
addressing review comments for client
janardhankrishna-sai Mar 7, 2025
7b97c65
Merge branch 'master' into refactor-tracing-metrics
janardhanvissa Mar 7, 2025
e8b0180
addressing review comments for server
janardhankrishna-sai Mar 7, 2025
f70274b
fixing vet issue
janardhankrishna-sai Mar 7, 2025
5e607c5
updating newServerStatsHandler
janardhankrishna-sai Mar 7, 2025
a850532
updating var name
janardhankrishna-sai Mar 7, 2025
0f39f8d
moving tracing code to a separate file and adding comments
janardhankrishna-sai Mar 10, 2025
2a914e3
removing unused isMetricsEnabled()
janardhankrishna-sai Mar 10, 2025
05b4278
addressing review comments and updating doc string
janardhankrishna-sai Mar 10, 2025
105efe9
addressing review comments
janardhankrishna-sai Mar 10, 2025
3d78d59
creating helper func for rpcinfo
janardhankrishna-sai Mar 10, 2025
44fce2d
addressing review comments
janardhankrishna-sai Mar 13, 2025
e5f50b7
updating logger
janardhankrishna-sai Mar 13, 2025
ffcc0fa
Merge branch 'master' into refactor-tracing-metrics
janardhanvissa Mar 13, 2025
67e5b24
update unaryInterceptor
janardhankrishna-sai Mar 13, 2025
8cf7f15
revert doc string
janardhankrishna-sai Mar 13, 2025
7949501
fixing vet issue
janardhankrishna-sai Mar 13, 2025
ad22a22
update clientStatsHandler unaryInterceptor
janardhankrishna-sai Mar 13, 2025
dd179fb
updating verbosity level and removing redundant code
janardhankrishna-sai Mar 13, 2025
3201805
reverting logger.Info from logger.Error
janardhankrishna-sai Mar 13, 2025
a3915ef
Update server_metrics.go
janardhanvissa Mar 13, 2025
1ff5159
addressing review comments
janardhankrishna-sai Mar 18, 2025
4133dd3
updating the comment
janardhankrishna-sai Mar 18, 2025
f8e0cac
addressing review comments
janardhankrishna-sai Mar 20, 2025
0b89cf6
fixing vet issue
janardhankrishna-sai Mar 20, 2025
42c08a0
fixing linter issue
janardhankrishna-sai Mar 20, 2025
5bb48b3
addressing review comments
janardhankrishna-sai Mar 25, 2025
fb8025d
addressing review comments
janardhankrishna-sai Mar 26, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
addressing review comments
  • Loading branch information
janardhankrishna-sai committed Mar 18, 2025
commit 1ff51595de22ba3c9c4fc7e36303170c5d2ca565
4 changes: 1 addition & 3 deletions stats/opentelemetry/client_metrics.go
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,6 @@ func (h *clientStatsHandler) initializeMetrics() {
rm.registerMetrics(metrics, meter)
}

// unaryInterceptor records metrics for unary RPC calls.
func (h *clientStatsHandler) unaryInterceptor(ctx context.Context, method string, req, reply any, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
ci := getCallInfo(ctx)
if ci == nil {
Expand Down Expand Up @@ -106,7 +105,6 @@ func determineMethod(method string, opts ...grpc.CallOption) string {
return "other"
}

// streamInterceptor records metrics for streaming RPC calls.
func (h *clientStatsHandler) streamInterceptor(ctx context.Context, desc *grpc.StreamDesc, cc *grpc.ClientConn, method string, streamer grpc.Streamer, opts ...grpc.CallOption) (grpc.ClientStream, error) {
ci := getCallInfo(ctx)
if ci == nil {
Expand Down Expand Up @@ -233,7 +231,7 @@ func (h *clientStatsHandler) setLabelsFromPluginOption(ai *attemptInfo, incoming
func (h *clientStatsHandler) processRPCEnd(ctx context.Context, ai *attemptInfo, e *stats.End) {
ci := getCallInfo(ctx)
if ci == nil {
logger.Info("ctx passed into client side stats handler metrics event handling has no metrics data present")
logger.Error("ctx passed into client side stats handler metrics event handling has no metrics data present")
return
}
latency := float64(time.Since(ai.startTime)) / float64(time.Second)
Expand Down
44 changes: 27 additions & 17 deletions stats/opentelemetry/client_tracing.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,13 @@
"context"
"log"
"strings"
"time"

otelcodes "go.opentelemetry.io/otel/codes"
"go.opentelemetry.io/otel/trace"
"google.golang.org/grpc"
grpccodes "google.golang.org/grpc/codes"
istats "google.golang.org/grpc/internal/stats"
"google.golang.org/grpc/stats"
otelinternaltracing "google.golang.org/grpc/stats/opentelemetry/internal/tracing"
"google.golang.org/grpc/status"
Expand All @@ -38,23 +40,21 @@

func (h *clientTracingHandler) initializeTraces() {
if h.options.TraceOptions.TracerProvider == nil {
log.Printf("TraceProvider is not provided in client trace options")
log.Printf("TracerProvider is not provided in client TraceOptions")
return
}

Check warning on line 45 in stats/opentelemetry/client_tracing.go

View check run for this annotation

Codecov / codecov/patch

stats/opentelemetry/client_tracing.go#L43-L45

Added lines #L43 - L45 were not covered by tests
h.options.TraceOptions.TracerProvider.Tracer(tracerName, trace.WithInstrumentationVersion(grpc.Version))
}

// unaryInterceptor records traces for unary RPC calls.
func (h *clientTracingHandler) unaryInterceptor(ctx context.Context, method string, req, reply any, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
ci := getCallInfo(ctx)
if ci == nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As above.

logger.Info("ctx passed into client tracing handler unary interceptor has no call info data present")
logger.Info("callInfo not present in context in clientTracingHandler unaryInterceptor")
ci = &callInfo{
target: cc.CanonicalTarget(),
method: determineMethod(method, opts...),
}
ctx = setCallInfo(ctx, ci)
}

Check warning on line 57 in stats/opentelemetry/client_tracing.go

View check run for this annotation

Codecov / codecov/patch

stats/opentelemetry/client_tracing.go#L51-L57

Added lines #L51 - L57 were not covered by tests

var span trace.Span
ctx, span = h.createCallTraceSpan(ctx, method)
Expand All @@ -63,23 +63,20 @@
return err
}

// streamInterceptor records traces for streaming RPC calls.
func (h *clientTracingHandler) streamInterceptor(ctx context.Context, desc *grpc.StreamDesc, cc *grpc.ClientConn, method string, streamer grpc.Streamer, opts ...grpc.CallOption) (grpc.ClientStream, error) {
ci := getCallInfo(ctx)
if ci == nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As above.

logger.Info("ctx passed into client tracing handler stream interceptor has no call info data present")
logger.Info("callInfo not present in context in clientTracingHandler streamInterceptor")
ci = &callInfo{
target: cc.CanonicalTarget(),
method: determineMethod(method, opts...),
}
ctx = setCallInfo(ctx, ci)
}

Check warning on line 75 in stats/opentelemetry/client_tracing.go

View check run for this annotation

Codecov / codecov/patch

stats/opentelemetry/client_tracing.go#L69-L75

Added lines #L69 - L75 were not covered by tests

var span trace.Span
ctx, span = h.createCallTraceSpan(ctx, method)
callback := func(err error) {
h.perCallTraces(err, span)
}
callback := func(err error) { h.perCallTraces(err, span) }
opts = append([]grpc.CallOption{grpc.OnFinish(callback)}, opts...)
return streamer(ctx, desc, cc, method, opts...)
}
Expand Down Expand Up @@ -128,16 +125,29 @@
func (h *clientTracingHandler) HandleConn(context.Context, stats.ConnStats) {}

// TagRPC implements per RPC attempt context management for traces.
func (h *clientTracingHandler) TagRPC(ctx context.Context, _ *stats.RPCTagInfo) context.Context {
// Fetch the rpcInfo set by a previously registered stats handler
// (like clientStatsHandler). Assumes this handler runs after one
// that sets the rpcInfo in the context.
func (h *clientTracingHandler) TagRPC(ctx context.Context, info *stats.RPCTagInfo) context.Context {
// Fetch the rpcInfo set by a previously registered stats handler.
ri := getRPCInfo(ctx)
if ri == nil {
logger.Info("ctx passed into client side tracing stats handler has no client attempt data present")
return ctx
var ai *attemptInfo
if ri != nil {
ai = ri.ai
} else {
labels := istats.GetLabels(ctx)
if labels == nil {
labels = &istats.Labels{
TelemetryLabels: map[string]string{
"grpc.lb.locality": "",
},
}
ctx = istats.SetLabels(ctx, labels)
}
ai = &attemptInfo{
startTime: time.Now(),
xdsLabels: labels.TelemetryLabels,
method: removeLeadingSlash(info.FullMethodName),
}

Check warning on line 148 in stats/opentelemetry/client_tracing.go

View check run for this annotation

Codecov / codecov/patch

stats/opentelemetry/client_tracing.go#L135-L148

Added lines #L135 - L148 were not covered by tests
}
ctx, ai := h.traceTagRPC(ctx, ri.ai)
ctx, ai = h.traceTagRPC(ctx, ai)
return setRPCInfo(ctx, &rpcInfo{ai: ai})
}

Expand All @@ -148,8 +158,8 @@
// that sets the rpcInfo in the context.
ri := getRPCInfo(ctx)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think Both HandleRPC and TagRPC use getRPCInfo() with logging and nil handling. Consider creating a common function to streamline this logic?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

if ri == nil {
logger.Error("ctx passed into client side tracing stats handler has no client attempt data present")
return
}

Check warning on line 163 in stats/opentelemetry/client_tracing.go

View check run for this annotation

Codecov / codecov/patch

stats/opentelemetry/client_tracing.go#L161-L163

Added lines #L161 - L163 were not covered by tests
populateSpan(rs, ri.ai)
}
8 changes: 4 additions & 4 deletions stats/opentelemetry/opentelemetry.go
Original file line number Diff line number Diff line change
Expand Up @@ -114,8 +114,8 @@ type MetricsOptions struct {
// configured for an individual metric turned on, the API call in this component
// will create a default view for that metric.
//
// For tracing, provide a TracerProvider in trace options.
// If no TracerProvider is provided, tracing will be no-op.
// For the traces supported by this instrumentation code, provide an
// implementation of a TextMapPropagator and OpenTelemetry TracerProvider.
func DialOption(o Options) grpc.DialOption {
csh := &clientStatsHandler{options: o}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't it possible to have metrics disabled and tracing enabled?

csh.initializeMetrics()
Expand Down Expand Up @@ -143,8 +143,8 @@ var joinServerOptions = internal.JoinServerOptions.(func(...grpc.ServerOption) g
// configured for an individual metric turned on, the API call in this component
// will create a default view for that metric.
//
// For tracing, provide a TracerProvider in trace options.
// If no TracerProvider is provided, tracing will be no-op.
// For the traces supported by this instrumentation code, provide an
// implementation of a TextMapPropagator and OpenTelemetry TracerProvider.
func ServerOption(o Options) grpc.ServerOption {
ssh := &serverStatsHandler{options: o}
ssh.initializeMetrics()
Expand Down
2 changes: 0 additions & 2 deletions stats/opentelemetry/server_metrics.go
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,6 @@ func (s *attachLabelsTransportStream) SendHeader(md metadata.MD) error {
return s.ServerTransportStream.SendHeader(md)
}

// unaryInterceptor records metrics for unary RPC calls.
func (h *serverStatsHandler) unaryInterceptor(ctx context.Context, req any, _ *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (any, error) {
var metadataExchangeLabels metadata.MD
if h.options.MetricsOptions.pluginOption != nil {
Expand Down Expand Up @@ -152,7 +151,6 @@ func (s *attachLabelsStream) SendMsg(m any) error {
return s.ServerStream.SendMsg(m)
}

// streamInterceptor records metrics for streaming RPC calls.
func (h *serverStatsHandler) streamInterceptor(srv any, ss grpc.ServerStream, _ *grpc.StreamServerInfo, handler grpc.StreamHandler) error {
var metadataExchangeLabels metadata.MD
if h.options.MetricsOptions.pluginOption != nil {
Expand Down
45 changes: 32 additions & 13 deletions stats/opentelemetry/server_tracing.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,11 @@
"context"
"log"
"strings"
"time"

"go.opentelemetry.io/otel/trace"
"google.golang.org/grpc"
"google.golang.org/grpc/internal"
"google.golang.org/grpc/stats"
otelinternaltracing "google.golang.org/grpc/stats/opentelemetry/internal/tracing"
)
Expand All @@ -33,10 +35,9 @@

func (h *serverTracingHandler) initializeTraces() {
if h.options.TraceOptions.TracerProvider == nil {
log.Printf("TraceProvider is not provided in server trace options")
log.Printf("TracerProvider is not provided in server TraceOptions")
return
}

Check warning on line 40 in stats/opentelemetry/server_tracing.go

View check run for this annotation

Codecov / codecov/patch

stats/opentelemetry/server_tracing.go#L38-L40

Added lines #L38 - L40 were not covered by tests
h.options.TraceOptions.TracerProvider.Tracer(tracerName, trace.WithInstrumentationVersion(grpc.Version))
}

func (h *serverTracingHandler) unaryInterceptor(ctx context.Context, req any, _ *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (any, error) {
Expand All @@ -48,16 +49,36 @@
}

// TagRPC implements per RPC attempt context management for traces.
func (h *serverTracingHandler) TagRPC(ctx context.Context, _ *stats.RPCTagInfo) context.Context {
// Fetch the rpcInfo set by a previously registered stats handler
// (like serverStatsHandler). Assumes this handler runs after one
// that sets the rpcInfo in the context.
func (h *serverTracingHandler) TagRPC(ctx context.Context, info *stats.RPCTagInfo) context.Context {
// Fetch the rpcInfo set by a previously registered stats handler.
ri := getRPCInfo(ctx)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add comments here and move the logic. It's possible to make a common function for retrieving getRPCInfo from the context.

Copy link
Contributor Author

@janardhanvissa janardhanvissa Mar 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make sense, done

if ri == nil {
logger.Info("ctx passed into server side tracing stats handler has no server call data present")
return ctx
var ai *attemptInfo
if ri != nil {
ai = ri.ai
} else {
method := info.FullMethodName
if h.options.MetricsOptions.MethodAttributeFilter != nil {
if !h.options.MetricsOptions.MethodAttributeFilter(method) {
method = "other"
}

Check warning on line 63 in stats/opentelemetry/server_tracing.go

View check run for this annotation

Codecov / codecov/patch

stats/opentelemetry/server_tracing.go#L59-L63

Added lines #L59 - L63 were not covered by tests
}
server := internal.ServerFromContext.(func(context.Context) *grpc.Server)(ctx)
if server == nil { // Shouldn't happen, defensive programming.
logger.Error("ctx passed into server side stats handler has no grpc server ref")
method = "other"
} else {
isRegisteredMethod := internal.IsRegisteredMethod.(func(*grpc.Server, string) bool)
if !isRegisteredMethod(server, method) {
method = "other"
}

Check warning on line 73 in stats/opentelemetry/server_tracing.go

View check run for this annotation

Codecov / codecov/patch

stats/opentelemetry/server_tracing.go#L65-L73

Added lines #L65 - L73 were not covered by tests
}

ai = &attemptInfo{
startTime: time.Now(),
method: removeLeadingSlash(method),
}

Check warning on line 79 in stats/opentelemetry/server_tracing.go

View check run for this annotation

Codecov / codecov/patch

stats/opentelemetry/server_tracing.go#L76-L79

Added lines #L76 - L79 were not covered by tests
}
ctx, ai := h.traceTagRPC(ctx, ri.ai)
ctx, ai = h.traceTagRPC(ctx, ai)
return setRPCInfo(ctx, &rpcInfo{ai: ai})
}

Expand All @@ -83,14 +104,12 @@

// HandleRPC handles per-RPC attempt stats events for tracing.
func (h *serverTracingHandler) HandleRPC(ctx context.Context, rs stats.RPCStats) {
// Fetch the rpcInfo set by a previously registered stats handler
// (like serverStatsHandler). Assumes this handler runs after one
// that sets the rpcInfo in the context.
// Fetch the rpcInfo set by a previously registered stats handler.
ri := getRPCInfo(ctx)
if ri == nil {
logger.Error("ctx passed into server side tracing stats handler has no server call data present")
return
}

Check warning on line 112 in stats/opentelemetry/server_tracing.go

View check run for this annotation

Codecov / codecov/patch

stats/opentelemetry/server_tracing.go#L110-L112

Added lines #L110 - L112 were not covered by tests
populateSpan(rs, ri.ai)
}

Expand Down
Loading