Add ability to auto-instrument with the OTel Java SDK #1704

grcevski · 2025-02-26T19:28:19Z

The Hotspot JVM allows for dynamic loading of agents and this provides a way for auto-instrumentation of Java applications without the use of eBPF, but by loading the OpenTelemetry Java agent dynamically. This PR introduces an experimental support for using the OpenTelemetry agent instead of eBPF when instrumenting Java applications.

The way this is implemented is as follows:

Beyla does its normal scan for services to instrument based on the discovery criteria.
If a Java service is discovered (and the experimental OTel agent support is enabled), we enter the namespace of the Java service and then run a few jcmd commands.
First we check if the version of the JVM is supported. At the moment JDK 25 is not supported by OTel instrumentation, since Java 25 is still in early-access mode.
Next, we walk the classes loaded by the JVM and verify if it doesn't already load the OpenTelemetry agent.
Finally, we copy the OpenTelemetry agent to the temporary directory on the running container and dynamically load the agent, while passing a translation of the Beyla options to the OTLP Java agent options.

codecov · 2025-02-26T19:41:01Z

Codecov Report

Attention: Patch coverage is 60.51661% with 107 lines in your changes missing coverage. Please review.

Project coverage is 69.68%. Comparing base (998ab56) to head (740f620).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/internal/otelsdk/sdk_inject.go	51.25%	80 Missing and 17 partials ⚠️
pkg/export/otel/common.go	60.00%	4 Missing and 4 partials ⚠️
pkg/export/otel/grafana.go	66.66%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1704      +/-   ##
==========================================
+ Coverage   65.29%   69.68%   +4.38%     
==========================================
  Files         196      210      +14     
  Lines       20086    21282    +1196     
==========================================
+ Hits        13116    14831    +1715     
+ Misses       6152     5721     -431     
+ Partials      818      730      -88

Flag	Coverage Δ
integration-test	`54.47% <58.67%> (+0.05%)`	⬆️
k8s-integration-test	`53.93% <17.34%> (-0.62%)`	⬇️
oats-test	`34.79% <15.49%> (-0.29%)`	⬇️
unittests	`45.77% <14.39%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

rafaelroquetto

This is legit great!
I left a bunch of nits/suggestions, feel free to ignore. I guess the only potential issue is the assumption of "." as the current working directory - let me know what you think

pkg/export/otel/traces.go

rafaelroquetto · 2025-02-26T21:40:04Z

pkg/internal/discover/attacher.go

+					sdkInstrumented := false
+					if ta.sdkInjectionPossible(&instr.Obj) {
+						if err := ta.sdkInjector.NewExecutable(&instr.Obj); err == nil {
+							sdkInstrumented = true
+						}
+					}


just a cosmetic nit - if you wrap this into a function, you can reword this as

sdkInstrumented := ta.trySDKInjection(&instr.Obj)

pkg/internal/otelsdk/sdk_inject.go

mariomac

That's amazing!!!

I'd raise some concerns/questions, not to be addressed right now in this PR, but to think about in the future:

What happens if we stop Beyla? Is the agent unloaded or it keeps running and instrumenting until the service is restarted?
If Beyla is restarted with a new configuration (for example, a new OTEL endpoint or credentials), the new configuration should be applied in the target service. This might require a way to know when a service has been instrumented "manually" or when the instrumentation has been injected from Beyla.
Having the JAR file embedded in our repository might look a bit obscure for some users. I'd discuss the option of either compile it from the sources or, download the binary from a trusted environment.

But anyway, this is an amazing leap forward!

mariomac · 2025-02-27T09:02:55Z

Dockerfile

@@ -39,6 +40,7 @@ COPY --from=builder /opt/app-root/bin/beyla .
 COPY --from=builder /opt/app-root/LICENSE .
 COPY --from=builder /opt/app-root/NOTICE .
 COPY --from=builder /opt/app-root/third_party_licenses.csv .
+COPY --from=builder /opt/app-root/opentelemetry-javaagent.jar .


As something to improve in the future, we should find a way track the javaagent version and include it in our vulnerability scans.

pkg/export/otel/traces.go

mariomac · 2025-02-27T09:07:56Z

pkg/internal/otelsdk/sdk_inject.go

+	"github.com/grafana/beyla/v2/pkg/internal/svc"
+)
+
+const OTelJDKAgent = "opentelemetry-javaagent.jar"


As a future task, I'd set this value configurable so the user can provide its own version of the java-agent (e.g. a patched version, or an updated version to temporarily fix a discovered vulnerability).

fstab · 2025-02-27T11:20:17Z

not to be addressed right now in this PR, but to think about in the future

Actually, I think we should address the issues @mariomac brought up before we merge this PR. So far Beyla is low risk: If instrumentation causes issues, you can always stop Beyla and everything's back to normal. If that changes because attaching the JVM agent can't be undone, we need an idea what to do if the agent causes issues or if there is a misconfiguration (wrong OTLP endpoint, too high sampling rate, etc).

…t into docker_utils

Generalise endpoint resolution func

grcevski · 2025-02-28T00:27:20Z

What happens if we stop Beyla? Is the agent unloaded or it keeps running and instrumenting until the service is restarted?

Good point, I think we can work on fixing this in follow-up PRs. I have some ideas, I think we should be able to supply set of options to make the agent effectively disabled.

If Beyla is restarted with a new configuration (for example, a new OTEL endpoint or credentials), the new configuration should be applied in the target service. This might require a way to know when a service has been instrumented "manually" or when the instrumentation has been injected from Beyla.

Great suggestion, I think we can do this. Right now we avoid double instrumenting, but we can do better than this. We should be able to tell if this agent was loaded on the command line or not, so to tell if it was managed by Beyla or not.

Having the JAR file embedded in our repository might look a bit obscure for some users. I'd discuss the option of either compile it from the sources or, download the binary from a trusted environment.

Rafael pushed changes to this PR and resolved this issue and the stuff that was bugging him :).

rafaelroquetto

Groovy!

grcevski · 2025-02-28T00:29:18Z

not to be addressed right now in this PR, but to think about in the future

Actually, I think we should address the issues @mariomac brought up before we merge this PR. So far Beyla is low risk: If instrumentation causes issues, you can always stop Beyla and everything's back to normal. If that changes because attaching the JVM agent can't be undone, we need an idea what to do if the agent causes issues or if there is a misconfiguration (wrong OTLP endpoint, too high sampling rate, etc).

We are not going to document this option for now, so that we can work on making it more solid. As you and Mario mentioned there are edge-cases we need to consider. But if we did it all at once it would be a much larger PR and complex. This way we can iterate on this until we are confident we can make this option suitable for customers to turn on.

marctc · 2025-02-28T14:22:42Z

pkg/config/ebpf_tracer.go

@@ -59,4 +59,7 @@ type EBPFTracer struct {

 	// Enables debug printing of the protocol data
 	ProtocolDebug bool `yaml:"protocol_debug_print" env:"BEYLA_PROTOCOL_DEBUG_PRINT"`
+
+	// Enables Java instrumentation with the OpenTelemetry JDK Agent
+	UseOTelSDKForJava bool `yaml:"use_otel_sdk_for_java" env:"BEYLA_USE_OTEL_SDK_FOR_JAVA"`


Reading the code above, I assume that enabling this would only generate OTEL metrics or traces. I think we should validate that if Prometheus is enabled this option can't be enabled.

I think this happens right now:
https://github.com/grafana/beyla/pull/1704/files#diff-d6000537195fb5cd7aa9e0b6bea705c332ce53d5db9e070e39dffcdc191562e3R65

With prometheus scrape Metrics will be false, so we'll program it to send traces only.

grcevski added 7 commits February 24, 2025 16:57

WIP: Attaching the OTel SDK to java processes

9667d6c

Fix todos

895e54e

Update dockerfiles, add test configs

af7dd9e

Fixes and integration tests

e252dc6

ensure we don't instrument JDK 25

696eb90

Merge branch 'main' into docker_utils

9abd50e

refactoring and linting

cefd5cf

grcevski requested a review from a team as a code owner February 26, 2025 19:28

grcevski added 2 commits February 26, 2025 14:31

remove stale todo

27c68ca

update licenses

0bba876

update makefile

e61da9e

rafaelroquetto reviewed Feb 26, 2025

View reviewed changes

mariomac approved these changes Feb 27, 2025

View reviewed changes

rafaelroquetto and others added 13 commits February 27, 2025 13:32

Merge branch 'main' into docker_utils

b06feb9

Pull java agent from upstream and embed it

f798e2d

Fix linter

1a4dfbb

Merge pull request #8 from grafana/java_agent

74474ec

Merge branch 'main' into docker_utils

5b6c7b8

merge

b6039a0

ignore files

8f4acab

Merge branch 'docker_utils' of github.com:grcevski/ebpf-autoinstrumen…

039c17d

…t into docker_utils

Fix linter - ioutil -> os

396da18

Fix tests

9283c78

Merge pull request #9 from grafana/java_agent_linter_fix

0a86c8b

Generalise endpoint resolution func

c3c67ab

Merge pull request #10 from grafana/otel_gen_fix

740f620

Generalise endpoint resolution func

rafaelroquetto approved these changes Feb 28, 2025

View reviewed changes

marctc reviewed Feb 28, 2025

View reviewed changes

grcevski merged commit c9291b2 into grafana:main Feb 28, 2025
12 checks passed

grcevski deleted the docker_utils branch February 28, 2025 16:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ability to auto-instrument with the OTel Java SDK #1704

Add ability to auto-instrument with the OTel Java SDK #1704

grcevski commented Feb 26, 2025

codecov bot commented Feb 26, 2025 •

edited

Loading

rafaelroquetto left a comment •

edited

Loading

rafaelroquetto Feb 26, 2025

mariomac left a comment

mariomac Feb 27, 2025

mariomac Feb 27, 2025

fstab commented Feb 27, 2025 •

edited

Loading

grcevski commented Feb 28, 2025 •

edited by marctc

Loading

rafaelroquetto left a comment

grcevski commented Feb 28, 2025

marctc Feb 28, 2025

grcevski Feb 28, 2025

Add ability to auto-instrument with the OTel Java SDK #1704

Add ability to auto-instrument with the OTel Java SDK #1704

Conversation

grcevski commented Feb 26, 2025

codecov bot commented Feb 26, 2025 • edited Loading

Codecov Report

rafaelroquetto left a comment • edited Loading

Choose a reason for hiding this comment

rafaelroquetto Feb 26, 2025

Choose a reason for hiding this comment

mariomac left a comment

Choose a reason for hiding this comment

mariomac Feb 27, 2025

Choose a reason for hiding this comment

mariomac Feb 27, 2025

Choose a reason for hiding this comment

fstab commented Feb 27, 2025 • edited Loading

grcevski commented Feb 28, 2025 • edited by marctc Loading

rafaelroquetto left a comment

Choose a reason for hiding this comment

grcevski commented Feb 28, 2025

marctc Feb 28, 2025

Choose a reason for hiding this comment

grcevski Feb 28, 2025

Choose a reason for hiding this comment

codecov bot commented Feb 26, 2025 •

edited

Loading

rafaelroquetto left a comment •

edited

Loading

fstab commented Feb 27, 2025 •

edited

Loading

grcevski commented Feb 28, 2025 •

edited by marctc

Loading