K8s auto-instrumentation details and example (Java) (#54)

* instrumenting java added * changes in generic instrumenting-applications.md file
elastic · Nov 28, 2024 · 27cace5 · 27cace5
1 parent 41da320
commit 27cace5
Show file tree

Hide file tree

Showing 2 changed files with 191 additions and 4 deletions.
diff --git a/docs/kubernetes/operator/instrumenting-applications.md b/docs/kubernetes/operator/instrumenting-applications.md
@@ -12,14 +12,15 @@ This section provides guidance and examples for applications instrumentation in
 
 In Kubernetes environments with the OpenTelemetry Operator, [**automatic (or zero-code) instrumentation**](https://opentelemetry.io/docs/kubernetes/operator/automatic/) simplifies the process by injecting and configuring instrumentation libraries into the targeted Pods.
 
-On the other hand, **manual instrumentation** with OpenTelemetry involves adding specific OpenTelemetry SDKs and APIs directly into your application’s code. This approach provides more granular control over what and how data is captured, allowing you to customize trace spans, metrics, and logging based on your application’s logic.
+On the other hand, **manual instrumentation** with OpenTelemetry allows you to customize trace spans, metrics, and logging directly in your application’s code. This approach provides more granular control over what and how data is captured.
 
 ## Table of contents
 
 - [Supported languages](#supported-languages)
 - [Prerequisites](#prerequisites)
 - [Auto-instrumentation basics](#auto-instrumentation-basics)
 - [Configuring auto-instrumentation](#configuring-auto-instrumentation)
+- [How auto-instrumentation works](#how-auto-instrumentation-works)
 - [Advanced configuration](#advanced-configuration)
 - [Manual instrumentation](#manual-instrumentation)
 
@@ -110,9 +111,9 @@ where ``<LANGUAGE>`` is one of: `go` , `java`, `nodejs`, `python`, `dotnet`
 
 > [!NOTE]
 > Ensure you add the annotations at Pod level and not directly at the workload `spec` level (Deployment, Job, etc.).
-> Ensure the annotation value must points to an existing `Instrumentation` object.
+> Ensure the annotation value points to an existing `Instrumentation` object.
 
-Alternatively, you can enable auto-instrumentation by adding the annotation at the **namespace level**. This approach automatically applies instrumentation to all Pods within the specified namespace.
+Alternatively, you can enable auto-instrumentation by adding the annotation at **namespace level**. This approach automatically applies instrumentation to all Pods within the specified namespace.
 
 ```yaml
 apiVersion: v1
@@ -133,11 +134,17 @@ In case you have multiple Instrumentation objects with different settings or ima
 
 The possible values for the annotation are detailed in the [Operator documentation](https://opentelemetry.io/docs/kubernetes/operator/automatic/#add-annotations-to-existing-deployments). For reference purposes, the values are:
 
-- `"true"`: to inject Instrumentation resource with default name from the current namespace.
+- `"true"`: to inject Instrumentation instance with `default` name from the current namespace.
 - `"my-instrumentation"`: to inject Instrumentation instance with name `"my-instrumentation"` in the current namespace.
 - `"my-other-namespace/my-instrumentation"`: to inject Instrumentation instance with name `"my-instrumentation"` from another namespace `"my-other-namespace"`.
 - `"false"`: do not inject.
 
+For details on instrumenting specific languages, refer to:
+
+- [Instrumenting Java](./instrumenting-java.md)
+- [Instrumenting Python](./instrumenting-python.md)
+- [Instrumenting Dotnet](./instrumenting-dotnet.md)
+
 ### Namespace based annotations example
 
 The following example creates a namespace with an annotation to instrument all Pods of the namespace with `java` libraries.
@@ -152,6 +159,32 @@ kubectl annotate namespace java-apps instrumentation.opentelemetry.io/inject-jav
 kubectl run otel-test -n java-apps --env OTEL_INSTRUMENTATION_METHODS_INCLUDE="test.Testing[methodB]" --image docker.elastic.co/demos/apm/k8s-webhook-test
 ```
 
+## Verify auto-instrumentation
+
+After adding the annotation and restarting the Pods, run `kubectl describe` on your application Pod to verify the SDK has been properly attached.
+
+Ensure that the `init container`, `volume`, and `environment variables` described in [how auto-instrumentation works](#how-auto-instrumentation-works) have been successfully injected into the Pod.
+
+## How auto-instrumentation works
+
+The OpenTelemetry Operator automates the process of instrumenting applications by injecting the necessary libraries and configuration into the application Pods.
+The process may vary slightly depending on the language, but it generally involves the following steps:
+
+- **Adding an init container**:
+
+  The operator adds an init container into the Pod. This container is responsible for copying the OpenTelemetry instrumentation library and make it accessible to the main application container.
+
+- **Creating a shared volume**:
+
+  The operator creates an `emptyDir` shared volume within the Pod, and mounts it in both containers. This volume serves as the medium for sharing the instrumentation library between the init container and the application container.
+
+- **Configuring the main container**:
+
+  The operator injects environment variables into the main application container to configure OpenTelemetry settings (for example, `OTEL_EXPORTER_OTLP_ENDPOINT` or `OTEL_TRACES_SAMPLER`). Additionally, it links the instrumentation library to the application using mechanisms specific to the language runtime, such as:
+    - **For Java**: The library is linked through the `javaagent` option using the JAVA_TOOL_OPTIONS environment variable.
+    - **For Node.js**: The library is linked through the `NODE_OPTIONS` environment variable.
+    - **For Python**: The operator uses the `PYTHONPATH` environment variable to load the library [sitecustomize](https://docs.python.org/es/dev/library/site.html#module-sitecustomize) module.
+
 ## Advanced configuration
 
 You can apply OTEL specific configuration to your applications at two different levels:

diff --git a/docs/kubernetes/operator/instrumenting-java.md b/docs/kubernetes/operator/instrumenting-java.md
@@ -0,0 +1,154 @@
+# Instrumenting Java applications with EDOT SDKs on Kubernetes
+
+This document focuses on instrumenting Java applications on Kubernetes, using the OpenTelemetry Operator, Elastic Distribution of OpenTelemetry (EDOT) Collectors, and the [EDOT Java](https://github.com/elastic/elastic-otel-java) SDK.
+
+- For general knowledge about the EDOT Java SDK, refer to the [getting started guide](https://github.com/elastic/elastic-otel-java/blob/main/docs/get-started.md).
+- For Java auto-instrumentation specifics, refer to [OpenTelemetry Operator Java auto-instrumentation](https://opentelemetry.io/docs/kubernetes/operator/automatic/#java).
+- For general information about instrumenting applications on kubernetes, refer to [instrumenting applications](./instrumenting-applications.md).
+
+## Java agent extensions consideration
+
+The operator supports a configuration that installs [Java agent extensions](https://opentelemetry.io/docs/zero-code/java/agent/extensions/) in `Instrumentation` objects. The extension needs to be available in an image. Refer to [using extensions with the OpenTelemetry Java agent](https://www.elastic.co/observability-labs/blog/using-the-otel-operator-for-injecting-elastic-agents#using-an-extension-with-the-opentelemetry-java-agent) for an example of adding an extension to an agent.
+
+## Instrument a Java app with EDOT Java SDK on Kubernetes 
+
+In this example, you'll learn how to:
+
+- Enable auto-instrumentation of a Java application using one of the following supported methods:
+  - Adding an annotation to the deployment Pods.
+  - Adding an annotation to the namespace.
+- Verify that auto-instrumentation libraries are injected and configured correctly.
+- Confirm data is flowing to **Kibana Observability**.
+
+For this example, we assume the application you're instrumenting is a deployment named `java-app` running in the `java-ns` namespace.
+
+1. Ensure you have successfully [installed the OpenTelemetry Operator](./README.md), and confirm that the following `Instrumentation` object exists in the system:
+
+```bash
+$ kubectl get instrumentation -n opentelemetry-operator-system
+NAME                      AGE    ENDPOINT                                                                                                
+elastic-instrumentation   107s   http://opentelemetry-kube-stack-daemon-collector.opentelemetry-operator-system.svc.cluster.local:4318
+```
+> [!NOTE]
+> If your `Instrumentation` object has a different name or is created in a different namespace, you will have to adapt the annotation value in the next step.
+
+2. Enable auto-instrumentation of your Java application using one of the following methods:
+
+  - Edit your application workload definition and include the annotation under `spec.template.metadata.annotations`:
+
+    ```yaml
+    kind: Deployment
+    metadata:
+      name: java-app
+      namespace: java-ns
+    spec:
+    ...
+      template:
+        metadata:
+    ...
+          annotations:
+            instrumentation.opentelemetry.io/inject-java: opentelemetry-operator-system/elastic-instrumentation
+    ...
+    ```
+
+  - Alternatively, add the annotation at namespace level to apply auto-instrumentation in all Pods of the namespace:
+
+    ```bash
+    kubectl annotate namespace java-ns instrumentation.opentelemetry.io/inject-java=opentelemetry-operator-system/elastic-instrumentation
+    ```
+
+3. Restart application
+
+  Once the annotation has been set, restart the application to create new Pods and inject the instrumentation libraries:
+
+    ```bash
+    kubectl rollout restart deployment java-app -n java
+    ```
+
+4. Verify the [auto-instrumentation resources](./instrumenting-applications.md#how-auto-instrumentation-works) are injected in the Pod:
+
+  Run a `kubectl describe` of one of your application Pods and check:
+
+  - There should be an init container named `opentelemetry-auto-instrumentation-java` in the Pod:
+
+    ```bash
+    $ kubectl describe pod java-app-8d84c47b8-8h5z2 -n java
+    Name:             java-app-8d84c47b8-8h5z2
+    Namespace:        java-ns
+    ...
+    ...
+    Init Containers:
+      opentelemetry-auto-instrumentation-java:
+        Container ID:  containerd://cbf67d7ca1bd62c25614b905a11e81405bed6fd215f2df21f84b90fd0279230b
+        Image:         docker.elastic.co/observability/elastic-otel-javaagent:1.0.0
+        Image ID:      docker.elastic.co/observability/elastic-otel-javaagent@sha256:28d65d04a329c8d5545ed579d6c17f0d74800b7b1c5875e75e0efd29e210566a
+        Port:          <none>
+        Host Port:     <none>
+        Command:
+          cp
+          /javaagent.jar
+          /otel-auto-instrumentation-java/javaagent.jar
+        State:          Terminated
+          Reason:       Completed
+          Exit Code:    0
+          Started:      Wed, 13 Nov 2024 15:47:02 +0100
+          Finished:     Wed, 13 Nov 2024 15:47:03 +0100
+        Ready:          True
+        Restart Count:  0
+        Environment:    <none>
+        Mounts:
+          /otel-auto-instrumentation-java from opentelemetry-auto-instrumentation-java (rw)
+          /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-swhn5 (ro)
+    ```
+
+  - The main container of the deployment is using the SDK as `javaagent`: 
+
+    ```bash
+    ...
+    Containers:
+      java-app:
+        Environment:
+    ...
+          JAVA_TOOL_OPTIONS:                      -javaagent:/otel-auto-instrumentation-java/javaagent.jar
+          OTEL_SERVICE_NAME:                     java-app
+          OTEL_EXPORTER_OTLP_ENDPOINT:           http://opentelemetry-kube-stack-daemon-collector.opentelemetry-operator-system.svc.cluster.local:4318
+    ...
+    ```
+
+  - The Pod has an `EmptyDir` volume named `opentelemetry-auto-instrumentation-java` mounted in both the main and the init containers in path `/otel-auto-instrumentation-java`.
+
+    ```bash
+    Init Containers:
+      opentelemetry-auto-instrumentation-java:
+    ...
+        Mounts:
+          /otel-auto-instrumentation-java from opentelemetry-auto-instrumentation-java (rw)
+    Containers:
+      java-app:
+    ...  
+        Mounts:
+          /otel-auto-instrumentation-java from opentelemetry-auto-instrumentation-java (rw)
+    ...
+    Volumes:
+    ...
+      opentelemetry-auto-instrumentation-java:
+        Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
+    ```
+
+  Ensure the environment variable `OTEL_EXPORTER_OTLP_ENDPOINT` points to a valid endpoint and there's network communication between the Pod and the endpoint.
+
+5. Confirm data is flowing to **Kibana**:
+
+  - Open **Observability** -> **Applications** -> **Service inventory**, and determine if:
+    - The application appears in the list of services.
+    - The application shows transactions and metrics.
+
+  - For application container logs, open **Kibana Discover** and filter for your Pods' logs. In the provided example, we could filter for them with either of the following:
+    - `k8s.deployment.name: "java-app"` (**adapt the query filter to your use case**)
+    - `k8s.pod.name: java-app*` (**adapt the query filter to your use case**)
+
+  Note that the container logs are not provided by the instrumentation library, but by the DaemonSet collector deployed as part of the [operator installation](./README.md).
+
+## Troubleshooting
+
+- Refer to [troubleshoot auto-instrumentation](./troubleshoot-auto-instrumentation.md) for further analysis.