docs(observability): introduce observing ai workflows documentation

Includes a broader refactoring of telemetry plugin docs
firebase · Feb 10, 2025 · ffae50b · ffae50b
1 parent 8dc9ed8
commit ffae50b
Show file tree

Hide file tree

Showing 11 changed files with 171 additions and 66 deletions.
diff --git a/docs/_guides.yaml b/docs/_guides.yaml
@@ -41,8 +41,8 @@ toc:
     path: /docs/genkit/multi-agent
   - title: Evaluation
     path: /docs/genkit/evaluation
-  - title: Observability & monitoring
-    path: /docs/genkit/monitoring
+  - title: Local Observability
+    path: /docs/genkit/local-observability
 
   - heading: Deploying AI workflows
   - title: Deploy with Firebase
@@ -53,6 +53,14 @@ toc:
     path: /docs/genkit/deploy-node
   - title: Authorization and integrity
     path: /docs/genkit/auth
+
+  - heading: Observing AI workflows
+  - title: Getting Started
+    path: /docs/genkit/observability/getting-started
+  - title: Advanced Setup
+    path: /docs/genkit/observability/advanced
+  - title: Troubleshooting
+    path: /docs/genkit/observability/troubleshooting
 
   - heading: Writing plugins
   - title: Overview

diff --git a/docs/deploy-node.md b/docs/deploy-node.md
@@ -1,5 +1,7 @@
-<!-- TODO: Add Next docs too. Maybe we need a web-hosting page that deploy-node
-     and cloud-run links to, which links to express, next, and maybe cloud functions>
+<!-- 
+TODO: Add Next docs too. Maybe we need a web-hosting page that deploy-node
+and cloud-run links to, which links to express, next, and maybe cloud functions 
+-->
 # Deploy flows to any Node.js platform
 
 Firebase Genkit has built-in integrations that help you deploy your flows to

diff --git a/docs/local-observability.md b/docs/local-observability.md
@@ -0,0 +1,22 @@
+# Observability
+
+Genkit provides a robust set of built-in observability features, including tracing and metrics collection powered by [OpenTelemetry](https://opentelemetry.io/). For local observability (e.g. during the development phase), the Genkit Developer UI provides detailed trace viewing and debugging capabilities. For production observability, we provide Genkit Monitoring in the Firebase console via the Firebase plugin. Alternatively, you can export your OpenTelemetry data to the observability tooling of your choice.
+
+## Tracing & Metrics
+
+Genkit automatically collects traces and metrics without requiring explicit configuration, allowing you to observe and debug your Genkit code's behavior in the Developer UI. These traces are stored locally, enabling you to analyze your Genkit flows step-by-step with detailed input/output logging and statistics. In production, traces and metrics can be exported to Firebase Genkit Monitoring for further analysis.
+
+## Logging
+
+Genkit also provides a centralized logging system that can be configured using the logging module. One advantage of using the Genkit provided logger is that logs will automatically be exported to Genkit Monitoring when the Firebase Telemetry plugin is enabled.
+
+```typescript
+import { logger } from 'genkit/logging';
+
+// Set the desired log level
+logger.setLogLevel('debug');
+```
+
+## Production Observability
+
+The [Genkit Monitoring](https://console.firebase.google.com/project/_/genai_monitoring) dashboard helps you to understand the overall health of your Genkit features, as well as debug stability and content issues that may point to problems with your LLM prompts and/or Genkit Flows. See the [Getting Started](./observability/getting-started.md) guide for more details.
diff --git a/docs/monitoring.md b/docs/monitoring.md
diff --git a/docs/observability/advanced.md b/docs/observability/advanced.md
@@ -0,0 +1 @@
+# Genkit Monitoring - Advanced Configuration
diff --git a/docs/observability/configuration/auth.md b/docs/observability/configuration/auth.md
diff --git a/docs/observability/configuration/telemetry-config.md b/docs/observability/configuration/telemetry-config.md
diff --git a/docs/observability/getting-started.md b/docs/observability/getting-started.md
@@ -0,0 +1,89 @@
+# Get started with Genkit Monitoring
+
+This quickstart guide describes how to set up Firebase Genkit Monitoring for your deployed Genkit features, so that you can collect and view real-time telemetry data. With Firebase Genkit Monitoring, you get visibility into how your Genkit features are performing in production.
+
+Key capabilities include:
+
+* Viewing quantitative metrics like Genkit feature latency, errors, and token usage
+* Inspecting traces to see your Genkit's feature steps, inputs, and outputs, to help with debugging and quality improvement
+* Exporting production traces to run evals within Genkit
+
+Setting up Genkit Monitoring requires completing tasks in both your codebase and on the Google Cloud Console.
+
+## Before you begin
+
+1. If you haven't already, create a Firebase project. 
+
+    In the [Firebase console](https://console.firebase.google.com), click **Add a project**, then follow the on-screen instructions. You can create a new project or add Firebase services to an already existing Google Cloud project.
+2. Ensure your project is on the [Blaze pricing plan](https://firebase.google.com/pricing). 
+
+    Genkit Monitoring relies on telemetry data written to Google Cloud Logging, Metrics, and Trace which are paid services. View the [Google Cloud Observability](http://cloud/stackdriver/pricing#cloud-monitoring-pricing) page for pricing details and to learn about available free tier limits.
+3. Write a Genkit feature by following the [Get Started Guide](https://firebase.google.com/docs/genkit/get-started) and prepare your code for deployment by using one of the following guides:
+    1. [Deploy with Firebase](../firebase.md)
+    2. [Deploy with Cloud Run](../cloud-run.md)
+    3. [Deploy to any Node.js platform](../deploy-node.md)
+
+
+## Step 1. Add the Firebase plugin
+
+Install the `@genkit-ai/firebase` plugin in your project:
+
+```
+npm i –save @genkit-ai/firebase
+```
+
+Import `enableFirebaseTelemetry` into your Genkit configuration file (i.e. where `genkit(...)` is initalized) and call it:
+
+```typescript
+enableFirebaseTelemetry({
+  forceDevExport: false // set to true for local testing
+});
+```
+
+## Step 2. Enable the required APIs
+
+Make sure that the following APIs are enabled for your GCP project:
+
+* [Cloud Logging API](https://console.cloud.google.com/apis/library/logging.googleapis.com)
+* [Cloud Trace API](https://console.cloud.google.com/apis/library/cloudtrace.googleapis.com)
+* [Cloud Monitoring API](https://console.cloud.google.com/apis/library/monitoring.googleapis.com)
+
+These APIs should be listed in the [API dashboard](https://console.cloud.google.com/apis/dashboard) for your project.
+
+## Step 3. Set up permissions
+
+The Firebase plugin needs to authenticate with Google Cloud Logging, Metrics, and Trace services using a _service account_.
+
+Grant the following roles to whichever service account is configured to run your code within the [Google Cloud IAM Console](https://console.cloud.google.com/iam-admin/iam). For Firebase Functions and/or Cloud Run, this is typically the “default compute service account”.
+
+* **Monitoring Metric Writer** (`roles/monitoring.metricWriter`)
+* **Cloud Trace Agent** (`roles/cloudtrace.agent`)
+* **Logs Writer** (`roles/logging.logWriter`)
+
+## Step 4. (Optional) Test your configuration locally
+
+Before deploying, you can run your Genkit code locally to confirm that telemetry data is being collected and viewable in the Genkit Monitoring dashboard.
+
+1. In your Genkit code, set `forceDevExport` to `true` to send telemetry from your local environment.
+2. Use your service account to authenticate and test your configuration.
+
+    > [!TIP]
+    > In order to impersonate the service account, you will need to have the `roles/iam.serviceAccountTokenCreator` [IAM role](https://console.cloud.google.com/iam-admin/iam) applied to your user account.
+
+    With the [Google Cloud CLI tool](https://cloud.google.com/sdk/docs/install?authuser=0), authenticate using the service account:
+
+    ```
+    gcloud auth application-default login --impersonate-service-account <SERVICE_ACCT_EMAIL>
+    ```
+3. Run and invoke your Genkit feature, and then view metrics on the [Genkit Monitoring dashboard](https://firebase.google.com/project/_/genai_monitoring). Allow for up to 5 minutes to collect the first metric (based on the default `metricExportIntervalMillis` setting in the telemetry configuration).
+4. If metrics are not appearing in the Genkit Monitoring dashboard, view the [Troubleshooting](./troubleshooting.md) guide for steps to debug.
+
+## Step 5.  Re-build and deploy code
+
+Re-build, deploy, and invoke your Genkit feature to start collecting data. Once Genkit Monitoring receives your metrics, they can be viewed by visiting the [Genkit Monitoring dashboard](https://firebase.google.com/project/_/genai_monitoring)
+
+Note: It may take up to 5 minutes to collect the first metric (based on the default `metricExportIntervalMillis` setting in the telemetry configuration).
+
+## Next Steps:
+
+View [advanced configuration](./advanced.md) options for the Firebase telemetry plugin.
diff --git a/docs/observability/troubleshooting.md b/docs/observability/troubleshooting.md
@@ -0,0 +1,40 @@
+# Genkit Monitoring - Troubleshooting
+
+## I can’t see traces and/or metrics in Firebase Genkit Monitoring
+
+1. Ensure that the following APIs are enabled for your underlying GCP project:
+    * [Cloud Logging API](https://console.cloud.google.com/apis/library/logging.googleapis.com)
+    * [Cloud Trace API](https://console.cloud.google.com/apis/library/cloudtrace.googleapis.com)
+    * [Cloud Monitoring API](https://console.cloud.google.com/apis/library/monitoring.googleapis.com)
+2. Ensure that the following roles are applied to the service account that is running your code (or service account that has been configured as part of the plugin options) in [Cloud IAM](https://console.cloud.google.com/iam-admin/iam).
+    * **Monitoring Metric Writer** (`roles/monitoring.metricWriter`)
+    * **Cloud Trace Agent** (`roles/cloudtrace.agent`)
+    * **Logs Writer** (`roles/logging.logWriter`)
+3. Inspect the application logs for errors writing to Cloud Logging, Cloud Trace, and/or Cloud Monitoring. On GCP infrastructure (e.g. Firebase Functions, Cloud Run, etc), even when telemetry is misconfigured, logs to stdout/stderr are automatically ingested by the Cloud Logging Agent, allowing you to diagnose issues in the in the [Cloud Logging Console](https://console.cloud.google.com/logs). 
+4. Debug locally:
+
+    Enable dev export:
+
+    ```typescript
+    enableFirebaseTelemetry({
+      forceDevExport: true
+    });
+    ```
+
+    To test with your personal user credentials, authenticate with Google Cloud via the [gcloud CLI](https://cloud.google.com/sdk/docs/install). This can help diagnose enabled/disabled APIs, but will not test the production service account permissions:
+
+    ```
+    gcloud auth application-default login
+    ```
+
+	Alternatively, impersonating the service account will allow you to test production-like access. You will need to have the `roles/iam.serviceAccountTokenCreator` IAM role applied to your user account in order to impersonate service accounts:
+
+    ```
+    gcloud auth application-default login --impersonate-service-account <SERVICE_ACCT_EMAIL>
+    ```
+
+	See the [ADC](https://cloud.google.com/docs/authentication/set-up-adc-local-dev-environment) documentation for more information.
+
+### Telemetry upload reliability in Firebase Functions / Cloud Run
+
+When Genkit is hosted in Google Cloud Run (including Firebase Functions), telemetry data upload may be less reliable as the container switches to the "idle" [lifecycle state](https://cloud.google.com/blog/topics/developers-practitioners/lifecycle-container-cloud-run). If higher reliability is important to you, consider changing [CPU allocation](https://cloud.google.com/run/docs/configuring/cpu-allocation) to "always allocated" in the Google Cloud Console. Note that this impacts pricing.
diff --git a/docs/plugins/google-cloud.md b/docs/plugins/google-cloud.md
@@ -1,10 +1,6 @@
 # Google Cloud plugin
 
-The Google Cloud plugin exports Firebase Genkit telemetry and logging data to
-the 
-[Cloud Observability](https://cloud.google.com/products/operations)
-suite, which powers the 
-[Firebase AI Monitoring](https://forms.gle/Lp5S1NxbZUXsWc457) dashboard.
+The Google Cloud plugin exports Firebase Genkit telemetry and logging data to the [Cloud Observability](https://cloud.google.com/products/operations) suite.
 
 ## Installation
 
@@ -164,10 +160,10 @@ For cases where exporting all traces isn't practical, OpenTelemetry allows trace
 
 There are four preconfigured samplers:
 
-- [AlwaysOnSampler](https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk/trace/src/main/java/io/opentelemetry/sdk/trace/samplers/AlwaysOnSampler.java) - samples all traces
-- [AlwaysOffSampler](https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk/trace/src/main/java/io/opentelemetry/sdk/trace/samplers/AlwaysOffSampler.java) - samples no traces
-- [ParentBased](https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk/trace/src/main/java/io/opentelemetry/sdk/trace/samplers/ParentBasedSampler.java) - samples based on parent span
-- [TraceIdRatioBased](https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk/trace/src/main/java/io/opentelemetry/sdk/trace/samplers/TraceIdRatioBasedSampler.java) - samples a configurable percentage of traces
+- [AlwaysOnSampler](https://github.com/open-telemetry/opentelemetry-js/blob/main/packages/opentelemetry-sdk-trace-base/src/sampler/AlwaysOnSampler.ts) - samples all traces
+- [AlwaysOffSampler](https://github.com/open-telemetry/opentelemetry-js/blob/main/packages/opentelemetry-sdk-trace-base/src/sampler/AlwaysOffSampler.ts) - samples no traces
+- [ParentBased](https://github.com/open-telemetry/opentelemetry-js/blob/main/packages/opentelemetry-sdk-trace-base/src/sampler/ParentBasedSampler.ts) - samples based on parent span
+- [TraceIdRatioBased](https://github.com/open-telemetry/opentelemetry-js/blob/main/packages/opentelemetry-sdk-trace-base/src/sampler/TraceIdRatioBasedSampler.ts) - samples a configurable percentage of traces
 
 #### autoInstrumentation & autoInstrumentationConfig
 

diff --git a/js/plugins/firebase/README.md b/js/plugins/firebase/README.md
@@ -21,12 +21,6 @@ const ai = genkit({
 });
 ```
 
-## Troubleshooting
-
-### Telemetry upload reliability in Firebase Functions / Cloud Run
-
-When Genkit is hosted in Google Cloud Run (including Firebase Functions), telemetry data upload may be less reliable as the container switches to the "idle" [lifecycle state](https://cloud.google.com/blog/topics/developers-practitioners/lifecycle-container-cloud-run). If higher reliability is important to you, consider changing [CPU allocation](https://cloud.google.com/run/docs/configuring/cpu-allocation) to "always allocated" in the Google Cloud Console. Note that this impacts pricing.
-
 The sources for this package are in the main [Genkit](https://github.com/firebase/genkit) repo. Please file issues and pull requests against that repo.
 
 Usage information and reference details can be found in [Genkit documentation](https://firebase.google.com/docs/genkit).