Skip to content

Commit

Permalink
feat: GCP logging and friends πŸ„β€β™‚οΈ (#2369)
Browse files Browse the repository at this point in the history
  • Loading branch information
talboren authored Oct 31, 2024
1 parent ec19ae4 commit 23d4c65
Show file tree
Hide file tree
Showing 14 changed files with 433 additions and 57 deletions.
1 change: 1 addition & 0 deletions docs/mint.json
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,7 @@
"providers/documentation/netdata-provider",
"providers/documentation/new-relic-provider",
"providers/documentation/ntfy-provider",
"providers/documentation/openai-provider",
"providers/documentation/openobserve-provider",
"providers/documentation/openshift-provider",
"providers/documentation/opsgenie-provider",
Expand Down
117 changes: 76 additions & 41 deletions docs/providers/documentation/gcpmonitoring-provider.mdx
Original file line number Diff line number Diff line change
@@ -1,78 +1,113 @@
---
title: "GCP Monitoring"
sidebarTitle: "GCP Monitoring Provider"
description: "GCP Monitoringing provider allows you to get alerts from Azure Monitoring via webhooks."
description: "GCP Monitoring provider allows you to get alerts and logs from GCP Monitoring via webhooks and log queries."
---

## Overview
The GCP Monitoring Provider enables seamless integration between Keep and GCP Monitoring, allowing alerts from GCP Monitoring to be directly sent to Keep through webhook configurations. This integration ensures that critical alerts are efficiently managed and responded to within Keep's platform.

The GCP Monitoring Provider enables seamless integration between Keep and GCP Monitoring, allowing alerts from GCP Monitoring to be directly sent to Keep through webhook configurations. In addition to alerts, the provider now supports querying log entries from GCP Logging, enabling a comprehensive view of alerts and associated logs within Keep's platform.

## Connecting GCP Monitoring to Keep
To connect GCP Monitoring to Keep, you'll need to configure a webhook as a notification channel in GCP Monitoring and then link it to the desired alert policy.

### Alert Integration via Webhook

To connect GCP Monitoring alerts to Keep, configure a webhook as a notification channel in GCP Monitoring and link it to the desired alert policy.

### Step 1: Access Notification Channels

Log in to the Google Cloud Platform console.
Navigate to **Monitoring > Alerting > Notification channels**.

<Frame
width="100"
height="200">
<img height="10" src="/images/gcpmonitoring_1.png" />
<Frame width="100" height="200">
<img height="10" src="/images/gcpmonitoring_1.png" />
</Frame>

### Step 2: Add a New Webhook

Within the Webhooks section, click on **ADD NEW**.

<Frame
width="100"
height="200">
<img height="10" src="/images/gcpmonitoring_2.png" />
<Frame width="100" height="200">
<img height="10" src="/images/gcpmonitoring_2.png" />
</Frame>

### Step 3: Configure the Webhook

In the Endpoint URL field, enter the webhook URL provided by Keep.
- For Display Name, use keep-gcpmonitoring-webhook-integration.
- Enable Use HTTP Basic Auth and input the following credentials:
- Auth Username: **api_key**
- Auth Password: **%YOURAPIKEY%**

<Frame
width="100"
height="200">
<img height="10" src="/images/gcpmonitoring_3.png" />

- **Display Name**: keep-gcpmonitoring-webhook-integration
- Enable **Use HTTP Basic Auth** and input the following credentials:
- **Auth Username**: `api_key`
- **Auth Password**: `%YOURAPIKEY%`

<Frame width="100" height="200">
<img height="10" src="/images/gcpmonitoring_3.png" />
</Frame>

### Step 4: Save the Webhook Configuration
- Click on Save to store the webhook configuration.

- Click **Save** to store the webhook configuration.

### Step 5: Associate the Webhook with an Alert Policy

Navigate to the alert policy you wish to send notifications from to Keep.
- Click on Edit.
- Under "Notifications and name," find the Notification Channels section and select the keep-gcpmonitoring-webhook-integration channel you created.
- Save the changes by clicking on SAVE POLICY.

<Frame
width="100"
height="200">
<img height="10" src="/images/gcpmonitoring_4.png" />
</Frame>

- Click **Edit**.
- Under "Notifications and name," find the **Notification Channels** section and select the `keep-gcpmonitoring-webhook-integration` channel you created.
- Save the changes by clicking on **SAVE POLICY**.

<Frame width="100" height="200">
<img height="10" src="/images/gcpmonitoring_4.png" />
</Frame>

<Frame
width="100"
height="200">
<img height="10" src="/images/gcpmonitoring_5.png" />
<Frame width="100" height="200">
<img height="10" src="/images/gcpmonitoring_5.png" />
</Frame>

### Step 6: Review the alert in Keep
### Step 6: Review the Alert in Keep

<Frame
width="100"
height="200">
<img height="10" src="/images/gcpmonitoring_6.png" />
Once the setup is complete, alerts from GCP Monitoring will start appearing in Keep.

<Frame width="100" height="200">
<img height="10" src="/images/gcpmonitoring_6.png" />
</Frame>

## Log Query Integration

The GCP Monitoring Provider also supports querying logs from GCP Logging, allowing you to fetch log entries based on specific filters. This is helpful for enriching alert data with related logs or for monitoring specific events in Keep.

### Authentication Requirements

To enable log querying, you need to provide a service account JSON file with the `logs.viewer` role. This service account should be configured in the `authentication` section of your GCP Monitoring Provider configuration.

### Querying Logs

The provider’s `query` function supports filtering logs based on criteria such as resource type, severity, or specific keywords. You can specify a time range for querying logs using `timedelta_in_days`, and control the number of entries with `page_size`.

#### Example Usage

Here’s an example of how you might use the provider to query log entries:

```python
query(filter='resource.type="cloud_run_revision" AND severity="ERROR"', timedelta_in_days=1)
```

This will return logs of severity β€œERROR” related to Cloud Run revisions from the past day.

#### Log Scopes

To read logs, the provider requires the following IAM role:

β€’ roles/logs.viewer - Allows the provider to read log entries.

#### Post Installation Validation

To validate both alerts and logs, follow these steps:

1. Alert Validation: Test the webhook by triggering an alert in GCP Monitoring and confirm it appears in Keep.
2. Log Query Validation: Execute a simple log query and verify that log entries are returned as expected.

### Useful Links
- [GCP Monitoring Notification Channels](https://cloud.google.com/monitoring/support/notification-options)
- [GCP Monitoring Alerting](https://cloud.google.com/monitoring/alerts)

- [GCP Monitoring Notification Channels](https://cloud.google.com/monitoring/support/notification-options)
- [GCP Monitoring Alerting](https://cloud.google.com/monitoring/alerts)
38 changes: 38 additions & 0 deletions docs/providers/documentation/openai-provider.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
---
title: "OpenAI Provider"
description: "The OpenAI Provider allows for integrating OpenAI's language models into Keep."
---

<Tip>
The OpenAI Provider supports querying GPT language models for prompt-based
interactions.
</Tip>

## Inputs

The OpenAI Provider supports the following functions:

- `query`: Interact with OpenAI's models by sending prompts and receiving responses
- `model`: The model to be used, defaults to `gpt-3.5-turbo`

## Outputs

Currently, the OpenAI Provider outputs the response from the model based on the prompt provided.

## Authentication Parameters

To use the OpenAI Provider, you'll need an API Key, and optionally, an Organization ID from OpenAI. The required parameters for authentication are:

- **api_key** (required): Your OpenAI Platform API Key.
- **organization_id** (optional): Your OpenAI Platform Organization ID.

## Connecting with the Provider

To connect to OpenAI, you'll need to obtain an API Key and (optionally) an Organization ID:

1. Log in to your OpenAI account at [OpenAI Platform](https://platform.openai.com).
2. Go to the **API Keys** section.
3. Click on **Create new secret key** to generate a key for Keep.
4. (Optional) Retrieve your **Organization ID** under **Organization settings** if you’re part of multiple organizations.

Use the generated API key in the `authentication` section of your OpenAI Provider configuration.
2 changes: 2 additions & 0 deletions docs/providers/documentation/slack-provider.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,8 @@ The `notify` function take following parameters as inputs:
- `message`: Required. Message text to send to Slack
- `blocks`: Optional. Array of interactive components like inputs, buttons
- `channel`: Optional. The channel ID to send to if using the OAuth integration.
- `thread_timestamp`: Optional. The timestamp of the thread to update if using the OAuth integration.
- `slack_timestamp`: Optional. The timestamp of the message to update if using the OAuth integration.


## Useful Links
Expand Down
8 changes: 8 additions & 0 deletions docs/providers/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -436,6 +436,14 @@ By leveraging Keep Providers, users are able to deeply integrate Keep with the t
}
></Card>

<Card
title="OpenAI"
href="/providers/documentation/openai-provider"
icon={
<img src="https://img.logo.dev/openai.com?token=pk_dfXfZBoKQMGDTIgqu7LvYg" />
}
></Card>

<Card
title="OpenObserve"
href="/providers/documentation/openobserve-provider"
Expand Down
41 changes: 41 additions & 0 deletions examples/workflows/gcp_logging_open_ai.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
id: enrich-gcp-alert
description: Enriched GCP Alert
disabled: false
triggers:
- type: manual
- filters:
- key: source
value: gcpmonitoring
type: alert
consts: {}
name: 5a76aa52-4e0f-43c3-85ff-5603229c5d7e
owners: []
services: []
steps:
- name: gcpmonitoring-step
provider:
config: "{{ providers.gcp }}"
type: gcpmonitoring
with:
as_json: false
filter: resource.type = "cloud_run_revision" {{alert.traceId}}
page_size: 1000
raw: false
timedelta_in_days: 1
- name: openai-step
provider:
config: "{{ providers.openai }}"
type: openai
with:
prompt:
"You are a very talented engineer that receives context from GCP logs
about an endpoint that returned 500 status code and reports back the root
cause analysis. Here is the context: keep.json_dumps({{steps.gcpmonitoring-step.results}}) (it is a JSON list of log entries from GCP Logging).
In your answer, also provide the log entry that made you conclude the root cause and specify what your certainty level is that it is the root cause. (between 1-10, where 1 is low and 10 is high)"
actions:
- name: slack-action
provider:
config: "{{ providers.slack }}"
type: slack
with:
message: "{{steps.openai-step.results}}"
Binary file added keep-ui/public/icons/openai-icon.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
7 changes: 6 additions & 1 deletion keep/api/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -190,8 +190,13 @@ def get_app(
app = FastAPI(
title="Keep API",
description="Rest API powering https://platform.keephq.dev and friends πŸ„β€β™€οΈ",
version="0.1.0",
version=KEEP_VERSION,
)

@app.get("/")
async def root():
return {"message": app.description, "version": KEEP_VERSION}

app.add_middleware(RawContextMiddleware, plugins=(plugins.RequestIdPlugin(),))
app.add_middleware(
GZipMiddleware, minimum_size=30 * 1024 * 1024
Expand Down
Loading

0 comments on commit 23d4c65

Please sign in to comment.