Merge branch 'main' into feat-1770-add-time-frame

Signed-off-by: Rajesh Jonnalagadda <[email protected]>

.gitignore

@@ -205,4 +205,7 @@ playwright_dump_*.html
 playwright_dump_*.png
 
 ee/experimental/ai_temp/*
-!ee/experimental/ai_temp/.gitkeep
+,e!ee/experimental/ai_temp/.gitkeep
+
+oauth2.cfg
+scripts/keep_slack_bot.py

docs/mint.json

@@ -45,6 +45,7 @@
           ]
         },
         "overview/maintenance-windows",
+        "overview/deduplication",
         "overview/examples"
       ]
     },
@@ -85,12 +86,9 @@
       "group": "Platform",
       "pages": [
         "platform/overview",
-        "platform/providers",
         "platform/alerts",
         "platform/alertseverityandstatus",
-        "platform/workflows",
         "platform/settings",
-        "platform/metrics",
         "platform/support"
       ]
     },
@@ -187,7 +185,6 @@
       "group": "Workflows",
       "pages": [
         "workflows/overview",
-        "workflows/getting-started",
         {
           "group": "Syntax",
           "pages": [

docs/overview/deduplication.mdx

@@ -0,0 +1,107 @@
+---
+title: "Alert Deduplication"
+---
+
+## Overview
+
+Alert deduplication is a crucial feature in Keep that helps reduce noise and streamline incident management by grouping similar alerts together. This process ensures that your team isn't overwhelmed by a flood of notifications for what is essentially the same issue, allowing for more efficient and focused incident response.
+
+## Glossary
+
+- **Deduplication Rule**: A set of criteria used to determine if alerts should be grouped together.
+- **Partial Deduplication**: Correlates instances of alerts into single alerts, considering the case of the same alert with different statuses (e.g., firing and resolved). This is the default mode where specified fields are used to identify and group related alerts.
+- **Fingerprint Fields**: Specific alert attributes used to identify similar alerts.
+- **Full Deduplication**: A mode where alerts are considered identical if all fields match exactly (except those explicitly ignored). This helps avoid system overload by discarding duplicate alerts.
+- **Ignore Fields**: In full deduplication mode, these are fields that are not considered when comparing alerts.
+
+## Deduplication Types
+
+### Partial Deduplication
+Partial deduplication allows you to specify certain fields (fingerprint fields) that are used to identify similar alerts. Alerts with matching values in these specified fields are considered duplicates and are grouped together. This method is flexible and allows for fine-tuned control over how alerts are deduplicated.
+
+Every provider integrated with Keep comes with pre-built partial deduplication rule tailored to that provider's specific alert format and common use cases.
+The default fingerprint fields defined using `FINGERPRINT_FIELDS` attributes in the provider code (e.g. [datadog provider](https://github.com/keephq/keep/blob/main/keep/providers/datadog_provider/datadog_provider.py#L188) or [gcp monitoring provder](https://github.com/keephq/keep/blob/main/keep/providers/gcpmonitoring_provider/gcpmonitoring_provider.py#L52)).
+
+### Full Deduplication
+When full deduplication is enabled, Keep will also discard exact same events (excluding ignore fields). This mode considers all fields of an alert when determining duplicates, except for explicitly ignored fields.
+
+By default, exact similar events excluding lastReceived time are fully deduplicated and discarded. This helps prevent system overload from repeated identical alerts.
+
+## Real Examples of Alerts and Results
+
+### Example 1: Partial Deduplication
+
+**Rule** - Deduplicate based on 'service' and 'error_message' fields.
+
+```json
+# alert 1
+{
+    "service": "payment",
+    "error_message": "Database connection failed",
+    "severity": "high",
+    "lastReceived": "2023-05-01T10:00:00Z"
+}
+# alert 2
+{
+    "service": "payment",
+    "error_message": "Database connection failed",
+    "severity": "critical",
+    "lastReceived": "2023-05-01T10:05:00Z"
+}
+# alert 3
+{
+    "service": "auth",
+    "error_message": "Invalid token",
+    "severity": "medium",
+    "lastReceived": "2023-05-01T10:10:00Z"
+}
+```
+
+**Result**:
+- Alerts 1 and 2 are deduplicated into a single alert, fields are updated.
+- Alert 3 remains separate as it has a different service and error message.
+
+### Example 2: Full Deduplication
+
+**Rule**: Full deduplication with 'timestamp' as an ignore field
+
+**Incoming Alerts**:
+
+```json
+
+# alert 1
+{
+    service: "api",
+    error: "Rate limit exceeded",
+    user_id: "12345",
+    lastReceived: "2023-05-02T14:00:00Z"
+}
+# alert 2 (discarded as its identical)
+{
+    service: "api",
+    error: "Rate limit exceeded",
+    user_id: "12345",
+    lastReceived: "2023-05-02T14:01:00Z"
+}
+# alert 3
+{
+    service: "api",
+    error: "Rate limit exceeded",
+    user_id: "67890",
+    lastReceived: "2023-05-02T14:02:00Z"
+}
+```
+
+**Result**:
+- Alerts 1 and 2 are deduplicated as they are identical except for the ignored timestamp field.
+- Alert 3 remains separate due to the different user_id.
+
+## How It Works
+
+Keep's deduplication process follows these steps:
+
+1. **Alert Ingestion**: Every alert received by Keep is first ingested into the system.
+
+2. **Enrichment**: After ingestion, each alert undergoes an enrichment process. This step adds additional context or information to the alert, enhancing its value and usefulness.
+
+3. **Deduplication**: Following enrichment, Keep's alert deduplicator comes into play. It applies the defined deduplication rules to the enriched alerts.

docs/overview/examples.mdx

@@ -14,7 +14,7 @@ In this example we will utilze:
 
 This example consists of two steps:
 1. Connect your tools - Datadog, OpsGenie and Postgres.
-2. Create a workflow that is triggered by the alert, runs an SQL query, and decides whether to create an incident.  Once the workflow is created, you can upload it via the [Workflows](/platform/workflows) page.
+2. Create a workflow that is triggered by the alert, runs an SQL query, and decides whether to create an incident.  Once the workflow is created, you can upload it via the [Workflows](https://docs.keephq.dev/workflows/overview) page.
 ```yaml
 alert:
   id: enterprise-tier-alerts

docs/overview/introduction.mdx

@@ -23,4 +23,4 @@ Keep helps with every step of the alert lifecycle:
 5. Summarization - Keep summarizes incidents based on past incidents and a knowledge base (Keep Enterprise only).
 
 ## How does Keep integrate with alerts?
-Alerts can either be [pulled](/platform/alerts#pulled-alerts) by Keep or [pushed](/platform/alerts#pushed-alerts) into it. Keep also offers zero-click alert instrumentation through [webhook installation](/platform/providers#webhook-integration).
+Alerts can either be [pulled](/platform/alerts#pulled-alerts) by Keep or [pushed](/platform/alerts#pushed-alerts) into it. Keep also offers zero-click alert instrumentation through [webhook installation](/providers/overview).

docs/overview/keyconcepts.mdx

@@ -27,7 +27,7 @@ For example, consider Promethues. If you want to push alerts from Promethues to
 'https://api.keephq.dev/alerts/event/prometheus' using API key authentication. Each Provider implements Push mechanism and is documented under the specific Provider page.
 
 #### Push alerts to Keep (Automatic)
-In compatible tools, Keep can automatically integrate with the alerting policy of the source tool and add itself as an alert destination. You can learn more about Webhook Integration [here](/platform/providers#webhook-integration).
+In compatible tools, Keep can automatically integrate with the alerting policy of the source tool and add itself as an alert destination. You can learn more about Webhook Integration [here](/providers/overview).
 Please note that this will slightly modify your monitors/notification policy.
 
 ### Pull alerts by Keep

docs/platform/overview.mdx

@@ -9,7 +9,7 @@ The platform is accessible on https://platform.keephq.dev and let you start the
 
 The platform is currently built on top of:
 
-1. [Providers](/platform/providers) - connect your stack to Keep.
+1. [Providers](/providers/overview) - connect your stack to Keep.
 2. [Alerts](/platform/alerts) - single pane of glass for your alerts.
-3. [Workflows](/platform/workflows) - create automations on top of your alerts (or regardless).
+3. [Workflows](/workflows/overview) - create automations on top of your alerts (or regardless).
 4. [Settings](/platform/settings) - the settings page (add users, etc).

keep-ui/app/deduplication/DeduplicationPlaceholder.tsx

@@ -0,0 +1,30 @@
+import { Fragment, useState } from "react";
+import { Button, Card, Subtitle, Title } from "@tremor/react";
+// import { CorrelationSidebar } from "./CorrelationSidebar";
+import { DeduplicationSankey } from "./DeduplicationSankey";
+
+export const DeduplicationPlaceholder = () => {
+  const [isSidebarOpen, setIsSidebarOpen] = useState(false);
+
+  const onCorrelationClick = () => {
+    setIsSidebarOpen(true);
+  };
+
+  return (
+    <Fragment>
+      <Card className="flex flex-col items-center justify-center gap-y-8 h-full">
+        <div className="text-center space-y-3">
+          <Title className="text-2xl">No Deduplications Yet</Title>
+          <Subtitle className="text-gray-400">
+            Reduce noise by creatiing deduplications.
+          </Subtitle>
+          <Subtitle className="text-gray-400">
+            Start sending alerts or connect providers to create deduplication
+            rules.
+          </Subtitle>
+        </div>
+        <DeduplicationSankey className="max-w-full" />
+      </Card>
+    </Fragment>
+  );
+};