Merge branch 'main' into feat-dashboards-re

docs/deployment/configuration.mdx

@@ -69,7 +69,7 @@ Resource provisioning settings control how Keep sets up initial resources. This
 <Info>
 Authentication configuration determines how Keep verifies user identities and manages access control. These settings are essential for securing your Keep instance and integrating with various authentication providers.
 </Info>
-<Tip>For specifc authentication type configuration, please see [authentication docs](/deployment/authentication/overview).</Tip>
+<Tip>For specific authentication type configuration, please see [authentication docs](/deployment/authentication/overview).</Tip>
 
 
 | Env var | Purpose | Required | Default Value | Valid options |

docs/deployment/ecs.mdx

@@ -102,7 +102,7 @@ sidebarTitle: "AWS ECS"
             - Configuration Type: Configure at task definition creation
             - Volume type: EFS
             - Storage configurations:
-                - File system ID: Select an exisiting EFS filesystem or create a new one
+                - File system ID: Select an existing EFS filesystem or create a new one
                 - Root Directory: /
                 ![Volume Configuration](/images/ecs-task-def-backend5.png)
             - Container mount points:

docs/deployment/kubernetes/installation.mdx

@@ -46,8 +46,11 @@ To read about more installation options, see [ingress-nginx installation docs](h
 </Info>
 ```bash
 # simplest way to install
+# we set snippet-annotations to true to allow rewrites
+#   see https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#allow-snippet-annotations
 helm upgrade --install ingress-nginx ingress-nginx \
   --repo https://kubernetes.github.io/ingress-nginx \
+  --set controller.config.allow-snippet-annotations=true \
   --namespace ingress-nginx --create-namespace
 ```
 

docs/deployment/kubernetes/overview.mdx

@@ -14,5 +14,6 @@ We maintain an opinionated, batteries-included Helm chart, but you can customize
 ## Next steps
 - Install Keep on [Kubernetes](/deployment/kubernetes/installation).
 - Keep's [Helm Chart](https://github.com/keephq/helm-charts).
+- Keep with [Kubernetes Secret Manager](/deployment/secret-manager#kubernetes-secret-manager)
 - Deep dive to Keep's kubernetes [Architecture](/deployment/kubernetes/architecture).
 - Install Keep on [OpenShift](/deployment/kubernetes/openshift).

docs/deployment/secret-manager.mdx

@@ -59,18 +59,97 @@ Usage:
 
 ## Kubernetes Secret Manager
 
-The `KubernetesSecretManager` interfaces with Kubernetes' native secrets system. It manages secrets within a specified Kubernetes namespace and is designed to operate within a Kubernetes cluster.
+### Overview
 
-Configuration:
+The `KubernetesSecretManager` interfaces with Kubernetes' native secrets system.
+
+It manages secrets within a specified Kubernetes namespace and is designed to operate within a Kubernetes cluster.
+
+### Configuration
 
-Set `K8S_NAMESPACE` environment variable to specify the Kubernetes namespace. Defaults to default if not set. Assumes Kubernetes configurations (like service account tokens) are properly set up when running within a cluster.
+- `SECRET_MANAGER_TYPE=k8s`
+- `K8S_NAMESPACE=keep` - environment variable to specify the Kubernetes namespace. Defaults to `.metadata.namespace` if not set. Assumes Kubernetes configurations (like service account tokens) are properly set up when running within a cluster.
 
 Usage:
 
 - Secrets are stored as Kubernetes Secret objects.
 - Provides functionalities to create, retrieve, and delete Kubernetes secrets.
 - Handles base64 encoding and decoding as required by Kubernetes.
 
+### Environment Variables From Secrets
+The Kubernetes Secret Manager integration allows Keep to fetch environment variables from Kubernetes Secrets.
+
+For sensitive environment variables, such as `DATABASE_CONNECTION_STRING`, it is recommended to store as a secret:
+
+#### Creating Database Connection Secret
+```bash
+# Create the base64 encoded string without newline
+CONNECTION_STRING_B64=$(echo -n "mysql+pymysql://user:password@host:3306/dbname" | base64)
+
+# Create the Kubernetes secret
+kubectl create secret generic keep-db-secret \
+  --namespace=keep \
+  --from-literal=connection_string=$(echo -n "mysql+pymysql://user:password@host:3306/dbname" | base64)
+
+# Or using a YAML file:
+cat <<EOF | kubectl apply -f -
+apiVersion: v1
+kind: Secret
+metadata:
+  name: keep-db-secret
+  namespace: keep
+type: Opaque
+data:
+  connection_string: $(echo -n "mysql+pymysql://user:password@host:3306/dbname" | base64)
+EOF
+```
+
+#### Update the helm Values.yaml
+
+After creating the secret, update the `values.yaml` so the helm chart will inject the secret as env var:
+```bash
+backend:
+  enabled: true
+  waitForDatabase: true
+  databaseConnectionStringFromSecret:
+    enabled: true  # Enable using secret for database connection
+    secretName: "keep-db-secret"  # Name of the secret we created
+    secretKey: "connection_string"  # Key in the secret containing our connection string
+```
+
+#### Apply with Helm
+
+```bash
+# If installing for the first time
+helm install keep keephq/keep \
+  -f values.yaml \
+  --namespace keep
+
+# If updating existing installation
+helm upgrade keep keephq/keep \
+  -f values.yaml \
+  --namespace keep
+```
+
+#### Verify the installation
+
+Check if the secret is properly created:
+```bash
+kubectl get secret keep-db-secret -n keep
+```
+
+Verify the content of the secret is correct:
+```bash
+kubectl get secret keep-db-secret -n keep -o jsonpath='{.data.connection_string}' | base64 -d
+```
+
+Verify the pod using the secret:
+```bash
+kubectl get pod -n keep -l app.kubernetes.io/component=backend -o yaml | grep DATABASE_CONNECTION_STRING -A 5
+```
+
+
+
 ## GCP Secret Manager
 
 The `GcpSecretManager` utilizes Google Cloud's Secret Manager service for secret management. It requires setting up with Google Cloud credentials and a project ID.

docs/deployment/stress-testing.mdx

@@ -27,7 +27,7 @@ The primary parameters that affect the specification requirements for Keep are:
 3. **Number of Workflows**: How many automation run as a result of alert.
 
 ### Main Components:
-- **Keep Backend** - API and buisness logic. A container that serves FastAPI on top of gunicorn.
+- **Keep Backend** - API and business logic. A container that serves FastAPI on top of gunicorn.
 - **Keep Frontend** - Web app. A container that serves the react app.
 - **Database** - Stores the alerts and any other operational data.
 - **Elasticsearch** (opt out by default) - Stores alerts as document for better search performance.

docs/overview/comparisons.mdx

@@ -17,7 +17,7 @@ Keep is different because it’s able to correlate alerts between different obse
 |                                       | Keep                                                           | Alternative                  |
 | ------------------------------------- | -------------------------------------------------------------- | ---------------------------- |
 | Aggregates alerts from one platform | ✅                                                           | ✅                            |
-| Aggregates alerts from mutliple platforms | ✅                                                             | ❌                            |
+| Aggregates alerts from multiple platforms | ✅                                                             | ❌                            |
 | Correlates alerts between multiple sources | ✅                                                    | ❌                            |
 | Alerts enrichment                     | ✅                                                             | ❌                            |
 | Open source                           | ✅                                                             | ❌                            |

docs/overview/deduplication.mdx

@@ -20,7 +20,7 @@ Alert deduplication is a crucial feature in Keep that helps reduce noise and str
 Partial deduplication allows you to specify certain fields (fingerprint fields) that are used to identify similar alerts. Alerts with matching values in these specified fields are considered duplicates and are grouped together. This method is flexible and allows for fine-tuned control over how alerts are deduplicated.
 
 Every provider integrated with Keep comes with pre-built partial deduplication rule tailored to that provider's specific alert format and common use cases.
-The default fingerprint fields defined using `FINGERPRINT_FIELDS` attributes in the provider code (e.g. [datadog provider](https://github.com/keephq/keep/blob/main/keep/providers/datadog_provider/datadog_provider.py#L188) or [gcp monitoring provder](https://github.com/keephq/keep/blob/main/keep/providers/gcpmonitoring_provider/gcpmonitoring_provider.py#L52)).
+The default fingerprint fields defined using `FINGERPRINT_FIELDS` attributes in the provider code (e.g. [datadog provider](https://github.com/keephq/keep/blob/main/keep/providers/datadog_provider/datadog_provider.py#L188) or [gcp monitoring provider](https://github.com/keephq/keep/blob/main/keep/providers/gcpmonitoring_provider/gcpmonitoring_provider.py#L52)).
 
 ### Full Deduplication
 When full deduplication is enabled, Keep will also discard exact same events (excluding ignore fields). This mode considers all fields of an alert when determining duplicates, except for explicitly ignored fields.

examples/workflows/cron-digest-alerts.yml

@@ -2,6 +2,7 @@ workflow:
   id: alerts-daily-digest
   description: run alerts digest twice a day (on 11:00 and 14:00)
   triggers:
+    - type: manual
     - type: interval
       cron: 0 11,14 * * *
   steps:
@@ -10,18 +11,16 @@ workflow:
       provider:
         type: keep
         with:
-          filters:
-          # filter out alerts that are closed
-          - key: status
-            value: open
+          version: 2
+          filter: "status == 'firing'"
           timerange:
-            from: "{{ state.workflows.alerts-daily-digest.last_run_time }}"
+            from: "{{ last_workflow_run_time }}"
             to: now
   actions:
     - name: send-digest
       foreach: "{{ steps.get-alerts.results }}"
       provider:
-        type: slack
-        config: "{{ providers.slack }}"
+        type: console
+        config: "{{ providers.console }}"
         with:
-          message: "Open alert: {{ foreach.value.name }}"
+          message: "Open alerts: {{ foreach.value.name }}"

keep-ui/app/incidents/[id]/incident-activity.tsx

@@ -15,7 +15,7 @@ import {
 import { AuditEvent, useAlerts } from "@/utils/hooks/useAlerts";
 import Loading from "@/app/loading";
 import { useCallback, useState, useEffect } from "react";
-import { getApiURL } from "@/utils/apiUrl";
+import { useApiUrl } from "@/utils/hooks/useConfig";
 import { useSession } from "next-auth/react";
 import { KeyedMutator } from "swr";
 import { toast } from "react-toastify";
@@ -59,7 +59,6 @@ export function IncidentActivityChronoItem({ activity }: { activity: any }) {
   );
 }
 
-
 export function IncidentActivityChronoItemComment({
   incident,
   mutator,
@@ -68,7 +67,7 @@ export function IncidentActivityChronoItemComment({
   mutator: KeyedMutator<AuditEvent[]>;
 }) {
   const [comment, setComment] = useState("");
-  const apiUrl = getApiURL();
+  const apiUrl = useApiUrl();
   const { data: session } = useSession();
 
   const onSubmit = useCallback(async () => {

keep/api/routes/alerts.py

@@ -4,7 +4,7 @@
 import json
 import logging
 import os
-from typing import Optional, List
+from typing import List, Optional
 
 import celpy
 from arq import ArqRedis
@@ -25,7 +25,12 @@
 from keep.api.consts import KEEP_ARQ_QUEUE_BASIC
 from keep.api.core.config import config
 from keep.api.core.db import get_alert_audit as get_alert_audit_db
-from keep.api.core.db import get_alerts_by_fingerprint, get_enrichment, get_last_alerts, get_alerts_metrics_by_provider
+from keep.api.core.db import (
+    get_alerts_by_fingerprint,
+    get_alerts_metrics_by_provider,
+    get_enrichment,
+    get_last_alerts,
+)
 from keep.api.core.dependencies import extract_generic_body, get_pusher_client
 from keep.api.core.elastic import ElasticClient
 from keep.api.models.alert import (
@@ -37,15 +42,15 @@
 from keep.api.models.alert_audit import AlertAuditDto
 from keep.api.models.db.alert import AlertActionType
 from keep.api.models.search_alert import SearchAlertsRequest
+from keep.api.models.time_stamp import TimeStampFilter
 from keep.api.tasks.process_event_task import process_event
 from keep.api.utils.email_utils import EmailTemplates, send_email
 from keep.api.utils.enrichment_helpers import convert_db_alerts_to_dto_alerts
+from keep.api.utils.time_stamp_helpers import get_time_stamp_filter
 from keep.identitymanager.authenticatedentity import AuthenticatedEntity
 from keep.identitymanager.identitymanagerfactory import IdentityManagerFactory
 from keep.providers.providers_factory import ProvidersFactory
 from keep.searchengine.searchengine import SearchEngine
-from keep.api.utils.time_stamp_helpers import get_time_stamp_filter
-from keep.api.models.time_stamp import TimeStampFilter
 
 router = APIRouter()
 logger = logging.getLogger(__name__)
@@ -447,19 +452,23 @@ def enrich_alert(
     authenticated_entity: AuthenticatedEntity = Depends(
         IdentityManagerFactory.get_auth_verifier(["write:alert"])
     ),
+    dispose_on_new_alert: Optional[bool] = Query(
+        False, description="Dispose on new alert"
+    ),
 ) -> dict[str, str]:
-    return _enrich_alert(enrich_data, authenticated_entity=authenticated_entity)
+    return _enrich_alert(
+        enrich_data,
+        authenticated_entity=authenticated_entity,
+        dispose_on_new_alert=dispose_on_new_alert,
+    )
 
 
 def _enrich_alert(
     enrich_data: EnrichAlertRequestBody,
-    pusher_client: Pusher = Depends(get_pusher_client),
     authenticated_entity: AuthenticatedEntity = Depends(
         IdentityManagerFactory.get_auth_verifier(["write:alert"])
     ),
-    dispose_on_new_alert: Optional[bool] = Query(
-        False, description="Dispose on new alert"
-    ),
+    dispose_on_new_alert: Optional[bool] = False,
 ) -> dict[str, str]:
     tenant_id = authenticated_entity.tenant_id
     logger.info(
@@ -469,7 +478,6 @@ def _enrich_alert(
             "tenant_id": tenant_id,
         },
     )
-
     try:
         enrichement_bl = EnrichmentsBl(tenant_id)
         # Shahar: TODO, change to the specific action type, good enough for now
@@ -530,6 +538,7 @@ def _enrich_alert(
             logger.exception("Failed to push alert to elasticsearch")
             pass
         # use pusher to push the enriched alert to the client
+        pusher_client = get_pusher_client()
         if pusher_client:
             logger.info("Telling client to poll alerts")
             try:
@@ -770,17 +779,15 @@ def get_alert_quality(
 ):
     logger.info(
         "Fetching alert quality metrics per provider",
-        extra={
-            "tenant_id": authenticated_entity.tenant_id,
-            "fields": fields
-        },
-
+        extra={"tenant_id": authenticated_entity.tenant_id, "fields": fields},
     )
     start_date = time_stamp.lower_timestamp if time_stamp else None
     end_date = time_stamp.upper_timestamp if time_stamp else None
     db_alerts_quality = get_alerts_metrics_by_provider(
-        tenant_id=authenticated_entity.tenant_id, start_date=start_date, end_date=end_date,
-        fields=fields
+        tenant_id=authenticated_entity.tenant_id,
+        start_date=start_date,
+        end_date=end_date,
+        fields=fields,
     )
-    
+
     return db_alerts_quality

keep/contextmanager/contextmanager.py

@@ -35,6 +35,7 @@ def __init__(self, tenant_id, workflow_id=None, workflow_execution_id=None):
             self.click_context = {}
         # last workflow context
         self.last_workflow_execution_results = {}
+        self.last_workflow_run_time = None
         if self.workflow_id:
             try:
                 last_workflow_execution = get_last_workflow_execution_by_workflow_id(
@@ -44,6 +45,7 @@ def __init__(self, tenant_id, workflow_id=None, workflow_execution_id=None):
                     self.last_workflow_execution_results = (
                         last_workflow_execution.results
                     )
+                    self.last_workflow_run_time = last_workflow_execution.started
             except Exception:
                 self.logger.exception("Failed to get last workflow execution")
                 pass
@@ -130,6 +132,7 @@ def get_full_context(self, exclude_providers=False, exclude_env=False):
             "foreach": self.foreach_context,
             "event": self.event_context,
             "last_workflow_results": self.last_workflow_execution_results,
+            "last_workflow_run_time": self.last_workflow_run_time,
             "alert": self.event_context,  # this is an alias so workflows will be able to use alert.source
             "incident": self.incident_context,  # this is an alias so workflows will be able to use alert.source
             "consts": self.consts_context,