feat: BaseIncidentProvider, PagerDuty incidents and process_incident

docs/providers/documentation/pagerduty-provider.mdx

@@ -1,12 +1,16 @@
 ---
 title: "Pagerduty Provider"
-description: "Pagerduty Provider is a provider that allows to create incidents or post events to Pagerduty."
+description: "Pagerduty Provider allows integration with PagerDuty to create, manage, and synchronize incidents and alerts within Keep."
 ---
 
+## Description
+
+The Pagerduty Provider enables integration with PagerDuty to create, manage, and synchronize incidents and alerts within Keep. It supports both direct API key authentication and OAuth2, allowing greater flexibility for secure integration.
+
 ## Inputs
 
 - `title`: str: Title of the alert or incident.
-- `alert_body`: str: UTF-8 string of custom message for alert. Shown in incident body for events, and in the body for incidents.
+- `alert_body`: dict: https://developer.pagerduty.com/api-reference/a7d81b0e9200f-create-an-incident#request-body
 - `dedup`: str | None: Any string, max 255 characters, used to deduplicate alerts for events.
 - `service_id`: str: ID of the service for incidents.
 - `body`: dict: Body of the incident.
@@ -15,39 +19,92 @@ description: "Pagerduty Provider is a provider that allows to create incidents o
 
 ## Authentication Parameters
 
-The `api_key` or `routing_key` are required for connecting to the Pagerduty provider. You can obtain them as described in the "Connecting with the Provider" section.
+PagerDuty supports two authentication methods:
 
-Routing key, which is an integration or ruleset key. API key, which is a user or team API key.
+1. **API Key** - A user or team key, accessible through **Configuration > API Access** in PagerDuty.
+2. **OAuth2** - Supports installation with OAuth2, with access and refresh tokens managed within Keep.
 
 ## Connecting with the Provider
 
-To use the PagerdutyProvider, you'll need to provide either a routing_key or an api_key.
+To connect Keep to PagerDuty:
+
+- **Routing Key**: Use for event posting via the PagerDuty Events API.
+- **API Key**: Use for incident creation and management through the PagerDuty Incidents API.
+- **OAuth2**: Token management handled automatically by Keep.
+
+<Frame>
+  <img src="/images/connect-to-pagerduty.png" />
+</Frame>
 
+<Note>
 You can find your integration key or routing key in the PagerDuty web app under **Configuration** > **Integrations**, and select the integration you want to use.
 You can find your API key in the PagerDuty web app under **Configuration** > **API Access**.
 
-The routing_key is used to post events to Pagerduty using the events API.
+The routing_key is used to post events to PagerDuty using the events API.
 The api_key is used to create incidents using the incidents API.
 
+</Note>
+
+### Enabling OAuth in the open-source version
+
+If you would like to use OAuth in the open-source, where you self-host Keep, you can do so by following these step:
+1. Create a PagerDuty account
+2. In the account page, go to **Integrations** > **App Registration**
+<Frame>
+  <img src="/images/pagerduty-app-registration.png" />
+</Frame>
+3. Click on **New App** blue button on the top right
+4. Fill in the required fields
+5. Select "OAuth 2.0" in the Functionality section and click **Next**
+6. In the Redirect URL, you need to add Keep's PagerDuty OAuth2 redirect URL, which is based on your deployments URL. For example, if Keep is deployed at http://localhost:3000, the redirect URL is http://localhost:3000/providers/oauth2/pagerduty
+<Frame>
+  <img src="/images/pagerduty-redirect-url.png" />
+</Frame>
+7. In the Authorization section, select **Scoped OAuth** and select the following scopes:
+- Abilities: Read Access
+- Incidents: Read/Write Access
+- Services: Read/Write Access
+- Webhook Subscriptions: Read/Write Access
+8. Click on **Register App** blue button on the bottom right
+9. Copy the **Client ID** and **Client Secret** from the OAuth 2.0 Client Information modal and set the `PAGERDUTY_CLIENT_ID` and `PAGERDUTY_CLIENT_SECRET` environment variables in your Keep backend deployment.
+<Frame>
+  <img src="/images/pagerduty-oauth2-credentials.png" />
+</Frame>
+
+## PagerDuty Webhook Integration
+
+By default, when Keep installs itself as a webhook integration, it subscribes to all incident events ("Account Scope").
+
+<Frame>
+  <img src="/images/pagerduty-account-scope.png" />
+</Frame>
+
+If you wish to limit Keep to some specific services, you can do so by selecting the **Service** scope and selecting the services you want to subscribe to.
+
+<Frame>
+  <img src="/images/pagerduty-service-scope.png" />
+</Frame>
+
+Find this page under **Integrations** > **Generic Webhooks (v3)**
+
 ## Scopes
 
 Certain scopes may be required to perform specific actions or queries via the Pagerduty Provider. Below is a summary of relevant scopes and their use cases:
 
 - incidents_read (Incidents Read)
-    Required: True
-    Description: View incidents.
+  Required: True
+  Description: View incidents.
 - incidents_write (Incidents Write)
-    Required: False
-    Description: Write incidents.
+  Required: False
+  Description: Write incidents.
 - webhook_subscriptions_read (Webhook Subscriptions Read)
-    Required: False
-    Description: View webhook subscriptions.
-    (*Required for auto-webhook integration)
+  Required: False
+  Description: View webhook subscriptions.
+  (\*Required for auto-webhook integration)
 - webhook_subscriptions_write (Webhook Subscriptions Write)
-    Required: False
-    Description: Write webhook subscriptions.
-    (*Required for auto-webhook integration)
-
+  Required: False
+  Description: Write webhook subscriptions.
+  (\*Required for auto-webhook integration)
 
 ## Notes
 
@@ -59,6 +116,7 @@ An expired trial while using the free version of PagerDuty may result in the "pa
 
 The webhook integration adds Keep as a destination within the "Integrations" API within Pagerduty.
 This grants Keep access to the following scopes within Pagerduty:
+
 - `webhook_subscriptions_read`
 - `webhook_subscriptions_write`
 

keep-ui/app/globals.css

@@ -25,7 +25,7 @@
 .page-container {
   @apply p-4 mx-auto w-full;
   @screen xl {
-    @apply p-10;
+    @apply p-6;
   }
 }
 

keep-ui/app/providers/filter-context/constants.ts

@@ -7,6 +7,7 @@ export const PROVIDER_LABELS: Record<TProviderLabels, string> = {
   ticketing: 'Ticketing',
   data: 'Data',
   queue: 'Queue',
+  incident: 'Incident'
 }
 
 export const PROVIDER_LABELS_KEYS = Object.keys(PROVIDER_LABELS);

keep-ui/app/providers/oauth2/[providerType]/page.tsx

@@ -16,6 +16,8 @@ export default async function InstallFromOAuth({
   const apiUrl = getApiURL();
   const cookieStore = cookies();
   const verifier = cookieStore.get("verifier");
+  const installWebhook = cookieStore.get("oauth2_install_webhook");
+  const pullingEnabled = cookieStore.get("oauth2_pulling_enabled");
 
   const response = await fetch(
     `${apiUrl}/providers/install/oauth2/${params.providerType}`,
@@ -29,6 +31,8 @@ export default async function InstallFromOAuth({
         ...searchParams,
         redirect_uri: `${process.env.NEXTAUTH_URL}/providers/oauth2/${params.providerType}`,
         verifier: verifier ? verifier.value : null,
+        install_webhook: installWebhook ? installWebhook.value : false,
+        pulling_enabled: pullingEnabled ? pullingEnabled.value : false,
       }),
       cache: "no-store",
     }

keep-ui/app/providers/provider-form.tsx

@@ -220,6 +220,8 @@ const ProviderForm = ({
     e.preventDefault();
     const verifier = generateRandomString();
     cookieCutter.set("verifier", verifier);
+    cookieCutter.set("oauth2_install_webhook", formValues.install_webhook);
+    cookieCutter.set("oauth2_pulling_enabled", formValues.pulling_enabled);
     const verifierChallenge = base64urlencode(await sha256(verifier));
 
     let oauth2Url = provider.oauth2_url;

keep-ui/app/providers/provider-tile.tsx

@@ -11,6 +11,7 @@ import {
   BellAlertIcon,
   ChatBubbleBottomCenterIcon,
   CircleStackIcon,
+  ExclamationTriangleIcon,
   QueueListIcon,
   TicketIcon,
   MapIcon,
@@ -101,6 +102,8 @@ function getIconForTag(tag: TProviderLabels) {
       return QueueListIcon;
     case "topology":
       return MapIcon;
+    case "incident":
+      return ExclamationTriangleIcon;
     default:
       return ChatBubbleBottomCenterIcon;
   }

keep-ui/app/providers/providers.tsx

@@ -53,6 +53,7 @@ interface AlertDistritbuionData {
 
 export type TProviderLabels =
   | "alert"
+  | "incident"
   | "topology"
   | "messaging"
   | "ticketing"

keep-ui/features/incident-list/ui/incident-table-component.tsx

@@ -51,8 +51,8 @@ const SortableHeaderCell = ({ header, children }: SortableHeaderCellProps) => {
                 column.getNextSortingOrder() === "asc"
                   ? "Sort ascending"
                   : column.getNextSortingOrder() === "desc"
-                    ? "Sort descending"
-                    : "Clear sort"
+                  ? "Sort descending"
+                  : "Clear sort"
               }
               icon={
                 column.getIsSorted()
@@ -77,14 +77,17 @@ export const IncidentTableComponent = (props: Props) => {
   return (
     <Table>
       <TableHead>
-        {table.getHeaderGroups().map((headerGroup) => (
+        {table.getHeaderGroups().map((headerGroup, index) => (
           <TableRow
             className="border-b border-tremor-border dark:border-dark-tremor-border"
-            key={headerGroup.id}
+            key={`${headerGroup.id}-${index}`}
           >
-            {headerGroup.headers.map((header) => {
+            {headerGroup.headers.map((header, index) => {
               return (
-                <SortableHeaderCell header={header} key={header.id}>
+                <SortableHeaderCell
+                  header={header}
+                  key={`${header.id}-${index}`}
+                >
                   {flexRender(
                     header.column.columnDef.header,
                     header.getContext()

keep/api/arq_worker.py

@@ -33,6 +33,10 @@
             "keep.api.tasks.process_topology_task.async_process_topology",
             KEEP_ARQ_QUEUE_BASIC,
         ),
+        (
+            "keep.api.tasks.process_incident_task.async_process_incident",
+            KEEP_ARQ_QUEUE_BASIC,
+        ),
     ]
 
 if KEEP_ARQ_TASK_POOL in [KEEP_ARQ_TASK_POOL_ALL, KEEP_ARQ_TASK_POOL_AI]:
@@ -63,7 +67,8 @@
 
 FUNCTIONS: list = (
     [
-        import_string(background_function) for background_function in list(ARQ_BACKGROUND_FUNCTIONS)
+        import_string(background_function)
+        for background_function in list(ARQ_BACKGROUND_FUNCTIONS)
     ]
     if ARQ_BACKGROUND_FUNCTIONS is not None
     else list()
@@ -85,15 +90,15 @@ def get_arq_worker(queue_name: str) -> Worker:
     expires = config(
         "ARQ_EXPIRES", cast=int, default=3600
     )  # the default length of time from when a job is expected to start after which the job expires, making it shorter to avoid clogging
-    expires_ai = config(
-        "ARQ_EXPIRES_AI", cast=int, default=3600*1000
-    )
+    expires_ai = config("ARQ_EXPIRES_AI", cast=int, default=3600 * 1000)
     # generate a worker id so each worker will have a different health check key
     worker_id = str(uuid4()).replace("-", "")
     worker = create_worker(
         WorkerSettings,
         keep_result=keep_result,
-        expires_extra_ms=expires_ai if KEEP_ARQ_TASK_POOL == KEEP_ARQ_TASK_POOL_AI else expires,
+        expires_extra_ms=(
+            expires_ai if KEEP_ARQ_TASK_POOL == KEEP_ARQ_TASK_POOL_AI else expires
+        ),
         queue_name=queue_name,
         health_check_key=f"{queue_name}:{worker_id}:health-check",
     )

keep/api/bl/ai_suggestion_bl.py

@@ -324,7 +324,7 @@ async def commit_incidents(
             try:
                 # Create the incident
                 incident_dto = IncidentDto.parse_obj(incident_with_feedback["incident"])
-                created_incident = incident_bl.create_incident(incident_dto)
+                created_incident = incident_bl.create_incident(incident_dto, generated_from_ai=True)
 
                 # Add alerts to the created incident
                 alert_ids = [

keep/api/bl/incidents_bl.py

@@ -12,13 +12,13 @@
 from keep.api.arq_pool import get_pool
 from keep.api.core.db import (
     add_alerts_to_incident_by_incident_id,
-    create_incident_from_dto,
     delete_incident_by_id,
     get_incident_alerts_by_incident_id,
     get_incident_by_id,
     get_incident_unique_fingerprint_count,
     remove_alerts_to_incident_by_incident_id,
     update_incident_from_dto_by_id,
+    create_incident_from_dto,
 )
 from keep.api.core.elastic import ElasticClient
 from keep.api.models.alert import IncidentDto, IncidentDtoIn
@@ -52,12 +52,26 @@ def __init__(
         self.ee_enabled = os.environ.get("EE_ENABLED", "false").lower() == "true"
         self.redis = os.environ.get("REDIS", "false") == "true"
 
-    def create_incident(self, incident_dto: IncidentDtoIn) -> IncidentDto:
+    def create_incident(
+        self, incident_dto: IncidentDtoIn, generated_from_ai: bool = False
+    ) -> IncidentDto:
+        """
+        Creates a new incident.
+
+        Args:
+            incident_dto (IncidentDtoIn): The data transfer object containing the details of the incident to be created.
+            generated_from_ai (bool, optional): Indicates if the incident was generated by Keep's AI. Defaults to False.
+
+        Returns:
+            IncidentDto: The newly created incident object, containing details of the incident.
+        """
         self.logger.info(
             "Creating incident",
             extra={"incident_dto": incident_dto.dict(), "tenant_id": self.tenant_id},
         )
-        incident = create_incident_from_dto(self.tenant_id, incident_dto)
+        incident = create_incident_from_dto(
+            self.tenant_id, incident_dto, generated_from_ai=generated_from_ai
+        )
         self.logger.info(
             "Incident created",
             extra={"incident_id": incident.id, "tenant_id": self.tenant_id},