Merge branch 'main' into Sparkle37Front

blog/src/pages/2023-08-31-boost-customer-satisfaction.mdx

@@ -17,6 +17,8 @@ export const Line = () => (
 
 ## Boost Customer Satisfaction with Custom AI Assistants {{ date: '2023-08-31T00:00Z', id: '2023-08-31-boost-customer-satisfaction' }}
 
+![](@/media/2023-08-31-boost-customer-satisfaction/cs-happy.png)
+
 Based on our experience rolling out Dust assistants to augment our users’ customer service teams, we’ve compiled below some best practices to serve this use case better. Please reach out at [email protected] with any questions!
 
 We’ve repeatedly seen that a general-purpose assistant with access to the entire company data is not optimal for customer service teams. The more data you give access to an assistant, the lower the chance of retrieving the correct information at the right time. Exploring this tension with our customers, we’ve discovered that customer service use cases are generally better supported by focused custom assistants that serve a dedicated set of instructions. The assistant then becomes a specialized tool more than a general helper.
@@ -38,6 +40,8 @@ For each, we provide recommendations on the instructions, model configuration, a
 - We prompt the assistant not to answer the question if a definite answer cannot be found in the retrieved information.
 - The assistant should not be used to answer an external user/customer question directly but to answer a question by a customer service agent about curated internal company knowledge while answering an external user/customer question.
 
+![](@/media/2023-08-31-boost-customer-satisfaction/customer-satisfaction-screenshot.png)
+
 ### Assistant specification
 
 **Instructions**

blog/src/pages/2023-09-28-automating-sql-query.mdx

@@ -17,6 +17,8 @@ export const Line = () => (
 
 ## Automate SQL Query Generation with a Custom Dust Assistant {{ date: '2023-09-28T00:00Z', id: '2023-09-28-automating-sql-query' }}
 
+![](@/media/2023-09-28-automating-sql-query/sql-header.png)
+
 Take the load off your data analysts with a Dust assistant specialized in SQL.
 
 From wide-scale rollouts, we've seen immense value in AI assistants that automatically generate SQL queries.
@@ -33,6 +35,8 @@ While this use case is straightforward, feel free to ask any questions. With the
 - We prompt the assistant by providing a full description of the database schemas, as well as a general translation of business terms to table and field names if necessary. This name correspondence is useful if table or column names are not clearly aligned with the general business terminology used within the company. As an example: it can be useful to define what your company consider an active user and similar concepts.
 - If your company is large, the number of tables may be prohibitively large (say 100+). In that case, we encourage shipping one custom assistant per team or function focusing on the tables that are useful to their general use cases. We sugget to not exceed 50 table definitions per assistant. We’re here to help, get in touch in case of doubt: [email protected]
 
+![](@/media/2023-09-28-automating-sql-query/sql-screenshot.png)
+
 In the instructions below we prompt the model to follow instructions specific to our systems. You can obviously adapt these instructions to your own systems.
 
 ### Assistant specification

blog/src/pages/2023-10-23-track-project-status.mdx

@@ -17,6 +17,8 @@ export const Line = () => (
 
 ## Build an AI Assistant to Track Internal Project Status {{ date: '2023-10-23T00:00Z', id: '2023-10-23-track-project-status' }}
 
+![](@/media/2023-10-23-track-project-status/track-project-img.png)
+
 Stay on top of key projects with a custom Dust assistant.
 
 From rolling out Dust and assistants across companies, we've seen tremendous value in creating AI assistants that provide project status updates.
@@ -34,6 +36,8 @@ Please reach out to [email protected] with any questions!
 - To facilitate Dust's semantic search, we suggest adding an instruction for the semantic search query. This will give more guidance to your assistant on where to focus.
 - Finally, we included a template of the desired format. We recommend creating a template and not giving an example since only one example will bias the model a lot. You can add examples at the end of the instructions but limit them to 3-4.
 
+![](@/media/2023-10-23-track-project-status/track-project-screenshot.png)
+
 In the instructions below, we prompt the model to follow instructions specific to our systems. You can adapt these instructions to your own systems.
 
 ### Assistant specification

blog/src/pages/index.mdx

@@ -10,6 +10,8 @@ export async function getStaticProps() {
 
 ---
 
+![](@/media/2023-10-23-track-project-status/track-project-img.png)
+
 ## Build an AI Assistant to Track Internal Project Status {{ date: '2023-10-23T00:00Z', id: '2023-10-23-track-project-status' }}
 
 Stay on top of key projects with a custom Dust assistant.
@@ -20,6 +22,8 @@ This best practice outlines how to build an assistant that delivers timely, rele
 
 ---
 
+![](@/media/2023-09-28-automating-sql-query/sql-header.png)
+
 ## Automate SQL Query Generation with a Custom Dust Assistant {{ date: '2023-09-28T00:00Z', id: '2023-09-28-automating-sql-query' }}
 
 Take the load off your data analysts with a Dust assistant specialized in SQL.
@@ -30,6 +34,8 @@ This article outlines building a Dust Custom assistant that outputs analysis-rea
 
 ---
 
+![](@/media/2023-08-31-boost-customer-satisfaction/cs-happy.png)
+
 ## Boost Customer Satisfaction with Custom AI Assistants {{ date: '2023-08-31T00:00Z', id: '2023-08-31-boost-customer-satisfaction' }}
 
 Based on our experience rolling out Dust assistants to augment our users’ customer service teams, we’ve compiled below some best practices to serve this use case better. Please reach out at [email protected] with any questions!

connectors/migrations/20231109_incident_gdrive_non_deleted_files.ts

@@ -0,0 +1,131 @@
+import { Sequelize } from "sequelize";
+
+import { dataSourceConfigFromConnector } from "@connectors/lib/api/data_source_config";
+import { deleteFromDataSource } from "@connectors/lib/data_sources";
+import { Connector, GoogleDriveFiles } from "@connectors/lib/models";
+
+// To be run from connectors with `CORE_DATABASE_URI` and `FRONT_DATABASE_URI` set.
+const { CORE_DATABASE_URI, FRONT_DATABASE_URI, LIVE = false } = process.env;
+
+async function main() {
+  const core_sequelize = new Sequelize(CORE_DATABASE_URI as string, {
+    logging: false,
+  });
+  const front_sequelize = new Sequelize(FRONT_DATABASE_URI as string, {
+    logging: false,
+  });
+
+  const gDriveConnectors = await Connector.findAll({
+    where: {
+      type: "google_drive",
+    },
+  });
+
+  console.log(`Processing ${gDriveConnectors.length} google drive connectors`);
+
+  for (const c of gDriveConnectors) {
+    const files = await GoogleDriveFiles.findAll({
+      where: {
+        connectorId: c.id,
+      },
+      attributes: ["id", "driveFileId", "dustFileId"],
+    });
+
+    console.log(`Connector ${c.id}: found ${files.length} files`);
+
+    const fileHash = files.reduce((acc, f) => {
+      acc[f.dustFileId] = f.id;
+      return acc;
+    }, {} as { [key: string]: number });
+
+    // find dustProjectId from front based on workspaceId and connectorName
+    const dsData = await front_sequelize.query(
+      'SELECT * FROM data_sources WHERE "connectorId" = :connectorId',
+      {
+        replacements: {
+          connectorId: c.id.toString(),
+        },
+      }
+    );
+
+    if (dsData[0].length === 0) {
+      throw new Error(`No data source found for connector ${c.id}`);
+    }
+    const ds = dsData[0][0] as { dustAPIProjectId: string };
+    const dustAPIProjectId = parseInt(ds.dustAPIProjectId);
+
+    console.log(`Found dustAPIProjectId: ${dustAPIProjectId}`);
+
+    const coreDsData = await core_sequelize.query(
+      `SELECT * FROM data_sources WHERE "project" = :dustAPIProjectId`,
+      {
+        replacements: {
+          dustAPIProjectId: dustAPIProjectId,
+        },
+      }
+    );
+
+    if (coreDsData[0].length === 0) {
+      throw new Error(
+        `No core data source found for dustAPIProjectId ${dustAPIProjectId}`
+      );
+    }
+    const coreDs = coreDsData[0][0] as { data_source_id: string; id: number };
+    const coreDsId = coreDs.id;
+
+    console.log(
+      `Found core DustDataSource: ${coreDsId} ${coreDs.data_source_id}`
+    );
+
+    const coreDocumentsData = await core_sequelize.query(
+      `SELECT * FROM data_sources_documents WHERE "data_source" = :coreDsId AND status = 'latest'`,
+      {
+        replacements: {
+          coreDsId: coreDsId,
+        },
+      }
+    );
+
+    const documents = coreDocumentsData[0] as {
+      id: number;
+      document_id: string;
+      tags_array: string[];
+    }[];
+
+    console.log(`Found ${documents.length} core documents`);
+
+    const documentsToDelete = documents.filter((d) => {
+      return fileHash[d.document_id] === undefined;
+    });
+
+    for (const d of documentsToDelete) {
+      console.log(
+        `Deleting document ${d.id} ${d.document_id}, tags #${d.tags_array.join(
+          ", #"
+        )}`
+      );
+      if (LIVE) {
+        await deleteDocument(c, d.document_id);
+      }
+    }
+
+    console.log(
+      `>>> Found ${documentsToDelete.length} documents to delete for workspace ${c.workspaceId}`
+    );
+  }
+}
+
+async function deleteDocument(connector: Connector, fileId: string) {
+  const dataSourceConfig = dataSourceConfigFromConnector(connector);
+  await deleteFromDataSource(dataSourceConfig, fileId);
+}
+
+main()
+  .then(() => {
+    console.log("Done");
+    process.exit(0);
+  })
+  .catch((err) => {
+    console.error(err);
+    process.exit(1);
+  });

connectors/src/connectors/github/lib/github_graphql.ts

@@ -34,9 +34,12 @@ const DiscussionCommentNodeSchema = t.type({
   bodyText: t.string,
   createdAt: t.string,
   updatedAt: t.string,
-  author: t.type({
-    login: t.string,
-  }),
+  author: t.union([
+    t.type({
+      login: t.string,
+    }),
+    t.null,
+  ]),
 });
 
 export type DiscussionCommentNode = t.TypeOf<

connectors/src/connectors/github/temporal/activities.ts

@@ -235,7 +235,9 @@ export async function githubUpsertDiscussionActivity(
       if (comment.isAnswer) {
         renderedDiscussion += "[ACCEPTED ANSWER] ";
       }
-      renderedDiscussion += `${comment.author.login}: ${comment.bodyText}||`;
+      renderedDiscussion += `${comment.author?.login || "Unknown author"}: ${
+        comment.bodyText
+      }||`;
       let nextChildCursor: string | null = null;
       for (;;) {
         const { cursor: childCursor, comments: childComments } =
@@ -245,7 +247,9 @@ export async function githubUpsertDiscussionActivity(
             nextChildCursor
           );
         for (const childComment of childComments) {
-          renderedDiscussion += `----${childComment.author.login}: ${childComment.bodyText}||`;
+          renderedDiscussion += `----${
+            childComment.author?.login || "Unknown author"
+          }: ${childComment.bodyText}||`;
         }
 
         if (!childCursor) {

connectors/src/connectors/slack/index.ts

@@ -1,4 +1,9 @@
-import { WebClient } from "@slack/web-api";
+import {
+  CodedError,
+  ErrorCode,
+  WebAPIPlatformError,
+  WebClient,
+} from "@slack/web-api";
 import PQueue from "p-queue";
 
 import { ConnectorPermissionRetriever } from "@connectors/connectors/interface";
@@ -235,8 +240,9 @@ export async function cleanupSlackConnector(
         return new Err(new Error("SLACK_CLIENT_SECRET is not defined"));
       }
 
+      const slackClient = await getSlackClient(connector.id);
       try {
-        const slackClient = await getSlackClient(connector.id);
+        await slackClient.auth.test();
         const deleteRes = await slackClient.apps.uninstall({
           client_id: SLACK_CLIENT_ID,
           client_secret: SLACK_CLIENT_SECRET,
@@ -249,10 +255,25 @@ export async function cleanupSlackConnector(
           );
         }
       } catch (e) {
-        logger.error(
-          { connectorId: connectorId, error: e },
-          "Could not uninstall the Slack app from the user's workspace."
-        );
+        const slackError = e as CodedError;
+        let shouldThrow = true;
+
+        if (slackError.code === ErrorCode.PlatformError) {
+          const platformError = e as WebAPIPlatformError;
+          if (platformError.data.error === "account_inactive") {
+            shouldThrow = false;
+            logger.info(
+              {
+                connectorId,
+              },
+              `Slack auth is invalid, skipping uninstallation of the Slack app`
+            );
+          }
+        }
+
+        if (shouldThrow) {
+          throw e;
+        }
       }
 
       const nangoRes = await nangoDeleteConnection(