feat(Gong) - Add transcripts sync #11231

aubin-tchoi · 2025-03-05T14:52:41Z

Description

This PR implements the sync of Gong transcripts.

The transcript are all available under a single folder named "Transcripts", which is not selectable.
A table gong_transcripts is added by this PR.

The transcripts are synced as follows:

A page of transcripts are fetched using the /calls/transcript endpoint.
The corresponding metadata is fetched using the /calls/extensive endpoint.
The participants are fetched from connectors' table gong_users, fetching Gong's API if not in db.
The data_source_document is created and upserted.

The following tags are attached to the document:

title.
createdAt.
language (ISO-639-2B: eng, fre, spa, ger, and ita).
media (audio or video).
scope (Internal, External).
direction (Inbound, Outbound, Conference).
the participant's emails if we have them.

Tests

Tested locally.

Risk

N/A.

Deploy Plan

Apply migration_60.sql.
Deploy connectors.
Deploy front.

github-actions · 2025-03-05T14:54:22Z

	Warnings
⚠️	Files in `**/lib/models/` have been modified and the PR has the `migration-ack` label. Don't forget to run the migration from prodbox.

Generated by 🚫 dangerJS against c2fcece

# Conflicts: # connectors/src/connectors/gong/lib/gong_api.ts # connectors/src/connectors/gong/temporal/activities.ts

flvndvd

LGTM once comments have been addressed 🔥

flvndvd · 2025-03-05T18:07:50Z

connectors/migrations/db/migration_60.sql

+    "createdAt"   TIMESTAMP WITH TIME ZONE NOT NULL,
+    "updatedAt"   TIMESTAMP WITH TIME ZONE NOT NULL,
+    "callId"      TEXT                     NOT NULL,
+    "title"       TEXT                     NOT NULL,


Can title be bigger than 255 characters?

we don't have occurrences in our gong but I would assume it's not really bounded

Let's truncate in the resource then.

flvndvd · 2025-03-05T18:08:22Z

connectors/migrations/db/migration_60.sql

+CREATE INDEX "gong_transcripts_connector_id" ON "gong_transcripts" ("connectorId");
+CREATE UNIQUE INDEX "gong_transcripts_connector_id_call_id" ON "gong_transcripts" ("connectorId", "callId");


You don't need the first index, the second one (which is a composite index) will also be used when querying only on connectorId.

flvndvd · 2025-03-05T18:08:52Z

connectors/src/connectors/gong/index.ts

+      folderId: makeGongTranscriptFolderInternalId(connector),
+      parents: [makeGongTranscriptFolderInternalId(connector)],
+      parentId: null,
+      title: "Transcripts",


nit
No magic variable.

flvndvd · 2025-03-05T18:09:22Z

connectors/src/connectors/gong/lib/gong_api.ts

+export const GongParticipantCodec = t.intersection([
+  t.type({
+    speakerId: t.union([t.string, t.null]),
+    userId: t.union([t.string, t.null, t.undefined]),


Are we sure it can both null or undefined? Feels like a very strange API 😅.

mb it can only be undefined, forgot to clean this up

flvndvd · 2025-03-05T18:09:55Z

connectors/src/connectors/gong/lib/gong_api.ts

+        started: t.string,
+        duration: t.number,
+        title: t.string,
+        media: t.union([t.literal("Video"), t.literal("Audio")]),


I'd suggest removing some fields from here and only keeping the one we use to avoid validation error.

flvndvd · 2025-03-05T18:14:39Z

connectors/src/connectors/gong/lib/upserts.ts

+  await upsertDataSourceDocument({
+    dataSourceConfig,
+    documentId,
+    documentContent: await renderDocumentTitleAndContent({


nit No await in object, let's extract them.

flvndvd · 2025-03-05T18:15:41Z

connectors/src/connectors/gong/lib/upserts.ts

+      `language:${transcriptMetadata.metaData.language}`, // The language codes (as defined by ISO-639-2B): eng, fre, spa, ger, and ita.
+      `media:${transcriptMetadata.metaData.media}`,
+      `scope:${transcriptMetadata.metaData.scope}`,
+      ...participants.map((p) => p.email),


I feel like this lives out the participant that are not internal to the connected Gong's workspace. Can we still include them?

hmm in any case these participants are retrieved using getGongUsers, meaning they are exposed by Gong's API so I would say it's fine since they are not local to the meeting and exist somewhere in gong

flvndvd · 2025-03-05T18:17:33Z

connectors/src/lib/models/gong.ts

+      { fields: ["connectorId"] },
+      { fields: ["connectorId", "callId"], unique: true },


Same as above, one composite index serves the two needs.

Suggested change

{ fields: ["connectorId"] },

{ fields: ["connectorId", "callId"], unique: true },

{ fields: ["connectorId", "callId"], unique: true },

flvndvd · 2025-03-05T18:17:57Z

connectors/src/resources/connector/gong.ts

@@ -32,6 +36,14 @@ export class GongConnectorStrategy
    connector: ConnectorResource,
    transaction: Transaction
  ): Promise<void> {
+    await GongUserModel.destroy({


flvndvd · 2025-03-05T18:18:35Z

connectors/src/resources/gong_resources.ts

+
+  toJSON(): Record<string, unknown> {
+    return {
+      id: this.id,


nit

Let's add the Gong call id, no?

This reverts commit c6392a9.

This reverts commit 4facc2f.

…fallback

aubin-tchoi force-pushed the gong-transcripts branch from 2f5d755 to e93525e Compare March 5, 2025 15:13

aubin-tchoi added the migration-ack 📁 Label to acknowledge that a migration is required. label Mar 5, 2025

aubin-tchoi force-pushed the gong-transcripts branch 2 times, most recently from 3ae9bdc to f31e5ba Compare March 5, 2025 15:57

aubin-tchoi added 12 commits March 5, 2025 17:15

add method getCallsMetadata

f200516

add incomplete TranscriptModel

eba06fb

add callId to TranscriptModel

22e499d

add function to upsert transcript

4b47f55

add the transcript upserts

0e87e71

add missing fields from gong_transcripts table

c88eed7

add the email addresses to the tags

a4ddebe

add GongTranscriptModel to the synced models

7c11ea7

add index on connectorId, callId

3b64095

add migration file

fdfefa9

add a top level folder with all the transcripts

8e02302

separate the two migration files

a7cad16

aubin-tchoi force-pushed the gong-transcripts branch from f31e5ba to a7cad16 Compare March 5, 2025 16:19

aubin-tchoi added 6 commits March 5, 2025 17:20

lint

8fa2e93

lint

7dc7b29

prevent selecting the Gong transcripts folder

7c54fe7

remove the empty function shouldSyncTranscript (we're not doing it)

d8b5f76

rename speakers into participants

38c5709

remove unnecessary fields from GongParticipantCodec

36e025f

aubin-tchoi force-pushed the gong-transcripts branch from 288000b to 36e025f Compare March 5, 2025 16:48

aubin-tchoi added 5 commits March 5, 2025 17:49

increment migration file index

6af1b4e

Merge branch 'main' into gong-transcripts

73753b8

# Conflicts: # connectors/src/connectors/gong/lib/gong_api.ts # connectors/src/connectors/gong/temporal/activities.ts

add todo

bb9a7f9

add participant metadata

6cfb19c

lint

bd65280

update limitations warning

35551e2

aubin-tchoi requested a review from flvndvd March 5, 2025 17:30

aubin-tchoi added 5 commits March 5, 2025 18:35

fix document content structure

8cccc34

fix how tags are preprocessed

6055163

prevent redunding the tag name in the prefix

f3b2630

exhaust the cursor when getting transcript metadata

87c1df6

delete users and transcripts on strategy delete

56eb911

flvndvd approved these changes Mar 5, 2025

View reviewed changes

aubin-tchoi added 17 commits March 5, 2025 22:23

fix GongParticipantCodec, making userId not nullable

25aeced

remove unnecessary index

6546107

add variable TRANSCRIPTS_FOLDER_TITLE

6a62ad9

extract variables for the document content

84589da

add missing fields from GongTranscriptResource.toJSON

3d31fea

remove unused methid

1175fce

prevent refetches of the OAuth token

4facc2f

refactor: move getGongAccessToken into utils

c6392a9

Revert "refactor: move getGongAccessToken into utils"

a3ceb8a

This reverts commit c6392a9.

Revert "prevent refetches of the OAuth token"

203ba83

This reverts commit 4facc2f.

add the speaker's email provided in the call/extensive endpoint as a …

7402be7

…fallback

fix codec for gong participants

fa2e424

add comment

f643f33

add a comment

18a2e9d

add another comment

b061f9e

remove unused type

e5abd3c

add the direction to the tags

c2fcece

aubin-tchoi merged commit 1eaaff9 into main Mar 5, 2025
6 checks passed

aubin-tchoi deleted the gong-transcripts branch March 5, 2025 22:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(Gong) - Add transcripts sync #11231

feat(Gong) - Add transcripts sync #11231

aubin-tchoi commented Mar 5, 2025 •

edited

Loading

github-actions bot commented Mar 5, 2025 •

edited

Loading

flvndvd left a comment

flvndvd Mar 5, 2025

aubin-tchoi Mar 5, 2025

flvndvd Mar 6, 2025

flvndvd Mar 5, 2025

flvndvd Mar 5, 2025

flvndvd Mar 5, 2025

aubin-tchoi Mar 5, 2025

flvndvd Mar 5, 2025

flvndvd Mar 5, 2025

flvndvd Mar 5, 2025

aubin-tchoi Mar 5, 2025

flvndvd Mar 5, 2025

flvndvd Mar 5, 2025

flvndvd Mar 5, 2025

		CREATE INDEX "gong_transcripts_connector_id" ON "gong_transcripts" ("connectorId");
		CREATE UNIQUE INDEX "gong_transcripts_connector_id_call_id" ON "gong_transcripts" ("connectorId", "callId");

		{ fields: ["connectorId"] },
		{ fields: ["connectorId", "callId"], unique: true },

feat(Gong) - Add transcripts sync #11231

feat(Gong) - Add transcripts sync #11231

Conversation

aubin-tchoi commented Mar 5, 2025 • edited Loading

Description

Tests

Risk

Deploy Plan

github-actions bot commented Mar 5, 2025 • edited Loading

flvndvd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aubin-tchoi commented Mar 5, 2025 •

edited

Loading

github-actions bot commented Mar 5, 2025 •

edited

Loading