-
Notifications
You must be signed in to change notification settings - Fork 111
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[connectors] Add a CLI command confluence upsert-page
#9358
Merged
Merged
Changes from all commits
Commits
Show all changes
18 commits
Select commit
Hold shift + click to select a range
bf29b28
wip
aubin-tchoi 80ae7d2
refactor: move page upsertion into a dedicated function
aubin-tchoi 24b70f5
feat: add page upsertion to the CLI command
aubin-tchoi ae07706
add return values for the CLI command
aubin-tchoi 2f76d26
skip pages that have a skip reason in the db
aubin-tchoi 9055b09
add the correct parents
aubin-tchoi d4a6f0e
add a command to upsert multiple pages, reading from a JSON file
aubin-tchoi 39c5ad0
add db upserts too
aubin-tchoi cbff48a
add confluence to the admin cli
aubin-tchoi eeaf46e
fix ConfluenceUpsertPageResponseSchema
aubin-tchoi 042c5a7
create an activity for the full upsert
aubin-tchoi 27f2d97
move cli force-upserts to workflows
aubin-tchoi 2be5a29
remove some exports and improve types
aubin-tchoi a385951
remove unused import
aubin-tchoi 647f687
cleanup
aubin-tchoi f2d142e
uniformize logs
aubin-tchoi 436d7e4
:sparkles:
aubin-tchoi 19716ba
fix incorrect import
aubin-tchoi File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,121 @@ | ||
import type { | ||
AdminSuccessResponseType, | ||
ConfluenceCommandType, | ||
ConfluenceUpsertPageResponseType, | ||
} from "@dust-tt/types"; | ||
import assert from "assert"; | ||
import fs from "fs/promises"; | ||
|
||
import { QUEUE_NAME } from "@connectors/connectors/confluence/temporal/config"; | ||
import { | ||
confluenceUpsertPagesWithFullParentsWorkflow, | ||
confluenceUpsertPageWithFullParentsWorkflow, | ||
} from "@connectors/connectors/confluence/temporal/workflows"; | ||
import { getTemporalClient } from "@connectors/lib/temporal"; | ||
import { default as topLogger } from "@connectors/logger/logger"; | ||
|
||
export const confluence = async ({ | ||
command, | ||
args, | ||
}: ConfluenceCommandType): Promise< | ||
AdminSuccessResponseType | ConfluenceUpsertPageResponseType | ||
> => { | ||
const logger = topLogger.child({ majorCommand: "confluence", command, args }); | ||
switch (command) { | ||
case "upsert-page": { | ||
if (!args.connectorId) { | ||
throw new Error("Missing --connectorId argument"); | ||
} | ||
if (!args.pageId) { | ||
throw new Error("Missing --pageId argument"); | ||
} | ||
const { connectorId, pageId } = args; | ||
|
||
const client = await getTemporalClient(); | ||
const workflow = await client.workflow.start( | ||
confluenceUpsertPageWithFullParentsWorkflow, | ||
{ | ||
args: [{ connectorId, pageId }], | ||
taskQueue: QUEUE_NAME, | ||
workflowId: `confluence-upsert-page-${connectorId}-${pageId}`, | ||
searchAttributes: { connectorId: [connectorId] }, | ||
memo: { connectorId }, | ||
} | ||
); | ||
|
||
const { workflowId } = workflow; | ||
const temporalNamespace = process.env.TEMPORAL_NAMESPACE; | ||
if (!temporalNamespace) { | ||
logger.info(`[Admin] Started temporal workflow with id: ${workflowId}`); | ||
} else { | ||
logger.info( | ||
`[Admin] Started temporal workflow with id: ${workflowId} - https://cloud.temporal.io/namespaces/${temporalNamespace}/workflows/${workflowId}` | ||
); | ||
} | ||
return { | ||
workflowId, | ||
workflowUrl: temporalNamespace | ||
? `https://cloud.temporal.io/namespaces/${temporalNamespace}/workflows/${workflowId}` | ||
: undefined, | ||
}; | ||
} | ||
case "upsert-pages": { | ||
if (!args.connectorId) { | ||
throw new Error("Missing --connectorId argument"); | ||
} | ||
if (!args.file) { | ||
throw new Error("Missing --file argument"); | ||
} | ||
if (!args.keyInFile) { | ||
throw new Error("Missing --keyInFile argument"); | ||
} | ||
const connectorId = args.connectorId; | ||
const file = args.file; | ||
const keyInFile = args.keyInFile; | ||
|
||
// parsing the JSON file | ||
const fileContent = await fs.readFile(file, "utf-8"); | ||
const jsonArray = JSON.parse(fileContent); | ||
assert(Array.isArray(jsonArray), "The file content is not an array."); | ||
|
||
const pageIds = jsonArray.map((entry) => { | ||
assert( | ||
keyInFile in entry, | ||
`Key "${keyInFile}" not found in entry ${JSON.stringify(entry)}` | ||
); | ||
return entry[keyInFile]; | ||
}); | ||
|
||
const client = await getTemporalClient(); | ||
const workflow = await client.workflow.start( | ||
confluenceUpsertPagesWithFullParentsWorkflow, | ||
{ | ||
args: [{ connectorId, pageIds }], | ||
taskQueue: QUEUE_NAME, | ||
workflowId: `confluence-upsert-pages-${connectorId}`, | ||
searchAttributes: { connectorId: [connectorId] }, | ||
memo: { connectorId }, | ||
} | ||
); | ||
|
||
const { workflowId } = workflow; | ||
const temporalNamespace = process.env.TEMPORAL_NAMESPACE; | ||
if (!temporalNamespace) { | ||
logger.info(`[Admin] Started temporal workflow with id: ${workflowId}`); | ||
} else { | ||
logger.info( | ||
`[Admin] Started temporal workflow with id: ${workflowId} - https://cloud.temporal.io/namespaces/${temporalNamespace}/workflows/${workflowId}` | ||
); | ||
} | ||
return { | ||
workflowId, | ||
workflowUrl: temporalNamespace | ||
? `https://cloud.temporal.io/namespaces/${temporalNamespace}/workflows/${workflowId}` | ||
: undefined, | ||
}; | ||
} | ||
|
||
default: | ||
throw new Error("Unknown Confluence command: " + command); | ||
} | ||
}; |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's a bit of a risk of having some divergence between the cli code and the activities code, we generally try to call an activity from here even if that means it's async.
All that being said I'm fine with having a direct implementation. Any chance we can dry a bit these 2 commands?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yep I agree, I was contemplating adding a new activity but in the end we won't do that in the workflow (upsert and check parents right after)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could be an option on the activity + a new workflow to do just one page with that option set
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hmm interesting
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sounds good, doing that 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
actually it's a bit too convoluted to use the same activity: one fetches the page + upserts without parent and is optimized for the workflow (it does some version checking), the other one fetches the page, fetches the space using the page data, resolves parents and does an upsert with parents.
I added a separate activity to do the
upsertWithFullParents
and added workflows since we might want to do these with retries and historySGTY @spolu ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yess!