[connectors] Upsert and backfill Confluence spaces as data_source_folders
#9402
Conversation
aubin-tchoi
force-pushed
the
confluence-folders
branch
from
6129354 to
d44c317
Compare
| await upsertFolderNode({ | ||
| dataSourceConfig, | ||
| folderId: internalId, | ||
| parents: [internalId], | ||
| title: confluenceSpace?.name ?? confluenceId, | ||
| }); | ||
|
|
There was a problem hiding this comment.
ok to make the user wait for this operation to complete?
we could technically do this in the workflow but it leads to higher risks of having the table confluence_spaces and the data_source_folders out of sync IMHO
There was a problem hiding this comment.
I'd actually rather do it in the activities, that's where we upsert in core
There was a problem hiding this comment.
alright, doing that 👍
| import { concurrentExecutor } from "@connectors/lib/async_utils"; | ||
| import { |
There was a problem hiding this comment.
Just checking: it seems there are multiple levels in confluence hierarchy
cf picture: test > test 123 > some files
Are those subspaces ? are those properly taken into account?
There was a problem hiding this comment.
these are pages, which are not folders but can be parents to other pages
There was a problem hiding this comment.
does this break anything regarding kwsearch?
There was a problem hiding this comment.
as per IRL: we're good 👍
| await upsertFolderNode({ | ||
| dataSourceConfig, | ||
| folderId: internalId, | ||
| parents: [internalId], | ||
| title: confluenceSpace?.name ?? confluenceId, | ||
| }); | ||
|
|
There was a problem hiding this comment.
I'd actually rather do it in the activities, that's where we upsert in core
| await deleteFolderNode({ dataSourceConfig, folderId: internalId }); | ||
|
|
There was a problem hiding this comment.
@philipperolet would you rather delete from core first and then from connectors or the opposite? no strong opinion here
There was a problem hiding this comment.
as per IRL: core first 👍
|
@philipperolet I moved the calls to front -> core to activities, can you check again please? 🙏 |
Description
data_sources_folders, I also get the nodes indata_sources_nodeswithmime_typeapplication/vnd.dust.folder.Risk
Deploy Plan