Skip to content

Commit

Permalink
[Slack] API: fetch public and private remote channels separately (#8775)
Browse files Browse the repository at this point in the history
Description
---
Fixes dust-tt/tasks#1655 (again)

This PR fixes the issues caused by the `conversations.list` rate
limiting

Investigating the above, it was shown (tested on a production
workspace) that fetching both public and private channels at the same
time changes the fetch behavior (likely due to their internal
implementation)
- Fetching both public and private =>  ~400 calls to
`conversations.list` (small pagination batches)
- Fetching public => ~10 calls, then private ~1 call (large pagination
batches)

As such, the PR fetches them separately, which will fix the issue

Risks
---
low: simple, tested change

Deploy
---
connectors
  • Loading branch information
philipperolet authored Nov 20, 2024
1 parent 95decd2 commit ff91734
Showing 1 changed file with 15 additions and 1 deletion.
16 changes: 15 additions & 1 deletion connectors/src/connectors/slack/temporal/activities.ts
Original file line number Diff line number Diff line change
Expand Up @@ -81,14 +81,28 @@ export const getChannels = cacheWithRedis(
async function _getChannelsUncached(
connectorId: ModelId,
joinedOnly: boolean
): Promise<Channel[]> {
return Promise.all([
_getTypedChannelsUncached(connectorId, joinedOnly, "public_channel"),
_getTypedChannelsUncached(connectorId, joinedOnly, "private_channel"),
]).then(([publicChannels, privateChannels]) => [
...publicChannels,
...privateChannels,
]);
}

async function _getTypedChannelsUncached(
connectorId: ModelId,
joinedOnly: boolean,
types: "public_channel" | "private_channel"
): Promise<Channel[]> {
const client = await getSlackClient(connectorId);
const allChannels = [];
let nextCursor: string | undefined = undefined;
let nbCalls = 0;
do {
const c: ConversationsListResponse = await client.conversations.list({
types: "public_channel, private_channel",
types,
// despite the limit being 1000, slack may return fewer channels
// we observed ~50 channels per call at times see https://github.com/dust-tt/tasks/issues/1655
limit: 999,
Expand Down

0 comments on commit ff91734

Please sign in to comment.