Skip to content

Commit

Permalink
Refactoring and trying to make concurrency work
Browse files Browse the repository at this point in the history
Mailchimp's request limit is making it hard to implement concurrency. Will try other techniques soon
  • Loading branch information
hdoro committed Nov 21, 2018
1 parent b02b567 commit 3155b17
Show file tree
Hide file tree
Showing 7 changed files with 43 additions and 64 deletions.
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,5 @@ gatsby-node.js
fetchAllCampaigns.js
fetchContent.js
helpers.js
yarn-error.log
yarn-error.log
backup
6 changes: 3 additions & 3 deletions .npmignore
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
src
.babelrc
.editorconfig
.git
tsconfig.json
tslint.json
.prettierrc
.gitignore
.gitignore
yarn-error.log
backup
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@ The process to save campaigns in `gatsby-node.js` is as follows:
## TODO

- Explore better ways to cache the content.
- Concurrent requests in a way that respects Mailchimp's 10 requests/minute limitation

## License

Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "gatsby-source-mailchimp",
"version": "0.6.0",
"version": "0.6.1",
"description": "Source plugin to fetch campaigns from Mailchimp into Gatsby",
"author": "Henrique Cavalieri <@hcavalieri>",
"license": "MIT",
Expand Down
41 changes: 0 additions & 41 deletions src/fetchAllCampaigns.ts

This file was deleted.

53 changes: 35 additions & 18 deletions src/gatsby-node.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
import axios, { AxiosPromise } from 'axios';
import { validateConfig } from './configValidation';
import { colorizeLog, consoleColors } from './helpers';
import { fetchAllCampaigns } from './fetchAllCampaigns';

export interface IPluginOptions {
plugins: any[];
Expand Down Expand Up @@ -88,19 +87,30 @@ export const sourceNodes = async (
}

if (data.total_items > campaigns.length && count === 0) {
campaigns = await fetchAllCampaigns({
campaigns,
totalItems: data.total_items,
count,
defaultCount,
authParams,
fields: campaignFields,
campaignsURL,
});
const reqLength = campaigns.length;
const extraTimesToFetch = Math.ceil(data.total_items / reqLength);
const reqArray = Array.from(
{ length: extraTimesToFetch },
(v, i) => i * reqLength
);
for (const t of reqArray) {
const newBatch = await axios.get(campaignsURL, {
...authParams,
params: {
status: 'sent',
offset: t,
fields: campaignFields,
count: count || defaultCount,
sort_field: 'send_time',
sort_dir: 'DESC',
},
});
campaigns = [...campaigns, ...newBatch.data.campaigns];
}
}
console.timeEnd(colorizeLog('\nMailchimp campaigns fetched in'));

let campaignRequests: AxiosPromise[] = [];
let campaignRequests: any[] = [];
let campaignsMetadata: any[] = [];
console.time(colorizeLog('\nMailchimp campaign content fetched in'));
for (const c of campaigns) {
Expand Down Expand Up @@ -136,15 +146,23 @@ export const sourceNodes = async (
params: {
fields: contentFields.join(','),
},
})
];
campaignsMetadata = [
...campaignsMetadata,
c,
}),
];
campaignsMetadata = [...campaignsMetadata, c];
}

const campaignsContent = await Promise.all(campaignRequests);
let campaignsContent: any = [];

const concurrReq = 3;
const reqSegments = Array.from(
{ length: Math.ceil(campaignRequests.length / concurrReq) },
(v, i) => i * concurrReq
);
for (const t of reqSegments) {
const requests = campaignRequests.slice(t, t + concurrReq);
const newContent = await Promise.all(requests);
campaignsContent = [...campaignsContent, newContent];
}

for (let i = 0; i < campaignsContent.length; i++) {
const meta = campaignsMetadata[i];
Expand Down Expand Up @@ -172,5 +190,4 @@ export const sourceNodes = async (
createNode(campaignNode);
}
console.timeEnd(colorizeLog('\nMailchimp campaign content fetched in'));

};
1 change: 1 addition & 0 deletions tsconfig.json
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
},
"exclude": [
"node_modules",
"backup",
"build",
"scripts",
"acceptance-tests",
Expand Down

0 comments on commit 3155b17

Please sign in to comment.