Splits daily tasks into multiple tasks #18
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
So, we recently have seen some issues with consistency in the "daily tasks" task that runs for the stacks project daily at 12:00 PM ET. We've seen it fail multiple days in a row, and fail one day but not the next. I have seen this firsthand by remotely triggering the daily task and watching it fail due to excessive memory usage while processing data.
I am far from an expert in rails, or how it uses memory, but I suspect that crunching all of this data can lead to leaky memory or incorrectly retained memory. An easy optimization in that case is to split these tasks out from one big task which is run on a heroku performance-L dyno, into multiple tasks where each task or set of tasks will have more headroom. My plan is to run the heavier snapshot-generating tasks on performance-L dynos and the rest on standard-2x dynos.
The scheduling would largely be the same, with the "sync" tasks running perhaps at 11:50 AM, just before the rest of the tasks were triggered at 12:00 PM. Cost-wise I don't expect this to have much of an impact, maybe a few dollars at most monthly.