Speed up rebuilds

[chrismytton]

Problem

Rebuilds are currently taking an average of about 10 minutes, which is slower than they're coming in, so they get backed up and fall slowly out of sync.

Proposed solution

Most of the time for a rebuild is spent doing a fresh clone of the everypolitician-data (which is multi-GB) and doing a full bundle install from scratch. If we switch to using a persistent disk to store a copy of the repo and the bundler dependencies then we can avoid all that overhead and skip straight to doing the actual rebuild.

I think it would be prudent to do a test of this locally to check there's nothing unexpected that comes up.

Acceptance criteria

Doing a full rebuild of a country should take seconds rather than minutes.

Prerequisites

#62

[chrismytton]

Another potential way to speed up rebuilds would be to compare the source files over http before we clone the repo. This would allow us to skip the rebuild entirely if no source files have changed.

As a bonus this would also be useful for the pull request summarizer, which also does comparisons of files over HTTP.

[chrismytton]

(I've updated the title of this issue to state the problem rather than the solution 😄 )

Now that #71 has been merged builds are mostly under 5 minutes, yay! 🎉

I think there is still more time that can be gained in some areas though:

• Use a local copy of the everypolitician-data repo and do a git fetch to get latest updates before building rather than having to perform a full clone.
• Keep a cache of bundler dependencies to cut down the time spent in bundle install
• Skip the rebuild script if there are no changes in the source CSVs. This will require extracting RemoteSource::Morph from everypolitician-data's lib directory into a separate gem.
• Cleaning up the everypolitician-data Gemfile - Clean up everypolitician-data Gemfile everypolitician#585