Support streaming downloads

[ivgiuliani]

This adds support for a streaming interface for uploads a downloads, though only downloads are supported at the moment.

Requirements considered:

• being able to easily access the content of the files in memory as we download it: this is so we can do processing of the content as the file gets downloaded
• keep the interface similar to the existing (not stream-based) download/upload, whilst making it easy to iterate on individual chunks: this is to reduce cognitive load on the library

Both GCS and S3 do not natively support streaming[1], so streaming is emulated using partial range downloads. The streaming interface is meant to be as compatible as possible with the non-streaming version:

Non-streamed download

This is the existing interface, no change there. It returns a hash with :bucket, :key and :content.

BucketStore.for("inmemory://bucket/file").download
=> {:bucket=>"bucket", :key=>"file", :content=>...}

Streamed download

The streaming version introduces a .stream for .download/.upload methods. .download takes an optional chunk_size argument to control the size of the chunks (with 4Mb being the default otherwise). .upload is not implemented as part of this PR, but .download returns an enumerable object where each object returned is a lazily downloaded chunk of at most chunk_size. The major difference in this case is that the content is returned outside of the main hash, as this allows to iterate more easily on individual chunks:

BucketStore.for("inmemory://bucket/file").stream.download
=> #<Enumerator: ...>

BucketStore.for("inmemory://bucket/file").stream.download.map { |item| item }
=> [[{:bucket=>"bucket", :key=>"file"}, "chunk 1"], [{:bucket=>"bucket", :key=>"file"}, "chunk 2"], ...]

BucketStore.for("inmemory://bucket/file").stream.download.map { |_, chunk| chunk }
=> ["chunk 1", "chunk 2", ...]

---

[1] sort of. S3's sdk does, in fact, support streaming downloads. Moreso, there's two ways it does this, but neither of them works well enough for our use cases. The first way is to stream directly to a file-like object (e.g. StringIO), however due to how this is implemented we can't access the content of the file in-memory as we download it. The second way is to use blocks, however when using blocks to download objects, the Ruby sdk will NOT retry failed requests after the first chunk of data has been yielded. Doing so could cause file corruption on the client end by starting over mid-stream. See https://aws.amazon.com/blogs/developer/downloading-objects-from-amazon-s3-using-the-aws-sdk-for-ruby/ for details.

[ahjmorton]

Suggested change

	def stream_download(bucket:, key:, chunk_size: nil)
	chunk_size ||= DEFAULT_STREAM_CHUNK_SIZE_BYTES
	def stream_download(bucket:, key:, chunk_size: DEFAULT_STREAM_CHUNK_SIZE_BYTES)

minor / nitpick, just curious why we're not using the default arg functionality

[ivgiuliani]

I think it's a PEBKAC...