Streaming large files using request lib

[mile95]

In the future, one can imagine that we would like to stream large files to a manager. When sending large files using the request lib (without streaming) is it very likely that python (the program) runs out of memory and we get a MemoryError. To avoid this one can use the requests_toolbelt package for streaming files in the request.post method.

Here is one working example where we stream the content of a file:

from requests_toolbelt import MultipartEncoder

payload = {'field0': "value0", 'field1': "value1"}
files = {'file': ("filename", open('mylargefile.tar', 'rb'), 'application/octet-stream')}

m = MultipartEncoder(fields=dict(files, **payload))
headers = get_base_headers() # Get the required headers.
headers['content-type'] = m.content_type 

response = request.post(
    "url",
     data=m,
     headers=headers
)

In the endpoint, using file: UploadFile in the argument will get the whole streamed file.