What's the correct way to recover from an Azure Storage “connection reset” 10054 error, when called from Python SDK?

[rfernand2]

We have a python app that uses Azure Storage, calling the BlockBlobService and AppendBlobService modules of the azure-storage-blob==2.1.0 package. When running a large number of simultaneous instances of our python app, we start seeing many occurrences of the following error:
ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
What is the best practice for recovering from these types of errors? Our normal retry function just fails on all 25 of its retries when this error is encountered. Is there a way to request that the connection to Azure is reset within a retry function?

[mitchdenny]

@rfernand2 I am transferring this issue over to the https://github.com/azure/azure-sdk-for-python repository since it is Python specific.

[rfernand2]

OK, thanks.

[amishra-dev]

Hi Roland (@rfernand2 ), are you still hitting this issue?

[tasherif-msft]

Hi @rfernand2, you have the ability to use a response hook (essentially a callback function) that would enable you to do something with the response you get before the retry is kicked off once again.
Using a raw_response_hook looks like the following:

counter = 0
def fail_response(response):
    if response.http_response.status_code == 404:
        counter += 1
bsc = BlobServiceClient(url, credential=your_credential, retry_total=4)
container = bsc.get_container_client(container_name)
with self.assertRaises(Exception):
    container.create_container(raw_response_hook=fail_response)

In this case the counter will keep incrementing before retrying.
Let me know if you need further more specific help on your use case!

[rfernand2]

Hi @tasherif-msft, thanks for the response. I'm not sure I understand how the above code would do anything to cause the RETRY to succeed. As I mentioned above, once I see this "connection reset" error, it always fails the 25 normal retries. Is there something special about a raw_response_hook that would automatically reestablish the connection?

[tasherif-msft]

Hi @rfernand2 I just noticed that you're using version 2.1.0 which has been deprecated for some time now.
This repository has the track 2 SDK which I highly recommend you upgrade to. Let me know if this is a possibility for you.
@xiafu-msft he is using track 1 SDK if you have any input