Excessive retries

[colingreen-payroc]

This recent commit add rettry logic to RestRequester...

25498dd#diff-28552eb7d57bcb8b7f43eb5706654e031aaac0cf623630bbdfb8f7c9b61a93cf

However, there is already a retry loop in HttpUtil.SendAsync(). The new retry loop in RestRequester causeses calls the the inner try loop, and both have exponential back-off, thus causing very long overall wait for retries.

Separately, the retry loop in HttpUtil.SendAsync() has this code:

                    catch (Exception e)
                    {
                        if (cancellationToken.IsCancellationRequested)
                        {
                            logger.Debug("SF rest request timeout or explicit cancel called.");
                            cancellationToken.ThrowIfCancellationRequested();
                        }
                        else if (childCts != null && childCts.Token.IsCancellationRequested)
                        {
                            logger.Warn("Http request timeout. Retry the request");
                            totalRetryTime += (int)httpTimeout.TotalSeconds;
                        }
                        else
                        {
                            //TODO: Should probably check to see if the error is recoverable or transient.
                            logger.Warn("Error occurred during request, retrying...", e);
                        }
                    }

The final else block there causes an infinite retry loop for all exceptions other than cancellation token related exceptions. E.g. a HTTP request fails to connect then this loop is infinite. I suggest at least honouring the disableRetries boolean setting, like so:

                        else
                        {
                            //TODO: Should probably check to see if the error is recoverable or transient.
                            if(disableRetry)
                            {
                                throw;
                            }

                            logger.Warn("Error occurred during request, retrying...", e);
                        }

[sfc-gh-nwhite]

@colgreen-payroc Do you have a simple reproduction scenario and perhaps a log showing the behavior? I'm going to work on reproducing it (by blocking access to a stage URL for result sets), but if you have a concrete repro and/or logs, that may help.

[colingreen-payroc]

@sfc-gh-nwhite the main scenario of concern for me is when using connection pooling. With that enabled, if I make one or more a successful queries then there will be a connection in the pool.

If you then make snowflake unavailable (e.g. unplug network cable is one way), and then make a new query, it will use the connection from the pool, and will enter the infinite retry loop.

The initial connection itself doesn't have the infinite loop, i.e., if you unplug the network cable before there is a connection in the pool, then you will get a connection failure exception (which is fine). So it's fairly easy to recreate for me.

My company won't allow me to use this package without somehow avoiding the infinite loop, as that could cause an outage of our app. And I can see know workaround, so I've had to fork the repo to make the change in my second code block in my first comment.

Thanks.

[sfc-gh-dszmolka]

since the initial issue submission, various enhancements and bugfixes were released. Please let us know if you still experience the issue with the recent versions of the Snowflake .NET driver and we can continue troubleshooting.
Thank you !