Add retry on specified httpx network errors

[peytonrunyan]

Begins to address #7512 and #7442

This pull request moves adds retry logic for exceptions encountered when sending requests with prefect/client/base.py::PrefectHttpxClient. It also moves the logic for the retry from the send method to _send_with_retry, which takes a partial of the send function with the associated args and kwargs, a list of exceptions to retry under, and a list of status codes to retry under.

Example

Checklist

• This pull request references any related issue by including "closes <link to issue>"
  • If no issue exists and your change is not a small fix, please create an issue first.
• This pull request includes tests or only affects documentation.
• This pull request includes a label categorizing the change e.g. fix, feature, enhancement

[netlify]

✅ Deploy Preview for prefect-orion ready!

Name	Link
🔨 Latest commit	3719fbe
🔍 Latest deploy log	https://app.netlify.com/sites/prefect-orion/deploys/6388505d5377a1000866f70a
😎 Deploy Preview	https://deploy-preview-7593--prefect-orion.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[jakekaplan]

nice work! left a couple questions

[jakekaplan]

I am not certain but could this be:

while (not response or retry):

It looks like retry is being set on the same statements that the while loop is conditional on? I might be missing a case where the reverse doesn't make sense. Or maybe the retry variable is unneeded? I can't quite put my finger on it, but there could be a way to simplify this or make it a little more easily readable😅

[peytonrunyan]

Is the new arrangement more readable?

[jakekaplan]

I'm so sorry, I think my original comment was a little vague and unhelpful! I think a combination of the conditionals underneath the try/except/else/finally structure makes it harder to follow the code paths. Maybe something like this could work? (Just to qualify a bit I don't know if this works and the solution as written is great, just seeing if it's possible to consolidate a bit)

        try_count = 0
        response = None

        while True:
            try_count += 1
            retry_seconds = None
            retry = try_count <= self.RETRY_MAX

            try:
                response = await request()
            except retry_exceptions:
                if not retry:
                    raise
                retry_seconds = 2**try_count
            else:
                if response.status_code in retry_codes:
                    retry_seconds = float(response.headers.get("Retry-After", 2**try_count))

            if retry and retry_seconds is not None:
                await anyio.sleep(retry_seconds)
            else:
                break

        return response

[zanieb]

@jakekaplan want to give it a quick look again?

[peytonrunyan]

@madkinsz thanks a ton for jumping in on this!