Bug: Incompatibility with Requests 2.32.1 causing connection to Docker daemon to fail #577

NikiforovG · 2024-05-21T17:54:50Z

Describe the bug

I experienced an issue when running pytest tests involving the Testcontainers Python library in my environment. The Docker client library was unable to connect to the Docker daemon and threw an error: 'Error while fetching server API version: Not supported URL scheme http+docker'.

To Reproduce

The issue seems to present itself when the following conditions are met:

Running tests involving the use of a test container (Testcontainers Python library) spawned through pytest.
Having requests library updated to version 2.32.1.

Rolling back the requests library to version 2.31.0 resolved my issue, which leads me to believe that the issue is with requests 2.32.1 compatibility.

Runtime environment

Python 3.11.7
pytest==8.2.0
testcontainers==3.4.4
requests==2.32.1 (issue occurs)
requests==2.31.0 (issue does not occur)

test setup failed
self = <docker.transport.unixconn.UnixHTTPAdapter object at 0x1092a9f50>
request = <PreparedRequest [GET]>, stream = False, timeout = 60, verify = True
cert = None, proxies = OrderedDict()

    def send(
        self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None
    ):
        """Sends PreparedRequest object. Returns Response object.
    
        :param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
        :param stream: (optional) Whether to stream the request content.
        :param timeout: (optional) How long to wait for the server to send
            data before giving up, as a float, or a :ref:`(connect timeout,
            read timeout) <timeouts>` tuple.
        :type timeout: float or tuple or urllib3 Timeout object
        :param verify: (optional) Either a boolean, in which case it controls whether
            we verify the server's TLS certificate, or a string, in which case it
            must be a path to a CA bundle to use
        :param cert: (optional) Any user-provided SSL certificate to be trusted.
        :param proxies: (optional) The proxies dictionary to apply to the request.
        :rtype: requests.Response
        """
    
        try:
>           conn = self._get_connection(request, verify, proxies=proxies, cert=cert)

../../../../Library/Caches/pypoetry/virtualenvs/task-consumer-u30wUliX-py3.11/lib/python3.11/site-packages/requests/adapters.py:532: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../Library/Caches/pypoetry/virtualenvs/task-consumer-u30wUliX-py3.11/lib/python3.11/site-packages/requests/adapters.py:400: in _get_connection
    conn = self.poolmanager.connection_from_host(
../../../../Library/Caches/pypoetry/virtualenvs/task-consumer-u30wUliX-py3.11/lib/python3.11/site-packages/urllib3/poolmanager.py:304: in connection_from_host
    return self.connection_from_context(request_context)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <urllib3.poolmanager.PoolManager object at 0x1092aa450>
request_context = {'block': False, 'cert_reqs': 'CERT_REQUIRED', 'host': 'localhost', 'maxsize': 10, ...}

    def connection_from_context(
        self, request_context: dict[str, typing.Any]
    ) -> HTTPConnectionPool:
        """
        Get a :class:`urllib3.connectionpool.ConnectionPool` based on the request context.
    
        ``request_context`` must at least contain the ``scheme`` key and its
        value must be a key in ``key_fn_by_scheme`` instance variable.
        """
        if "strict" in request_context:
            warnings.warn(
                "The 'strict' parameter is no longer needed on Python 3+. "
                "This will raise an error in urllib3 v2.1.0.",
                DeprecationWarning,
            )
            request_context.pop("strict")
    
        scheme = request_context["scheme"].lower()
        pool_key_constructor = self.key_fn_by_scheme.get(scheme)
        if not pool_key_constructor:
>           raise URLSchemeUnknown(scheme)
E           urllib3.exceptions.URLSchemeUnknown: Not supported URL scheme http+docker

../../../../Library/Caches/pypoetry/virtualenvs/task-consumer-u30wUliX-py3.11/lib/python3.11/site-packages/urllib3/poolmanager.py:326: URLSchemeUnknown

During handling of the above exception, another exception occurred:

self = <docker.api.client.APIClient object at 0x1092a8690>

    def _retrieve_server_version(self):
        try:
>           return self.version(api_version=False)["ApiVersion"]

../../../../Library/Caches/pypoetry/virtualenvs/task-consumer-u30wUliX-py3.11/lib/python3.11/site-packages/docker/api/client.py:213: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../Library/Caches/pypoetry/virtualenvs/task-consumer-u30wUliX-py3.11/lib/python3.11/site-packages/docker/api/daemon.py:181: in version
    return self._result(self._get(url), json=True)
../../../../Library/Caches/pypoetry/virtualenvs/task-consumer-u30wUliX-py3.11/lib/python3.11/site-packages/docker/utils/decorators.py:44: in inner
    return f(self, *args, **kwargs)
../../../../Library/Caches/pypoetry/virtualenvs/task-consumer-u30wUliX-py3.11/lib/python3.11/site-packages/docker/api/client.py:236: in _get
    return self.get(url, **self._set_request_timeout(kwargs))
../../../../Library/Caches/pypoetry/virtualenvs/task-consumer-u30wUliX-py3.11/lib/python3.11/site-packages/requests/sessions.py:602: in get
    return self.request("GET", url, **kwargs)
../../../../Library/Caches/pypoetry/virtualenvs/task-consumer-u30wUliX-py3.11/lib/python3.11/site-packages/requests/sessions.py:589: in request
    resp = self.send(prep, **send_kwargs)
../../../../Library/Caches/pypoetry/virtualenvs/task-consumer-u30wUliX-py3.11/lib/python3.11/site-packages/requests/sessions.py:703: in send
    r = adapter.send(request, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <docker.transport.unixconn.UnixHTTPAdapter object at 0x1092a9f50>
request = <PreparedRequest [GET]>, stream = False, timeout = 60, verify = True
cert = None, proxies = OrderedDict()

    def send(
        self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None
    ):
        """Sends PreparedRequest object. Returns Response object.
    
        :param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
        :param stream: (optional) Whether to stream the request content.
        :param timeout: (optional) How long to wait for the server to send
            data before giving up, as a float, or a :ref:`(connect timeout,
            read timeout) <timeouts>` tuple.
        :type timeout: float or tuple or urllib3 Timeout object
        :param verify: (optional) Either a boolean, in which case it controls whether
            we verify the server's TLS certificate, or a string, in which case it
            must be a path to a CA bundle to use
        :param cert: (optional) Any user-provided SSL certificate to be trusted.
        :param proxies: (optional) The proxies dictionary to apply to the request.
        :rtype: requests.Response
        """
    
        try:
            conn = self._get_connection(request, verify, proxies=proxies, cert=cert)
        except LocationValueError as e:
>           raise InvalidURL(e, request=request)
E           requests.exceptions.InvalidURL: Not supported URL scheme http+docker

../../../../Library/Caches/pypoetry/virtualenvs/task-consumer-u30wUliX-py3.11/lib/python3.11/site-packages/requests/adapters.py:534: InvalidURL

The above exception was the direct cause of the following exception:

    @pytest.fixture(scope="session")
    def postgres_container() -> Generator[PostgresContainer, None, None]:
>       with PostgresContainer("postgres:16", dbname='test', driver=None) as postgres:

tests/conftest.py:10: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../Library/Caches/pypoetry/virtualenvs/task-consumer-u30wUliX-py3.11/lib/python3.11/site-packages/testcontainers/postgres/__init__.py:61: in __init__
    super().__init__(image=image, **kwargs)
../../../../Library/Caches/pypoetry/virtualenvs/task-consumer-u30wUliX-py3.11/lib/python3.11/site-packages/testcontainers/core/container.py:46: in __init__
    self._docker = DockerClient(**(docker_client_kw or {}))
../../../../Library/Caches/pypoetry/virtualenvs/task-consumer-u30wUliX-py3.11/lib/python3.11/site-packages/testcontainers/core/docker_client.py:56: in __init__
    self.client = docker.from_env(**kwargs)
../../../../Library/Caches/pypoetry/virtualenvs/task-consumer-u30wUliX-py3.11/lib/python3.11/site-packages/docker/client.py:94: in from_env
    return cls(
../../../../Library/Caches/pypoetry/virtualenvs/task-consumer-u30wUliX-py3.11/lib/python3.11/site-packages/docker/client.py:45: in __init__
    self.api = APIClient(*args, **kwargs)
../../../../Library/Caches/pypoetry/virtualenvs/task-consumer-u30wUliX-py3.11/lib/python3.11/site-packages/docker/api/client.py:197: in __init__
    self._version = self._retrieve_server_version()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <docker.api.client.APIClient object at 0x1092a8690>

    def _retrieve_server_version(self):
        try:
            return self.version(api_version=False)["ApiVersion"]
        except KeyError as ke:
            raise DockerException(
                'Invalid response from docker daemon: key "ApiVersion"'
                ' is missing.'
            ) from ke
        except Exception as e:
>           raise DockerException(
                f'Error while fetching server API version: {e}'
            ) from e
E           docker.errors.DockerException: Error while fetching server API version: Not supported URL scheme http+docker

../../../../Library/Caches/pypoetry/virtualenvs/task-consumer-u30wUliX-py3.11/lib/python3.11/site-packages/docker/api/client.py:220: DockerException

The text was updated successfully, but these errors were encountered:

ErwinJapie · 2024-05-21T18:29:37Z

Running into the exact same issue as of today.

alexanderankin · 2024-05-21T18:31:35Z

Any relevant issues on requests library side?

…

On Tue, May 21, 2024, 2:30 PM Erwin Boshart ***@***.***> wrote: Running into the exact issue as of today. — Reply to this email directly, view it on GitHub <#577 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACECGJCP7I7NYCIIYVKPU2TZDOHCRAVCNFSM6AAAAABICC6H6CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRTGIYDGNRYG4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>

alexanderankin · 2024-05-21T19:44:04Z

related issues:

alexanderankin · 2024-05-23T11:28:51Z

there has been a release https://pypi.org/project/docker/#history - perhaps if someone can retest and update here if its still an issue, then we can close

ErwinJapie · 2024-05-27T09:17:42Z

I can confirm this issue has been resolved for us after the update.

kiview · 2024-05-27T14:51:43Z

Thanks you @ErwinJapie, I'll close this now, since from my understanding, it is essentially resolved through the docker-py release.

Upgrading requests (#192) is failing due to issue in docker, which was fixed in 7.1.0: testcontainers/testcontainers-python#577. By updating both, we expect things to work smoothly. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

tarilabs mentioned this issue May 27, 2024

chore(deps): bump docker from 7.0.0 to 7.1.0 #583

Closed

kiview closed this as completed May 27, 2024

thpierce mentioned this issue May 27, 2024

Upgrade requests and docker aws-observability/aws-otel-python-instrumentation#196

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug: Incompatibility with Requests 2.32.1 causing connection to Docker daemon to fail #577

Bug: Incompatibility with Requests 2.32.1 causing connection to Docker daemon to fail #577

NikiforovG commented May 21, 2024 •

edited by alexanderankin

Loading

ErwinJapie commented May 21, 2024 •

edited

Loading

alexanderankin commented May 21, 2024 via email

alexanderankin commented May 21, 2024 •

edited

Loading

alexanderankin commented May 23, 2024

ErwinJapie commented May 27, 2024

kiview commented May 27, 2024

Bug: Incompatibility with Requests 2.32.1 causing connection to Docker daemon to fail #577

Bug: Incompatibility with Requests 2.32.1 causing connection to Docker daemon to fail #577

Comments

NikiforovG commented May 21, 2024 • edited by alexanderankin Loading

ErwinJapie commented May 21, 2024 • edited Loading

alexanderankin commented May 21, 2024 via email

alexanderankin commented May 21, 2024 • edited Loading

alexanderankin commented May 23, 2024

ErwinJapie commented May 27, 2024

kiview commented May 27, 2024

NikiforovG commented May 21, 2024 •

edited by alexanderankin

Loading

ErwinJapie commented May 21, 2024 •

edited

Loading

alexanderankin commented May 21, 2024 •

edited

Loading