Parity with requests: support https:// proxied over https:// #833

mborsetti · 2023-10-31T21:22:07Z

mborsetti
Oct 31, 2023

Executing the following using HTTPX:

import httpx

response = httpx.get('https://api.ipify.org', proxies='https://proxyname.com')
print(response.status_code)

Throws a RuntimeError ("not supported by httpcore package"):

Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.1.3\plugins\python-ce\helpers\pydev\pydevd.py", line 1500, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.1.3\plugins\python-ce\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:\Users\anonymous\AppData\Roaming\JetBrains\PyCharmCE2023.2\scratches\scratch_2.py", line 4, in <module>
    response = httpx.get('https://api.ipify.org', proxies='https://proxyname.com')
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_api.py", line 189, in get
    return request(
           ^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_api.py", line 100, in request
    return client.request(
           ^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 814, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 901, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 929, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 966, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 1002, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_transports\default.py", line 228, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_sync\connection_pool.py", line 262, in handle_request
    raise exc
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_sync\connection_pool.py", line 245, in handle_request
    response = connection.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_sync\http_proxy.py", line 317, in handle_request
    stream = stream.start_tls(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_backends\sync.py", line 149, in start_tls
    raise RuntimeError(
RuntimeError: Attempted to add a TLS layer on top of the existing TLS stream, which is not supported by httpcore package

Similar code using requests works just fine:

import requests

response = requests.get('https://api.ipify.org', proxies={'https': 'https://proxyname.com'})
print(response.status_code)

Result:

karpetrosyan · 2023-11-01T04:44:02Z

karpetrosyan
Nov 1, 2023
Collaborator

Just upgrade your httpx version to 0.25.0.

3 replies

mborsetti Nov 1, 2023
Author

Hello @karpetrosyan,

Thanks for your reply.

Just upgrade your httpx version to 0.25.0.

I am running 0.25.0, as confirmed by the below code:

import httpx

response = httpx.get('https://api.ipify.org', proxies='https://proxyname.com')
print(response.status_code)

httpx.__version__='0.25.0'
Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.1.3\plugins\python-ce\helpers\pydev\pydevd.py", line 1500, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.1.3\plugins\python-ce\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:\Users\anonymous\AppData\Roaming\JetBrains\PyCharmCE2023.2\scratches\scratch_2.py", line 4, in <module>
    response = httpx.get('https://api.ipify.org', proxies='https://proxyname.com')
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_api.py", line 189, in get
    return request(
           ^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_api.py", line 100, in request
    return client.request(
           ^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 814, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 901, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 929, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 966, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 1002, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_transports\default.py", line 228, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_sync\connection_pool.py", line 262, in handle_request
    raise exc
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_sync\connection_pool.py", line 245, in handle_request
    response = connection.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_sync\http_proxy.py", line 317, in handle_request
    stream = stream.start_tls(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_backends\sync.py", line 149, in start_tls
    raise RuntimeError(
RuntimeError: Attempted to add a TLS layer on top of the existing TLS stream, which is not supported by httpcore package

From my tests:

HTTPX allows an http:// (unencrypted) request to be proxied through a https:// (encrypted) proxy
HTTPX raises an httpcore RuntimeError when an https:// (encrypted) request is proxied through an https:// (encrypted) proxy

requests supports use case 2, as per code in my initial post.

Can HTTPX/httpcore be fixed to achieve parity?

karpetrosyan Nov 2, 2023
Collaborator

Thank you so much @mborsetti! This is a bug that will be fixed in the next release.
See: #840

karpetrosyan Nov 2, 2023
Collaborator

Resolved in #840

mborsetti · 2023-11-02T18:01:16Z

mborsetti
Nov 2, 2023
Author

@karpetrosyan Thanks for the quick action!

1 reply

tomchristie Nov 3, 2023
Maintainer

Would you like to retry now?

mborsetti · 2023-11-03T16:25:42Z

mborsetti
Nov 3, 2023
Author

Thanks for the fast turnaround, but ... hmmm?

Same code as above:

httpx.__version__='0.25.1'
Traceback (most recent call last):
  File "C:\Users\anonymous\AppData\Roaming\JetBrains\PyCharmCE2023.2\scratches\scratch_2.py", line 7, in <module>
    response = httpx.get('https://api.ipify.org', proxies='https://propxyname.com')
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_api.py", line 189, in get
    return request(
           ^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_api.py", line 100, in request
    return client.request(
           ^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 814, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 901, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 929, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 966, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_client.py", line 1002, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpx\_transports\default.py", line 228, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_sync\connection_pool.py", line 262, in handle_request
    raise exc
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_sync\connection_pool.py", line 245, in handle_request
    response = connection.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_sync\http_proxy.py", line 317, in handle_request
    stream = stream.start_tls(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python312\Lib\site-packages\httpcore\_backends\sync.py", line 149, in start_tls
    raise RuntimeError(
RuntimeError: Attempted to add a TLS layer on top of the existing TLS stream, which is not supported by httpcore package

I reran the version using requests to ensure there wasn't anything else going on, and that one still works.

0 replies

mborsetti · 2023-11-03T16:42:59Z

mborsetti
Nov 3, 2023
Author

Update: I thought of checking the version of httpcore, and so much for pip install --upgrade httpx (or maybe just bad timing), it was still the old version.

I've manually updated httpcore, and it's all working fine now!

Thanks for the quick turnaround!!

httpx.__version__='0.25.1'
httpcore.__version__='1.0.1'
<my response here>

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parity with requests: support https:// proxied over https:// #833

{{title}}

Replies: 4 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Parity with requests: support https:// proxied over https:// #833

mborsetti Oct 31, 2023

Replies: 4 comments · 4 replies

karpetrosyan Nov 1, 2023 Collaborator

mborsetti Nov 1, 2023 Author

karpetrosyan Nov 2, 2023 Collaborator

karpetrosyan Nov 2, 2023 Collaborator

mborsetti Nov 2, 2023 Author

tomchristie Nov 3, 2023 Maintainer

mborsetti Nov 3, 2023 Author

mborsetti Nov 3, 2023 Author

mborsetti
Oct 31, 2023

Replies: 4 comments 4 replies

karpetrosyan
Nov 1, 2023
Collaborator

mborsetti Nov 1, 2023
Author

karpetrosyan Nov 2, 2023
Collaborator

karpetrosyan Nov 2, 2023
Collaborator

mborsetti
Nov 2, 2023
Author

tomchristie Nov 3, 2023
Maintainer

mborsetti
Nov 3, 2023
Author

mborsetti
Nov 3, 2023
Author