HTTPX POST requests have different body sizes compared to Requests library, causing server errors

[denallo]

I've encountered an issue where the POST requests sent using the httpx library have a different body size compared to those sent using the requests library, resulting in server errors. Here is a detailed description and steps to reproduce the issue.

Reproduction Steps:

Send a POST request using the requests library with the following code:

import requests
import json

payload = b'{"messages": [{"content": "You are a helpful assistant.", "role": "system"}, {"content": "who is the winner of the us open", "role": "user"}], "model": "llama3.1:8b", "n": 1, "stream": false, "temperature": 0.7, "tools":[]}'
json_data = json.loads(payload.decode("utf-8"))

url = 'http://localhost:11434/v1/chat/completions'
headers = {
    'Content-Type': 'application/json'
}

response = requests.post(
    url=url,
    headers=headers,
    json=json_data,
    stream=False,
)
print(response.text)

Send a POST request using the httpx library with the following code:

import httpx
import json

payload = b'{"messages": [{"content": "You are a helpful assistant.", "role": "system"}, {"content": "who is the winner of the us open", "role": "user"}], "model": "llama3.1:8b", "n": 1, "stream": false, "temperature": 0.7, "tools":[]}'
json_data = json.loads(payload.decode("utf-8"))

url = 'http://localhost:11434/v1/chat/completions'
headers = {
    'Content-Type': 'application/json'
}

response = httpx.post(
    url, 
    headers=headers,
    json=json_data
)
print(response.text)

Compare the two requests using packet capture, and notice that httpx sends 3 bytes less than requests, it caused lossing of "[]}"

Environment Information:

httpx version: 0.27.2
requests version: 2.32.3
Python version: Python 3.11.9
Operating System: win10 22H2

Additional Information:

The port of 11434 was an ollama server, you can simply use python -m http.server 11434 to simulate it.

[tomchristie]

Closing as time wasting.

[denallo]

This situation was discovered while I was using LangChain, and I spent quite a bit of time single-step debugging from LangChain to ChatOpenAI and then to httpx, unwrapping layer by layer. In order to make the problem clearer, I simplified the code; yet, all these efforts to save others' time were ultimately met with a "Closing as time wasting." How ironic.

[denallo]

Alright, by reading https://www.python-httpx.org/quickstart/, I have gained a better understanding of this issue. The same data packet is truncated at the end when requesting via HTTP service, but it doesn't happen with HTTPS.

Here is the code:

import httpx
url = "http://httpbin.org/post"  # or "https://httpbin.org/post"
print(f'url: {url}')
json_data = {'integer': 123, 'boolean': True, 'list': ['a', 'b', 'c']}
response = httpx.post(
    url, 
    json=json_data
)

print(response.text)

When I make a request to HTTPS, the actual data sent is like this, which is great, no issues at all, excellent:

{
  "args": {},
  "data": "{\"integer\": 123, \"boolean\": true, \"list\": [\"a\", \"b\", \"c\"]}",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate",
    "Content-Length": "58",
    "Content-Type": "application/json",
    "Host": "httpbin.org",
    "User-Agent": "python-httpx/0.27.2",
    "X-Amzn-Trace-Id": "Root=1-6738291d-455ff2570a12254822da8a49"
  },
  "json": {
    "boolean": true,
    "integer": 123,
    "list": [
      "a",
      "b",
      "c"
    ]
  },
  "url": "https://httpbin.org/post"
}

However, when making a request to HTTP, it becomes like this (note the end of the data field):

{
  "args": {},
  "data": "\r\n\r\n{\"integer\": 123, \"boolean\": true, \"list\": [\"a\", \"b\", \"",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate",
    "Content-Length": "58",
    "Content-Type": "application/json",
    "Host": "httpbin.org",
    "User-Agent": "python-httpx/0.27.2",
    "X-Amzn-Trace-Id": "Root=1-6738294e-45a50c67728ef49b16b25820"
  },
  "json": null,
  "url": "http://httpbin.org/post"
}

[eij-iew]

The "json" in second response is null?

[denallo]

yeah, and the "data" field was begin at "\r\n\r\n" which was not in json_data

Just now, my friend helped me test it, and he said there was no issue on his end. He also asked if I had a proxy enabled on my side. After I turned off the proxy and tested again, sure enough, there was no problem.