Consider impact of proxy-initiated close on HTTP 407

[bemasc]

@ekinnear noted today that the performance impact of proxies closing the connection after returning HTTP 407 are not merely hypothetical, and are significant in practice. This draft currently says that proxies MAY adopt this behavior as a mitigation for potential misbehaving clients.

Some things we could do:

• Remove this MAY. After a closer look at some earlier anecdotes, we have not actually found a client that exhibits the misbehavior.
• Conduct some kind of study or outreach to see if we can find any misbehaving clients (i.e. CONNECT clients that send first-flight payload data without waiting for the 200 response).
• Define a request header like "Vulnerable-To-Request-Smuggling: ?0", by which the client promises that it is proxying TCP connections for a trusted TCP client or delaying payload data until after the success response, so there is no need for defensive measures.

[MikeBishop]

Definitely not the last one -- too close to the Evil Bit. I'd be inclined to leave this as MAY, but not stronger, and expand on the text a little bit. It's a defense that has a performance cost, and implementers need to decide whether they're willing to pay the price to have that defense. That's what MAY is -- we don't endorse it, we just note the option.

The right solution is to use HTTP/2 or HTTP/3, where this won't be an issue.

[bemasc]

By trolling through Github, I was able to find many examples of proxy clients that appear to have the wrong behavior:

• https://github.com/lonely-caat/stuff/blob/100daaee4d1707900abdb24300fcdc30356adb60/proxyssl.py#L54
• https://github.com/mzdluo123/bypass_block/blob/002dd2975615eb675a7c0dd8e65b15f79ace35ec/socks_client.py#L65
• https://github.com/qwj/python-proxy/blob/c3a8446c415d53a7a8addfbc5ea37ea27de80504/pproxy/proto.py#L331
• https://github.com/binux/pyproxy/blob/fc6050fdce364dc525b8ae8be51b2088e13b6fcd/pyproxy.py#L194

Most of these are not an "early send" optimization as I expected. Instead, most of them wait for the response but assume a "200 OK" without checking the status code.

These clients might all be intended to be used in contexts where this behavior doesn't create a vulnerability, but it does seem to suggest that there is an easy mistake here that justifies the MAY.