Mechanism for denying specific IPs after they have been resolved

[simonw]

Please describe your feature request:

I want to use httpx to retrieve URLs that have been provided for me by end users.

I don't want those end users to be able to hit private URLs - like the AWS private http://169.254.169.254/latest/meta-data/ endpoint.

I can check the URLs that they are providing, but I can't easily guarantee that they haven't set up their own something.evil.com DNS entry that resolves to 169.254.169.254. Even if I resolve the hostname myself and verify its IP there's an advanced form of attack called DNS rebinding where they could potentially use a short-TTL DNS server of their own and swap the IP for an evil IP after I've checked it but before httpx fetches it.

It would be great if HTTPX had some kind of documented mechanism - subclassing a transport, or some kind of event hook perhaps? - which was specifically designed to help me run my own check against the resolved IP address before the request was sent to it.

Once I've validated the IP I'd like to be confident that HTTPX was hitting that specific IP with host: hostname.com such that I know it's not hitting some private IP address.

Describe the use case of this feature:

Helping people safely build systems that make requests against URLs provided by untrusted users.

[GeorginaReeder]

Thanks for your feature request @simonw , we'll take a look into this!

[dogancanbakir]

@simonw Thanks for raising this issue. We already have a mechanism that enables you to allow or deny specific IPs or CIDRs:

-allow string[]                  list of allowed IPs/CIDRs for processing (can be specified as a file or comma-separated)
-deny string[]                   list of denied IPs/CIDRs for processing (can be specified as a file or comma-separated)

Does this meet your needs?