HttpClient proxy configuration per request

[jordillachmrf]

As explained here https://stackoverflow.com/questions/74893474/is-there-any-way-to-imlement-a-rotating-proxy-based-httpclient-in-reactor-netty it would be nice if reactor-netty HttpClient could be configured in a per request basis, instead of being "statically" configured as explained in the docs https://projectreactor.io/docs/netty/1.1.5/reference/index.html#_proxy_support_2

Motivation

Some of our clients are GDRP constrained and based on the location of their servers and our servers(eeuu vs eu) we get 451 status codes(legal reasons)

Desired solution

HttpClient client = HttpClient.create() .proxy((spec, req) -> spec.type(ProxyProvider.Proxy.HTTP) .host("proxy") .port(8080) .nonProxyHosts("localhost") .connectTimeoutMillis(20_000));
req -> HttpClientRequest, if HttpClientRequest would be available then we could route the request to our eeuu proxy or eu proxy based on the destination uri.

Considered alternatives

Have various HttpClients's, handle 451 in our code and fallback to the appropriate HttpClient at runtime

[pderop]

I will take a look at this issue soon.

[violetagg]

@jordillachmrf Can you elaborate more about the desired solution? You know the destination URI as this is something the the one that uses the HttpClient provides. Based on this you can then choose which HttpClient to use i.e. the one configured with proxy for x or the one configured with proxy for y.

[jordillachmrf]

And that's exactly how we are tackling this, but keeping the routing solution close the proxy configuration seems to us more appropriate than moving this logic one layer above.
We have various "services" that use these various proxied webClients, we have had to wrap these into an artificial layer to hide this complexity.
We were wondering that adding this flexibility when configuring the proxy would allow us to get rid of this

[violetagg]

@jordillachmrf I'm gonna add this to the general backlog. For the time being please use the described solution in my previous comment.
The team will discuss when and how this can be implemented. If you can work on this feature and provide a PR, this will be great.

[longkunbetter]

@violetagg Is somebody working on this problem?

[violetagg]

@longkunbetter We are not working on this one. I marked this with help wanted if somebody wants to work on this.

[punitdarira]

Hi @violetagg,
What would be the best way to implement this? ReqeustConfig like Apache HTTPClient or some other way? like a proxy() method builder itself

[violetagg]

@punitdarira Do you want to provide a PR? If yes - some rough idea is to plug the implementation in reactor.netty.http.client.HttpClientConnect.MonoHttpConnect#subscribe, in addition to something like what we have for headers (reactor.netty.http.client.HttpClient#headersWhen). I haven't tried that!

[violetagg]

If you would like to have a proxy per request then what should be the destination URL used for selecting proxy settings?

• the full URL i.e. http://host:port/path/...
• or just /path/...

[raccoonback]

@violetagg
Hello!
May I take charge of this issue and work on resolving it? 😊

[violetagg]

@violetagg Hello! May I take charge of this issue and work on resolving it? 😊

sure, go for it then

[raccoonback]

@violetagg
Hello!

I have a few questions regarding this issue:

1. Dynamic Proxy Configuration Scope

Is the functionality for configuring a dynamic proxy for each request supported only at the HTTP protocol level?
Or is it correct to assume that it is not supported at the TCP protocol level?

2. proxyWhen() Method Interface Design

I am referring to reactor.netty.http.client.HttpClientConnect.MonoHttpConnect#subscribe and reactor.netty.http.client.HttpClient#headersWhen to implement dynamic proxy configuration.
If I were to provide a proxyWhen() method that supports dynamic proxy configuration for each request, what type of parameter would be appropriate for the higher-order function? (In the example below, T)

For example, I am considering the following method signature. I would appreciate your feedback or any suggestions for improvement:

HttpClient.create()
  	.proxyWhen(
		(T whatFactor, ProxyProvider.TypeSpec spec) -> {
                            spec.type(ProxyProvider.Proxy.HTTP)
					.host("localhost")
					.port(8080)
					.connectTimeoutMillis(20_000)
                 }
	);

The HttpClientRequest mentioned above seems challenging to use since it appears to allow injection only after the connection is established.
(If I’m misunderstanding something, please feel free to let me know.)

Given this limitation, what kind of information or object would you recommend as an alternative for implementing dynamic proxy configuration?

I would greatly appreciate any additional guidance or ideas related to this topic.
Thank you!

[violetagg]

@raccoonback I haven't thought of a concrete implementation. Let me think about it and I'll come back to you.

[raccoonback]

@violetagg Thank you!

[violetagg]

@raccoonback

Is the functionality for configuring a dynamic proxy for each request supported only at the HTTP protocol level?
Or is it correct to assume that it is not supported at the TCP protocol level?

We want it only for HttpClient

If I were to provide a proxyWhen() method that supports dynamic proxy configuration for each request, what type of parameter would be appropriate for the higher-order function? (In the example below, T)

I think we can only provide HttpClientConfig, which I think is enough as one can obtain information for everything even the configured uri.

[raccoonback]

@violetagg
Thank you for confirming!
I will proceed with the work based on your input!