how to optimize or troubleshoot Apiman Gateway performance issue

[ronimhd]

I was performing load test to our Apiman deployment. The load test is performed using Locust. I compared the result of directly call to the API or call via Apiman Gateway.
The strange is, if we ran load test to direct call, RPS (Request per Second) is increased according to concurrent users in Locust. But if we ran load test to the API via Apiman Gateway, the Request per Second is stuck in 7. That it seems there is bottleneck.
What I need to check for Apiman Gateway configuration or another deployment option ?
Note: I deployed the Apiman 3.1.2 in Kubernetes, and the API is in the same K8s cluster

[msavy]

Start with --debug and attach your debugger, then have a look at why/where the Vert.x threads are stuck. Set your debugger not to pause threads.

It's usually a misbehaving custom policy, but there's no way for me to know from the information provided.

Something like Java Flight Recorder and/or monitoring tools are likely also useful.

[ronimhd]

fyi, I created a custom policy, but for this load test, I do not apply any policy, neither in Organization, API and Client level

[ronimhd]

Hi, I can increase performance by change in this part:

I believe, the issue is because there is only on HttpClient for a target end point, hence this is the bottleneck.
But is it optimum or proper solution ?

[msavy]

It may just be a case of increasing the permitted pool size for each HttpClient. There's nothing wrong with re-using existing HttpClient.

I guess your backend services must be HTTP 1.1?

Try setting maxPoolSize to something larger (and/or possible http2MaxPoolSize).

I'll try to make these easier to configure in Apiman 4.

[As a very small aside, please post code snippets as code blocks so that it is easier to read for accessibility and reasons.]