can't reach maximum performance of lettuce( spirng data redis reactive, webflux)

[mouse500]

I wrote a sample WebFlux application, which read just some data from redis and do small cpu job(md5 calculation), repeats several time.

It use spring data redis reactive

 'org.springframework.boot:spring-boot-starter-data-redis-reactive'

Codes that connects redis is like this.

 reactiveStringRedisTemplate.opsForValue().get(keyname)

redis server runs at localhost

 redis-server &

Full codes , you can find it here

https://github.com/mouse500/redperf

It's a simple codes

And for test, I call API (/wredis) with Jmeter to loadtest

Problem that I think is...

this application doesn't reach maximum TPS,

It reaches CPU around 40% on my local PC.

Even though it has more room of CPU, it doesn't work any harder

Why doesn't it use resource fully?

---

In case of some other method to connect redis,

(I put connection proxy ,written in nodejs)

It showed much much higher CPU and got much more TPS.

So I don't think it is not about Redis server performance.

Problem seems issue at "calling Redis from WebFlux application with lettuce"

---

How can I make this sample application show a maximum TPS ( CPU reach 100%) ?

What option can I try?

[mp911de]

This is a duplicate of #1292. Avoid high CPU workloads on I/O threads.

[mouse500]

Thank you for your answer.

Avoiding CPU workloads is not my test situation.
Some CPU workloads on this test codes is some simulation of real workloads.

Even though I put some CPU workload, it still has CPU room to work more. ( CPU level is not full, but just 40%)
Isn't this mean that this codes can work more( fake CPU workloads or even interact with redis server)?

Or,
your key point is ...
CPU workload need to be on another thread than lettuce I/O threads? (not CPU load it self?)
If then. how can I achieve this? Could you give a small guide for this? I'll try.

[mouse500]

ah. ok. i think i need to more think about #1292

[mouse500]

Avoiding cpu workload from I/O thread was a good clue to me
I tried several options(regarding putting publishOn, where, to which thread).
After all, putting publishOn to Schedules.parrellel() right after accessing redis via lettuce shows pretty better performance,

But,
still it doesn't touch CPU 100%.
I think server still have a room to work more. (now it shows CPU 70% level, yes, it is much better)
I want to make maximum performance regarding interacting with redis.

What resource could be a bottleneck ? ( this sample prj is quite simple)
What point am I missing?
Could someone give a small opinion about this?

[mp911de]

On a modern multi-core CPU, you won't see 100% usage when a single core is busy with CPU-bound workloads. The underlying issue is that when running CPU-heavy operations on I/O threads, then the threads are busy with your workload instead of communicating with Redis. A single connection is tied to a single I/O thread and a thread can do only one thing at a time.

[mouse500]

Thank you very much for your valuable explanation.
Your advice always give me a good inspiration and good guide.
Now let me give some more situation.

I think we have some confusion about CPU workload,
My intention was, that was just some small workload to simulate real case.(one time md5 hash calculation)
So, in this experiment. lets remove CPU workload..

What if I remove CPU workload?
Now the only thing codes do is accepting api call, and interact with redis(storing and fetching 15kb payload)?
Redis I/O therad can do more accessing redis? Because no more thing to do at thread? and now I/O thread can do more work for redis accessing?
can it show more TPS?

Experiment(, I just did) shows no change at TPS (with or without CPU workload)
It still shows almost same CPU level( a little bit lower because CPU workload remvoed)
almost same CPU level and same TPS(means, same Redis accessing count)

"On a modern multi-core CPU, you won't see 100%" => this cannot be reasonable, Computer Science need to show optimal performance if all other resource is enough, I think the lettuce need to show maximum performance at CPU 100%.

For comparison, when I test similar situation with only http outer call(WebClient accessing some nodejs service and that service accessing Redis with node's redis lib) , It shows maximum performance, WebFlux with WebClient + nodesjs accessing Redis => shows maximum TPS and it touches CPU 100%.
This is very reasonable, and this is my final target performance.

It seems there is some other bottleneck resource, I'm not sure as of now whether it is about lettuce thread or not.
Still I'm investigating.
If you have good opinion on this. I'm ready to listen always. Thank you very much.

---

btw, at that point(while load test), its thread dump shows....
12 all of "lettuce-eventExecutorLoop-3-..." thread 's stacktrace are like this.

  "threadState": "TIMED_WAITING",
  "stackTrace": [
    {
      "methodName": "park",
      "fileName": "Unsafe.java",
      "lineNumber": -2,
      "className": "sun.misc.Unsafe",
      "nativeMethod": true
    },
    {
      "methodName": "parkNanos",
      "fileName": "LockSupport.java",
      "lineNumber": 215,
      "className": "java.util.concurrent.locks.LockSupport",
      "nativeMethod": false
    },
    {
      "methodName": "awaitNanos",
      "fileName": "AbstractQueuedSynchronizer.java",
      "lineNumber": 2078,
      "className": "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject",
      "nativeMethod": false
    },
    {
      "methodName": "poll",
      "fileName": "LinkedBlockingQueue.java",
      "lineNumber": 467,
      "className": "java.util.concurrent.LinkedBlockingQueue",
      "nativeMethod": false
    },
   ......

and the "lettuce-epollEventLoop-5-1" is like this....
(and there is only one lettuce-epoolEventLoop thread)

  "threadState": "RUNNABLE",
  "stackTrace": [
    {
      "methodName": "unpark",
      "fileName": "Unsafe.java",
      "lineNumber": -2,
      "className": "sun.misc.Unsafe",
      "nativeMethod": true
    },
    {
      "methodName": "unpark",
      "fileName": "LockSupport.java",
      "lineNumber": 141,
      "className": "java.util.concurrent.locks.LockSupport",
      "nativeMethod": false
    },
    {
      "methodName": "unparkSuccessor",
      "fileName": "AbstractQueuedSynchronizer.java",
      "lineNumber": 662,
      "className": "java.util.concurrent.locks.AbstractQueuedSynchronizer",
      "nativeMethod": false
    },
    {
      "methodName": "release",
      "fileName": "AbstractQueuedSynchronizer.java",
      "lineNumber": 1264,
      "className": "java.util.concurrent.locks.AbstractQueuedSynchronizer",
      "nativeMethod": false
    },
    {
      "methodName": "unlock",
      "fileName": "ReentrantLock.java",
      "lineNumber": 457,
      "className": "java.util.concurrent.locks.ReentrantLock",
      "nativeMethod": false
    },

Can it be some clue for investigating situation?

[mouse500]

I'm back again.

Still I'm curious about achieving CPU max utilzation, but "avoiding CPU load from I/O thread" has good performance gain even though it is not maximum expected.
I'll apply it to my production codes, Thank you very much.

I'm quite happy with this valuable conversation.
Thank you