[filebeat]Suspect Redis connection pool configuration issue causing discrepancy between maximum connection settings and actual results

[wang-qijia]

Description:
I am experiencing an issue with the maximum connection settings in Redis not matching the expected behavior. I suspect that there might be a problem with the Redis connection pool configuration.

Specifically, I am using Filebeat to collect Redis slow logs, but the maximum number of connections that are being established does not align with the settings I have configured. This has led me to suspect that there might be an issue with the connection pool parameters for Redis.

Could someone please help me troubleshoot this issue? I would like to understand if my Redis connection pool configuration is incorrect or if there are any other potential reasons for this discrepancy.

Thank you!

[botelastic]

This issue doesn't have a Team:<team> label.

[shmsr]

@wang-qijia Thank you for opening an issue. Could you please elaborate and could you also please send the configuration?

[wang-qijia]

filebeat.inputs:
- type: redis
  hosts: ["localhost:6379"]
  timeout: 1
  scan_frequency: 2
  maxconn: 1


output.console:
  pretty: true

Maximum number of connections : https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-redis.html#redis-maxconn

[shmsr]

@wang-qijia If you see:

beats/filebeat/input/redis/input.go

Line 130 in e1aa970

MaxIdle: maxConn,

You'd notice the maxconn that you have provided is setting MaxIdle. There's another parameter that is configurable (although not exposed for you to set) is MaxActive.

Definition of MaxIdle and MaxActive in the context of the Redis client that we have chosen:

// Maximum number of idle connections in the pool.
MaxIdle int

// Maximum number of connections allocated by the pool at a given time.
// When zero, there is no limit on the number of connections in the pool.
MaxActive int

Ref: https://pkg.go.dev/github.com/gomodule/[email protected]/redis#Pool

A good thread on StackOverflow discussing about MaxActive and MaxIdle in general: https://stackoverflow.com/a/54085623/5821408

Also, if you read the definition of MaxActive it even says that when it is zero, there is no limit on the number of connections in the pool. I think that explains the situation you are facing. So it is working as expected as maxconn is MaxIdle.

[wang-qijia]

@wang-qijia If you see:

beats/filebeat/input/redis/input.go

Line 130 in e1aa970

MaxIdle: maxConn,

You'd notice the maxconn that you have provided is setting MaxIdle. There's another parameter that is configurable (although not exposed for you to set) is MaxActive.

Definition of MaxIdle and MaxActive in the context of the Redis client that we have chosen:

// Maximum number of idle connections in the pool.
MaxIdle int

// Maximum number of connections allocated by the pool at a given time.
// When zero, there is no limit on the number of connections in the pool.
MaxActive int

Ref: https://pkg.go.dev/github.com/gomodule/[email protected]/redis#Pool

A good thread on StackOverflow discussing about MaxActive and MaxIdle in general: https://stackoverflow.com/a/54085623/5821408

Also, if you read the definition of MaxActive it even says that when it is zero, there is no limit on the number of connections in the pool. I think that explains the situation you are facing. So it is working as expected as maxconn is MaxIdle.

Thank you for your response.

What I aim to achieve is to ensure that the maximum number of connections remains within the expected limit and that the slow log collection does not impact the core business operations.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!