Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

connection pool timeout - not reconnect #3289

Open
parsibox opened this issue Feb 25, 2025 · 13 comments
Open

connection pool timeout - not reconnect #3289

parsibox opened this issue Feb 25, 2025 · 13 comments

Comments

@parsibox
Copy link

hi
i use go 1.23 on linux 64 bit
today my redis has a problem and i restarted that but my golang code have error and not reconnected

this is my code

	RedisClient = redis.NewClient(&redis.Options{
		Addr:     ip + ":" + port, //"localhost:6379",
		Password: password,
		DB:       db,
		PoolSize: poolSize,
		ConnMaxLifetime: time.Minute * 10,
		DialTimeout:     time.Second * 10,
		ReadTimeout:     time.Second * 30,
		WriteTimeout:    time.Second * 10,
		MaxRetries:      3000,
		MinRetryBackoff: time.Second * 2,
		MaxRetryBackoff: time.Second * 10,
		OnConnect: func(ctx context.Context, cn *redis.Conn) error {
			logger.Info("ConRedis connected")
			return nil
		},
	})
	if err := RedisClient.Ping(context.Background()).Err(); err != nil {
		// Handle connection error
		logger.Error("ConRedis error")
	}

i get error : redis: connection pool timeout

why pool connection not renew?or not reconnect again?after i restarted my code it work !!!

@ndyakov
Copy link
Collaborator

ndyakov commented Feb 27, 2025

Hello @parsibox , what is the poolSize that you are using?

@parsibox
Copy link
Author

parsibox commented Feb 28, 2025

30
my redis only down for 15 min and after that i do not have any good connection on pool
all connection say "connection pool timeout" i think they should reconnect and work again

@monkey92t
Copy link
Collaborator

redis-server version?
go-redis version?

@y-kawawa
Copy link

y-kawawa commented Mar 10, 2025

I also encountered an issue where connection pool timeouts were not subject to retries.

  • client
    • go 1.23.7
    • go-redis v9.7.1
  • server
    • Valkey serverless engine version: 8
opts, _ := redis.ParseURL(url)
opts.DialTimeout = 1 * time.Second
opts.ReadTimeout = 1 * time.Second
opts.WriteTimeout = 1 * time.Second
opts.MaxRetries = 2

I suspect that the reason connection pool timeouts are not being retried is as follows:
ErrPoolTimeout is excluded in shouldRetry.

@monkey92t
Copy link
Collaborator

I also encountered an issue where connection pool timeouts were not subject to retries.

  • client

    • go 1.23.7
    • go-redis v9.7.1
  • server

    • Valkey serverless engine version: 8

opts, _ := redis.ParseURL(url)
opts.DialTimeout = 1 * time.Second
opts.ReadTimeout = 1 * time.Second
opts.WriteTimeout = 1 * time.Second
opts.MaxRetries = 2
I suspect that the reason connection pool timeouts are not being retried is as follows: ErrPoolTimeout is excluded in shouldRetry.

Indeed, this error was overlooked, and it can be fixed.

@monkey92t
Copy link
Collaborator

I also encountered an issue where connection pool timeouts were not subject to retries.

  • client

    • go 1.23.7
    • go-redis v9.7.1
  • server

    • Valkey serverless engine version: 8

opts, _ := redis.ParseURL(url)
opts.DialTimeout = 1 * time.Second
opts.ReadTimeout = 1 * time.Second
opts.WriteTimeout = 1 * time.Second
opts.MaxRetries = 2
I suspect that the reason connection pool timeouts are not being retried is as follows: ErrPoolTimeout is excluded in shouldRetry.

#3298

@parsibox
Copy link
Author

redis-server version? go-redis version?

Redis Version: 7.0.11
OS: Linux5.4.0-125-genericx86_64
github.com/redis/go-redis/v9 v9.7.1

@monkey92t
Copy link
Collaborator

redis-server version? go-redis version?

Redis Version: 7.0.11 OS: Linux5.4.0-125-genericx86_64 github.com/redis/go-redis/v9 v9.7.1

It is observed that you are using the latest version of go-redis and encountered the error "redis: connection pool timeout." This error corresponds to ErrPoolTimeout in go-redis (ErrPoolTimeout timed out waiting to get a connection from the connection pool), indicating that go-redis timed out while trying to obtain a connection from the connection pool. This suggests that there are no available connections in the pool, and the PoolSize limit has been reached. The timeout duration depends on opt.PoolTimeout (if not set, it defaults to ReadTimeout + 1s).

Your PoolSize is set to 30. From the scenario above, it can be inferred that your connection pool is exhausted. The connections might be waiting for read/write operations or could have been manually manipulated (e.g., via the Conn function). Could you provide an example that reproduces this issue?

@parsibox
Copy link
Author

redis-server version? go-redis version?

Redis Version: 7.0.11 OS: Linux5.4.0-125-genericx86_64 github.com/redis/go-redis/v9 v9.7.1

It is observed that you are using the latest version of go-redis and encountered the error "redis: connection pool timeout." This error corresponds to ErrPoolTimeout in go-redis (ErrPoolTimeout timed out waiting to get a connection from the connection pool), indicating that go-redis timed out while trying to obtain a connection from the connection pool. This suggests that there are no available connections in the pool, and the PoolSize limit has been reached. The timeout duration depends on opt.PoolTimeout (if not set, it defaults to ReadTimeout + 1s).  Your PoolSize is set to 30. From the scenario above, it can be inferred that your connection pool is exhausted. The connections might be waiting for read/write operations or could have been manually manipulated (e.g., via the Conn function). Could you provide an example that reproduces this issue?

i only first run a script in redis and that was heavy then i restart redis after that my redis pool connection get connection pool timeout
i think you should check first is there any live connection in connection pool or not if there is not then remove old connection from pool and create new connection in pool

@parsibox
Copy link
Author

or maybe there is a bug in remove died connection from pool

@monkey92t
Copy link
Collaborator

We cannot delete connections in the connection pool; they might be in use (or even in use for a long time), which is one of the connection pool's functions. Based on what you're describing, I believe heavy requests are causing the connection pool to be very busy. Am I understanding this correctly?

@joe-ton
Copy link

joe-ton commented Mar 18, 2025

Hey @parsibox, @monkey92t — I'd like to look into contributing here. I'm particularly interested in improving the robustness of the connection pool logic, specifically around handling timeouts, reconnect attempts, and stale connection management.

I've reviewed the discussion above and the details provided, and I'm ready to dig deeper into this. If there are specific areas of the codebase you think would benefit most from attention, please let me know. Otherwise, I'll start by investigating the handling of ErrPoolTimeout in the retry and connection pool logic.

Looking forward to helping here!

@ndyakov
Copy link
Collaborator

ndyakov commented Mar 24, 2025

@parsibox could you be able to test with the latest 9.8.0-beta.1 ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants