Having issue integrating bloom filter support using JedisPooled

[Ranjeet116966]

I am currently working on integrating Bloom filter support into one of our features. For this, I’m using JedisPooled for the Bloom filter-related operations, as I believe Jedis itself doesn’t support these methods. After deploying these changes to production, I'm encountering an error ("Unexpected end of stream" / JedisConnectionException) on every second request, causing those requests to fail. I’m unsure whether this issue is due to the size of the connection pool or because the timeouts are set too low for Jedis.

I am using JedisPooled in the same way it’s mentioned in the README file, where I create an instance of JedisPooled like this and pass the custom config I have:

JedisPooled jedis = new JedisPooled("localhost", 6379);

Then, I issue commands on this jedis instance.

Jedis version: 4.2.2
Redis version: 7.2.4

Here is my current Jedis configuration:

connection_timeout: 200
read_timeout: 3000
max_total: 40
max_idle: 5
max_wait: 100

Can someone please take a look and help me understand if i am missing something, and if somehow i can get rid of these errors TIA...

[sazzad16]

@Ranjeet116966 Provide a short code that reproduces the issue and add the full stacktrace.

[Ranjeet116966]

Here is the full stack trace, will provide the code snippet in sometime, Actually i am integrating bloom filter in some gem, that we consume on some other repo, all the config that we need to initialise the connection pool would be passed from consumer end, so will have to create a short code file to replicate the same behaviour

ERROR jruby.app- Connection handling error: Unexpected end of stream. redis.clients.jedis.util.RedisInputStream.ensureFill(redis/clients/jedis/util/RedisInputStream.java:205) redis.clients.jedis.util.RedisInputStream.readByte(redis/clients/jedis/util/RedisInputStream.java:46) redis.clients.jedis.Protocol.process(redis/clients/jedis/Protocol.java:126) redis.clients.jedis.Protocol.read(redis/clients/jedis/Protocol.java:192) redis.clients.jedis.Connection.readProtocolWithCheckingBroken(redis/clients/jedis/Connection.java:316) redis.clients.jedis.Connection.getOne(redis/clients/jedis/Connection.java:298) redis.clients.jedis.Connection.executeCommand(redis/clients/jedis/Connection.java:123) redis.clients.jedis.executors.DefaultCommandExecutor.executeCommand(redis/clients/jedis/executors/DefaultCommandExecutor.java:24) redis.clients.jedis.UnifiedJedis.executeCommand(redis/clients/jedis/UnifiedJedis.java:148) redis.clients.jedis.UnifiedJedis.bfExists(redis/clients/jedis/UnifiedJedis.java:3923) java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498) org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:470) org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:328)

[Ranjeet116966]

Here is the basic code snippet you can refer to, this is how i have also implemented, and i am getting the above error after deploying this code to prod

require 'java'

require 'redis.clients:jedis-4.2.2.jar'
require 'org.slf4j:slf4j-api-2.0.13.jar'
require 'org.apache.logging.log4j:log4j-slf4j2-impl-2.23.1.jar'

java_import 'redis.clients.jedis.JedisPooled'
java_import 'redis.clients.jedis.JedisPoolConfig'
java_import 'redis.clients.jedis.bloom.RedisBloomCommands'

connection_timeout = 200
read_timeout = 3000

def initialize_pool(connection_timeout, read_timeout)
  jedis_pool_config = JedisPoolConfig.new
  jedis_pool_config.setMaxTotal(5)
  jedis_pool_config.setMaxIdle(5)
  jedis_pool_config.setMaxWaitMillis(100)
  jedis_pool_config.setBlockWhenExhausted(true)

  JedisPooled.new(
    jedis_pool_config,
    java.net.URI.create("redis://localhost:6380"),
    connection_timeout,
    read_timeout
  )
end

def execute_bloom_filter_command(pool)
  pool.bfAdd('bloom_filter_key', 'item')
  puts pool.bfExists('bloom_filter_key', 'item')
end

pool = initialize_pool(connection_timeout, read_timeout)
execute_bloom_filter_command(pool)

[sazzad16]

@Ranjeet116966 Couldn't reproduce your issue. Can't even guess what could go wrong within localhost.

• What's the timeout in Redis server? I.g. redis-cli ==> CONFIG GET timeout or Jedis ==> configGet("timeout").
• Could you try increasing the connection timeout? I.e. as high the read_timeout. (Seems doesn't matter in case, but just to be safe.)
• Could you try an updated Jedis version? If the latest version is not possible, at least Jedis 4.4.8?

[Ranjeet116966]

Reproducing the error is difficult because it's not very frequent, some of the request are passing and some request are failing so error is very inconsistent in my testing env as well.

• Redis timeout is set to 0 i think

CONFIG GET timeout
1) "timeout"
2) "0"

Will try increasing theconnect_timeoutand if that doesn't help will also upgrade the jedis version, Did we change anything related to this issue, i didn't find anything like that in release notes

[sazzad16]

... some of the request are passing and some request are failing ...

What's the ratio of passing and failing?

[Ranjeet116966]

it's almost 50:50 so basically half of the total requests are failing with above error

[Ranjeet116966]

Seems like none of the above solution is working for me, have tried all of that and still seeing the same issue. Let me know if there is anything that can help TIA!!

[Ranjeet116966]

could this be related to above error - #3864 ?

[sazzad16]

No. It doesn't seem so.

[sazzad16]

I'm encountering an error ("Unexpected end of stream" / JedisConnectionException) on every second request, ...

Reproducing the error is difficult because it's not very frequent, ...

Humbly speaking, I find the above two statements conflicting.

[Ranjeet116966]

by not very frequent i mean it's not consistent because as i said 50% request are still passing

[Ranjeet116966]

Hi, a quick update on this: It seems that changing the client-output-buffer-limit for pubsub to 0 has resolved the issue. However, I'm not entirely sure why this fixed it, as I’m not using the pubsub client for my specific use case.

CONFIG get client-output-buffer-limit:

"client-output-buffer-limit"
"normal 0 0 0 slave 268435456 67108864 60 ¸ 0 0 0"

[Ranjeet116966]

Hi @sazzad16,

Is there a significant difference between JedisPool and JedisPooled class? I've been using JedisPooled for running Bloom filter module commands, but I'm encountering above error. Other Redis commands still use JedisPool, and I tried switching all Redis commands to JedisPooled, but I'm getting an "Unexpected end of stream" error. The same commands work fine with JedisPool, but not with JedisPooled.

[Ranjeet116966]

Okay thanks @sazzad16 then i will try to use JedisCluster class to see if that resolve this issue.

[ayansaha23]

did it resolve, I get error bfReserve is not a member of redis.clients.jedis.JedisPool?

[sazzad16]

@ayansaha23 seems to be different issue.

bfReserve is not a method in JedisPool class. You would have to used JedisPooled class for that.

If you issue is not even that, please create a new discussion with more information.

[ayansaha23]

yes thanks I got it to work using JedisPooled class

[Ranjeet116966]

@ayansaha23 can you please tell what version of jedis library you are using and redis version as well?

[Ranjeet116966]

@sazzad16 I tried using JedisCluster, and it worked without any errors. However, my Redis cluster has 3 masters and 3 slaves, but since I'm using only one key for the Bloom filter, I can't effectively utilize the space across those nodes. Is it possible to use the JedisCluster class with a non-sharded Redis setup, and what would be the implications of doing so?

[ggivo]

@Ranjeet116966
No, JedisCluster is designed for Redis Cluster. It will try to load slot distribution on creation and will produce an error when connected to non-cluster env.

Have you tested with more recent Jedis version, as suggested before?
Error from initial issue report "Unexpected end of stream. " suggests that connection is forcibly closed while there is ongoing operation. I also tried to reproduce the mentioned error but without success.

Jedis : 4.4.8
Redis : docker run -p 6379:6379 redis/redis-stack:7.4.0-v1

    public static void testBloomFilters(String uri) {
        ConnectionPoolConfig poolConfig  = new ConnectionPoolConfig();
        poolConfig.setMaxTotal(5);
        poolConfig.setMaxIdle(5);
        poolConfig.setMaxWaitMillis(100);
        poolConfig.setBlockWhenExhausted(true);

        try (JedisPooled jedis = new JedisPooled(poolConfig, URI.create(uri),100,100);) {

            for (int i = 0; i < 100; i++) {
                jedis.bfAdd("bloom_filter_key", "item");
                boolean resp = jedis.bfExists("bloom_filter_key", "item");
                System.out.println("Response from Redis: " + resp);
            }

        } catch (Exception e) {
            System.err.println("Error connecting to Redis: " + e.getMessage());
        }
    }

[Ranjeet116966]

@ggivo yes i have tried these versions - 5.1.0, 5.2.0 but unfortunately nothing helped !!

[Ranjeet116966]

@ggivo one more thing which class should i use to setup the connection pool, right now i see you are using ConnectionPoolConfig and i have used JedisPoolConfig and when looking at the codebase seems like both classes are extending GenericObjectPoolConfig so which one is recommended approach for configuring connection pool?

[ggivo]

JedisPooled expects GenericObjectPoolConfig<Connection>. ConnectionPoolConfig is an OOB implementation providing defaults settings for theGenericObjectPoolConfig<Connection> pool and is used with JedisPooled.

Legacy 'JedisPool' class expects GenericObjectPoolConfig<Jedis> . JedisPoolConfig is an OOB implementation providing defaults for the GenericObjectPoolConfig<Jedis> pool and is used with 'JedisPool'

[ggivo]

I am not able to provide suggestions without having the issue isolated & reliably reproducible.
It might also be related to the ruby Java interoperability. For sure, the provided example script (considering it is using the latest Jedis version) is invoking a respective constructor using an object from the wrong type.

UnifiedJedis is the base class used both for clustered and non-clustered env depending on how it is created/initialized.
JedisPooled provides a pool of connections toward individual nodes.

[Ranjeet116966]

@ggivo I don't think this is a constructor issue anymore, i am passing the correct constructor now and we are already using jedispool and jediscluster class for our sharded and non sharded redis it's been working fine for us. we are using jruby and this is how we are instantiating JedisCluster and it's working fine in prod env -

` basic_cache_config = config['basic_cache']
          jedis_pool_config = ConnectionPoolConfig.new
          jedis_pool_config.setMaxTotal(basic_cache_config['connection_pool']['max_total'])
          jedis_pool_config.setMaxIdle(basic_cache_config['connection_pool']['max_idle'])
          jedis_pool_config.setMaxWaitMillis(basic_cache_config['connection_pool']['max_wait'])
          jedis_pool_config.setBlockWhenExhausted(true)

          jedis_cluster_nodes = java.util.HashSet.new
          redis_cluster_hosts_ports.each do |redis_cluster_host_port|
            host_and_port = HostAndPort.new(redis_cluster_connection[:host], redis_cluster_connection[:port])
            jedis_cluster_nodes.add(host_and_port)
          end

          JedisCluster.new(
            jedis_cluster_nodes,
            # Timeouts are in milliseconds.
            basic_cache_config['read_timeout'],
            jedis_pool_config
          )`

and something similar i am doing for instantiating JedisPooled class so it's doesn't seem to be an env issue because other two classes are working fine, and JedisCluster is also working with bloom filter, the only class that is not working for me intermittently is JedisPooled

basic_cache_config = config['basic_cache']
         jedis_pool_config = JedisPoolConfig.new
         jedis_pool_config.setMaxTotal(basic_cache_config['connection_pool']['max_total'])
         jedis_pool_config.setMaxIdle(basic_cache_config['connection_pool']['max_idle'])
         jedis_pool_config.setMaxWaitMillis(basic_cache_config['connection_pool']['max_wait'])
         jedis_pool_config.setBlockWhenExhausted(true)

     

           JedisPooled.new(
             jedis_pool_config,
             java.net.URI.create("redis://#{redis_connection[:host]}:#{redis_connection[:port]}"),
             basic_cache_config['connection_timeout'],
             basic_cache_config['read_timeout']
           )

I am suspecting this could be the cause - #1029 (comment) but even setting up redis timeout to 0 and keeping no idle connection on pool is not working i am still seeing few failures.

[ggivo]

@Ranjeet116966
Your latest example still uses the wrong type for jedis_pool_config.
There is no constructor in JedisPooled that accepts JedisPoolConfig. You need to provide ConnectionPoolConfigsimilar to the example with JedisCluster,

[Ranjeet116966]

@ggivo sorry shared the old code i am using ConnectionPoolConfig only but even that is not working for me -

` basic_cache_config = config['basic_cache']
          jedis_pool_config = ConnectionPoolConfig.new
          jedis_pool_config.setMaxTotal(basic_cache_config['connection_pool']['max_total'])
          jedis_pool_config.setMaxIdle(basic_cache_config['connection_pool']['max_idle'])
          jedis_pool_config.setMaxWaitMillis(basic_cache_config['connection_pool']['max_wait'])
          jedis_pool_config.setBlockWhenExhausted(true)
          jedis_pool_config.setTestWhileIdle(true)

            JedisPooled.new(
              jedis_pool_config,
              java.net.URI.create("redis://#{redis_connection[:host]}:#{redis_connection[:port]}"),
              basic_cache_config['connection_timeout'],
              basic_cache_config['read_timeout']
            )`

[Ranjeet116966]

@ggivo do you think this could be the reason - I am suspecting this could be the cause - #1029 (comment) but even setting up redis timeout to 0 and keeping no idle connection on pool is not working i am still seeing few failures.

[Ranjeet116966]

Hi @garydgregory,

I wanted to tag you in this discussion to check if the issue I’m facing could be related to the Apache Commons Pool library. I noticed that a similar issue was raised in the past: Jedis Issue #3106.

Could you confirm if any changes were made to Apache Commons Pool that might be relevant here?

For reference, I am using Apache Commons Pool version 2.11.1.

Thanks in advance!

[garydgregory]

Hello @Ranjeet116966

I can't really say anything aside from try the current version of Commons Pool (2.12.1) and you could also try Commons Collections 4.5.0-M3 if you want to try our Bloom Filter implementation.

If you think there is an issue in Commons Pool, I'll need a reproducible test case either as a GitHub PR or an attachment to a Jira issue, see https://commons.apache.org/proper/commons-pool/issue-management.html

HTH

[Ranjeet116966]

Thanks, @garydgregory, for the response! I'll try out the latest version of Commons Pool and also take a look at the Bloom filter implementation in Commons Collections 4.5.0-M3.

I have a question regarding how this would work with Redis. Currently, I’m using the RedisBloom module, which I install on the Redis server and interact with using the Jedis client. Given this setup, how would Commons Collections' Bloom filter implementation integrate with Redis?

Would there be additional steps needed to make it work in my current environment?

[garydgregory]

Hello @Ranjeet116966

The Apache Commons Collections Bloom Filter package is standalone, so you must do whatever Jedis needs to integrate it.

• Introduction: https://commons.apache.org/proper/commons-collections/bloomFilters/intro.html
• Package Javadoc: https://commons.apache.org/proper/commons-collections/apidocs/org/apache/commons/collections4/bloomfilter/package-summary.html