Question: Lettuce Find Keys Slow (KeyScanCursor)

[jfuehner]

I asked this question on StackOverflow back in April but haven't received any responses. Was hoping perhaps someone here may be able to assist...

https://stackoverflow.com/questions/67003553/lettuce-find-keys-slow-keyscancursor

I have been using the below code with Lettuce/Redis for a while now but as our key store grows I have noticed a significant slow down in finding keys based on a match pattern.

ScanArgs scanArgs = ScanArgs.Builder.limit(50).match("test*");
RedisAdvancedClusterCommands<byte[], byte[]> commands = getPoolConnection().sync();
KeyScanCursor<byte[]> cursor = commands.scan(scanArgs);

List<byte[]> retVal = new ArrayList<>();
while(!cursor.isFinished()) {
   retVal.addAll(cursor.getKeys());

   cursor = commands.scan(cursor, scanArgs);
}

return retVal;

If I increase the limit from say 50 to 5000 the performance is better. However, I am just wondering if that is the correct thing to do or if there is just a more performant implementation I could be using to find keys based on a pattern?

[mp911de]

I moved the ticket into discussions.

Generally, SCAN is the appropriate command to scan the entire keyspace. Batch sizes are a trade-off between server roundtrips and keeping the server busy. Since Lettuce isn't involved into how many keys you store inside Redis, there isn't much you can do from a client perspective. You should see a similar behavior with redis-cli and I suggest asking this question on the Redis mailing list.