Add support for Redis Cluster

[maeln]

Hello!

This PR aim to add the support for Redis Cluster: https://redis.io/docs/management/scaling/
This is different than Redis Sentinel, which is already supported.

Since this is a bit more complicated to use than a normal redis instance, I try to put more complete example in the documentation.
Redis Cluster is supported since version 4.1 of redis-py, so it should be pretty stable.

Open to feedback :)

[zzzeek]

hey there -

how would this be tested? Are there directives you can add to tox.ini to run redis in this mode?

[maeln]

Hi,
It should be testable by creating a redis cluster locally, using for example the cli in the redis documentation: https://redis.io/docs/management/scaling/#create-a-redis-cluster

I tested using one redis cluster i had laying around and it was working.
I will try to to write a small test procedure tomorow and I will look if anything is needed for the tox file

EDIT: I see that it is not the first time :p #135

[zzzeek]

if you can get it into the tox file then we can just test on CI

[maeln]

To make it work, I need to update pifpaf so that they can spawn a redis-cluster. I am working on it, and i will submit a patch to them

[maeln]

Made a PR to pifpaf to add Redis Cluster support: jd/pifpaf#163

[zzzeek]

wow, thanks for that effort. the pifpaf project seems kind of dead lately so hopefully someone can accept that PR. otherwise we might have to go without CI testing here for now

[sqla-tester]

OK, this is sqla-tester setting up my work on behalf of zzzeek to try to get revision 6875e4e of this pull request into gerrit so we can run tests and reviews and stuff

[sqla-tester]

New Gerrit review created for change 6875e4e: https://gerrit.sqlalchemy.org/c/sqlalchemy/dogpile.cache/+/5033

[maeln]

Seems like I have a few formatting issue to fix. I will take care of it soon

[zzzeek]

OK, can you also add a changelog file? see a commit like 074cb77 to see an example of what this file looks like. this would be :tags: feature, redis and :tickets: 250 since there is no separate issue here. feel free to credit yourself in the changelog.

[maeln]

Will do!

[maeln]

I wrote a changelog, it is a bit dry but I didn't know what more to say, and I also check the generated doc + run tox. Everything seems fine :) . Hopefully pifpaf merge the PR and we can add some tests. I was in contact with jd a few months ago on a completely separate thing, so maybe I can try to send him a quick mail, but I think founding his company as kept him quite busy, so we will see.

[zzzeek]

oh is that what he did huh.

[sqla-tester]

OK, this is sqla-tester setting up my work on behalf of zzzeek to try to get revision d112f02 of this pull request into gerrit so we can run tests and reviews and stuff

[sqla-tester]

Patchset d112f02 added to existing Gerrit review https://gerrit.sqlalchemy.org/c/sqlalchemy/dogpile.cache/+/5033

[zzzeek]

either way, this is a totally new backend that wont break anything, it's only a few lines of startup code, and im sure you're using it on your end so it's not too controversial to just merge it

[CaselIT]

seem we could spin up a docker image with a redis cluster https://github.com/Grokzen/docker-redis-cluster

[maeln]

If you want to test locally, you should be able to spawn a working redis cluster with the pifpaf fork I made : https://github.com/maeln/pifpaf

Using the command :

python -m pifpaf run redis --cluster

[zzzeek]

any news on this?

[maeln]

Hello! Sadly, no news on the pifpaf side of things :/ . I don't know if there is any other way forward if we want to merge it

[zzzeek]

well we'd just merge it. you've tested this on your end in a real program right?

also @CaselIT we might need to find an alternative to pifpaf, maybe home roll

[CaselIT]

maybe home roll

zifzaf? 😨

are there no alternative?

[CaselIT]

Also not sure what's the state of pifpaf, but it seems it's still active, meaning there were commits to it recently (last month)

seem we could spin up a docker image with a redis cluster https://github.com/Grokzen/docker-redis-cluster

would this be an option mike?

[zzzeek]

I would assume everyone has shell scripts that run containers these days

[sqla-tester]

Michael Bayer (zzzeek) wrote:

We can't get a CI run as pifpaf does not support the backend, however this is a totally new add that does not affect any existing code so we can just merge this.

View this in Gerrit at https://gerrit.sqlalchemy.org/c/sqlalchemy/dogpile.cache/+/5033

[sqla-tester]

Gerrit review https://gerrit.sqlalchemy.org/c/sqlalchemy/dogpile.cache/+/5033 has been merged. Congratulations! :)

[maeln]

Hi! Sorry for the delay, life got in the way ...

@zzzeek Yes, I tested it in a real product using a simple local setup (spawning manually 3 redis server in cluster mode) and the Scaleway redis cluster managed service. I also tested on a more synthetic setup and so far everything was working great.

Thank you for the merge :) . Things are a bit complicated for me these days, but when it settles down, I will try to go back to the pif paf PR to see if I can get it to be merged