eredis_cluster

eredis_cluster is a wrapper for eredis to support cluster mode of Redis 3.0.0+

Contents

• History
• Usage
• Compilation and tests
• Configuration
• Troubleshooting
• See also

History

This project was started by Adrien Moreau in 2015. In 2021, maintainance was taken over by the Nordix Foundation (backed by Ericsson) and the Hex package is released from the Nordix fork since 0.6.0.

See also CHANGELOG.md.

Usage

For the full reference manual, see the generated documentation in doc/eredis_cluster.md.

%% Start the application and, if init nodes are defined in the application
%% configuration, connect to the cluster (otherwise use connect/1,2)
eredis_cluster:start().

%% Simple command
eredis_cluster:q(["GET","abc"]).

%% Pipeline
eredis_cluster:qp([["LPUSH", "a", "a"], ["LPUSH", "a", "b"], ["LPUSH", "a", "c"]]).

%% Pipeline in multiple node (keys are sorted by node, a pipeline request is
%% made on each node, then the result is aggregated and returned. The response
%% keep the command order
eredis_cluster:qmn([["GET", "a"], ["GET", "b"], ["GET", "c"]]).

%% Transaction (a pipeline wrapped in MULTI-EXEC; returns the result of EXEC)
eredis_cluster:transaction([["LPUSH", "a", "a"], ["LPUSH", "a", "b"], ["LPUSH", "a", "c"]]).

%% Transaction Function
TransactionFun = fun(Worker) ->
    eredis_cluster:qw(Worker, ["WATCH", "abc"]),
    {ok, Var} = eredis_cluster:qw(Worker, ["GET", "abc"]),

    %% Do something with Var %%
    Var2 = binary_to_integer(Var) + 1,

    {ok, Result} = eredis_cluster:qw(Worker,[["MULTI"], ["SET", "abc", Var2], ["EXEC"]]),
    lists:last(Result)
end,
eredis_cluster:transaction(TransactionFun, "abc").

%% Optimistic Locking Transaction
Function = fun(GetResult) ->
    {ok, Var} = GetResult,
    Var2 = binary_to_integer(Var) + 1,
    {[["SET", Key, Var2]], Var2}
end,
Result = optimistic_locking_transaction(Key, ["GET", Key], Function),
{ok, {TransactionResult, CustomVar}} = Result.

%% Atomic Key update (using optimistic locking transaction)
Fun = fun(Var) -> binary_to_integer(Var) + 1 end,
eredis_cluster:update_key("abc", Fun).

%% Atomic Field update (using optimistic locking transaction)
Fun = fun(Var) -> binary_to_integer(Var) + 1 end,
eredis_cluster:update_hash_field("abc", "efg", Fun).

%% Pre-load Lua script on all nodes
Script = "return redis.call('set', KEYS[1], ARGV[1]);",
{ok, ScriptHash} = eredis_cluster:load_script(Script),

%% Execute pre-loaded script based on hash (EVALSHA) on the node where
%% the key "abs" is, with a fallback to load it if needed.
eredis_cluster:eval(Script, ScriptHash, ["abc"], ["123"]).

%% Flush DB
eredis_cluster:flushdb().

%% Query on all cluster server
eredis_cluster:qa(["FLUSHDB"]).

%% Execute a query on the server containing the key "TEST"
eredis_cluster:qk(["FLUSHDB"], "TEST").

Multi-cluster

If you need to work with multiple Redis clusters in the same application, the functions connect/3, disconnect/1, q/2, qk/3, qa/2, qa2/2, qmn/2, transaction/3, get_pool_by_command/2, get_pool_by_key/2, get_all_pools/1 accept a named cluster parameter. Multi-cluster support was added in eredis_cluster 0.7.0.

eredis_cluster:start().
eredis_cluster:connect(mycluster, [{"127.0.0.1", 30001},
                                   {"127.0.0.1", 30002}], []).

{ok, Result} = eredis_cluster:q(mycluster, ["GET", "foo"]),

%% Query on all cluster nodes
eredis_cluster:qa(mycluster, ["FLUSHDB"]).

%% Execute a query on the server containing the key "TEST"
eredis_cluster:qk(mycluster, ["FLUSHDB"], "TEST").

%% Scan one of the nodes in a cluster
[MyclusterNode1 | _] = eredis_cluster:get_all_pools(mycluster),
eredis_cluster:qn(["SCAN", 0, "COUNT", 10], MyclusterNode1),

%% Transaction (See above for TransactionFun example)
eredis_cluster:transaction(TransactionFun, mycluster, "abc").

Compilation and tests

The directory contains a Makefile that uses rebar3.

Setup Redis clusters and start the tests using following commands:

make        # ... or rebar3 compile
make start  # Start local Redis clusters using Docker
make test   # Run tests towards the clusters
make stop   # Teardown the Redis clusters

Configuration

To configure the Redis cluster client, you can use an application variable (probably in your app.config):

{eredis_cluster,
    [
        {init_nodes,[
            {"127.0.0.1", 30001},
            {"127.0.0.1", 30002}
        ]},
        {pool_size, 5},
        {pool_max_overflow, 10},
        {username, "redis_user"},
        {password, "redis_pw"},
        {socket_options, [{send_timeout, 500},
                          {send_timeout_close, true},
                          {nodelay, true},
                          {keepalive, true}]},
        {tls, [{cacertfile, "ca.crt"}, ...]}
    ]
}

You don't need to specify all nodes of your configuration as eredis_cluster will retrieve them through the command CLUSTER SLOTS at runtime.

Configuration parameters

• init_nodes: List of Redis nodes to fetch cluster information from. Default: []

• pool_size: Number of connected clients to each Redis node. Default: 10

• pool_max_overflow: Max number of extra clients that can be started when the pool is exhausted. Default: 0

• username: Username for Redis ACL authentication. Alternatives are a 0-ary function that returns the username, a string or iodata or the atom undefined for no username. Default: undefined

  Note: A 0-ary function is preferred as it prevents secrets from appearing in logs and stacktraces.

• password: Password for Redis ACL authentication. Alternatives are a 0-ary function that returns the password, a string or iodata or the atom undefined for no password. Default: undefined

  Note: A 0-ary function is preferred as it prevents secrets from appearing in logs and stacktraces.

• socket_options: Extra socket options. Enables selecting host interface or performance tuning. Default: []. However, there are some options added by default by the underlying eredis client.

  The socket is set to {active, N} with N = 10 by default. This can be tuned by including {active, N} where N must be an integer or true. Active once and false are not supported.

  We suggest the following socket options, in order to efficiently detect a failing connection and trigger an update of the cluster topology in case of failover and other Redis Cluster events:

  • {send_timeout, Timeout} and {send_timeout_close, true}: Makes sure send errors are not ignored. Triggers a reconnect on send errors.

  • {nodelay, true}: Send TCP packets immediately. A Redis command is sent as a whole, not in pieces, so there is no point delaying the send.

  • {keepalive, true}: Enables TCP keepalive. Enabled by default on Linux.

  • Raw Linux-specific options can help to detect a hanging socket quickly. See Linux kernel file for details: include/uapi/linux/tcp.h. This example sets very strict values for near real-time failure detected. Consider choosing values that make sense for your application:

    %% TCP_USER_TIMEOUT (in milliseconds); should be > TCP_KEEPIDLE if
    %% keepalive is enabled:
    {raw, 6, 18, <<1100:32/native>>},
    %% {keepalive, true} must be set for these to take effect:
    {raw, 6, 4, <<1:32/native>>},  % TCP_KEEPIDLE (in seconds)
    {raw, 6, 5, <<1:32/native>>},  % TCP_KEEPINTVL (in seconds)
    {raw, 6, 6, <<1:32/native>>}   % TCP_KEEPCNT

• tls: Enable TLS/SSL and use specified TLSOptions. Default: TLS not enabled.

Configuring via API

An alternative is to set configurations programmatically via set_env() and eredis_cluster:connect/1.

application:set_env(eredis_cluster, pool_size, 5),
application:set_env(eredis_cluster, pool_max_overflow, 10),
application:set_env(eredis_cluster, password, "redis_pw"),
application:set_env(eredis_cluster, socket_options, [{send_timeout, 6000}]),
application:set_env(eredis_cluster, tls, [{cacertfile, "ca.crt"},
                                          {certfile, "client.crt"},
                                          {keyfile, "client.key"}]),

%% Set initial nodes and perform a controlled connect
eredis_cluster:connect([{"127.0.0.1", 30001},
                        {"127.0.0.1", 30002}]).

Configuring using connect/2

It is also possible to give options while doing a connect using eredis_cluster:connect/2. The given options will override options set via application configuration, i.e will be prepended to the property list.

Options = [{tls, [{cacertfile, "ca.crt"},
                  {certfile, "client.crt"},
                  {keyfile, "client.key"}]}],
eredis_cluster:connect([{"127.0.0.1", 30001},
                        {"127.0.0.1", 30002}], Options).

Troubleshooting

The following Redis-log indicates that Redis accepts TLS, but the client is not configured for TLS.

# Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number (conn: fd=12)

Debug logging for TLS connections can be enabled in eredis_cluster by the connect option: {log_level, debug}

See also

• Generated documentation: doc/eredis_cluster.md
• Dependencies:
  • Poolboy
  • Eredis