Merge pull request #2 from antirez/unstable

update from master

Author: ushachar

.github/workflows/ci.yml

@@ -0,0 +1,29 @@
+name: CI
+
+on: [push, pull_request]
+
+jobs:
+  test-ubuntu-latest:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v1
+    - name: make
+      run: make
+    - name: test
+      run: |
+        sudo apt-get install tcl8.5
+        ./runtest --clients 2 --verbose
+
+  build-ubuntu-old:
+    runs-on: ubuntu-16.04
+    steps:
+    - uses: actions/checkout@v1
+    - name: make
+      run: make
+
+  build-macos-latest:
+    runs-on: macos-latest
+    steps:
+    - uses: actions/checkout@v1
+    - name: make
+      run: make

.gitignore

@@ -1,5 +1,8 @@
 .*.swp
 *.o
+*.xo
+*.so
+*.d
 *.log
 dump.rdb
 redis-benchmark
@@ -28,3 +31,5 @@ deps/lua/src/liblua.a
 .prerequisites
 *.dSYM
 Makefile.dep
+.vscode/*
+.idea/*

README.md

@@ -35,6 +35,11 @@ It is as simple as:
 
     % make
 
+To build with TLS support, you'll need OpenSSL development libraries (e.g.
+libssl-dev on Debian/Ubuntu) and run:
+
+    % make BUILD_TLS=yes
+
 You can run a 32 bit Redis binary using:
 
     % make 32bit
@@ -43,6 +48,13 @@ After building Redis, it is a good idea to test it using:
 
     % make test
 
+If TLS is built, running the tests with TLS enabled (you will need `tcl-tls`
+installed):
+
+    % ./utils/gen-test-certs.sh
+    % ./runtest --tls
+
+
 Fixing build problems with dependencies or cached build options
 ---------
 
@@ -125,6 +137,12 @@ as options using the command line. Examples:
 All the options in redis.conf are also supported as options using the command
 line, with exactly the same name.
 
+Running Redis with TLS:
+------------------
+
+Please consult the [TLS.md](TLS.md) file for more information on
+how to use Redis with TLS.
+
 Playing with Redis
 ------------------
 
@@ -406,7 +424,7 @@ replicas, or to continue the replication after a disconnection.
 Other C files
 ---
 
-* `t_hash.c`, `t_list.c`, `t_set.c`, `t_string.c` and `t_zset.c` contains the implementation of the Redis data types. They implement both an API to access a given data type, and the client commands implementations for these data types.
+* `t_hash.c`, `t_list.c`, `t_set.c`, `t_string.c`, `t_zset.c` and `t_stream.c` contains the implementation of the Redis data types. They implement both an API to access a given data type, and the client commands implementations for these data types.
 * `ae.c` implements the Redis event loop, it's a self contained library which is simple to read and understand.
 * `sds.c` is the Redis string library, check http://github.com/antirez/sds for more information.
 * `anet.c` is a library to use POSIX networking in a simpler way compared to the raw interface exposed by the kernel.

TLS.md

@@ -0,0 +1,89 @@
+TLS Support
+===========
+
+Getting Started
+---------------
+
+### Building
+
+To build with TLS support you'll need OpenSSL development libraries (e.g.
+libssl-dev on Debian/Ubuntu).
+
+Run `make BUILD_TLS=yes`.
+
+### Tests
+
+To run Redis test suite with TLS, you'll need TLS support for TCL (i.e.
+`tcl-tls` package on Debian/Ubuntu).
+
+1. Run `./utils/gen-test-certs.sh` to generate a root CA and a server
+   certificate.
+
+2. Run `./runtest --tls` or `./runtest-cluster --tls` to run Redis and Redis
+   Cluster tests in TLS mode.
+
+### Running manually
+
+To manually run a Redis server with TLS mode (assuming `gen-test-certs.sh` was
+invoked so sample certificates/keys are available):
+
+    ./src/redis-server --tls-port 6379 --port 0 \
+        --tls-cert-file ./tests/tls/redis.crt \
+        --tls-key-file ./tests/tls/redis.key \
+        --tls-ca-cert-file ./tests/tls/ca.crt
+
+To connect to this Redis server with `redis-cli`:
+
+    ./src/redis-cli --tls \
+        --cert ./tests/tls/redis.crt \
+        --key ./tests/tls/redis.key \
+        --cacert ./tests/tls/ca.crt
+
+This will disable TCP and enable TLS on port 6379. It's also possible to have
+both TCP and TLS available, but you'll need to assign different ports.
+
+To make a Replica connect to the master using TLS, use `--tls-replication yes`,
+and to make Redis Cluster use TLS across nodes use `--tls-cluster yes`.
+
+Connections
+-----------
+
+All socket operations now go through a connection abstraction layer that hides
+I/O and read/write event handling from the caller.
+
+**Multi-threading I/O is not currently supported for TLS**, as a TLS connection
+needs to do its own manipulation of AE events which is not thread safe. The
+solution is probably to manage independent AE loops for I/O threads and longer
+term association of connections with threads. This may potentially improve
+overall performance as well.
+
+Sync IO for TLS is currently implemented in a hackish way, i.e. making the
+socket blocking and configuring socket-level timeout.  This means the timeout
+value may not be so accurate, and there would be a lot of syscall overhead.
+However I believe that getting rid of syncio completely in favor of pure async
+work is probably a better move than trying to fix that. For replication it would
+probably not be so hard. For cluster keys migration it might be more difficult,
+but there are probably other good reasons to improve that part anyway.
+
+To-Do List
+----------
+
+- [ ] Add session caching support. Check if/how it's handled by clients to
+  assess how useful/important it is.
+- [ ] redis-benchmark support. The current implementation is a mix of using
+  hiredis for parsing and basic networking (establishing connections), but
+  directly manipulating sockets for most actions. This will need to be cleaned
+  up for proper TLS support. The best approach is probably to migrate to hiredis
+  async mode.
+- [ ] redis-cli `--slave` and `--rdb` support.
+
+Multi-port
+----------
+
+Consider the implications of allowing TLS to be configured on a separate port,
+making Redis listening on multiple ports:
+
+1. Startup banner port notification
+2. Proctitle
+3. How slaves announce themselves
+4. Cluster bus port calculation

deps/Makefile

@@ -41,9 +41,13 @@ distclean:
 
 .PHONY: distclean
 
+ifeq ($(BUILD_TLS),yes)
+    HIREDIS_MAKE_FLAGS = USE_SSL=1
+endif
+
 hiredis: .make-prerequisites
 	@printf '%b %b\n' $(MAKECOLOR)MAKE$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR)
-	cd hiredis && $(MAKE) static
+	cd hiredis && $(MAKE) static $(HIREDIS_MAKE_FLAGS)
 
 .PHONY: hiredis
 

deps/hiredis/.gitignore

@@ -5,3 +5,4 @@
 /*.dylib
 /*.a
 /*.pc
+*.dSYM

deps/hiredis/.travis.yml

@@ -26,20 +26,72 @@ addons:
     - libc6-dev-i386
     - libc6-dbg:i386
     - gcc-multilib
+    - g++-multilib
     - valgrind
 
 env:
-    - CFLAGS="-Werror"
-    - PRE="valgrind --track-origins=yes --leak-check=full"
-    - TARGET="32bit" TARGET_VARS="32bit-vars" CFLAGS="-Werror"
-    - TARGET="32bit" TARGET_VARS="32bit-vars" PRE="valgrind --track-origins=yes --leak-check=full"
+  - BITS="32"
+  - BITS="64"
 
-matrix:
-  exclude:
-    - os: osx
-      env: PRE="valgrind --track-origins=yes --leak-check=full"
+script:
+  - EXTRA_CMAKE_OPTS="-DENABLE_EXAMPLES:BOOL=ON -DHIREDIS_SSL:BOOL=ON";
+    if [ "$TRAVIS_OS_NAME" == "osx" ]; then
+      if [ "$BITS" == "32" ]; then
+        CFLAGS="-m32 -Werror";
+        CXXFLAGS="-m32 -Werror";
+        LDFLAGS="-m32";
+        EXTRA_CMAKE_OPTS=;
+      else
+        CFLAGS="-Werror";
+        CXXFLAGS="-Werror";
+      fi;
+    else
+      TEST_PREFIX="valgrind --track-origins=yes --leak-check=full";
+      if [ "$BITS" == "32" ]; then
+        CFLAGS="-m32 -Werror";
+        CXXFLAGS="-m32 -Werror";
+        LDFLAGS="-m32";
+        EXTRA_CMAKE_OPTS=;
+      else
+        CFLAGS="-Werror";
+        CXXFLAGS="-Werror";
+      fi;
+    fi;
+    export CFLAGS CXXFLAGS LDFLAGS TEST_PREFIX EXTRA_CMAKE_OPTS
+  - mkdir build/ && cd build/
+  - cmake .. ${EXTRA_CMAKE_OPTS}
+  - make VERBOSE=1
+  - ctest -V
 
-    - os: osx
-      env: TARGET="32bit" TARGET_VARS="32bit-vars" PRE="valgrind --track-origins=yes --leak-check=full"
+matrix:
+  include:
+    # Windows MinGW cross compile on Linux
+    - os: linux
+      dist: xenial
+      compiler: mingw
+      addons:
+        apt:
+          packages:
+            - ninja-build
+            - gcc-mingw-w64-x86-64
+            - g++-mingw-w64-x86-64
+      script:
+        - mkdir build && cd build
+        - CC=x86_64-w64-mingw32-gcc CXX=x86_64-w64-mingw32-g++ cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_BUILD_WITH_INSTALL_RPATH=on
+        - ninja -v
 
-script: make $TARGET CFLAGS="$CFLAGS" && make check PRE="$PRE" && make $TARGET_VARS hiredis-example
+    # Windows MSVC 2017
+    - os: windows
+      compiler: msvc
+      env:
+        - MATRIX_EVAL="CC=cl.exe && CXX=cl.exe"
+      before_install:
+        - eval "${MATRIX_EVAL}"
+      install:
+        - choco install ninja
+      script:
+        - mkdir build && cd build
+        - cmd.exe /C '"C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Auxiliary\Build\vcvarsall.bat" amd64 &&
+          cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release &&
+          ninja -v'
+        - ctest -V

deps/hiredis/CHANGELOG.md

@@ -12,6 +12,16 @@
   compare to other values, casting might be necessary or can be removed, if
   casting was applied before.
 
+### 0.x.x (unreleased)
+**BREAKING CHANGES**:
+
+* Change `redisReply.len` to `size_t`, as it denotes the the size of a string
+
+User code should compare this to `size_t` values as well.
+If it was used to compare to other values, casting might be necessary or can be removed, if casting was applied before.
+
+* `redisReplyObjectFunctions.createArray` now takes `size_t` for its length parameter.
+
 ### 0.14.0 (2018-09-25)
 
 * Make string2ll static to fix conflict with Redis (Tom Lee [c3188b])
@@ -50,8 +60,9 @@
 * Import latest upstream sds. This breaks applications that are linked against the old hiredis v0.13
 * Fix warnings, when compiled with -Wshadow
 * Make hiredis compile in Cygwin on Windows, now CI-tested
-
-**BREAKING CHANGES**:
+* Bulk and multi-bulk lengths less than -1 or greater than `LLONG_MAX` are now
+  protocol errors. This is consistent with the RESP specification. On 32-bit
+  platforms, the upper bound is lowered to `SIZE_MAX`.
 
 * Remove backwards compatibility macro's