Move to reactive approach

[kptfh]

As I can see that janusgraph is heavily relies on storage backend (the grate majority of request time may be spent communicating with storage).
Most of currently implemented storage backends(Cassandra, Elasticsearch) as well as most of the NoSQL DBs (Aerospike, Redis, Couchbase) provide async API and this process is accelerating.
My assumption is that janusgraph can benefit from moving to reactive architecture especially in terms of concurrent requests per node.

[farodin91]

Design decisions should be discussed on https://groups.google.com/group/janusgraph-dev

[porunov]

Recently in #3760 we implemented multi-queries support for CQL when we reuse Async DataStax Driver approach (will be available in JanusGraph 1.0.0).
We have seen immediate performance improvement as well as less resources usage when switching to that approach.
In #3170 we had a discussions with @cdegroc and @li-boxuan about current thread pools usages as well as moving to Async approach.
@cdegroc was interested in starting a discussion of moving to Async interface for slice queries.
I believe changing slice interface to async approach does make sense because it will open the door for new optimizations. For example, currently with multi-query we send multiple slice queries for different vertices, await finish all of them, cache the result, and move to the next step execution.
If our slice interface were Async (for example, returning CompletableFuture<EntryList> instead of EntryList), then we could potentially change multi-query to be async as well. In such case, I would imagine the multi-query interface would return Future results as well. I.e. instead of having Map<JanusGraphVertex, Iterable<JanusGraphVertexProperty>> properties() we could have Map<JanusGraphVertex, CompletableFuture<Iterable<JanusGraphVertexProperty>>> properties(). In such case, all the steps which execute multi-query don't need to await until the result for all queries is returned. Instead, we await the result for the current requested Traverser<Vertex> only and move to the next step immediately. Whenever we return back to the current step again, we would simply need to await for the next Traverser<Vertex>, but not for the result of other vertices requested in the multi-query.
I assume, if we change slice query as well as multi-query to Async interfaces - we may improve queries execution performance due to less time spending for IO await operations.

Notice, this will improve JanusGraph performance, but it doesn't fully allow to execute Gremlin queries Asynchronously.
We still need to have at least 1 thread executing 1 Gremlin query. If we want to execute 100 Gremlin queries in parallel - it will take at least 100 threads.
To execute Gremlin queries Asynchronously (i.e. to be able to execute 100 Gremlin queries using 1 thread) we need to change TinkerPop interface. That said, I think changing TinkerPop interface out of the scope of graph providers and should be discussed with the TinkerPop community. (See mailing list thread about adding Async capabilities to TinkerPop).

I propose to use this discussion thread to discuss JanusGraph internal Async capabilities improvements. I.e. changing slice interface to async interface, changing multi-query methods interface to async interface, and possibly changing other JanusGraph interfaces to async approach.

@cdegroc I wonder if you had any more thoughts about changing the slice interface to async interface. I would imagine it could require also changing our db-cache and / or transaction cache to store CompletableFuture<EntryList> instead of EntryList. Also, I assume it might require some slight changes in the approach how we handle exceptions. For example, currently storage providers internally decide which exceptions are considered to be PermanentBackendException and which exceptions should be considered as TemporaryBackendExceptions. If our getSlice interface is changed to CompletableFuture<EntryList> then I assume we might need to think about how we handle those exceptions later. I guess we could either chain CompletableFuture<EntryList> to return PermanentBackendException or TemporaryBackendExceptions wrapped in some kind of RuntimeException OR we could think about adding a method into KeyColumnValueStore which converts Throwable to PermanentBackendException or TemporaryBackendExceptions (i.e. something like BackendException convertToBackendException(Throwable throwable);).

[cdegroc]

I've quickly skimmed through #3760 and saw you found a way to introduce async calls without breaking the KeyColumnValueStore which is neat 👏 I'll try to bench it as soon as I can.

@cdegroc I wonder if you had any more thoughts about changing the slice interface to async interface. I would imagine it could require also changing our db-cache and / or transaction cache to store CompletableFuture instead of EntryList.

Indeed that's what we did, but we only patched backend implementations relevant to us.

Also, I assume it might require some slight changes in the approach how we handle exceptions. For example, currently storage providers internally decide which exceptions are considered to be PermanentBackendException and which exceptions should be considered as TemporaryBackendExceptions.

I think TemporaryBackendExceptions are handled by BackendOperation#executeDirect in our case? I don't think we have made changes to exception handling.

[cdegroc]

Here is the code (rebased on top of an old v0.6 branch): cdegroc@a594d5f
It was co-authored by @li-boxuan.