OTP 26 Review

[martinsumner]

As part of ongoing modernisation, work is now in progress on extending OTP support for Riak.

Branch Names

• nhse-develop-3.0. This branch supports 3.0 releases, with support for OTP 22 (having previously supported OTP 20 and OTP 22). There are no plans for further updates to this branch beyond tag riak-3.0.18-nhse.
• nhse-develop-3.2. This branch supports 3.2 releases, which presently support OTP 22 and OTP 24. Bug fixes will continue to be applied, but at some stage support for OTP 22 may be dropped.
• nhse-develop-3.4. This branch supports 3.4 releases, which will support OTP 24 and OTP 26. New features will be added to this branch until a nhse-develop-3.6 is available.

Performance Testing

Initial performance testing comparing 3.0, 3.2 and 3.4 in a CPU intensive test - uses HTTP API, leveled backend, im4gn instances, mid-size objects, a relatively high proportion of 2i queries with and without regular expressions:

The improvements between 3.0 and 3.2 are largely explained by leveled performance improvements due to refactoring following an eprof profiling exercises. There are no direct code changes resulting in the 3.2 -> 3.4 improvements, these are all related to code (most likely leveled code) running faster in OTP 26 when compared with OTP 24.

Functional Testing

All dependencies compile without warning or error in the develop-3.4 branch. There have been a few issues in riak_test tests:

• verify_busy_dist_port; a hangover from 3.2 release, this test requires a rewrite to remove the dependency on the riak_test_lager_backend.
• pipe_verify_restart_input_forwarding; fails due to the assertion to verify that the test was impacted by the prompted crash. Can be made to work by increasing the size of inputs by an order of magnitude.
• basic_command_line; requires a change to detect the new command prompt in OTP26.
• verify_dt_converge and verify_no_datatype_siblings; fails due to issues caused by the change in OTP26 of the default kernel setting of prevent_overlapping_partitions. The test will pass if prevent_overlapping_partitions is disabled.

Further testing

Although compilation has been checked across the dependencies, and the larger repos have been subject to xref/dialyzer/eunit tests under OTP26, there is still a need to run a full sweep of all dependencies confirming xref, dialyzer, eunit and eqc readiness.

Feature Proposals

Two features are currently proposed for the 3.4 release

Stricter If-Not-Modified-Since and If-None-Match support

In the Riak 3.0.13 release the behaviour when using If-None-Match or (X-Riak-)If-Not-Modified-Since was standardised between the PB and HTTP API. As part of the 3.4 release an option will be added to support this more strictly. With the enhanced strictness the API-level check would continue to be performed, but it will be supplemented by an additional check of the condition at the vnode PUT coordinator. As part of this change, when the option is enabled, and if and only if a conditional PUT is being managed, the choice of coordinator algorithm will change to one that always prefers the first primary in the preflist.

This change will still not offer strict guarantees. If there are concurrent updates, there may still be siblings formed if:

• different nodes hold a different view of the cluster state due to partitioning;
• updates are made into different (replicating) clusters;
• during the window when information about a node failure is propagated around the cluster;
• if a client fails to supply the conditional restrictions correctly.

The aim is to generally reduce the number of siblings created, in the face of concurrent updates, where this is a priority over the loss of potentially unwritten data.

Stricter If-Not-Modified-Since and If-None-Match support (alternative implementation)

An alternative implementation to strengthen conditional PUTs is described here - OpenRiak/riak_kv#33. This is currently the preferred method.

Support for a New 2i Query API supporting Filter Expressions

In Riak it is possible to have secondary index terms where additional attributes can be overloaded on to the sort key for further filtering via regular expressions.

e.g. an Index providing support for family name

fnidx_bin => ORWELL

can instead be overload with additional demographic information e.g.

fnidx_bin => ORWELL|19030625|19500121|#BLAIR|#GEORGE#ERIC#ARTHUR

So if looking for a family name OR.*L born between 1901 and 1904 with the given name ERIC

buckets\people\index\fnidx_bin\OR\OR~?term_regex="OR[^\\|]*L\\|190[1-3][0-9]{4}\\|[^\\|]*\\|[^\\|]*\\|[^\\|]*#ERIC"

There are a number of issues with this approach:

• Writing regular expressions are hard to get right (e.g. in the case above it will match records with the given name ERICA), and hard to automate from variable user input.
• Some comparisons are very hard to achieve via a regular expression (e.g. DOB between 19021025 and 19050116).
• The PCRE library used in erlang is subject to quadratic explosion depending on the complexity of the expression, and in any automated production of regular expressions in application code it is hard to guard against this.
• Handling unicode is non-trivial.

Other databases, such as AWS DynamoDB support a simple/natural filter language for projected attributes appended to sort keys on indexes. Such a language is far easier for developers, but presently there is no way of explicitly adding structured attributes onto index entries in Riak.

The proposal is to support in a new query API eval expressions and filter expressions. eval expressions will be a pipeline of functions that can be used to convert index entries into Key/Value mappings (to be treated as projected attributes). The filter expressions will allow results to be filtered based on those Key/Value mappings using a natural language filter expression.

i.e. the new Query API would allow for a JSON Query request like this:

POST /types/TypeName/buckets/BucketName/query
{
   "index_name" : "fnidx_bin",
  "start_term": "OR",
  "end_term": "OR~",
  "eval_expression" : "delim($term, \"|\", ($fn, $dob, $dod, $alt_fns, $gns)) | split($gns, \"#\", $gns)",
  "filter_expression" : "($dob BETWEEN \"1901010\" AND \"19031231\") AND (\"ERIC\" IN $gns) AND ends_with($fn, \"L\")",
  "return_type" : "keys"
}

Progress on the expression and eval language can be tracked in the associated leveled PR. The feature need not be leveled specific, i.e. eleveldb support should be implementable via the fold function - however, support for leveled will be the priority.

Other extensions are being considered, such as allowing for other values of return_type - keys, terms, count, count_by_term, count_by_key, {count_by_specific_term, $eval_key}.

It is a priority to make sure that the new expressions can be used on existing overloaded terms in different styles, so the flexibility of the eval pipeline is key.

Initial testing indicates that using these new evaluation and filter expressions is actually faster than using PCRE regular expressions in non-trivial cases, even when the PCRE expressions are compiled. Testing has also been undertaken using the RE2 library as an alternative to PCRE. Compiled RE2 expressions are quicker than both PCRE and eval/filter expressions, but compilations in currently available NIFs are not exportable between nodes. Consideration may be given to using RE2 again when it is made part of OTP28.

Potential Feature Reductions

The following features MAY be removed/reduced as part of 3.4 release:

• Consistent PUT (i.e. riak_ensemble) - PROBABLE;
• Map/Reduce (i.e riak_pipe) - POSSIBLE.

A develop-3.5 branch is being considered that will more aggressively deprecate/remove features to just the set required by NHSe. This MAY further remove the following:

• riak_repl;
• eleveldb backend, and all use of eleveldb by riak_core;
• Write-once in Riak KV;
• kv_index_hashtree support;
• Under-utilised features in riak_core.

OTP26-specific issues

OTP26 changes the default value for prevent_overlapping_partitions to true. Given Riak is intended to be tolerant of partitions, and doesn't genrrally make direct use of the global module (other than in riak_kv_entropy_info) - what is the correct value to be used in Riak?

Other priority issues

Some other priority issues are also intended to be addressed in the 3.4 life-cycle:

• Tidying Tictac Ecosystem for non-users basho/riak_kv#1883;
• Automated pruning back of vector clocks at the maximum limit (rather than pruning to the maximum)
• Equal not dominates and Merge basho/riak_kv#1846
• Make HEAD more efficient basho/riak_kv#1677
• Some http API improvements with regex compilation and other refcatoring and potentially more.