APIs and Clients

[martinsumner]

The subject of client development and maintenance has been raised on OpenRiak team calls, so I thought it would be useful to have a discussion available here.

Two APIs

Riak has two APIs at present - one HTTP-based, and one PB-based. There is then the option for each API to overlay TLS encryption and authentication. Some points of note:

• It had been basho's intention to deprecate the HTTP API, and move to a PB-only model.
• The PB API is notable "faster" than the HTTP API, but the delta is use-case dependent:
  • Processing HTTP requests and responses consumes extra CPU cycles, and so in CPU bound clusters there is also an impact on maximum throughput;
  • There is a PR to partially close the gap.
• There are inconsistencies in implementation between the PB and HTTP API, subtle differences in validation and parameter defaults:
  • Most riak_test system generally tests each service using only one of the APIs, and not both, and if both are tested the tests are not necessarily identical;
  • It is possible to break the HTTP API by using the PB API e.g. by adding a binary key via the PB API that will crash a request to receive it the HTTP API (or return an index query result that contains the query).
  • The integration of TLS is distinct between the two APIs, with HTTP requiring a separate listener that is HTTPS only, whereas PB implements TLS by negotiation on the in-clear channel. There are differences in the TLS features they support (no use of client authentication via certificates on HTTP).
• The HTTP API has significant external dependencies in webmachine and mochiweb.
• Unicode support in Riak is undocumented, and complicated by the existence of two APIs.

Anecdotally, it seems that most large users of Riak use the PB API, as performance matters, but use of the HTTP API exists. Preference for the HTTP API is generally because:

• There is a need to integrate with an async HTTP request handler.
• There is a broader operational desire for everything to be HTTP, and an investment in infrastructure (e.g. HTTP middleware such as WAFs, load-balancers) to support this.
• It was easier to get started with HTTP.

There would be advantages of having a single API in the future - but if so which one should it be?

The existence of two APIs complicates future choices about client development:

• presently some of the more developed clients attempt to abstract across the two protocol choices (e.g. the python client);
• some languages have separate clients for each protocol (e.g. the Erlang clients).

If two APIs are maintained, should there be a preferred API for clients to be developed against?

Clients

Previously, basho supported 8 different clients (Java, Ruby, C#, Python, PHP, Node.js, Erlang, Go), and there were around 100 community-led client projects. On basho's demise, the decision was made to maintain only the two Erlang clients. These clients were maintained for testing purposes only, so without effort to maintain usability.

The decision to drop client support was based on the following:

• A shortage of developer time and a need to prioritise effort.
• The belief that most users would ultimately want to write their own client to meet the needs of their own environment and coding styles (hence why there were over a hundred community clients).
• The API was shrinking, and with it the value in masking the complexity of the API via the client:
  • further there exists the issue of whether masking things like transferring the Encoded clocks from GETs to PUTs, actually hindered developers at start-up, as it prevented them from experiencing things they needed to understand.
• The clients that did exist, had a habit of containing hidden complexity and introducing new failure scenarios. The assumptions of the client developer were not always aligned with the assumptions of the riak developer - and clients would do surprising things that would make Riak appear unreliable or slow.
  • Calling the stats endpoint to healthcheck a node at startup;
  • Implementing connection pooling that causes long-pauses on node failures;
  • Implementing hidden request queues within the client.

There remains though an appetite for clients, and a negative perception on the project created by all the open-source available clients being so far out-of-date.

As a workaround for having clients, documenting the Riak HTTP API in OpenAPI has been investigated. The initial results of that is that the existing HTTP API is not OpenAPI compatible. The OpenAPI specification does not allow for templated HTTP header names, and expects headers to be used to carry information about the request not the data - and this causes immediate problems with the use of headers for secondary indexes and object-specific metadata.

It is assumed at this point, that the resolution to this is not to have a second HTTP (third overall) API that is OpenAPI compatible.

If there are to be clients in the future though, what should be the priority for the development of those clients:

• Should clients be focused on HTTP API (easier to develop) or PB API (better performance, masking more complexity) or must they be dual API?
• What languages should be prioritised for clients, especially considering whether overall popularity of languages might be different to the popularity of languages within organisations facing problems of scale that might need a Riak?
  • n.b. One of the biggest known historic Riak implementations used the perl client.
• Should clients be non-functionally rich (which was the direction of travel for basho) i.e. implementing load-balancing, connection-management, server healthchecks.
  • Could it be better to support standard HAProxy/NGINX configurations for logging, filtering and load-balancing requests rather than loading that logic into each client.
• Should clients be opinionated about the features they expose, restricting just to those features we would prefer to be used (i.e. remove strong consistency, search, map/reduce, write-once).
• Should clients be opinionated about the options they expose (i.e. expose node_confirms, sync_on_write in preference to dw, pw values - focus on exposing the usable/explainable options).
• Should clients be consistent between languages in their implementation.
• Should the focus be on clients that are easy to develop/maintain (e.g. Gleam, Elixir clients that can lean on the maintained Erlang client).
• Should clients have a suite of tests that can and will be tested as part of Riak release, and what should the scope of those tests be i.e. should there be a standardised test spec for clients.
• What are the non-functional requirements for client testing, and should clients be supported that are not know to be soak-tested in production environments?
• How should the documentation of clients be integrated into the documentation of Riak? Is copying/pasting client API documentation into Riak documentation maintainable?
• Should clients include admin/operational commands, and those that fall in the grey area (e.g. safe bucket listing via aae_fold)?