APIs and Clients

[martinsumner]

The subject of client development and maintenance has been raised on OpenRiak team calls, so I thought it would be useful to have a discussion available here.

Two APIs

Riak has two APIs at present - one HTTP-based, and one PB-based. There is then the option for each API to overlay TLS encryption and authentication. Some points of note:

• It had been basho's intention to deprecate the HTTP API, and move to a PB-only model.
• The PB API is notable "faster" than the HTTP API, but the delta is use-case dependent:
  • Processing HTTP requests and responses consumes extra CPU cycles, and so in CPU bound clusters there is also an impact on maximum throughput;
  • There is a PR to partially close the gap.
• There are inconsistencies in implementation between the PB and HTTP API, subtle differences in validation and parameter defaults:
  • Most riak_test system generally tests each service using only one of the APIs, and not both, and if both are tested the tests are not necessarily identical;
  • It is possible to break the HTTP API by using the PB API e.g. by adding a binary key via the PB API that will crash a request to receive it the HTTP API (or return an index query result that contains the query).
  • The integration of TLS is distinct between the two APIs, with HTTP requiring a separate listener that is HTTPS only, whereas PB implements TLS by negotiation on the in-clear channel. There are differences in the TLS features they support (no use of client authentication via certificates on HTTP).
• The HTTP API has significant external dependencies in webmachine and mochiweb.
• Unicode support in Riak is undocumented, and complicated by the existence of two APIs.

Anecdotally, it seems that most large users of Riak use the PB API, as performance matters, but use of the HTTP API exists. Preference for the HTTP API is generally because:

• There is a need to integrate with an async HTTP request handler.
• There is a broader operational desire for everything to be HTTP, and an investment in infrastructure (e.g. HTTP middleware such as WAFs, load-balancers) to support this.
• It was easier to get started with HTTP.

There would be advantages of having a single API in the future - but if so which one should it be?

The existence of two APIs complicates future choices about client development:

• presently some of the more developed clients attempt to abstract across the two protocol choices (e.g. the python client);
• some languages have separate clients for each protocol (e.g. the Erlang clients).

If two APIs are maintained, should there be a preferred API for clients to be developed against?

Clients

Previously, basho supported 8 different clients (Java, Ruby, C#, Python, PHP, Node.js, Erlang, Go), and there were around 100 community-led client projects. On basho's demise, the decision was made to maintain only the two Erlang clients. These clients were maintained for testing purposes only, so without effort to maintain usability.

The decision to drop client support was based on the following:

• A shortage of developer time and a need to prioritise effort.
• The belief that most users would ultimately want to write their own client to meet the needs of their own environment and coding styles (hence why there were over a hundred community clients).
• The API was shrinking, and with it the value in masking the complexity of the API via the client:
  • further there exists the issue of whether masking things like transferring the Encoded clocks from GETs to PUTs, actually hindered developers at start-up, as it prevented them from experiencing things they needed to understand.
• The clients that did exist, had a habit of containing hidden complexity and introducing new failure scenarios. The assumptions of the client developer were not always aligned with the assumptions of the riak developer - and clients would do surprising things that would make Riak appear unreliable or slow.
  • Calling the stats endpoint to healthcheck a node at startup;
  • Implementing connection pooling that causes long-pauses on node failures;
  • Implementing hidden request queues within the client.

There remains though an appetite for clients, and a negative perception on the project created by all the open-source available clients being so far out-of-date.

As a workaround for having clients, documenting the Riak HTTP API in OpenAPI has been investigated. The initial results of that is that the existing HTTP API is not OpenAPI compatible. The OpenAPI specification does not allow for templated HTTP header names, and expects headers to be used to carry information about the request not the data - and this causes immediate problems with the use of headers for secondary indexes and object-specific metadata.

It is assumed at this point, that the resolution to this is not to have a second HTTP (third overall) API that is OpenAPI compatible.

If there are to be clients in the future though, what should be the priority for the development of those clients:

• Should clients be focused on HTTP API (easier to develop) or PB API (better performance, masking more complexity) or must they be dual API?
• What languages should be prioritised for clients, especially considering whether overall popularity of languages might be different to the popularity of languages within organisations facing problems of scale that might need a Riak?
  • n.b. One of the biggest known historic Riak implementations used the perl client.
• Should clients be non-functionally rich (which was the direction of travel for basho) i.e. implementing load-balancing, connection-management, server healthchecks.
  • Could it be better to support standard HAProxy/NGINX configurations for logging, filtering and load-balancing requests rather than loading that logic into each client.
• Should clients be opinionated about the features they expose, restricting just to those features we would prefer to be used (i.e. remove strong consistency, search, map/reduce, write-once).
• Should clients be opinionated about the options they expose (i.e. expose node_confirms, sync_on_write in preference to dw, pw values - focus on exposing the usable/explainable options).
• Should clients be consistent between languages in their implementation.
• Should the focus be on clients that are easy to develop/maintain (e.g. Gleam, Elixir clients that can lean on the maintained Erlang client).
• Should clients have a suite of tests that can and will be tested as part of Riak release, and what should the scope of those tests be i.e. should there be a standardised test spec for clients.
• What are the non-functional requirements for client testing, and should clients be supported that are not know to be soak-tested in production environments?
• How should the documentation of clients be integrated into the documentation of Riak? Is copying/pasting client API documentation into Riak documentation maintainable?
• Should clients include admin/operational commands, and those that fall in the grey area (e.g. safe bucket listing via aae_fold)?

[martinsumner]

One way forward may be to look at the mapping between Protocol Buffers and JSON - https://protobuf.dev/programming-guides/json/.

OTP now has native JSON support, which has been subject to significant optimisation effort. The main third-party GPB library in Erlang - https://github.com/tomas-abrahamsson/gpb - supports JSON <-> PB conversion. There is widespread support for libraries converting between the formats in different languages.

Might there be a situation where from the perspective of Riak KV there is no need to support two APIs, but simply two transports for the same API i.e. transport JSON via HTTP, and GPB via PB. The HTTP API would only use headers/URLs to describe the request, should that description be required by intermediary HTTP infrastructure - the HTTP interface in riak_api will simply validate that the request in the message body is as described in the headers (e.g. buckets, message type match), and then the two inputs downstream would be treated the same.

The upsides would be:

• Two transports, but without duplication of validation/parsing logic across two APIs.
• Backwards compatible with existing clients (PB still works as-is).
• Easy to add HTTP support into existing clients (PB -> JSON conversion).
• Removing/upgrading webmachine/mochijson dependencies is easier as functional requirements simplified.
• More efficient handling of secondary indexes, avoiding issues with HTTP header limitations (e.g. lower-casing, language specific limits on header counts and sizes).

The downsides would be:

• Binary handling via HTTP would be less efficient - binary values would now need to be base64 encoded.
• It would be unnatural and less readable for anyone using JSON than the existing API as their object format (e.g. a GET response would no longer be human readable as the value would need to be compressed/encoded).
• Anyone with a bespoke HTTP-only client would need to convert their code (e.g. some users just use language-native HTTP libraries for Riak requests).

Perhaps in Riak 4.0 there should be three ways of interfacing to Riak:

• PB API;
• JSON API via HTTP, which is mapped to the PB API;
• A limited subset of the legacy API (GET object, PUT object of JSON objects only) which just converted within riak_api to/from a PB request.

[martinsumner]

Following further discussion within the development community, a potential outline way forward was discussed.

• The PB API will be supported in Riak 4.0, but deprecated, and scheduled for retirement in Riak 4.2.
• The scope of the API will be reduced to Object, Query, AAEFold in Riak 4.0.
• In the HTTP Object API there will be two options for the passing of metadata:
  • Using HTTP Request Headers as present;
  • Embedded within the value where the separation of object metadata and object value will be controlled by a callback within the merge_strategy.

So it should be possible from this building block to support either human readable passing of metadata and value (e.g. an OpenAPI friendly version) or an efficient binary encoding definable by the user (e.g. Erlang External Term Format). Extending to support those formats would be possible by defining and adding a merge_strategy - it would not require any changes to the HTTP API.

[martinsumner]

One other topic is to how to implement the HTTP API going forward. Currently mochiweb/webmachine is used, and there are concerns that:

• This carries an overhead of unused capability;
• There are efficiency issues dues to atom()/binary()/string() conversion and repeated lowercase comparisons needs in HTTP headers (and some requests may have a large number of HTTP headers);
• Does not support HTTP 2 and other potential protocols that may be required in the future.

There are three basic options:

• Optimise the existing mochiweb/webmachine implementation, particularly with regards to HTTP header handling.
• Switch to Cowboy (and gain from HTTP 2 support).
• Build a simple local alternative (inspired by elli), focused on maximising efficiency and minimising overhead.

[churcho]

• Optimise the existing mochiweb/webmachine implementation, particularly with regards to HTTP header handling.
• Switch to Cowboy (and gain from HTTP 2 support).
• Build a simple local alternative (inspired by elli), focused on maximising efficiency and minimising overhead.

Has there been any consensus on these?

Is this 4.0 planned or coming in the 3.x?
Who is taking lead on this that I can reach out to with more questions about it?

Super green on Erlang but proficient in Elixir. I want to see if I can explore some ideas I am tinkering with around clients.

[martinsumner]

No consensus on the way forward yet, but the expectation is to make progress clients and the APIs for Riak 4.0. I'm not sure who will be taking the lead, but discussions will continue on here, through the erlef openriak slack channel and our monthly working group calls. Please get involved Churchill!

[martinsumner]

There are references earlier in the discussion to the potential need for HTTP 2 support. Although this seems an obvious progression, it is not necessarily a good idea for Riak.

The HTTP 2 protocol is promoted as an improvement in performance over HTTP 1.1, and performance is important to Riak. However, almost all performance improvements are aimed at the problem of serving a web page consisting of multiple objects - e.g. multiplexing and prioritisation. Riak, though, is a strict REST model where connections are used for a single request/response at a time.

There may be a role for multiplexing if Riak is sat behind a HTTP 2 proxy, but there is unlikely to be a huge difference with Riak being a HTTP 1.1 proxy with an over-provisioned connection pool.

There is support for improved header compression with the HPACK compression protocol. HTTP 1.1 does not specifically support header compression. Some Riak requests have large numbers of HTTP headers (due to index entries being compressed), so there could be value here. However, compression is a trade-off of CPU for bandwidth, so there can be no certainty that such compression would be noticeable advantageous.

There is also the potential performance overhead of introducing HTTP 2.0 support. Implementing HTTP 2 required Cowboy to split the processing of requests across two processes, with a potentially significant performance penalty - https://stressgrid.com/blog/cowboy_performance_part_2/, erlang/otp#9423. In the debate over performance on erlang forums, the claims of improved performance relative to cowboy seem to stem from the additional inter-process communication required by Cowboy to to introduce HTTP 2.

Although there may be good reasons to adopt Cowboy, and HTTP 2.0 - we should be cautious about investing time in this path under the assumption that performance will improve. There are no guarantees of improved performance, and indeed a risk of performance degradation.

[martinsumner]

Some further analysis on improving performance on the HTTP API through potential changes the underlying HTTP implementation - OpenRiak/riak_kv#133.