APIs and Clients #11
Replies: 3 comments 4 replies
-
|
One way forward may be to look at the mapping between Protocol Buffers and JSON - https://protobuf.dev/programming-guides/json/. OTP now has native JSON support, which has been subject to significant optimisation effort. The main third-party GPB library in Erlang - https://github.com/tomas-abrahamsson/gpb - supports JSON <-> PB conversion. There is widespread support for libraries converting between the formats in different languages. Might there be a situation where from the perspective of Riak KV there is no need to support two APIs, but simply two transports for the same API i.e. transport JSON via HTTP, and GPB via PB. The HTTP API would only use headers/URLs to describe the request, should that description be required by intermediary HTTP infrastructure - the HTTP interface in riak_api will simply validate that the request in the message body is as described in the headers (e.g. buckets, message type match), and then the two inputs downstream would be treated the same. The upsides would be:
The downsides would be:
Perhaps in Riak 4.0 there should be three ways of interfacing to Riak:
|
Beta Was this translation helpful? Give feedback.
-
|
There are references earlier in the discussion to the potential need for HTTP 2 support. Although this seems an obvious progression, it is not necessarily a good idea for Riak. The HTTP 2 protocol is promoted as an improvement in performance over HTTP 1.1, and performance is important to Riak. However, almost all performance improvements are aimed at the problem of serving a web page consisting of multiple objects - e.g. multiplexing and prioritisation. Riak, though, is a strict REST model where connections are used for a single request/response at a time. There may be a role for multiplexing if Riak is sat behind a HTTP 2 proxy, but there is unlikely to be a huge difference with Riak being a HTTP 1.1 proxy with an over-provisioned connection pool. There is support for improved header compression with the HPACK compression protocol. HTTP 1.1 does not specifically support header compression. Some Riak requests have large numbers of HTTP headers (due to index entries being compressed), so there could be value here. However, compression is a trade-off of CPU for bandwidth, so there can be no certainty that such compression would be noticeable advantageous. There is also the potential performance overhead of introducing HTTP 2.0 support. Implementing HTTP 2 required Cowboy to split the processing of requests across two processes, with a potentially significant performance penalty - https://stressgrid.com/blog/cowboy_performance_part_2/, erlang/otp#9423. In the debate over performance on erlang forums, the claims of improved performance relative to cowboy seem to stem from the additional inter-process communication required by Cowboy to to introduce HTTP 2. Although there may be good reasons to adopt Cowboy, and HTTP 2.0 - we should be cautious about investing time in this path under the assumption that performance will improve. There are no guarantees of improved performance, and indeed a risk of performance degradation. |
Beta Was this translation helpful? Give feedback.
-
|
Some further analysis on improving performance on the HTTP API through potential changes the underlying HTTP implementation - OpenRiak/riak_kv#133. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
The subject of client development and maintenance has been raised on OpenRiak team calls, so I thought it would be useful to have a discussion available here.
Two APIs
Riak has two APIs at present - one HTTP-based, and one PB-based. There is then the option for each API to overlay TLS encryption and authentication. Some points of note:
riak_testsystem generally tests each service using only one of the APIs, and not both, and if both are tested the tests are not necessarily identical;Anecdotally, it seems that most large users of Riak use the PB API, as performance matters, but use of the HTTP API exists. Preference for the HTTP API is generally because:
There would be advantages of having a single API in the future - but if so which one should it be?
The existence of two APIs complicates future choices about client development:
If two APIs are maintained, should there be a preferred API for clients to be developed against?
Clients
Previously, basho supported 8 different clients (Java, Ruby, C#, Python, PHP, Node.js, Erlang, Go), and there were around 100 community-led client projects. On basho's demise, the decision was made to maintain only the two Erlang clients. These clients were maintained for testing purposes only, so without effort to maintain usability.
The decision to drop client support was based on the following:
There remains though an appetite for clients, and a negative perception on the project created by all the open-source available clients being so far out-of-date.
As a workaround for having clients, documenting the Riak HTTP API in OpenAPI has been investigated. The initial results of that is that the existing HTTP API is not OpenAPI compatible. The OpenAPI specification does not allow for templated HTTP header names, and expects headers to be used to carry information about the request not the data - and this causes immediate problems with the use of headers for secondary indexes and object-specific metadata.
It is assumed at this point, that the resolution to this is not to have a second HTTP (third overall) API that is OpenAPI compatible.
If there are to be clients in the future though, what should be the priority for the development of those clients:
node_confirms,sync_on_writein preference todw,pwvalues - focus on exposing the usable/explainable options).Beta Was this translation helpful? Give feedback.
All reactions