[Design] Proxy full-node api

[r4mmer]

Summary

Proxy calls to the full-node and use API Gateway cache to mitigate the number of calls that the full-node has to handle.

Motivation

Some endpoints which are not expensive on the ful-node may cause a denial of service if they are called too many times on a short period.
So if we only call once and serve from the cache for all other hundreds (or more) calls on the period the full-node will be able to handle bursts of traffic more efficiently.

Guide-level explanation

Since we will not add any special handling of the response the only thing we need to discuss for each endpoint is cache.

Endpoints

/v1a/thin_wallet/token_history

• cache 10 seconds
• cache key: token_uid + count + hash + page + timestamp

The pagination arguments will be used as cache key so that we return a consistent pagination for this api.

Obs: This is called through the wallet-lib (TokenDetail screen)

/v1a/thin_wallet/token

Will be split in 2 endpoints, one for fetching a single token and the other to fetch a paginated list of tokens.

• Single token:
  • cache 10 seconds
  • cache key: id
• List tokens:
  • cache 1 hour
  • cache key: none

Obs: This is called through the wallet-lib (TokenDetail and TokenList screens)

* This may need to be revised pending the token api design.

/v1a/transaction

Will be split in 2 endpoints, one for fetching a single tx and the other to fetch a paginated list of txs.

• Single TX:
  • cache 10 seconds
  • cache key: id
• List TX:
  • cache 10 seconds
  • cache key: type + count + hash + page + timestamp

/v1a/dashboard_tx

• cache 10 seconds
• cache key: block + tx

New blocks and transactions are always arriving but this call is not time-sensitive.
A delay of a few seconds that the cache would introduce would not change the experience of the end-user.
Since this is called on the first page of the explorer it's excepted to be called a lot during bursts of traffic.

/v1a/transaction_acc_weight

• cache 5 seconds
• cache key: id

This changes constantly but a delay of a few seconds will not affect the user experience.

/v1a/version

• cache time 1 hour
• cache key: none

Since the response only changes when deploying a new version the cache will not affect this endpoint.

Referenve-level explanation

All APIs will be transparent to the explorer-service, no treatment of the data will be made.
The cache will be handled by the API Gateway and configured on the serverless.yml for the lambda of the endpoint.
The only logic needed will be the parameter handling (extract parameters from querystring and pass to the internal method).

[andreabadesso]

Looks good to me, approved

[luislhl]

How the endpoints of the full-node were selected for this design? I think this should be described.

Will be split in 2 endpoints

What do you mean by this? We will have 2 different endpoints in API Gateway both proxying to the same endpoint in the full-node?

cache key: none (or block+tx)

I didn't understand if we will or will not use a cache key. Apparently it makes sense to use.

/v1a/version

When this is called in the Explorer?

[r4mmer]

How the endpoints of the full-node were selected for this design? I think this should be described.

Of all endpoints the explorer uses, the only ones not included here are:

• /v1a/push_tx
  • should not be cached
• /v1a/decode_tx
  • there's an active issue to use the wallet-lib instead of this api
• /v1a/thin_wallet/address_balance and /v1a/thin_wallet/address_search
  • covered on another issue
• /v1a/graphviz
  • the use on the explorer wasn't standard, so I'm hesitant to include on a standardized proxy api

What do you mean by this? We will have 2 different endpoints in API Gateway both proxying to the same endpoint in the full-node?

Yes, the token and transaction work differently depending on which parameters are used.
If you pass an id (for tx_id or token_uid) it will retrieve that transaction or token, but if id is not present it will assume a list operation.
For the transaction the list operation has some pagination parameters (and a filter by type).
For the token there are no parameters.

Since the parameters are so different the cache key cannot be the same for the get and list operations, which is why it makes more sense to split in 2 lambdas.

I didn't understand if we will or will not use a cache key. Apparently it makes sense to use.

It does make sense to use, but on the explorer the block and tx parameters for this api are hard-coded, so all requests will retrieve the same amount of each.
If all requests retrieve the same amount, it makes sense to not use one as well, but i'm changing to using so the design won't be ambiguous.

When this is called in the Explorer?

This api is called here.
It's basically to set the network from the explorer to the same network of the backend (full-node) and check if the full-node version is allowed.

[luislhl]

Approved

[obiyankenobi]

/v1a/thin_wallet/token

• List tokens:
  • cache 10 seconds
  • cache key: none

AFAIK, this list does not change anymore, as we only return the first 200 elements. So we could add a very long cache time.

/v1a/dashboard_tx

• cache 10 seconds
• cache key: block + tx

From what I remember, we load the txs using this API and then listen to new ones on the ws. If we have this 10s cache, we might miss a few ones. Not sure if it's a big deal or how often this would happen, but we need to be aware of that.

[r4mmer]

AFAIK, this list does not change anymore, as we only return the first 200 elements. So we could add a very long cache time.

Yes, changed to 1h, but a design to change this is high on the priority list.

From what I remember, we load the txs using this API and then listen to new ones on the ws. If we have this 10s cache, we might miss a few ones. Not sure if it's a big deal or how often this would happen, but we need to be aware of that.

It's very likely that some TXs may be missed We can create an issue later to make a dashboard_tx api on the explorer-service so that the user won't miss any TX.
Until then this proxy will protect the full-node during peak traffic.

[pedroferreira1]

For me the design is approved, just want to confirm one thing. All our API calls from the explorer will go to explorer service now instead of directly to the full node, right? We have 2 steps to implement this design, (i) create the cache attributes on API Gateway and change the explorer code to call explorer service API, right? Would be good to have a task breakdown.

Yes, changed to 1h, but a design to change this is high on the priority list.

I agree with this time, but is critical that you add a note in the Token List design about that. Before we forget this and keep using this long time cache.

It's very likely that some TXs may be missed We can create an issue later to make a dashboard_tx api on the explorer-service so that the user won't miss any TX.

I don't think I like 10 seconds of cache in this API. The full node would crash when receiving 1,000 requests/s in this endpoint but 10s of cache is too much in my opinion.

I would go with at most 3s (maybe 1s of cache is enough). Not sure what @jansegre @msbrogli @luislhl think about it, they also have lot of experience with the full node consumption.

[luislhl]

I would go with at most 3s (maybe 1s of cache is enough). Not sure what @jansegre @msbrogli @luislhl think about it, they also have lot of experience with the full node consumption.

I think I agree with 10s being too much. Maybe 5s is enough in my opinion.

But at the same time, I think the impact of missing a tx because of this won't be so great. It will only make the user reload the page because his/her tx didn't appear, and then he/she would get it correctly.