The common API

[pepyakin]

There are several rollup stacks out there. E.g. #19 #8 #9 and sovereign-sdk.

Sov is Rust. Others in Go. Probably there may be yet others in completely different languages (although, not very likely). We don't want to reimplement all the intricacies in every adapter. Therefore, it may be worthwhile to push most of the heavy lifting into what we called previous an API.

There are several approaches to build such an API:

• One of them is to try to come up with a common denominator API that could be used to implement every adapter we would encounter.
• Another way, is to just provide API specifically for each adapter, i.e. ad-hoc API, Like, e.g. /sovereign_v1/finalized_at/:height and /rollkit_v2/get_blob/:id.

While the common denominator approach seems cleaner/more elegant/more aesthetically pleasing, I think the ad-hoc API approach is the way to go, at least for the beginning.

I think getting the common denominator API is tricky. Every new adapter may change the whole thing completely and we may still need to add the ad-hoc things for specific adapters.

I am pretty sure we will have some breaking changes initially. With the ad-hoc APIs we don't have to spend too much time to foresee all the possible use-cases, we just implement the current API in the form of RPC and that's it. There still may be a new adapter that challenges some foundational assumptions, but that would affect the internal API/the implementations which should be easier to change compared to the common denominator RPC which a public API. Such a change may affect the adapters who were perfectly happy with the previous version of the API (most of them probably).

The elegance of the common denominator API also comes at a cost: while ad-hoc API will return all the required information for the adapter call in one go, the common denominator API may require several calls and further logic on the adapter side.

OTOH, the ad-hoc APIs allow making the adapters dumb. All the heavy lifting is performed in the implementation of API. Making the adapters dumb is desirable because it makes the whole system more amenable for testing. Arguably, dumb adapters don't require as much testing besides some integration tests and we can cover the common Rust logic more extensively than we could ever do in adapters.

The ad-hoc API has a downside, that probably the adapters will have to be in-tree. I think this is fine. We can provide the adapters for select rollup stacks. For example I wonder if #9 #8 #19 cover a big chunk of rollups already. Realistically, we cannot expect that stack devs will line up to implement integrations with sugondat, so we will have to kinda do it ourselves at least initially.

Also, picking the ad-hoc API way doesn't mean we should stick forever to it. Maybe, if there's enough demand, we could add a common denominator API later on, when we gain enough experience and things may stabilize.

With all that said, I will present my current vision of the common denominator API below. The reason is, it should be obvious how to structure the ad-hoc APIs, however, as I mentioned, the common denominator API is less obvious. Also, perhaps, this API may inform under the hood abstractions for the ad-hoc API approach.

header(height) → AbridgedHeader

/// Get the blob by it's ID. 
get_blob(id) → Option<Blob>

/// Returns the inidices of blob extrinsics for the given namespace.
get_blobs(height, Namespace) → [ExtrinsicIndex]

/// Same as the above, but also returns a proof that the returned extrinsic index vector:
/// (a) doesn't omit any extrinsic index (exhaustive)
/// (b) every extrinsic index represents a blob of the specified namespace.
get_blobs_with_proof(height, Namespace) → [ExtrinsicIndex], Proof

validate(id, proof) → bool


submit(Blob, Namespace) → id

Notes:

• height — a finalized block number. In every API so far it was u64 and I am ok with it. In JSON we may consider using String to encode it.
• id is an identifier of an included blob, e.g. height and extrinsic index or height and extrinsic hash.
  • The is a subtle difference between picking index or hash for the ID in that hash can be obtained before submitting but index can be only obtained after the blob landed. Arguably, it doesn't matter much because height has to be obtained after landing anyway.
  • Substrate default RPC doesn't provide extrinsic-by-hash API and it's not trivial to add since that would require integrating the corresponding index. Hence height is required.
• AbridgedHeader consists of:
  • hash
  • parent
  • nmt_root
  • timestamp
  • ...
• Blob = Vec
• Namespace, TBD by Determine Namespace ID size #12

[gabriele-0201]

we just implement the current API in the form of RPC and that's it.

I really like this approach
There are only a couple of rollups stack so implement each da-interface as an rpc should not be really difficult (maybe going deeper we will discover the opposite, but for now seems a good approach)

The ad-hoc API has a downside, that probably the adapters will have to be in-tree.

Why keeping them in tree is needed?

(referring to the common denominator API) this API may inform under the hood abstractions for the ad-hoc API approach.

we could also proceed with both, on the sugondat-chain the more complex thing is to setup the RPCs that are able to able to use some runtime-apis so maybe created the base layer having both: common divisor and ah-hoc api should not be that complex

I will present my current vision of the common denominator API

Asking to the da for ALL blobs of a namespace inside a specific block, is split in a number of rpc calls equals to 1+n where n is the number of blobs submitted in the block, why not something like get_blobs(height, Namespace) -> Vec<Blobs>? because it seems to me a standard behavior for a rollup and it could decrease the amount of calls to 1

[pepyakin]

Why keeping them in tree is needed?

well, to add another adapter would mean that the devs of that rollup stack will have to fork the project and implement their set of endpoints (assuming that they don't want to use the existing endpoints for other adapters). Then they either keep it out of tree or upstream it making it in-tree. Another alternative, is that we provide an extension point somewhere, but that feels kinda overkill and goes against the idea of having a single binary as I described in #24. But possible.

we could also proceed with both, on the sugondat-chain the more complex thing is to setup the RPCs that are able to able to use some runtime-apis so maybe created the base layer having both: common divisor and ah-hoc api should not be that complex

TBH, I am not sure if I follow. But generally going for both adhoc and the common denominator doesn't make sense to me, because we will have to spend time on it to make common denominator API reasonable (or do a half-assed approach), then support both versions. It's a no go.

Asking to the da for ALL blobs of a namespace inside a specific block, is split in a number of rpc calls equals to 1+n where n is the number of blobs submitted in the block, why not something like get_blobs(height, Namespace) -> Vec? because it seems to me a standard behavior for a rollup and it could decrease the amount of calls to 1

Well, yes, that's the problem I was trying to describe. There is a natural tradeoff between the flexibility of the API and its efficiency. Maybe, it is the common way to ask blobs in such a way, and it does seem to me this way as well, but what are you going to do when that's not the case? Probably, you will add the bare versions of the API. But now you will have to maintain all of those new APIs (you don't want to break older clients do you?). Are those efficiency gains worth it? Maybe or maybe not.

That also shows the advantage of the ad-hoc APIs. You don't touch the old APIs, if the adapter changes you add new API, if something requires change, that something is not public API.

[gabriele-0201]

Then they either keep it out of tree or upstream it making it in-tree

the added set of RPCs is upstreamed, not the adapter code itself, right ?¿

because we will have to spend time on it to make common denominator API reasonable

yeah, that's true, in my initial equation I imagined the difficulty of creating a common denominator API too low.... for sure satisfy all different adapters is not easy

But now you will have to maintain all of those new APIs (you don't want to break older clients do you?). Are those efficiency gains worth it? Maybe or maybe not.

I see your point now... decide something like this require some security on the functioning of the system and right now we don't have this

That also shows the advantage of the ad-hoc APIs. You don't touch the old APIs, if the adapter changes you add new API, if something requires change, that something is not public API.

yeah, and this could also fit with the shim described in #24 where it changes overtime with the adapters evolving

[pepyakin]

the added set of RPCs is upstreamed, not the adapter code itself, right ?¿

yep!

[rphmeier]

Do I have this right? The proposed architecture is that every adapter is a thin wrapper around a spawned child process which is the API worker and which implements APIs for every adapter to be served over some serialized query/subscription protocol like JSON-RPC.

Internally these APIs would be implemented with something like the common-denominator API, which is itself not exposed.

[pepyakin]

Almost.

around a spawned child process

I think spawning a child process is a unnecessary complication. I was envisioning that the shim process would be launched by a docker-compose or another similar program.

which implements APIs for every adapter

Yes, althought I am not sure if those should be enabled/exposed by default. Maybe for e.g. opstack you may need to pass an --expose-opstack, but not sure yet.

JSON-RPC

I was hoping to avoid JSON-RPC if possible. That's tangential anyway.

Internally these APIs would be implemented with something like the common-denominator API, which is itself not exposed.

Yeah, kinda, but not necessarily. It should be just reasonably designed.

[pepyakin]

Closed by #32