Multi-language bindings for qos_core

[alxiong]

May I ask if there's any plan to add bindings of qos_core and qos_nsm to expose the protocol and the state machine (and secure entropy gen) so that app binaries written in other languages can safely communicate over the vscok?

(Alternatively, I guess I'm looking for a go implementation of QuorumOS protocol, bindings or native impl, either should work)

I saw #18 briefly mentioned a similar intention but lacking context of whether why a serialization format is sufficient for multi-language support.
Is the idea doing IPC between a qos_core-wrapping rust app and my go app with language-agnostic serialization (like cbor?)
If so, could be nice to add a centralized documentation of the structs/message schema (and the concrete format chosen) that I need to unmarshal in the other language.

[emostov]

Hey @alxiong -

Important note: We're planning a significant refactor soon that will allow apps to communicate directly over VSOCK instead of being proxied by QOS protocol. That said, here's how things work today:

Current data flow: See the readme here

You probably don't need full bindings:

• Your Go app can use whatever serialization it wants - no QOS-specific codecs needed
• Your app just needs to know which unix socket to accept connections on
• NSM API: Go libraries already exist (e.g., https://github.com/hf/nsm)
• State machine: You don't need access since apps only start after it reaches a terminal state

You might need qos_p256 bindings to use the Quorum Key the enclave boots with.

How it works today:
On the host side, you send a proxy request - this is an enum variant serialized in borsh that wraps your app-specific serialized data. QOS protocol receives this request and sends just the data portion to your app over the unix socket. We have some sample host-side utils for this here (from a Rust gRPC host, just to illustrate usage).

Borsh is supported in Go, so you could recreate the borsh serialization if needed, but I think you can avoid it entirely with the solution below.

Proposed solution for your use case:
We could add a simple /proxy-request endpoint to qos_host that takes a body of bytes, wraps them in ProtocolMsg::ProxyRequest, sends it to QOS protocol, and returns the response. This would let you keep all Go-specific serialization on the host side without dealing with borsh.

[alxiong]

This is super helpful! @emostov
The linked diagram in Readme and your clarification makes perfect sense.

re: proposed solution
i actually didn't notice the distinctions of what you proposed and what's be illustrated in the current enclave data flow. I thought the enclave app (in go) will still have to unmarshal the serialized app data parsed by the QOS protocol?

This would let you keep all Go-specific serialization on the host side without dealing with borsh.

iiuc, the communication between host (qos_host) and the QOS protocol (in rust inside enclave) via vsock needs serialization like borsh. At the same time, the communication between QOS protocol (qos_core) and my enclave app (in go) are not two separate process, thus also need serialized bytes for IPC.
did i misunderstood something?

[emostov]

@alxiong assuming data is the serialized bytes for a request to your app

1. Go client sends POST /proxy-request { "data": "0x123abc" } to qos_host
2. qos_host will create ProtocolMsg::ProxyRequest { data } and send that to QOS protocol over VSOCK
3. QOS protocol will receive ProtocolMsg::ProxyRequest { data } and send only data over a unix socket to your Go app
4. Your Go app will read just data bytes from over the unix socket and deserialize using your apps specific codec
5. Your Go app will do some stuff and then create a response and serialize that with its app specific codec
6. You Go app will take those serialized bytes, lets call it data_response and send it over the unix socket
7. QOS protocol will receive data_response over unix socket and wrap it in ProtocolMsg::ProxyResponse { data_response } and then send that back over the VSOCK
8. qos_host will receive ProtocolMsg::ProxyResponse { data_response } and respond back to your Go client with the response body { "data_response": "0x456def" }

Since we are doing planning some big refactors and want to make sure we are supporting more generically different types of apps, it would be helpful to understand what is your specific use case and requirements?

[alxiong]

thanks for the detail workflow description, i believe we are on the same page now!

what is your specific use case and requirements

I want to run Geth's state transition function (STF) inside TEE. this require authenticated storage (as it writes to leveldb's persistent storage volume) and require accepting new blocks from a trusted data oracle (as qos already support TLS fetcher enclave).
in my mind, we are not exactly stateless (unlike the signing service). the geth could be long-running processing many blocks and keep running for days before shutting down.

we thought about using encrypted db for tamper resistant storage, but we decide to go with a simpler authenticated storage (via MAC tags on db entries for now) due to:

• disk encryption to encrypt container (via dm-crypt kernel module by modifying the default kernel) takes large memory resource, basically i unload the entire geth storage in my memory.. 🙃
• we haven't implemented the key forwarding as you guys already provide, and we want the new updated PCR0 or new enclave to continue accessing the plaintext db entries.
• having per-enclave/pcr0 decryption key would mean that KMS keep track of whitelist of measurements and their decryption keys, so the disk encryption is using symmetric cipher key, and we need to pass an encrypted AES-key to the whitelisted enclave after verifying their boot attestation at KMS. and this whole workflow is quite TEE-specific (or at least AWS-specific).

with authenticated db, whenever we read, abstractly, we fetch from persistent storage the key-value entry and its associated tags generated by this enclave during its bootstrapping stage. Tags are only generated and verifiable by the enclave who holds the HMAC key inside the enclave)
We design the workflow to be as TEE-agnostic as possible.

We are looking into the amazing library qos provides and validating the prospect of switching to your stack.

Happy to get your opinion and input on these design choices. We are also happy to hop on a call to further explain.

cc @Sneh1999 @zacshowa

[emostov]

Makes sense - we have talked about similar persistence setups internally but due to other competing priorities have not been able to dive in deeper. Ideally one day we can have a suite of enclave friendly DB primitives :)

For both the authentication and encryption based patterns it sounds like a secret that is shareable between enclaves across EIF (prc0) and application versions would simplify the design (these values are both tracked in what we call the manifest). The quorum key secret would be perfect here - the boot standard and then key forward flows built into QOS would make propagating this secret a first class primitive. You can then derive from that secret an authentication keys, symmetric encryption keys etc. The secret is stored on the file system in the enclave, so you can use standard Go logic to read it in while Geth is running.

Do you know how you are planning on having Geth communicate with outside of the enclave? Do you want to simulate normal networking interfaces? Or can you get all the data to run over a single or few VSOCK connections?

[alxiong]

a secret that is shareable between enclaves across EIF (prc0) and application versions would simplify the design...
The quorum key secret would be perfect here

absolutely, this quorum app-specific (not enclave/prc0-bounded) secret with app-bound lifespan is exactly what I want.

You can then derive from that secret an authentication keys, symmetric encryption keys etc.

exactly. we also took the approach of using KDF to get various key materials for different purposes.

how you are planning on having Geth communicate with outside of the enclave?

right now, we are running over a few vsock connections. 1 for geth http rpc, 1 for websocket rpc, 1 for NFS, 1 for kms, 1 for supervisord for group control and graceful shutdown. (roughly enumerating here, could miss sth or skipping over minor diff of ingress and egress)

We are all ears when it comes to what's the most convenient and robust design here.

ah, speaking of which, the graceful shutdown is another concern: we want to close app first (which has defer/Drop logic to close db connections), vsock second, and enclave instance third. Otherwise we might left with a limbo state. Sometimes when signaling TERM to the running enclave process (on the host side), the graceful shutdown is buggy. We would appreciate tooling or library to make this code less error-prone.

Another need is logging. we use enclaver for a while since proxying logs outside is out-of-box (but enclaver doesn't have reproducible build, and frankly we want to avoid using it). Does qos has any handy utils or config that helps with debug mode logging?