[Improvement] Nydusd config written to disk

[Fricounet]

Problem Summary

Currently, the config needed by nydusd to start is written to disk by the snapshotter. This is a bit of a security concern because the potential registry access credentials are written in this file.
Although someone getting root access on the host would be able to do the same things as the nydus-snapshotter to generate its own credentials, it would still be better to not write them to disk at all.

Proposed Improvement

I think there are a few things to discuss there:

1. What do we write to disk? Nothing at all? Or the nydusd config but missing the creds that will have to be passed using another method
2. If we don't write the config at all, how do we ensure nydusd gets access to its config fast enough so that it can start properly?
3. If we only omit the credentials, what method should we use to transmit this information early enough that nydusd hasn't started making registry api calls?

I don't have a strong opinion for 1.
For 3, @imeoer proposed in #690 that we orchestrate calling the new nydusd API PUT /api/v1/config depending where we are in the startup process and the nydusd kind (upgrade, failover, first start). I think this would work although I'm a bit wary of the orchestration complexity there. I don't think this approach would work well for 2. though because nydusd socket is started a bit late during the startup process.

There's is another idea I got and that I wanted to share which is about using kernel FIFOs. FIFO would ensure that the config (or creds for that matter) are never persisted to disk while keeping orchestration tasks simple because the snapshotter can write to the FIFO before nydusd is even started and nydusd can wait on the FIFO when it starts to make sure it gets its config in a timely manner.

Happy to discuss more on this!

Additional Context

Issue created from the discussions we had in #690

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[imeoer]

Hi @Fricounet, thanks for the issue.

If we relax the restriction that the API calls must be unidirectional (snapshotter → nydusd), nydusd could call snapshotter-provided APIs — such as for auth retrieval. This is already done today during failover and hot upgrades: the snapshotter runs a temporary uds-based HTTP server, listened to via func (su *Supervisor) SendStatesTimeout, and nydusd acts as an HTTP client to it (for sending / receiving the fd of opened fuse device) — using the nydusd --supervisor flag to specify the UDS socket path.

[Fricounet]

I think that could work as well. If we can reuse some existing logic currently used for the supervisor, that'd be even better.

[imeoer]

@Fricounet My thought is that nydusd should not request this temporary supervisor server, because it is specifically designed for failover and upgrade purposes, and is temporarily assigned to each nydusd instance. and after use, it will be closed, transmitting other data over it would mess up the send / recv of the fuse fd.

Maybe snapshotter can listen a independent uds server, also add a configuration to the registry backend:

{
  "device": {
    "backend": {
      "type": "registry",
      "config": {
        ...
        "auth_server": "/path/to/snapshotter-auth-server.sock"
        ...
      }
    },
    ...
  },
  ...
}

Which one do you think is better compared to this?

[Fricounet]

At this point I think the proposal in the other thread might look more robust. We'd need to test if giving the token to nydusd after the socket is ready doesn't lead to final errors. But if it doesn't, it does look like it would be more robust and simple to understand than adding a new UDS server

[imeoer]

@Fricounet Okay, let me continue to take a look at this.