Skip to content

Commit 7174e76

Browse files
authored
feat: cocoon vm exec via vsock + cocoon-agent (#25)
* feat: vsock and vm exec via cocoon-agent Default-on hybrid vsock UDS for every CH/FC VM at <runDir>/vsock.uds with guest CID 3 (constant — per-VM isolation comes from distinct socket paths). ToVM populates VsockSocket on running VMs. cocoon vm exec dials the UDS, performs the CONNECT <port> handshake, and hands the resulting stream to cocoon-agent client.Run for full kubectl-exec semantics (stdin/stdout/stderr streaming, exit-code propagation). Restore/clone patch the snapshot's vsock.socket to the new VM's runDir so restored CH VMs keep agent reachability. * refactor: split hypervisor/backend.go by operation Move methods out of the 1000-line monolith into per-operation files matching SKILL.md's 'one responsibility per file' rule. backend.go keeps the const block, Backend/BackendConfig types, and the *Spec hooks; everything else moves to inspect.go, state.go, create.go, clone.go, restore.go, start.go, stop.go, helpers.go (new), plus appended sections in snapshot.go and gc.go.
1 parent ac727a5 commit 7174e76

32 files changed

Lines changed: 1402 additions & 895 deletions

KNOWN_ISSUES.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,16 @@ Clone regenerates cidata for **network reconfiguration only** — it does not in
4848

4949
This is by design: clone restores the VM's exact state including all account settings.
5050

51+
## `vm exec` requires vsock — legacy VMs and Windows excluded
52+
53+
`cocoon vm exec` dials the cocoon-agent inside the guest over hybrid vsock. Three caveats:
54+
55+
- **Legacy VMs (created before vsock support landed)** have no vsock UDS bound. `vm inspect` omits `vsock_socket` and `vm exec` returns `vsock not configured for this VM (recreate the VM to enable agent exec)`. Recreate the VM to gain exec capability.
56+
- **Windows guests** (`--windows`) have no cocoon-agent build yet (v0.1 is Linux-only). `vm exec` short-circuits with a clear error.
57+
- **FC clone vsock requires FC ≥ v1.16** (the `vsock_override` field on `PUT /snapshot/load` was merged post-v1.15). Older FC rejects the field with `unknown field vsock_override`. Cocoon sends the field only on clone (omitted on same-VM restore for FC < v1.16 compatibility); upgrade FC to clone-with-vsock.
58+
59+
Race window: `cocoon vm run X && cocoon vm exec X -- cmd` may fail with `read CONNECT reply: EOF` if the in-guest agent hasn't started yet. The error includes the hint `(cocoon-agent may still be starting; retry shortly)`. Wait ~5–10s after `vm run` returns.
60+
5161
## Clone/restore disk queue count is immutable
5262

5363
When cloning or restoring a VM with a different `--cpu` value, the disk `num_queues` (one queue per vCPU) retains the snapshot's original value. This is because `num_queues` is part of the virtio-blk device state baked into the binary snapshot — changing it in `config.json` causes Cloud Hypervisor to crash on `vm.restore`.

README.md

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,7 @@ cocoon
138138
│ ├── list (alias: ls) List VMs with status
139139
│ ├── inspect VM Show detailed VM info (JSON)
140140
│ ├── console [flags] VM Attach interactive console
141+
│ ├── exec [flags] VM -- CMD Run a command in a running VM via cocoon-agent (vsock)
141142
│ ├── rm [flags] VM [VM...] Delete VM(s) (--force to stop first)
142143
│ ├── restore [flags] VM SNAP Restore a running VM to a snapshot
143144
│ ├── status [VM...] Watch VM status in real time
@@ -292,6 +293,25 @@ Applies to `cocoon vm debug`:
292293
| ---------------- | -------- | ------------------------------------------------- |
293294
| `--escape-char` | `^]` | Escape character (single char or `^X` caret notation) |
294295

296+
### Exec Flags
297+
298+
`cocoon vm exec` runs a command inside a running VM via the cocoon-agent (vsock, no SSH). Stdin/stdout/stderr stream like `kubectl exec`; the host shell sees the guest command's exit code.
299+
300+
| Flag | Default | Description |
301+
| -------------- | ------- | -------------------------------------------------- |
302+
| `--env`, `-e` | | Extra env var `KEY=VALUE` (repeatable) |
303+
304+
```
305+
$ cocoon vm exec myvm -- uname -n
306+
myvm
307+
$ echo hello | cocoon vm exec myvm -- cat
308+
hello
309+
$ cocoon vm exec -e FOO=bar myvm -- sh -c 'echo $FOO'
310+
bar
311+
```
312+
313+
Requires cocoon-agent to be running inside the guest (already baked into the official `ghcr.io/cocoonstack/cocoon/ubuntu:24.04` image and started via systemd). Windows guests are not yet supported.
314+
295315
### List Flags
296316

297317
Applies to `cocoon vm list`, `cocoon image list`, and `cocoon snapshot list`:

cmd/vm/commands.go

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ type Actions interface {
1818
List(cmd *cobra.Command, args []string) error
1919
Inspect(cmd *cobra.Command, args []string) error
2020
Console(cmd *cobra.Command, args []string) error
21+
Exec(cmd *cobra.Command, args []string) error
2122
RM(cmd *cobra.Command, args []string) error
2223
Restore(cmd *cobra.Command, args []string) error
2324
Debug(cmd *cobra.Command, args []string) error
@@ -103,6 +104,14 @@ func Command(h Actions) *cobra.Command {
103104
}
104105
consoleCmd.Flags().String("escape-char", "^]", "escape character (single char or ^X caret notation)")
105106

107+
execCmd := &cobra.Command{
108+
Use: "exec [flags] VM -- COMMAND [ARGS...]",
109+
Short: "Run a command inside a running VM via cocoon-agent (vsock)",
110+
Args: cobra.MinimumNArgs(2),
111+
RunE: h.Exec,
112+
}
113+
execCmd.Flags().StringArrayP("env", "e", nil, "extra env var KEY=VALUE (repeatable)")
114+
106115
rmCmd := &cobra.Command{
107116
Use: "rm [flags] VM [VM...]",
108117
Short: "Delete VM(s) (--force to stop running VMs first)",
@@ -164,6 +173,7 @@ func Command(h Actions) *cobra.Command {
164173
listCmd,
165174
inspectCmd,
166175
consoleCmd,
176+
execCmd,
167177
rmCmd,
168178
restoreCmd,
169179
debugCmd,

cmd/vm/exec.go

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
package vm
2+
3+
import (
4+
"errors"
5+
"fmt"
6+
"io"
7+
"net"
8+
"os"
9+
"strings"
10+
11+
"github.com/spf13/cobra"
12+
13+
"github.com/cocoonstack/cocoon-agent/client"
14+
cmdcore "github.com/cocoonstack/cocoon/cmd/core"
15+
"github.com/cocoonstack/cocoon/hypervisor"
16+
"github.com/cocoonstack/cocoon/types"
17+
)
18+
19+
const hybridVsockReplyMax = 256
20+
21+
// ErrVsockNotConfigured is returned for VMs predating vsock support (e.g. restored from a legacy snapshot).
22+
var ErrVsockNotConfigured = errors.New("vsock not configured for this VM")
23+
24+
// ExecExitError carries the agent child's exit code for host-shell propagation.
25+
type ExecExitError struct{ Code int }
26+
27+
func (e *ExecExitError) Error() string { return fmt.Sprintf("exit code %d", e.Code) }
28+
29+
func (h Handler) Exec(cmd *cobra.Command, args []string) error {
30+
ctx, conf, err := h.Init(cmd)
31+
if err != nil {
32+
return err
33+
}
34+
ref, argv := args[0], args[1:]
35+
if len(argv) > 0 && argv[0] == "--" {
36+
argv = argv[1:]
37+
}
38+
if len(argv) == 0 {
39+
return fmt.Errorf("exec: no command given")
40+
}
41+
42+
hyper, err := cmdcore.FindHypervisor(ctx, conf, ref)
43+
if err != nil {
44+
return fmt.Errorf("exec: %w", err)
45+
}
46+
info, err := hyper.Inspect(ctx, ref)
47+
if err != nil {
48+
return fmt.Errorf("exec: inspect: %w", err)
49+
}
50+
if info.State != types.VMStateRunning {
51+
return fmt.Errorf("exec: %w", hypervisor.ErrNotRunning)
52+
}
53+
if info.VsockSocket == "" {
54+
return fmt.Errorf("exec: %w (recreate the VM to enable agent exec)", ErrVsockNotConfigured)
55+
}
56+
if info.Config.Windows {
57+
return fmt.Errorf("exec: cocoon-agent is not yet available on Windows guests")
58+
}
59+
60+
envPairs, _ := cmd.Flags().GetStringArray("env")
61+
env, err := parseExecEnv(envPairs)
62+
if err != nil {
63+
return err
64+
}
65+
66+
conn, err := dialHybridVsock(info.VsockSocket, hypervisor.VsockAgentPort)
67+
if err != nil {
68+
return fmt.Errorf("exec: dial agent: %w (cocoon-agent may still be starting; retry shortly)", err)
69+
}
70+
defer conn.Close() //nolint:errcheck
71+
72+
code, err := client.Run(ctx, conn, argv, env, os.Stdin, os.Stdout, os.Stderr)
73+
if err != nil {
74+
return fmt.Errorf("exec: %w", err)
75+
}
76+
if code != 0 {
77+
return &ExecExitError{Code: code}
78+
}
79+
return nil
80+
}
81+
82+
func parseExecEnv(pairs []string) (map[string]string, error) {
83+
if len(pairs) == 0 {
84+
return nil, nil
85+
}
86+
out := make(map[string]string, len(pairs))
87+
for _, p := range pairs {
88+
k, v, ok := strings.Cut(p, "=")
89+
if !ok || k == "" {
90+
return nil, fmt.Errorf("--env %q must be KEY=VALUE", p)
91+
}
92+
out[k] = v
93+
}
94+
return out, nil
95+
}
96+
97+
// dialHybridVsock dials the host UDS and runs the CONNECT-port handshake (CH/FC share the dialect).
98+
func dialHybridVsock(socketPath string, port uint32) (io.ReadWriteCloser, error) {
99+
conn, err := net.Dial("unix", socketPath)
100+
if err != nil {
101+
return nil, err
102+
}
103+
if _, werr := fmt.Fprintf(conn, "CONNECT %d\n", port); werr != nil {
104+
_ = conn.Close()
105+
return nil, fmt.Errorf("write CONNECT: %w", werr)
106+
}
107+
reply, err := readHybridVsockReply(conn)
108+
if err != nil {
109+
_ = conn.Close()
110+
return nil, fmt.Errorf("read CONNECT reply: %w", err)
111+
}
112+
if !strings.HasPrefix(reply, "OK ") {
113+
_ = conn.Close()
114+
return nil, fmt.Errorf("hybrid vsock CONNECT %d: %s", port, strings.TrimSpace(reply))
115+
}
116+
return conn, nil
117+
}
118+
119+
// readHybridVsockReply reads one '\n'-terminated line byte-by-byte; bufio would over-read into the agent's first frame.
120+
func readHybridVsockReply(r io.Reader) (string, error) {
121+
buf := make([]byte, 0, 32)
122+
one := make([]byte, 1)
123+
for {
124+
n, err := r.Read(one)
125+
if n > 0 {
126+
buf = append(buf, one[0])
127+
if one[0] == '\n' {
128+
return string(buf), nil
129+
}
130+
if len(buf) >= hybridVsockReplyMax {
131+
return "", fmt.Errorf("reply line exceeds %d bytes", hybridVsockReplyMax)
132+
}
133+
}
134+
if err != nil {
135+
return "", err
136+
}
137+
}
138+
}

cmd/vm/exec_test.go

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
package vm
2+
3+
import (
4+
"net"
5+
"os"
6+
"strings"
7+
"testing"
8+
)
9+
10+
func TestParseExecEnv(t *testing.T) {
11+
tests := []struct {
12+
name string
13+
pairs []string
14+
want map[string]string
15+
wantErr string
16+
}{
17+
{name: "empty", pairs: nil, want: nil},
18+
{name: "single", pairs: []string{"FOO=bar"}, want: map[string]string{"FOO": "bar"}},
19+
{name: "multi", pairs: []string{"A=1", "B=2"}, want: map[string]string{"A": "1", "B": "2"}},
20+
{name: "value with =", pairs: []string{"URL=https://x?a=1"}, want: map[string]string{"URL": "https://x?a=1"}},
21+
{name: "empty value allowed", pairs: []string{"K="}, want: map[string]string{"K": ""}},
22+
{name: "missing =", pairs: []string{"FOO"}, wantErr: "must be KEY=VALUE"},
23+
{name: "empty key", pairs: []string{"=v"}, wantErr: "must be KEY=VALUE"},
24+
}
25+
for _, tt := range tests {
26+
t.Run(tt.name, func(t *testing.T) {
27+
got, err := parseExecEnv(tt.pairs)
28+
if tt.wantErr != "" {
29+
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
30+
t.Fatalf("got err=%v, want contains %q", err, tt.wantErr)
31+
}
32+
return
33+
}
34+
if err != nil {
35+
t.Fatalf("unexpected err: %v", err)
36+
}
37+
if len(got) != len(tt.want) {
38+
t.Fatalf("got %v, want %v", got, tt.want)
39+
}
40+
for k, v := range tt.want {
41+
if got[k] != v {
42+
t.Errorf("got[%q]=%q, want %q", k, got[k], v)
43+
}
44+
}
45+
})
46+
}
47+
}
48+
49+
func TestReadHybridVsockReply(t *testing.T) {
50+
tests := []struct {
51+
name string
52+
input string
53+
want string
54+
wantErr string
55+
}{
56+
{name: "ok line", input: "OK 1024\n", want: "OK 1024\n"},
57+
{name: "stops at newline (next bytes preserved)", input: "OK 5\nLEFTOVER", want: "OK 5\n"},
58+
{name: "no newline EOF", input: "OK 5", wantErr: "EOF"},
59+
{name: "overflow", input: strings.Repeat("a", hybridVsockReplyMax+1), wantErr: "exceeds"},
60+
}
61+
for _, tt := range tests {
62+
t.Run(tt.name, func(t *testing.T) {
63+
got, err := readHybridVsockReply(strings.NewReader(tt.input))
64+
if tt.wantErr != "" {
65+
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
66+
t.Fatalf("got err=%v, want contains %q", err, tt.wantErr)
67+
}
68+
return
69+
}
70+
if err != nil {
71+
t.Fatalf("unexpected err: %v", err)
72+
}
73+
if got != tt.want {
74+
t.Errorf("got %q, want %q", got, tt.want)
75+
}
76+
})
77+
}
78+
}
79+
80+
// TestDialHybridVsock_ConnectHandshake spins up an in-process listener that
81+
// speaks the CH/FC hybrid vsock dialect (CONNECT <port>\n → OK <port>\n).
82+
func TestDialHybridVsock_ConnectHandshake(t *testing.T) {
83+
// macOS caps unix socket paths at ~104 bytes, so t.TempDir() (long
84+
// /var/folders/... path) can overflow. Use os.CreateTemp + immediate unlink.
85+
f, err := os.CreateTemp("", "vsock-*.uds")
86+
if err != nil {
87+
t.Fatalf("create temp: %v", err)
88+
}
89+
sockPath := f.Name()
90+
_ = f.Close()
91+
_ = os.Remove(sockPath)
92+
defer os.Remove(sockPath) //nolint:errcheck
93+
94+
ln, err := net.Listen("unix", sockPath)
95+
if err != nil {
96+
t.Fatalf("listen: %v", err)
97+
}
98+
defer ln.Close() //nolint:errcheck
99+
100+
tests := []struct {
101+
name string
102+
reply string
103+
wantErr string
104+
}{
105+
{name: "OK accepted", reply: "OK 9001\n"},
106+
{name: "rejected", reply: "Failed\n", wantErr: "Failed"},
107+
}
108+
for _, tt := range tests {
109+
t.Run(tt.name, func(t *testing.T) {
110+
done := make(chan struct{})
111+
go func() {
112+
defer close(done)
113+
server, aErr := ln.Accept()
114+
if aErr != nil {
115+
return
116+
}
117+
defer server.Close() //nolint:errcheck
118+
buf := make([]byte, 64)
119+
n, _ := server.Read(buf)
120+
want := "CONNECT 1024\n"
121+
if string(buf[:n]) != want {
122+
t.Errorf("server got %q, want %q", buf[:n], want)
123+
}
124+
_, _ = server.Write([]byte(tt.reply))
125+
}()
126+
127+
conn, err := dialHybridVsock(sockPath, 1024)
128+
if tt.wantErr != "" {
129+
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
130+
t.Fatalf("got err=%v, want contains %q", err, tt.wantErr)
131+
}
132+
<-done
133+
return
134+
}
135+
if err != nil {
136+
t.Fatalf("unexpected err: %v", err)
137+
}
138+
_ = conn.Close()
139+
<-done
140+
})
141+
}
142+
}

go.mod

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ module github.com/cocoonstack/cocoon
33
go 1.25.6
44

55
require (
6+
github.com/cocoonstack/cocoon-agent v0.1.1-0.20260505130343-db13d35d7b13
67
github.com/containernetworking/cni v1.3.0
78
github.com/containernetworking/plugins v1.9.0
89
github.com/creack/pty v1.1.24

go.sum

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@ github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b h1:r6VH0faHjZe
1010
github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b/go.mod h1:Vz9DsVWQQhf3vs21MhPMZpMGSht7O/2vFW2xusFUVOs=
1111
github.com/cockroachdb/redact v1.1.5 h1:u1PMllDkdFfPWaNGMyLD1+so+aq3uUItthCFqzwPJ30=
1212
github.com/cockroachdb/redact v1.1.5/go.mod h1:BVNblN9mBWFyMyqK1k3AAiSxhvhfK2oOZZ2lK+dpvRg=
13+
github.com/cocoonstack/cocoon-agent v0.1.1-0.20260505130343-db13d35d7b13 h1:/hJYLC0uOuK6umFiWVXxwH9BbmTUUsx57neYQCP5WqE=
14+
github.com/cocoonstack/cocoon-agent v0.1.1-0.20260505130343-db13d35d7b13/go.mod h1:s2pwhiZASfKOsJdsoLeIoZ8dxV6PrzNZfCsTSHWvCjM=
1315
github.com/containerd/stargz-snapshotter/estargz v0.18.2 h1:yXkZFYIzz3eoLwlTUZKz2iQ4MrckBxJjkmD16ynUTrw=
1416
github.com/containerd/stargz-snapshotter/estargz v0.18.2/go.mod h1:XyVU5tcJ3PRpkA9XS2T5us6Eg35yM0214Y+wvrZTBrY=
1517
github.com/containernetworking/cni v1.3.0 h1:v6EpN8RznAZj9765HhXQrtXgX+ECGebEYEmnuFjskwo=

0 commit comments

Comments
 (0)