Skip to content

Commit c25d54c

Browse files
authored
embedded DHCP version, no dnsmasq anymore, no 253 routes anymore (#1)
* daemon + dhcp server, no dnsmasq anymore * refactor and cleanup * fix docs * fix round 1 * running ver
1 parent 665e62e commit c25d54c

31 files changed

Lines changed: 1315 additions & 657 deletions

README.md

Lines changed: 100 additions & 140 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,20 @@
11
# cocoon-net
22

3-
VPC-native networking setup for [Cocoon](https://github.com/cocoonstack/cocoon) VM nodes on GKE and Volcengine. It provisions cloud networking resources and configures the Linux host so that Windows and Linux VMs obtain VPC-routable IPs directly via DHCP — no overlay network, no iptables DNAT, no `kubectl port-forward` required.
3+
VPC-native networking for [Cocoon](https://github.com/cocoonstack/cocoon) VM nodes. Provisions cloud networking resources and runs an embedded DHCP server so VMs obtain VPC-routable IPs directly -- no overlay network, no iptables DNAT, no external DHCP server dependency.
44

55
## Overview
66

7-
- **Platform auto-detection** -- identifies GKE or Volcengine via instance metadata
7+
- **Embedded DHCP server** on `cni0` bridge, replacing the external DHCP server dependency
8+
- **Dynamic /32 host routes** added on DHCP lease, removed on expiry
9+
- **Platform auto-detection** via instance metadata (GKE or Volcengine)
810
- **Cloud resource provisioning** -- GKE alias IP ranges or Volcengine ENI secondary IPs
9-
- **Host networking** -- cni0 bridge, dnsmasq DHCP, /32 host routes, iptables FORWARD + NAT
11+
- **Host networking** -- cni0 bridge, sysctl, iptables FORWARD + NAT
1012
- **CNI integration** -- generates conflist for Kubernetes pod networking
11-
- **State management** -- persists pool state to `/var/lib/cocoon/net/pool.json`
12-
- **Adopt mode** -- bring existing hand-provisioned nodes under management without cloud API calls
13-
- **Idempotent re-runs** -- skips config writes and service restarts when nothing changed
14-
- **Dry-run mode** -- preview all changes before applying
13+
- **State management** -- pool state persisted to `/var/lib/cocoon/net/pool.json`
14+
- **Adopt mode** -- bring existing hand-provisioned nodes under management
15+
- **Daemon mode** -- runs as a long-lived systemd service
1516

16-
### Supported platforms
17+
### Supported Platforms
1718

1819
| Platform | Mechanism | Max IPs/node |
1920
|---|---|---|
@@ -22,205 +23,164 @@ VPC-native networking setup for [Cocoon](https://github.com/cocoonstack/cocoon)
2223

2324
## Architecture
2425

25-
On each node, `cocoon-net init` runs through these steps:
26-
27-
1. **Detect** the cloud platform via instance metadata (auto, or `--platform` flag).
28-
2. **Provision** cloud networking:
29-
- **GKE**: adds a secondary IP range to the subnet, assigns alias IPs to the instance NIC, fixes GCE guest-agent route hijack.
30-
- **Volcengine**: creates a dedicated /24 subnet, creates and attaches 7 secondary ENIs, assigns 20 secondary private IPs per ENI.
31-
3. **Configure** the node:
32-
- Creates `cni0` bridge with gateway IP.
33-
- Generates `/etc/dnsmasq-cni.d/cni0.conf` with contiguous DHCP ranges and restarts `dnsmasq-cni`.
34-
- Adds `/32` host routes for each VM IP pointing to `cni0`.
35-
- Installs iptables FORWARD rules between secondary NICs and `cni0`, plus NAT MASQUERADE for outbound.
36-
- Applies sysctl (`ip_forward=1`, `rp_filter=0`).
37-
4. **Generate** `/etc/cni/net.d/30-dnsmasq-dhcp.conflist`.
38-
5. **Save** pool state to `/var/lib/cocoon/net/pool.json`.
26+
```
27+
cocoon-net init cocoon-net daemon
28+
| |
29+
v v
30+
Cloud provisioning Node setup (sysctl, bridge, iptables, CNI conflist)
31+
(alias IPs / ENIs) |
32+
| v
33+
v DHCP server on cni0
34+
pool.json <---------- |
35+
v
36+
On lease: add /32 route
37+
On release: del /32 route
38+
```
3939

40-
## Installation
40+
**Two-phase operation**:
41+
42+
1. `cocoon-net init` (or `adopt`) -- one-time cloud provisioning + state persistence
43+
2. `cocoon-net daemon` -- long-running service: node setup + DHCP + dynamic routing
4144

42-
### Download
45+
## Installation
4346

4447
```bash
4548
curl -sL https://github.com/cocoonstack/cocoon-net/releases/latest/download/cocoon-net_Linux_x86_64.tar.gz | tar xz
4649
sudo install -m 0755 cocoon-net /usr/local/bin/
4750
```
4851

49-
### Build from source
52+
Build from source:
5053

5154
```bash
5255
git clone https://github.com/cocoonstack/cocoon-net.git
5356
cd cocoon-net
54-
make build # produces ./cocoon-net
55-
```
56-
57-
## Configuration
58-
59-
### Flags
60-
61-
| Flag | Default | Description |
62-
|---|---|---|
63-
| `--platform` | auto-detect | Force platform (`gke` or `volcengine`) |
64-
| `--node-name` | (required) | Virtual node name (e.g. `cocoon-pool`) |
65-
| `--subnet` | (required) | VM subnet CIDR (e.g. `172.20.100.0/24`) |
66-
| `--pool-size` | `140` (init) / `253` (adopt) | Number of IPs in the pool |
67-
| `--gateway` | first IP in subnet | Gateway IP on `cni0` |
68-
| `--primary-nic` | auto-detect | Host primary NIC (`ens4` on GKE, `eth0` on Volcengine) |
69-
| `--dns` | `8.8.8.8,1.1.1.1` | Comma-separated DNS servers for DHCP clients |
70-
| `--state-dir` | `/var/lib/cocoon/net` | State directory for `pool.json` |
71-
| `--dry-run` | `false` | Show what would be done, without making changes |
72-
| `--manage-iptables` | `false` | (adopt only) Let cocoon-net write FORWARD + NAT rules |
73-
74-
### Environment variables
75-
76-
| Variable | Default | Description |
77-
|---|---|---|
78-
| `COCOON_NET_LOG_LEVEL` | `info` | Log level (`debug`, `info`, `warn`, `error`) |
79-
80-
### Credentials
81-
82-
**GKE**
83-
84-
Uses application default credentials or the GCE instance service account. Requires `roles/compute.networkAdmin` or equivalent.
85-
86-
**Volcengine**
87-
88-
Reads from `~/.volcengine/config.json`:
89-
90-
```json
91-
{
92-
"access_key_id": "AKxxxx",
93-
"secret_access_key": "xxxx",
94-
"region": "cn-hongkong"
95-
}
96-
```
97-
98-
Or environment variables:
99-
100-
```bash
101-
export VOLCENGINE_ACCESS_KEY_ID=AKxxxx
102-
export VOLCENGINE_SECRET_ACCESS_KEY=xxxx
103-
export VOLCENGINE_REGION=cn-hongkong
57+
make build
10458
```
10559

10660
## Usage
10761

108-
### init — full node setup
62+
### init -- provision cloud networking
10963

11064
```bash
11165
sudo cocoon-net init \
66+
--platform gke \
11267
--node-name cocoon-pool \
11368
--subnet 172.20.100.0/24 \
11469
--pool-size 140
11570
```
11671

117-
With all flags:
72+
### daemon -- run DHCP server (systemd service)
11873

11974
```bash
120-
sudo cocoon-net init \
121-
--node-name cocoon-pool \
122-
--subnet 172.20.100.0/24 \
123-
--pool-size 140 \
124-
--gateway 172.20.100.1 \
125-
--platform volcengine \
126-
--primary-nic eth0 \
127-
--dns "8.8.8.8,1.1.1.1" \
128-
--state-dir /var/lib/cocoon/net
75+
sudo cocoon-net daemon
12976
```
13077

131-
Dry run (show what would be done):
78+
The daemon loads the pool from `pool.json`, configures host networking, and starts the embedded DHCP server. Host routes are managed dynamically: added when a VM gets a lease, removed when the lease expires.
13279

133-
```bash
134-
sudo cocoon-net init --node-name cocoon-pool --subnet 172.20.100.0/24 --dry-run
80+
Systemd unit:
81+
82+
```ini
83+
[Unit]
84+
Description=cocoon-net VPC networking daemon
85+
After=network-online.target
86+
Wants=network-online.target
87+
88+
[Service]
89+
ExecStart=/usr/local/bin/cocoon-net daemon
90+
Restart=always
91+
RestartSec=5
92+
93+
[Install]
94+
WantedBy=multi-user.target
13595
```
13696

137-
### adopt bring an existing node under cocoon-net management
97+
### adopt -- bring an existing node under management
13898

139-
For nodes whose cloud networking (alias IP range or secondary ENIs) was already provisioned by hand, `adopt` runs only the host-side configuration (bridge, sysctl, routes, dnsmasq, CNI conflist) and writes the pool state file — without calling any cloud APIs.
99+
For nodes whose cloud networking was already provisioned by hand:
140100

141101
```bash
142102
sudo cocoon-net adopt \
103+
--platform gke \
143104
--node-name cocoon-pool \
144105
--subnet 172.20.0.0/24
145106
```
146107

147-
By default, adopt preserves the host's existing iptables rules. Opt in to let cocoon-net manage them:
108+
### status -- show pool state
148109

149110
```bash
150-
sudo cocoon-net adopt \
151-
--node-name cocoon-pool \
152-
--subnet 172.20.0.0/24 \
153-
--manage-iptables
111+
cocoon-net status
154112
```
155113

156-
Cloud-side teardown must be done manually on adopted nodes — cocoon-net will not undo what it did not provision.
157-
158-
### status — show pool state
114+
### teardown -- remove cloud networking resources
159115

160116
```bash
161-
sudo cocoon-net status
117+
sudo cocoon-net teardown
162118
```
163119

164-
Output:
120+
## Flags
165121

166-
```
167-
Platform: volcengine
168-
Node: cocoon-pool
169-
Subnet: 172.20.100.0/24
170-
Gateway: 172.20.100.1
171-
IPs: 140
172-
Updated: 2026-04-04T06:00:00Z
173-
ENIs: 7
174-
SubnetID: subnet-xxx
175-
```
122+
| Flag | Default | Description |
123+
|---|---|---|
124+
| `--platform` | (required) | Cloud platform (`gke` or `volcengine`) |
125+
| `--node-name` | (required) | Virtual node name |
126+
| `--subnet` | (required) | VM subnet CIDR (e.g. `172.20.100.0/24`) |
127+
| `--pool-size` | `140` (init) / `253` (adopt) | Number of IPs in the pool |
128+
| `--gateway` | first IP in subnet | Gateway IP on `cni0` |
129+
| `--primary-nic` | auto-detect | Host primary NIC |
130+
| `--dns` | `8.8.8.8,1.1.1.1` | DNS servers for DHCP clients |
131+
| `--state-dir` | `/var/lib/cocoon/net` | State directory for `pool.json` |
132+
| `--lease-file` | `/var/lib/cocoon/net/leases.json` | DHCP lease persistence file |
133+
| `--dry-run` | `false` | Preview changes without applying |
134+
| `--skip-iptables` | `false` | (daemon) Skip iptables setup |
135+
| `--manage-iptables` | `false` | (adopt) Let cocoon-net write iptables rules |
176136

177-
### teardown — remove cloud networking
137+
### Environment Variables
178138

179-
```bash
180-
sudo cocoon-net teardown
181-
```
139+
| Variable | Default | Description |
140+
|---|---|---|
141+
| `COCOON_NET_LOG_LEVEL` | `info` | Log level (`debug`, `info`, `warn`, `error`) |
182142

183-
### CNI integration
143+
## CNI Integration
184144

185-
Both `init` and `adopt` generate `/etc/cni/net.d/30-dnsmasq-dhcp.conflist`:
145+
Both `init` and `adopt` generate `/etc/cni/net.d/30-cocoon-dhcp.conflist`:
186146

187147
```json
188148
{
189149
"cniVersion": "1.0.0",
190-
"name": "dnsmasq-dhcp",
191-
"plugins": [
192-
{
193-
"type": "bridge",
194-
"bridge": "cni0",
195-
"isGateway": false,
196-
"ipMasq": false,
197-
"ipam": {}
198-
}
199-
]
150+
"name": "cocoon-dhcp",
151+
"plugins": [{
152+
"type": "bridge",
153+
"bridge": "cni0",
154+
"isGateway": false,
155+
"ipMasq": false,
156+
"ipam": {}
157+
}]
200158
}
201159
```
202160

203-
The CNI IPAM is intentionally empty — Windows guests obtain their IP directly from dnsmasq running on `cni0`. Using `"type": "dhcp"` would cause a dual-DHCP conflict.
204-
205-
In a CocoonSet, use `network: dnsmasq-dhcp` to route VM pods through this CNI:
161+
IPAM is intentionally empty -- VMs obtain IPs from the embedded DHCP server. In a CocoonSet:
206162

207163
```yaml
208164
spec:
209165
agent:
210-
network: dnsmasq-dhcp
166+
network: cocoon-dhcp
211167
os: windows
212168
```
213169
170+
## Credentials
171+
172+
**GKE**: Uses application default credentials or GCE instance service account (`roles/compute.networkAdmin`).
173+
174+
**Volcengine**: Reads from `~/.volcengine/config.json` or environment variables (`VOLCENGINE_ACCESS_KEY_ID`, `VOLCENGINE_SECRET_ACCESS_KEY`, `VOLCENGINE_REGION`).
175+
214176
## Development
215177

216178
```bash
217-
make build # build binary
218-
make test # run tests with coverage
219-
make lint # run golangci-lint (linux + darwin)
220-
make fmt # format code with gofumpt and goimports
221-
make deps # tidy modules
222-
make clean # remove artifacts
223-
make help # show all targets
179+
make build # build binary
180+
make test # run tests with coverage
181+
make lint # golangci-lint (linux + darwin)
182+
make fmt # gofumpt + goimports
183+
make help # show all targets
224184
```
225185

226186
### Guides
@@ -232,11 +192,11 @@ make help # show all targets
232192

233193
| Project | Role |
234194
|---|---|
195+
| [cocoon](https://github.com/cocoonstack/cocoon) | MicroVM engine (Cloud Hypervisor + Firecracker) |
235196
| [cocoon-common](https://github.com/cocoonstack/cocoon-common) | Shared metadata, Kubernetes, and logging helpers |
236197
| [cocoon-operator](https://github.com/cocoonstack/cocoon-operator) | CocoonSet and Hibernation CRDs |
237198
| [cocoon-webhook](https://github.com/cocoonstack/cocoon-webhook) | Admission webhook for sticky scheduling |
238-
| [epoch](https://github.com/cocoonstack/epoch) | Snapshot storage backend |
239-
| [vk-cocoon](https://github.com/cocoonstack/vk-cocoon) | Virtual kubelet provider managing VM lifecycle |
199+
| [vk-cocoon](https://github.com/cocoonstack/vk-cocoon) | Virtual kubelet provider |
240200

241201
## License
242202

0 commit comments

Comments
 (0)