Skip to content

Commit eea38cb

Browse files
Copilotbcho
Copilot
and
bcho
authored
Add agent e2e node config scenarios (#170)
* Initial plan * Add agent e2e node config matrix Agent-Logs-Url: https://github.com/Azure/unbounded/sessions/95afa82b-ea29-442e-810d-674a5d2980d3 Co-authored-by: bcho <1975118+bcho@users.noreply.github.com> * Address agent e2e review feedback Agent-Logs-Url: https://github.com/Azure/unbounded/sessions/bf7223a8-8101-4d8e-94c4-6f60a177756b Co-authored-by: bcho <1975118+bcho@users.noreply.github.com> * Address agent e2e follow-up feedback Agent-Logs-Url: https://github.com/Azure/unbounded/sessions/442e0e77-0155-41c7-82c5-4b62834be2b7 Co-authored-by: bcho <1975118+bcho@users.noreply.github.com> * Discover agent node configs in e2e job Agent-Logs-Url: https://github.com/Azure/unbounded/sessions/6951efd5-d828-4068-8b85-f5c2cc79ddc5 Co-authored-by: bcho <1975118+bcho@users.noreply.github.com> * Parallelize agent node config e2e scenarios Agent-Logs-Url: https://github.com/Azure/unbounded/sessions/f57fcc1a-6b19-4828-a063-9085a9e0b487 Co-authored-by: bcho <1975118+bcho@users.noreply.github.com> * Extract machina setup and type node configs Agent-Logs-Url: https://github.com/Azure/unbounded/sessions/dfdf4bdc-d5a8-4652-aa3d-f5fe82aa2f3a Co-authored-by: bcho <1975118+bcho@users.noreply.github.com> * Ensure machina setup creates e2e state dir Agent-Logs-Url: https://github.com/Azure/unbounded/sessions/b02edbe4-11d3-4395-9282-950750113c50 Co-authored-by: bcho <1975118+bcho@users.noreply.github.com> * Assign unique MACs to parallel e2e VMs Agent-Logs-Url: https://github.com/Azure/unbounded/sessions/caa37a46-997e-4f3d-96de-d55c1710e5a8 Co-authored-by: bcho <1975118+bcho@users.noreply.github.com> * Fix parallel agent config e2e isolation Agent-Logs-Url: https://github.com/Azure/unbounded/sessions/3ec2c073-d4c3-4d5b-994c-2390bb69403c Co-authored-by: bcho <1975118+bcho@users.noreply.github.com> * Stabilize agent config e2e node readiness * Avoid logging agent bootstrap script * Remove sensitive e2e log values * Avoid logging e2e status message bodies --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: bcho <1975118+bcho@users.noreply.github.com> Co-authored-by: hbc <bahe@microsoft.com> Co-authored-by: hbc <me@hbc.rocks>
1 parent 990e2cd commit eea38cb

11 files changed

Lines changed: 897 additions & 267 deletions

File tree

‎.github/actions/agent-e2e-kind-control-plane/action.yaml‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
name: Setup agent e2e Kind control plane
2+
description: Set up KVM, dependencies, a Kind control plane, and VM bridge networking for agent e2e tests.
3+
inputs:
4+
cluster-name:
5+
description: Kind cluster name.
6+
required: true
7+
vm-subnet:
8+
description: VM subnet prefix.
9+
required: true
10+
create-vm:
11+
description: Whether to launch the default e2e VM.
12+
required: false
13+
default: "true"
14+
runs:
15+
using: composite
16+
steps:
17+
- name: Enable KVM
18+
shell: bash
19+
run: |
20+
echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
21+
sudo udevadm control --reload-rules
22+
sudo udevadm trigger --name-match=kvm
23+
24+
- name: Set up Go
25+
uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0
26+
with:
27+
go-version-file: go.mod
28+
29+
- name: Install system dependencies
30+
shell: bash
31+
run: |
32+
sudo apt-get update
33+
sudo apt-get install -y --no-install-recommends \
34+
qemu-system-x86 qemu-utils genisoimage \
35+
iptables
36+
37+
- name: Create Kind cluster
38+
uses: helm/kind-action@ef37e7f390d99f746eb8b610417061a60e82a6cc # v1.14.0
39+
with:
40+
cluster_name: ${{ inputs['cluster-name'] }}
41+
version: v0.29.0
42+
43+
- name: Configure Kind cluster networking for VM
44+
shell: bash
45+
env:
46+
KIND_CLUSTER_NAME: ${{ inputs['cluster-name'] }}
47+
run: |
48+
set -euo pipefail
49+
KIND_CONTAINER="${KIND_CLUSTER_NAME}-control-plane"
50+
KIND_IP=$(docker inspect "${KIND_CONTAINER}" \
51+
--format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}')
52+
if [[ -z "${KIND_IP}" ]]; then
53+
echo "::error::Could not determine Kind control-plane container IP"
54+
exit 1
55+
fi
56+
echo "[INFO] Kind control-plane IP: ${KIND_IP}"
57+
BRIDGE="virbr-e2e"
58+
59+
sudo iptables -I FORWARD -i "${BRIDGE}" -j ACCEPT
60+
sudo iptables -I FORWARD -o "${BRIDGE}" -j ACCEPT
61+
sudo iptables -t raw -I PREROUTING -i "${BRIDGE}" -j ACCEPT
62+
63+
echo "[INFO] Patching kindnet DaemonSet for VM-reachable control plane endpoint..."
64+
PATCH=$(cat <<EOF
65+
{
66+
"spec": {
67+
"template": {
68+
"spec": {
69+
"containers": [{
70+
"name": "kindnet-cni",
71+
"env": [
72+
{"name": "CONTROL_PLANE_ENDPOINT", "value": "${KIND_IP}:6443"}
73+
]
74+
}]
75+
}
76+
}
77+
}
78+
}
79+
EOF
80+
)
81+
kubectl -n kube-system patch daemonset kindnet \
82+
--type=strategic -p "${PATCH}"
83+
84+
echo "[INFO] Waiting for kindnet rollout..."
85+
kubectl -n kube-system rollout status daemonset/kindnet --timeout=60s
86+
87+
- name: Create QEMU VM
88+
if: ${{ inputs['create-vm'] == 'true' }}
89+
shell: bash
90+
run: python3 ./hack/agent/e2e-kind/e2e.py --verbose create-vm
91+
92+
- name: Create VM bridge
93+
if: ${{ inputs['create-vm'] != 'true' }}
94+
shell: bash
95+
run: python3 ./hack/agent/e2e-kind/e2e.py --verbose create-vm-bridge
96+
97+
- name: Attach Kind container to VM bridge
98+
shell: bash
99+
env:
100+
KIND_CLUSTER_NAME: ${{ inputs['cluster-name'] }}
101+
VM_SUBNET: ${{ inputs['vm-subnet'] }}
102+
run: |
103+
set -euo pipefail
104+
KIND_CONTAINER="${KIND_CLUSTER_NAME}-control-plane"
105+
BRIDGE="virbr-e2e"
106+
107+
echo "[INFO] Attaching Kind container to ${BRIDGE} bridge..."
108+
KIND_PID=$(docker inspect "${KIND_CONTAINER}" --format '{{.State.Pid}}')
109+
sudo ip link delete veth-kind-e2e 2>/dev/null || true
110+
sudo ip link add veth-kind-e2e type veth peer name eth-e2e
111+
sudo ip link set veth-kind-e2e master "${BRIDGE}"
112+
sudo ip link set veth-kind-e2e up
113+
sudo ip link set eth-e2e netns "${KIND_PID}"
114+
sudo nsenter -t "${KIND_PID}" -n ip addr add "${VM_SUBNET}.2/24" dev eth-e2e
115+
sudo nsenter -t "${KIND_PID}" -n ip link set eth-e2e up

‎.github/actions/agent-e2e-kind-logs/action.yaml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
name: Collect agent e2e Kind logs
2+
description: Collect and upload diagnostics for agent e2e Kind jobs.
3+
inputs:
4+
artifact-name:
5+
description: Name of the uploaded log artifact.
6+
required: true
7+
node-configs:
8+
description: Whether to collect logs for discovered node config scenarios.
9+
required: false
10+
default: "false"
11+
runs:
12+
using: composite
13+
steps:
14+
- name: Collect logs
15+
shell: bash
16+
env:
17+
COLLECT_NODE_CONFIG_LOGS: ${{ inputs['node-configs'] }}
18+
run: python3 ./hack/agent/e2e-kind/e2e.py --verbose collect-logs
19+
20+
- name: Upload logs
21+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
22+
with:
23+
name: ${{ inputs['artifact-name'] }}
24+
path: logs/
25+
retention-days: 30

‎.github/actions/agent-e2e-machina-setup/action.yaml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
name: Set up agent e2e machina resources
2+
description: Install Machine CRDs and start/validate the machina controller for agent e2e tests.
3+
runs:
4+
using: composite
5+
steps:
6+
- name: Install Machine CRD
7+
shell: bash
8+
run: python3 ./hack/agent/e2e-kind/e2e.py --verbose install-machine-crd
9+
10+
- name: Start machina controller
11+
shell: bash
12+
run: python3 ./hack/agent/e2e-kind/e2e.py --verbose start-machina-controller
13+
14+
- name: Validate machina controller
15+
shell: bash
16+
run: python3 ./hack/agent/e2e-kind/e2e.py --verbose validate-machina-controller

‎.github/workflows/agent-e2e-kind.yaml‎

Lines changed: 45 additions & 169 deletions
Original file line numberDiff line numberDiff line change
@@ -60,110 +60,14 @@ jobs:
6060
- name: Checkout
6161
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
6262

63-
- name: Enable KVM
64-
run: |
65-
echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
66-
sudo udevadm control --reload-rules
67-
sudo udevadm trigger --name-match=kvm
68-
69-
- name: Set up Go
70-
uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0
63+
- name: Set up test control plane
64+
uses: ./.github/actions/agent-e2e-kind-control-plane
7165
with:
72-
go-version-file: go.mod
66+
cluster-name: ${{ env.KIND_CLUSTER_NAME }}
67+
vm-subnet: ${{ env.VM_SUBNET }}
7368

74-
- name: Install system dependencies
75-
run: |
76-
sudo apt-get update
77-
sudo apt-get install -y --no-install-recommends \
78-
qemu-system-x86 qemu-utils genisoimage \
79-
iptables
80-
81-
- name: Create Kind cluster
82-
uses: helm/kind-action@ef37e7f390d99f746eb8b610417061a60e82a6cc # v1.14.0
83-
with:
84-
cluster_name: ${{ env.KIND_CLUSTER_NAME }}
85-
version: v0.29.0
86-
87-
- name: Configure Kind cluster networking for VM
88-
run: |
89-
set -euo pipefail
90-
KIND_CONTAINER="${KIND_CLUSTER_NAME}-control-plane"
91-
KIND_IP=$(docker inspect "${KIND_CONTAINER}" \
92-
--format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}')
93-
if [[ -z "${KIND_IP}" ]]; then
94-
echo "::error::Could not determine Kind control-plane container IP"
95-
exit 1
96-
fi
97-
echo "[INFO] Kind control-plane IP: ${KIND_IP}"
98-
BRIDGE="virbr-e2e"
99-
100-
# Allow forwarding between the VM bridge and Docker bridge.
101-
sudo iptables -I FORWARD -i "${BRIDGE}" -j ACCEPT
102-
sudo iptables -I FORWARD -o "${BRIDGE}" -j ACCEPT
103-
104-
# Docker may insert raw PREROUTING DROP rules that block non-Docker
105-
# traffic to container IPs. Insert an ACCEPT so the VM can reach the
106-
# Kind API server.
107-
sudo iptables -t raw -I PREROUTING -i "${BRIDGE}" -j ACCEPT
108-
109-
# Patch kindnet so CONTROL_PLANE_ENDPOINT uses the container IP instead
110-
# of the hostname (which is unresolvable from the VM).
111-
echo "[INFO] Patching kindnet DaemonSet for VM-reachable control plane endpoint..."
112-
PATCH=$(cat <<EOF
113-
{
114-
"spec": {
115-
"template": {
116-
"spec": {
117-
"containers": [{
118-
"name": "kindnet-cni",
119-
"env": [
120-
{"name": "CONTROL_PLANE_ENDPOINT", "value": "${KIND_IP}:6443"}
121-
]
122-
}]
123-
}
124-
}
125-
}
126-
}
127-
EOF
128-
)
129-
kubectl -n kube-system patch daemonset kindnet \
130-
--type=strategic -p "${PATCH}"
131-
132-
echo "[INFO] Waiting for kindnet rollout..."
133-
kubectl -n kube-system rollout status daemonset/kindnet --timeout=60s
134-
135-
- name: Create QEMU VM
136-
run: python3 ./hack/agent/e2e-kind/e2e.py --verbose create-vm
137-
138-
- name: Attach Kind container to VM bridge
139-
run: |
140-
set -euo pipefail
141-
KIND_CONTAINER="${KIND_CLUSTER_NAME}-control-plane"
142-
BRIDGE="virbr-e2e"
143-
144-
# Connect the Kind container directly to the VM bridge via a veth
145-
# pair so that the VM subnet is directly reachable at L2. This is
146-
# required because kindnet adds routes of the form
147-
# "10.244.x.0/24 via <nodeIP>" and the kernel rejects these when
148-
# the gateway is only reachable via an indirect route.
149-
echo "[INFO] Attaching Kind container to ${BRIDGE} bridge..."
150-
KIND_PID=$(docker inspect "${KIND_CONTAINER}" --format '{{.State.Pid}}')
151-
sudo ip link delete veth-kind-e2e 2>/dev/null || true
152-
sudo ip link add veth-kind-e2e type veth peer name eth-e2e
153-
sudo ip link set veth-kind-e2e master "${BRIDGE}"
154-
sudo ip link set veth-kind-e2e up
155-
sudo ip link set eth-e2e netns "${KIND_PID}"
156-
sudo nsenter -t "${KIND_PID}" -n ip addr add "${VM_SUBNET}.2/24" dev eth-e2e
157-
sudo nsenter -t "${KIND_PID}" -n ip link set eth-e2e up
158-
159-
- name: Install Machine CRD
160-
run: python3 ./hack/agent/e2e-kind/e2e.py --verbose install-machine-crd
161-
162-
- name: Start machina controller
163-
run: python3 ./hack/agent/e2e-kind/e2e.py --verbose start-machina-controller
164-
165-
- name: Validate machina controller
166-
run: python3 ./hack/agent/e2e-kind/e2e.py --verbose validate-machina-controller
69+
- name: Set up machina resources
70+
uses: ./.github/actions/agent-e2e-machina-setup
16771

16872
# No Machine CR pre-created; agent self-registers during bootstrap.
16973
- name: Run agent to join VM to cluster
@@ -227,77 +131,49 @@ jobs:
227131
- name: Validate node repave upgrade
228132
run: python3 ./hack/agent/e2e-kind/e2e.py --verbose validate-node-repave-upgrade
229133

230-
- name: Collect VM logs
134+
- name: Collect logs
231135
if: always()
232-
run: |
233-
mkdir -p logs
234-
VM_DIR=".vm-e2e"
235-
# Collect VM serial console log
236-
cp "${VM_DIR}/${VM_NAME}.log" logs/vm-serial.log 2>/dev/null || true
237-
SSH="ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=5 -i ${VM_DIR}/ssh/id_ed25519 ubuntu@${VM_IP}"
238-
# Collect full journal from the VM host (best-effort)
239-
$SSH "sudo journalctl --no-pager -l" > logs/vm-journal.log 2>/dev/null || true
240-
# Collect unbounded-agent logs from the VM host
241-
$SSH "sudo journalctl -u unbounded-agent --no-pager -l" > logs/vm-unbounded-agent.log 2>/dev/null || true
242-
# Collect unbounded-agent-daemon logs
243-
$SSH "sudo journalctl -u unbounded-agent-daemon --no-pager -l" > logs/vm-unbounded-agent-daemon.log 2>/dev/null || true
244-
cp ".vm-e2e/machina-controller.log" logs/machina-controller.log 2>/dev/null || true
245-
# Kubelet and containerd run inside the nspawn container, not on
246-
# the host. Use 'journalctl -M <machine>' to read the container
247-
# journal, and fall back to machinectl shell if that doesn't work.
248-
# The nspawn machine name is fixed to kube1/kube2 (decoupled from
249-
# the Kubernetes node name).
250-
$SSH "sudo machinectl list --no-pager" > logs/vm-machines.txt 2>/dev/null || true
251-
for MACHINE in kube1 kube2; do
252-
$SSH "sudo journalctl -M ${MACHINE} --no-pager -l" > logs/nspawn-${MACHINE}-journal.log 2>/dev/null || true
253-
$SSH "sudo journalctl -M ${MACHINE} -u kubelet --no-pager -l" > logs/nspawn-${MACHINE}-kubelet.log 2>/dev/null || true
254-
$SSH "sudo journalctl -M ${MACHINE} -u containerd --no-pager -l" > logs/nspawn-${MACHINE}-containerd.log 2>/dev/null || true
255-
$SSH "sudo machinectl status ${MACHINE} --no-pager" > logs/vm-machine-${MACHINE}-status.txt 2>/dev/null || true
256-
$SSH "sudo machinectl shell ${MACHINE} /usr/bin/systemctl list-units --no-pager" > logs/nspawn-${MACHINE}-units.txt 2>/dev/null || true
257-
done
258-
259-
- name: Collect cluster state
136+
uses: ./.github/actions/agent-e2e-kind-logs
137+
with:
138+
artifact-name: agent-e2e-kind-logs
139+
140+
- name: Cleanup
260141
if: always()
261-
run: |
262-
mkdir -p logs
263-
kubectl get nodes -o wide > logs/nodes.txt 2>&1 || true
264-
kubectl describe nodes > logs/nodes-describe.txt 2>&1 || true
265-
kubectl get pods -A -o wide > logs/pods.txt 2>&1 || true
266-
kubectl get events -A --sort-by='.lastTimestamp' > logs/events.txt 2>&1 || true
267-
# Collect Machine CRs (if CRD is installed)
268-
kubectl get machines -o wide > logs/machines.txt 2>&1 || true
269-
kubectl get machines -o yaml > logs/machines-full.yaml 2>&1 || true
270-
kubectl get machineconfigurations -o wide > logs/machineconfigurations.txt 2>&1 || true
271-
kubectl get machineconfigurations -o yaml > logs/machineconfigurations-full.yaml 2>&1 || true
272-
kubectl get machineconfigurationversions -o wide > logs/machineconfigurationversions.txt 2>&1 || true
273-
kubectl get machineconfigurationversions -o yaml > logs/machineconfigurationversions-full.yaml 2>&1 || true
274-
kubectl get machineoperations -o wide > logs/machineoperations.txt 2>&1 || true
275-
kubectl get machineoperations -o yaml > logs/machineoperations-full.yaml 2>&1 || true
276-
# Collect kubelet logs from the Kind control-plane
277-
docker exec kind-control-plane journalctl -u kubelet --no-pager -l > logs/kind-kubelet.log 2>&1 || true
278-
# Collect kube-apiserver logs (critical for diagnosing TLS bootstrap / RBAC issues)
279-
docker exec kind-control-plane crictl logs $(docker exec kind-control-plane crictl ps -a --name kube-apiserver -q 2>/dev/null | head -1) > logs/kube-apiserver.log 2>&1 || true
280-
# Dump all ClusterRoleBindings to see what RBAC kubeadm/Kind created
281-
kubectl get clusterrolebindings -o wide > logs/clusterrolebindings.txt 2>&1 || true
282-
kubectl get clusterrolebindings -o yaml > logs/clusterrolebindings-full.yaml 2>&1 || true
283-
# List CSRs to see if the kubelet attempted TLS bootstrap
284-
kubectl get csr -o wide > logs/csrs.txt 2>&1 || true
285-
kubectl describe csr > logs/csrs-describe.txt 2>&1 || true
286-
# Dump bootstrap token secrets (redact token-secret values)
287-
kubectl get secrets -n kube-system -l 'kubernetes.io/legacy-token-last-used' -o wide > logs/bootstrap-tokens.txt 2>&1 || true
288-
kubectl get secrets -n kube-system --field-selector type=bootstrap.kubernetes.io/token -o yaml > logs/bootstrap-token-secrets.yaml 2>&1 || true
289-
# Collect workload test pod details
290-
kubectl describe pods -n e2e-workload-test > logs/workload-pods-describe.txt 2>&1 || true
291-
kubectl logs -n e2e-workload-test --all-containers --prefix e2e-hello > logs/workload-hello.log 2>&1 || true
292-
kubectl logs -n e2e-workload-test --all-containers --prefix e2e-dns-test > logs/workload-dns.log 2>&1 || true
293-
294-
- name: Upload logs
295-
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
142+
run: python3 ./hack/agent/e2e-kind/e2e.py --verbose cleanup
143+
144+
agent-config-e2e:
145+
name: agent config e2e
146+
runs-on: ubuntu-24.04
147+
timeout-minutes: 60
148+
env:
149+
KIND_CLUSTER_NAME: agent-config-e2e
150+
VM_NAME: agent-config-e2e
151+
VM_SUBNET: "192.168.110"
152+
VM_IP: "192.168.110.10"
153+
AGENT_MACHINE_NAME: agent-config-e2e
154+
steps:
155+
- name: Checkout
156+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
157+
158+
- name: Set up test control plane
159+
uses: ./.github/actions/agent-e2e-kind-control-plane
160+
with:
161+
cluster-name: ${{ env.KIND_CLUSTER_NAME }}
162+
vm-subnet: ${{ env.VM_SUBNET }}
163+
create-vm: "false"
164+
165+
- name: Set up machina resources
166+
uses: ./.github/actions/agent-e2e-machina-setup
167+
168+
- name: Discover and validate node configs
169+
run: python3 ./hack/agent/e2e-kind/e2e.py --verbose validate-node-configs
170+
171+
- name: Collect logs
296172
if: always()
173+
uses: ./.github/actions/agent-e2e-kind-logs
297174
with:
298-
name: agent-e2e-kind-logs
299-
path: logs/
300-
retention-days: 30
175+
artifact-name: agent-config-e2e-logs
176+
node-configs: "true"
301177

302178
- name: Cleanup
303179
if: always()

0 commit comments

Comments
 (0)