Skip to content

Add agent e2e node config scenarios#170

Merged
bcho merged 16 commits into
mainfrom
copilot/add-e2e-test-setup-nodes
May 22, 2026
Merged

Add agent e2e node config scenarios#170
bcho merged 16 commits into
mainfrom
copilot/add-e2e-test-setup-nodes

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 20, 2026

The agent e2e coverage only exercised one node bootstrap shape. This adds minimal node config e2e scenarios so CI can validate node join, workload scheduling, and repave with alternate kubelet labels, taints, and node IP settings.

  • E2E config variants

    • Added file-based node config scenarios under hack/agent/e2e-kind/node-configs/.
    • Added --node-config support to the e2e harness so scenarios can pass configured labels, taints, and node IP through manual-bootstrap.
  • Validation

    • Added validate-node-config to assert expected labels, taints, and Node internal IP on the Kubernetes Node.
    • Extended Machine CR validation to confirm self-registered config matches the bootstrap inputs.
    • Preserves configured labels through repave validation.
  • CI coverage

    • Keeps the existing full agent-e2e job as a single default run.
    • Adds a separate agent-config-e2e job with distinct Kind/VM settings for node config scenarios.
    • Discovers node config scenario files from hack/agent/e2e-kind/node-configs/ and validates them in parallel against the same Kind control plane.
    • Extracts shared Kind control-plane and VM bridge setup into a reusable workflow action.
    • Extracts shared machina CRD/controller setup into a reusable workflow action.
    • Extracts shared log collection and upload into a reusable workflow action.
    • Includes labels-and-taints and node-ip scenarios.

Example variant:

{
  "name": "labels-and-taints",
  "nodeLabels": {
    "e2e.unbounded-cloud.io/config": "labels-and-taints"
  },
  "registerWithTaints": [
    "e2e.unbounded-cloud.io/dedicated=agent:NoSchedule"
  ]
}

Agent-Logs-Url: https://github.com/Azure/unbounded/sessions/95afa82b-ea29-442e-810d-674a5d2980d3

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>
Copilot AI changed the title [WIP] Add minimal e2e test setup for creating nodes with different settings Add agent e2e node config matrix May 20, 2026
Copilot finished work on behalf of bcho May 20, 2026 17:23
Copilot AI requested a review from bcho May 20, 2026 17:23
Copy link
Copy Markdown
Member

@bcho bcho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread .github/workflows/agent-e2e-kind.yaml
Comment thread .github/workflows/agent-e2e-kind.yaml Outdated
Agent-Logs-Url: https://github.com/Azure/unbounded/sessions/bf7223a8-8101-4d8e-94c4-6f60a177756b

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>
Copilot finished work on behalf of bcho May 20, 2026 17:52
Copilot AI requested a review from bcho May 20, 2026 17:52
Copy link
Copy Markdown
Member

@bcho bcho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread hack/agent/e2e-kind/node-configs/labels-and-taints.json
Comment thread .github/workflows/agent-e2e-kind.yaml Outdated
Comment thread hack/agent/e2e-kind/e2e.py Outdated
Comment thread hack/agent/e2e-kind/e2e.py Outdated
Copy link
Copy Markdown
Member

@bcho bcho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread .github/workflows/agent-e2e-kind.yaml Outdated
Copilot finished work on behalf of bcho May 20, 2026 18:05
Copilot AI requested a review from bcho May 20, 2026 18:05
Copilot AI changed the title Add agent e2e node config matrix Add agent e2e node config scenarios May 20, 2026
Copilot finished work on behalf of bcho May 20, 2026 18:12
Comment thread .github/workflows/agent-e2e-kind.yaml Outdated
Comment thread .github/workflows/agent-e2e-kind.yaml Outdated
Comment thread hack/agent/e2e-kind/node-configs/labels-and-taints.json
Copy link
Copy Markdown
Contributor Author

Copilot AI commented May 20, 2026

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • 192.0.2.1
    • Triggering command: /tmp/go-build1326072227/b1340/controller.test /tmp/go-build1326072227/b1340/controller.test -test.testlogfile=/tmp/go-build1326072227/b1340/testlog.txt -test.paniconexit0 -test.timeout=10m0s -o 6072227/b876/_pk-errorsas portcfg ux-amd64/pkg/tool/linux_amd64/vet -p roller-runtime/p-atomic mpile ux-amd64/pkg/too-buildtags -o -tools@v0.4.8/si-errorsas mpile ux-amd64/pkg/tool/linux_amd64/vet -p /cobra t ux-amd64/pkg/tooorigin (packet block)
  • invalid-ip
    • Triggering command: /tmp/go-build1326072227/b1340/controller.test /tmp/go-build1326072227/b1340/controller.test -test.testlogfile=/tmp/go-build1326072227/b1340/testlog.txt -test.paniconexit0 -test.timeout=10m0s -o 6072227/b876/_pk-errorsas portcfg ux-amd64/pkg/tool/linux_amd64/vet -p roller-runtime/p-atomic mpile ux-amd64/pkg/too-buildtags -o -tools@v0.4.8/si-errorsas mpile ux-amd64/pkg/tool/linux_amd64/vet -p /cobra t ux-amd64/pkg/tooorigin (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot AI requested a review from bcho May 20, 2026 18:38
Copilot finished work on behalf of bcho May 20, 2026 18:38
Copy link
Copy Markdown
Member

@bcho bcho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

e2e test is failing

Copy link
Copy Markdown
Member

@bcho bcho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor Author

Copilot AI commented May 21, 2026

e2e test is failing

Fixed in f55274c. The config e2e failure showed both parallel VM nodes stuck NotReady because kindnet was CrashLoopBackOff; the parallel QEMU VMs were using the default duplicate NIC MAC on the shared bridge. The e2e launcher now assigns a stable unique MAC per VM IP.

Copilot finished work on behalf of bcho May 21, 2026 05:05
Copilot AI requested a review from bcho May 21, 2026 05:05
Comment thread hack/agent/e2e-kind/e2e.py Dismissed
Copy link
Copy Markdown
Contributor Author

Copilot AI commented May 21, 2026

@copilot

I checked the current runs. The latest e2e run for f55274c is action_required with no jobs created, so it needs workflow approval/re-run before the fix can be validated. The still-running e2e run is for older commit 0f4fb7d and does not include the MAC fix.

Copilot finished work on behalf of bcho May 21, 2026 05:06
Copy link
Copy Markdown
Member

@bcho bcho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor Author

Copilot AI commented May 21, 2026

https://github.com/Azure/unbounded/actions/runs/26206569938/job/77108680919?pr=170 it is still failing @copilot

Fixed in d4845d8. The parallel config scenarios were still sharing the same site-scoped bootstrap token lookup, and repave was looking for per-node MachineConfigurations that the shared setup had not created. The scenarios now use isolated e2e site names and create the per-node MachineConfiguration when needed before repave validation.

Copilot finished work on behalf of bcho May 21, 2026 07:22
Copilot AI requested a review from bcho May 21, 2026 07:22
@bcho bcho marked this pull request as ready for review May 21, 2026 21:22
@bcho bcho requested a review from a team May 21, 2026 21:22
@bcho bcho enabled auto-merge (squash) May 21, 2026 21:23
@bcho bcho merged commit eea38cb into main May 22, 2026
25 checks passed
@bcho bcho deleted the copilot/add-e2e-test-setup-nodes branch May 22, 2026 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[agent] Add minimal e2e test setup for creating node with different settings

3 participants