Skip to content

api: "Failed to place sandbox" gives no diagnostic - expose why no node accepted the sandbox #3553

Description

@AdaAibaby

Problem

When sandbox creation fails because no node can host it, the client receives a generic 500 error:

This tells the user nothing about why placement failed. The actual cause is buried in server logs as an internal error and is never surfaced. Common root causes include:

  • CPU model mismatch: the template was built on a newer CPU generation (e.g. Emerald Rapids, model 207) but all cluster nodes run an older generation (e.g. Ice Lake, model 106). Cross-generation placement is asymmetric — an n2 build can resume on an n4 node, but not the reverse.
  • Label mismatch: the team or template requires scheduling labels (e.g. gpu, fast-disk) that no available node carries.
  • All nodes not-accepting: nodes are draining, unhealthy, or saturated.

Without seeing which filter eliminated all nodes it is impossible to self-diagnose or file a useful bug report.

Root cause

BestOfK.chooseNode returns a FailedToPlaceSandboxError whose Error() string includes machine=... and labels=... constraints, but create_instance.go stores that in the internal Err field and returns a hardcoded ClientMsg: "Failed to place sandbox" that never reaches the client.

Proposed fix

  1. Track per-filter rejection counts inside BestOfK.sample() (not-accepting, cpu-incompatible, label-filtered, excluded).
  2. Embed those counts plus the build CPU constraints in FailedToPlaceSandboxError.Error().
  3. Propagate err.Error() to ClientMsg in create_instance.go.

Example new error message:

Failed to place sandbox: no compatible node found (38 nodes checked: 0 not-accepting, 38 cpu-incompatible, 0 label-filtered, 0 excluded); build cpu: arch=x86_64 family=6 model=207

This lets the caller immediately identify a CPU-generation mismatch or label gap without digging through server logs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions