Skip to content

tester agent JSON schema validation always fails in stream-json mode #458

Description

@jmadore-payfacto

Summary

When using the conductor-bootstrap config (which triggers debug-workflow), the tester agent consistently fails with JSON schema validation errors even though the agent completes successfully and the underlying work is correct.

Error

Error: Agent tester output failed JSON schema validation:
  #/required must have required property 'approved'
  #/required must have required property 'summary'
  #/required must have required property 'commandResult'
  #/required must have required property 'rootCauseVerification'
  #/required must have required property 'testVerification'

Root Cause

The tester agent in debug-workflow.json runs with stream-json output format (for live log visibility). In this mode, zeroshot validates the structured_output field of the final stream-json result against the agent's output schema after the fact.

However, because --json-schema is silently ignored when outputFormat !== 'json' (see runner.jsresolveJsonSchema), Claude is never told about the required schema shape. Claude produces its standard stream-json structured_output with {"summary": "...", "result": "..."} instead of the schema-required fields (approved, commandResult, rootCauseVerification, testVerification).

The result: schema validation always fails for the tester agent, causing the cluster to reject valid work and retry indefinitely.

Evidence

From task log, Claude's structured_output:

{
  "summary": "Fix already applied in iteration 1; tester rejection was tester agent crash, not code issue",
  "result": "ROOT CAUSE FIX: ..."
}

The agent itself noted: "The previous rejection was due to the tester agent crashing with JSON schema validation errors — the code fix was correct and already applied."

Expected Behavior

Either:

  1. The tester agent should use strictSchema: true in debug-workflow.json so --output-format json --json-schema <schema> is passed to Claude, enforcing the schema natively, or
  2. When a jsonSchema is defined on an agent, the system prompt should inject the schema shape so Claude knows to produce it even in stream-json mode

Config Used

conductor-bootstrap (out of the box, no customization)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions