Optional deterministic, unit-testable ground truth for rule-shaped scenarios (Tln)

[OpakAlex]

Motivation

AssetOpsBench scores trajectories with an LLM judge (Llama-4-Maverick-17B, 6 dimensions). That's the right tool for open-ended answers, but a large share of the 460+ scenarios have acceptance criteria that are really rules over the returned facts — "each recommended work order carries a primary failure code", "route this condition to the right failure mode", "forecast horizon ≥ 24h". For those, an LLM judge is non-deterministic, un-unit-testable, costs tokens per evaluation, and can't explain why it scored as it did.

The ground-truth schema already splits deterministic: true from deterministic: false + characteristic_form. This proposes an optional, additive ground-truth check type for the rule-expressible cases, backed by Tln — a deterministic, unit-testable rule/workflow language. It sits alongside the existing JSON ground truth and LLM judge; nothing is removed and the Python agents are untouched.

What it looks like

Their example utterance (deterministic: false today, judge-scored):

"When an anomaly happens for equipment CWC04009, can you recommend top three work orders?"

The criterion — three ranked WOs, each with a primary failure code — is a rule, tested hermetically with mocked tools (no model, no cluster):

detect "Anomaly needs a work-order recommendation" {
  for records where type == "anomaly" and attr "equipment" != ""
  flag matching items
  remediate { tool "wo" "recommend" { equipment attr "equipment"  top_n 3 } }
}

test "Anomaly on CWC04009 asks for the top-3 recommendations" {
  given { record 1 type "anomaly"  attr 1 "equipment" "CWC04009" }
  mock tool "wo" "recommend" { returns { count 3 } }
  when detect "Anomaly needs a work-order recommendation"
  expect {
    flagged 1
    tool_called "wo" "recommend" with { equipment == "CWC04009"  top_n == 3 }
  }
}

FMSR failure-mode routing is kNN classification, not prose, so it's a better ground truth as a deterministic classifier than a judge:

classify "Failure mode" {
  for records where type == "incident" and status == "open"
  features [attr "vibration", attr "temp"]
  trained_on records where type == "incident" and status == "resolved"
  label_attr "root_cause"
  confidence >= 0.8
}

tln explain prints exactly which records fired and why, so a failing scenario is debuggable — unlike a judge score.

Mapping

AssetOpsBench	Tln
execution_steps + execution_links	workflow (step/depends_on, topo-sorted, cycle-checked)
deterministic: true ground truth	detect/rule + .tln.test with mock tool / tool_called … with
deterministic: false + characteristic_form (judge)	detect rule over response facts — deterministic + explainable
FMSR routing	classify with a confidence >= gate

Non-goals

• Not replacing natural-language utterances, the execution DAG, or the LLM judge for genuinely prose-shaped answers.
• Not mandatory; opt-in per scenario.
• Pure retrieval utterances (e.g. "What IoT sites are available?") stay as simple structural checks — Tln earns its place only when the criterion is a rule or a classifier.

Cost, honestly

Tln ships as a single static Go binary (tln test <policy> <test>); the evaluator would shell out to it — a real dependency to weigh in a Python repo, which is why this is proposed as optional. Tln is open-source; license is compatible with AssetOpsBench's Apache-2.0.

Offer

Happy to open a PR with a small demo folder: an optional evaluator plus 3–5 existing scenarios (the two above + a couple more WO/FMSR cases) converted as reference, and a short docs/guideline/tln_ground_truth.md on when to use it vs. the judge. Would the maintainers be open to that?