Skip to content

Commit c5f1498

Browse files
authored
refactor(rubric): parse rubric at build time, eliminate _rubric_id context key (#84)
- Move Rubric parsing from runtime to build time (DSL, deserializer). Node.getRubric() now returns typed Rubric instead of raw String. RubricEngine.evaluate() takes Rubric directly — no repository lookup, no _rubric_id context-key round-trip. - Keep RubricRepository as placeholder for future DB-backed store. Delete 58 trivial/tautological tests (record properties, HashMap wrappers). Resolve: #74 Signed-off-by: Aleksandr Suvorov <asuvorov@hensu.io>
1 parent 737bc22 commit c5f1498

73 files changed

Lines changed: 451 additions & 1973 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

README.md

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -112,8 +112,6 @@ fun contentPipeline() = workflow("content-pipeline") {
112112
agent("reviewer") { role = "Content Reviewer"; model = Models.GEMINI_3_1_PRO }
113113
}
114114

115-
rubrics { rubric("content-quality", "content-quality.md") }
116-
117115
state {
118116
input("topic", VarType.STRING)
119117
variable("draft", VarType.STRING, "the full written article text")
@@ -126,7 +124,7 @@ fun contentPipeline() = workflow("content-pipeline") {
126124
agent = "writer"
127125
prompt = "Write a short article about {topic}. {recommendation}"
128126
writes("draft")
129-
rubric = "content-quality"
127+
rubric = "content-quality.md"
130128
onScore {
131129
whenScore lessThan 70.0 goto "write" // score too low – retry
132130
}

docs/developer-guide-core.md

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -182,7 +182,7 @@ map keyed by node ID. For `StandardNode`s with `writes` declared, it routes the
182182
- **Single write**: attempts to parse the response as JSON and extract the declared key; falls back to the full raw text if the key is absent or the output is not valid JSON.
183183
- **Multiple writes**: the response is parsed as JSON and each declared key is extracted into context.
184184

185-
**`RubricPostProcessor`** — If a node has a `rubricId`, this processor evaluates the output against the rubric's
185+
**`RubricPostProcessor`** — If a node has a `Rubric`, this processor evaluates the output against the rubric's
186186
criteria using the `RubricEngine`. It stores the evaluation result in the state for use by `ScoreTransition` rules. If
187187
the evaluation fails and no explicit transition handles the low score, it can trigger an **auto-backtrack** to a prior
188188
step, enabling self-correcting loops. On auto-backtrack, sets `state.nodeRedirected = true` — downstream processors
@@ -710,15 +710,16 @@ The rubric engine evaluates output quality against defined criteria with score-b
710710

711711
| Component | Description |
712712
|----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------|
713-
| `RubricEngine` | Orchestrates evaluation using repository and evaluator |
714-
| `RubricRepository` | Stores rubric definitions (in-memory by default) |
713+
| `RubricEngine` | Orchestrates evaluation using evaluator |
715714
| `RubricEvaluator` | Evaluates output against criteria |
716715
| `ScoreExtractingEvaluator` | Reads the `score` engine variable from context; accumulates `recommendation` feedback for failing criteria into `_rubric_criterion_feedback` |
717716
| `Rubric` | Immutable definition with pass threshold and weighted criteria |
718717
| `Criterion` | Single evaluation dimension with weight and minimum score |
719718

720719
### How Evaluation Works
721720

721+
Rubrics are parsed at build time and stored directly on the node as typed `Rubric` objects.
722+
722723
`ScoreExtractingEvaluator` reads the `score` engine variable directly from the execution context. The score is extracted automatically by `OutputExtractionPostProcessor` whenever the node has a `ScoreTransition` — no JSON parsing is needed in the evaluator itself.
723724

724725
If the score falls below a criterion's minimum and a `recommendation` engine variable is present in context, the text is appended to `_rubric_criterion_feedback`. `RubricPostProcessor` uses that list to assemble a combined backtrack context update for self-correcting loops.
@@ -861,7 +862,7 @@ Each injector fires when **either** of two conditions is met:
861862

862863
```mermaid
863864
flowchart LR
864-
r(["RubricPromptInjector\n· rubricId != null"]) --> s(["ScoreVariableInjector\n· ScoreTransition or\nconsensus branch"])
865+
r(["RubricPromptInjector\n· rubric != null"]) --> s(["ScoreVariableInjector\n· ScoreTransition or\nconsensus branch"])
865866
s --> a(["ApprovalVariableInjector\n· ApprovalTransition or\nconsensus branch"])
866867
a --> rec(["RecommendationVariable\nInjector\n· Score/Approval or\nconsensus branch"])
867868
rec --> w(["WritesVariableInjector\n· has writes()"])

docs/developer-guide-server.md

Lines changed: 4 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -970,7 +970,6 @@ Every integration test extends `IntegrationTestBase`, which provides:
970970
| `pushAndExecuteWithMcp(workflow, ctx, endpoint)` | Executes with MCP-enabled tenant context |
971971
| `registerStub(key, response)` | Programmatic stub registration by node ID or agent ID |
972972
| `registerStub(scenario, key, response)` | Scenario-specific stub registration |
973-
| `resolveRubricPath(resourceName)` | Copies classpath rubric to temp file for `RubricParser` |
974973

975974
#### Writing an Integration Test
976975

@@ -1098,28 +1097,10 @@ Place workflow definitions in `src/test/resources/workflows/`. Use `model: "stub
10981097

10991098
#### Rubric Testing
11001099

1101-
Pre-register parsed rubrics so the executor skips filesystem path resolution:
1102-
1103-
```java
1104-
private Rubric parseAndRegisterRubric(String rubricId, String resourceName) {
1105-
String rubricPath = resolveRubricPath(resourceName);
1106-
Rubric parsed = RubricParser.parse(Path.of(rubricPath));
1107-
1108-
Rubric rubric = Rubric.builder()
1109-
.id(rubricId)
1110-
.name(parsed.getName())
1111-
.version(parsed.getVersion())
1112-
.type(parsed.getType())
1113-
.passThreshold(parsed.getPassThreshold())
1114-
.criteria(parsed.getCriteria())
1115-
.build();
1116-
1117-
hensuEnvironment.getRubricRepository().save(rubric);
1118-
return rubric;
1119-
}
1120-
```
1121-
1122-
Place rubric markdown files in `src/test/resources/rubrics/`.
1100+
Rubrics are parsed at build time and stored directly on workflow nodes. Workflow JSON fixtures
1101+
include inline rubric content in the `"rubric"` field, which the deserializer parses into typed
1102+
`Rubric` objects at load time. No separate registration step is needed — just call
1103+
`loadWorkflow("fixture.json")` and the rubric is ready.
11231104

11241105
#### Repository Tests (Testcontainers)
11251106

docs/dsl-reference.md

Lines changed: 18 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -51,10 +51,6 @@ fun myWorkflow() = workflow("WorkflowName") {
5151
// Agent definitions
5252
}
5353

54-
rubrics {
55-
// Rubric references (optional)
56-
}
57-
5854
config {
5955
// Execution settings (optional)
6056
}
@@ -78,7 +74,6 @@ fun myWorkflow() = workflow("WorkflowName") {
7874
|---------------|--------------------------------------------------------------------------------------------------------------------------|
7975
| `state { }` | Optional typed state schema. When declared, enables load-time validation of `writes` and `{variable}` prompt references. |
8076
| `agents { }` | Agent definitions (models, roles, temperatures) |
81-
| `rubrics { }` | Rubric file references for quality evaluation |
8277
| `config { }` | Workflow execution settings |
8378
| `graph { }` | Node graph (required) |
8479

@@ -155,7 +150,7 @@ Standard nodes execute an agent with a prompt and transition based on the result
155150
node("node-id") {
156151
agent = "agent-id"
157152
prompt = "Your prompt with {placeholders}"
158-
rubric = "rubric-id" // Optional
153+
rubric = "rubric-id.md" // Optional
159154
writes("param1", "param2") // Optional — declare state variables this node produces
160155

161156
review(ReviewMode.OPTIONAL) // Optional
@@ -169,11 +164,11 @@ node("node-id") {
169164

170165
#### Standard Node Properties
171166

172-
| Property | Type | Required | Description |
173-
|-------------|---------|----------|-----------------------------------------|
174-
| `agent` | String? | Yes | ID of the agent to execute |
175-
| `prompt` | String? | Yes | Prompt template or `.md` file reference |
176-
| `rubric` | String? | No | ID of rubric to evaluate output quality |
167+
| Property | Type | Required | Description |
168+
|----------|---------|----------|----------------------------------------------------------------|
169+
| `agent` | String? | Yes | ID of the agent to execute |
170+
| `prompt` | String? | Yes | Inline prompt template or `.md` file reference from `prompts/` |
171+
| `rubric` | String? | No | Inline rubric content or `.md` file reference from `rubrics/` |
177172

178173
#### Standard Node Functions
179174

@@ -219,13 +214,13 @@ parallel("review-committee") {
219214

220215
#### Branch Properties
221216

222-
| Property | Type | Required | Description |
223-
|------------|---------------|----------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
224-
| `agent` | String | Yes | ID of the agent to execute |
225-
| `prompt` | String? | No | Prompt template or `.md` file reference |
226-
| `rubric` | String? | No | ID of rubric for branch evaluation. When set, the rubric's pass/fail result determines the branch's consensus vote (APPROVE/REJECT), overriding text-based heuristics |
227-
| `weight` | Double | No | Vote weight for `WEIGHTED_VOTE` consensus strategy. Higher values give more influence to the branch score (default: 1.0) |
228-
| `yields()` | vararg String | No | State variable names this branch produces as structured domain output. The agent's JSON response must include these fields; the engine extracts and merges them into workflow state |
217+
| Property | Type | Required | Description |
218+
|------------|---------------|----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
219+
| `agent` | String | Yes | ID of the agent to execute |
220+
| `prompt` | String? | No | Inline prompt template or `.md` file reference from `prompts/` |
221+
| `rubric` | String? | No | Inline rubric content or `.md` file reference from `rubrics/`. When set, the rubric's pass/fail result determines the branch's consensus vote (APPROVE/REJECT), overriding text-based heuristics |
222+
| `weight` | Double | No | Vote weight for `WEIGHTED_VOTE` consensus strategy. Higher values give more influence to the branch score (default: 1.0) |
223+
| `yields()` | vararg String | No | State variable names this branch produces as structured domain output. The agent's JSON response must include these fields; the engine extracts and merges them into workflow state |
229224

230225
#### Rubric-Based Consensus
231226

@@ -335,7 +330,7 @@ generic("validate-input") {
335330
"required" to true
336331
}
337332

338-
rubric = "validation-rubric" // Optional
333+
rubric = "validation-rubric.md" // Optional
339334

340335
onSuccess goto "process"
341336
onFailure retry 2 otherwise "error"
@@ -725,7 +720,7 @@ Because `recommendation` is an engine variable, you reference it as a `{placehol
725720
node("score-content") {
726721
agent = "reviewer"
727722
prompt = "Review the article: {article}\nOutput a score and feedback."
728-
rubric = "content-quality"
723+
rubric = "content-quality.md"
729724

730725
onScore {
731726
whenScore greaterThanOrEqual 80.0 goto "publish"
@@ -746,24 +741,15 @@ node("revise") {
746741
```
747742

748743
## Rubrics
744+
Rubrics define quality evaluation criteria for node outputs. Rubric files should be placed in the `rubrics/` directory of your working directory.
749745

750-
Rubrics define quality evaluation criteria for node outputs.
751-
752-
```kotlin
753-
rubrics {
754-
rubric("quality-check", "quality.md") // rubrics/quality.md
755-
rubric("pr-review", "templates/pr.md") // rubrics/templates/pr.md
756-
rubric("docs") { file = "documentation.md" } // Alternative syntax
757-
}
758-
```
759-
760-
Reference rubrics in nodes:
746+
To use a rubric, specify the filename in the node's `rubric` property:
761747

762748
```kotlin
763749
node("review") {
764750
agent = "reviewer"
765751
prompt = "Review this code"
766-
rubric = "quality-check"
752+
rubric = "content-quality.md" // References rubrics/content-quality.md
767753

768754
onScore {
769755
whenScore greaterThanOrEqual 80.0 goto "approve"
@@ -1094,10 +1080,6 @@ fun contentPipeline() = workflow("ContentPipeline") {
10941080
}
10951081
}
10961082

1097-
rubrics {
1098-
rubric("content-quality", "content-quality.md")
1099-
}
1100-
11011083
graph {
11021084
start at "research"
11031085

docs/unified-architecture.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -760,7 +760,7 @@ CDI wiring, `WorkflowExecutor`, `TenantContext` — against in-memory repositori
760760
profile disables PostgreSQL, Flyway, and the scheduler (no Docker required).
761761

762762
All integration tests extend `IntegrationTestBase`, which provides CDI injection, per-test
763-
state cleanup, and helpers (`registerStub`, `pushAndExecute`, `resolveRubricPath`).
763+
state cleanup, and helpers (`registerStub`, `pushAndExecute`).
764764

765765
### Repository Tests (Testcontainers PostgreSQL)
766766

hensu-cli/src/main/java/io/hensu/cli/review/ReviewTerminal.java

Lines changed: 6 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -190,20 +190,18 @@ private void displayHelp() {
190190
private ReviewDecision handleBacktrack(ReviewData data) {
191191
List<ReviewData.StepInfo> steps = data.historySteps();
192192

193-
if (steps == null || steps.size() <= 1) {
193+
if (steps == null || steps.isEmpty()) {
194194
println(styles.warn("No previous steps available to backtrack to."));
195195
return null;
196196
}
197197

198-
List<ReviewData.StepInfo> validSteps = steps.subList(0, steps.size() - 1);
199-
200198
println("");
201199
println(styles.boxTopWithLabel(styles.dim("backtrack target")));
202200
println("");
203201

204-
for (int i = validSteps.size() - 1; i >= 0; i--) {
205-
ReviewData.StepInfo step = validSteps.get(i);
206-
int displayNum = validSteps.size() - i;
202+
for (int i = steps.size() - 1; i >= 0; i--) {
203+
ReviewData.StepInfo step = steps.get(i);
204+
int displayNum = steps.size() - i;
207205
boolean ok = "SUCCESS".equals(step.status());
208206
String status = styles.successOrError(ok ? "OK" : "FAIL", ok);
209207
println(String.format(" [%d] %s (%s)", displayNum, step.nodeId(), status));
@@ -217,12 +215,12 @@ private ReviewDecision handleBacktrack(ReviewData data) {
217215
try {
218216
int choice = Integer.parseInt(input);
219217
if (choice == 0) return null;
220-
if (choice < 1 || choice > validSteps.size()) {
218+
if (choice < 1 || choice > steps.size()) {
221219
println(styles.error("Invalid choice. Please select a number from the list."));
222220
return null;
223221
}
224222

225-
ReviewData.StepInfo targetStep = validSteps.get(validSteps.size() - choice);
223+
ReviewData.StepInfo targetStep = steps.get(steps.size() - choice);
226224

227225
print("Reason for backtracking (optional): ");
228226
String reason = readInput();

hensu-cli/src/main/java/io/hensu/cli/visualizer/TextVisualizationFormat.java

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -120,11 +120,14 @@ private String renderNode(Node node, String nodeId, String indent, AnsiStyles st
120120
"agent",
121121
styles.bold(standardNode.getAgentId())));
122122
}
123-
if (standardNode.getRubricId() != null) {
123+
if (standardNode.getRubric() != null) {
124124
sb.append(
125125
String.format(
126126
"%s%s %-9s %s%n",
127-
indent, styles.boxMid(), "rubric", standardNode.getRubricId()));
127+
indent,
128+
styles.boxMid(),
129+
"rubric",
130+
standardNode.getRubric().getCriteria().size() + " criteria"));
128131
}
129132
if (standardNode.getReviewConfig() != null) {
130133
sb.append(
@@ -271,11 +274,14 @@ private String renderNode(Node node, String nodeId, String indent, AnsiStyles st
271274
styles.accent(String.valueOf(genericNode.getConfig().size()))
272275
+ " entries"));
273276
}
274-
if (genericNode.getRubricId() != null) {
277+
if (genericNode.getRubric() != null) {
275278
sb.append(
276279
String.format(
277280
"%s%s %-9s %s%n",
278-
indent, styles.boxMid(), "rubric", genericNode.getRubricId()));
281+
indent,
282+
styles.boxMid(),
283+
"rubric",
284+
genericNode.getRubric().getCriteria().size() + " criteria"));
279285
}
280286
appendTransitions(sb, indent, genericNode.getTransitionRules(), styles);
281287
}

0 commit comments

Comments
 (0)