Change the way we deal with scenario state within `Traces` for `Checks` `Interactions` and their attributes

[linear]

Current State

- Interaction (core/trace.py:10): stores inputs, outputs, metadata but does not store interactionspec, or checks
- Trace.with_interaction(spec) (core/trace.py:134): calls spec.generate(trace) but discards the interactionspec object after so we can't use this info in subsequent interactions/checks
- Check.run(trace) (core/check.py): returns CheckResult — no inputs tracked, not stored in trace
- ScenarioRunner (scenarios/runner.py): stores check results in TestCaseResult, but these are disconnected from Trace
- JSONPath resolution (core/extraction.py:78): uses trace.model_dump() — will automatically expose any new fields added to Interaction

[linear]

OSS-16

[davidberenstein1957]

I feel this is also a bit related to #2267, and the general vibe of lack of controll and vieww over the state during a scenario.

[henchaves]

Overall, I agree with this lack of control and view for most of the classes you mentioned. To be defined what we should refactor after #2269 is merged

[debu-sinha]

Now that the Interact API (#2270) has landed, this seems like the natural time to revisit this.

I've been working on trace-level evaluation in another framework and ran into the same disconnect: check results and the trace they evaluate end up in separate structures, so you lose the ability to ask "what was evaluated, and what did we think about it?" from a single object.

One thing I'm curious about: with the annotations dict added in #2268, is the plan to use that as the place where check results get stored on the trace? That would make the trace a single source of truth for both execution and evaluation, and JSONPath resolution would pick up the results automatically. Or is there a separate mechanism being considered?

[henchaves]

Now that the Interact API (#2270) has landed, this seems like the natural time to revisit this.

I've been working on trace-level evaluation in another framework and ran into the same disconnect: check results and the trace they evaluate end up in separate structures, so you lose the ability to ask "what was evaluated, and what did we think about it?" from a single object.

One thing I'm curious about: with the annotations dict added in #2268, is the plan to use that as the place where check results get stored on the trace? That would make the trace a single source of truth for both execution and evaluation, and JSONPath resolution would pick up the results automatically. Or is there a separate mechanism being considered?

Hi @debu-sinha,
Thanks for commenting on this. Would you be available for a quick call so we can better understand your pain points with other eval frameworks? If so, please let me know whether you're on our Discord channel and what your username is, so I can contact you directly there.

Regarding your question about annotations, they were introduced primarily to make it possible to pass static values to a check (e.g., annotations = {"threshold": 0.8}), and inside Check.run it would use trace.annotations.threshold to decide whether the check should pass. However, we're open to discussing better ways to link checks with the interactions they run on.

[debu-sinha]

@henchaves just joined Discord, username is debusinha. I'm traveling next week (Tue-Thu) but free Friday onward. Happy to find a time that works.