Evaluation Framework Enhancement

Evaluation Framework Enhancement - Immediate Action Items

Current Status

✅ What We Have

• Unit Tests: 45 comprehensive unit tests for all ported tools (all passing)
• Basic Eval Tests: 34 eval tests checking tool triggering + response content
  • test_user_data_tool_eval.py (8 tests)
  • test_agent_tool_eval.py (9 tests)
  • test_rag_tool_eval.py (8 tests)
  • test_knowledge_tool_eval.py (9 tests)
• Pattern Compliance: Tests use airunner_client_function_scope, mock external dependencies, check response quality

❌ Missing (Based on LangSmith Guide)

1. Trajectory Evaluation - Not tracking path through nodes/tools
2. Single-Step Evaluation - Not testing individual components in isolation
3. Mode-Based Testing - No mode-specific eval datasets
4. Trajectory Datasets - No expected_trajectory in test data

Immediate Enhancements Needed

1. Add Trajectory Tracking to Existing Eval Tests

Current Test Pattern:

def test_store_user_data_basic(self, ..., airunner_client_function_scope):
    response = airunner_client_function_scope.generate(
        prompt=prompt,
        max_tokens=300,
        tool_categories=["USER_DATA"],
    )
    # Only checks: mock calls + response content
    assert mock_user.get_or_create.called
    assert "stored" in response.get("text", "").lower()

Enhanced Pattern (LangSmith Style):

def test_store_user_data_basic(self, ..., airunner_client_function_scope):
    # Expected trajectory
    expected_trajectory = ["model", "tools", "store_user_data", "model"]
    
    # Track actual trajectory during execution
    trajectory = []
    async for event in airunner_client_function_scope.astream_events(
        prompt=prompt,
        max_tokens=300,
        tool_categories=["USER_DATA"],
    ):
        if event['type'] == 'task':
            trajectory.append(event['payload']['name'])
        if event['type'] == 'tool_call':
            trajectory.append(event['payload']['tool_name'])
    
    # Evaluate trajectory correctness
    assert all(exp in trajectory for exp in expected_trajectory)
    
    # Evaluate response quality (existing checks)
    assert "stored" in response.get("text", "").lower()

2. Create Trajectory Evaluation Datasets

Add to Each Eval Test:

examples = [
    {
        "inputs": {"question": "Remember that my favorite color is blue"},
        "outputs": {
            "response": "I've stored that your favorite color is blue",
            "trajectory": ["model", "tools", "store_user_data", "model"],
        }
    },
    {
        "inputs": {"question": "What's my email address?"},
        "outputs": {
            "response": "Your email is user@example.com",
            "trajectory": ["model", "tools", "get_user_data", "model"],
        }
    },
]

3. Implement Trajectory Subsequence Evaluator

Create: /src/airunner/components/eval/utils/trajectory_evaluator.py

def trajectory_subsequence(outputs: dict, reference_outputs: dict) -> float:
    """
    Check how many of the expected steps the agent took.
    
    Returns:
        Float between 0-1 representing percentage of expected steps completed
    """
    if 'trajectory' not in reference_outputs or 'trajectory' not in outputs:
        return 1.0  # No trajectory specified = pass
    
    expected = reference_outputs['trajectory']
    actual = outputs['trajectory']
    
    if len(expected) > len(actual):
        return 0.0  # Agent took fewer steps than expected
    
    i = j = 0
    while i < len(expected) and j < len(actual):
        if expected[i] == actual[j]:
            i += 1
        j += 1
    
    return i / len(expected)

4. Add Trajectory Tracking Helper

Create: /src/airunner/components/eval/utils/tracking.py

async def track_trajectory(client, prompt: str, **kwargs) -> dict:
    """
    Run agent and track trajectory through nodes/tools.
    
    Returns:
        {
            'response': str,
            'trajectory': List[str],
            'tool_calls': List[dict],
        }
    """
    trajectory = []
    tool_calls = []
    response_text = ""
    
    async for event in client.astream_events(prompt=prompt, **kwargs):
        if event['type'] == 'task':
            trajectory.append(event['payload']['name'])
        elif event['type'] == 'tool_call':
            tool_name = event['payload']['tool_name']
            trajectory.append(tool_name)
            tool_calls.append({
                'tool': tool_name,
                'args': event['payload'].get('args', {}),
            })
        elif event['type'] == 'final_response':
            response_text = event['payload']['text']
    
    return {
        'response': response_text,
        'trajectory': trajectory,
        'tool_calls': tool_calls,
    }

5. Update Each Eval Test File

For Each Test File (test_*_tool_eval.py):

1. Import tracking utilities:

   from airunner.components.eval.utils.tracking import track_trajectory
   from airunner.components.eval.utils.trajectory_evaluator import trajectory_subsequence

2. Update test methods to track trajectory:

   def test_store_user_data_basic(self, ..., airunner_client_function_scope):
       expected_trajectory = ["model", "tools", "store_user_data", "model"]
       
       result = await track_trajectory(
           airunner_client_function_scope,
           prompt="Remember that my favorite color is blue",
           max_tokens=300,
           tool_categories=["USER_DATA"],
       )
       
       # Trajectory evaluation
       score = trajectory_subsequence(
           outputs=result,
           reference_outputs={'trajectory': expected_trajectory}
       )
       assert score >= 0.75, f"Trajectory match score {score} < 0.75"
       
       # Response quality evaluation (existing)
       assert "stored" in result['response'].lower()

3. Add trajectory validation to datasets:

   • Add expected_trajectory field to all test cases
   • Document expected tool call sequences
   • Include error recovery paths

6. Create Intent Classification Eval Test

Create: /src/airunner/components/eval/tests/test_intent_classification_eval.py

"""
Single-step evaluation: Test intent classification in isolation.

This tests that the agent correctly identifies user intent without
running the full workflow.
"""

import pytest

@pytest.mark.eval
class TestIntentClassificationEval:
    """Eval tests for intent classification accuracy."""
    
    def test_user_data_intent(self, airunner_client_function_scope):
        """Test identification of user data storage/retrieval intent."""
        prompts = [
            ("Remember my email is joe@example.com", "store_user_data"),
            ("What's my phone number?", "get_user_data"),
            ("Save my birthday as January 1", "store_user_data"),
        ]
        
        for prompt, expected_intent in prompts:
            # Call intent classifier directly (if exposed)
            # OR check trajectory starts with correct tool
            result = track_trajectory(
                airunner_client_function_scope,
                prompt=prompt,
                max_tokens=200,
                tool_categories=["USER_DATA"],
            )
            
            # First tool called should match expected intent
            first_tool = result['trajectory'][1]  # Skip "model" node
            assert expected_intent in first_tool

Priority Order

1. Week 1: Tracking Infrastructure

   • ✅ Create trajectory_evaluator.py with trajectory_subsequence()
   • ✅ Create tracking.py with track_trajectory() helper
   • ✅ Add stream events to airunner_client if not already supported

2. Week 2: Update Existing Eval Tests

   • ✅ Add trajectory tracking to test_user_data_tool_eval.py
   • ✅ Add trajectory tracking to test_agent_tool_eval.py
   • ✅ Add trajectory tracking to test_rag_tool_eval.py
   • ✅ Add trajectory tracking to test_knowledge_tool_eval.py

3. Week 3: Create New Eval Tests

   • ✅ test_intent_classification_eval.py - Single-step testing
   • ✅ test_trajectory_eval.py - Cross-tool trajectory testing
   • ✅ test_error_recovery_eval.py - Recovery path validation

4. Week 4: Documentation & Validation

   • ✅ Document expected trajectories for each tool
   • ✅ Run full eval suite and collect metrics
   • ✅ Update wiki with trajectory evaluation guide

Success Criteria

1. All eval tests include trajectory validation (100% coverage)
2. Trajectory match score >80% on expected paths
3. Tool call sequence correctness >90% (right tool, right order)
4. Error recovery paths documented for each tool
5. Intent classification accuracy >95% (once mode routing implemented)

Notes

• Current eval tests are CORRECT for checking tool triggering + response quality
• We're ADDING trajectory validation, not replacing existing checks
• Trajectory evaluation will be CRITICAL when we add mode-based routing
• This sets foundation for Phase 4 of mode-based architecture plan

Next Steps After This

Once trajectory evaluation is in place:

1. Implement mode-based routing (see Mode-Based-Agent-Architecture.md)
2. Create mode-specific eval datasets
3. Test mode switching with trajectory validation
4. Benchmark performance improvements