RFE Judge - LLM Output Comparison Tool

An intelligent agent that evaluates and compares two LLM-generated Request for Feature Enhancement (RFE) documents using Claude AI.

Overview

This tool uses Claude to assess RFE quality across five key metrics:

1. Clarity of Purpose and Stakeholder Alignment - How clearly the RFE defines the problem and user story
2. Structural Completeness and Organization - Quality of document structure and formatting
3. Actionability and Testability - Presence of testable requirements and acceptance criteria
4. Language Quality and Communicative Tone - Professional tone and linguistic precision
5. Role Consistency and Perspective - Maintenance of assigned role perspective

Each metric is scored 1-5, and the tool automatically determines which output is superior based on total scores.

Setup

1. Install Dependencies

pip install -r requirements.txt

2. Set Your Anthropic API Key

export ANTHROPIC_API_KEY='your-api-key-here'

Or add it to your ~/.bashrc or ~/.zshrc:

echo 'export ANTHROPIC_API_KEY="your-api-key-here"' >> ~/.zshrc
source ~/.zshrc

Usage

Input File Format

Create a JSON file with the following structure:

{
  "original_prompt": "The prompt that was used to generate the RFEs",
  "assigned_role": "Product Manager for OpenShift (optional)",
  "output_a": {
    "label": "GPT-4 Output",
    "content": "... full RFE text ..."
  },
  "output_b": {
    "label": "Claude Sonnet Output",
    "content": "... full RFE text ..."
  }
}

Run the Evaluation

python rfe_judge.py example_input.json

Output

The tool will:

1. Evaluate both outputs on all 5 criteria
2. Display scores in the terminal
3. Save detailed results to <input_filename>_results.json

Example output:

Evaluating RFE outputs...
Output A: GPT-4 Output
Output B: Claude Sonnet Output

Evaluating Output A...
Output A Total Score: 23/25

Evaluating Output B...
Output B Total Score: 15/25

Results saved to: example_input_results.json

============================================================
FINAL RESULTS
============================================================
Output A (GPT-4 Output): 23/25
Output B (Claude Sonnet Output): 15/25

Winner: GPT-4 Output (by 8 points)
============================================================

Results File Format

The results JSON file contains:

{
  "input_metadata": {
    "original_prompt": "...",
    "assigned_role": "...",
    "output_a_label": "...",
    "output_b_label": "..."
  },
  "output_a": {
    "scores": {
      "clarity_of_purpose": {
        "score": 4,
        "reasoning": "The RFE clearly defines..."
      },
      ...
    },
    "total_score": 23
  },
  "output_b": {
    "scores": { ... },
    "total_score": 15
  },
  "winner": "output_a",
  "score_difference": 8
}

Evaluation Criteria Details

1. Clarity of Purpose and Stakeholder Alignment (1-5)

• 1: Vague problem and unclear user/stakeholder
• 5: Clearly defines the user, the problem, and the desired outcome/goal

2. Structural Completeness and Organization (1-5)

• 1: Unformatted wall of text with no clear sections
• 5: Perfectly structured with logical headings, clear sections, and excellent flow

3. Actionability and Testability (1-5)

• 1: Lacks any definable acceptance criteria or next steps
• 5: Includes clear, generic, and testable requirements that define 'done'

4. Language Quality and Communicative Tone (1-5)

• 1: Ambiguous, overly verbose, or unprofessional language
• 5: Concise, precise, and highly professional tone

5. Role Consistency and Perspective (1-5)

• 1: Shows no distinguishable difference from a default RFE
• 5: Clearly frames the entire request using the assigned role's unique concerns and priorities

Tips for Best Results

1. Include the assigned role: Adding the role helps evaluate criterion #5 more accurately
2. Use realistic RFE outputs: The tool works best with complete RFE documents
3. Consistent formatting: Ensure both outputs are in their raw form (don't manually edit)
4. Review the reasoning: The detailed reasoning in the results file provides insights beyond just scores

Customization

To modify the evaluation criteria, edit the CRITERIA dictionary in rfe_judge.py:

CRITERIA = {
    "clarity_of_purpose": {
        "name": "Clarity of Purpose and Stakeholder Alignment",
        "description": "...",
        "prompt": "..."
    },
    # Add or modify criteria here
}

Troubleshooting

Error: "ANTHROPIC_API_KEY not found"

• Make sure you've set the environment variable correctly
• Check with: echo $ANTHROPIC_API_KEY

Error: "Error parsing JSON response"

• This is rare but can happen if Claude's response format is unexpected
• The error message will show the raw response for debugging

Low scores across all criteria:

• Ensure the RFE outputs are complete and not truncated
• Check that the original_prompt accurately represents what was used

License

MIT