MATH Level5 Accuracy Investigation

MATH Level 5 Accuracy Investigation

Date: October 31, 2025
Goal: Achieve 10/10 (100%) accuracy on MATH Level 5 benchmark
Current Status: 5/10 (50%) accuracy

Summary

Investigated and fixed multiple issues with MATH Level 5 evaluation, achieving reliable 50% accuracy. Tools and complex prompting strategies actually hurt performance instead of helping.

Key Findings

1. Test Metric Was Deceptive ❌

• Problem: Original test counted score >= 0.80 as "correct"
• Result: Reported 90% (9/10) when real accuracy was only 50% (5/10)
• Fix: Changed to ONLY use normalized exact matching

2. LLM Evaluator is Unreliable ❌

• Problem: LLM-based evaluator gave score 1.00 to completely wrong answers
• Examples:
  • Problem 5: Model answered 45, expected 87.5 → Evaluator gave 1.00
  • Problem 8: Model returned 3x3 matrix, expected 3x1 vector → Evaluator gave 1.00
• Fix: Removed evaluator from scoring logic, use only exact string matching

3. Matrix Extraction Bug 🐛

• Problem: re.search extracted FIRST matrix instead of LAST
• Impact: Problem 6 failed even though model gave correct answer
• Fix: Changed to re.findall(...matrices...)[-1] to get LAST matrix

4. LaTeX Normalization Issues 🔤

• Problem: \frac{1}{5} didn't match 1/5, \\\\ line breaks not handled
• Fix: Added normalization for:
  • \frac{a}{b} → a/b
  • \\\\ → space
  • Whitespace collapse

5. Response Truncation 📏

• Problem: Model responses incomplete, cut off mid-calculation
• Fix: Implemented continuation loop (max 10 attempts)
• Result: Helped on some problems but not enough to solve them

6. CRITICAL: Tools Hurt Performance ⚠️

• Baseline (no tools): 5/10 = 50% accuracy ✅
• With tools (MATH + ANALYSIS): 4/10 = 40% accuracy ❌
• Enhanced prompting + tools: 0/10 = 0% (just outputs tool JSON) ❌
• Conclusion: Tools add complexity and confuse the model

Test Results

Passing Problems (5/10)

• ✅ Problem 2: Polar coordinates (Precalculus)
• ✅ Problem 3: Trigonometric minimization (Precalculus)
• ✅ Problem 4: Reflection matrix (Precalculus)
• ✅ Problem 6: Vector reflection (Precalculus) - Fixed with matrix extraction
• ✅ Problem 10: Determinant calculation (Precalculus)

Failing Problems (5/10)

• ❌ Problem 1: Vector orthogonality

  • Expected: \frac{3}{8}
  • Got: \frac{3}{\sqrt{10}}
  • Issue: Wrong algebraic manipulation

• ❌ Problem 5: Trigonometric sum

  • Expected: 87.5
  • Got: 45
  • Issue: Incorrect summation formula

• ❌ Problem 7: Trigonometric equation

  • Expected: 18
  • Got: Wrong value
  • Issue: Calculation error

• ❌ Problem 8: Matrix normal vector

  • Expected: 3x1 vector \begin{pmatrix} ... \end{pmatrix}
  • Got: 3x3 matrix
  • Issue: Misunderstood problem requirements

• ❌ Problem 9: Trigonometric reduction

  • Expected: Tuple (3,-3,1)
  • Got: Wrong tuple (3,3,2)
  • Issue: Sign error

Attempted Strategies

What We Tried

1. Enhanced Pólya Prompting - Made things worse (0% with tools)
2. Few-Shot Learning - Test hung, couldn't complete
3. Self-Correction - Test hung, couldn't complete
4. Temperature Tuning - Test hung, couldn't complete
5. Continuation Loop - Helped with truncation but didn't fix wrong answers

What Worked

• ✅ Simple baseline (no tools): 50% accuracy
• ✅ Exact string matching: Reliable metric
• ✅ Matrix extraction fix: Fixed Problem 6
• ✅ LaTeX normalization: Better answer matching

What Made Things Worse

• ❌ Tools (MATH/ANALYSIS): Reduced accuracy to 40%
• ❌ Complex prompting: Model just outputs tool JSON
• ❌ Higher max_tokens: Doesn't help if reasoning is wrong

Code Changes

Files Modified

1. benchmark_datasets/__init__.py

   • Fixed extract_numeric_answer() to get LAST matrix
   • Enhanced normalize_answer() with LaTeX patterns

2. test_math_level5.py

   • Added continuation loop detection and retry logic
   • Changed scoring to exact match only
   • Increased timeout 300s → 600s
   • Added comprehensive debug output

3. New diagnostic tests:

   • test_token_limits.py
   • test_truncation_investigation.py

Next Steps to Reach 100%

Short-term (Incremental Improvements)

1. Analyze failing problems - Understand WHY these 5 fail
2. Problem-specific prompting - Different strategies per problem type
3. Multi-sampling + voting - Generate 3 answers, pick most common
4. Chain-of-thought - Force explicit step-by-step reasoning

Medium-term (Model Improvements)

1. Fine-tuning - Train model specifically on MATH dataset
2. Larger model - Try Qwen2.5-14B or 32B
3. Ensemble methods - Combine multiple model predictions

Long-term (Fundamental Changes)

1. Symbolic math engine - Use SymPy for calculations
2. Hybrid approach - LLM for reasoning, symbolic for computation
3. Iterative refinement - Multiple correction passes

Recommendations

For this PR:

• ✅ Commit the fixes we made (matrix extraction, normalization, continuation)
• ✅ Document 50% baseline accuracy as starting point
• ⚠️ Do NOT include unreliable LLM evaluator in scoring
• ⚠️ Do NOT use tools for MATH dataset (they hurt performance)

For future work:

• Focus on understanding the 5 failing problems
• Try simpler approaches before complex ones
• Baseline (no tools) is the foundation to build on
• Consider fine-tuning on MATH dataset for better results

Technical Details

Model: Qwen2.5-7B-Instruct (4-bit quantized)
Hardware: NVIDIA RTX 5080, 16GB VRAM
Parameters: temperature=0.0, max_tokens=4096, no tools
Test Duration: ~6.5 minutes for 10 problems
Evaluation Method: Normalized exact string matching

Commit Hash

• Initial fixes: 934a71b0

References

• MATH Dataset: https://github.com/hendrycks/math
• Evaluation code: src/airunner/components/eval/tests/test_math_level5.py
• Answer extraction: src/airunner/components/eval/benchmark_datasets/__init__.py