Skip to content

BENCHMARK_V4.md

Latest commit

 

History

History
64 lines (53 loc) · 3.12 KB

BENCHMARK_V4.md

File metadata and controls

64 lines (53 loc) · 3.12 KB
Raw

V4 Benchmark: 10 Queries × 3 MCPs = 30 Runs

Date: 2026-05-31
Configs:

  • MCP1: academix_academic_search_papers — limit=15, sort="relevance"
  • MCP2: paper-search_search_papers — sources=all, max_results_per_source=5
  • MCP3: research_search_literature — max_results=15

BEST_SOURCES for MCP3: arxiv, semantic, crossref, unpaywall, openaire OpenAlex backend: academix (sole, source_hits floor=2) Precision weights: openaire+2, scopus+2, springer+2, academix+1, arxiv+1, semantic+1 Ranking: source_hits → relevance_score (+citation boost) → abstract → citation_count → year Citation walk: forward + backward, most-cited papers, 10 per walk, source_hits floor=2 relevance_score filter: >=3 (removes papers with zero term match + <50 citations)

Summary Table

# Query MCP1 Rel MCP2 Rel MCP3 Rel Winner
Q1 generative AI feedback accuracy second language writing 7 5 5 MCP1
Q2 ChatGPT written corrective feedback EFL revision quality 10 7 6 MCP1
Q3 peer teacher LLM feedback comparison accuracy second language writing 5 5 7 MCP3
Q4 LLM self-assessment bias self-enhancement writing evaluation 2 4 7 MCP3
Q5 DeBERTa fine-tuned text classification educational feedback 1 3 7 MCP3
Q6 feedback literacy intervention training evaluative judgement 3 8 11 MCP3
Q7 metacognitive laziness AI student learning Fan 2 9 9 MCP3/MCP2 tie
Q8 hybrid AI human feedback writing accuracy L2 4 12 11 MCP2
Q9 spaced micro-learning feedback evaluation writing 1 5 9 MCP3
Q10 uptake taxonomy feedback L2 writing revision decisions 5 6 7 MCP3
Total 40 64 79 MCP3 7/10

Precision

MCP1 MCP2 MCP3
26.7% 15.8% 52.7%

MCP2 Source Performance (across all 10 queries)

Source Returned Precision Note
arxiv 5/5 ~50% Best consistent source
semantic 2.5/5 ~60% Variable but high-precision when hits
crossref 5/5 ~40% Mixed quality
openalex 5/5 ~30% Duplicates MCP1
pmc 2.5/5 ~10% Biomedical noise
europepmc 4.5/5 ~10% Biomedical noise
core 0.8/5 0% Dead
biorxiv 5/5 0% Neuroscience only
medrxiv 5/5 0% Epidemiology only
8 others 0/5 — Dead sources
zenodo 0/5 crashes Error every query

Key Findings

  1. MCP3 wins 7/10 queries with 52.7% precision — the clear tool for this domain
  2. MCP1 wins on broad queries (Q1, Q2) where citation-weighted relevance surfaces landmark papers
  3. MCP2 wins Q8 (hybrid feedback) via Semantic Scholar — MCP3's only missed query
  4. relevance_score filter (≥3) removes ~55% of noise with zero false negatives
  5. bioRxiv and medRxiv are pure noise in MCP2 for all education queries — 0% precision
  6. zenodo crashes every query — should be excluded by default
  7. 8 of 21 MCP2 sources are dead for this domain