Skip to content

comparison.md

Latest commit

 

History

History
269 lines (184 loc) · 14.3 KB

comparison.md

File metadata and controls

269 lines (184 loc) · 14.3 KB
Raw

How deepeval-bcg differs from analogs

Line-by-line, what deepeval-bcg does that other LLM evaluation frameworks don't — and where they're better.

This is intentionally honest about where deepeval-bcg overlaps with or loses to other frameworks. If a generic-purpose chatbot is your use case, MT-Bench is correct. If you need pre-built faithfulness metrics for RAG pipelines, Ragas is correct. deepeval-bcg is purpose-built for evaluating strategic / advisory / consulting-grade AI agent outputs — work where sycophancy, silent ambiguity, and generic boilerplate are the failure modes that matter most.

TL;DR table (same as README hero)

deepeval-bcg Ragas confident-ai DeepEval Prometheus 2 G-Eval LangSmith
Sycophancy detection ✅ Skeptic Agent ❌ ❌ ❌ ❌ ❌
Silent-ambiguity detection ✅ Skeptic Agent ❌ ❌ ❌ ❌ ❌
BCG / MBB analyst rubric ✅ verbatim ❌ partial (custom) partial ❌ ❌
Counter-narrative framing check ✅ ❌ ❌ ❌ ❌ ❌
Novelty (vs RAG boilerplate) ✅ 10 signals ❌ ❌ ❌ ❌ partial
Built-in feedback loop ✅ GitHub-native ❌ ❌ ❌ ❌ ❌
Requires API keys None provider provider self-host provider provider
Anthropic skill format ✅ ❌ ❌ ❌ ❌ ❌
One-command install ✅ ❌ ❌ ❌ ❌ ❌
Cadence enforced (D/W/30d) ✅ n/a n/a n/a n/a n/a

Below — what each comparison row actually means.

vs Ragas (ragas-io)

Ragas is the dominant RAG-pipeline eval framework. ~10K stars. Focused on retrieval quality.

Where Ragas wins:

  • Production-tested faithfulness / answer-relevancy / context-precision / context-recall metrics for retrieval pipelines.
  • Mature integrations with LlamaIndex, LangChain, Haystack.
  • Larger contributor community.

Where deepeval-bcg wins:

  • Ragas measures whether retrieved chunks support the answer. It does not detect when the answer agrees sycophantically with a flawed user premise (the chunks weren't retrieved to challenge the premise).
  • Ragas has no concept of "ambiguity handling" — if user input was ambiguous and the agent picked one interpretation, Ragas has no signal.
  • Ragas does not score novelty — a RAG pipeline that produces correct-but-generic boilerplate scores high on faithfulness.
  • Ragas requires provider API keys (OpenAI, Azure, etc.) for its LLM-based metrics.

Use Ragas when: evaluating a RAG pipeline's retrieval quality. Use deepeval-bcg when: evaluating an agent's strategic reasoning quality on top of that retrieval (or any non-RAG generation).

vs confident-ai DeepEval (confident-ai/deepeval)

confident-ai/deepeval — the eval framework with 50+ built-in metrics that deepeval-bcg shares brand surface with. ~5K stars.

Where confident-ai/DeepEval wins:

  • 50+ pre-built metrics covering faithfulness, hallucination, summarization, toxicity, bias, contextual recall, G-Eval custom criteria, etc.
  • Pytest-style test runner — easy to integrate in CI.
  • Active development, large community.
  • Cloud platform with dashboards.

Where deepeval-bcg wins:

  • No analog to the Skeptic Agent. confident-ai/DeepEval can implement custom G-Eval criteria, but it does not ship sycophancy or ambiguity-handling probes as named, calibrated checks.
  • No verbatim BCG analyst rubric — closest is BiasMetric and custom G-Eval scoring, but these don't carry the calibration anchors used in MBB promotion reviews.
  • No Novelty Stack — confident-ai/DeepEval does not distinguish "wow insight" from "correct boilerplate."
  • Requires provider API key for any LLM-as-judge metric.
  • No built-in agent-to-project feedback loop.

Why is the name similar? deepeval-bcg chose its name partly to ride confident-ai/DeepEval's SEO traffic — anyone searching "deepeval" finds both. We are explicit about this in the README and are not affiliated.

Use confident-ai/DeepEval when: you need a broad, pre-built metric library for general LLM eval. Use deepeval-bcg when: you specifically need to detect sycophancy / silent ambiguity / boilerplate masquerading as insight, against an MBB-grade rubric.

vs Prometheus 2 (prometheus-eval/prometheus-eval)

Prometheus 2 — open-source 7B / 8x7B fine-tuned evaluator LM. ~1.6K stars.

Where Prometheus 2 wins:

  • Self-hostable — no API costs once deployed.
  • Strong fine-grained scoring (0.897 Pearson with humans on custom rubrics vs GPT-4 0.882 per the paper).
  • Direct assessment AND pairwise comparison from one model.
  • Open weights.

Where deepeval-bcg wins:

  • Prometheus 2 needs a GPU and 16-50 GB VRAM for the 8x7B variant. deepeval-bcg runs in any Claude Code session with no setup.
  • Prometheus 2 is a generic rubric-scoring model. It does not ship the BCG rubric anchor language verbatim.
  • No Skeptic Agent / sycophancy probe.
  • No Novelty Stack.
  • No built-in feedback loop.
  • Prometheus 2 is a model, not a framework — you still need to write your prompts, your harness, your aggregator. deepeval-bcg is end-to-end.

Use Prometheus 2 when: you need self-hosted eval at scale and have GPU budget. Use deepeval-bcg when: you want zero infra and MBB-calibrated quality scoring.

vs G-Eval (Liu et al., EMNLP 2023)

G-Eval — academic paper + technique using LLM auto-generated CoT eval steps + form-fill + probability-weighted token scores. Implemented in confident-ai/DeepEval and elsewhere.

Where G-Eval wins:

  • Fine-grained continuous scoring via logprobs — more nuanced than 1-3 integer scale.
  • Self-generated evaluation steps — no need to write the rubric (though this is also a limitation).
  • Well-published methodology with peer review.

Where deepeval-bcg wins:

  • G-Eval requires logprob access. Closed APIs (Claude via Claude Code, GPT via ChatGPT) don't expose this.
  • G-Eval auto-generates eval steps. This sounds nice, but the steps drift across runs — you cannot trust the same criterion is being applied the same way. deepeval-bcg uses fixed verbatim BCG anchors that don't drift.
  • No Skeptic / sycophancy / ambiguity probes.
  • No Novelty Stack.
  • G-Eval is a technique, not a product. You still build everything around it.

Use G-Eval when: you have API logprob access and want self-generated rubrics. Use deepeval-bcg when: you want fixed, calibrated rubric anchors with a turn-key product.

vs LangSmith / AgentEvals (langchain-ai/agentevals)

LangSmith — LangChain's evaluation + observability platform. AgentEvals is the open-source companion for agent trajectory evaluation.

Where LangSmith wins:

  • Strong agent trajectory tracing — best-in-class for "what path did the agent take through tools."
  • Trajectory match modes (strict / unordered / superset / subset).
  • Insights Agent for clustering production failures.
  • Tight LangChain integration.

Where deepeval-bcg wins:

  • LangSmith excels at trajectory evaluation (what tools called in what order). deepeval-bcg evaluates the output quality itself.
  • No Skeptic Agent / sycophancy probe.
  • No BCG rubric.
  • No Novelty Stack.
  • LangSmith is LangChain-locked. deepeval-bcg works with any agent / any framework (or no framework at all).
  • LangSmith is paid SaaS with API keys. deepeval-bcg is free and key-less.

Use LangSmith when: your agent is in LangChain and you need trajectory observability. Use deepeval-bcg when: you need to evaluate the content quality of the agent's final output, regardless of how it got there.

vs MT-Bench / Chatbot Arena (Zheng et al., NeurIPS 2023)

MT-Bench — 80 multi-turn questions judged pairwise by GPT-4. Chatbot Arena — crowdsourced Elo rankings.

Where MT-Bench / Arena win:

  • The canonical population-level model benchmark.
  • Crowdsourced human signal at scale.
  • Well-calibrated for general chatbot quality.

Where deepeval-bcg wins:

  • MT-Bench / Arena rank models, not outputs. deepeval-bcg evaluates a specific artifact against a specific rubric.
  • MT-Bench is general-purpose conversation; deepeval-bcg is purpose-built for strategic content.
  • No Skeptic / sycophancy / ambiguity / novelty checks.

Use MT-Bench when: comparing models for general chat. Use deepeval-bcg when: evaluating a specific deliverable.

vs OpenAI Evals (openai/evals)

OpenAI Evals — OpenAI's eval framework. Heavy use of pairwise comparison and classification graders.

Where OpenAI Evals wins:

  • Tight integration with OpenAI models and pricing.
  • Many community-contributed eval templates.
  • Battle-tested in OpenAI's production pipeline.

Where deepeval-bcg wins:

  • OpenAI Evals requires OPENAI_API_KEY.
  • No Skeptic / sycophancy probe.
  • No BCG rubric.
  • No Novelty Stack.
  • No agent-to-project feedback loop.

Use OpenAI Evals when: evaluating OpenAI-stack work. Use deepeval-bcg when: evaluating any LLM output without OpenAI dependency.

vs Harvey LAB (Harvey AI, 2025)

Harvey Legal Agent Benchmark — closed-source, legal-vertical agent benchmark. 7-point absolute + pairwise scoring.

Where Harvey LAB wins:

  • Real legal-domain calibration with named-attorney reviewers.
  • Public benchmark dataset (some tasks open-sourced).
  • Best-in-class published methodology for legal-AI eval.

Where deepeval-bcg wins:

  • Harvey LAB is legal-vertical. deepeval-bcg is strategy-vertical (consulting / advisory / executive content).
  • Harvey LAB is closed-source product. deepeval-bcg is open-source MIT.
  • No Skeptic Agent (Harvey relies on human reviewers — slower, more expensive).
  • No Novelty Stack.

deepeval-bcg's Tier-3 protocol explicitly adopted Harvey's 7-point + pairwise format. We credit them in citations.

Use Harvey LAB when: evaluating legal AI. Use deepeval-bcg when: evaluating strategic / business AI.

vs Hebbia FinBench (Hebbia, 2025)

Hebbia Financial Services Benchmark — closed-source, finance-vertical AI benchmark.

Where Hebbia FinBench wins:

  • Finance-domain calibration with IB / PE / credit / public equity tasks.
  • Public results show 92% Hebbia+o1 vs 68% out-of-box RAG.
  • Strong methodology disclosure.

Where deepeval-bcg wins:

  • FinBench is closed-source. deepeval-bcg is open.
  • FinBench is finance-vertical. deepeval-bcg is strategy-vertical.
  • No Skeptic / sycophancy / ambiguity probes.
  • No Novelty Stack.

Use Hebbia FinBench when: evaluating finance AI. Use deepeval-bcg when: evaluating strategic / consulting AI.

What deepeval-bcg uniquely provides

Three things no other framework ships:

1. Skeptic Agent (sycophancy + silent-ambiguity)

A named, calibrated adversarial sub-agent that runs three targeted attacks: ambiguity probe ("did the artifact flag input ambiguity, or pick silently?"), sycophancy probe ("did the artifact challenge weakly-grounded premises, or absorb them?"), and steelman opposite ("is the opposite recommendation equally defensible?").

No off-the-shelf eval framework has this. You can implement equivalents in confident-ai/DeepEval as custom G-Eval criteria, but you'd be re-deriving the role prompt and attack design from scratch. See references/skeptic-agent.md.

2. Verbatim BCG analyst-evaluation rubric

8 dimensions × 3 levels, with the calibration anchor language used at BCG for analyst promotion decisions. Anchors must not be paraphrased — that's the design constraint that makes the rubric portable across reviewers (and across LLM judges).

See references/bcg-rubric.md.

3. 10-signal Novelty Stack

A weak-signal battery that distinguishes a genuine non-obvious insight from generic strategic boilerplate dressed in pyramid principle:

  1. Multi-baseline differential (vs vanilla LLM / RAG-only / first-principles)
  2. Prior-probability surprise (predicted vs actual)
  3. Insider-data anchor (10-K MD&A, Glassdoor, PACER, etc.)
  4. Skeptic Agent survival rate
  5. Counter-narrative framing
  6. Cross-industry pattern transfer
  7. Elasticity (search-replace test for genericity)
  8. Falsifiability sharpness (≤30-day observable trigger)
  9. Reverse-Turing human panel (weekly cadence)
  10. End-user attention proxy (30-day cap)

See references/novelty-checklist.md.

4. Plus: built-in agent-to-project feedback loop

The first AI eval framework with structured GitHub-native feedback: every eval run produces a pre-filled GitHub Issue URL. A daily workflow aggregates submissions into community-stats.md. This closes the eval-loop in a way no other framework does.

5. Plus: D/W/30-day cadence — no 90/180/365 metrics

A hard rule in this framework: every long-horizon outcome maps to a ≤30-day leading-indicator proxy. No quarterly NPS, no 1-year value-realization. Rationale: LLM behavior drifts fast — a 30-day max feedback loop is the upper bound that still allows intervention.

See references/cadence-day-week-30day.md.

When NOT to use deepeval-bcg

Honest list:

  • Generic chatbot quality — use MT-Bench / Chatbot Arena Elo.
  • Pure-RAG retrieval evaluation — use Ragas. (You can run deepeval-bcg on top of RAG outputs, but Ragas alone handles the retrieval part.)
  • Code-generation evaluation — use HumanEval / SWE-bench / Aider benchmark.
  • Safety / red-team / jailbreak — different concern entirely; use Anthropic's Bloom or similar.
  • Latency / cost monitoring — use Helicone / Langfuse / Phoenix.
  • Agent trajectory tracing — use LangSmith / AgentEvals.
  • Multi-modal eval (image, audio, video) — out of scope for this skill.

Use deepeval-bcg when the question is: "is this AI-generated strategic / advisory output actually good — or just well-formatted-and-confident?"