Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
100 lines (70 loc) · 4.5 KB

CONTRIBUTING.md

File metadata and controls

100 lines (70 loc) · 4.5 KB
Raw

Contributing to deepeval-bcg

Thanks for considering a contribution. This project is built on real engagement output, real failure modes, and verbatim MBB-grade calibration — contributions that move it forward in any of those directions are welcome.

High-value contribution areas

Area Why it matters
New vertical rubric anchors Healthcare, finance, retail, energy, infrastructure each need calibrated rubric anchors. The core 8-dim rubric is industry-agnostic; vertical overlays sharpen it.
Tier-1 NLI contradiction check Embedding similarity catches paraphrase drift but not inversion. NLI-based contradiction detection between artifact and upstream context is the next obvious add.
Cross-language anchors Verbatim BCG anchors are English. Russian, Mandarin, German, Spanish, Arabic translations valued (must preserve calibration, not just translate literally).
New novelty signals 10 signals shipped; counterfactual replay attribution (AgenTracer-style) and adversarial probe survival are obvious next signals.
Non-Claude-Code runtime Currently Claude-native via Claude Code. Adapters for plain Anthropic SDK, Workbench, or other runtimes appreciated — but keep the no-required-keys core.
Golden-set entries Real engagement artifacts (anonymized) labelled by an experienced reviewer expand the regression set. Aim 50–200 per archetype.
Feedback issue parsers The aggregator currently does simple keyword extraction. Better feedback-issue NLP improves community stats.

How to contribute

Reporting a bug

Open an issue with Bug template.

Suggesting a feature

Open an issue with Feature template.

Submitting eval feedback (from running the skill)

Every /deepeval-run emits a pre-filled GitHub Issue URL with verdict context already filled. Click it, edit/confirm, submit. This is the preferred way to give us calibration signal — see Eval feedback template.

Code / docs contributions

  1. Fork the repo.
  2. Create a branch: git checkout -b your-feature.
  3. Make changes. Keep style: stdlib-only for T0/T1 where possible; no API keys; cadence rule (D/W/30d max) preserved.
  4. Test: 
    python3 .claude/skills/deepeval/scripts/eval_tier0.py --artifact examples/amazon-eval-2026-05-17/verdict.md
python3 .claude/skills/deepeval/scripts/eval_tier1.py --artifact examples/amazon-eval-2026-05-17/verdict.md
  5. Submit PR with description of (a) what changed, (b) failure mode it addresses, (c) backward-compatibility impact.

Verbatim rubric anchor changes

Be careful here. The BCG rubric anchors are calibrated language. PRs that paraphrase anchor text will be declined unless they include:

  • Evidence that the original anchor was misinterpreted in ≥3 evals
  • Proposed new anchor + Krippendorff α data showing improved inter-rater agreement
  • Migration path for past evals

This is the same bar BCG uses for its own rubric revisions.

Repo layout note — dual skill location

The skill exists in two places:

  • .claude/skills/deepeval/ — used by the shell installer (install.sh) and by Claude Code when this repo itself is opened as a working directory.
  • plugin/skills/deepeval/ — used by the Claude Code plugin marketplace flow (/plugin marketplace add EvXata/deepeval-bcg).

Both must stay in sync. When you edit one, mirror the change:

rsync -a --delete .claude/skills/deepeval/ plugin/skills/deepeval/

See docs/publishing.md for the full publication channels and an optional pre-commit hook that enforces sync.

Style guide

  • Code: PEP 8 for Python, 4-space indent, type hints when helpful.
  • Docs: GitHub-flavored Markdown, no emoji in code/configs, em-dash — not double-hyphen.
  • Commit messages: imperative present tense ("add cross-doc consistency check"), 72-char first line, optional body explaining why.
  • No promotional adjectives in skill/doc bodies. The product is its outputs, not its self-description.

Code of conduct

By contributing you agree to the Contributor Covenant.

License

Contributions are MIT-licensed alongside the rest of the project. By opening a PR you confirm you have the right to submit the work under MIT.

Maintainers

@EvXata is the primary maintainer. PRs are reviewed within ~7 days. For urgent issues (security, data leak in golden-set), label urgent — those are reviewed within 24h.