Overview
Add the standard retrieval-quality metrics (Recall@K, Precision@K, HitRate@K, MRR, NDCG@K, and InfAP) as first-class Check classes in the giskard.checks library. Currently, evaluation workflows (including the rag-evaluator skill) have to wrap each metric formula in a custom FnCheck, duplicating the implementation across user code and skill documentation. Promoting these to built-ins gives users a consistent, tested API for retrieval evaluation and eliminates a class of common implementation mistakes.
To-Do
- Implement six new Check classes in
giskard.checks (e.g. giskard.checks.builtin.retrieval): RecallAtK, PrecisionAtK, HitRateAtK, MRR, NDCGAtK, InfAP.
- Each check accepts
k where applicable, a threshold parameter for pass/fail, and JSONPath keys for the labelled relevant doc IDs and the retrieved doc IDs (e.g., relevant_ids_key, retrieved_ids_key).
- Support the three scoring strategies documented in the
rag-evaluator skill's retrieval-metrics.md Section 4: Strict (default), Cosine similarity, and LLM-judged relevance. Strategy selection should be a constructor option, with sensible defaults.
- Add unit tests covering: empty
relevant_ids, empty retrieved_ids, perfect retrieval, partial overlap, ranking sensitivity (for MRR / NDCG), and sparse-label cases (for InfAP).
- Add API reference documentation for each new check, including worked examples that mirror current
FnCheck recipes for an easy upgrade path.
- Update the
rag-evaluator skill (retrieval-metrics.md and examples.md) to use the new built-ins; deprecate or remove the custom FnCheck wrapper recipes once the built-ins ship.
Acceptance Criteria
- All six retrieval metrics importable from
giskard.checks and usable directly in .check(...) calls without writing custom FnCheck wrappers.
- Each check supports configurable
k, threshold, JSONPath keys for relevant/retrieved IDs, and a scoring-strategy option (Strict / Cosine / LLM-judged).
- Unit test coverage includes the edge cases listed in the To-Do above and is green in CI.
- Reference documentation is published on the Giskard Docs website, with worked examples for each metric.
- The
rag-evaluator skill no longer ships inline FnCheck reimplementations of these metrics; it points users at the built-ins.
Overview
Add the standard retrieval-quality metrics (Recall@K, Precision@K, HitRate@K, MRR, NDCG@K, and InfAP) as first-class
Checkclasses in thegiskard.checkslibrary. Currently, evaluation workflows (including therag-evaluatorskill) have to wrap each metric formula in a customFnCheck, duplicating the implementation across user code and skill documentation. Promoting these to built-ins gives users a consistent, tested API for retrieval evaluation and eliminates a class of common implementation mistakes.To-Do
giskard.checks(e.g.giskard.checks.builtin.retrieval):RecallAtK,PrecisionAtK,HitRateAtK,MRR,NDCGAtK,InfAP.kwhere applicable, athresholdparameter for pass/fail, and JSONPath keys for the labelled relevant doc IDs and the retrieved doc IDs (e.g.,relevant_ids_key,retrieved_ids_key).rag-evaluatorskill'sretrieval-metrics.mdSection 4: Strict (default), Cosine similarity, and LLM-judged relevance. Strategy selection should be a constructor option, with sensible defaults.relevant_ids, emptyretrieved_ids, perfect retrieval, partial overlap, ranking sensitivity (for MRR / NDCG), and sparse-label cases (for InfAP).FnCheckrecipes for an easy upgrade path.rag-evaluatorskill (retrieval-metrics.mdandexamples.md) to use the new built-ins; deprecate or remove the customFnCheckwrapper recipes once the built-ins ship.Acceptance Criteria
giskard.checksand usable directly in.check(...)calls without writing customFnCheckwrappers.k,threshold, JSONPath keys for relevant/retrieved IDs, and a scoring-strategy option (Strict / Cosine / LLM-judged).rag-evaluatorskill no longer ships inlineFnCheckreimplementations of these metrics; it points users at the built-ins.