Skip to content

Add built-in retrieval quality checks #2445

Description

@henchaves

Overview

Add the standard retrieval-quality metrics (Recall@K, Precision@K, HitRate@K, MRR, NDCG@K, and InfAP) as first-class Check classes in the giskard.checks library. Currently, evaluation workflows (including the rag-evaluator skill) have to wrap each metric formula in a custom FnCheck, duplicating the implementation across user code and skill documentation. Promoting these to built-ins gives users a consistent, tested API for retrieval evaluation and eliminates a class of common implementation mistakes.

To-Do

  • Implement six new Check classes in giskard.checks (e.g. giskard.checks.builtin.retrieval): RecallAtK, PrecisionAtK, HitRateAtK, MRR, NDCGAtK, InfAP.
  • Each check accepts k where applicable, a threshold parameter for pass/fail, and JSONPath keys for the labelled relevant doc IDs and the retrieved doc IDs (e.g., relevant_ids_key, retrieved_ids_key).
  • Support the three scoring strategies documented in the rag-evaluator skill's retrieval-metrics.md Section 4: Strict (default), Cosine similarity, and LLM-judged relevance. Strategy selection should be a constructor option, with sensible defaults.
  • Add unit tests covering: empty relevant_ids, empty retrieved_ids, perfect retrieval, partial overlap, ranking sensitivity (for MRR / NDCG), and sparse-label cases (for InfAP).
  • Add API reference documentation for each new check, including worked examples that mirror current FnCheck recipes for an easy upgrade path.
  • Update the rag-evaluator skill (retrieval-metrics.md and examples.md) to use the new built-ins; deprecate or remove the custom FnCheck wrapper recipes once the built-ins ship.

Acceptance Criteria

  • All six retrieval metrics importable from giskard.checks and usable directly in .check(...) calls without writing custom FnCheck wrappers.
  • Each check supports configurable k, threshold, JSONPath keys for relevant/retrieved IDs, and a scoring-strategy option (Strict / Cosine / LLM-judged).
  • Unit test coverage includes the edge cases listed in the To-Do above and is green in CI.
  • Reference documentation is published on the Giskard Docs website, with worked examples for each metric.
  • The rag-evaluator skill no longer ships inline FnCheck reimplementations of these metrics; it points users at the built-ins.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions