generated from amazon-archives/__template_Apache-2.0
-
Notifications
You must be signed in to change notification settings - Fork 11
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Problem Statement
Currently there's no way to evaluate whether RAG responses are grounded in the retrieved context. The existing FaithfulnessEvaluator checks against conversation history, but RAG systems need validation against the actual context retrieved from vector stores.
Proposed Solution
Add a ContextualFaithfulnessEvaluator that validates responses against a retrieval_context field on test Cases.
Use Case
When using RAG, I need to detect hallucinations:
case = Case(
input="What is the refund policy?",
retrieval_context=[
"Refunds available within 30 days of purchase.",
"Items must be unopened for full refund."
]
)The evaluator would then score how grounded the response is in relation to the retrieval context defined in the Case.
Alternatives Solutions
No response
Additional Context
No response
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request