Skip to content

Commit

Permalink
Fix a few old groundedness references (#1139)
Browse files Browse the repository at this point in the history
* Update selecting_components.md

* Update MultiQueryRetrievalLangchain.ipynb

* Update random_evaluation.ipynb

* Update canopy_quickstart.ipynb
  • Loading branch information
joshreini1 authored May 15, 2024
1 parent 95d8d0b commit c424cd4
Show file tree
Hide file tree
Showing 4 changed files with 27 additions and 36 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -9,21 +9,25 @@ to refer to parts of an LLM stack trace and use those when defining evaluations.
For example, the following lens refers to the input to the retrieve step of the
app called query.

```python
Select.RecordCalls.retrieve.args.query
```
!!! example

```python
Select.RecordCalls.retrieve.args.query
```

Such lenses can then be used to define evaluations as so:

```python
# Context relevance between question and each context chunk.
f_context_relevance = (
Feedback(provider.context_relevance_with_cot_reasons, name = "Context Relevance")
.on(Select.RecordCalls.retrieve.args.query)
.on(Select.RecordCalls.retrieve.rets)
.aggregate(np.mean)
)
```
!!! example

```python
# Context relevance between question and each context chunk.
f_context_relevance = (
Feedback(provider.context_relevance_with_cot_reasons, name = "Context Relevance")
.on(Select.RecordCalls.retrieve.args.query)
.on(Select.RecordCalls.retrieve.rets)
.aggregate(np.mean)
)
```

In most cases, the Select object produces only a single item but can also
address multiple items.
Expand All @@ -35,18 +39,15 @@ the documents returned by the `retrieve` method. These items can be evaluated se
as shown above, or can be collected into an array for evaluation with `.collect()`.
This is most commonly used for groundedness evaluations.

Example:
!!! example

```python
grounded = Groundedness(groundedness_provider=provider)

f_groundedness = (
Feedback(grounded.groundedness_measure_with_cot_reasons, name = "Groundedness")
.on(Select.RecordCalls.retrieve.rets.collect())
.on_output()
.aggregate(grounded.grounded_statements_aggregator)
)
```
```python
f_groundedness = (
Feedback(provider.groundedness_measure_with_cot_reasons, name = "Groundedness")
.on(Select.RecordCalls.retrieve.rets.collect())
.on_output()
)
```

Selectors can also access multiple calls to the same component. In agentic applications,
this is an increasingly common practice. For example, an agent could complete multiple
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,6 @@
"from trulens_eval.feedback.provider import OpenAI\n",
"import logging\n",
"from trulens_eval.app import App\n",
"from trulens_eval.feedback import Groundedness\n",
"from langchain_core.output_parsers import StrOutputParser\n",
"from langchain_core.runnables import RunnablePassthrough\n",
"from langchain import hub"
Expand Down Expand Up @@ -184,14 +183,11 @@
"\n",
"context = App.select_context(rag_chain)\n",
"\n",
"\n",
"grounded = Groundedness(groundedness_provider=OpenAI())\n",
"# Define a groundedness feedback function\n",
"f_groundedness = (\n",
" Feedback(grounded.groundedness_measure_with_cot_reasons)\n",
" Feedback(provider.groundedness_measure_with_cot_reasons)\n",
" .on(context.collect()) # collect context chunks into a list\n",
" .on_output()\n",
" .aggregate(grounded.grounded_statements_aggregator)\n",
")\n",
"\n",
"# Question/answer relevance between overall question and answer.\n",
Expand Down
6 changes: 1 addition & 5 deletions trulens_eval/examples/experimental/random_evaluation.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -203,22 +203,18 @@
"outputs": [],
"source": [
"from trulens_eval import Feedback, Select\n",
"from trulens_eval.feedback import Groundedness\n",
"from trulens_eval.feedback.provider.openai import OpenAI as fOpenAI\n",
"\n",
"import numpy as np\n",
"\n",
"# Initialize provider class\n",
"fopenai = fOpenAI()\n",
"\n",
"grounded = Groundedness(groundedness_provider=fopenai)\n",
"\n",
"# Define a groundedness feedback function\n",
"f_groundedness = (\n",
" Feedback(grounded.groundedness_measure_with_cot_reasons, name = \"Groundedness\")\n",
" Feedback(provider.groundedness_measure_with_cot_reasons, name = \"Groundedness\")\n",
" .on(Select.RecordCalls.retrieve.rets.collect())\n",
" .on_output()\n",
" .aggregate(grounded.grounded_statements_aggregator)\n",
")\n",
"\n",
"# Question/answer relevance between overall question and answer.\n",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -440,7 +440,6 @@
],
"source": [
"from trulens_eval import Feedback, Select\n",
"from trulens_eval.feedback import Groundedness\n",
"from trulens_eval.feedback.provider.openai import OpenAI as fOpenAI\n",
"import numpy as np\n",
"\n",
Expand All @@ -455,10 +454,9 @@
"\n",
"# Define a groundedness feedback function\n",
"f_groundedness = (\n",
" Feedback(grounded.groundedness_measure_with_cot_reasons, name = \"Groundedness\", higher_is_better=True)\n",
" Feedback(provider.groundedness_measure_with_cot_reasons, name = \"Groundedness\", higher_is_better=True)\n",
" .on(context.collect())\n",
" .on(output)\n",
" .aggregate(grounded.grounded_statements_aggregator)\n",
")\n",
"\n",
"# Question/answer relevance between overall question and answer.\n",
Expand Down

0 comments on commit c424cd4

Please sign in to comment.