TruLens Eval v0.25.0

What's Changed

• add nemo guardrails integrations by @piotrm0 in #824
• Improve regex matching for structured output extraction from LLM feedback function providers by @daniel-huang-1230 in #954
• Better selection of main input/output by @joshreini1 in #938
• makefile targets for release process by @piotrm0 in #934
• Documentation structure and heading pages by @piotrm0 in #945
• update tru virtual docs by @piotrm0 in #949

Bug Fixes

• Fix release test pipeline by @joshreini1 in #962
• Enable test set generation for llama-index by @joshreini1 in #865
• fix links in docs by @piotrm0 in #963
• adjust all_tools symlink by @joshreini1 in #965
• instrumentation notebook updates and fixes by @piotrm0 in #953

Full Changelog: trulens-eval-0.24.1...trulens-eval-0.25.0

TruLens Eval v0.24.1

Library containing evaluations of LLM Applications

Changelog

• Bugfixes.

Bug Fixes

• Updated langchain and llama_index examples to avoid deprecated or outdated functions.
• Removed unneeded debug statements.

trulens-eval-0.24.0

What's Changed

• cost tracking tests and litellm cost tracking by @piotrm0 in #910
• check packages on init by @piotrm0 in #917
• Increase provider test coverage to Huggingface feedback provider by @venkatkakoju in #919
• upgrade Llama-Index integration to 0.10 by @joshreini1 in #891
• Update issue templates by @joshreini1 in #923
• async handling adjustments by @piotrm0 in #918

New Contributors

• @venkatkakoju made their first contribution in #919

Full Changelog: trulens-eval-0.23.0...trulens-eval-0.24.0

trulens-eval-0.23.0

What's Changed

• Improvements to Comprehensiveness Feedback by @joshreini1 in #901
• Deeper Instrumentation for Hybrid Retrievers by @joshreini1 in #873

Examples

• Evaluating Hybrid and Ensemble Retrievers by @joshreini1 in #873

Bug Fixes

• various documentation fixes by @piotrm0 in #907
• fix forward refs with model_rebuild by @piotrm0 in #905
• cleanup by @piotrm0 in #880
• fix for in-memory sqlite params by @piotrm0 in #904
• Fix use case colab links by @joshreini1 in #900
• few site-related fixes to recently merged pr by @piotrm0 in #903
• Capture multi-line criteria and reasons for COT feedbacks by @joshreini1 in #901

Full Changelog: trulens-eval-0.22.2...trulens-eval-0.23.0

trulens-eval-0.22.2

What's Changed

• Metadata error type validation by @aaronvarghese in #888
• Update prompting to improve agreement_measure reliability by @daniel-huang-1230 in #884
• Temporary pin to llama-index version by @joshreini1 in #893
• Better exceptions optional imports by @piotrm0 in #892
• Fix bug to enable bedrock provider models by @joshreini1 in #887

New Contributors

• @aaronvarghese made their first contribution in #888

Full Changelog: trulens-eval-0.22.1...trulens-eval-0.22.2

trulens-eval-0.22.1

What's Changed

Changelog

• Improved scalability by adding in a non-threaded pacer by @piotrm0 in #874
• Fixed warning due to streamlit params -> query params by @joshreini1 in #860
• Improved deferred evaluation and status checks by @piotrm0 in #807 and #879
• Allow different schemas for Bedrock provider calls by @joshreini1 in #878

Examples

• Randomly run evals based on record_id hash by @joshreini1 in #850

Bugfixes

• Fix bug in cost tracking system by @piotrm0 in #866
• make feedback_results backwards compatible by @piotrm0 in #881
• fix st.query_params by @joshreini1 in #883

Full Changelog: trulens-eval-0.22.0...trulens-eval-0.22.1

trulens-eval-0.22.0

What's Changed

• Releases/rc trulens eval 0.21.0 by @joshreini1 in #830
• factor out common error message from app types by @piotrm0 in #832
• Add user-facing methods to use with custom functions by @joshreini1 in #847
• add testing with older python versions by @piotrm0 in #841
• Add calibration tests for OpenAI by @joshreini1 in #838

Examples

• Update example of usage with AzureOpenAI by @joshreini1 in #847

Bug Fixes

• fix ellipsis issue by @piotrm0 in #840 and #843
• Fix typing issues for stock feedback functions by @joshreini1 in #838

Full Changelog: trulens-eval-0.21.0...trulens-eval-0.22.0

TruLens Eval v0.21.0

What's changed

• Deduplicated sync/async methods by @piotrm0 in #793
• Refactored groundedness methods by @joshreini1 in #801
• Error on deprecated passthrough methods by @piotrm0 in #803
• Virtual models for logging and evaluating existing data by @piotrm0 in #806
• Rename summarization quality to comprehensiveness by @joshreini1 in #816
• Delete long deprecated TruApp and TruDB by @piotrm0 in #817
• Enable async unit tests by @piotrm0 in #831
• Add generation of test cases by @joshreini1 in #705

Examples

• Expand evaluation docs by @joshreini1 in #823 including:
  • Running Feedback Functions
  • Feedback Function Selectors
  • Feedback Function Providers
  • Feedback Implementations
  • Generating Test Cases
  • Feedback Evaluations

Bug Fixes

• Add metadata display and application tag display in UI by @joshreini1 in #797
• Fixed issue with float precision by @joshreini1 in #798
• Fix typo in openai moderation - sexual minors by @joshreini1 in #815
• Include reasoning in summarization eval by @joshreini1 in #815
• Make OpenAI optional by @joshreini1 in #827

New contributors

• @vivekgangasani made their first contribution to update AWS jumpstart examples in #795

Notes

• When feedback mode is set to WITH_APP_THREAD, feedback may be computed more eagerly than expected.

TruLens Eval v0.20.3

What's changed

• Improved system prompt for ground truth evaluations by @daniel-huang-1230 in #782
• Added pydantic validation to feedback functions by @daniel-huang-1230 in #782
• Changed bedrock default model from titan-text-lite to titan-text-express by @joshreini1 in #788
• Documentation on optional packages to use with TruLens by @piotrm0 in #787

Examples

• Update langchain retrieval agents example to show validation of tool selection and tool input, work with openai > 1.0 by @piotrm0 @joshreini1 in #778
• Update AWS examples with changes to Bedrock API, other bug fixes by @joshreini1 in #788

Bug Fixes

• Removed optional imports from utility import (defined by init) by @piotrm0 in #786
• Fix missing f-strings in warning when app_id not yet present in the logging db by @andrewisplinghoff in #790

New contributors

• @andrewisplinghoff made their first contribution in #790

TruLens Eval v0.20.2

What's changed

• Changed bedrock default model to "amazon.titan-text-lite-v1" by @rajib76 in #774
• Documented outstanding tech debt by @piotrm0 in #771

Bug Fixes

• Fix paths to components if an app gets garbage collected by @piotrm0 in #771
• Wrapped openai imports and jupyter notebook related imports in optional contexts by @piotrm0 in #770
• Fix to issubclass by @piotrm0 in #770
• Address issue with select_context where the retriever is hidden by an excluded attribute by @piotrm0 in #768

New contributors

• @rajib76 made their first contribution in #774