TruLens Eval v0.29.0

Breaking Changes

In this release, we re-aligned the groundedness feedback function with other LLM-based feedback functions. It's now faster and easier to define a groundedness feedback function, and can be done with a standard LLM provider rather than importing groundedness on its own. In addition, the custom groundedness aggregation required is now done by default.

Before:

from trulens_eval.feedback.provider.openai import OpenAI
from trulens_eval.feedback import Groundedness

provider = OpenAI() # or any other LLM-based provider
grounded = Groundedness(groundedness_provider=provider)
f_groundedness = (
    Feedback(grounded.groundedness_measure_with_cot_reasons, name = "Groundedness")
    .on(Select.RecordCalls.retrieve.rets.collect())
    .on_output()
    .aggregate(grounded.grounded_statements_aggregator)
)

After:

provider = OpenAI()
f_groundedness = (
    Feedback(provider.groundedness_measure_with_cot_reasons, name = "Groundedness")
    .on(Select.RecordCalls.retrieve.rets.collect())
    .on_output()
)

This change also applies to the NLI-based groundedness feedback function available from the Huggingface provider.

Before:

from trulens_eval.feedback.provider.openai import Huggingface
from trulens_eval.feedback import Groundedness

from trulens_eval.feedback.provider import Huggingface
huggingface_provider = Huggingface()
grounded = Groundedness(groundedness_provider=huggingface_provider)

f_groundedness = (
    Feedback(grounded.groundedness_measure_with_cot_reasons, name = "Groundedness")
    .on(Select.RecordCalls.retrieve.rets.collect())
    .on_output()
    .aggregate(grounded.grounded_statements_aggregator)
)

After:

from trulens_eval.feedback import Feedback
from trulens_eval.feedback.provider.hugs = Huggingface

huggingface_provider = Huggingface()
    
f_groundedness = (
    Feedback(huggingface_provider.groundedness_measure_with_nli, name = "Groundedness")
    .on(Select.RecordCalls.retrieve.rets.collect())
    .on_output()
)

In addition to the change described above, below you can find the full release description.

What's Changed

• update groundedness prompt by @bpmcgough in #1112
• Default names for rag triad utility by @joshreini1 in #1122
• Unify groundedness interface by @joshreini1 in #1135

Bug Fixes

• Fixed bug with trace view initialization when no feedback functions exist by @walnutdust in #1108
• Remove references to running moderation endpoint on AzureOpenAI by @joshreini1 in #1116
• swap rag utility (qs)relevance by @piotrm0 in #1120
• Fix Link in Readme by @timbmg in #1128
• chore: remove unused code cell by @stokedout in #1113
• trurails: update to getattr by @joshreini1 in #1130
• Fix typo in README.md by @eltociear in #1136
• fix rag triad and awaitable calls by @piotrm0 in #1110
• Remove placeholder feedback for asynchronous responses by @arn-tru in #1127
• Stop iteration streams in openai cost tracking by @piotrm0 in #1138

Examples

• Show OSS models (and tracking) in LiteLLM application by @joshreini1 in #1109

New Contributors

• @stokedout made their first contribution in #1113
• @timbmg made their first contribution in #1128
• @bpmcgough made their first contribution in #1112
• @eltociear made their first contribution in #1136

Full Changelog: trulens-eval-0.28.0...trulens-eval-0.29.0

TruLens Eval v0.28.0

What's Changed

• Meta-eval / feedback functions benchmarking notebooks, ranking-based eval utils, and docs update by @daniel-huang-1230 in #991
• App delete functionality added by @arn-tru in #1061
• Added test coverage to langchain provider by @arn-tru in #1062
• Configurable table prefix by @piotrm0 in #971
• Add example systemd service file by @piotrm0 in #1072

Bug fixes

• Queue fixed for python version lower than 3.9 by @arn-tru in #1066
• Fix test-tru by @piotrm0 in #1070
• Removed broken tests by @arn-tru in #1076
• Fix legacy db missing abstract method by @piotrm0 in #1077
• Release test fixes by @piotrm0 in #1078
• Docs fixes by @piotrm0 in #1075

Examples

• MongoDB Atlas quickstart by @joshreini1 in #1056
• OpenAI Assistants API (quickstart) by @joshreini1 in #1041

Full Changelog: trulens-eval-0.27.2...trulens-eval-0.28.0

trulens-eval-0.27.2

Bug Fix

• add missing pprint import by @joshreini1 in #1054

Full Changelog: trulens-eval-0.27.1...trulens-eval-0.27.2

trulens-eval-0.27.1

What's changed

• Add if_missing. by @piotrm0 in #1038
• Added feedback button to trulens by @arn-tru in #1046

Documentation updates

• pipelines readme by @piotrm0 in #1030
• docs | standards on proper names by @markdavidmc0 in #997
• docs glossary by @piotrm0 in #1029
• Fix TruLens docs link in hybrid retriever notebook by @daniel-huang-1230 in #1035
• docs README by @joshreini1 in #1034
• docs: fix typo by @nicoloboschi in #1036
• more pipelines docs by @piotrm0 in #1033
• Fix azure docs pipeline by @joshreini1 in #1037
• Docs updates for feedback, instrumentation apis, examples by @joshreini1 in #1032
• Proper names and glossary expansion in docs by @piotrm0 in #1042

Bug fixes

• Import improvements, fix version conflicts by @joshreini1 in #1047
• Fix import and favicon by @arn-tru in #1049
• remove pkg_resources and distutils by @piotrm0 in #1052
• pin streamlit-aggrid version by @piotrm0 in #1043

New Contributors

• @markdavidmc0 made their first contribution in #997
• @nicoloboschi made their first contribution in #1036

Full Changelog: trulens-eval-0.27.0...trulens-eval-0.27.1

trulens-eval-0.27.0

What's Changed

• Python 3.12 support by @joshreini1 in #1012
• Design guidelines for contributors @piotrm0 in #1015
• Pull request template by @piotrm0 in #1021
• Handle utf8 encoding issues in trulens database @arn-tru in #1023
• Split system and user prompts for feedback functions by @joshreini1 in #1018
• Enable Meta and Mistral models on AWS Bedrock by @joshreini1 in #1018
• Added support for Langchain MultiQueryRetriever by @sayedsohan in #1014
• Add Vectara Hallucination Detection Model by @Josephrp in #950
• Parametrize temperature for create chat completion by @daniel-huang-1230 in #1026

Examples

• Update LiteLLM quickstart to show TogetherAI model usage by @joshreini1 in #1018
• Add Claude-3 as a feedback provider example by @joshreini1 in #1018
• Notebook to show evaluation of Langchain MultiQueryRetriever by @sayedsohan in #1014
• Added example to show usage of Vectara Hallucination Detection Model by @Josephrp in #950

New Contributors

• @arn-tru made their first contribution in #1023
• @sayedsohan made their first contribution in #1014
• @Josephrp made their first contribution in #950

Full Changelog: trulens-eval-0.26.0...trulens-eval-0.27.0

trulens-eval-0.26.0

What's Changed

• QS Relevance -> Context Relevance by @joshreini1 in #977
• Verify feedback selectors on recorder init by @piotrm0 in #961
• Relax llama version by @joshreini1 in #985
• Allow VirtualRecords to have multiple calls to the same component. by @piotrm0 in #988
• Allow Feedback.run with args even if they had selectors specified. by @piotrm0 in #1003

Documentation

• update doc-building requirements by @piotrm0 in #990
• docs updates/additions by @piotrm0 in #996
• Update feedback docs by @joshreini1 in #999
• doc usage formatting by @piotrm0 in #1002

Examples

• Existing data quickstart by @joshreini1 in #976
• Adds Azure Quickstart for LangChain by @ingridstevens in #984

Bug Fixes

• fix more docs links by @piotrm0 in #987
• Fix broken colab links by @joshreini1 in #994

Full Changelog: trulens-eval-0.25.1...trulens-eval-0.26.0

TruLens Eval v0.25.1

What's Changed

• Examples and Bug Fixes

Examples

• canopy quickstart by @joshreini1 in #925
• add virtual app setup by @joshreini1 in #967

Bug fixes

• Fix release pipeline: allow deserialization for faiss example by @joshreini1 in #978
• Fix TruChain instrumentation issue introduced in 0.24.1 by @joshreini1 in #974
• fix links in docs by @piotrm0 in #968
• Update typo in pip install on azure_openai.ipynb by @ingridstevens in #973
• fix colab link - langchain ensemble notebook by @joshreini1 in #980

Full Changelog: trulens-eval-0.25.0...trulens-eval-0.25.1

TruLens Eval v0.25.0

What's Changed

• add nemo guardrails integrations by @piotrm0 in #824
• Improve regex matching for structured output extraction from LLM feedback function providers by @daniel-huang-1230 in #954
• Better selection of main input/output by @joshreini1 in #938
• makefile targets for release process by @piotrm0 in #934
• Documentation structure and heading pages by @piotrm0 in #945
• update tru virtual docs by @piotrm0 in #949

Bug Fixes

• Fix release test pipeline by @joshreini1 in #962
• Enable test set generation for llama-index by @joshreini1 in #865
• fix links in docs by @piotrm0 in #963
• adjust all_tools symlink by @joshreini1 in #965
• instrumentation notebook updates and fixes by @piotrm0 in #953

Full Changelog: trulens-eval-0.24.1...trulens-eval-0.25.0

TruLens Eval v0.24.1

Library containing evaluations of LLM Applications

Changelog

• Bugfixes.

Bug Fixes

• Updated langchain and llama_index examples to avoid deprecated or outdated functions.
• Removed unneeded debug statements.

trulens-eval-0.24.0

What's Changed

• cost tracking tests and litellm cost tracking by @piotrm0 in #910
• check packages on init by @piotrm0 in #917
• Increase provider test coverage to Huggingface feedback provider by @venkatkakoju in #919
• upgrade Llama-Index integration to 0.10 by @joshreini1 in #891
• Update issue templates by @joshreini1 in #923
• async handling adjustments by @piotrm0 in #918

New Contributors

• @venkatkakoju made their first contribution in #919

Full Changelog: trulens-eval-0.23.0...trulens-eval-0.24.0