TruLens v1.0.1

Moving to TruLens v1: Reliable and Modular Logging and Evaluation

TruLens v1 is here! Read the announcement blog and check out our migration guide for moving from trulens_eval.

TruLens 1.0 Announcement Blog

TruLens-Eval v0.33.1

Bug Fixes

• Fix for punkt breaking change to groundedness feedback function
• Fix incompatibility with latest pip version

TruLens-Eval v0.32.1

Bug Fixes

• Fix for punkt breaking change to groundedness feedback function
• Fix incompatibility with latest pip version

TruLens-Eval v0.31.1

Bug Fixes

• Fix for punkt breaking change to groundedness feedback function
• Fix incompatibility with latest pip version

TruLens-Eval v0.33.0

What's Changed

• timeouts for wait_for_feedback_results by @sfc-gh-pmardziel in #1267
• TruLens Streamlit components by @sfc-gh-jreini in #1224
• Run the dashboard on an unused port by default by @sfc-gh-jreini in #1280 and @sfc-gh-jreini in #1275

Documentation Updates

• Reflect Snowflake SQLAlchemy Release in "Connect to Snowflake" Docs by @sfc-gh-jreini in #1281
• Update guardrails examples by @sfc-gh-jreini in #1275

Bug Fixes

• Remove duplicated tests by @sfc-gh-dkurokawa in #1283
• fix LlamaIndex streaming response import by @sfc-gh-chu in #1276

Full Changelog: trulens-eval-0.32.0...trulens-eval-0.33.0

TruLens-Eval v0.32.0

What's Changed

• Context filtering guardrails by @sfc-gh-jreini in #1192
• Query optimizations for TruLens dashboard resulting in 4-32x benchmarked speedups by @sfc-gh-chu in #1216
• Logging in Snowflake database by @sfc-gh-chu in #1216
• Snowflake Cortex feedback provider by @sfc-gh-dhuang in #1202
• improve langchain prompting using native messages by @nicoloboschi in #1194
• fix groundedness with no supporting evidence by @nicoloboschi in #1193
• Improve Microsecond support by @sfc-gh-gtokernliang in #1195
• SkipEval exception by @sfc-gh-pmardziel in #1200
• Update pull_request_template.md by @sfc-gh-jreini in #1234
• Use rounding instead of flooring in feedback score extraction by @sfc-gh-dhuang in #1244

Documentation

• Benchmarking Snowflake arctic-instruct feedback function of groundedness by @sfc-gh-dhuang in #1185
• Evaluation Benchmarks Page by @sfc-gh-jreini in #1190
• Documentation for snowflake sqlalchemy implementation by @sfc-gh-chu in #1216*
• Documentation for logging in snowflake database by @sfc-gh-chu in #1216
• Documentation for cortex provider by @sfc-gh-dhuang in #1202

Examples

• Context filtering guardrails added to quickstarts by @sfc-gh-jreini in #1192
• Update Arctic model notebook to use new Cortex provider by @sfc-gh-dhuang in #1202
• New example showing cortex finetuning by @sfc-gh-dhuang in #1202
• show how to add cost/latency/usage details in virtual records by @sfc-gh-jreini in #1197

Bug Fixes

• Enable formatting during PR build. Also format code that wasn't formatted. by @sfc-gh-dkurokawa in #1212
• Fix test cases generation - normalization step for SummEval score by @sfc-gh-dhuang in #1217
• Enable regex to extract floats in score generation by @sfc-gh-dhuang in #1223
• Fix cost tracking in OpenAI and LiteLLM endpoints by @sfc-gh-dhuang in #1228
• remove deprecated legacy caching by @sfc-gh-jreini in #1233
• Remove remaining streamlit legacy caching by @JushBJJ in #1246

trulens-eval-0.31.0

What's Changed

• Parallelize groundedness LLM calls for speedup by @sfc-gh-dhuang in #1180
• Option for quieter deferred evaluation by @epinzur in #1178
• Support for langchain >=0.2.x retrievers via instrumenting the invoke method by @nicoloboschi in #1187

Examples

• ❄️ Snowflake Arctic quickstart by @joshreini1 in #1156

Bug fixes

• Fix a few more old groundedness references + llamaindex agent toolspec import by @daniel-huang-1230 in #1161
• Very minor fix of print statement by @sfc-gh-dhuang in #1173
• Fix sidebar logo formatting by @sfc-gh-chu in https://github.com/truera/trulens/pull/1169\
• [bugfix] prevent stack overflow in jsonify by @piotrm0 in #1176

Full Changelog: trulens-eval-0.30.1...trulens-eval-0.31.0

trulens-eval-0.30.1

What's Changed

• update comprehensiveness by @daniel-huang-1230 and @joshreini1 in #1064
• glossary additions by @piotrm0 in #1144

Bug Fixes

• Add langchain-community to optional requirements by @joshreini1 in #1146
• Checks for use of openai endpoint by @piotrm0 in #1154

Full Changelog: trulens-eval-0.29.0...trulens-eval-0.30.1

TruLens Eval v0.29.0

Breaking Changes

In this release, we re-aligned the groundedness feedback function with other LLM-based feedback functions. It's now faster and easier to define a groundedness feedback function, and can be done with a standard LLM provider rather than importing groundedness on its own. In addition, the custom groundedness aggregation required is now done by default.

Before:

from trulens_eval.feedback.provider.openai import OpenAI
from trulens_eval.feedback import Groundedness

provider = OpenAI() # or any other LLM-based provider
grounded = Groundedness(groundedness_provider=provider)
f_groundedness = (
    Feedback(grounded.groundedness_measure_with_cot_reasons, name = "Groundedness")
    .on(Select.RecordCalls.retrieve.rets.collect())
    .on_output()
    .aggregate(grounded.grounded_statements_aggregator)
)

After:

provider = OpenAI()
f_groundedness = (
    Feedback(provider.groundedness_measure_with_cot_reasons, name = "Groundedness")
    .on(Select.RecordCalls.retrieve.rets.collect())
    .on_output()
)

This change also applies to the NLI-based groundedness feedback function available from the Huggingface provider.

Before:

from trulens_eval.feedback.provider.openai import Huggingface
from trulens_eval.feedback import Groundedness

from trulens_eval.feedback.provider import Huggingface
huggingface_provider = Huggingface()
grounded = Groundedness(groundedness_provider=huggingface_provider)

f_groundedness = (
    Feedback(grounded.groundedness_measure_with_cot_reasons, name = "Groundedness")
    .on(Select.RecordCalls.retrieve.rets.collect())
    .on_output()
    .aggregate(grounded.grounded_statements_aggregator)
)

After:

from trulens_eval.feedback import Feedback
from trulens_eval.feedback.provider.hugs = Huggingface

huggingface_provider = Huggingface()
    
f_groundedness = (
    Feedback(huggingface_provider.groundedness_measure_with_nli, name = "Groundedness")
    .on(Select.RecordCalls.retrieve.rets.collect())
    .on_output()
)

In addition to the change described above, below you can find the full release description.

What's Changed

• update groundedness prompt by @bpmcgough in #1112
• Default names for rag triad utility by @joshreini1 in #1122
• Unify groundedness interface by @joshreini1 in #1135

Bug Fixes

• Fixed bug with trace view initialization when no feedback functions exist by @walnutdust in #1108
• Remove references to running moderation endpoint on AzureOpenAI by @joshreini1 in #1116
• swap rag utility (qs)relevance by @piotrm0 in #1120
• Fix Link in Readme by @timbmg in #1128
• chore: remove unused code cell by @stokedout in #1113
• trurails: update to getattr by @joshreini1 in #1130
• Fix typo in README.md by @eltociear in #1136
• fix rag triad and awaitable calls by @piotrm0 in #1110
• Remove placeholder feedback for asynchronous responses by @arn-tru in #1127
• Stop iteration streams in openai cost tracking by @piotrm0 in #1138

Examples

• Show OSS models (and tracking) in LiteLLM application by @joshreini1 in #1109

New Contributors

• @stokedout made their first contribution in #1113
• @timbmg made their first contribution in #1128
• @bpmcgough made their first contribution in #1112
• @eltociear made their first contribution in #1136

Full Changelog: trulens-eval-0.28.0...trulens-eval-0.29.0

TruLens Eval v0.28.0

What's Changed

• Meta-eval / feedback functions benchmarking notebooks, ranking-based eval utils, and docs update by @daniel-huang-1230 in #991
• App delete functionality added by @arn-tru in #1061
• Added test coverage to langchain provider by @arn-tru in #1062
• Configurable table prefix by @piotrm0 in #971
• Add example systemd service file by @piotrm0 in #1072

Bug fixes

• Queue fixed for python version lower than 3.9 by @arn-tru in #1066
• Fix test-tru by @piotrm0 in #1070
• Removed broken tests by @arn-tru in #1076
• Fix legacy db missing abstract method by @piotrm0 in #1077
• Release test fixes by @piotrm0 in #1078
• Docs fixes by @piotrm0 in #1075

Examples

• MongoDB Atlas quickstart by @joshreini1 in #1056
• OpenAI Assistants API (quickstart) by @joshreini1 in #1041

Full Changelog: trulens-eval-0.27.2...trulens-eval-0.28.0