Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes and adjustment in rag metrics and related inference engines #1466

Merged
merged 33 commits into from
Jan 13, 2025

Conversation

lilacheden
Copy link
Member

  1. add new classification engines and remove obsolete models
  2. propagate score_prefix from metricPipeline to its metric
  3. Adjust autorag metrics to unitxt flow with rag.response_generation task
  4. Add AzureOpenAIInferenceEngine and make rag llmaj use it

@yoavkatz
Copy link
Member

yoavkatz commented Jan 5, 2025

Lilach. Thank you for this work. To complete the work, I think we should have metric links and metrics lists (let's think of the names, I'm not sure about them).

metrics.rag.end_to_end.recommended_cpu_only:
metrics.rag.end_to_end.recommended_cpu_only.faithfulness -> link to recommended metric
metrics.rag.end_to_end.recommended_cpu_only.context_relevance -> link to recommended metric
metrics.rag.end_to_end.recommended_cpu_only.answer_correctness -> link to recommended metric
metrics.rag.end_to_end.recommended_cpu_only.context_correctness -> link to recommended metric
metrics.rag.end_to_end.recommended_cpu_only.answer_relevance -> link to recommended metric

metrics.rag.end_to_end.recommended_local_llm (can we think of a better name?)
...
metrics.rag.end_to_end.recommended_llm_as_judge
....

And then update the example to use the new metrics.

unitxt/examples/evaluate_rag_end_to_end_dataset_with_given_predictions.py

@coveralls
Copy link

coveralls commented Jan 5, 2025

Coverage Status

coverage: 80.088% (+0.03%) from 80.057%
when pulling 056a313 on metrics_fix
into 21482b9 on main.

@lilacheden lilacheden enabled auto-merge (squash) January 13, 2025 13:44
@lilacheden lilacheden merged commit 1350d56 into main Jan 13, 2025
18 of 20 checks passed
@lilacheden lilacheden deleted the metrics_fix branch January 13, 2025 14:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants