-
Notifications
You must be signed in to change notification settings - Fork 53
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fixes and adjustment in rag metrics and related inference engines #1466
Conversation
lilacheden
commented
Jan 2, 2025
- add new classification engines and remove obsolete models
- propagate score_prefix from metricPipeline to its metric
- Adjust autorag metrics to unitxt flow with rag.response_generation task
- Add AzureOpenAIInferenceEngine and make rag llmaj use it
…t in bam Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Lilach. Thank you for this work. To complete the work, I think we should have metric links and metrics lists (let's think of the names, I'm not sure about them). metrics.rag.end_to_end.recommended_cpu_only: metrics.rag.end_to_end.recommended_local_llm (can we think of a better name?) And then update the example to use the new metrics. unitxt/examples/evaluate_rag_end_to_end_dataset_with_given_predictions.py |
fba3f2a
to
1e6c3e4
Compare
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>
Signed-off-by: lilacheden <[email protected]>