The OPEA Evaluation Working Group is chartered to identify standardized methodologies and frameworks for evaluating the RAG pipeline, to aid in the benchmarking of the individual components and the end-to-end solution.
The Evaluation will comprise of both Quantitative and Qualitative metrics in the domains of Performance, Safety, Trustworthiness and Scalability.
- Methodology and Eval Frameworks
- Performance – Focus on metrics/KPIs for each component and End to end
- Trustworthiness - Ability to guarantee quality, security, robustness & relevance to Government or other policies
- Scalability / Enterprise Readiness - Ability to be used in production in enterprise environments