MMLU Benchmarks

[gagika]

MMLU Benchmark Script (0-shot)

Tests

Tested for llama2-7b and llama3.1-8b

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed.

[RissyRan]

This is great! Do we have logs to see how the test results looks like?

[gagika]

This is great! Do we have logs to see how the test results looks like?

Here is the output for llama3.1-8b with default 0-shot prompt:

Final accuracy on MMLU dataset: 0.6413

Subcategory Accuracies:
Accuracy for subcategory 'politics': 0.7963
Accuracy for subcategory 'other': 0.6876
Accuracy for subcategory 'culture': 0.8283
Accuracy for subcategory 'history': 0.7860
Accuracy for subcategory 'biology': 0.7819
Accuracy for subcategory 'psychology': 0.7640
Accuracy for subcategory 'physics': 0.5172
Accuracy for subcategory 'engineering': 0.6345
Accuracy for subcategory 'philosophy': 0.5323
Accuracy for subcategory 'health': 0.7000
Accuracy for subcategory 'economics': 0.6415
Accuracy for subcategory 'computer science': 0.5947
Accuracy for subcategory 'law': 0.5309
Accuracy for subcategory 'math': 0.4474
Accuracy for subcategory 'chemistry': 0.5314
Accuracy for subcategory 'business': 0.8101
Accuracy for subcategory 'geography': 0.7727

Category Accuracies:
Accuracy for category 'STEM': 0.5500
Accuracy for category 'humanities': 0.5819
Accuracy for category 'social sciences': 0.7488
Accuracy for category 'other (business, health, misc.)': 0.7104