HellaSwag (Open LLM Leaderboard v1)
General QA
The 10,042 HellaSwag validation contexts as scored by the Open LLM Leaderboard v1: 10-shot under lighteval's port of lm-evaluation-harness, choosing among four endings by length-normalized log-likelihood (acc_norm). It fixes the scoring protocol that hellaswag-unspecified-scoring-protocol leaves open, and as a fixed harness measurement it forms its own factor group.
Top models (higher is better)
| Model | Score |
|---|---|
| Smaug-72B-v0.1 | 89.3 |
| Mixtral 8x22B Instruct | 89.1 |
| Tülu 2 DPO 70B | 89.0 |
| Falcon 180B | 88.9 |
| DBRX Instruct | 88.8 |
| Mixtral 8x22B | 88.7 |
| C4AI Command R+ | 88.6 |
| Solar 10.7B Instruct V1.0 | 88.2 |
| Llama 3 70B | 88.0 |
| Mixtral 8x7B Instruct | 87.6 |
| Qwen1.5 110B | 87.5 |
| Llama 2 70B | 87.3 |
| Qwen2-72B | 87.3 |
| DeepSeek LLM 67B Base | 87.1 |
| Command R | 87.0 |
| DeepSeek LLM 67B Chat | 86.8 |
| Nous Hermes 2 Mixtral 8x7B SFT | 86.7 |
| Mixtral 8x7B | 86.5 |
| Qwen1.5 72B Chat | 86.4 |
| Stable Beluga 2 | 86.4 |
| Qwen1.5 72B | 86.0 |
| Yi 1.5 34B Chat | 86.0 |
| Qwen 72B | 85.9 |
| Llama 2 70B Chat | 85.9 |
| Phi-3 Medium 4K Instruct | 85.8 |
| Yi-34B | 85.7 |
| Llama 3 70B Instruct | 85.7 |
| Snorkel-Mistral-PairRM-DPO | 85.6 |
| Nous Hermes 2 Yi 34B | 85.5 |
| Qwen1.5 32B Chat | 85.5 |
| Falcon 40B | 85.3 |
| Qwen1.5 32B | 85.0 |
| OpenChat 3.5 1210 | 84.9 |
| Phi-3 Medium 128K Instruct | 84.9 |
| Mistral 7B Instruct v0.2 | 84.9 |
| Qwen1.5 110B Chat | 84.7 |
| Zephyr 7B Beta | 84.4 |
| Falcon 40B Instruct | 84.3 |
| Yi 34B Chat | 84.2 |
| Zephyr 7B Alpha | 84.0 |