WinoGrande (Open LLM Leaderboard v1)
General QA
WinoGrande's 1,267-item validation set as scored by the Open LLM Leaderboard v1: 5-shot under lighteval's port of lm-evaluation-harness, choosing between the two fillers by the log-likelihood of the rest of the sentence. A fixed harness template is a different measurement from the lab-reported runs in winogrande, so it forms its own factor group.
Top models (higher is better)
| Model | Score |
|---|---|
| C4AI Command R+ | 85.4 |
| Llama 3 70B | 85.3 |
| Mixtral 8x22B Instruct | 85.2 |
| Smaug-72B-v0.1 | 85.1 |
| Mixtral 8x22B | 85.0 |
| Qwen2-72B | 84.5 |
| DeepSeek LLM 67B Chat | 84.2 |
| Qwen1.5 110B | 84.1 |
| DeepSeek LLM 67B Base | 84.1 |
| Llama 2 70B | 83.7 |
| Solar 10.7B Instruct V1.0 | 83.6 |
| Tülu 2 DPO 70B | 83.3 |
| Yi-34B | 83.0 |
| Qwen1.5 72B | 83.0 |
| Nous Hermes 2 Mixtral 8x7B SFT | 83.0 |
| Nous Hermes 2 Yi 34B | 83.0 |
| Stable Beluga 2 | 83.0 |
| Llama 3 70B Instruct | 82.9 |
| Qwen 72B | 82.5 |
| DBRX Instruct | 82.1 |
| Mixtral 8x7B | 81.7 |
| Yi 1.5 34B Chat | 81.6 |
| Command R | 81.5 |
| Qwen1.5 32B | 81.3 |
| Falcon 40B | 81.3 |
| Mixtral 8x7B Instruct | 81.1 |
| OpenChat 3.5 1210 | 80.7 |
| Llama 2 70B Chat | 80.5 |
| Yi 34B Chat | 80.1 |
| Qwen1.5 72B Chat | 79.1 |
| Zephyr 7B Alpha | 78.6 |
| Gemma 7B | 78.5 |
| Llama 3 8B | 78.5 |
| Mistral 7B v0.1 Base | 78.4 |
| Falcon2-11B | 78.3 |
| Neural Chat 7B v3-1 | 78.0 |
| Zephyr 7B Beta | 77.7 |
| Snorkel-Mistral-PairRM-DPO | 77.6 |
| Yi-9B | 77.5 |
| Qwen1.5 110B Chat | 77.3 |