ARC-Challenge (Open LLM Leaderboard v1)
General QA
The 1,172-question ARC-Challenge test set as scored by Hugging Face's Open LLM Leaderboard v1: 25-shot under lighteval's port of lm-evaluation-harness, choosing an option by length-normalized log-likelihood (acc_norm). Uniform guessing over each question's own three to five options gives 25.0156427759 percent. A fixed harness template is a different measurement from the lab-reported runs in arc-ai2, so it forms its own factor group.
Top models (higher is better)
| Model | Score |
|---|---|
| Smaug-72B-v0.1 | 76.0 |
| Mixtral 8x22B Instruct | 72.7 |
| Tülu 2 DPO 70B | 72.1 |
| Qwen1.5 110B Chat | 72.0 |
| Llama 3 70B Instruct | 71.4 |
| Stable Beluga 2 | 71.1 |
| Solar 10.7B Instruct V1.0 | 71.1 |
| C4AI Command R+ | 71.0 |
| Mixtral 8x22B | 70.6 |
| Yi 1.5 34B Chat | 70.5 |
| Mixtral 8x7B Instruct | 70.1 |
| Qwen1.5 110B | 70.0 |
| Falcon 180B | 69.8 |
| Nous Hermes 2 Mixtral 8x7B SFT | 69.7 |
| Qwen2-72B | 68.8 |
| Llama 3 70B | 68.8 |
| Qwen1.5 72B Chat | 68.5 |
| DBRX Instruct | 67.8 |
| DeepSeek LLM 67B Chat | 67.7 |
| Llama 2 70B | 67.3 |
| Phi-3 Medium 4K Instruct | 67.3 |
| Nous Hermes 2 Yi 34B | 66.9 |
| Phi-3 Medium 128K Instruct | 66.5 |
| Mixtral 8x7B | 66.4 |
| Neural Chat 7B v3-1 | 66.3 |
| Qwen1.5 32B Chat | 66.0 |
| Snorkel-Mistral-PairRM-DPO | 66.0 |
| Qwen1.5 72B | 65.9 |
| Command R | 65.5 |
| Yi 34B Chat | 65.4 |
| DeepSeek LLM 67B Base | 65.4 |
| Qwen 72B | 65.2 |
| OpenChat 3.5 1210 | 64.9 |
| Yi-34B | 64.6 |
| Llama 2 70B Chat | 64.6 |
| Yi 1.5 9B Chat | 63.7 |
| Qwen1.5 32B | 63.4 |
| Phi-3 Mini 128K Instruct | 63.1 |
| Mistral 7B Instruct v0.2 | 63.1 |
| Phi-3-mini-4k-instruct | 63.0 |