TruthfulQA MC2 (Open LLM Leaderboard v1)
Safety
TruthfulQA's 817 questions in the multiple-true-answer format as scored by the Open LLM Leaderboard v1: the normalized probability a model assigns to the true reference answers, averaged over questions, under the harness's fixed six-example prompt, which the leaderboard lists as 0-shot. It is a diagnostic: base models score 31 to 45 whatever their size (GPT-2 124M 40.7, LLaMA-65B 43.4, GPT-NeoX-20B 31.6), so it tracks preference tuning rather than capability.
Top models (higher is better)
| Model | Score |
|---|---|
| Smaug-72B-v0.1 | 76.7 |
| Solar 10.7B Instruct V1.0 | 71.4 |
| Snorkel-Mistral-PairRM-DPO | 70.9 |
| Mistral 7B Instruct v0.2 | 68.3 |
| Mixtral 8x22B Instruct | 68.1 |
| DBRX Instruct | 67.0 |
| Qwen1.5 32B Chat | 66.9 |
| Qwen1.5 110B Chat | 65.9 |
| Tülu 2 DPO 70B | 65.8 |
| Mixtral 8x7B Instruct | 65.0 |
| Qwen1.5 72B Chat | 63.9 |
| Yi 1.5 34B Chat | 62.2 |
| Llama 3 70B Instruct | 61.8 |
| Nous Hermes 2 Yi 34B | 60.4 |
| Qwen1.5 14B Chat | 60.4 |
| Qwen 72B | 60.2 |
| Phi-3-mini-4k-instruct | 59.9 |
| Qwen1.5 72B | 59.6 |
| Neural Chat 7B v3-1 | 59.5 |
| Stable Beluga 2 | 59.4 |
| Zephyr 7B Alpha | 57.9 |
| Phi-3 Medium 4K Instruct | 57.7 |
| Zephyr 7B Beta | 57.4 |
| Qwen1.5 32B | 57.4 |
| WizardLM-2 7B | 57.0 |
| C4AI Command R+ | 56.3 |
| Mistral 7B Instruct v0.1 | 56.3 |
| Yi-34B | 56.2 |
| Vicuna 33B v1.3 | 56.2 |
| Quyen Pro V0.1 | 55.8 |
| DeepSeek LLM 67B Chat | 55.8 |
| Yi 34B Chat | 55.4 |
| Qwen2-72B | 54.7 |
| Phi-3 Medium 128K Instruct | 54.6 |
| Phi-3 Mini 128K Instruct | 54.1 |
| Qwen1.5 7B Chat | 53.7 |
| Qwen2-7B | 53.1 |
| Llama 2 70B Chat | 52.8 |
| Yi 1.5 9B Chat | 52.7 |
| InternLM 20B | 52.6 |