TheoremQA
Math · 2023-05-21
TheoremQA is a theorem-driven question-answering benchmark of 800 expert-curated questions covering about 350 theorems from mathematics, physics, electrical engineering and computer science, and finance. Scores are answer accuracy.
Top models (higher is better)
| Model | Score |
|---|---|
| Qwen2 72B Instruct | 44.4 |
| Qwen2.5 32B | 44.1 |
| Qwen2.5-Coder-32B | 43.1 |
| Qwen2.5 14B | 43.0 |
| Llama 3 70B Instruct | 42.5 |
| Qwen2.5 72B | 42.4 |
| Mixtral 8x22B Instruct | 40.8 |
| Qwen2.5-Coder-14B | 39.6 |
| Qwen2.5 7B | 36.0 |
| Mixtral 8x22B | 35.9 |
| Gemma 2 27B | 35.8 |
| Qwen1.5 110B | 34.9 |
| Qwen2.5 Coder 7B | 34.0 |
| Qwen2-57B-A14B | 33.5 |
| Qwen2-57B-A14B Instruct | 33.1 |
| Qwen1.5 32B Chat | 30.9 |
| Qwen2.5-Coder-3B | 29.5 |
| Qwen1.5 72B | 29.3 |
| DeepSeek-Coder-V2-Lite-Base | 29.3 |
| Gemma 2 9B | 28.9 |
| Qwen1.5 32B | 28.8 |
| Qwen1.5 72B Chat | 28.8 |
| Qwen2.5 3B | 27.4 |
| Qwen2 7B Instruct | 25.3 |
| Mixtral 8x7B | 23.2 |
| Llama 3 8B Instruct | 23.0 |
| Llama 3 8B | 22.1 |
| Qwen2.5 1.5B | 22.1 |
| Gemma 7B | 21.5 |
| Qwen2.5 Coder 1.5B | 21.4 |
| StarCoder2-15B | 20.5 |
| Qwen1.5 110B Chat | 18.8 |
| DeepSeek-Coder-Base 33B | 17.5 |
| StarCoder2-7B | 16.0 |
| Qwen2.5 0.5B | 16.0 |
| CodeQwen1.5 7B | 15.8 |
| Qwen2.5 Coder 0.5B | 14.3 |
| Qwen1.5 7B | 14.2 |
| Qwen1.5 7B Chat | 14.1 |
| DeepSeek-Coder-Base 6.7B | 13.6 |