MedMCQA (Open Medical-LLM)
Science · 2024-04-19
MedMCQA collects Indian medical entrance-exam questions across 21 subjects. The Open Medical-LLM Leaderboard scores the 4,183-question validation split under lm-evaluation-harness with a fixed prompt, a different protocol from lab-reported runs, so it forms its own factor group. Four answer options give a 25 percent guessing floor.
Top models (higher is better)
| Model | Score |
|---|---|
| Llama 3 8B | 57.5 |
| Llama 3 8B Instruct | 56.8 |
| Qwen1.5 7B | 50.2 |
| Gemma 7B | 49.3 |
| Mistral 7B v0.1 Base | 48.2 |
| Solar 10.7B Instruct V1.0 | 47.5 |
| Zephyr 7B Beta | 45.7 |
| Qwen1.5 7B Chat | 45.3 |
| Gemma 1.1 7B IT | 44.4 |
| Gemma 7B IT | 42.2 |
| Mistral 7B Instruct v0.1 | 41.2 |
| Phi-1.5 | 34.3 |
| Gemma 2B | 30.2 |
| Falcon 7B | 27.2 |