MedQA 4-options (Open Medical-LLM)
Science · 2024-04-19
MedQA draws US Medical Licensing Examination style questions. This is the four-option variant's 1,273-question test split, scored by the Open Medical-LLM Leaderboard under lm-evaluation-harness with a fixed prompt. That protocol differs from lab-reported MedQA runs and from the Vals demographic-bias variants Atlas already carries, so it forms its own factor group.
Top models (higher is better)
| Model | Score |
|---|---|
| Llama 3 8B Instruct | 60.7 |
| Llama 3 8B | 59.7 |
| Solar 10.7B Instruct V1.0 | 54.7 |
| Zephyr 7B Beta | 51.5 |
| Mistral 7B v0.1 Base | 50.8 |
| Qwen1.5 7B | 48.9 |
| Gemma 7B | 48.7 |
| Gemma 1.1 7B IT | 45.7 |
| Qwen1.5 7B Chat | 45.4 |
| Mistral 7B Instruct v0.1 | 43.5 |
| Gemma 7B IT | 40.4 |
| Phi-1.5 | 31.1 |
| Gemma 2B | 29.2 |
| Falcon 7B | 27.9 |