MM-Vet
Multimodal · 2023-08-04
MM-Vet has 218 open-ended questions that require integrating recognition, OCR, knowledge, language generation, spatial awareness and math; a GPT-4-class judge scores each answer from 0 to 1 and the result is the mean on a 0-100 scale. This row holds release-reported scores under each publisher's harness, with the judge recorded in notes; it is a component of the fixed-harness OpenVLM family (mmvet-vlmevalkit).
Top models (higher is better)
| Model | Score |
|---|---|
| Kimi-VL-A3B-Thinking-2506 | 78.1 |
| Qwen2 VL 72B Instruct | 74.0 |
| Gemini 3.5 Flash-Lite | 73.3 |
| InternVL2.5-78B | 72.3 |
| Gemma 3 27B IT | 71.0 |
| Kimi-VL-A3B-Thinking | 69.5 |
| GPT-4o (2024-05-13) | 69.1 |
| Qwen2.5-VL-7B-Instruct | 67.1 |
| GPT-5.6 Luna | 67.1 |
| GPT-4o Mini | 66.9 |
| Kimi-VL-A3B-Instruct | 66.7 |
| Gemma 3 12B IT | 64.9 |
| Gemma 4 12B IT | 64.1 |
| Qwen2 VL 7B Instruct | 62.0 |
| Qwen2.5-VL-3B-Instruct | 61.8 |
| MiniCPM-o 2.6 | 60.0 |
| MiniCPM-V 2.6 | 60.0 |
| Qwen3.5-4B | 59.9 |
| Llama 3.2 11B Vision Instruct | 57.6 |
| LFM3-VL-2B | 57.4 |
| Qwen3.5-2B | 53.4 |
| Qwen2 VL 2B Instruct | 49.5 |
| LFM2-VL-1.6B | 47.8 |
| Gemma 4 E2B IT | 44.8 |
| LFM2.5-VL-450M | 41.1 |
| LFM2-VL-450M | 33.9 |