VSI-Bench
Multimodal · 2024-12-18
VSI-Bench, from Thinking in Space, tests visual-spatial intelligence from egocentric indoor videos with over 5,000 questions on ScanNet, ScanNet++ and ARKitScenes videos in eight tasks (object counting, absolute and relative distance, object and room size, relative direction, route planning and appearance order). Multiple-choice tasks use exact-match accuracy and numerical tasks use mean relative accuracy over confidence thresholds; the overall is the unweighted mean of the eight task scores. VSI-Bench-Debiased is a separate pruned subset.
Top models (higher is better)
| Model | Score |
|---|---|
| LFM3-VL-2B | 63.3 |
| Qwen3.5-4B | 50.5 |
| GPT-5.6 Luna | 49.2 |
| Qwen3.5-2B | 39.8 |
| Gemini 3.5 Flash-Lite | 36.5 |
| Gemma 4 12B IT | 31.2 |
| Gemma 4 E2B IT | 21.3 |