OpenEQA (episodic memory)
Multimodal · 2024-04-11
OpenEQA's episodic-memory track (EM-EQA) asks 1,636 open-vocabulary questions about 152 indoor episode histories from ScanNet and HM3D, which the model sees as a video or frame sequence before answering. Answers are graded by LLM-Match: an LLM judge rates each answer 1-5 against human references, rescaled to 0-100 and averaged; the canonical judge is GPT-4.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-5.6 Luna | 78.5 |
| Gemini 3.5 Flash-Lite | 77.5 |
| LFM3-VL-2B | 76.3 |
| Qwen3.5-4B | 72.8 |
| Qwen3.5-2B | 63.8 |
| Gemma 4 12B IT | 56.4 |
| Gemma 4 E2B IT | 43.2 |