FRAMES
Search · 2024-09-19
FRAMES (Factuality, Retrieval, And reasoning MEasurement Set) is Google's 824-question multi-hop benchmark whose answers require combining facts from two to fifteen Wikipedia articles, with numerical, tabular, temporal and constraint reasoning. It is run either as long-context reading over supplied articles or as an agentic task with search tools; tool use is recorded on each row. Scores report answer accuracy. Higher is better.
Top models (higher is better)
| Model | Score |
|---|---|
| GPT-4.1 | 87.4 |
| Kimi K2 Thinking | 87.0 |
| DeepSeek-R1-0528 | 83.0 |
| DeepSeek-R1 | 82.5 |
| DeepSeek-R1-Zero | 82.3 |
| GPT-4o (2024-05-13) | 80.5 |
| DeepSeek-V3-0324 | 79.2 |
| Kimi K2 Instruct | 77.1 |
| o1-mini | 76.9 |
| Sonnet 4 | 76.3 |
| DeepSeek-V3 | 73.3 |
| Claude 3.5 Sonnet (Oct 2024) | 72.5 |
| Llama 3.1 405B Instruct | 70.0 |
| Qwen2.5 72B Instruct | 69.8 |
| DeepSeek-V2-Chat | 66.9 |
| DeepSeek-V2.5 | 65.4 |
| Kimi K2 Instruct 0905 | 58.1 |