InfiniteBench En.MC
General QA · 2024-02-21
The English multiple-choice task of InfiniteBench (longbook_choice_eng): four-option questions about long novels whose key entities have been replaced, averaging about 184K tokens per book, so the answer must come from the supplied text. Scores are accuracy; sources may truncate the book to fit a 128K-token window.
Top models (higher is better)
| Model | Score |
|---|---|
| Llama 3.1 405B Instruct | 83.4 |
| GPT-4o (source unspecified) | 82.5 |
| Llama 3.1 70B Instruct | 78.2 |
| GPT-4 0125 Preview | 72.1 |
| Llama 3.2 3B Instruct | 63.3 |
| Phi-3.5 Mini Instruct | 39.2 |
| Llama 3.2 1B Instruct | 38.0 |