InfiniteBench En.QA

Name: InfiniteBench En.QA
Published: 2024-02-21

General QA · 2024-02-21

The English free-form question-answering task of InfiniteBench (longbook_qa_eng): questions about long novels with replaced key entities, averaging about 192K tokens per book. Answers are scored by token-level F1 against the reference; sources may truncate the book to fit a 128K-token window.

Top models (higher is better)

Model	Score
Llama 3.1 70B Instruct	36.7
Llama 3.1 405B Instruct	30.5
Llama 3.2 1B Instruct	20.3
Llama 3.2 3B Instruct	19.8
GPT-4o (source unspecified)	19.1
GPT-4 0125 Preview	15.7
Claude 3.5 Sonnet (June 2024)	11.3
Phi-3.5 Mini Instruct	11.3