ARC-Challenge (Cloze Likelihood Scoring)
General QA · 2018-03-14
The 1,172-question ARC-Challenge test set scored in the cloze formulation: each answer option's text is scored by its likelihood as a continuation of the question, usually length-normalized, without the options listed in the prompt, as lm-evaluation-harness and the Open LLM Leaderboard v1 do. This runs about 20 points below the multiple-choice formulation behind most ARC-Challenge reports, which lists the options and scores the answer letter (Llama-3-70B: 68.8 against 93.0), so these reports are a fallback component of the ARC family rather than pooled with ARC-Challenge. Reports are filed here when the source states this scoring or its table reproduces Open LLM Leaderboard v1 values.
Benchmark source: arXiv