OpenBookQA (HELM Lite, 5-shot multiple choice)
Science · 2023-12-19
HELM Lite's OpenbookQA run: the 500-question OpenBookQA test split posed five-shot as a single-letter multiple-choice question (HELM method multiple_choice_joint, temperature 0, one output token) and scored by exact match. Asked this way, capable models reach the mid-90s, while likelihood scoring of the same questions plateaus near 50%, so this protocol is its own factor group.