SimpleQA F1
General QA · 2024-10-30
OpenAI's SimpleQA set of 4,326 short fact-seeking questions, scored by the F1 aggregate that OpenAI's simple-evals also reports: the harmonic mean of overall accuracy and accuracy conditional on attempted answers, on a 0-100 scale. Google's Gemini 2.5 technical report states that its SimpleQA figures use this metric. It rewards abstaining over guessing, so it is distinct from the percentage-correct metric and cannot be treated as 4,326 binary successes.