AlpacaEval 2.0 (raw win rate)
Chat & Writing · 2024-04-01
The raw, uncontrolled AlpacaEval 2.0 win rate against the reference model, judged by GPT-4-Turbo over 805 instructions. This is the weighted rate, averaging the judge's per-comparison preference probability rather than counting discrete wins, so it is a mean of per-item scores over the 805 comparisons rather than a pass count. Carried as a component of the length-controlled factor group: it re-measures the same comparisons without the length adjustment, and scores several points lower for verbose models as a result. A handful of models were judged on 801 to 804 comparisons instead of the full 805 because individual generations failed; the declared count is the full set and each row records its own total in the notes.