MMLU-Pro (Open LLM Leaderboard v2)
General QA · 2024-06-26
MMLU-Pro extends MMLU to up to ten answer options with expert review to reduce noise. The Open LLM harness scores only each question's actual options; uniform guessing on the 12,032-question test set gives 11.087694718844986 percent at official dataset revision b189ec765aa7ed75c8acfea42df31fdae71f97be. Scored by HuggingFace's Open LLM Leaderboard v2 under lm-evaluation-harness with a fixed prompt template and no chain-of-thought, which is a different measurement protocol from lab-reported and Artificial Analysis runs, so it forms its own factor group. Raw micro-accuracy is imported, not the leaderboard's baseline-normalized column.