← All benchmarks

GSM1k (paper LM Evaluation Harness protocol)

Math · 2024-05-01

GSM1k as scored in Scale AI's GSM1k paper (arXiv 2405.00332v4, the NeurIPS 2024 Datasets and Benchmarks camera-ready, Appendix F, standard prompt). The paper uses a fork of EleutherAI's LM Evaluation Harness at default settings except a 1,000-token generation limit, five examples drawn at random from the GSM8K train set, and the last number in the response compared with the answer, so answers that do not follow the five-shot format are marked wrong; open models ran at temperature 0 under vLLM or Hugging Face and API models through LiteLLM at default settings between April 16 and July 10, 2024. The strict extraction puts these scores well below Scale's leaderboard values for the same models (Claude 3 Opus 82.4 against 95.19), so they form their own family rather than joining gsm1k. The paper describes 1,250 problems in v1 to v3 and 1,205 in v4, so no item count is asserted.

Top models (higher is better)