LLM Chess Elo

Games · 2025-12-01

LLM Chess (Maxim Saplin and co-authors) has a language model play Black in an agentic multi-turn dialog against a Random Player or the Komodo Dragon chess engine. Each move is a dialog of actions (get_current_board, get_legal_moves, make_move in UCI notation); a model that makes too many illegal moves or actions in one move's dialog (the limit is three) loses the game, and games stop at 200 moves. The Elo is a one-dimensional maximum-likelihood estimate over aggregated (opponent Elo, wins, draws, losses) blocks, anchored to Dragon skill levels at Elo = 125 × (level + 1) on the chess.com scale, with the Random Player calibrated against Dragon first; it is reported with a 95% confidence interval, and the number of games varies by model. It is an unbounded scale of its own and is not comparable with the chessbench.ai Elo (chessbench-ai-elo), which uses a different harness, opponents and anchors.

Benchmark source: GitHub

Top models (higher is better)

Model	Score	Source
GPT-6 Astra	1614	LLM Chess leaderboard: gpt-6-astra-high · LLM Chess Authors
GPT-6.1 Sol	1612	LLM Chess leaderboard: gpt-6.1-sol-xhigh · LLM Chess Authors
Gemini 4 Argon	1609	LLM Chess leaderboard: gemini-4-argon-high · LLM Chess Authors
Claude Opus 5.5	1575	LLM Chess leaderboard: claude-opus-5-5-high · LLM Chess Authors
GPT-5.6 Sol	1550	LLM Chess leaderboard: gpt-5.6-sol-xhigh · LLM Chess Authors
Gemini 3.8 Flash	1546	LLM Chess leaderboard: gemini-3.8-flash-high · LLM Chess Authors
GPT-6 Sol	1545	LLM Chess leaderboard: gpt-6-sol-xhigh · LLM Chess Authors
GPT-5.5	1532	LLM Chess leaderboard: gpt-5.5-medium · LLM Chess Authors
Gemini 3.1 Pro Preview	1511	LLM Chess leaderboard: gemini-3.1-pro-preview-high · LLM Chess Authors
GPT-5.4	1322	LLM Chess leaderboard: gpt-5.4-high · LLM Chess Authors
Mercury 3	1308	LLM Chess leaderboard: inception-mercury-3-high · LLM Chess Authors
GPT-5.6 Terra	1293	LLM Chess leaderboard: gpt-5.6-terra-xhigh · LLM Chess Authors
Claude Opus 5	1285	LLM Chess leaderboard: claude-opus-5_adaptive-thinking-high · LLM Chess Authors
GPT-6 Luna	1219	LLM Chess leaderboard: gpt-6-luna-max · LLM Chess Authors
Claude Sonnet 5.5	1195	LLM Chess leaderboard: claude-sonnet-5-5-high · LLM Chess Authors
GPT-5.6 Luna	1185	LLM Chess leaderboard: gpt-5.6-luna-xhigh · LLM Chess Authors
Claude Haiku 5.5	1143	LLM Chess leaderboard: claude-haiku-5-5-high · LLM Chess Authors
Gemini 3.7 Flash	1107	LLM Chess leaderboard: gemini-3.7-flash-high · LLM Chess Authors
Gemini 3.6 Flash	1094	LLM Chess leaderboard: gemini-3.6-flash-high · LLM Chess Authors
DeepSeek V4.1 Flash	1087	LLM Chess leaderboard: deepseek-v4.1-flash-2026-09-10-high · LLM Chess Authors
GPT-5	1086	LLM Chess leaderboard: gpt-5-medium · LLM Chess Authors
GPT-5.2	1085	LLM Chess leaderboard: gpt-5.2-medium · LLM Chess Authors
GPT-5.3-Codex	1053	LLM Chess leaderboard: gpt-5.3-codex-medium · LLM Chess Authors
Gemini 3 Pro Preview	1034	LLM Chess leaderboard: gemini-3-pro-preview-high · LLM Chess Authors
Gemini 3.5 Flash	953	LLM Chess leaderboard: gemini-3.5-flash-medium · LLM Chess Authors
GPT-5.1	948	LLM Chess leaderboard: gpt-5.1-medium · LLM Chess Authors
Grok 4.20	843	LLM Chess leaderboard: grok-4-20-reasoning · LLM Chess Authors
Grok 4.7	822	LLM Chess leaderboard: grok-4.7-high · LLM Chess Authors
Sonnet 5	806	LLM Chess leaderboard: claude-sonnet-5-high · LLM Chess Authors
Grok 4.1 Fast	800	LLM Chess leaderboard: grok-4-1-fast-reasoning · LLM Chess Authors
o3	778	LLM Chess leaderboard: o3-medium · LLM Chess Authors
GPT-5 Mini	759	LLM Chess leaderboard: gpt-5-mini-high · LLM Chess Authors
Grok 4 Fast	735	LLM Chess leaderboard: grok-4-fast-reasoning · LLM Chess Authors
Opus 4.7	691	LLM Chess leaderboard: claude-opus-4-7-high · LLM Chess Authors
GPT-5 Nano	683	LLM Chess leaderboard: gpt-5-nano-high · LLM Chess Authors
GLM-5.3	605	LLM Chess leaderboard: glm-5.3-high · LLM Chess Authors
GLM-5.3 Flash	572	LLM Chess leaderboard: glm-5.3-flash-high · LLM Chess Authors
GPT-5.1-Codex mini	544	LLM Chess leaderboard: gpt-5.1-codex-mini-high · LLM Chess Authors
GPT-5.1-Codex	504	LLM Chess leaderboard: gpt-5.1-codex-low · LLM Chess Authors
Opus 4.8	450	LLM Chess leaderboard: claude-opus-4-8-high · LLM Chess Authors