Coding

Name: Coding
Published: 2024-05-29

Code · 2024-05-29

Elo rating on Scale SEAL's coding prompt set across programming tasks and languages.

Model	Score
o1-mini	1237
o3-mini	1137
GPT-4o (2024-11-20)	1132
o1 Preview	1123
Gemini 2.0 Flash Experimental	1111
Gemini Exp-1206	1109
Gemini 2.0 Flash Thinking Experimental 01-21	1108
DeepSeek-R1	1100
o1 (2024-12-17)	1083
Claude 3.5 Sonnet (June 2024)	1079
GPT-4o (2024-08-06)	1045
GPT-4o (2024-05-13)	1036
GPT-4 0125 Preview	1034
Mistral Large 2 (Instruct 2407)	1029
Llama 3.1 405B Instruct	1022
Gemini 1.5 Pro Experimental 0827	1007
Gemini 1.5 Pro Preview 0514	994
GPT-4 Turbo (1106 Preview)	992
DeepSeek-V3	985
Llama 3.2 90B Vision Instruct	984
Opus 3	959
Gemini 1.5 Flash Preview 0514	943
Gemini 1.5 Pro Preview 0409	891
Claude 3 Sonnet	879
Llama 3 70B Instruct	871
Mistral Large 1.0	811
Gemini 1.0 Pro	685
Code Llama Instruct 34B	598