Atlas

Compare Models

GPT-6 Sol vs Grok 4.7

GPT-6 Sol leads by 3.2 capability points (≈95% interval +0.9 to +5.4): a clear lead over Grok 4.7 · P(higher) ≈ 100%

Wins 138 of 218 shared benchmarks (60 lost, 20 tied)

Capability vs. relative cost
GPT-6 SolGrok 4.7Closed weightsOpen weights

Each line runs through one model’s priced reasoning settings, cheapest to most expensive; grey points are other models’ priced settings in the same range. Results whose reasoning effort the source didn’t report are left out wherever a model has named effort levels. Capability and relative cost are the published estimates (relative cost is cost per task as a multiple of GPT-5 (high)); hover a setting, or use the arrow keys, to set it beside the other model’s same setting.

GPT-6 SolvsGrok 4.7capability vs. relative costData Nov 15, 2026Village LLM Atlasatlas.village.dev

Benchmark wins138 – 60 · 218 shared, 20 tied

Comparing matching setups only (106 benchmarks excluded) · suites counted once
← GPT-6 SolGrok 4.7 →

GPT-6 SolvsGrok 4.7wins by categoryData Nov 15, 2026Data: Vals AI, Artificial Analysis, Mercor and 46 othersVillage LLM Atlasatlas.village.dev

Margins

Each benchmark scaled 0–100% (its declared range, else the field’s range); Δ is Grok 4.7 minus GPT-6 Sol.

Vending-Bench Arena: Round 13Games105307930−100.0
CyberGym-E2E-AA (Artificial Analysis)Cybersecurity0.074.0+74.0
FrontierMath Tier 4 (v2) - PrivateMath90.017.1−72.9
ProofBench v1.1Math83.026.0−57.0
BrokenArXiv 05/2026Math87.040.0−47.0
ArXivMath 08/2026 (Native Harnesses)Math93.046.5−46.5
BrokenArXiv 08/2026 (Native Harnesses)Math78.635.1−43.4
BrokenArXiv 06/2026Math95.452.8−42.6
AA-PersonalityBench Expressiveness indexChat & Writing33.475.3+42.3
Harvey's Legal Agent Benchmark - Capital MarketsLegal0.040.0+40.0
ChartographyMultimodal53.614.7−38.9
Furniture Assembly BenchmarkMultimodal58.320.8−37.5
Olam Social Poker EloGames15421508−37.0
FrontierMath Tiers 1-3 (v2) - PrivateMath89.853.0−36.8
LAB-Bench LitQA2 (SecureBio)Science51.816.9−34.9
DeepSWE v1.1Code66.633.3−33.3
AA-PersonalityBench Conviction indexChat & Writing50.582.9+32.7
Terminal-Bench 2.1 Extended (Mercor held-out)Code42.411.8−30.6
DeepsecBench — Revalidated V1Code40.910.9−30.0
DeepsecBench — Revalidated V1 (233 issues)Code40.710.9−29.9
LAB-Bench ProtocolQA (SecureBio)Science81.151.4−29.7
LLM Chess EloGames1545822−29.7
BuseyBench SVGMultimodal7.54.6−29.6
ArXivMath 05/2026Math90.060.8−29.2
ALE-BenchCode26371823−28.8
ARC-AGI-2General QA89.661.4−28.2
Mystery Game PuzzlesGames56.029.0−27.0
Toloka Arena (v3)Agents58.031.0−27.0
SRE Bench — Fully SolvedCode30.53.8−26.7
EnterpriseBench: CoreCraftProfessional Work43.168.2+25.1
IOI (Vals v2)Code82.657.7−24.9
BrowseComp Extended (Mercor held-out)Search63.239.5−23.7
Vending-Bench 2Games1442810537−23.6
AA-PersonalityBench Informativeness indexChat & Writing10.533.4+23.1
Humanity's Last Exam DiamondGeneral QA45.223.6−21.6
ArXivMath 06/2026Math91.070.1−20.8
SM BenchSafety66.086.3+20.3
AA-3D v1.0 EloCode1131992−20.2
TaxCalcBench TY25Finance & Accounting28.08.0−20.0
Harvey's Legal Agent Benchmark - Antitrust CompetitionLegal0.020.0+20.0
Harvey's Legal Agent Benchmark - Corporate M&ALegal0.020.0+20.0
Harvey's Legal Agent Benchmark - Energy Natural ResourcesLegal0.020.0+20.0
Harvey's Legal Agent Benchmark - Funds Asset ManagementLegal0.020.0+20.0
Harvey's Legal Agent Benchmark - Healthcare Life SciencesLegal0.020.0+20.0
Harvey's Legal Agent Benchmark - Intellectual PropertyLegal0.020.0+20.0
Harvey's Legal Agent Benchmark - Litigation Dispute ResolutionLegal0.020.0+20.0
Terminal-Bench Science 0.1 (Vals)Science31.411.4−20.0
Artificial Analysis Cyber Index v1Indexes36.856.4+19.6
DeepsecBench-AA (Artificial Analysis)Cybersecurity46.326.9−19.4
AA-PersonalityBench Transparency indexChat & Writing39.158.1+19.2
HardGeoBench Avg DistanceMultimodal14972104−18.7
CritPt v2 (agentic, avg@5)Science80.962.3−18.6
LAB-Bench CloningScenarios (SecureBio)Science98.580.1−18.4
Legal Research BenchLegal28.847.1+18.3
Short Form Bench v1.0Games23942578+18.2
AA-PersonalityBench Organization indexChat & Writing12.429.6+17.3
GeneBench-Pro Public Release SubsetScience26.010.0−16.0
MedXpertQA MM Extended (Mercor held-out)Health & Medicine68.052.0−16.0
SWE Atlas Codebase QnA Extended (Mercor held-out)Code31.315.3−16.0
SWE-bench Verified Extended (Mercor held-out)Code39.355.3+16.0
Tax Agent BenchFinance & Accounting53.069.0+16.0
LMArena Agent Arena Net ImprovementAgents8.84.0−15.8
DistressBenchSafety68.052.4−15.6
Harvey LAB (Mercor run, undisclosed ~213-task cohort)Legal4.119.5+15.4
KernelBench Mega Normalized (RTX PRO 6000)Code2.718.0+15.3
AA-Briefcase v1.1 EloAgents13591636+15.2
Chess Puzzles (task version 2.0)Games56.041.0−15.0
MBCT (SecureBio)Science61.146.1−15.0
Bio Capabilities Index (SecureBio)Indexes161141−14.8
HPCT (SecureBio)Science67.853.7−14.1
GBENCH One-shot Coding GScoreGames70.156.3−13.8
BrowseComp (Mercor 130-question subset)Search89.475.8−13.6
ChessBench EloGames13031076−13.5
Roboflow Vision Evals (October 2026, five tasks)Multimodal76.963.5−13.4
Extended NYT Connections 940 (Quadratic v1)Games90.176.8−13.3
CritPt (avg@5)Science30.918.0−12.9
MentalHealthBenchSafety53.941.3−12.6
Code MigrationCode57.244.8−12.4
MedXpertQA MM (200-question set)Health & Medicine81.369.0−12.3
CadQueryEval (Corrected Geometry)Code100.088.0−12.0
DualEntry Accounting AI BenchmarkFinance & Accounting76.288.1+11.9
Financial Coaching BenchFinance & Accounting62.674.1+11.5
WeirdML v3Science19.78.5−11.3
Ruby LLM Benchmarks — Program FixerCode79.068.0−11.1
GDPval-AA v2.1Professional Work14661717+11.1
Conceptual Reasoning IndexIndexes71.063.1−10.9
τ³-Banking (Mercor run, bm25_grep retrieval)Agents43.032.6−10.4
Roboflow Vision Evals (September 2026 protocol)Multimodal82.371.9−10.4
Vals Science IndexIndexes45.034.8−10.2
HardGeoBench <=100miMultimodal37.527.5−10.0
LMCAGeneral QA59.149.4−9.7
AA-PersonalityBench Warmth indexChat & Writing54.344.8−9.6
Clipping Bench v1.0Games1240384−9.2
Public Benefits Bench v1.1Professional Work56.665.6+9.0
HardGeoBench <=10miMultimodal26.317.5−8.7
Artificial Analysis Intelligence Index v5.0.1Indexes44.235.5−8.7
CyberBench v1.1 OverallCybersecurity78.069.5−8.5
LegalReasonBenchLegal31.940.4+8.5
APEX-SWECode45.053.4+8.4
ACCoRDGeneral QA82.474.0−8.4
LMArena Agent Arena Net Improvement (Steering Burden)Agents6.92.6−8.2
WellnessBenchSafety56.048.0−8.0
YC-BenchGames14417931666389+7.8
LiveBench 2.0Agents81.974.1−7.8
AA-PersonalityBench Slop indexChat & Writing1.08.6+7.7
ITBench-AAAgents49.442.1−7.4
MedScribeHealth & Medicine82.089.4+7.3
VCT v2 (SecureBio)Science49.342.0−7.3
GameBench 2Games79.472.3−7.1
LMArena Agent Arena SteerabilityAgents14.50.5−7.0
Harvey HLAB — Criteria Pass RateLegal83.590.5+7.0
FrontierCode 2.0 (Artificial Analysis)Code35.628.7−6.9
Epoch Capabilities Index (ECI)Indexes164156−6.4
GPQA-Bio (SecureBio)Science88.982.5−6.4
MetagenomicsBench V0Science42.048.3+6.3
Terminal-Bench 4.0Code43.837.6−6.2
LMArena Agent Arena Steering BurdenAgents10.1-2.3−6.2
Terminal-Bench-Science 0.1Science17.611.4−6.2
DeepResearch Bench II (Mercor 130-task set)Search42.949.0+6.1
HealthBenchHealth & Medicine48.254.2+5.9
Harvey LAB-AA v1.1 (hallucination-gated)Legal3.69.4+5.8
AA-PersonalityBench Imagination indexChat & Writing29.635.3+5.8
Creative Writing v3 EloChat & Writing21262012−5.8
CharXiv Extended (Mercor held-out)Multimodal83.878.2−5.6
BioMysteryBench (Vals Terminus 2)Science74.869.3−5.6
ARC-AGI-1 Semi-PrivateGeneral QA95.590.2−5.3
GDP.xlsxProfessional Work17.122.3+5.2
DAYJOB: FinanceFinance & Accounting9.314.5+5.2
MysteryMechanismScience30.225.2−5.0
Arena Alignment IndexSafety87.682.7−4.9
DAYJOB: HealthcareHealth & Medicine3.68.4+4.8
Hermes IndexIndexes44.139.3−4.8
SimpleQA Verified (Epoch anti-abstention)General QA60.756.0−4.7
Terminal-Bench 4.0 (Artificial Analysis)Code30.325.8−4.5
Excel Modeling BenchmarkFinance & Accounting71.567.0−4.5
SWE-TogetherCode60.665.1+4.5
APEX-Accounting 1.1Finance & Accounting59.755.3−4.4
Blueprint-Bench 2Multimodal0.40.3−4.4
MMMU ProMultimodal82.578.2−4.4
CWE-Bench-AA (Artificial Analysis)Cybersecurity64.268.3+4.2
SAGEMath44.840.8−4.0
Eval Connections One-Shot with TrapsGames89.085.0−4.0
Veracode GenAI Code Security (Fall 2026)Cybersecurity73.069.0−4.0
Artificial Analysis Progress Index v1Indexes334323−4.0
AutomationBench-AA 1.0.6Agents61.765.6+3.9
Design Arena Overall Code CategoriesCode12931270−3.8
AA-LCR v1.1Search83.780.0−3.7
ChessBench AccuracyGames77.474.0−3.6
LMArena Agent Arena Praise vs ComplaintAgents10.03.1−3.5
Finance Agent v2Finance & Accounting49.052.3+3.2
Humanity's Last Exam (Text-Only)General QA46.343.1−3.2
KernelBench CUDA Normalized (RTX PRO 6000)Code42.045.2+3.1
CharXiv (descriptive and reasoning combined)Multimodal95.092.0−3.0
SWE Atlas - Codebase QnACode44.147.0+2.9
WCB (SecureBio)Science60.557.6−2.9
LMArena Text Arena EloChat & Writing14571442−2.6
Vals Index v2.1Indexes57.554.9−2.6
HardGeoBench Country AccuracyMultimodal46.343.8−2.5
SWE-bench MultilingualCode81.784.2+2.5
MedCodeHealth & Medicine47.149.6+2.5
AA-Omniscience IndexIndexes27.132.0+2.5
ARC-AGI-3 (Semi-Private)General QA9.57.0−2.4
Vals Index v2Indexes62.660.2−2.4
CVDP (517-problem set)Code52.754.9+2.2
SpeciEval (Pooled English Runs)Safety98.596.4−2.1
SpeciEval (Refusal-Aware Retries, Historical)Safety98.596.4−2.1
AA-LCR Extended (Mercor held-out)Search70.268.2−2.0
GDP.pdf (Artificial Analysis, All-pass)Professional Work25.223.2−2.0
OTIS Mock AIME 2024-2025Math100.098.1−1.9
ChessBench CoherenceGames88.286.3−1.9
ForecastBench Preliminary Dataset Brier IndexForecasting62.660.7−1.9
ComplexConstraintsChat & Writing50.848.9−1.9
HealthBench (length-adjusted, Artificial Analysis)Health & Medicine54.256.0+1.8
APEX-1 (Mercor, mean score)Professional Work63.665.4+1.8
PubMedQA (SecureBio)Health & Medicine80.979.2−1.7
Vibe Code Bench v1.1Code87.886.2−1.6
ProgramBenchCode2.00.5−1.5
FrontierMath ErdősMath1.50.0−1.5
OncologyBench Strict (25 tasks, October 9 rubric)Health & Medicine20.018.7−1.3
DTBenchGeneral QA97.396.0−1.3
BioSimBench 1.0Science13.612.3−1.3
SWE-bench VerifiedCode80.379.1−1.2
WMDP-Bio (SecureBio)Science90.088.8−1.2
Artificial Analysis Intelligence Index v4.3.2Indexes47.646.4−1.2
Medical Long Context Reasoning (MLCR-AA)Health & Medicine16.115.0−1.1
Hermes BenchAgents73.574.4+0.9
LMArena Agent Arena Bash ReliabilityAgents10.99.3−0.8
AdvancedIFChat & Writing80.980.2−0.7
Creative Writing v3 Rubric ScoreChat & Writing16.115.9−0.5
Terminal-Bench 2.1Code85.485.0−0.4
dfbench Detection RecallCybersecurity59.359.0−0.3
LMArena Agent Arena Confirmed SuccessAgents4.55.1+0.3
Longform Creative WritingChat & Writing82.883.0+0.2
SciCode 1.0.1 (Artificial Analysis)Code57.657.8+0.1
MMLU ProGeneral QA88.988.8−0.1
MMLU-Bio (SecureBio)Science90.890.9+0.1
LMArena Agent Arena Bash RecoveryAgents7.07.2+0.1
LMArena Agent Arena Tool HallucinationAgents1.01.00.0
AIME 2025Math100.0100.00.0
APEX-Agents 1.1Professional Work68.768.70.0
FrontierCUA pass@1Agents20.020.00.0
Harvey's Legal Agent Benchmark - Arbitration International Dispute ResolutionLegal0.00.00.0
Harvey's Legal Agent Benchmark - Banking FinanceLegal0.00.00.0
Harvey's Legal Agent Benchmark - Bankruptcy RestructuringLegal0.00.00.0
Harvey's Legal Agent Benchmark - Corporate GovernanceLegal0.00.00.0
Harvey's Legal Agent Benchmark - Data Privacy CybersecurityLegal0.00.00.0
Harvey's Legal Agent Benchmark - Emerging Companies Venture CapitalLegal0.00.00.0
Harvey's Legal Agent Benchmark - Employment LaborLegal0.00.00.0
Harvey's Legal Agent Benchmark - Environmental ESGLegal0.00.00.0
Harvey's Legal Agent Benchmark - ImmigrationLegal0.00.00.0
Harvey's Legal Agent Benchmark - InsuranceLegal0.00.00.0
Harvey's Legal Agent Benchmark - International Trade SanctionsLegal0.00.00.0
Harvey's Legal Agent Benchmark - Real EstateLegal0.00.00.0
Harvey's Legal Agent Benchmark - Structured Finance SecuritizationLegal0.00.00.0
Harvey's Legal Agent Benchmark - TaxLegal0.00.00.0
Harvey's Legal Agent Benchmark - Trusts Estates Private ClientLegal0.00.00.0
Harvey's Legal Agent Benchmark - White Collar Defense InvestigationsLegal0.00.00.0
ObviousBenchGeneral QA99.399.30.0

GPT-6 SolvsGrok 4.7normalised score marginsData Nov 15, 2026Data: Vals AI, Artificial Analysis, Mercor and 46 othersVillage LLM Atlasatlas.village.dev

Win counts use each benchmark’s declared score direction. Represented suite slices and the derived Capability score are excluded from headline totals. Missing or incompatible setups are excluded, never counted as losses.
Loading Atlas data…