Atlas

Compare Models

Opus 4.8 vs Grok 4.7

Grok 4.7 leads by 0.9 capability points (≈95% interval −1.1 to +2.9): statistically tied · P(higher) ≈ 80%

Wins 90 of 192 shared benchmarks (85 lost, 17 tied)

Capability vs. relative cost
Opus 4.8Grok 4.7Closed weightsOpen weights

Each line runs through one model’s priced reasoning settings, cheapest to most expensive; grey points are other models’ priced settings in the same range. Results whose reasoning effort the source didn’t report are left out wherever a model has named effort levels. Capability and relative cost are the published estimates (relative cost is cost per task as a multiple of GPT-5 (high)); hover a setting, or use the arrow keys, to set it beside the other model’s same setting.

Opus 4.8vsGrok 4.7capability vs. relative costData Nov 15, 2026Village LLM Atlasatlas.village.dev

Benchmark wins85 – 90 · 192 shared, 17 tied

Comparing matching setups only (14 benchmarks excluded) · suites counted once
← Opus 4.8Grok 4.7 →

Opus 4.8vsGrok 4.7wins by categoryData Nov 15, 2026Data: Vals AI, Mercor, benchmarks.bio and 47 othersVillage LLM Atlasatlas.village.dev

Margins

Each benchmark scaled 0–100% (its declared range, else the field’s range); Δ is Grok 4.7 minus Opus 4.8.

ABC-Bench fragment design (SecureBio)Science19.8100.0+80.2
FrontierMath Tier 4 (v2) - PrivateMath56.117.1−39.0
LAB-Bench LitQA2 (SecureBio)Science48.716.9−31.8
Medical Long Context Reasoning (MLCR-AA)Health & Medicine45.615.0−30.6
Concept Synth INDUCTION Challenge64 Round 1General QA6.334.4+28.1
FrontierMath Tiers 1-3 (v2) - PrivateMath80.053.0−27.0
HardGeoBench Avg DistanceMultimodal12452104−26.5
LAB-Bench ProtocolQA (SecureBio)Science76.951.4−25.5
KernelBench Mega Normalized (RTX PRO 6000)Code40.618.0−22.6
Furniture Assembly BenchmarkMultimodal42.520.8−21.7
Terminal-Bench 2.1 Extended (Mercor held-out)Code32.711.8−20.9
KernelBench CUDA Normalized (RTX PRO 6000)Code24.945.2+20.3
CadQueryEval (Corrected Geometry)Code68.088.0+20.0
Harvey's Legal Agent Benchmark - Antitrust CompetitionLegal0.020.0+20.0
Harvey's Legal Agent Benchmark - Capital MarketsLegal20.040.0+20.0
Harvey's Legal Agent Benchmark - Corporate M&ALegal0.020.0+20.0
Harvey's Legal Agent Benchmark - Data Privacy CybersecurityLegal20.00.0−20.0
Harvey's Legal Agent Benchmark - Emerging Companies Venture CapitalLegal20.00.0−20.0
Harvey's Legal Agent Benchmark - Funds Asset ManagementLegal0.020.0+20.0
Harvey's Legal Agent Benchmark - Healthcare Life SciencesLegal40.020.0−20.0
Harvey's Legal Agent Benchmark - Intellectual PropertyLegal0.020.0+20.0
Harvey's Legal Agent Benchmark - Litigation Dispute ResolutionLegal0.020.0+20.0
Harvey's Legal Agent Benchmark - Structured Finance SecuritizationLegal20.00.0−20.0
AutomationBench-AA 1.0.6Agents45.665.6+20.0
DistressBenchSafety71.452.4−19.0
AA-Briefcase v1.1 EloAgents13201657+18.5
Blueprint-Bench 2Multimodal0.10.3+18.0
BrokenArXiv 05/2026Math23.040.0+17.0
DeepSWE v1.1Code54.471.0+16.6
Bio Capabilities Index (SecureBio)Indexes164141−16.4
EnterpriseBench: CoreCraftProfessional Work52.368.2+15.9
LAB-Bench CloningScenarios (SecureBio)Science95.980.1−15.8
LLM Chess EloGames450822+15.3
FrontierCode 2.0 - Pass RateCode17.132.3+15.2
Toloka Arena (v1)Agents54.339.3−15.0
Extended NYT Connections 940 (Quadratic v1)Games91.176.8−14.3
SAGEMath54.840.8−14.0
Terminal-Bench 4.0Code23.637.6+13.9
FrontierCode 2.0 - ScoreCode15.328.9+13.6
SM BenchSafety72.986.3+13.4
Bug Hunt BenchCode14.327.6+13.3
ComplexConstraintsChat & Writing35.648.9+13.3
SpeciEval (Pooled English Runs)Safety84.396.4+12.1
SpeciEval (Refusal-Aware Retries, Historical)Safety84.396.4+12.1
TaxCalcBench TY25Finance & Accounting20.08.0−12.0
GDPval-AA v2.1Professional Work14611718+11.3
LMArena Code Arena WebDevCode15551641+11.2
BrokenArXiv 06/2026Math41.752.8+11.1
Chess Puzzles (task version 2.0)Games52.041.0−11.0
Olam Social Poker EloGames15181508−10.9
BioSecBench-Surveillance V1 Clean RerunsScience37.348.0+10.7
ARC-AGI-2General QA72.161.4−10.7
TxBench Antibody Discovery V1 Clean RerunsScience32.442.8+10.5
MBCT (SecureBio)Science56.446.1−10.3
AA-AnalystAgent (pass^5)Professional Work45.055.0+10.0
HardGeoBench <=10miMultimodal27.517.5−10.0
HardGeoBench Country AccuracyMultimodal53.843.8−10.0
GameBench 2Games62.572.3+9.9
ALE-BenchCode21021823−9.9
RedlineBenchLegal39.749.5+9.8
ScBench-Long V1 Clean RerunsScience21.011.4−9.7
APEX-SWECode43.953.4+9.5
BrowseComp Extended (Mercor held-out)Search30.039.5+9.5
Vision Arena OverallMultimodal12861249−9.5
HPCT (SecureBio)Science63.153.7−9.4
LMArena Agent Arena Net ImprovementAgents6.94.0−9.4
WCB (SecureBio)Science66.757.6−9.1
HardGeoBench <=100miMultimodal36.327.5−8.7
SWE Atlas Codebase QnA Extended (Mercor held-out)Code24.015.3−8.7
Arena Alignment IndexSafety74.582.7+8.2
LMCAGeneral QA57.549.4−8.1
GeneBench-Pro Public Release SubsetScience18.010.0−8.0
SWE-bench Verified Extended (Mercor held-out)Code63.355.3−8.0
Conceptual Reasoning IndexIndexes68.363.1−7.1
Mystery Game PuzzlesGames36.029.0−7.0
LMArena Text Arena EloChat & Writing14821442−7.0
ChessBench EloGames9611076+6.8
ChessBench CoherenceGames79.786.3+6.6
AdvancedIFChat & Writing73.780.2+6.5
BuseyBench SVGMultimodal5.24.6−6.4
TxBench Oligo Discovery V1 Clean RerunsScience33.940.3+6.4
Finance Agent v2 - Financial ModelingFinance & Accounting23.317.7−5.7
Finance Agent v2 - Market AnalysisFinance & Accounting67.261.8−5.4
Roboflow Vision Evals (October 2026, five tasks)Multimodal58.363.5+5.2
AA-LCR Extended (Mercor held-out)Search63.268.2+5.0
RuneBenchGames5.15.4+5.0
Ruby LLM Benchmarks — Program FixerCode72.668.0−4.7
Artificial Analysis Intelligence Index v4.3.2Indexes41.846.4+4.7
HealthBenchHealth & Medicine58.854.2−4.6
LMArena Agent Arena SteerabilityAgents9.60.5−4.6
DeepResearch Bench II (Mercor 130-task set)Search44.649.0+4.4
BioSecBench-Function V1 Clean RerunsScience39.043.3+4.4
GPQA DiamondGeneral QA88.492.7+4.4
APEX-Agents 1.1Professional Work64.468.7+4.3
ArXivMath 05/2026Math65.060.8−4.2
Tax Agent BenchFinance & Accounting64.969.0+4.1
DeepsecBench — Revalidated V1Code6.910.9+4.0
Terminal-Bench 4.0 (Artificial Analysis)Code21.725.8+4.0
DeepsecBench — Revalidated V1 (233 issues)Code6.910.9+4.0
VCT v2 (SecureBio)Science45.942.0−3.9
Vals RSI Index v1.1 (four tasks)Agents24.020.2−3.8
MineBench — Bradley-Terry RatingGames17861848+3.8
LLM Creative Story WritingChat & Writing0.80.4−3.7
MedCodeHealth & Medicine53.249.6−3.7
MedScribeHealth & Medicine85.889.4+3.6
SWE-TogetherCode61.565.1+3.6
MedXpertQA MM (200-question set)Health & Medicine72.569.0−3.5
WMDP-Bio (SecureBio)Science92.388.8−3.5
Vibe Code Bench v1.1Code82.786.2+3.5
Legal Research BenchLegal43.847.1+3.4
SciCode 1.0.1 (Artificial Analysis)Code54.457.8+3.4
ACCoRDGeneral QA77.474.0−3.3
PubMedQA (SecureBio)Health & Medicine75.979.2+3.3
Roboflow Vision Evals (September 2026 protocol)Multimodal68.771.9+3.2
Epoch Capabilities Index (ECI)Indexes160156−3.1
VariantBench V1 Clean RerunsScience42.445.5+3.1
Senior SWE-BenchCode30.527.4−3.1
Finance Agent v2 - General Quantitative AnalysisFinance & Accounting71.668.6−3.1
SimpleQA Verified (Epoch anti-abstention)General QA53.056.0+3.0
Terminal-Bench 2.1Code82.085.0+3.0
Design Arena Overall Code CategoriesCode12521270+3.0
SpatialBench-Long V1 Clean RerunsScience27.424.4−3.0
APEX-1 (Mercor, mean score)Professional Work62.765.4+2.7
LiveBench 2.0Agents76.874.1−2.7
OncologyBench Strict (25 tasks, October 9 rubric)Health & Medicine16.018.7+2.7
Harvey HLAB — Criteria Pass RateLegal87.990.5+2.6
Finance Agent v2 - ComparablesFinance & Accounting44.942.2−2.6
SpatialBench V1 Clean RerunsScience67.069.6+2.6
SWE Atlas - Codebase QnACode44.447.0+2.6
Finance Agent v2 - Earnings AnalysisFinance & Accounting72.174.6+2.5
Public Benefits Bench v1.1Professional Work68.165.6−2.5
Code MigrationCode47.344.8−2.4
Excel Modeling BenchmarkFinance & Accounting69.467.0−2.4
ARC-AGI-1 Semi-PrivateGeneral QA92.590.2−2.3
Finance Agent v2 - General Qualitative AnalysisFinance & Accounting66.468.7+2.3
GBENCH One-shot Coding GScoreGames54.056.3+2.3
ScBench V1 Clean RerunsScience56.258.5+2.2
Finance Agent v2 - AdjustmentsFinance & Accounting48.446.3−2.1
BrowseComp (Mercor 130-question subset)Search73.775.8+2.1
ABC-Bench liquid handling (SecureBio)Science100.098.0−2.0
ChessBench AccuracyGames72.474.0+1.7
AIME 2025Math98.3100.0+1.7
Humanity's Last Exam (Text-Only)General QA44.843.1−1.7
AA-Omniscience IndexIndexes28.832.0+1.6
GPQA-Bio (SecureBio)Science80.982.5+1.6
LLM Debate BenchmarkChat & Writing16311620−1.6
YC-BenchGames17113891666389−1.6
Artificial Analysis Progress Index v1Indexes318323+1.5
CharXiv Extended (Mercor held-out)Multimodal76.778.2+1.5
Terminal-Bench Science 0.1 (Vals)Science10.011.4+1.4
Finance Agent v2 - All PassFinance & Accounting43.542.1−1.4
MetagenomicsBench V1Science50.048.7−1.3
APEX-AccountingFinance & Accounting48.046.7−1.3
MetagenomicsBench V0Science50.048.7−1.3
Finance Agent v2 - Disclosure AnalysisFinance & Accounting65.764.5−1.2
CharXiv (descriptive and reasoning combined)Multimodal93.292.0−1.2
MMLU ProGeneral QA90.088.8−1.2
GDP.pdfProfessional Work24.022.8−1.2
DTBenchGeneral QA94.996.0+1.1
ForecastBench Preliminary Dataset Brier IndexForecasting61.660.7−0.9
LegalBenchLegal83.684.4+0.8
Vals Index v2Indexes60.960.2−0.7
Humanity's Last Exam DiamondGeneral QA23.123.6+0.5
ProgramBenchCode1.00.5−0.5
MedXpertQA MM Extended (Mercor held-out)Health & Medicine51.652.0+0.4
TxBench Preclinical Pharmacology V1 Clean RerunsScience54.354.7+0.3
Finance Agent v2 - PrecedentsFinance & Accounting25.625.9+0.3
EpiBench V1 Clean RerunsScience20.420.8+0.3
OTIS Mock AIME 2024-2025Math98.398.1−0.3
FrontierCode 1.1 Extended - Pass RateCode65.565.2−0.3
FrontierCode 1.1 Extended - ScoreCode59.659.4−0.2
ARC-AGI-3 (Semi-Private)General QA1.51.7+0.2
ArXivMath 06/2026Math70.070.1+0.2
Vals Index v2.1Indexes55.154.9−0.2
LMArena Agent Arena Bash RecoveryAgents7.27.20.0
Harvey's Legal Agent Benchmark - Arbitration International Dispute ResolutionLegal0.00.00.0
Harvey's Legal Agent Benchmark - Banking FinanceLegal0.00.00.0
Harvey's Legal Agent Benchmark - Bankruptcy RestructuringLegal0.00.00.0
Harvey's Legal Agent Benchmark - Corporate GovernanceLegal0.00.00.0
Harvey's Legal Agent Benchmark - Employment LaborLegal0.00.00.0
Harvey's Legal Agent Benchmark - Energy Natural ResourcesLegal20.020.00.0
Harvey's Legal Agent Benchmark - Environmental ESGLegal0.00.00.0
Harvey's Legal Agent Benchmark - ImmigrationLegal0.00.00.0
Harvey's Legal Agent Benchmark - InsuranceLegal0.00.00.0
Harvey's Legal Agent Benchmark - International Trade SanctionsLegal0.00.00.0
Harvey's Legal Agent Benchmark - Real EstateLegal0.00.00.0
Harvey's Legal Agent Benchmark - TaxLegal0.00.00.0
Harvey's Legal Agent Benchmark - Trusts Estates Private ClientLegal0.00.00.0
Harvey's Legal Agent Benchmark - White Collar Defense InvestigationsLegal0.00.00.0
MMLU-Bio (SecureBio)Science90.990.90.0
ObviousBenchGeneral QA99.399.30.0
ReactBench (39 tasks)Code32.332.30.0

Opus 4.8vsGrok 4.7normalised score marginsData Nov 15, 2026Data: Vals AI, Mercor, benchmarks.bio and 47 othersVillage LLM Atlasatlas.village.dev

Win counts use each benchmark’s declared score direction. Represented suite slices and the derived Capability score are excluded from headline totals. Missing or incompatible setups are excluded, never counted as losses.
Loading Atlas data…