Atlas

Compare Models

Opus 4.8 vs GPT-6 Sol

GPT-6 Sol leads by 4.0 capability points (≈95% interval +1.9 to +6.2): a clear lead over Opus 4.8 · P(higher) ≈ 100%

Wins 116 of 168 shared benchmarks (51 lost, 1 tied)

Capability vs. relative cost
Opus 4.8GPT-6 SolClosed weightsOpen weights

Each line runs through one model’s priced reasoning settings, cheapest to most expensive; grey points are other models’ priced settings in the same range. Results whose reasoning effort the source didn’t report are left out wherever a model has named effort levels. Capability and relative cost are the published estimates (relative cost is cost per task as a multiple of GPT-5 (high)); hover a setting, or use the arrow keys, to set it beside the other model’s same setting.

Opus 4.8vsGPT-6 Solcapability vs. relative costData Nov 15, 2026Village LLM Atlasatlas.village.dev

Benchmark wins51 – 116 · 168 shared, 1 tied

Comparing matching setups only (88 benchmarks excluded) · suites counted once
← Opus 4.8GPT-6 Sol →

Opus 4.8vsGPT-6 Solwins by categoryData Nov 15, 2026Data: Vals AI, Mercor, benchmarks.bio and 49 othersVillage LLM Atlasatlas.village.dev

Margins

Each benchmark scaled 0–100% (its declared range, else the field’s range); Δ is GPT-6 Sol minus Opus 4.8.

BrokenArXiv 05/2026Math23.087.0+64.0
VoxelBench Text RatingGames16272394+54.6
BrokenArXiv 06/2026Math41.795.4+53.7
LLM Chess EloGames4501545+45.0
KernelBench Mega Normalized (RTX PRO 6000)Code40.62.7−37.9
BullshitBench v2 LeaderboardSafety95.059.0−36.0
DeepsecBench — Revalidated V1Code6.940.9+34.0
FrontierMath Tier 4 (v2) - PrivateMath56.190.0+33.9
DeepsecBench — Revalidated V1 (233 issues)Code6.940.7+33.9
BrowseComp Extended (Mercor held-out)Search30.063.2+33.2
CadQueryEval (Corrected Geometry)Code68.0100.0+32.0
Medical Long Context Reasoning (MLCR-AA)Health & Medicine45.616.1−29.4
Olam Social Poker EloGames15181542+26.1
ArXivMath 05/2026Math65.090.0+25.0
SWE-bench Verified Extended (Mercor held-out)Code63.339.3−24.0
ZeroBench (Main questions, pass@5)Multimodal17.041.0+24.0
BuseyBench SVGMultimodal5.27.5+23.2
Blueprint-Bench 2Multimodal0.10.4+22.4
Terminal-Bench 4.0 (Artificial Analysis)Code21.743.9+22.2
Humanity's Last Exam DiamondGeneral QA23.145.2+22.1
Terminal-Bench Science 0.1 (Vals)Science10.031.4+21.4
Terminal-Bench 4.0Code23.244.4+21.2
ArXivMath 06/2026Math70.091.0+21.0
MMMU-Pro Extended (Mercor held-out)Multimodal31.451.9+20.5
ChessBench EloGames9611303+20.3
Mystery Game PuzzlesGames36.056.0+20.0
FACTS Benchmark SuiteGeneral QA53.572.5+19.0
ALE-BenchCode21022637+18.9
Roboflow Vision Evals (October 2026, five tasks)Multimodal58.376.9+18.6
RiemannBenchMath47.264.8+17.6
ARC-AGI-2General QA72.189.6+17.5
KernelBench CUDA Normalized (RTX PRO 6000)Code24.942.0+17.1
GameBench 2Games62.579.4+17.0
MedXpertQA MM Extended (Mercor held-out)Health & Medicine51.668.0+16.4
AutomationBench 1.0.6 (Zapier)Agents16.933.2+16.3
AutomationBench-AA 1.0.6Agents45.661.7+16.1
GBENCH One-shot Coding GScoreGames54.070.1+16.1
Furniture Assembly BenchmarkMultimodal42.558.3+15.8
BrowseComp (Mercor 130-question subset)Search73.789.4+15.7
ComplexConstraintsChat & Writing35.650.8+15.2
Legal Research BenchLegal43.828.8−14.9
SpeciEval (Pooled English Runs)Safety84.398.5+14.2
SpeciEval (Refusal-Aware Retries, Historical)Safety84.398.5+14.2
Roboflow Vision Evals (September 2026 protocol)Multimodal68.782.3+13.6
Arena Alignment IndexSafety74.587.6+13.1
Finance Agent v2 - General Qualitative AnalysisFinance & Accounting66.453.6−12.8
SUPERChemScience60.372.8+12.5
DeepSWE v1.1Code54.466.6+12.2
HieroglyphBenchMultimodal14.026.0+12.0
ScBench-Long V1 Clean RerunsScience16.528.4+11.9
Tax Agent BenchFinance & Accounting64.953.0−11.9
Public Benefits Bench v1.1Professional Work68.156.6−11.5
Terminal-Bench 2.1Code71.983.1+11.2
HealthBenchHealth & Medicine58.848.2−10.6
Finance Agent v2 - Disclosure AnalysisFinance & Accounting65.755.6−10.1
SAGEMath54.844.8−10.0
Code MigrationCode47.357.2+9.9
FrontierMath Tiers 1-3 (v2) - PrivateMath80.089.8+9.8
Terminal-Bench 2.1 Extended (Mercor held-out)Code32.742.4+9.7
YC-BenchGames17113891441793−9.4
Finance Agent v2 - Market AnalysisFinance & Accounting67.258.0−9.3
EnterpriseBench: CoreCraftProfessional Work52.343.1−9.2
MedXpertQA MM (200-question set)Health & Medicine72.581.3+8.8
AA-Briefcase v1.1 EloAgents13201476+8.5
ChessBench CoherenceGames79.788.2+8.5
SimpleBenchGeneral QA64.873.1+8.3
TaxCalcBench TY25Finance & Accounting20.028.0+8.0
GeneBench-Pro Public Release SubsetScience18.026.0+8.0
GPQA-Bio (SecureBio)Science80.988.9+8.0
Harvey's Legal Agent BenchmarkLegal9.61.7−7.9
HardGeoBench Avg DistanceMultimodal12451497−7.8
Humanity's Sixth SenseMultimodal23.531.2+7.7
SimpleQA Verified (Epoch anti-abstention)General QA53.060.7+7.7
Finance Agent v2 - All PassFinance & Accounting43.536.0−7.5
HardGeoBench Country AccuracyMultimodal53.846.3−7.5
SWE Atlas Codebase QnA Extended (Mercor held-out)Code24.031.3+7.3
AdvancedIFChat & Writing73.780.9+7.2
CharXiv Extended (Mercor held-out)Multimodal76.783.8+7.1
AA-LCR Extended (Mercor held-out)Search63.270.2+7.0
SM BenchSafety72.966.0−7.0
NoCoT-Bench NCRI 15.2General QA102115+6.9
TxBench Antibody Discovery V1 Clean RerunsScience30.237.0+6.8
Design Arena Overall Code CategoriesCode12521293+6.8
Ruby LLM Benchmarks — Program FixerCode72.679.0+6.4
LMArena Agent Arena Net ImprovementAgents6.98.8+6.4
MetagenomicsBench V1Science48.342.0−6.3
MetagenomicsBench V0Science48.342.0−6.3
WCB (SecureBio)Science66.760.5−6.2
MedCodeHealth & Medicine53.247.1−6.1
Artificial Analysis Intelligence Index v4.3.2Indexes41.847.6+5.8
VCT (SecureBio)Science48.254.0+5.8
Artificial Analysis Progress Index v1Indexes318334+5.5
ChessBench AccuracyGames72.477.4+5.4
TxBench Preclinical Pharmacology V1 Clean RerunsScience57.763.0+5.3
TxBench Oligo Discovery V1 Clean RerunsScience35.340.6+5.3
Finance Agent v2 - AdjustmentsFinance & Accounting48.443.2−5.1
LiveBench 2.0Agents76.881.9+5.1
Vibe Code Bench v1.1Code82.787.8+5.1
ACCoRDGeneral QA77.482.4+5.0
PubMedQA (SecureBio)Health & Medicine75.980.9+5.0
GlobeBenchGeneral QA20.025.0+5.0
MBCT (SecureBio)Science56.461.1+4.7
HPCT (SecureBio)Science63.167.8+4.7
LisanBench Difficulty-WeightedGames26943118+4.5
ScBench V1 Clean RerunsScience57.161.5+4.4
LMArena Text Arena EloChat & Writing14821457−4.4
APEX-Agents 1.1Professional Work64.468.7+4.3
SciCode (main problems)Science16.912.7−4.2
LAB-Bench ProtocolQA (SecureBio)Science76.981.1+4.2
Finance Agent v2 - Financial ModelingFinance & Accounting23.319.3−4.0
Finance Agent v2 - ComparablesFinance & Accounting44.940.8−4.0
Chess Puzzles (task version 2.0)Games52.056.0+4.0
OncologyBench Strict (25 tasks, October 9 rubric)Health & Medicine16.020.0+4.0
Wiz Cyber Model Arena v3.9Cybersecurity65.469.3+3.9
LisanBench Path LengthGames94638222−3.9
BioSecBench-Function V1 Clean RerunsScience43.139.3−3.8
EpiBench V1 Clean RerunsScience25.822.0−3.8
Conceptual Reasoning IndexIndexes68.371.0+3.8
MedScribeHealth & Medicine85.882.0−3.7
VariantBench V1 Clean RerunsScience42.438.7−3.7
DistressBenchSafety71.468.0−3.5
VCT v2 (SecureBio)Science45.949.3+3.4
Epoch Capabilities Index (ECI)Indexes160164+3.3
SciCode 1.0.1 (Artificial Analysis)Code54.457.6+3.2
LAB-Bench LitQA2 (SecureBio)Science48.751.8+3.1
ARC-AGI-1 Semi-PrivateGeneral QA92.595.5+3.0
Finance Agent v2 - PrecedentsFinance & Accounting25.628.5+2.9
LAB-Bench CloningScenarios (SecureBio)Science95.998.5+2.6
LMArena Agent Arena SteerabilityAgents9.614.5+2.5
Vals Index v2.1Indexes55.157.5+2.4
GDP.pdf (Artificial Analysis, All-pass)Professional Work22.825.2+2.4
DTBenchGeneral QA94.997.3+2.4
GDPval-AA v2.1Professional Work14611514+2.3
WMDP-Bio (SecureBio)Science92.390.0−2.3
Excel Modeling BenchmarkFinance & Accounting69.471.5+2.2
Gray Swan IPI Attack Success (k=15)Safety8.010.1−2.1
CharXiv (descriptive and reasoning combined)Multimodal93.295.0+1.8
SpatialBench-Long V1 Clean RerunsScience33.331.6−1.8
SpatialBench V1 Clean RerunsScience65.867.5+1.7
AIME 2025Math98.3100.0+1.7
DeepResearch Bench II (Mercor 130-task set)Search44.642.9−1.7
OTIS Mock AIME 2024-2025Math98.3100.0+1.7
Vals Index v2Indexes60.962.6+1.7
Finance Agent v2 - General Quantitative AnalysisFinance & Accounting71.670.0−1.6
Bio Capabilities Index (SecureBio)Indexes164161−1.6
LMCAGeneral QA57.559.1+1.6
Humanity's Last Exam (Text-Only)General QA44.846.3+1.5
HardGeoBench <=10miMultimodal27.526.3−1.3
HardGeoBench <=100miMultimodal36.337.5+1.3
APEX-SWECode43.945.0+1.1
MMLU ProGeneral QA90.088.9−1.1
NoCoT-Bench NCKI (kspine v3)General QA117119+1.1
Extended NYT Connections 940 (Quadratic v1)Games91.190.1−1.0
ForecastBench Preliminary Dataset Brier IndexForecasting61.662.6+1.0
GDPval (Mercor 213-task rubric score)Professional Work84.583.5−1.0
ProgramBenchCode1.02.0+1.0
HealthBench ProfessionalHealth & Medicine60.361.2+0.9
APEX-1 (Mercor, mean score)Professional Work62.763.6+0.9
SWE-TogetherCode61.560.6−0.9
BioSecBench-Surveillance V1 Clean RerunsScience46.947.7+0.9
AA-Omniscience IndexIndexes28.827.1−0.8
ARC-AGI-3 (Semi-Private)General QA1.50.9−0.6
Finance Agent v2 - Earnings AnalysisFinance & Accounting72.172.4+0.3
SWE Atlas - Codebase QnACode44.444.1−0.3
GDPval Extended (Mercor held-out rubric score)Professional Work58.958.7−0.2
MMLU-Bio (SecureBio)Science90.990.8−0.1
LMArena Agent Arena Bash RecoveryAgents7.27.0−0.1
ObviousBenchGeneral QA99.399.30.0

Opus 4.8vsGPT-6 Solnormalised score marginsData Nov 15, 2026Data: Vals AI, Mercor, benchmarks.bio and 49 othersVillage LLM Atlasatlas.village.dev

Win counts use each benchmark’s declared score direction. Represented suite slices and the derived Capability score are excluded from headline totals. Missing or incompatible setups are excluded, never counted as losses.
Loading Atlas data…