Atlas

Compare Models

GPT-6 Astra vs GPT-6 Sol

GPT-6 Astra leads by 3.5 capability points (≈95% interval +1.0 to +6.0): a clear lead over GPT-6 Sol · P(higher) ≈ 100%

Wins 225 of 293 shared benchmarks (37 lost, 31 tied)

Capability vs. relative cost
GPT-6 AstraGPT-6 SolClosed weightsOpen weights

Each line runs through one model’s priced reasoning settings, cheapest to most expensive; grey points are other models’ priced settings in the same range. Results whose reasoning effort the source didn’t report are left out wherever a model has named effort levels. Capability and relative cost are the published estimates (relative cost is cost per task as a multiple of GPT-5 (high)); hover a setting, or use the arrow keys, to set it beside the other model’s same setting.

GPT-6 AstravsGPT-6 Solcapability vs. relative costData Nov 15, 2026Village LLM Atlasatlas.village.dev

Benchmark wins225 – 37 · 293 shared, 31 tied

Comparing matching setups only (11 benchmarks excluded) · suites counted once
← GPT-6 AstraGPT-6 Sol →

GPT-6 AstravsGPT-6 Solwins by categoryData Nov 15, 2026Data: Vals AI, Artificial Analysis, Mercor and 77 othersVillage LLM Atlasatlas.village.dev

Margins

Each benchmark scaled 0–100% (its declared range, else the field’s range); Δ is GPT-6 Sol minus GPT-6 Astra.

Clipping Bench v1.0Games95141240−89.0
ARC-AGI-3 (Semi-Private)General QA98.623.0−75.5
AA-PersonalityBench Organization indexChat & Writing69.512.4−57.7
AA-PersonalityBench Transparency indexChat & Writing94.339.1−55.8
AA-PersonalityBench Imagination indexChat & Writing77.229.6−48.1
Search ArenaSearch12311174−45.2
Terminal-Bench-Science 0.1Science59.517.1−42.4
Short Form Bench v1.0Games28032394−40.5
CyberBench v1.1 OverallCybersecurity41.178.0+36.9
Terminal-Bench Science 0.1 (Vals)Science65.731.4−34.3
Terminal-Bench 4.0 (Artificial Analysis)Code59.630.3−29.3
Mystery Game PuzzlesGames84.056.0−28.0
ChessBench EloGames17551303−26.9
SRE Bench — Fully SolvedCode56.930.5−26.3
ExploitBench - Internal Port (June-August 2026)Cybersecurity31.55.5−26.0
AA-PersonalityBench Expressiveness indexChat & Writing58.133.4−25.0
SpireClimbGames34.09.2−24.8
GlobeBenchGeneral QA49.625.0−24.6
CheatBenchSafety47.471.9−24.6
NoCoT-Bench NCRI 15.2General QA159115−24.4
MysteryMechanismScience53.230.2−23.0
EBR-bench v4 (card ban, single agent)Games76.253.3−22.9
WeirdML v3Science42.219.7−22.5
Humanity's Sixth SenseMultimodal53.631.2−22.4
FrontierCUA pass@1Agents42.220.0−22.2
Furniture Assembly BenchmarkMultimodal80.058.3−21.7
AA-PersonalityBench Warmth indexChat & Writing75.354.3−21.2
ExploitGym - release-reported percentage, cohort unspecifiedCybersecurity42.422.1−20.3
Vals Science IndexIndexes65.145.0−20.1
Harvey's Legal Agent Benchmark - Corporate M&ALegal20.00.0−20.0
Harvey's Legal Agent Benchmark - Trusts Estates Private ClientLegal20.00.0−20.0
AA-PersonalityBench Informativeness indexChat & Writing29.610.5−19.2
Medical Long Context Reasoning (MLCR-AA)Health & Medicine35.016.1−18.9
ALE-BenchCode31642633−18.8
Humanity's Last Exam DiamondGeneral QA63.845.2−18.6
HardGeoBench Avg DistanceMultimodal9081497−18.2
ZeroBench Main Questions pass^5Multimodal35.017.0−18.0
Chess Puzzles (task version 2.0)Games74.056.0−18.0
ChartographyMultimodal71.053.6−17.4
IOI (Vals v2)Code100.082.6−17.4
VoxelBench Text RatingGames26252394−16.5
Biopharma Bench V0.1Health & Medicine70.454.3−16.1
CWE-bench v1 Programmatic Pass@1Cybersecurity68.052.0−16.0
ProofBench v1.1Math99.083.0−16.0
TaxCalcBench TY25Finance & Accounting44.028.0−16.0
AA-PersonalityBench Slop indexChat & Writing16.21.0−15.4
BullshitBench v2 LeaderboardSafety69.054.0−15.0
MazeBench ASCIIGames16.01.0−15.0
OpenAI internal computer-use safety - adverse outcome rateSafety2.417.4−15.0
Decision-Bench v1.1Professional Work51.536.5−15.0
Bug Hunt BenchCode42.927.9−14.9
GameBench 2Games94.379.4−14.9
SimpleQA Verified (Epoch anti-abstention)General QA75.660.7−14.9
Finance Agent v2 - General Qualitative AnalysisFinance & Accounting68.453.6−14.8
CAD ArenaAgents67.152.4−14.7
Arena Image-to-WebDevCode17221655−14.5
LMArena Agent Arena Net Improvement (Steering Burden)Agents14.56.9−14.4
ZeroBench Main Mean pass@1 (five samples)Multimodal43.228.8−14.4
HieroglyphBenchMultimodal40.026.0−14.0
HardGeoBench Country AccuracyMultimodal60.046.3−13.7
HardGeoBench <=100miMultimodal51.337.5−13.7
Blueprint-Bench 2Multimodal0.50.4−12.8
Artificial Analysis Intelligence Index v5.0.1Indexes56.944.2−12.7
RuneBenchGames7.36.4−12.5
DAYJOB: FinanceFinance & Accounting21.59.3−12.2
Hermes IndexIndexes56.344.1−12.2
HealthBench Hard (updated implementation)Health & Medicine34.222.1−12.1
LMArena Code Arena WebDevCode17641673−12.0
TxBench Antibody Discovery V1 Clean RerunsScience48.937.0−11.9
HardGeoBench <=10miMultimodal37.526.3−11.2
ZeroBench (Main questions, pass@5)Multimodal52.041.0−11.0
LMArena Agent Arena Praise vs ComplaintAgents31.710.0−10.8
Legal Research BenchLegal39.428.8−10.6
Code MigrationCode67.757.2−10.5
SimpleBenchGeneral QA83.673.1−10.5
Tax Agent BenchFinance & Accounting63.353.0−10.3
GeneBench-Pro Public Release SubsetScience36.026.0−10.0
Professional Reasoning Benchmark - LegalLegal48.438.6−9.8
DeepsecBench-AA (Artificial Analysis)Cybersecurity36.946.3+9.4
BioSecBench-Function V1 Clean RerunsScience48.639.3−9.4
FrontierCode 2.0 - Pass RateCode47.638.3−9.2
GBENCH One-shot Coding GScoreGames79.270.1−9.1
FrontierCode 2.0 - ScoreCode44.034.9−9.1
Toloka Arena (v3)Agents67.058.0−9.0
FrontierPhysics v0.1Science23.214.3−8.9
ChessBench CoherenceGames97.088.2−8.8
Terminal-Bench 4.0Code58.249.4−8.8
Finance Agent v2 - General Quantitative AnalysisFinance & Accounting78.870.0−8.8
VCT v2 (SecureBio)Science58.049.3−8.7
Olam Social Poker EloGames15501542−8.7
HealthBenchHealth & Medicine56.948.2−8.7
TxBench Oligo Discovery V1 Clean RerunsScience49.240.6−8.6
NoCoT-Bench NCKI (kspine v3)General QA131119−8.6
HealthBench ProfessionalHealth & Medicine69.761.2−8.5
AA-Omniscience IndexIndexes43.727.1−8.3
LAB-Bench ProtocolQA (SecureBio)Science89.381.1−8.2
AutomationBench 1.0.6 (Zapier)Agents41.433.2−8.2
OSWorld 2 (Aug 8) offline - reported partial reward, step limit unspecifiedAgents72.664.4−8.2
ChessBench AccuracyGames85.077.4−8.1
MMMU-Pro Extended (Mercor held-out)Multimodal60.051.9−8.1
ChessBenchGames70.062.0−8.0
Extended NYT Connections 940 (Quadratic v1)Games98.190.1−8.0
DAYJOB: HealthcareHealth & Medicine11.63.6−8.0
WellnessBenchSafety64.056.0−8.0
LMArena Agent Arena SteerabilityAgents-1.414.5+8.0
VariantBench V1 Clean RerunsScience46.638.7−7.9
SpatialBench-Long V1 Clean RerunsScience39.331.6−7.7
MetagenomicsBench V0Science49.742.0−7.7
MetagenomicsBench V1Science49.742.0−7.7
LegalReasonBenchLegal39.531.9−7.6
FrontierMath Tier 4 (v2) - PrivateMath97.690.0−7.6
Finance Agent v2 - Market AnalysisFinance & Accounting65.558.0−7.6
DeepSWE v1.1Code74.166.6−7.5
BigFinanceBench private holdout (final-answer accuracy)Finance & Accounting54.647.1−7.5
SUPERChemScience80.372.8−7.5
BioSecBench-Structure V1 (58 evaluations)Science21.313.8−7.5
DeepResearch Bench II (Mercor 130-task set)Search50.142.9−7.2
MedXpertQA MM Extended (Mercor held-out)Health & Medicine60.868.0+7.2
RiemannBenchMath72.064.8−7.2
ComplexConstraintsChat & Writing57.750.8−6.9
SciCode (main problems)Science19.612.7−6.9
AutomationBench-AA 1.0.6Agents68.561.7−6.8
Humanity's Last Exam (Text-Only)General QA53.146.3−6.8
LibraryDesignBenchCode45.138.5−6.6
HealthBench Hard (length-adjusted)Health & Medicine36.630.1−6.5
Roboflow Vision Evals (October 2026, five tasks)Multimodal83.476.9−6.5
APEX-Agents 1.1Professional Work75.168.7−6.4
DistressBenchSafety74.368.0−6.4
EpiBench V1 Clean RerunsScience28.322.0−6.3
GDP.pdfProfessional Work34.228.0−6.2
Finance Agent v2 - Disclosure AnalysisFinance & Accounting61.755.6−6.0
MedScribeHealth & Medicine87.982.0−5.9
GDP.xlsxProfessional Work22.917.1−5.8
AA-PersonalityBench Conviction indexChat & Writing44.850.5+5.8
APEX-1 (Mercor, mean score)Professional Work69.363.6−5.7
MedXpertQA MM (200-question set)Health & Medicine86.881.3−5.5
Artificial Analysis Progress Index v1Indexes350334−5.5
ARC-AGI-2General QA95.089.6−5.4
GDPval Extended (Mercor held-out rubric score)Professional Work64.158.7−5.4
AA-3D v1.0 EloCode11681131−5.4
Bedside BenchHealth & Medicine93.488.1−5.3
LMCAGeneral QA64.459.1−5.3
LMArena Agent Arena Net ImprovementAgents10.48.8−5.2
AA-Briefcase v1.1 EloAgents15691476−5.1
Harvey LAB (Mercor run, undisclosed ~213-task cohort)Legal9.24.1−5.1
Finance Agent v2 - AdjustmentsFinance & Accounting48.343.2−5.1
Artificial Analysis Intelligence Index v4.3.2Indexes52.747.6−5.0
APEX-Accounting 1.1Finance & Accounting64.759.7−5.0
ArXivMath 05/2026Math95.090.0−5.0
DualEntry Accounting AI BenchmarkFinance & Accounting81.276.2−5.0
Eval Connections One-Shot with TrapsGames94.089.0−5.0
Harvey LAB-AA v1.1 (hallucination-gated)Legal8.63.6−5.0
Artificial Analysis Coding Agent Index v1.5Indexes61.656.7−5.0
Financial Coaching BenchFinance & Accounting67.562.6−4.9
BrowseComp (Mercor 130-question subset)Search94.289.4−4.8
LMArena Agent Arena Confirmed SuccessAgents14.04.5−4.8
MentalHealthBenchSafety58.754.2−4.5
VCT (SecureBio)Science58.554.0−4.5
Harvey LAB held-out set (Fireworks run)Legal5.10.7−4.4
BioMysteryBench (Vals Terminus 2)Science79.374.8−4.4
FrontierCode 1.1 Extended - Pass RateCode70.666.3−4.3
Roboflow Vision Evals (September 2026 protocol)Multimodal86.682.3−4.3
Finance Agent v2 - All PassFinance & Accounting40.336.0−4.3
LiveBench 2.0Agents86.181.9−4.2
Epoch Capabilities Index (ECI)Indexes169164−4.2
Terminal-Bench 2.1Code87.383.1−4.1
Vals Index v2Indexes66.662.6−4.0
BrokenArXiv 05/2026Math91.087.0−4.0
Harvey LAB: In-House Contracting (held-out)Legal5.31.3−4.0
Harvey HLAB — Criteria Pass RateLegal87.583.5−4.0
SM BenchSafety69.966.0−3.9
Conceptual Reasoning IndexIndexes73.971.0−3.9
Ruby LLM Benchmarks — Program FixerCode75.179.0+3.9
MMMU ProMultimodal86.482.5−3.9
Nimble Web ContextSearch72.676.5+3.9
FrontierMath Tiers 1-3 (v2) - PrivateMath93.789.8−3.9
FrontierCode 1.1 Extended - ScoreCode64.560.7−3.8
τ³-Banking (Mercor run, bm25_grep retrieval)Agents46.743.0−3.7
SWE Atlas - Codebase QnACode47.844.1−3.7
BrokenArXiv 06/2026Math99.195.4−3.7
TxBench Preclinical Pharmacology V1 Clean RerunsScience59.363.0+3.7
AA-LCR v1.1Search80.083.7+3.7
APEX-SWECode48.645.0−3.6
ProgramBenchCode5.52.0−3.5
ArXivMath 06/2026Math94.491.0−3.5
Artificial Analysis Cyber Index v1Indexes33.436.8+3.4
GDPval-AA v2.1Professional Work15871514−3.2
GDPval (Mercor 213-task rubric score)Professional Work86.783.5−3.2
FACTS Benchmark SuiteGeneral QA75.772.5−3.2
DeepsecBench — Revalidated V1Code37.840.9+3.1
DeepsecBench — Revalidated V1 (233 issues)Code37.640.7+3.1
InferenceBenchCode7.97.6−3.1
Finance Agent v2 - ComparablesFinance & Accounting37.840.8+3.1
LMArena Text Arena EloChat & Writing14741457−3.0
ARC-AGI-1 Semi-PrivateGeneral QA98.595.5−3.0
LAB-Bench LitQA2 (SecureBio)Science54.851.8−3.0
Epiq LAER CorporateBenchProfessional Work82.079.0−3.0
Finance Agent v2 - Earnings AnalysisFinance & Accounting69.572.4+2.9
BioSecBench-Refusal V1 Clean RerunsSafety40.837.9−2.9
LLM Chess EloGames16141545−2.8
MBCT (SecureBio)Science63.961.1−2.8
Agents' Last Exam (ALE)Agents34.231.6−2.6
MMLU ProGeneral QA91.588.9−2.6
LisanBench Path LengthGames73938222+2.6
Creative Writing v3 EloChat & Writing21772126−2.6
VulcanBench Frontier v4Code89.386.8−2.5
ScBench V1 Clean RerunsScience63.961.5−2.4
LMArena Agent Arena Steering BurdenAgents14.810.1−2.3
BioSimBench 1.0Science15.913.6−2.3
MacroscopeBenchCode78.080.3+2.3
Qodo AI Code Review Benchmark (125 PRs)Code46.648.9+2.3
Finance Agent v2 - Financial ModelingFinance & Accounting21.619.3−2.3
CritPt v2 (agentic, avg@5)Science83.180.9−2.3
Vectara Hallucination LeaderboardSafety8.76.5+2.2
CVDP (517-problem set)Code54.952.7−2.2
Bio Capabilities Index (SecureBio)Indexes164161−2.2
τ³-Banking (Fireworks, banking_knowledge)Agents44.847.0+2.1
WCB (SecureBio)Science62.660.5−2.1
BuseyBench SVGMultimodal7.37.5+2.1
Wiz Cyber Model Arena v3.9Cybersecurity69.271.2+2.0
SpeciEval (Pooled English Runs)Safety96.598.5+1.9
SpeciEval (Refusal-Aware Retries, Historical)Safety96.598.5+1.9
Finance Agent v2 - PrecedentsFinance & Accounting30.328.5−1.8
Gray Swan IPI Attack Success (k=15)Safety8.510.1−1.6
FrontierCS Algorithmic v1.1Code54.052.4−1.6
SAGEMath46.444.8−1.6
LMArena Agent Arena Bash ReliabilityAgents7.810.9+1.6
GPQA-Bio (SecureBio)Science87.488.9+1.5
LAB-Bench CloningScenarios (SecureBio)Science100.098.5−1.5
FrontierMath ErdősMath2.91.5−1.5
MedCodeHealth & Medicine48.547.1−1.4
OncologyBench Strict (25 tasks, October 9 rubric)Health & Medicine21.320.0−1.3
FORTRESS Average Risk Score (ARS)Safety12.113.5−1.3
CharXiv Extended (Mercor held-out)Multimodal85.183.8−1.3
HPCT (SecureBio)Science69.067.8−1.2
SciCode 1.0.1 (Artificial Analysis)Code56.557.6+1.2
ScBench-Long V1 Clean RerunsScience27.328.4+1.1
AdvancedIFChat & Writing82.080.9−1.1
CharXiv (descriptive and reasoning combined)Multimodal96.195.0−1.1
Vals RSI Index v1.1 (four tasks)Agents27.128.1+1.0
CritPt (avg@5)Science31.730.9−0.9
ITBench-AAAgents48.649.4+0.8
CWE-Bench-AA (Artificial Analysis)Cybersecurity63.364.2+0.8
WMDP-Bio (SecureBio)Science90.890.0−0.8
BioSecBench-Surveillance V1 Clean RerunsScience48.547.7−0.8
LMArena Agent Arena Bash RecoveryAgents5.57.0+0.8
SAB RankingForecasting-0.0-0.1−0.7
ACCoRDGeneral QA81.782.4+0.7
ObviousBenchGeneral QA100.099.3−0.7
SWE-Marathon v1.1Code42.543.1+0.6
Creative Writing v3 Rubric ScoreChat & Writing15.916.1+0.6
EnterpriseBench: CoreCraftProfessional Work42.643.1+0.5
MMLU-Bio (SecureBio)Science91.390.8−0.5
SAB ResearchFinance & Accounting95.295.6+0.4
ForecastBench Preliminary Dataset Brier IndexForecasting62.962.6−0.3
PubMedQA (SecureBio)Health & Medicine81.280.9−0.3
Arena Alignment IndexSafety87.887.6−0.2
Excel Modeling BenchmarkFinance & Accounting71.771.5−0.2
Hermes BenchAgents73.773.5−0.2
Quant-Arena (iterative total score)Finance & Accounting51.351.1−0.2
LisanBench Difficulty-WeightedGames31133118+0.1
LMArena Agent Arena Tool HallucinationAgents1.01.00.0
AIME 2025Math100.0100.00.0
CadQueryEval (Corrected Geometry)Code100.0100.00.0
CyberGym-E2E-AA (Artificial Analysis)Cybersecurity0.00.00.0
DTBenchGeneral QA97.397.30.0
Harvey's Legal Agent Benchmark - Antitrust CompetitionLegal0.00.00.0
Harvey's Legal Agent Benchmark - Arbitration International Dispute ResolutionLegal0.00.00.0
Harvey's Legal Agent Benchmark - Banking FinanceLegal0.00.00.0
Harvey's Legal Agent Benchmark - Bankruptcy RestructuringLegal0.00.00.0
Harvey's Legal Agent Benchmark - Capital MarketsLegal0.00.00.0
Harvey's Legal Agent Benchmark - Corporate GovernanceLegal0.00.00.0
Harvey's Legal Agent Benchmark - Data Privacy CybersecurityLegal0.00.00.0
Harvey's Legal Agent Benchmark - Emerging Companies Venture CapitalLegal0.00.00.0
Harvey's Legal Agent Benchmark - Employment LaborLegal0.00.00.0
Harvey's Legal Agent Benchmark - Energy Natural ResourcesLegal0.00.00.0
Harvey's Legal Agent Benchmark - Environmental ESGLegal0.00.00.0
Harvey's Legal Agent Benchmark - Funds Asset ManagementLegal0.00.00.0
Harvey's Legal Agent Benchmark - Healthcare Life SciencesLegal0.00.00.0
Harvey's Legal Agent Benchmark - ImmigrationLegal0.00.00.0
Harvey's Legal Agent Benchmark - InsuranceLegal0.00.00.0
Harvey's Legal Agent Benchmark - Intellectual PropertyLegal0.00.00.0
Harvey's Legal Agent Benchmark - International Trade SanctionsLegal0.00.00.0
Harvey's Legal Agent Benchmark - Litigation Dispute ResolutionLegal0.00.00.0
Harvey's Legal Agent Benchmark - Real EstateLegal0.00.00.0
Harvey's Legal Agent Benchmark - Structured Finance SecuritizationLegal0.00.00.0
Harvey's Legal Agent Benchmark - TaxLegal0.00.00.0
Harvey's Legal Agent Benchmark - White Collar Defense InvestigationsLegal0.00.00.0
Longform Creative WritingChat & Writing82.882.80.0
Next.js AI Agent EvaluationsAgents100.0100.00.0
OTIS Mock AIME 2024-2025Math100.0100.00.0
SpatialBench V1 Clean RerunsScience67.567.50.0
SWE-TogetherCode60.660.60.0

GPT-6 AstravsGPT-6 Solnormalised score marginsData Nov 15, 2026Data: Vals AI, Artificial Analysis, Mercor and 77 othersVillage LLM Atlasatlas.village.dev

Win counts use each benchmark’s declared score direction. Represented suite slices and the derived Capability score are excluded from headline totals. Missing or incompatible setups are excluded, never counted as losses.
Loading Atlas data…