Atlas

Compare Models

Opus 4.8 vs Gemini 3.7 Flash

Opus 4.8 leads by 0.3 capability points (≈95% interval −1.6 to +2.3): statistically tied · P(higher) ≈ 64%

Gemini 3.7 Flash wins 124 of 249 shared benchmarks (107 lost, 18 tied)

Capability vs. relative cost
Opus 4.8Gemini 3.7 FlashClosed weightsOpen weights

Each line runs through one model’s priced reasoning settings, cheapest to most expensive; grey points are other models’ priced settings in the same range. Results whose reasoning effort the source didn’t report are left out wherever a model has named effort levels. Capability and relative cost are the published estimates (relative cost is cost per task as a multiple of GPT-5 (high)); hover a setting, or use the arrow keys, to set it beside the other model’s same setting.

Opus 4.8vsGemini 3.7 Flashcapability vs. relative costData Nov 15, 2026Village LLM Atlasatlas.village.dev

Benchmark wins107 – 124 · 249 shared, 18 tied

Comparing matching setups only (36 benchmarks excluded) · suites counted once
← Opus 4.8Gemini 3.7 Flash →

Opus 4.8vsGemini 3.7 Flashwins by categoryData Nov 15, 2026Data: Vals AI, Mercor, benchmarks.bio and 56 othersVillage LLM Atlasatlas.village.dev

Margins

Each benchmark scaled 0–100% (its declared range, else the field’s range); Δ is Gemini 3.7 Flash minus Opus 4.8.

HPCT Refusal (CAIS)Safety85.013.0−72.0
ATLAS-Education (StudentBench) Lesson PlanningProfessional Work0.9-0.4−54.8
Olam Social Poker EloGames15181468−54.3
ChessBench — Historical Output RepairGames24.070.7+46.7
BabyVisionMultimodal28.473.5+45.1
HieroglyphBenchMultimodal14.055.0+41.0
MASKSafety94.453.7−40.7
VCT-Refusal (CAIS)Safety41.24.8−36.4
SWE-bench Verified Extended (Mercor held-out)Code63.330.0−33.3
BrokenArXiv 06/2026Math41.711.1−30.6
Medical Long Context Reasoning (MLCR-AA)Health & Medicine45.615.0−30.6
LLM Chess EloGames4501107+27.0
LMArena Agent Arena Net ImprovementAgents6.9-1.3−27.0
SWE-Marathon v1.1Code48.822.5−26.2
LLM Debate BenchmarkChat & Writing16311455−25.0
MindCube TinyMultimodal64.988.4+23.5
Roboflow Vision Evals (October 2026, five tasks)Multimodal58.380.1+21.8
Harvey's Legal Agent Benchmark - Banking FinanceLegal0.020.0+20.0
Harvey's Legal Agent Benchmark - Corporate GovernanceLegal0.020.0+20.0
Harvey's Legal Agent Benchmark - Data Privacy CybersecurityLegal20.00.0−20.0
Harvey's Legal Agent Benchmark - Emerging Companies Venture CapitalLegal20.00.0−20.0
Harvey's Legal Agent Benchmark - Energy Natural ResourcesLegal20.040.0+20.0
Harvey's Legal Agent Benchmark - Healthcare Life SciencesLegal40.020.0−20.0
Harvey's Legal Agent Benchmark - InsuranceLegal0.020.0+20.0
Harvey's Legal Agent Benchmark - Structured Finance SecuritizationLegal20.00.0−20.0
FrontierMath Tier 4 (v2) - PrivateMath56.136.6−19.5
Hemingway-benchChat & Writing10381105+18.6
ERQAMultimodal59.577.7+18.2
LVBenchMultimodal67.385.4+18.1
ChessBench EloGames9611265+18.1
HLE Extended (Mercor held-out)General QA49.431.6−17.8
FACTS Benchmark SuiteGeneral QA53.571.2+17.7
Terminal-Bench 4.0Code23.66.1−17.6
ArXivMath 05/2026Math65.047.5−17.5
DeepSWE v1.1Code48.765.5+16.8
MetagenomicsBench V1Science48.331.7−16.7
MetagenomicsBench V0Science48.331.7−16.6
LLM Creative Story WritingChat & Writing0.8-0.7−16.6
Roboflow Vision Evals (September 2026 protocol)Multimodal68.785.2+16.5
AutomationBench-AA 1.0.6Agents45.662.0+16.4
Humanity's Sixth SenseMultimodal23.539.8+16.3
SimpleQA Verified (Epoch anti-abstention)General QA53.069.2+16.2
Antidote: Everyday EditionSafety10301075+16.1
CadQueryEval (Corrected Geometry)Code68.084.0+16.0
Furniture Assembly BenchmarkMultimodal42.526.7−15.8
Senior SWE-BenchCode30.514.7−15.8
BrokenArXiv 04/2026Math34.419.3−15.2
AA-AnalystAgent (pass^5)Professional Work45.060.0+15.0
APEX-Agents 1.1Professional Work64.479.4+15.0
AgentVQAMultimodal44.058.5+14.5
BrokenArXiv 05/2026Math23.08.5−14.5
AA-LCR Extended (Mercor held-out)Search63.248.9−14.3
BuseyBench SVGMultimodal5.26.6+14.3
Chess Puzzles (task version 2.0)Games52.066.0+14.0
AA-Briefcase EloAgents13541116−13.9
MRCR v2 (8-needle, Context Arena) - 128K pointwiseGeneral QA75.288.6+13.4
EnigmaEvalGeneral QA20.533.9+13.4
TxBench Antibody Discovery V1 Clean RerunsScience30.243.5+13.4
NoCoT-Bench NCRI 15.2General QA102126+13.2
IntPhys 2Multimodal55.168.2+13.1
Arena Code WebDev — FullstackCode15801629+12.9
TextQuests Game ProgressGames40.152.9+12.8
ARC-AGI-2General QA72.184.6+12.5
SUPERChemScience60.372.8+12.5
Code MigrationCode47.334.8−12.5
Vibe Code Bench v1.1Code82.770.4−12.3
Terminal-Bench 2.1 Extended (Mercor held-out)Code32.744.4+11.7
TextQuests Average Cumulative HarmSafety21.024.4−11.6
AHK-EvalCode80.691.7+11.1
MedXpertQA MM Extended (Mercor held-out)Health & Medicine51.662.7+11.1
HANDBOOK.mdProfessional Work21.910.8−11.1
VariantBench V1 Clean RerunsScience42.431.4−11.0
MedXpertQA MM (200-question set)Health & Medicine72.583.5+11.0
Concept Synth INDUCTION Challenge64 Round 1General QA6.317.2+10.9
BioTIER RefusalSafety94.784.4−10.3
Finance Agent v2 - General Quantitative AnalysisFinance & Accounting71.681.8+10.2
SpatialViz-BenchMultimodal65.275.3+10.1
LAB-Bench ProtocolQA (SecureBio)Science76.986.8+9.9
LiveBench 2.0Agents76.867.2−9.6
BioSecBench-Function V1 Clean RerunsScience43.133.7−9.4
Legal Research BenchLegal43.834.6−9.1
WCB (SecureBio)Science66.758.0−8.7
PCT Political Inconsistency (CAIS)Safety53.845.1+8.7
Design Arena Overall Code CategoriesCode12521304+8.6
Agent Red Teaming k=100 (CAIS)Safety47.038.5+8.5
AutoformBenchMath44.052.5+8.5
Arena Code WebDev — Data & AnalyticsCode15201577+8.4
FrontierMath Tiers 1-3 (v2) - PrivateMath80.071.6−8.4
Finance Agent v2 - PrecedentsFinance & Accounting25.633.8+8.2
Terminal-Bench 4.0 (Artificial Analysis)Code21.713.6−8.1
Humanity's Last Exam DiamondGeneral QA23.131.2+8.1
RiemannBenchMath47.239.2−8.0
ALE-BenchCode21021876−8.0
SWE-bench VerifiedCode88.680.8−7.8
SM BenchSafety72.980.6+7.7
ArXivMath 06/2026Math70.062.5−7.5
Finance Agent v2 - Market AnalysisFinance & Accounting67.274.5+7.3
ChessBench AccuracyGames72.479.2+7.3
Tax Agent BenchFinance & Accounting64.957.7−7.2
Arena Code WebDev — HTMLCode15421593+7.2
LMCAGeneral QA57.550.4−7.1
Finance Agent v2 - Earnings AnalysisFinance & Accounting72.179.1+7.0
ComplexConstraintsChat & Writing35.642.4+6.8
SpatialBench V1 Clean RerunsScience65.859.1−6.7
Bug Hunt BenchCode14.321.0+6.7
SkillsBench with SkillsAgents59.265.9+6.7
ScBench V1 Clean RerunsScience57.150.4−6.7
EnterpriseOps-Gym-AAAgents44.050.4+6.4
LMArena Agent Arena Bash RecoveryAgents7.2-5.4−6.3
Finance Agent v2 - AdjustmentsFinance & Accounting48.454.7+6.3
Wiz Cyber Model Arena v3.9Cybersecurity65.459.1−6.3
xbench-ScienceQAScience76.470.2−6.2
MRCR v2 (8-needle, Context Arena) - 128K AUCGeneral QA87.893.9+6.1
SpatialBench-Long V0 VerifiedScience28.822.7−6.1
CharXiv ReasoningMultimodal78.584.5+6.0
TaxCalcBench TY25Finance & Accounting20.014.0−6.0
Vals RSI Index v1.1 (four tasks)Agents24.018.3−5.7
RedlineBenchLegal39.745.3+5.6
SAGEMath54.849.2−5.6
BrowseComp (Mercor 130-question subset)Search73.779.2+5.5
LAB-Bench LitQA2 (SecureBio)Science48.754.2+5.5
SciCode 1.0.1 (Artificial Analysis)Code54.459.8+5.4
BioSecBench-Surveillance V1 Clean RerunsScience46.941.5−5.4
Finance Agent v2 - ComparablesFinance & Accounting44.950.3+5.4
Machiavelli Deception (CAIS)Safety83.785.5−5.4
Conceptual Reasoning IndexIndexes68.364.6−5.1
ArXivMath 04/2026Math62.557.5−5.0
MMMU-Pro Extended (Mercor held-out)Multimodal31.436.4+5.0
LMArena Agent Arena SteerabilityAgents9.6-0.0−4.8
SpeciEval (Pooled English Runs)Safety84.389.0+4.7
SpeciEval (Refusal-Aware Retries, Historical)Safety84.389.0+4.7
GDPval-AA v2Professional Work15291433−4.7
Arena Code WebDev — SimulationsCode15761618+4.7
BioSecBench-Function V0Science44.640.1−4.5
BrowseComp Extended (Mercor held-out)Search30.025.5−4.5
TxBench Antibody Discovery V0Science45.650.0+4.4
APEX-1 (Mercor, mean score)Professional Work62.767.0+4.3
Terminal-Bench Science 0.1 (Vals)Science10.05.7−4.3
Arena Code WebDev — Consumer ProductCode15411574+4.3
Humanity's Last ExamGeneral QA49.845.6−4.2
Arena Code WebDev — Reference-Based DesignCode15771608+4.2
Arena Code WebDev — FrontendCode15601592+4.1
BabaIsBench Stage 0: The IntroGames96.0100.0+4.0
ZeroBench (Main questions, pass@5, with tools)Multimodal34.030.0−4.0
GeneBench-Pro Public Release SubsetScience18.014.0−4.0
Finance Agent v2 - All PassFinance & Accounting43.547.5+3.9
Vals Index v2.1Indexes55.151.3−3.8
EpiBench V1 Clean RerunsScience25.822.0−3.8
Terminal-Bench 2.1Code82.085.8+3.8
LegalBenchLegal83.687.3+3.7
Arena Code WebDev — ReactCode15561584+3.7
EnterpriseBench: CoreCraftProfessional Work52.355.9+3.6
SpatialBench-Long V1 Clean RerunsScience33.329.8−3.6
GPQA DiamondGeneral QA91.995.5+3.5
Arena Code WebDev — GamingCode16071638+3.5
ChessBench CoherenceGames79.783.2+3.5
LAB-Bench CloningScenarios (SecureBio)Science95.999.4+3.5
Remote Labor Index (RLI)Professional Work8.35.0−3.3
FrontierCode 1.1 Extended - ScoreCode59.656.3−3.3
Arena Code WebDev — Content Creation ToolsCode15541574+3.3
MortgageTax (Vals all 1,258 documents)Finance & Accounting69.966.7−3.3
FrontierCode 1.1 Extended - Pass RateCode65.562.3−3.2
ARC-AGI-1 Semi-PrivateGeneral QA92.595.5+3.0
WordleBenchGames94.097.0+3.0
Artificial Analysis Intelligence Index v4.3.2Indexes41.839.1−2.7
NoCoT-Bench NCKI (kspine v3)General QA117121+2.7
VCT v2 (SecureBio)Science45.948.6+2.7
Artificial Analysis Intelligence Index v4.3Indexes42.039.4−2.6
GDPval-AA v2.1Professional Work14611404−2.5
DistressBenchSafety71.469.0−2.5
CharXiv Extended (Mercor held-out)Multimodal76.779.1+2.4
MMMU-Pro (Vals 4-Option)Multimodal86.689.0+2.4
TxBench Preclinical Pharmacology V1 Clean RerunsScience57.755.3−2.3
ScBench-Long V1 Clean RerunsScience16.518.8+2.3
CyberArenaCybersecurity65.067.1+2.2
CharXiv (descriptive and reasoning combined)Multimodal93.295.3+2.1
Finance Agent v2 - Disclosure AnalysisFinance & Accounting65.767.7+2.0
SpatialBench V0 VerifiedScience66.464.3−2.0
Vision Arena OverallMultimodal12861294+2.0
Excel Modeling BenchmarkFinance & Accounting69.471.3+2.0
APEX-SWECode43.942.0−1.9
DTBenchGeneral QA94.996.8+1.9
MedScribeHealth & Medicine85.883.9−1.8
Document ArenaProfessional Work14751476+1.7
GlobeBenchGeneral QA20.021.8+1.7
Vals Index v2Indexes60.959.3−1.6
HPCT (SecureBio)Science63.164.7+1.6
AIME 2025Math98.396.7−1.6
ScBench-Long V0 VerifiedScience25.824.2−1.5
GBENCH One-shot Coding GScoreGames54.052.6−1.4
BioMysteryBench - Human SolvableScience88.587.1−1.4
MBCT (SecureBio)Science56.455.0−1.4
OncologyBench Strict (25 tasks, October 9 rubric)Health & Medicine16.014.7−1.3
LMArena Text Arena EloChat & Writing14821489+1.3
MineBench — Bradley-Terry RatingGames17861765−1.3
𝜏³-BankingAgents34.235.5+1.2
SkillsBench without SkillsAgents43.744.9+1.2
DeepResearch Bench II (Mercor 130-task set)Search44.643.4−1.2
Gray Swan IPI Attack Success (k=15)Safety8.09.2−1.2
Epoch Capabilities Index (ECI)Indexes160158−1.2
AA-Omniscience IndexIndexes28.826.5−1.1
Finance Agent v2 - General Qualitative AnalysisFinance & Accounting66.465.3−1.1
OTIS Mock AIME 2024-2025Math98.397.2−1.1
BioMysteryBench - Human DifficultScience42.443.5+1.1
GDPval Extended (Mercor held-out rubric score)Professional Work58.957.8−1.1
VCT (SecureBio)Science48.249.3+1.1
Bio Capabilities Index (SecureBio)Indexes164162−1.1
Mystery Game PuzzlesGames36.037.0+1.0
ProgramBenchCode1.00.0−1.0
PG-LLM-50Science0.40.3−1.0
ATLAS-Education (StudentBench) Learning GainProfessional Work16.014.1−1.0
TaxEval v2Finance & Accounting75.674.7−0.9
Arena Code WebDev — Brand & MarketingCode15471553+0.9
Finance Agent v2 - Financial ModelingFinance & Accounting23.324.2+0.8
LiveCodeBench (Vals Public) - MediumCode89.890.6+0.8
Harvey HLAB — Criteria Pass RateLegal87.988.6+0.7
GDPval (Mercor 213-task rubric score)Professional Work84.585.2+0.7
GPQA-Bio (SecureBio)Science80.981.6+0.7
ObviousBenchGeneral QA99.3100.0+0.7
MMLU ProGeneral QA90.090.6+0.6
MMLU-Bio (SecureBio)Science90.991.4+0.5
ACCoRDGeneral QA77.477.0−0.4
Artificial Analysis Progress Index v1Indexes318317−0.3
Toloka Arena (v1)Agents54.354.6+0.3
TxBench Oligo Discovery V1 Clean RerunsScience35.335.6+0.3
GDP.pdfProfessional Work24.023.8−0.2
MedCodeHealth & Medicine53.253.4+0.2
TxBench Oligo Discovery V0Science39.439.5+0.1
Arena Alignment IndexSafety74.574.4−0.1
ForecastBench Preliminary Dataset Brier IndexForecasting61.661.7+0.1
Harvey LAB (Mercor run, 1,749 public tasks)Legal9.49.3−0.1
ForecastBench Tournament Overall Brier IndexForecasting66.266.20.0
Harvey's Legal Agent Benchmark - Antitrust CompetitionLegal0.00.00.0
Harvey's Legal Agent Benchmark - Arbitration International Dispute ResolutionLegal0.00.00.0
Harvey's Legal Agent Benchmark - Bankruptcy RestructuringLegal0.00.00.0
Harvey's Legal Agent Benchmark - Capital MarketsLegal20.020.00.0
Harvey's Legal Agent Benchmark - Corporate M&ALegal0.00.00.0
Harvey's Legal Agent Benchmark - Employment LaborLegal0.00.00.0
Harvey's Legal Agent Benchmark - Environmental ESGLegal0.00.00.0
Harvey's Legal Agent Benchmark - Funds Asset ManagementLegal0.00.00.0
Harvey's Legal Agent Benchmark - ImmigrationLegal0.00.00.0
Harvey's Legal Agent Benchmark - Intellectual PropertyLegal0.00.00.0
Harvey's Legal Agent Benchmark - International Trade SanctionsLegal0.00.00.0
Harvey's Legal Agent Benchmark - Litigation Dispute ResolutionLegal0.00.00.0
Harvey's Legal Agent Benchmark - Real EstateLegal0.00.00.0
Harvey's Legal Agent Benchmark - TaxLegal0.00.00.0
Harvey's Legal Agent Benchmark - Trusts Estates Private ClientLegal0.00.00.0
Harvey's Legal Agent Benchmark - White Collar Defense InvestigationsLegal0.00.00.0
LiveCodeBench (Vals Public) - EasyCode99.199.10.0

Opus 4.8vsGemini 3.7 Flashnormalised score marginsData Nov 15, 2026Data: Vals AI, Mercor, benchmarks.bio and 56 othersVillage LLM Atlasatlas.village.dev

Win counts use each benchmark’s declared score direction. Represented suite slices and the derived Capability score are excluded from headline totals. Missing or incompatible setups are excluded, never counted as losses.
Loading Atlas data…