Nemotron 3 Super 120B A12B Base
NVIDIA · 2026-03-11 · 123.6B parameters
Nemotron 3 Super 120B A12B Base is NVIDIA's pretrained (not post-trained) LatentMoE hybrid Mamba-2/Transformer base model with multi-token prediction layers, the base of Nemotron 3 Super 120B A12B. The BF16 checkpoint contains exactly 123,611,012,096 parameters in 88 layers with 512 routed experts of which 22 are active per token; NVIDIA reports about 12B active parameters. It was pretrained over a 25-trillion-token horizon with an NVFP4-based recipe and supports up to 1M tokens of context.
Benchmark scores
| Benchmark | Score |
|---|---|
| AGIEval English | 77.9 |
| ARC-Challenge | 96.1 |
| Capability | 128.2 |
| GPQA Diamond | 60.0 |
| GSM8K | 90.7 |
| MATH | 84.8 |
| MATH Level 5 | 70.0 |
| MGSM Chinese | 85.6 |
| MGSM French | 85.6 |
| MGSM German | 90.4 |
| MGSM Japanese | 81.6 |
| MGSM Russian | 91.2 |
| MGSM Spanish | 88.0 |
| MMLU | 86.0 |
| PIQA | 85.5 |
| WinoGrande | 78.9 |