← All models

Nemotron-4 340B Base

NVIDIA · 2024-06-14

Nemotron-4 340B Base is NVIDIA's open-access 340B-class dense decoder-only base model, released on June 14, 2024 together with Instruct and Reward variants for synthetic-data generation. NVIDIA reports 9.4 billion embedding and 331.6 billion non-embedding parameters (about 341 billion in total; no exact checkpoint count is published), 96 layers with grouped-query attention, partial rotary embeddings, squared-ReLU MLPs, untied embeddings and a 256,000-token SentencePiece vocabulary. It was pretrained on 9 trillion tokens (8 trillion followed by 1 trillion of continued pretraining) with a 4,096-token context and is distributed in NeMo format under the NVIDIA Open Model License.

Benchmark scores