← All models

Mistral-NeMo-Minitron 8B Base

NVIDIA · 2024-08-21 · 8.4B parameters

Mistral-NeMo-Minitron 8B Base is NVIDIA's open-weight 8B base language model obtained by width-pruning Mistral NeMo 12B and distilling it on 380 billion tokens, released on August 21, 2024; the BF16 checkpoint contains exactly 8,414,105,600 parameters. It keeps a 40-layer decoder with grouped-query attention, RoPE and a 131,072-token Tekken tokenizer vocabulary, with an 8,192-token context. It is released under the NVIDIA Open Model License.

Benchmark scores