← All models

Moonlight 16B A3B

Moonshot AI · 2025-02-22 · 16B parameters

Moonlight-16B-A3B is Moonshot AI's open-weight base Mixture-of-Experts language model, released in February 2025 with the Muon is Scalable for LLM Training report to show that the Muon optimizer scales to large-model pre-training. It follows the DeepSeek-V3-style MoE architecture with MLA attention (64 routed experts, six selected per token), about 16B total and 3B activated parameters (the card lists 15.29B and 2.24B excluding embeddings), was trained on 5.7 trillion tokens, and supports an 8K context. Its BF16 tensors contain 15,960,111,936 parameters.

Benchmark scores