← All models

DeepSeek-V2-Lite

DeepSeek · 2024-05-16 · 15.7B parameters

DeepSeek-V2-Lite is the small MLA/DeepSeekMoE base model released in May 2024 alongside the DeepSeek-V2 paper's appendix. Its released weights contain 15,706,484,224 parameters (DeepSeek's rounded 2.4B activated per token), with 27 layers, 2 shared and 64 routed experts per MoE layer and 6 routed experts active; it was trained from scratch on 5.7T tokens of the V2 corpus and extended to a documented 32K context, although its YaRN configuration declares 163,840 positions.

Benchmark scores