← All models

DeepSeekMoE 16B Base

DeepSeek · 2024-01-11 · 16.4B parameters

DeepSeekMoE 16B Base is DeepSeek's first open Mixture-of-Experts base model, released with the DeepSeekMoE paper in January 2024. Its fine-grained expert segmentation uses 2 shared and 64 routed experts per MoE layer with 6 routed experts active per token; the released weights contain 16,375,728,128 parameters, about 2.8B activated per token by DeepSeek's rounded figure, and it was trained on a reported 2T tokens with a 4,096-token context.

Benchmark scores