DeepSeekMoE 16B Chat
DeepSeek · 2024-01-11 · 16.4B parameters
DeepSeekMoE 16B Chat is the supervised-fine-tuned chat checkpoint of DeepSeekMoE 16B Base, released with the DeepSeekMoE paper in January 2024. It keeps the base model's fine-grained MoE design (2 shared and 64 routed experts, 6 routed active per token), 16,375,728,128 released parameters and 4,096-token context.
Benchmark scores
| Benchmark | Score |
|---|---|
| AGIEval | 19.7 |
| ARC-Challenge | 50.0 |
| ARC-Easy | 69.9 |
| BBH (BIG-Bench Hard) | 42.2 |
| C-Eval | 40.0 |
| Capability | 90.2 |
| CLUEWSC | 68.2 |
| CMATH | 67.9 |
| CMMLU | 49.3 |
| GSM8K | 62.2 |
| HellaSwag (Unspecified Scoring Protocol) | 72.2 |
| HumanEval | 45.7 |
| MATH | 15.2 |
| MBPP | 46.2 |
| MMLU | 47.2 |
| Natural Questions (open-domain, closed-book) | 35.1 |
| PIQA | 79.7 |
| RACE-High | 50.6 |
| RACE-Middle | 64.8 |
| TriviaQA | 63.3 |
| WinoGrande | 69.0 |