DeepSeek LLM 67B Base
DeepSeek · 2023-11-29 · 67.4B parameters
DeepSeek LLM 67B Base is the 67B dense base model of DeepSeek's first general LLM release, made public on November 29, 2023. It is a 95-layer decoder with grouped-query attention (64 query and 8 key-value heads) and a 4,096-token context, pretrained from scratch on a reported 2 trillion mainly English and Chinese tokens; it shares the 67,425,001,472-parameter architecture of DeepSeek LLM 67B Chat.
Benchmark scores
| Benchmark | Score |
|---|---|
| AGIEval | 41.3 |
| BBH (BIG-Bench Hard) | 68.7 |
| C-Eval | 66.1 |
| C3 | 75.3 |
| Capability | 105.8 |
| CCPM | 88.5 |
| ChID | 92.1 |
| CLUEWSC | 81.0 |
| CMATH | 63.0 |
| CMMLU | 70.8 |
| CMRC 2018 | 73.4 |
| CRUXEval-I | 42.5 |
| CRUXEval-O | 41.0 |
| GSM8K | 63.4 |
| HumanEval-Mul | 37.2 |
| MATH | 18.7 |
| MBPP | 57.4 |
| MMLU | 71.3 |
| Natural Questions (open-domain, closed-book) | 36.6 |
| OpenBookQA | 60.2 |
| PIQA | 83.6 |
| RACE-High | 50.7 |
| RACE-Middle | 69.9 |
| TriviaQA | 78.9 |