Qwen1.5 MoE A2.7B
Alibaba · 2024-03-28 · 14.3B parameters
Qwen1.5-MoE-A2.7B is the base (pretrained) Mixture-of-Experts checkpoint of Alibaba's Qwen1.5 family, released on March 28, 2024 and upcycled from Qwen-1.8B. It has 14,315,784,192 total parameters and activates about 2.7 billion per token, using 60 routed experts with 4 selected per token plus shared experts; its shipped configuration sets 8,192 maximum positions. Qwen1.5-MoE-A2.7B-Chat is its aligned counterpart.
Benchmark scores
| Benchmark | Score |
|---|---|
| BBH (BIG-Bench Hard) | 39.1 |
| C-Eval | 79.2 |
| Capability | 93.3 |
| CMMLU | 79.2 |
| GSM8K | 61.5 |
| HumanEval | 34.2 |
| MATH | 21.9 |
| MBPP | 36.6 |
| MMLU | 62.5 |