Qwen2-57B-A14B
Alibaba · 2024-06-07 · 57.4B parameters
Qwen2-57B-A14B is the base (pretrained) Mixture-of-Experts checkpoint of Alibaba's Qwen2 series, released on June 7, 2024. It has 57,408,658,944 total parameters and activates about 14 billion per token, routing each token to 8 of 64 experts plus a shared expert, and Qwen documents a 65,536-token context length; Qwen2-57B-A14B-Instruct is its instruction-tuned counterpart.
Benchmark scores
| Benchmark | Score |
|---|---|
| ARC-Challenge | 64.1 |
| BBH (BIG-Bench Hard) | 67.0 |
| C-Eval | 87.7 |
| Capability | 109.6 |
| CMMLU | 88.5 |
| GPQA - release-reported, subset unspecified | 34.3 |
| GSM8K | 80.7 |
| HellaSwag (Unspecified Scoring Protocol) | 85.2 |
| HumanEval | 53.0 |
| HumanEval+ | 46.3 |
| MATH | 43.0 |
| MBPP | 71.9 |
| MBPP+ | 57.4 |
| MMLU | 76.5 |
| MMLU Pro | 43.0 |
| MMLU STEM | 69.8 |
| MMLU-Redux | 72.4 |
| MultiPL-E (HumanEval, 8-language average) | 49.8 |
| TheoremQA | 33.5 |
| TruthfulQA (Unspecified Metric) | 57.7 |
| WinoGrande | 79.5 |