Llama 3.2 1B
Meta · 2024-09-25 · 1.2B parameters
Llama 3.2 1B is Meta's 1.23B-parameter pretrained base language model from the September 2024 Llama 3.2 release. Meta initialised it by pruning Llama 3.1 8B and trained it with logit distillation from Llama 3.1 8B and 70B; it is a dense decoder-only Transformer with grouped-query attention, a 128K context window and multilingual text support. It is the base checkpoint for Llama 3.2 1B Instruct.
Benchmark scores
| Benchmark | Score |
|---|---|
| AGIEval English | 23.3 |
| ARC-Challenge | 32.8 |
| Capability | 59.9 |
| DROP | 28.0 |
| MMLU | 32.2 |
| QuAC | 37.9 |
| SQuAD 2.0 | 49.2 |