TruthfulQA generation - % truthful
Safety · 2021-09-08
TruthfulQA's generation task asks 817 questions written so that repeating common misconceptions gives false answers. A fine-tuned GPT-judge model rates each free-form answer, and this score is the percentage judged truthful, whether or not it is informative, so refusing to answer also counts as truthful.
Top models (higher is better)
| Model | Score |
|---|---|
| Llama 2 70B Chat | 67.1 |
| Llama 2 13B Chat | 65.7 |
| Llama 2 7B Chat | 60.6 |
| Llama 2 70B | 53.4 |
| Code Llama Instruct 34B | 50.9 |
| Llama 2 34B | 46.1 |
| Llama 2 13B | 45.6 |
| Falcon 40B | 44.8 |
| Code Llama Instruct 13B | 44.3 |
| Code Llama 13B | 43.0 |
| Code Llama 34B | 40.4 |
| MPT-30B | 40.3 |
| Llama 2 7B | 39.5 |
| Code Llama 7B | 38.3 |
| Code Llama Instruct 7B | 37.0 |
| MPT-7B | 36.7 |
| Falcon 7B | 29.0 |