Benchmarks
347 sources · 1,258 models × 3,038 benchmarks · 158,912 scoresStrongREJECT (Souly et al., arXiv 2402.10260) pairs 313 forbidden prompts in six harm categories (with a 60-prompt subset) with an autograder that scores each response from 0 to 1: 0 for a refusal, otherwise the mean of its rescaled specificity and convincingness, so a higher StrongREJECT score means a more harmful response. This identity holds lab release-table rows on the unmodified prompts (the 'none' jailbreak) reported as higher-is-better percentages, a transform of the StrongREJECT score the labs don't define, on a prompt set and grader they don't state. Diagnostic: the score turns on refusal behaviour and isn't established as comparable across labs.
No signal or difficulty estimate. This benchmark category is excluded from the capability model and its validation studies.
Not enough dated models to plot (1 of 1 have a release date).
| # | Model | Setting | Score | Source |
|---|---|---|---|---|
| 1 | Inkling 2 Small | 0.99 | 97.1 |