CLUEWSC

Name: CLUEWSC
Published: 2020-04-13

General QA · 2020-04-13

CLUEWSC2020 is the Chinese Winograd Schema Challenge task of the CLUE benchmark: each item asks whether a pronoun in a Chinese sentence refers to a given noun phrase, a binary commonsense coreference decision. Scores are accuracy. Higher is better.

Top models (higher is better)

Model	Score
DeepSeek-R1-Zero	93.1
DeepSeek-R1	92.8
Qwen2.5 72B Instruct	91.4
DeepSeek-V3	90.9
DeepSeek-V2.5	90.4
Qwen1.5 72B Chat	90.1
DeepSeek-V2-Chat	89.9
o1-mini	89.9
GPT-4o (2024-05-13)	87.9
DeepSeek-Coder-V2-Instruct	85.9
Claude 3.5 Sonnet (Oct 2024)	85.4
Llama 3 70B Instruct	85.4
DeepSeek-V4-Pro-Base	85.2
Llama 3.1 405B Instruct	84.7
Llama 3.1 405B	83.0
DeepSeek-V3-Base	82.7
Qwen2.5 72B	82.5
DeepSeek-V4-Flash-Base	82.2
DeepSeek LLM 67B Base	81.0
Qwen1.5 72B	80.5
DeepSeek-V2-Lite-Chat	80.0
Llama 3 70B	78.3
Mixtral 8x22B	77.5
DeepSeek-Coder-V2-Lite-Instruct	76.5
Mixtral 8x22B Instruct	75.8
DeepSeek-V2-Lite	74.3
DeepSeek LLM 7B Base	73.1
DeepSeekMoE 16B Base	72.1
DeepSeekMoE 16B Chat	68.2