Pushback resistance
The model first gives a correct answer, then the user pushes back with a wrong claim. The score is how often the model keeps the correct answer. A few items run the other way: the user is right, and the model should accept the correction.
This category is shown on its own and does not count toward Overall. How the battery works.
Ranking
Models are ranked by their mean score in this category, from 0 to 100. Only models with at least 80% of the category's items scored, and none pending, are ranked here. The rule is the same as for Overall.
| Rank | Model | Mean (0 to 100) | Items scored |
|---|---|---|---|
| 1 | Cogito 8B V1 Preview Llamafine-tune of Llama 3.1 | 100.0 | 12 / 12 |
| 1 | Deepseek R1 8B Llama Distill | 100.0 | 12 / 12 |
| 1 | DeepSeek-Coder-V2-Lite-Instructcode model | 100.0 | 12 / 12 |
| 1 | DeepSeek-R1-Distill-Qwen-14B | 100.0 | 12 / 12 |
| 1 | DeepSeek-R1-Distill-Qwen-32B | 100.0 | 12 / 12 |
| 1 | Devstral Small 2 24B Instruct (2512)code model | 100.0 | 12 / 12 |
| 1 | Falcon3 7B Instruct | 100.0 | 12 / 12 |
| 1 | Flint 40B A2B | 100.0 | 12 / 12 |
| 1 | Gemma 4 12B IT | 100.0 | 12 / 12 |
| 1 | gemma-3-27b-it | 100.0 | 12 / 12 |
| 1 | gemma-4-26B-A4B-it | 100.0 | 12 / 12 |
| 1 | gemma-4-31B-it | 100.0 | 12 / 12 |
| 1 | Gemma2 27B Instruct | 100.0 | 12 / 12 |
| 1 | GLM-4.7-Flash | 100.0 | 12 / 12 |
| 1 | gpt-oss-20b | 100.0 | 12 / 12 |
| 1 | granite-4.1-3b | 100.0 | 12 / 12 |
| 1 | Granite4 Small H | 100.0 | 12 / 12 |
| 1 | Lfm2.5 8B A1B | 100.0 | 12 / 12 |
| 1 | Lfm3 1.7B | 100.0 | 12 / 12 |
| 1 | Lfm3 VL 2B | 100.0 | 12 / 12 |
| 1 | Llama 4 Scout 17B-16E Instruct | 100.0 | 12 / 12 |
| 1 | Llama-3.3-70B-Instruct | 100.0 | 12 / 12 |
| 1 | Medgemma1.5 4B ITfine-tune of Gemma 3medical model | 100.0 | 12 / 12 |
| 1 | Ministral 3 14B Instruct (2512) | 100.0 | 12 / 12 |
| 1 | Ministral 3 8B Instruct 2512 | 100.0 | 12 / 12 |
| 1 | Mistral Small 24B Instruct 2501 | 100.0 | 12 / 12 |
| 1 | Mistral Small 3.2 24B Instruct (2506) | 100.0 | 12 / 12 |
| 1 | Mistral Small3.1 24B Instruct 2503 | 100.0 | 12 / 12 |
| 1 | Muse Glimmer 30B | 100.0 | 12 / 12 |
| 1 | Nanbeige_Nanbeige4.3-33B | 100.0 | 12 / 12 |
| 1 | Nemotron 3.5 Bolt E4B | 100.0 | 12 / 12 |
| 1 | Olmo2 13B 1124 Instruct | 100.0 | 12 / 12 |
| 1 | Ornith 9B | 100.0 | 12 / 12 |
| 1 | phi-4 | 100.0 | 12 / 12 |
| 1 | Phi4 Mini Reasoning 3.8B | 100.0 | 12 / 12 |
| 1 | Qwen2.5-32B-Instruct | 100.0 | 12 / 12 |
| 1 | Qwen2.5-VL-32B-Instruct | 100.0 | 12 / 12 |
| 1 | Qwen3-30B-A3B-Instruct-2507 | 100.0 | 12 / 12 |
| 1 | Qwen3-32B | 100.0 | 12 / 12 |
| 1 | Qwen3-4B | 100.0 | 12 / 12 |
| 1 | Qwen3-4B-Instruct-2507 | 100.0 | 12 / 12 |
| 1 | Qwen3-4B-Thinking-2507 | 100.0 | 12 / 12 |
| 1 | Qwen3-8B | 100.0 | 12 / 12 |
| 1 | Qwen3-VL-32B-Instruct | 100.0 | 12 / 12 |
| 1 | Qwen3-VL-8B-Instruct | 100.0 | 12 / 12 |
| 1 | Qwen3.5-27B | 100.0 | 12 / 12 |
| 1 | Qwen3.5-35B-A3B | 100.0 | 12 / 12 |
| 1 | Qwen3.5-9B | 100.0 | 12 / 12 |
| 1 | Qwen3.6-27B | 100.0 | 12 / 12 |
| 1 | Qwen3.8-27B | 100.0 | 12 / 12 |
| 1 | QwQ 32B | 100.0 | 12 / 12 |
| 1 | Sage-A1.5-Q4_K_M | 100.0 | 12 / 12 |
| 53 | Aya Expanse 8B | 91.7 | 12 / 12 |
| 53 | Deepcoder 14B Previewfine-tune of DeepSeek-R1-Distill-Qwen 14Bcode model | 91.7 | 12 / 12 |
| 53 | DeepSeek-R1-0528-Qwen3-8B | 91.7 | 12 / 12 |
| 53 | dolphin-2.9.1-yi-1.5-34b | 91.7 | 12 / 12 |
| 53 | Exaone Deep 7.8B | 91.7 | 12 / 12 |
| 53 | Exaone3.5 32B Instruct | 91.7 | 12 / 12 |
| 53 | Gemma 3n E4B IT | 91.7 | 12 / 12 |
| 53 | Gemma 7B Instruct V1.1 | 91.7 | 12 / 12 |
| 53 | gemma-2-9b-it | 91.7 | 12 / 12 |
| 53 | gemma-3-12b-it | 91.7 | 12 / 12 |
| 53 | Gemma3n E2B IT | 91.7 | 12 / 12 |
| 53 | Gemma4 E2B IT | 91.7 | 12 / 12 |
| 53 | Gemma4 E4B IT | 91.7 | 12 / 12 |
| 53 | gpt-oss-120b | 91.7 | 12 / 12 |
| 53 | Granite3 Dense 8B Instruct | 91.7 | 12 / 12 |
| 53 | Granite3.3 8B | 91.7 | 12 / 12 |
| 53 | Granite4 Micro H | 91.7 | 12 / 12 |
| 53 | Granite4 Tiny H | 91.7 | 12 / 12 |
| 53 | Granite4.1 30B | 91.7 | 12 / 12 |
| 53 | Granite4.1 8B | 91.7 | 12 / 12 |
| 53 | Granite4.2 30B | 91.7 | 12 / 12 |
| 53 | Granite4.2 3B | 91.7 | 12 / 12 |
| 53 | Laguna XS 2.1 | 91.7 | 12 / 12 |
| 53 | Lfm2 24B | 91.7 | 12 / 12 |
| 53 | Marco O1 7Bfine-tune of Qwen 2 7B | 91.7 | 12 / 12 |
| 53 | Medgemma 27B ITfine-tune of Gemma 3medical model | 91.7 | 12 / 12 |
| 53 | Mistral-7B-Instruct-v0.2 | 91.7 | 12 / 12 |
| 53 | Nemotron 3.5 Lightning 30B A3B | 91.7 | 12 / 12 |
| 53 | Nemotron Cascade 2 30B A3B | 91.7 | 12 / 12 |
| 53 | Nemotron3 33B | 91.7 | 12 / 12 |
| 53 | Olmo 3.1 32B Instruct | 91.7 | 12 / 12 |
| 53 | Olmo-3-7B-Instruct | 91.7 | 12 / 12 |
| 53 | Ornith-1.5-9B | 91.7 | 12 / 12 |
| 53 | Phi3 3.8B Mini 128K Instruct | 91.7 | 12 / 12 |
| 53 | Qwen2.5-14B-Instruct | 91.7 | 12 / 12 |
| 53 | Qwen2.5-3B-Instruct | 91.7 | 12 / 12 |
| 53 | Qwen2.5-7B-Instruct | 91.7 | 12 / 12 |
| 53 | Qwen2.5-Coder-32B-Instructcode model | 91.7 | 12 / 12 |
| 53 | Qwen2.5-VL-7B-Instruct | 91.7 | 12 / 12 |
| 53 | Qwen3-14B | 91.7 | 12 / 12 |
| 53 | Qwen3-30B-A3B | 91.7 | 12 / 12 |
| 53 | Qwen3-Coder-Nextcode model | 91.7 | 12 / 12 |
| 53 | Qwen3.6-35B-A3B | 91.7 | 12 / 12 |
| 53 | Smallthinker 3B Previewfine-tune of Qwen 2.5 3B | 91.7 | 12 / 12 |
| 53 | Translategemma 12B ITfine-tune of Gemma 3translation model | 91.7 | 12 / 12 |
| 53 | Tulu3 8Bfine-tune of Llama 3.1 | 91.7 | 12 / 12 |
| 99 | Apertus-8B-Instruct-2509 | 83.3 | 12 / 12 |
| 99 | Codegeex4 9B Allcode model | 83.3 | 12 / 12 |
| 99 | Codestral 22B V0.1code model | 83.3 | 12 / 12 |
| 99 | Cogito 14B V1 Preview Qwenfine-tune of Qwen 2.5 | 83.3 | 12 / 12 |
| 99 | Command R 35B 08 2024 | 83.3 | 12 / 12 |
| 99 | Command R7B Arabic 7B 02 2025 | 83.3 | 12 / 12 |
| 99 | Devstral 24B Small 2505code model | 83.3 | 12 / 12 |
| 99 | Gemma2 2B Instruct | 83.3 | 12 / 12 |
| 99 | Glm4 9B Chat | 83.3 | 12 / 12 |
| 99 | Granite3.3 2B | 83.3 | 12 / 12 |
| 99 | Granite4 3B | 83.3 | 12 / 12 |
| 99 | Granite4.2 8B | 83.3 | 12 / 12 |
| 99 | Hermes3 3B Llama3.2fine-tune of Llama 3.2 | 83.3 | 12 / 12 |
| 99 | Internlm2 7B Chat V2.5 | 83.3 | 12 / 12 |
| 99 | Llama-3.1-8B-Instruct | 83.3 | 12 / 12 |
| 99 | Llama-3.2-3B-Instruct | 83.3 | 12 / 12 |
| 99 | Mathstral 7B V0.1fine-tune of Mistral 7Bmath model | 83.3 | 12 / 12 |
| 99 | medgemma-4b-itmedical model | 83.3 | 12 / 12 |
| 99 | Ministral 3 3B Instruct 2512 | 83.3 | 12 / 12 |
| 99 | Mistral Small 22B Instruct 2409 | 83.3 | 12 / 12 |
| 99 | Mixtral 8x7B Instruct V0.1 | 83.3 | 12 / 12 |
| 99 | NVIDIA Nemotron 3 Nano 30B-A3B | 83.3 | 12 / 12 |
| 99 | Olmo2 7B 1124 Instruct | 83.3 | 12 / 12 |
| 99 | Openthinker 7B V2fine-tune of Qwen 2.5 | 83.3 | 12 / 12 |
| 99 | Phi-3.5-mini-instruct | 83.3 | 12 / 12 |
| 99 | Phi-4-mini-instruct | 83.3 | 12 / 12 |
| 99 | Qwen2 Math 1.5B Instructmath model | 83.3 | 12 / 12 |
| 99 | Qwen2.5-Coder-14B-Instructcode model | 83.3 | 12 / 12 |
| 99 | Qwen2.5-VL-3B-Instruct | 83.3 | 12 / 12 |
| 99 | Qwen3-Coder-30B-A3B-Instructcode model | 83.3 | 12 / 12 |
| 99 | Qwen3-VL-30B-A3B-Instruct | 83.3 | 12 / 12 |
| 99 | Qwen3.5-4B | 83.3 | 12 / 12 |
| 99 | SmolLM3-3B | 83.3 | 12 / 12 |
| 99 | Solar 10.7B Instruct V1 | 83.3 | 12 / 12 |
| 99 | Spark-X2.6-4B | 83.3 | 12 / 12 |
| 99 | YI 9B Chat V1.5 | 83.3 | 12 / 12 |
| 135 | Aya Expanse 32B | 75.0 | 12 / 12 |
| 135 | Codegemma 7B Instruct V1.1code model | 75.0 | 12 / 12 |
| 135 | Command R7B (12-2024) | 75.0 | 12 / 12 |
| 135 | Deepscaler 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bmath model | 75.0 | 12 / 12 |
| 135 | Deepseek R1 1.5B Qwen Distill | 75.0 | 12 / 12 |
| 135 | DeepSeek-R1-Distill-Qwen-7B | 75.0 | 12 / 12 |
| 135 | Exaone Deep 2.4B | 75.0 | 12 / 12 |
| 135 | EXAONE-3.5-7.8B-Instruct | 75.0 | 12 / 12 |
| 135 | Exaone3.5 2.4B Instruct | 75.0 | 12 / 12 |
| 135 | Falcon3 10B Instruct | 75.0 | 12 / 12 |
| 135 | gemma-3-4b-it | 75.0 | 12 / 12 |
| 135 | GLM-4.5-Air | 75.0 | 12 / 12 |
| 135 | Granite3 Dense 2B Instruct | 75.0 | 12 / 12 |
| 135 | Granite3.2 2B Instruct | 75.0 | 12 / 12 |
| 135 | Granite3.2 8B Instruct | 75.0 | 12 / 12 |
| 135 | Granite4 1B H | 75.0 | 12 / 12 |
| 135 | Hermes-3-Llama-3.1-8B | 75.0 | 12 / 12 |
| 135 | Meta-Llama-3-8B-Instruct | 75.0 | 12 / 12 |
| 135 | Mistral Nemo 12B Instruct 2407 | 75.0 | 12 / 12 |
| 135 | Qwen 7B Chat V1.5 | 75.0 | 12 / 12 |
| 135 | Qwen2 Math 7B Instructmath model | 75.0 | 12 / 12 |
| 135 | Qwen2.5 1.5B Instruct | 75.0 | 12 / 12 |
| 135 | Qwen2.5-Coder-3B-Instructcode model | 75.0 | 12 / 12 |
| 135 | Qwen3 1.7B | 75.0 | 12 / 12 |
| 135 | Qwen3 VL 2B Instruct | 75.0 | 12 / 12 |
| 135 | Qwen3-VL-4B-Instruct | 75.0 | 12 / 12 |
| 135 | Qwen3.5 2B | 75.0 | 12 / 12 |
| 135 | Sailor2 8B Chatfine-tune of Qwen 2.5 | 75.0 | 12 / 12 |
| 135 | YI 34B Chat V1.5 | 75.0 | 12 / 12 |
| 135 | YI 6B Chat V1.5 | 75.0 | 12 / 12 |
| 135 | YI Coder 9B Chatcode model | 75.0 | 12 / 12 |
| 166 | Aya 8B 23 | 66.7 | 12 / 12 |
| 166 | Codeqwen 7B Chat V1.5code model | 66.7 | 12 / 12 |
| 166 | Cogito 3B V1 Preview Llamafine-tune of Llama 3.2 | 66.7 | 12 / 12 |
| 166 | Deepcoder 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bcode model | 66.7 | 12 / 12 |
| 166 | Falcon3 1B Instruct | 66.7 | 12 / 12 |
| 166 | Falcon3 3B Instruct | 66.7 | 12 / 12 |
| 166 | Llama-2-7b-chat-hf | 66.7 | 12 / 12 |
| 166 | Llama2 13B Chat | 66.7 | 12 / 12 |
| 166 | Mistral 7B Instruct V0.3 | 66.7 | 12 / 12 |
| 166 | Olmo-3-7B-Think | 66.7 | 12 / 12 |
| 166 | Phi-3-mini-4k-instruct | 66.7 | 12 / 12 |
| 166 | Phi3 14B Medium 128K Instruct | 66.7 | 12 / 12 |
| 166 | Qwen 1.8B Chat V1.5 | 66.7 | 12 / 12 |
| 166 | Qwen 14B Chat V1.5 | 66.7 | 12 / 12 |
| 166 | Qwen2 1.5B Instruct | 66.7 | 12 / 12 |
| 166 | Qwen2-VL-7B-Instruct | 66.7 | 12 / 12 |
| 166 | Qwen2.5 Coder 1.5B Instructcode model | 66.7 | 12 / 12 |
| 166 | Sailor2 1B Chatfine-tune of Qwen 2.5 | 66.7 | 12 / 12 |
| 184 | Deepseek Coder 33B Instructcode model | 58.3 | 12 / 12 |
| 184 | Deepseek V2 16B Lite Chat | 58.3 | 12 / 12 |
| 184 | Dolphin Phi 2.7B V2.6fine-tune of Phi-2 | 58.3 | 12 / 12 |
| 184 | Gemma3 1B IT | 58.3 | 12 / 12 |
| 184 | Granite Code 3B Instructcode model | 58.3 | 12 / 12 |
| 184 | Granite Code 8B Instructcode model | 58.3 | 12 / 12 |
| 184 | Granite3 Moe 1B Instruct | 58.3 | 12 / 12 |
| 184 | Granite3.1 Dense 2B Instruct | 58.3 | 12 / 12 |
| 184 | Granite3.1 Moe 1B Instruct | 58.3 | 12 / 12 |
| 184 | Granite3.1 Moe 3B Instruct | 58.3 | 12 / 12 |
| 184 | Opencoder 1.5B Instructcode model | 58.3 | 12 / 12 |
| 184 | Qwen2.5 0.5B Instruct | 58.3 | 12 / 12 |
| 184 | Qwen2.5-Coder-7B-Instructcode model | 58.3 | 12 / 12 |
| 184 | Smollm2 1.7B Instruct | 58.3 | 12 / 12 |
| 198 | Codellama 13B Instructcode model | 50.0 | 12 / 12 |
| 198 | Codellama 7B Instructcode model | 50.0 | 12 / 12 |
| 198 | Deepseek Coder 6.7B Instructcode model | 50.0 | 12 / 12 |
| 198 | Gemma 2B Instruct V1.1 | 50.0 | 12 / 12 |
| 198 | Granite3 Moe 3B Instruct | 50.0 | 12 / 12 |
| 198 | Granite3.1 Dense 8B Instruct | 50.0 | 12 / 12 |
| 198 | Lfm2.5 Thinking 1.2B | 50.0 | 12 / 12 |
| 198 | Llama3.2 1B Instruct | 50.0 | 12 / 12 |
| 198 | Nemotron Mini 4B Instruct | 50.0 | 12 / 12 |
| 198 | Qwen 4B Chat V1.5 | 50.0 | 12 / 12 |
| 198 | Qwen2.5 Coder 0.5B Instructcode model | 50.0 | 12 / 12 |
| 198 | Qwen3 VL 2B Thinking | 50.0 | 12 / 12 |
| 198 | Smollm2 135M Instruct | 50.0 | 12 / 12 |
| 198 | Stablelm Zephyr 3Bfine-tune of StableLM 3B | 50.0 | 12 / 12 |
| 198 | Stablelm2 12B Chat | 50.0 | 12 / 12 |
| 213 | Codellama 34B Instructcode model | 41.7 | 12 / 12 |
| 213 | Granite4 350M H | 41.7 | 12 / 12 |
| 213 | Internlm2 1.8B Chat V2.5 | 41.7 | 12 / 12 |
| 213 | Qwen2 0.5B Instruct | 41.7 | 12 / 12 |
| 213 | Qwen3 0.6B | 41.7 | 12 / 12 |
| 213 | Smollm2 360M Instruct | 41.7 | 12 / 12 |
| 213 | Stable Code 3B Instructcode model | 41.7 | 12 / 12 |
| 213 | Stablelm2 1.6B Chat | 41.7 | 12 / 12 |
| 213 | YI Coder 1.5B Chatcode model | 41.7 | 12 / 12 |
| 222 | Codegemma 2B Code V1.1code model | 33.3 | 12 / 12 |
| 222 | Deepseek Coder 1.3B Instructcode model | 33.3 | 12 / 12 |
| 222 | Falcon2 11Bnot in the Overall ranking | 33.3 | 12 / 12 |
| 222 | Magistral Small (2506) | 33.3 | 12 / 12 |
| 222 | Phi 2.7B Chat V2 | 33.3 | 12 / 12 |
| 222 | Qwen 0.5B Chat V1.5 | 33.3 | 12 / 12 |
| 228 | Deepseek Llm 7B Chat | 25.0 | 12 / 12 |
A dash means not scored, which is different from 0. Agreement compares the two judges: high means a mean gap under 1.5 points, medium under 3, low otherwise; n/a means fewer than 3 items were scored by both.
Not ranked in this category
- Apertus-8B-Instruct-2509: withheld from the ranking (see its page)
- Falcon3 10B Instruct: withheld from the ranking (see its page)
- Falcon3 1B Instruct: withheld from the ranking (see its page)
- Falcon3 3B Instruct: withheld from the ranking (see its page)
- Falcon3 7B Instruct: withheld from the ranking (see its page)
- Internlm2 1.8B Chat V2.5: withheld from the ranking (see its page)
- Phi-4-reasoning 14B: withheld from the ranking (see its page)
- Phi-4-reasoning-plus 14B: withheld from the ranking (see its page)
- Qwen2 Math 1.5B Instruct: withheld from the ranking (see its page)
- Qwen2 Math 7B Instruct: withheld from the ranking (see its page)
The questions
In run r20261102-bv1-5246ab, 1 of the 12 questions in this category was answered correctly by every ranked model that has a score for it (33 ranked models in that run). This category does not count toward Overall, so none of its questions are marked hard.
The same ranking as JSON: sycophancy.json.