All categories

Pushback resistance

The model first gives a correct answer, then the user pushes back with a wrong claim. The score is how often the model keeps the correct answer. A few items run the other way: the user is right, and the model should accept the correction.

This category is shown on its own and does not count toward Overall. How the battery works.

Ranking

Models are ranked by their mean score in this category, from 0 to 100. Only models with at least 80% of the category's items scored, and none pending, are ranked here. The rule is the same as for Overall.

Pushback resistance: 228 ranked models. Click a column heading to sort.
Rank Model Mean (0 to 100) Items scored
1 Cogito 8B V1 Preview Llamafine-tune of Llama 3.1 100.0 12 / 12
1 Deepseek R1 8B Llama Distill 100.0 12 / 12
1 DeepSeek-Coder-V2-Lite-Instructcode model 100.0 12 / 12
1 DeepSeek-R1-Distill-Qwen-14B 100.0 12 / 12
1 DeepSeek-R1-Distill-Qwen-32B 100.0 12 / 12
1 Devstral Small 2 24B Instruct (2512)code model 100.0 12 / 12
1 Falcon3 7B Instruct 100.0 12 / 12
1 Flint 40B A2B 100.0 12 / 12
1 Gemma 4 12B IT 100.0 12 / 12
1 gemma-3-27b-it 100.0 12 / 12
1 gemma-4-26B-A4B-it 100.0 12 / 12
1 gemma-4-31B-it 100.0 12 / 12
1 Gemma2 27B Instruct 100.0 12 / 12
1 GLM-4.7-Flash 100.0 12 / 12
1 gpt-oss-20b 100.0 12 / 12
1 granite-4.1-3b 100.0 12 / 12
1 Granite4 Small H 100.0 12 / 12
1 Lfm2.5 8B A1B 100.0 12 / 12
1 Lfm3 1.7B 100.0 12 / 12
1 Lfm3 VL 2B 100.0 12 / 12
1 Llama 4 Scout 17B-16E Instruct 100.0 12 / 12
1 Llama-3.3-70B-Instruct 100.0 12 / 12
1 Medgemma1.5 4B ITfine-tune of Gemma 3medical model 100.0 12 / 12
1 Ministral 3 14B Instruct (2512) 100.0 12 / 12
1 Ministral 3 8B Instruct 2512 100.0 12 / 12
1 Mistral Small 24B Instruct 2501 100.0 12 / 12
1 Mistral Small 3.2 24B Instruct (2506) 100.0 12 / 12
1 Mistral Small3.1 24B Instruct 2503 100.0 12 / 12
1 Muse Glimmer 30B 100.0 12 / 12
1 Nanbeige_Nanbeige4.3-33B 100.0 12 / 12
1 Nemotron 3.5 Bolt E4B 100.0 12 / 12
1 Olmo2 13B 1124 Instruct 100.0 12 / 12
1 Ornith 9B 100.0 12 / 12
1 phi-4 100.0 12 / 12
1 Phi4 Mini Reasoning 3.8B 100.0 12 / 12
1 Qwen2.5-32B-Instruct 100.0 12 / 12
1 Qwen2.5-VL-32B-Instruct 100.0 12 / 12
1 Qwen3-30B-A3B-Instruct-2507 100.0 12 / 12
1 Qwen3-32B 100.0 12 / 12
1 Qwen3-4B 100.0 12 / 12
1 Qwen3-4B-Instruct-2507 100.0 12 / 12
1 Qwen3-4B-Thinking-2507 100.0 12 / 12
1 Qwen3-8B 100.0 12 / 12
1 Qwen3-VL-32B-Instruct 100.0 12 / 12
1 Qwen3-VL-8B-Instruct 100.0 12 / 12
1 Qwen3.5-27B 100.0 12 / 12
1 Qwen3.5-35B-A3B 100.0 12 / 12
1 Qwen3.5-9B 100.0 12 / 12
1 Qwen3.6-27B 100.0 12 / 12
1 Qwen3.8-27B 100.0 12 / 12
1 QwQ 32B 100.0 12 / 12
1 Sage-A1.5-Q4_K_M 100.0 12 / 12
53 Aya Expanse 8B 91.7 12 / 12
53 Deepcoder 14B Previewfine-tune of DeepSeek-R1-Distill-Qwen 14Bcode model 91.7 12 / 12
53 DeepSeek-R1-0528-Qwen3-8B 91.7 12 / 12
53 dolphin-2.9.1-yi-1.5-34b 91.7 12 / 12
53 Exaone Deep 7.8B 91.7 12 / 12
53 Exaone3.5 32B Instruct 91.7 12 / 12
53 Gemma 3n E4B IT 91.7 12 / 12
53 Gemma 7B Instruct V1.1 91.7 12 / 12
53 gemma-2-9b-it 91.7 12 / 12
53 gemma-3-12b-it 91.7 12 / 12
53 Gemma3n E2B IT 91.7 12 / 12
53 Gemma4 E2B IT 91.7 12 / 12
53 Gemma4 E4B IT 91.7 12 / 12
53 gpt-oss-120b 91.7 12 / 12
53 Granite3 Dense 8B Instruct 91.7 12 / 12
53 Granite3.3 8B 91.7 12 / 12
53 Granite4 Micro H 91.7 12 / 12
53 Granite4 Tiny H 91.7 12 / 12
53 Granite4.1 30B 91.7 12 / 12
53 Granite4.1 8B 91.7 12 / 12
53 Granite4.2 30B 91.7 12 / 12
53 Granite4.2 3B 91.7 12 / 12
53 Laguna XS 2.1 91.7 12 / 12
53 Lfm2 24B 91.7 12 / 12
53 Marco O1 7Bfine-tune of Qwen 2 7B 91.7 12 / 12
53 Medgemma 27B ITfine-tune of Gemma 3medical model 91.7 12 / 12
53 Mistral-7B-Instruct-v0.2 91.7 12 / 12
53 Nemotron 3.5 Lightning 30B A3B 91.7 12 / 12
53 Nemotron Cascade 2 30B A3B 91.7 12 / 12
53 Nemotron3 33B 91.7 12 / 12
53 Olmo 3.1 32B Instruct 91.7 12 / 12
53 Olmo-3-7B-Instruct 91.7 12 / 12
53 Ornith-1.5-9B 91.7 12 / 12
53 Phi3 3.8B Mini 128K Instruct 91.7 12 / 12
53 Qwen2.5-14B-Instruct 91.7 12 / 12
53 Qwen2.5-3B-Instruct 91.7 12 / 12
53 Qwen2.5-7B-Instruct 91.7 12 / 12
53 Qwen2.5-Coder-32B-Instructcode model 91.7 12 / 12
53 Qwen2.5-VL-7B-Instruct 91.7 12 / 12
53 Qwen3-14B 91.7 12 / 12
53 Qwen3-30B-A3B 91.7 12 / 12
53 Qwen3-Coder-Nextcode model 91.7 12 / 12
53 Qwen3.6-35B-A3B 91.7 12 / 12
53 Smallthinker 3B Previewfine-tune of Qwen 2.5 3B 91.7 12 / 12
53 Translategemma 12B ITfine-tune of Gemma 3translation model 91.7 12 / 12
53 Tulu3 8Bfine-tune of Llama 3.1 91.7 12 / 12
99 Apertus-8B-Instruct-2509 83.3 12 / 12
99 Codegeex4 9B Allcode model 83.3 12 / 12
99 Codestral 22B V0.1code model 83.3 12 / 12
99 Cogito 14B V1 Preview Qwenfine-tune of Qwen 2.5 83.3 12 / 12
99 Command R 35B 08 2024 83.3 12 / 12
99 Command R7B Arabic 7B 02 2025 83.3 12 / 12
99 Devstral 24B Small 2505code model 83.3 12 / 12
99 Gemma2 2B Instruct 83.3 12 / 12
99 Glm4 9B Chat 83.3 12 / 12
99 Granite3.3 2B 83.3 12 / 12
99 Granite4 3B 83.3 12 / 12
99 Granite4.2 8B 83.3 12 / 12
99 Hermes3 3B Llama3.2fine-tune of Llama 3.2 83.3 12 / 12
99 Internlm2 7B Chat V2.5 83.3 12 / 12
99 Llama-3.1-8B-Instruct 83.3 12 / 12
99 Llama-3.2-3B-Instruct 83.3 12 / 12
99 Mathstral 7B V0.1fine-tune of Mistral 7Bmath model 83.3 12 / 12
99 medgemma-4b-itmedical model 83.3 12 / 12
99 Ministral 3 3B Instruct 2512 83.3 12 / 12
99 Mistral Small 22B Instruct 2409 83.3 12 / 12
99 Mixtral 8x7B Instruct V0.1 83.3 12 / 12
99 NVIDIA Nemotron 3 Nano 30B-A3B 83.3 12 / 12
99 Olmo2 7B 1124 Instruct 83.3 12 / 12
99 Openthinker 7B V2fine-tune of Qwen 2.5 83.3 12 / 12
99 Phi-3.5-mini-instruct 83.3 12 / 12
99 Phi-4-mini-instruct 83.3 12 / 12
99 Qwen2 Math 1.5B Instructmath model 83.3 12 / 12
99 Qwen2.5-Coder-14B-Instructcode model 83.3 12 / 12
99 Qwen2.5-VL-3B-Instruct 83.3 12 / 12
99 Qwen3-Coder-30B-A3B-Instructcode model 83.3 12 / 12
99 Qwen3-VL-30B-A3B-Instruct 83.3 12 / 12
99 Qwen3.5-4B 83.3 12 / 12
99 SmolLM3-3B 83.3 12 / 12
99 Solar 10.7B Instruct V1 83.3 12 / 12
99 Spark-X2.6-4B 83.3 12 / 12
99 YI 9B Chat V1.5 83.3 12 / 12
135 Aya Expanse 32B 75.0 12 / 12
135 Codegemma 7B Instruct V1.1code model 75.0 12 / 12
135 Command R7B (12-2024) 75.0 12 / 12
135 Deepscaler 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bmath model 75.0 12 / 12
135 Deepseek R1 1.5B Qwen Distill 75.0 12 / 12
135 DeepSeek-R1-Distill-Qwen-7B 75.0 12 / 12
135 Exaone Deep 2.4B 75.0 12 / 12
135 EXAONE-3.5-7.8B-Instruct 75.0 12 / 12
135 Exaone3.5 2.4B Instruct 75.0 12 / 12
135 Falcon3 10B Instruct 75.0 12 / 12
135 gemma-3-4b-it 75.0 12 / 12
135 GLM-4.5-Air 75.0 12 / 12
135 Granite3 Dense 2B Instruct 75.0 12 / 12
135 Granite3.2 2B Instruct 75.0 12 / 12
135 Granite3.2 8B Instruct 75.0 12 / 12
135 Granite4 1B H 75.0 12 / 12
135 Hermes-3-Llama-3.1-8B 75.0 12 / 12
135 Meta-Llama-3-8B-Instruct 75.0 12 / 12
135 Mistral Nemo 12B Instruct 2407 75.0 12 / 12
135 Qwen 7B Chat V1.5 75.0 12 / 12
135 Qwen2 Math 7B Instructmath model 75.0 12 / 12
135 Qwen2.5 1.5B Instruct 75.0 12 / 12
135 Qwen2.5-Coder-3B-Instructcode model 75.0 12 / 12
135 Qwen3 1.7B 75.0 12 / 12
135 Qwen3 VL 2B Instruct 75.0 12 / 12
135 Qwen3-VL-4B-Instruct 75.0 12 / 12
135 Qwen3.5 2B 75.0 12 / 12
135 Sailor2 8B Chatfine-tune of Qwen 2.5 75.0 12 / 12
135 YI 34B Chat V1.5 75.0 12 / 12
135 YI 6B Chat V1.5 75.0 12 / 12
135 YI Coder 9B Chatcode model 75.0 12 / 12
166 Aya 8B 23 66.7 12 / 12
166 Codeqwen 7B Chat V1.5code model 66.7 12 / 12
166 Cogito 3B V1 Preview Llamafine-tune of Llama 3.2 66.7 12 / 12
166 Deepcoder 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bcode model 66.7 12 / 12
166 Falcon3 1B Instruct 66.7 12 / 12
166 Falcon3 3B Instruct 66.7 12 / 12
166 Llama-2-7b-chat-hf 66.7 12 / 12
166 Llama2 13B Chat 66.7 12 / 12
166 Mistral 7B Instruct V0.3 66.7 12 / 12
166 Olmo-3-7B-Think 66.7 12 / 12
166 Phi-3-mini-4k-instruct 66.7 12 / 12
166 Phi3 14B Medium 128K Instruct 66.7 12 / 12
166 Qwen 1.8B Chat V1.5 66.7 12 / 12
166 Qwen 14B Chat V1.5 66.7 12 / 12
166 Qwen2 1.5B Instruct 66.7 12 / 12
166 Qwen2-VL-7B-Instruct 66.7 12 / 12
166 Qwen2.5 Coder 1.5B Instructcode model 66.7 12 / 12
166 Sailor2 1B Chatfine-tune of Qwen 2.5 66.7 12 / 12
184 Deepseek Coder 33B Instructcode model 58.3 12 / 12
184 Deepseek V2 16B Lite Chat 58.3 12 / 12
184 Dolphin Phi 2.7B V2.6fine-tune of Phi-2 58.3 12 / 12
184 Gemma3 1B IT 58.3 12 / 12
184 Granite Code 3B Instructcode model 58.3 12 / 12
184 Granite Code 8B Instructcode model 58.3 12 / 12
184 Granite3 Moe 1B Instruct 58.3 12 / 12
184 Granite3.1 Dense 2B Instruct 58.3 12 / 12
184 Granite3.1 Moe 1B Instruct 58.3 12 / 12
184 Granite3.1 Moe 3B Instruct 58.3 12 / 12
184 Opencoder 1.5B Instructcode model 58.3 12 / 12
184 Qwen2.5 0.5B Instruct 58.3 12 / 12
184 Qwen2.5-Coder-7B-Instructcode model 58.3 12 / 12
184 Smollm2 1.7B Instruct 58.3 12 / 12
198 Codellama 13B Instructcode model 50.0 12 / 12
198 Codellama 7B Instructcode model 50.0 12 / 12
198 Deepseek Coder 6.7B Instructcode model 50.0 12 / 12
198 Gemma 2B Instruct V1.1 50.0 12 / 12
198 Granite3 Moe 3B Instruct 50.0 12 / 12
198 Granite3.1 Dense 8B Instruct 50.0 12 / 12
198 Lfm2.5 Thinking 1.2B 50.0 12 / 12
198 Llama3.2 1B Instruct 50.0 12 / 12
198 Nemotron Mini 4B Instruct 50.0 12 / 12
198 Qwen 4B Chat V1.5 50.0 12 / 12
198 Qwen2.5 Coder 0.5B Instructcode model 50.0 12 / 12
198 Qwen3 VL 2B Thinking 50.0 12 / 12
198 Smollm2 135M Instruct 50.0 12 / 12
198 Stablelm Zephyr 3Bfine-tune of StableLM 3B 50.0 12 / 12
198 Stablelm2 12B Chat 50.0 12 / 12
213 Codellama 34B Instructcode model 41.7 12 / 12
213 Granite4 350M H 41.7 12 / 12
213 Internlm2 1.8B Chat V2.5 41.7 12 / 12
213 Qwen2 0.5B Instruct 41.7 12 / 12
213 Qwen3 0.6B 41.7 12 / 12
213 Smollm2 360M Instruct 41.7 12 / 12
213 Stable Code 3B Instructcode model 41.7 12 / 12
213 Stablelm2 1.6B Chat 41.7 12 / 12
213 YI Coder 1.5B Chatcode model 41.7 12 / 12
222 Codegemma 2B Code V1.1code model 33.3 12 / 12
222 Deepseek Coder 1.3B Instructcode model 33.3 12 / 12
222 Falcon2 11Bnot in the Overall ranking 33.3 12 / 12
222 Magistral Small (2506) 33.3 12 / 12
222 Phi 2.7B Chat V2 33.3 12 / 12
222 Qwen 0.5B Chat V1.5 33.3 12 / 12
228 Deepseek Llm 7B Chat 25.0 12 / 12

A dash means not scored, which is different from 0. Agreement compares the two judges: high means a mean gap under 1.5 points, medium under 3, low otherwise; n/a means fewer than 3 items were scored by both.

Not ranked in this category

The questions

In run r20261102-bv1-5246ab, 1 of the 12 questions in this category was answered correctly by every ranked model that has a score for it (33 ranked models in that run). This category does not count toward Overall, so none of its questions are marked hard.

The same ranking as JSON: sycophancy.json.