All categories

Famous gotchas

Well-known trick questions. They are widely published, so a correct answer may be memorised rather than reasoned.

This category is shown on its own and does not count toward Overall. How the battery works.

Ranking

Models are ranked by their mean score in this category, from 0 to 100. Only models with at least 80% of the category's items scored, and none pending, are ranked here. The rule is the same as for Overall.

Famous gotchas: 228 ranked models. Click a column heading to sort.
Rank Model Mean (0 to 100) Items scored
1 gemma-4-26B-A4B-it 100.0 15 / 15
1 gpt-oss-120b 100.0 15 / 15
1 gpt-oss-20b 100.0 15 / 15
1 Muse Glimmer 30B 100.0 15 / 15
1 Nemotron 3.5 Lightning 30B A3B 100.0 15 / 15
1 Olmo 3.1 32B Instruct 100.0 15 / 15
1 Ornith-1.5-9B 100.0 15 / 15
1 Qwen3-4B 100.0 15 / 15
1 Qwen3-Coder-Nextcode model 100.0 15 / 15
1 QwQ 32B 100.0 15 / 15
11 DeepSeek-R1-Distill-Qwen-14B 93.3 15 / 15
11 DeepSeek-R1-Distill-Qwen-32B 93.3 15 / 15
11 Exaone Deep 2.4B 93.3 15 / 15
11 Exaone Deep 7.8B 93.3 15 / 15
11 Gemma 4 12B IT 93.3 15 / 15
11 gemma-3-27b-it 93.3 15 / 15
11 gemma-4-31B-it 93.3 15 / 15
11 Gemma4 E2B IT 93.3 15 / 15
11 Gemma4 E4B IT 93.3 15 / 15
11 GLM-4.5-Air 93.3 15 / 15
11 Granite4.2 30B 93.3 15 / 15
11 Laguna XS 2.1 93.3 15 / 15
11 Lfm3 1.7B 93.3 15 / 15
11 Llama 4 Scout 17B-16E Instruct 93.3 15 / 15
11 Llama-3.3-70B-Instruct 93.3 15 / 15
11 Mistral Small 3.2 24B Instruct (2506) 93.3 15 / 15
11 Nemotron 3.5 Bolt E4B 93.3 15 / 15
11 NVIDIA Nemotron 3 Nano 30B-A3B 93.3 15 / 15
11 Olmo-3-7B-Instruct 93.3 15 / 15
11 Ornith 9B 93.3 15 / 15
11 Phi4 Mini Reasoning 3.8B 93.3 15 / 15
11 Qwen3-30B-A3B 93.3 15 / 15
11 Qwen3-4B-Instruct-2507 93.3 15 / 15
11 Qwen3-4B-Thinking-2507 93.3 15 / 15
11 Qwen3-VL-32B-Instruct 93.3 15 / 15
11 Qwen3-VL-8B-Instruct 93.3 15 / 15
11 Qwen3.5-27B 93.3 15 / 15
11 Qwen3.5-35B-A3B 93.3 15 / 15
11 Qwen3.5-9B 93.3 15 / 15
11 Qwen3.6-27B 93.3 15 / 15
11 Qwen3.6-35B-A3B 93.3 15 / 15
11 Qwen3.8-27B 93.3 15 / 15
11 Sage-A1.5-Q4_K_M 93.3 15 / 15
44 Deepseek R1 8B Llama Distill 86.7 15 / 15
44 DeepSeek-R1-0528-Qwen3-8B 86.7 15 / 15
44 Falcon3 10B Instruct 86.7 15 / 15
44 Flint 40B A2B 86.7 15 / 15
44 Granite4.1 30B 86.7 15 / 15
44 Granite4.2 3B 86.7 15 / 15
44 Medgemma 27B ITfine-tune of Gemma 3medical model 86.7 15 / 15
44 Ministral 3 14B Instruct (2512) 86.7 15 / 15
44 Mistral Nemo 12B Instruct 2407 86.7 15 / 15
44 Mistral Small 24B Instruct 2501 86.7 15 / 15
44 Nemotron3 33B 86.7 15 / 15
44 Openthinker 7B V2fine-tune of Qwen 2.5 86.7 15 / 15
44 phi-4 86.7 15 / 15
44 Qwen2.5-14B-Instruct 86.7 15 / 15
44 Qwen2.5-VL-32B-Instruct 86.7 15 / 15
44 Qwen3-30B-A3B-Instruct-2507 86.7 15 / 15
44 Qwen3-32B 86.7 15 / 15
44 Qwen3-8B 86.7 15 / 15
44 Qwen3-VL-4B-Instruct 86.7 15 / 15
44 Qwen3.5-4B 86.7 15 / 15
44 Translategemma 12B ITfine-tune of Gemma 3translation model 86.7 15 / 15
65 Aya Expanse 8B 80.0 15 / 15
65 Cogito 14B V1 Preview Qwenfine-tune of Qwen 2.5 80.0 15 / 15
65 Command R7B (12-2024) 80.0 15 / 15
65 Deepcoder 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bcode model 80.0 15 / 15
65 DeepSeek-R1-Distill-Qwen-7B 80.0 15 / 15
65 Devstral 24B Small 2505code model 80.0 15 / 15
65 Devstral Small 2 24B Instruct (2512)code model 80.0 15 / 15
65 EXAONE-3.5-7.8B-Instruct 80.0 15 / 15
65 Exaone3.5 32B Instruct 80.0 15 / 15
65 Falcon3 7B Instruct 80.0 15 / 15
65 Gemma2 27B Instruct 80.0 15 / 15
65 Granite4 3B 80.0 15 / 15
65 Granite4 Small H 80.0 15 / 15
65 Granite4.1 8B 80.0 15 / 15
65 Lfm2 24B 80.0 15 / 15
65 Lfm2.5 8B A1B 80.0 15 / 15
65 Lfm2.5 Thinking 1.2B 80.0 15 / 15
65 Lfm3 VL 2B 80.0 15 / 15
65 Ministral 3 8B Instruct 2512 80.0 15 / 15
65 Nanbeige_Nanbeige4.3-33B 80.0 15 / 15
65 Nemotron Cascade 2 30B A3B 80.0 15 / 15
65 Olmo-3-7B-Think 80.0 15 / 15
65 Phi-3.5-mini-instruct 80.0 15 / 15
65 Phi-4-mini-instruct 80.0 15 / 15
65 Qwen2.5-32B-Instruct 80.0 15 / 15
65 Qwen2.5-Coder-32B-Instructcode model 80.0 15 / 15
65 Qwen2.5-Coder-7B-Instructcode model 80.0 15 / 15
65 Qwen3-14B 80.0 15 / 15
65 Qwen3-Coder-30B-A3B-Instructcode model 80.0 15 / 15
65 Qwen3-VL-30B-A3B-Instruct 80.0 15 / 15
65 YI 34B Chat V1.5 80.0 15 / 15
96 Cogito 8B V1 Preview Llamafine-tune of Llama 3.1 73.3 15 / 15
96 Deepseek R1 1.5B Qwen Distill 73.3 15 / 15
96 Gemma 3n E4B IT 73.3 15 / 15
96 gemma-2-9b-it 73.3 15 / 15
96 gemma-3-4b-it 73.3 15 / 15
96 Glm4 9B Chat 73.3 15 / 15
96 granite-4.1-3b 73.3 15 / 15
96 Granite3 Dense 8B Instruct 73.3 15 / 15
96 Granite3.2 8B Instruct 73.3 15 / 15
96 Granite3.3 8B 73.3 15 / 15
96 Granite4 Micro H 73.3 15 / 15
96 Medgemma1.5 4B ITfine-tune of Gemma 3medical model 73.3 15 / 15
96 Mixtral 8x7B Instruct V0.1 73.3 15 / 15
96 Phi3 14B Medium 128K Instruct 73.3 15 / 15
96 Qwen2.5-Coder-14B-Instructcode model 73.3 15 / 15
96 Qwen3 1.7B 73.3 15 / 15
96 Spark-X2.6-4B 73.3 15 / 15
113 Aya Expanse 32B 66.7 15 / 15
113 Codellama 13B Instructcode model 66.7 15 / 15
113 Command R7B Arabic 7B 02 2025 66.7 15 / 15
113 Deepscaler 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bmath model 66.7 15 / 15
113 dolphin-2.9.1-yi-1.5-34b 66.7 15 / 15
113 Exaone3.5 2.4B Instruct 66.7 15 / 15
113 Falcon3 3B Instruct 66.7 15 / 15
113 gemma-3-12b-it 66.7 15 / 15
113 Gemma2 2B Instruct 66.7 15 / 15
113 GLM-4.7-Flash 66.7 15 / 15
113 Granite Code 8B Instructcode model 66.7 15 / 15
113 Granite3.1 Dense 8B Instruct 66.7 15 / 15
113 Granite3.1 Moe 3B Instruct 66.7 15 / 15
113 Granite4.2 8B 66.7 15 / 15
113 Hermes-3-Llama-3.1-8B 66.7 15 / 15
113 Llama-3.1-8B-Instruct 66.7 15 / 15
113 Marco O1 7Bfine-tune of Qwen 2 7B 66.7 15 / 15
113 medgemma-4b-itmedical model 66.7 15 / 15
113 Mistral Small 22B Instruct 2409 66.7 15 / 15
113 Mistral Small3.1 24B Instruct 2503 66.7 15 / 15
113 Qwen 7B Chat V1.5 66.7 15 / 15
113 Qwen2.5-7B-Instruct 66.7 15 / 15
113 Qwen3.5 2B 66.7 15 / 15
113 Sailor2 8B Chatfine-tune of Qwen 2.5 66.7 15 / 15
137 Codestral 22B V0.1code model 60.0 15 / 15
137 Deepcoder 14B Previewfine-tune of DeepSeek-R1-Distill-Qwen 14Bcode model 60.0 15 / 15
137 Gemma 7B Instruct V1.1 60.0 15 / 15
137 Granite4 Tiny H 60.0 15 / 15
137 Llama-3.2-3B-Instruct 60.0 15 / 15
137 Mathstral 7B V0.1fine-tune of Mistral 7Bmath model 60.0 15 / 15
137 Meta-Llama-3-8B-Instruct 60.0 15 / 15
137 Mistral-7B-Instruct-v0.2 60.0 15 / 15
137 Phi-3-mini-4k-instruct 60.0 15 / 15
137 Qwen 14B Chat V1.5 60.0 15 / 15
137 Qwen2 1.5B Instruct 60.0 15 / 15
137 Qwen2 Math 7B Instructmath model 60.0 15 / 15
137 Qwen2-VL-7B-Instruct 60.0 15 / 15
137 Qwen2.5 1.5B Instruct 60.0 15 / 15
137 Qwen2.5-Coder-3B-Instructcode model 60.0 15 / 15
137 Qwen2.5-VL-3B-Instruct 60.0 15 / 15
137 Qwen2.5-VL-7B-Instruct 60.0 15 / 15
137 Qwen3 VL 2B Instruct 60.0 15 / 15
137 Smallthinker 3B Previewfine-tune of Qwen 2.5 3B 60.0 15 / 15
137 SmolLM3-3B 60.0 15 / 15
137 YI 6B Chat V1.5 60.0 15 / 15
137 YI 9B Chat V1.5 60.0 15 / 15
159 Cogito 3B V1 Preview Llamafine-tune of Llama 3.2 53.3 15 / 15
159 Deepseek Coder 33B Instructcode model 53.3 15 / 15
159 Dolphin Phi 2.7B V2.6fine-tune of Phi-2 53.3 15 / 15
159 Gemma3 1B IT 53.3 15 / 15
159 Gemma3n E2B IT 53.3 15 / 15
159 Granite3 Moe 3B Instruct 53.3 15 / 15
159 Granite4 1B H 53.3 15 / 15
159 Llama2 13B Chat 53.3 15 / 15
159 Ministral 3 3B Instruct 2512 53.3 15 / 15
159 Mistral 7B Instruct V0.3 53.3 15 / 15
159 Olmo2 13B 1124 Instruct 53.3 15 / 15
159 Phi3 3.8B Mini 128K Instruct 53.3 15 / 15
159 Qwen2.5 Coder 1.5B Instructcode model 53.3 15 / 15
159 Qwen2.5-3B-Instruct 53.3 15 / 15
159 Qwen3 VL 2B Thinking 53.3 15 / 15
159 YI Coder 9B Chatcode model 53.3 15 / 15
175 Apertus-8B-Instruct-2509 46.7 15 / 15
175 Codegemma 7B Instruct V1.1code model 46.7 15 / 15
175 Command R 35B 08 2024 46.7 15 / 15
175 DeepSeek-Coder-V2-Lite-Instructcode model 46.7 15 / 15
175 Granite3.2 2B Instruct 46.7 15 / 15
175 Granite3.3 2B 46.7 15 / 15
175 Internlm2 7B Chat V2.5 46.7 15 / 15
175 Nemotron Mini 4B Instruct 46.7 15 / 15
175 Olmo2 7B 1124 Instruct 46.7 15 / 15
175 Smollm2 360M Instruct 46.7 15 / 15
175 Tulu3 8Bfine-tune of Llama 3.1 46.7 15 / 15
186 Aya 8B 23 40.0 15 / 15
186 Codellama 7B Instructcode model 40.0 15 / 15
186 Codeqwen 7B Chat V1.5code model 40.0 15 / 15
186 Deepseek Llm 7B Chat 40.0 15 / 15
186 Deepseek V2 16B Lite Chat 40.0 15 / 15
186 Granite Code 3B Instructcode model 40.0 15 / 15
186 Granite3 Dense 2B Instruct 40.0 15 / 15
186 Granite3.1 Dense 2B Instruct 40.0 15 / 15
186 Granite3.1 Moe 1B Instruct 40.0 15 / 15
186 Phi 2.7B Chat V2 40.0 15 / 15
186 Qwen 1.8B Chat V1.5 40.0 15 / 15
186 Qwen2 Math 1.5B Instructmath model 40.0 15 / 15
186 Qwen2.5 0.5B Instruct 40.0 15 / 15
186 Smollm2 1.7B Instruct 40.0 15 / 15
186 Stable Code 3B Instructcode model 40.0 15 / 15
201 Codegeex4 9B Allcode model 33.3 15 / 15
201 Codellama 34B Instructcode model 33.3 15 / 15
201 Deepseek Coder 6.7B Instructcode model 33.3 15 / 15
201 Falcon2 11Bnot in the Overall ranking 33.3 15 / 15
201 Falcon3 1B Instruct 33.3 15 / 15
201 Granite3 Moe 1B Instruct 33.3 15 / 15
201 Hermes3 3B Llama3.2fine-tune of Llama 3.2 33.3 15 / 15
201 Internlm2 1.8B Chat V2.5 33.3 15 / 15
201 Llama-2-7b-chat-hf 33.3 15 / 15
201 Qwen2.5 Coder 0.5B Instructcode model 33.3 15 / 15
201 Qwen3 0.6B 33.3 15 / 15
201 Solar 10.7B Instruct V1 33.3 15 / 15
201 Stablelm2 12B Chat 33.3 15 / 15
201 YI Coder 1.5B Chatcode model 33.3 15 / 15
215 Gemma 2B Instruct V1.1 26.7 15 / 15
215 Magistral Small (2506) 26.7 15 / 15
215 Opencoder 1.5B Instructcode model 26.7 15 / 15
215 Qwen 0.5B Chat V1.5 26.7 15 / 15
215 Qwen2 0.5B Instruct 26.7 15 / 15
220 Granite4 350M H 20.0 15 / 15
220 Qwen 4B Chat V1.5 20.0 15 / 15
220 Stablelm Zephyr 3Bfine-tune of StableLM 3B 20.0 15 / 15
223 Llama3.2 1B Instruct 13.3 15 / 15
223 Stablelm2 1.6B Chat 13.3 15 / 15
225 Codegemma 2B Code V1.1code model 6.7 15 / 15
225 Deepseek Coder 1.3B Instructcode model 6.7 15 / 15
225 Sailor2 1B Chatfine-tune of Qwen 2.5 6.7 15 / 15
225 Smollm2 135M Instruct 6.7 15 / 15

A dash means not scored, which is different from 0. Agreement compares the two judges: high means a mean gap under 1.5 points, medium under 3, low otherwise; n/a means fewer than 3 items were scored by both.

Not ranked in this category

The questions

In run r20261102-bv1-5246ab, 1 of the 15 questions in this category was answered correctly by every ranked model that has a score for it (33 ranked models in that run). This category does not count toward Overall, so none of its questions are marked hard.

The same ranking as JSON: gotchas.json.