Reasoning
Multi-step reasoning questions that each have one exact answer. The model gives its final answer on the last line, and code compares it with the reference.
This category counts toward Overall. The two judge models also score every item, and their scores are shown here separately. How the battery works.
Ranking
Models are ranked by their mean score in this category, from 0 to 100. Only models with at least 80% of the category's items scored, and none pending, are ranked here. The rule is the same as for Overall.
| Rank | Model | Mean (0 to 100) | Items scored | Judge A mean | Judge B mean | Agreement |
|---|---|---|---|---|---|---|
| 1 | DeepSeek-R1-Distill-Qwen-32B | 97.1 | 35 / 35 | 8.9openai/gpt-5.6-luna | 8.8anthropic/claude-haiku-5.5 | high |
| 1 | gemma-4-31B-it | 97.1 | 35 / 35 | 9.4openai/gpt-5.6-luna | 9.1anthropic/claude-haiku-5.5 | high |
| 1 | gpt-oss-120b | 97.1 | 35 / 35 | 8.0anthropic/claude-haiku-5.5 | 8.2deepseek/deepseek-v4.1-flash | high |
| 1 | gpt-oss-20b | 97.1 | 35 / 35 | 7.1anthropic/claude-haiku-5.5 | 7.2deepseek/deepseek-v4.1-flash | high |
| 1 | Llama-3.3-70B-Instruct | 97.1 | 35 / 35 | 9.4openai/gpt-5.6-luna | 8.8anthropic/claude-haiku-5.5 | high |
| 6 | Gemma 4 12B IT | 94.3 | 35 / 35 | 9.5openai/gpt-5.6-luna | 8.9anthropic/claude-haiku-5.5 | high |
| 6 | gemma-4-26B-A4B-it | 94.3 | 35 / 35 | 9.1openai/gpt-5.6-luna | 8.8anthropic/claude-haiku-5.5 | high |
| 6 | Muse Glimmer 30B | 94.3 | 35 / 35 | 9.2openai/gpt-5.6-luna | 8.7anthropic/claude-haiku-5.5 | high |
| 6 | Phi4 Mini Reasoning 3.8B | 94.3 | 35 / 35 | 7.9openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | high |
| 6 | Qwen3-8B | 94.3 | 35 / 35 | 8.9openai/gpt-5.6-luna | 8.4anthropic/claude-haiku-5.5 | high |
| 6 | QwQ 32B | 94.3 | 35 / 35 | 8.1openai/gpt-5.6-luna | 7.6anthropic/claude-haiku-5.5 | high |
| 12 | Deepseek R1 8B Llama Distill | 91.4 | 35 / 35 | 8.1openai/gpt-5.6-luna | 8.0anthropic/claude-haiku-5.5 | high |
| 12 | DeepSeek-R1-Distill-Qwen-14B | 91.4 | 35 / 35 | 8.3openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | high |
| 12 | gemma-3-27b-it | 91.4 | 35 / 35 | 9.0openai/gpt-5.6-luna | 8.2anthropic/claude-haiku-5.5 | high |
| 12 | Granite4.1 8B | 91.4 | 35 / 35 | 9.0openai/gpt-5.6-luna | 8.7anthropic/claude-haiku-5.5 | high |
| 12 | Laguna XS 2.1 | 91.4 | 35 / 35 | 9.1openai/gpt-5.6-luna | 8.7anthropic/claude-haiku-5.5 | high |
| 12 | Llama 4 Scout 17B-16E Instruct | 91.4 | 35 / 35 | 8.7openai/gpt-5.6-luna | 8.2anthropic/claude-haiku-5.5 | high |
| 12 | Ornith 9B | 91.4 | 35 / 35 | 9.2openai/gpt-5.6-luna | 8.7anthropic/claude-haiku-5.5 | high |
| 12 | Ornith-1.5-9B | 91.4 | 35 / 35 | 9.1openai/gpt-5.6-luna | 8.5anthropic/claude-haiku-5.5 | high |
| 12 | phi-4 | 91.4 | 35 / 35 | 8.8openai/gpt-5.6-luna | 8.3anthropic/claude-haiku-5.5 | high |
| 12 | Qwen3-14B | 91.4 | 35 / 35 | 8.9openai/gpt-5.6-luna | 8.5anthropic/claude-haiku-5.5 | high |
| 12 | Qwen3-VL-4B-Instruct | 91.4 | 35 / 35 | 9.0openai/gpt-5.6-luna | 8.4anthropic/claude-haiku-5.5 | high |
| 12 | Qwen3-VL-8B-Instruct | 91.4 | 35 / 35 | 9.1openai/gpt-5.6-luna | 8.6anthropic/claude-haiku-5.5 | high |
| 24 | gemma-3-12b-it | 88.6 | 35 / 35 | 8.8openai/gpt-5.6-luna | 8.2anthropic/claude-haiku-5.5 | high |
| 24 | Gemma4 E2B IT | 88.6 | 35 / 35 | 8.6openai/gpt-5.6-luna | 8.4anthropic/claude-haiku-5.5 | high |
| 24 | Granite4 Small H | 88.6 | 35 / 35 | 8.9openai/gpt-5.6-luna | 8.4anthropic/claude-haiku-5.5 | high |
| 24 | Mistral Small3.1 24B Instruct 2503 | 88.6 | 35 / 35 | 8.5openai/gpt-5.6-luna | 8.0anthropic/claude-haiku-5.5 | high |
| 24 | Nanbeige_Nanbeige4.3-33B | 88.6 | 35 / 35 | 8.9openai/gpt-5.6-luna | 8.5anthropic/claude-haiku-5.5 | high |
| 24 | Openthinker 7B V2fine-tune of Qwen 2.5 | 88.6 | 35 / 35 | 7.6openai/gpt-5.6-luna | 7.5anthropic/claude-haiku-5.5 | high |
| 24 | Qwen2.5-14B-Instruct | 88.6 | 35 / 35 | 8.7openai/gpt-5.6-luna | 8.1anthropic/claude-haiku-5.5 | high |
| 24 | Qwen2.5-VL-32B-Instruct | 88.6 | 35 / 35 | 8.1openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | high |
| 24 | Qwen3-30B-A3B | 88.6 | 35 / 35 | 8.8openai/gpt-5.6-luna | 8.2anthropic/claude-haiku-5.5 | high |
| 24 | Qwen3-30B-A3B-Instruct-2507 | 88.6 | 35 / 35 | 8.8openai/gpt-5.6-luna | 8.5anthropic/claude-haiku-5.5 | high |
| 24 | Qwen3-4B | 88.6 | 35 / 35 | 8.8openai/gpt-5.6-luna | 8.4anthropic/claude-haiku-5.5 | high |
| 24 | Qwen3-Coder-30B-A3B-Instructcode model | 88.6 | 35 / 35 | 8.7openai/gpt-5.6-luna | 8.3anthropic/claude-haiku-5.5 | high |
| 24 | Qwen3-Coder-Nextcode model | 88.6 | 35 / 35 | 8.8openai/gpt-5.6-luna | 8.7anthropic/claude-haiku-5.5 | high |
| 24 | Qwen3-VL-32B-Instruct | 88.6 | 35 / 35 | 9.0openai/gpt-5.6-luna | 8.7anthropic/claude-haiku-5.5 | high |
| 24 | Qwen3.5-27B | 88.6 | 35 / 35 | 8.9openai/gpt-5.6-luna | 8.5anthropic/claude-haiku-5.5 | high |
| 24 | Qwen3.6-27B | 88.6 | 35 / 35 | 8.8openai/gpt-5.6-luna | 8.3anthropic/claude-haiku-5.5 | high |
| 24 | Qwen3.8-27B | 88.6 | 35 / 35 | 8.7openai/gpt-5.6-luna | 8.4anthropic/claude-haiku-5.5 | high |
| 24 | Sage-A1.5-Q4_K_M | 88.6 | 35 / 35 | 8.9openai/gpt-5.6-luna | 8.7anthropic/claude-haiku-5.5 | high |
| 42 | Devstral Small 2 24B Instruct (2512)code model | 85.7 | 35 / 35 | 8.5openai/gpt-5.6-luna | 8.0anthropic/claude-haiku-5.5 | high |
| 42 | Exaone Deep 7.8B | 85.7 | 35 / 35 | 7.0openai/gpt-5.6-luna | 6.5anthropic/claude-haiku-5.5 | high |
| 42 | Gemma4 E4B IT | 85.7 | 35 / 35 | 8.6openai/gpt-5.6-luna | 8.2anthropic/claude-haiku-5.5 | high |
| 42 | Lfm2.5 8B A1B | 85.7 | 35 / 35 | 7.8openai/gpt-5.6-luna | 6.9anthropic/claude-haiku-5.5 | high |
| 42 | Mistral Small 24B Instruct 2501 | 85.7 | 35 / 35 | 8.5openai/gpt-5.6-luna | 8.0anthropic/claude-haiku-5.5 | high |
| 42 | Nemotron 3.5 Lightning 30B A3B | 85.7 | 35 / 35 | 8.7openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | high |
| 42 | Qwen2.5-32B-Instruct | 85.7 | 35 / 35 | 8.6openai/gpt-5.6-luna | 8.0anthropic/claude-haiku-5.5 | high |
| 42 | Qwen3-32B | 85.7 | 35 / 35 | 8.7openai/gpt-5.6-luna | 8.1anthropic/claude-haiku-5.5 | high |
| 42 | Qwen3.5-35B-A3B | 85.7 | 35 / 35 | 8.7openai/gpt-5.6-luna | 8.2anthropic/claude-haiku-5.5 | high |
| 42 | Qwen3.5-4B | 85.7 | 35 / 35 | 8.6openai/gpt-5.6-luna | 8.2anthropic/claude-haiku-5.5 | high |
| 42 | Qwen3.5-9B | 85.7 | 35 / 35 | 8.7openai/gpt-5.6-luna | 8.2anthropic/claude-haiku-5.5 | high |
| 42 | Qwen3.6-35B-A3B | 85.7 | 35 / 35 | 8.7openai/gpt-5.6-luna | 8.1anthropic/claude-haiku-5.5 | high |
| 42 | Spark-X2.6-4B | 85.7 | 35 / 35 | 8.7openai/gpt-5.6-luna | 8.3anthropic/claude-haiku-5.5 | high |
| 55 | Cogito 14B V1 Preview Qwenfine-tune of Qwen 2.5 | 82.9 | 35 / 35 | 8.2openai/gpt-5.6-luna | 7.6anthropic/claude-haiku-5.5 | high |
| 55 | Exaone Deep 2.4B | 82.9 | 35 / 35 | 7.0openai/gpt-5.6-luna | 6.6anthropic/claude-haiku-5.5 | high |
| 55 | Exaone3.5 2.4B Instruct | 82.9 | 35 / 35 | 7.9openai/gpt-5.6-luna | 7.2anthropic/claude-haiku-5.5 | high |
| 55 | Gemma 3n E4B IT | 82.9 | 35 / 35 | 7.7openai/gpt-5.6-luna | 7.5anthropic/claude-haiku-5.5 | high |
| 55 | GLM-4.5-Air | 82.9 | 35 / 35 | 8.0openai/gpt-5.6-luna | 7.3anthropic/claude-haiku-5.5 | high |
| 55 | Granite4.1 30B | 82.9 | 35 / 35 | 8.6openai/gpt-5.6-luna | 8.0anthropic/claude-haiku-5.5 | high |
| 55 | Lfm2.5 Thinking 1.2B | 82.9 | 35 / 35 | 5.5openai/gpt-5.6-luna | 5.5anthropic/claude-haiku-5.5 | high |
| 55 | Medgemma 27B ITfine-tune of Gemma 3medical model | 82.9 | 35 / 35 | 7.3openai/gpt-5.6-luna | 7.6anthropic/claude-haiku-5.5 | high |
| 55 | Ministral 3 14B Instruct (2512) | 82.9 | 35 / 35 | 8.6openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | high |
| 55 | Ministral 3 8B Instruct 2512 | 82.9 | 35 / 35 | 8.2openai/gpt-5.6-luna | 7.6anthropic/claude-haiku-5.5 | high |
| 55 | Mistral Small 3.2 24B Instruct (2506) | 82.9 | 35 / 35 | 8.4openai/gpt-5.6-luna | 7.9anthropic/claude-haiku-5.5 | high |
| 55 | Nemotron 3.5 Bolt E4B | 82.9 | 35 / 35 | 8.3openai/gpt-5.6-luna | 8.0anthropic/claude-haiku-5.5 | high |
| 55 | Olmo 3.1 32B Instruct | 82.9 | 35 / 35 | 8.3openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | high |
| 55 | Qwen2.5-Coder-14B-Instructcode model | 82.9 | 35 / 35 | 8.1openai/gpt-5.6-luna | 7.6anthropic/claude-haiku-5.5 | high |
| 55 | Qwen2.5-Coder-7B-Instructcode model | 82.9 | 35 / 35 | 8.0openai/gpt-5.6-luna | 7.7anthropic/claude-haiku-5.5 | high |
| 55 | Qwen3-4B-Instruct-2507 | 82.9 | 35 / 35 | 8.4openai/gpt-5.6-luna | 7.9anthropic/claude-haiku-5.5 | high |
| 55 | Qwen3-VL-30B-A3B-Instruct | 82.9 | 35 / 35 | 8.6openai/gpt-5.6-luna | 8.1anthropic/claude-haiku-5.5 | high |
| 55 | Translategemma 12B ITfine-tune of Gemma 3translation model | 82.9 | 35 / 35 | 8.3openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | high |
| 73 | Cogito 8B V1 Preview Llamafine-tune of Llama 3.1 | 80.0 | 35 / 35 | 8.0openai/gpt-5.6-luna | 7.6anthropic/claude-haiku-5.5 | high |
| 73 | Deepcoder 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bcode model | 80.0 | 35 / 35 | 7.0openai/gpt-5.6-luna | 7.0anthropic/claude-haiku-5.5 | high |
| 73 | EXAONE-3.5-7.8B-Instruct | 80.0 | 35 / 35 | 8.3openai/gpt-5.6-luna | 7.6anthropic/claude-haiku-5.5 | high |
| 73 | Falcon3 10B Instruct | 80.0 | 35 / 35 | 8.1openai/gpt-5.6-luna | 7.4anthropic/claude-haiku-5.5 | high |
| 73 | Granite4.2 30B | 80.0 | 35 / 35 | 8.3openai/gpt-5.6-luna | 7.9anthropic/claude-haiku-5.5 | high |
| 73 | Nemotron3 33B | 80.0 | 35 / 35 | 7.7openai/gpt-5.6-luna | 7.3anthropic/claude-haiku-5.5 | high |
| 73 | Olmo-3-7B-Instruct | 80.0 | 35 / 35 | 8.2openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | high |
| 73 | Qwen2.5-Coder-32B-Instructcode model | 80.0 | 35 / 35 | 8.1openai/gpt-5.6-luna | 7.4anthropic/claude-haiku-5.5 | high |
| 81 | DeepSeek-R1-0528-Qwen3-8B | 77.1 | 35 / 35 | 7.5openai/gpt-5.6-luna | 6.9anthropic/claude-haiku-5.5 | high |
| 81 | DeepSeek-R1-Distill-Qwen-7B | 77.1 | 35 / 35 | 7.1openai/gpt-5.6-luna | 6.7anthropic/claude-haiku-5.5 | high |
| 81 | Devstral 24B Small 2505code model | 77.1 | 35 / 35 | 7.7openai/gpt-5.6-luna | 7.4anthropic/claude-haiku-5.5 | high |
| 81 | Exaone3.5 32B Instruct | 77.1 | 35 / 35 | 8.1openai/gpt-5.6-luna | 7.4anthropic/claude-haiku-5.5 | high |
| 81 | GLM-4.7-Flash | 77.1 | 35 / 35 | 7.4openai/gpt-5.6-luna | 6.6anthropic/claude-haiku-5.5 | high |
| 81 | Lfm2 24B | 77.1 | 35 / 35 | 7.5openai/gpt-5.6-luna | 7.2anthropic/claude-haiku-5.5 | high |
| 81 | Phi-3-mini-4k-instruct | 77.1 | 35 / 35 | 7.4openai/gpt-5.6-luna | 6.7anthropic/claude-haiku-5.5 | high |
| 81 | Phi-4-mini-instruct | 77.1 | 35 / 35 | 7.5openai/gpt-5.6-luna | 6.7anthropic/claude-haiku-5.5 | high |
| 81 | Phi3 14B Medium 128K Instruct | 77.1 | 35 / 35 | 7.7openai/gpt-5.6-luna | 6.9anthropic/claude-haiku-5.5 | high |
| 81 | Qwen2.5-7B-Instruct | 77.1 | 35 / 35 | 7.8openai/gpt-5.6-luna | 7.0anthropic/claude-haiku-5.5 | high |
| 91 | Command R7B Arabic 7B 02 2025 | 74.3 | 35 / 35 | 7.5openai/gpt-5.6-luna | 7.1anthropic/claude-haiku-5.5 | high |
| 91 | Deepcoder 14B Previewfine-tune of DeepSeek-R1-Distill-Qwen 14Bcode model | 74.3 | 35 / 35 | 6.7openai/gpt-5.6-luna | 6.2anthropic/claude-haiku-5.5 | high |
| 91 | Granite4 Tiny H | 74.3 | 35 / 35 | 7.6openai/gpt-5.6-luna | 6.9anthropic/claude-haiku-5.5 | high |
| 91 | Granite4.2 3B | 74.3 | 35 / 35 | 7.1openai/gpt-5.6-luna | 6.7anthropic/claude-haiku-5.5 | high |
| 91 | Lfm3 1.7B | 74.3 | 35 / 35 | 7.2openai/gpt-5.6-luna | 7.0anthropic/claude-haiku-5.5 | high |
| 91 | Ministral 3 3B Instruct 2512 | 74.3 | 35 / 35 | 7.5openai/gpt-5.6-luna | 6.6anthropic/claude-haiku-5.5 | high |
| 91 | Mistral Small 22B Instruct 2409 | 74.3 | 35 / 35 | 7.5openai/gpt-5.6-luna | 7.4anthropic/claude-haiku-5.5 | high |
| 91 | Qwen2.5-VL-7B-Instruct | 74.3 | 35 / 35 | 7.3openai/gpt-5.6-luna | 6.9anthropic/claude-haiku-5.5 | high |
| 91 | Qwen3-4B-Thinking-2507 | 74.3 | 35 / 35 | 7.3openai/gpt-5.6-luna | 6.9anthropic/claude-haiku-5.5 | high |
| 91 | Qwen3.5 2B | 74.3 | 35 / 35 | 7.3openai/gpt-5.6-luna | 6.8anthropic/claude-haiku-5.5 | high |
| 101 | Command R7B (12-2024) | 71.4 | 35 / 35 | 6.7openai/gpt-5.6-luna | 6.6anthropic/claude-haiku-5.5 | high |
| 101 | Deepseek R1 1.5B Qwen Distill | 71.4 | 35 / 35 | 6.1openai/gpt-5.6-luna | 6.2anthropic/claude-haiku-5.5 | high |
| 101 | dolphin-2.9.1-yi-1.5-34b | 71.4 | 35 / 35 | 7.5openai/gpt-5.6-luna | 6.9anthropic/claude-haiku-5.5 | high |
| 101 | Falcon3 7B Instruct | 71.4 | 35 / 35 | 7.3openai/gpt-5.6-luna | 7.0anthropic/claude-haiku-5.5 | high |
| 101 | Flint 40B A2B | 71.4 | 35 / 35 | 7.4openai/gpt-5.6-luna | 6.6anthropic/claude-haiku-5.5 | high |
| 101 | gemma-3-4b-it | 71.4 | 35 / 35 | 7.2openai/gpt-5.6-luna | 6.9anthropic/claude-haiku-5.5 | high |
| 101 | Marco O1 7Bfine-tune of Qwen 2 7B | 71.4 | 35 / 35 | 6.8openai/gpt-5.6-luna | 5.9anthropic/claude-haiku-5.5 | high |
| 101 | Qwen2 Math 7B Instructmath model | 71.4 | 35 / 35 | 6.5openai/gpt-5.6-luna | 6.6anthropic/claude-haiku-5.5 | high |
| 101 | Qwen2.5-3B-Instruct | 71.4 | 35 / 35 | 6.9openai/gpt-5.6-luna | 6.6anthropic/claude-haiku-5.5 | high |
| 101 | Qwen2.5-VL-3B-Instruct | 71.4 | 35 / 35 | 7.0openai/gpt-5.6-luna | 6.4anthropic/claude-haiku-5.5 | high |
| 101 | SmolLM3-3B | 71.4 | 35 / 35 | 7.3openai/gpt-5.6-luna | 6.6anthropic/claude-haiku-5.5 | high |
| 101 | YI 34B Chat V1.5 | 71.4 | 35 / 35 | 7.6openai/gpt-5.6-luna | 7.0anthropic/claude-haiku-5.5 | high |
| 113 | Deepscaler 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bmath model | 68.6 | 35 / 35 | 6.1openai/gpt-5.6-luna | 6.0anthropic/claude-haiku-5.5 | high |
| 113 | gemma-2-9b-it | 68.6 | 35 / 35 | 7.4openai/gpt-5.6-luna | 6.4anthropic/claude-haiku-5.5 | high |
| 113 | Gemma3n E2B IT | 68.6 | 35 / 35 | 6.4openai/gpt-5.6-luna | 6.4anthropic/claude-haiku-5.5 | high |
| 113 | Llama-3.2-3B-Instruct | 68.6 | 35 / 35 | 6.9openai/gpt-5.6-luna | 6.4anthropic/claude-haiku-5.5 | high |
| 113 | Nemotron Cascade 2 30B A3B | 68.6 | 35 / 35 | 6.7openai/gpt-5.6-luna | 6.3anthropic/claude-haiku-5.5 | high |
| 113 | Qwen3 1.7B | 68.6 | 35 / 35 | 7.0openai/gpt-5.6-luna | 6.4anthropic/claude-haiku-5.5 | high |
| 113 | YI 9B Chat V1.5 | 68.6 | 35 / 35 | 6.6openai/gpt-5.6-luna | 6.3anthropic/claude-haiku-5.5 | high |
| 120 | DeepSeek-Coder-V2-Lite-Instructcode model | 65.7 | 35 / 35 | 6.5openai/gpt-5.6-luna | 6.2anthropic/claude-haiku-5.5 | high |
| 120 | Gemma2 27B Instruct | 65.7 | 35 / 35 | 7.3openai/gpt-5.6-luna | 6.3anthropic/claude-haiku-5.5 | high |
| 120 | granite-4.1-3b | 65.7 | 35 / 35 | 6.7openai/gpt-5.6-luna | 6.1anthropic/claude-haiku-5.5 | high |
| 120 | Granite4.2 8B | 65.7 | 35 / 35 | 6.8openai/gpt-5.6-luna | 6.2anthropic/claude-haiku-5.5 | high |
| 120 | Internlm2 7B Chat V2.5 | 65.7 | 35 / 35 | 6.1openai/gpt-5.6-luna | 6.1anthropic/claude-haiku-5.5 | high |
| 120 | Mathstral 7B V0.1fine-tune of Mistral 7Bmath model | 65.7 | 35 / 35 | 6.1openai/gpt-5.6-luna | 6.3anthropic/claude-haiku-5.5 | high |
| 120 | medgemma-4b-itmedical model | 65.7 | 35 / 35 | 6.8openai/gpt-5.6-luna | 6.4anthropic/claude-haiku-5.5 | high |
| 120 | Medgemma1.5 4B ITfine-tune of Gemma 3medical model | 65.7 | 35 / 35 | 6.3openai/gpt-5.6-luna | 6.1anthropic/claude-haiku-5.5 | high |
| 120 | NVIDIA Nemotron 3 Nano 30B-A3B | 65.7 | 35 / 35 | 7.1openai/gpt-5.6-luna | 6.3anthropic/claude-haiku-5.5 | high |
| 120 | Qwen2 Math 1.5B Instructmath model | 65.7 | 35 / 35 | 5.9openai/gpt-5.6-luna | 6.1anthropic/claude-haiku-5.5 | high |
| 120 | Qwen2.5 Coder 1.5B Instructcode model | 65.7 | 35 / 35 | 5.9openai/gpt-5.6-luna | 5.6anthropic/claude-haiku-5.5 | high |
| 120 | Qwen3 VL 2B Instruct | 65.7 | 35 / 35 | 6.9openai/gpt-5.6-luna | 6.2anthropic/claude-haiku-5.5 | high |
| 132 | Granite3.3 8B | 62.9 | 35 / 35 | 6.5openai/gpt-5.6-luna | 6.2anthropic/claude-haiku-5.5 | high |
| 132 | Granite4 3B | 62.9 | 35 / 35 | 6.6openai/gpt-5.6-luna | 6.1anthropic/claude-haiku-5.5 | high |
| 132 | Granite4 Micro H | 62.9 | 35 / 35 | 6.9openai/gpt-5.6-luna | 6.2anthropic/claude-haiku-5.5 | high |
| 132 | Lfm3 VL 2B | 62.9 | 35 / 35 | 6.6openai/gpt-5.6-luna | 6.1anthropic/claude-haiku-5.5 | high |
| 132 | Olmo-3-7B-Think | 62.9 | 35 / 35 | 6.1openai/gpt-5.6-luna | 5.4anthropic/claude-haiku-5.5 | high |
| 132 | Qwen2.5 1.5B Instruct | 62.9 | 35 / 35 | 6.3openai/gpt-5.6-luna | 5.7anthropic/claude-haiku-5.5 | high |
| 138 | Gemma3 1B IT | 60.0 | 35 / 35 | 5.7openai/gpt-5.6-luna | 5.3anthropic/claude-haiku-5.5 | high |
| 138 | Glm4 9B Chat | 60.0 | 35 / 35 | 6.5openai/gpt-5.6-luna | 5.9anthropic/claude-haiku-5.5 | high |
| 138 | Llama-3.1-8B-Instruct | 60.0 | 35 / 35 | 6.6openai/gpt-5.6-luna | 5.9anthropic/claude-haiku-5.5 | high |
| 138 | Olmo2 13B 1124 Instruct | 60.0 | 35 / 35 | 6.5openai/gpt-5.6-luna | 6.0anthropic/claude-haiku-5.5 | high |
| 138 | Phi-3.5-mini-instruct | 60.0 | 35 / 35 | 6.2openai/gpt-5.6-luna | 5.6anthropic/claude-haiku-5.5 | high |
| 138 | Qwen3 VL 2B Thinking | 60.0 | 35 / 35 | 5.9openai/gpt-5.6-luna | 5.4anthropic/claude-haiku-5.5 | high |
| 138 | Tulu3 8Bfine-tune of Llama 3.1 | 60.0 | 35 / 35 | 6.1openai/gpt-5.6-luna | 5.5anthropic/claude-haiku-5.5 | high |
| 145 | Granite3.1 Dense 8B Instruct | 57.1 | 35 / 35 | 6.0openai/gpt-5.6-luna | 5.6anthropic/claude-haiku-5.5 | high |
| 145 | Granite3.3 2B | 57.1 | 35 / 35 | 6.4openai/gpt-5.6-luna | 5.9anthropic/claude-haiku-5.5 | high |
| 145 | Mistral Nemo 12B Instruct 2407 | 57.1 | 35 / 35 | 6.3openai/gpt-5.6-luna | 5.7anthropic/claude-haiku-5.5 | high |
| 145 | Qwen2.5-Coder-3B-Instructcode model | 57.1 | 35 / 35 | 5.9openai/gpt-5.6-luna | 5.7anthropic/claude-haiku-5.5 | high |
| 145 | Smallthinker 3B Previewfine-tune of Qwen 2.5 3B | 57.1 | 35 / 35 | 5.5openai/gpt-5.6-luna | 5.0anthropic/claude-haiku-5.5 | high |
| 150 | Aya Expanse 32B | 54.3 | 35 / 35 | 6.1openai/gpt-5.6-luna | 5.8anthropic/claude-haiku-5.5 | high |
| 150 | Aya Expanse 8B | 54.3 | 35 / 35 | 5.7openai/gpt-5.6-luna | 5.4anthropic/claude-haiku-5.5 | high |
| 150 | Granite3 Dense 8B Instruct | 54.3 | 35 / 35 | 5.7openai/gpt-5.6-luna | 5.4anthropic/claude-haiku-5.5 | high |
| 150 | Hermes-3-Llama-3.1-8B | 54.3 | 35 / 35 | 5.6openai/gpt-5.6-luna | 5.5anthropic/claude-haiku-5.5 | high |
| 150 | Phi3 3.8B Mini 128K Instruct | 54.3 | 35 / 35 | 6.1openai/gpt-5.6-luna | 5.3anthropic/claude-haiku-5.5 | high |
| 150 | Sailor2 8B Chatfine-tune of Qwen 2.5 | 54.3 | 35 / 35 | 5.8openai/gpt-5.6-luna | 5.5anthropic/claude-haiku-5.5 | high |
| 156 | Falcon3 3B Instruct | 51.4 | 35 / 35 | 5.7openai/gpt-5.6-luna | 5.1anthropic/claude-haiku-5.5 | high |
| 156 | Meta-Llama-3-8B-Instruct | 51.4 | 35 / 35 | 6.0openai/gpt-5.6-luna | 5.1anthropic/claude-haiku-5.5 | high |
| 156 | Olmo2 7B 1124 Instruct | 51.4 | 35 / 35 | 5.9openai/gpt-5.6-luna | 5.1anthropic/claude-haiku-5.5 | high |
| 159 | Codegeex4 9B Allcode model | 48.6 | 35 / 35 | 5.1openai/gpt-5.6-luna | 4.8anthropic/claude-haiku-5.5 | high |
| 159 | Command R 35B 08 2024 | 48.6 | 35 / 35 | 5.4openai/gpt-5.6-luna | 4.8anthropic/claude-haiku-5.5 | high |
| 159 | Granite3 Dense 2B Instruct | 48.6 | 35 / 35 | 5.5openai/gpt-5.6-luna | 4.7anthropic/claude-haiku-5.5 | high |
| 159 | Granite4 1B H | 48.6 | 35 / 35 | 4.7openai/gpt-5.6-luna | 4.1anthropic/claude-haiku-5.5 | high |
| 159 | Qwen 14B Chat V1.5 | 48.6 | 35 / 35 | 5.4openai/gpt-5.6-luna | 4.8anthropic/claude-haiku-5.5 | high |
| 164 | Codegemma 7B Instruct V1.1code model | 45.7 | 35 / 35 | 5.3openai/gpt-5.6-luna | 4.9anthropic/claude-haiku-5.5 | high |
| 164 | Codestral 22B V0.1code model | 45.7 | 35 / 35 | 5.3openai/gpt-5.6-luna | 4.9anthropic/claude-haiku-5.5 | high |
| 164 | Cogito 3B V1 Preview Llamafine-tune of Llama 3.2 | 45.7 | 35 / 35 | 5.4openai/gpt-5.6-luna | 5.0anthropic/claude-haiku-5.5 | high |
| 164 | Granite3.1 Dense 2B Instruct | 45.7 | 35 / 35 | 5.1openai/gpt-5.6-luna | 4.6anthropic/claude-haiku-5.5 | high |
| 164 | Granite3.2 8B Instruct | 45.7 | 35 / 35 | 5.5openai/gpt-5.6-luna | 4.9anthropic/claude-haiku-5.5 | high |
| 164 | Llama2 13B Chat | 45.7 | 35 / 35 | 4.8openai/gpt-5.6-luna | 4.3anthropic/claude-haiku-5.5 | high |
| 164 | Qwen 7B Chat V1.5 | 45.7 | 35 / 35 | 5.0openai/gpt-5.6-luna | 4.4anthropic/claude-haiku-5.5 | high |
| 164 | Qwen2-VL-7B-Instruct | 45.7 | 35 / 35 | 4.7openai/gpt-5.6-luna | 4.2anthropic/claude-haiku-5.5 | high |
| 164 | Solar 10.7B Instruct V1 | 45.7 | 35 / 35 | 5.0openai/gpt-5.6-luna | 4.5anthropic/claude-haiku-5.5 | high |
| 164 | Stablelm Zephyr 3Bfine-tune of StableLM 3B | 45.7 | 35 / 35 | 4.4openai/gpt-5.6-luna | 3.9anthropic/claude-haiku-5.5 | high |
| 164 | YI 6B Chat V1.5 | 45.7 | 35 / 35 | 4.8openai/gpt-5.6-luna | 4.3anthropic/claude-haiku-5.5 | high |
| 175 | Apertus-8B-Instruct-2509 | 42.9 | 35 / 35 | 5.1openai/gpt-5.6-luna | 4.6anthropic/claude-haiku-5.5 | high |
| 175 | Codellama 34B Instructcode model | 42.9 | 35 / 35 | 4.7openai/gpt-5.6-luna | 3.8anthropic/claude-haiku-5.5 | high |
| 175 | Gemma2 2B Instruct | 42.9 | 35 / 35 | 5.1openai/gpt-5.6-luna | 4.3anthropic/claude-haiku-5.5 | high |
| 175 | Granite3.2 2B Instruct | 42.9 | 35 / 35 | 5.0openai/gpt-5.6-luna | 4.3anthropic/claude-haiku-5.5 | high |
| 175 | Mixtral 8x7B Instruct V0.1 | 42.9 | 35 / 35 | 4.9openai/gpt-5.6-luna | 4.5anthropic/claude-haiku-5.5 | high |
| 180 | Falcon3 1B Instruct | 40.0 | 35 / 35 | 4.3openai/gpt-5.6-luna | 4.1anthropic/claude-haiku-5.5 | high |
| 180 | Gemma 7B Instruct V1.1 | 40.0 | 35 / 35 | 4.8openai/gpt-5.6-luna | 4.3anthropic/claude-haiku-5.5 | high |
| 180 | Granite Code 8B Instructcode model | 40.0 | 35 / 35 | 4.5openai/gpt-5.6-luna | 4.1anthropic/claude-haiku-5.5 | high |
| 180 | Mistral-7B-Instruct-v0.2 | 40.0 | 35 / 35 | 4.3openai/gpt-5.6-luna | 4.1anthropic/claude-haiku-5.5 | high |
| 184 | Codellama 7B Instructcode model | 37.1 | 35 / 35 | 4.2openai/gpt-5.6-luna | 3.7anthropic/claude-haiku-5.5 | high |
| 184 | Deepseek Llm 7B Chat | 37.1 | 35 / 35 | 4.5openai/gpt-5.6-luna | 4.1anthropic/claude-haiku-5.5 | high |
| 184 | Granite Code 3B Instructcode model | 37.1 | 35 / 35 | 3.9openai/gpt-5.6-luna | 3.4anthropic/claude-haiku-5.5 | high |
| 184 | Granite3 Moe 3B Instruct | 37.1 | 35 / 35 | 4.4openai/gpt-5.6-luna | 3.6anthropic/claude-haiku-5.5 | high |
| 184 | Granite3.1 Moe 3B Instruct | 37.1 | 35 / 35 | 4.5openai/gpt-5.6-luna | 3.9anthropic/claude-haiku-5.5 | high |
| 184 | Hermes3 3B Llama3.2fine-tune of Llama 3.2 | 37.1 | 35 / 35 | 4.4openai/gpt-5.6-luna | 3.9anthropic/claude-haiku-5.5 | high |
| 184 | Llama3.2 1B Instruct | 37.1 | 35 / 35 | 4.1openai/gpt-5.6-luna | 3.9anthropic/claude-haiku-5.5 | high |
| 184 | Mistral 7B Instruct V0.3 | 37.1 | 35 / 35 | 3.9openai/gpt-5.6-luna | 3.5anthropic/claude-haiku-5.5 | high |
| 184 | Qwen3 0.6B | 37.1 | 35 / 35 | 4.2openai/gpt-5.6-luna | 3.9anthropic/claude-haiku-5.5 | high |
| 193 | Aya 8B 23 | 34.3 | 35 / 35 | 3.5openai/gpt-5.6-luna | 2.8anthropic/claude-haiku-5.5 | high |
| 193 | Deepseek V2 16B Lite Chat | 34.3 | 35 / 35 | 4.3openai/gpt-5.6-luna | 4.0anthropic/claude-haiku-5.5 | high |
| 193 | Magistral Small (2506) | 34.3 | 35 / 35 | 2.9openai/gpt-5.6-luna | 3.0anthropic/claude-haiku-5.5 | high |
| 193 | Nemotron Mini 4B Instruct | 34.3 | 35 / 35 | 3.7openai/gpt-5.6-luna | 3.3anthropic/claude-haiku-5.5 | high |
| 193 | Phi 2.7B Chat V2 | 34.3 | 35 / 35 | 3.1openai/gpt-5.6-luna | 2.7anthropic/claude-haiku-5.5 | high |
| 198 | Codellama 13B Instructcode model | 31.4 | 35 / 35 | 3.9openai/gpt-5.6-luna | 3.4anthropic/claude-haiku-5.5 | high |
| 198 | Deepseek Coder 33B Instructcode model | 31.4 | 35 / 35 | 3.6openai/gpt-5.6-luna | 3.1anthropic/claude-haiku-5.5 | high |
| 198 | Dolphin Phi 2.7B V2.6fine-tune of Phi-2 | 31.4 | 35 / 35 | 4.2openai/gpt-5.6-luna | 3.7anthropic/claude-haiku-5.5 | high |
| 198 | Granite3.1 Moe 1B Instruct | 31.4 | 35 / 35 | 3.5openai/gpt-5.6-luna | 3.2anthropic/claude-haiku-5.5 | high |
| 198 | Internlm2 1.8B Chat V2.5 | 31.4 | 35 / 35 | 3.3openai/gpt-5.6-luna | 3.1anthropic/claude-haiku-5.5 | high |
| 198 | Qwen2 0.5B Instruct | 31.4 | 35 / 35 | 3.4openai/gpt-5.6-luna | 3.1anthropic/claude-haiku-5.5 | high |
| 198 | YI Coder 9B Chatcode model | 31.4 | 35 / 35 | 3.8openai/gpt-5.6-luna | 3.6anthropic/claude-haiku-5.5 | high |
| 205 | Qwen2 1.5B Instruct | 28.6 | 35 / 35 | 3.4openai/gpt-5.6-luna | 3.3anthropic/claude-haiku-5.5 | high |
| 205 | Qwen2.5 0.5B Instruct | 28.6 | 35 / 35 | 3.6openai/gpt-5.6-luna | 3.3anthropic/claude-haiku-5.5 | high |
| 205 | Smollm2 1.7B Instruct | 28.6 | 35 / 35 | 2.8openai/gpt-5.6-luna | 2.4anthropic/claude-haiku-5.5 | high |
| 208 | Codeqwen 7B Chat V1.5code model | 25.7 | 35 / 35 | 3.1openai/gpt-5.6-luna | 2.9anthropic/claude-haiku-5.5 | high |
| 208 | Gemma 2B Instruct V1.1 | 25.7 | 35 / 35 | 2.9openai/gpt-5.6-luna | 2.4anthropic/claude-haiku-5.5 | high |
| 208 | Llama-2-7b-chat-hf | 25.7 | 35 / 35 | 3.6openai/gpt-5.6-luna | 3.0anthropic/claude-haiku-5.5 | high |
| 211 | Falcon2 11Bnot in the Overall ranking | 22.9 | 35 / 35 | 2.9openai/gpt-5.6-luna | 2.2anthropic/claude-haiku-5.5 | high |
| 211 | Qwen2.5 Coder 0.5B Instructcode model | 22.9 | 35 / 35 | 2.7openai/gpt-5.6-luna | 2.1anthropic/claude-haiku-5.5 | high |
| 211 | Stable Code 3B Instructcode model | 22.9 | 35 / 35 | 2.9openai/gpt-5.6-luna | 2.3anthropic/claude-haiku-5.5 | high |
| 211 | Stablelm2 12B Chat | 22.9 | 35 / 35 | 3.2openai/gpt-5.6-luna | 2.8anthropic/claude-haiku-5.5 | high |
| 215 | Granite4 350M H | 20.0 | 35 / 35 | 2.1openai/gpt-5.6-luna | 1.9anthropic/claude-haiku-5.5 | high |
| 215 | Qwen 1.8B Chat V1.5 | 20.0 | 35 / 35 | 2.3openai/gpt-5.6-luna | 2.3anthropic/claude-haiku-5.5 | high |
| 217 | Codegemma 2B Code V1.1code model | 17.1 | 35 / 35 | 2.1openai/gpt-5.6-luna | 1.0anthropic/claude-haiku-5.5 | high |
| 217 | Granite3 Moe 1B Instruct | 17.1 | 35 / 35 | 2.4openai/gpt-5.6-luna | 2.2anthropic/claude-haiku-5.5 | high |
| 217 | YI Coder 1.5B Chatcode model | 17.1 | 35 / 35 | 2.6openai/gpt-5.6-luna | 2.1anthropic/claude-haiku-5.5 | high |
| 220 | Qwen 0.5B Chat V1.5 | 14.3 | 35 / 35 | 2.0openai/gpt-5.6-luna | 1.4anthropic/claude-haiku-5.5 | high |
| 221 | Qwen 4B Chat V1.5 | 11.4 | 35 / 35 | 1.8openai/gpt-5.6-luna | 1.3anthropic/claude-haiku-5.5 | high |
| 222 | Deepseek Coder 6.7B Instructcode model | 8.6 | 35 / 35 | 2.3openai/gpt-5.6-luna | 1.6anthropic/claude-haiku-5.5 | high |
| 222 | Sailor2 1B Chatfine-tune of Qwen 2.5 | 8.6 | 35 / 35 | 1.9openai/gpt-5.6-luna | 1.6anthropic/claude-haiku-5.5 | high |
| 222 | Smollm2 360M Instruct | 8.6 | 35 / 35 | 1.5openai/gpt-5.6-luna | 1.3anthropic/claude-haiku-5.5 | high |
| 222 | Stablelm2 1.6B Chat | 8.6 | 35 / 35 | 1.9openai/gpt-5.6-luna | 1.6anthropic/claude-haiku-5.5 | high |
| 226 | Deepseek Coder 1.3B Instructcode model | 5.7 | 35 / 35 | 1.7openai/gpt-5.6-luna | 1.1anthropic/claude-haiku-5.5 | high |
| 227 | Opencoder 1.5B Instructcode model | 2.9 | 35 / 35 | 1.7openai/gpt-5.6-luna | 1.5anthropic/claude-haiku-5.5 | high |
| 227 | Smollm2 135M Instruct | 2.9 | 35 / 35 | 1.1openai/gpt-5.6-luna | 0.7anthropic/claude-haiku-5.5 | high |
A dash means not scored, which is different from 0. Agreement compares the two judges: high means a mean gap under 1.5 points, medium under 3, low otherwise; n/a means fewer than 3 items were scored by both.
Not ranked in this category
- Apertus-8B-Instruct-2509: withheld from the ranking (see its page)
- Falcon3 10B Instruct: withheld from the ranking (see its page)
- Falcon3 1B Instruct: withheld from the ranking (see its page)
- Falcon3 3B Instruct: withheld from the ranking (see its page)
- Falcon3 7B Instruct: withheld from the ranking (see its page)
- Internlm2 1.8B Chat V2.5: withheld from the ranking (see its page)
- Phi-4-reasoning 14B: withheld from the ranking (see its page)
- Phi-4-reasoning-plus 14B: withheld from the ranking (see its page)
- Qwen2 Math 1.5B Instruct: withheld from the ranking (see its page)
- Qwen2 Math 7B Instruct: withheld from the ranking (see its page)
The questions
In run r20261102-bv1-5246ab, 5 of the 35 questions in this category were answered correctly by every ranked model that has a score for them (33 ranked models in that run). Hard questions are the ones that half or fewer of the ranked models of 3B and over answered correctly. This category has 7.
The same ranking as JSON: reasoning.json.