All categories

Reasoning

Multi-step reasoning questions that each have one exact answer. The model gives its final answer on the last line, and code compares it with the reference.

This category counts toward Overall. The two judge models also score every item, and their scores are shown here separately. How the battery works.

Ranking

Models are ranked by their mean score in this category, from 0 to 100. Only models with at least 80% of the category's items scored, and none pending, are ranked here. The rule is the same as for Overall.

Reasoning: 228 ranked models. Click a column heading to sort.
Rank Model Mean (0 to 100) Items scored Judge A mean Judge B mean Agreement
1 DeepSeek-R1-Distill-Qwen-32B 97.1 35 / 35 8.9openai/gpt-5.6-luna 8.8anthropic/claude-haiku-5.5 high
1 gemma-4-31B-it 97.1 35 / 35 9.4openai/gpt-5.6-luna 9.1anthropic/claude-haiku-5.5 high
1 gpt-oss-120b 97.1 35 / 35 8.0anthropic/claude-haiku-5.5 8.2deepseek/deepseek-v4.1-flash high
1 gpt-oss-20b 97.1 35 / 35 7.1anthropic/claude-haiku-5.5 7.2deepseek/deepseek-v4.1-flash high
1 Llama-3.3-70B-Instruct 97.1 35 / 35 9.4openai/gpt-5.6-luna 8.8anthropic/claude-haiku-5.5 high
6 Gemma 4 12B IT 94.3 35 / 35 9.5openai/gpt-5.6-luna 8.9anthropic/claude-haiku-5.5 high
6 gemma-4-26B-A4B-it 94.3 35 / 35 9.1openai/gpt-5.6-luna 8.8anthropic/claude-haiku-5.5 high
6 Muse Glimmer 30B 94.3 35 / 35 9.2openai/gpt-5.6-luna 8.7anthropic/claude-haiku-5.5 high
6 Phi4 Mini Reasoning 3.8B 94.3 35 / 35 7.9openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 high
6 Qwen3-8B 94.3 35 / 35 8.9openai/gpt-5.6-luna 8.4anthropic/claude-haiku-5.5 high
6 QwQ 32B 94.3 35 / 35 8.1openai/gpt-5.6-luna 7.6anthropic/claude-haiku-5.5 high
12 Deepseek R1 8B Llama Distill 91.4 35 / 35 8.1openai/gpt-5.6-luna 8.0anthropic/claude-haiku-5.5 high
12 DeepSeek-R1-Distill-Qwen-14B 91.4 35 / 35 8.3openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 high
12 gemma-3-27b-it 91.4 35 / 35 9.0openai/gpt-5.6-luna 8.2anthropic/claude-haiku-5.5 high
12 Granite4.1 8B 91.4 35 / 35 9.0openai/gpt-5.6-luna 8.7anthropic/claude-haiku-5.5 high
12 Laguna XS 2.1 91.4 35 / 35 9.1openai/gpt-5.6-luna 8.7anthropic/claude-haiku-5.5 high
12 Llama 4 Scout 17B-16E Instruct 91.4 35 / 35 8.7openai/gpt-5.6-luna 8.2anthropic/claude-haiku-5.5 high
12 Ornith 9B 91.4 35 / 35 9.2openai/gpt-5.6-luna 8.7anthropic/claude-haiku-5.5 high
12 Ornith-1.5-9B 91.4 35 / 35 9.1openai/gpt-5.6-luna 8.5anthropic/claude-haiku-5.5 high
12 phi-4 91.4 35 / 35 8.8openai/gpt-5.6-luna 8.3anthropic/claude-haiku-5.5 high
12 Qwen3-14B 91.4 35 / 35 8.9openai/gpt-5.6-luna 8.5anthropic/claude-haiku-5.5 high
12 Qwen3-VL-4B-Instruct 91.4 35 / 35 9.0openai/gpt-5.6-luna 8.4anthropic/claude-haiku-5.5 high
12 Qwen3-VL-8B-Instruct 91.4 35 / 35 9.1openai/gpt-5.6-luna 8.6anthropic/claude-haiku-5.5 high
24 gemma-3-12b-it 88.6 35 / 35 8.8openai/gpt-5.6-luna 8.2anthropic/claude-haiku-5.5 high
24 Gemma4 E2B IT 88.6 35 / 35 8.6openai/gpt-5.6-luna 8.4anthropic/claude-haiku-5.5 high
24 Granite4 Small H 88.6 35 / 35 8.9openai/gpt-5.6-luna 8.4anthropic/claude-haiku-5.5 high
24 Mistral Small3.1 24B Instruct 2503 88.6 35 / 35 8.5openai/gpt-5.6-luna 8.0anthropic/claude-haiku-5.5 high
24 Nanbeige_Nanbeige4.3-33B 88.6 35 / 35 8.9openai/gpt-5.6-luna 8.5anthropic/claude-haiku-5.5 high
24 Openthinker 7B V2fine-tune of Qwen 2.5 88.6 35 / 35 7.6openai/gpt-5.6-luna 7.5anthropic/claude-haiku-5.5 high
24 Qwen2.5-14B-Instruct 88.6 35 / 35 8.7openai/gpt-5.6-luna 8.1anthropic/claude-haiku-5.5 high
24 Qwen2.5-VL-32B-Instruct 88.6 35 / 35 8.1openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 high
24 Qwen3-30B-A3B 88.6 35 / 35 8.8openai/gpt-5.6-luna 8.2anthropic/claude-haiku-5.5 high
24 Qwen3-30B-A3B-Instruct-2507 88.6 35 / 35 8.8openai/gpt-5.6-luna 8.5anthropic/claude-haiku-5.5 high
24 Qwen3-4B 88.6 35 / 35 8.8openai/gpt-5.6-luna 8.4anthropic/claude-haiku-5.5 high
24 Qwen3-Coder-30B-A3B-Instructcode model 88.6 35 / 35 8.7openai/gpt-5.6-luna 8.3anthropic/claude-haiku-5.5 high
24 Qwen3-Coder-Nextcode model 88.6 35 / 35 8.8openai/gpt-5.6-luna 8.7anthropic/claude-haiku-5.5 high
24 Qwen3-VL-32B-Instruct 88.6 35 / 35 9.0openai/gpt-5.6-luna 8.7anthropic/claude-haiku-5.5 high
24 Qwen3.5-27B 88.6 35 / 35 8.9openai/gpt-5.6-luna 8.5anthropic/claude-haiku-5.5 high
24 Qwen3.6-27B 88.6 35 / 35 8.8openai/gpt-5.6-luna 8.3anthropic/claude-haiku-5.5 high
24 Qwen3.8-27B 88.6 35 / 35 8.7openai/gpt-5.6-luna 8.4anthropic/claude-haiku-5.5 high
24 Sage-A1.5-Q4_K_M 88.6 35 / 35 8.9openai/gpt-5.6-luna 8.7anthropic/claude-haiku-5.5 high
42 Devstral Small 2 24B Instruct (2512)code model 85.7 35 / 35 8.5openai/gpt-5.6-luna 8.0anthropic/claude-haiku-5.5 high
42 Exaone Deep 7.8B 85.7 35 / 35 7.0openai/gpt-5.6-luna 6.5anthropic/claude-haiku-5.5 high
42 Gemma4 E4B IT 85.7 35 / 35 8.6openai/gpt-5.6-luna 8.2anthropic/claude-haiku-5.5 high
42 Lfm2.5 8B A1B 85.7 35 / 35 7.8openai/gpt-5.6-luna 6.9anthropic/claude-haiku-5.5 high
42 Mistral Small 24B Instruct 2501 85.7 35 / 35 8.5openai/gpt-5.6-luna 8.0anthropic/claude-haiku-5.5 high
42 Nemotron 3.5 Lightning 30B A3B 85.7 35 / 35 8.7openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 high
42 Qwen2.5-32B-Instruct 85.7 35 / 35 8.6openai/gpt-5.6-luna 8.0anthropic/claude-haiku-5.5 high
42 Qwen3-32B 85.7 35 / 35 8.7openai/gpt-5.6-luna 8.1anthropic/claude-haiku-5.5 high
42 Qwen3.5-35B-A3B 85.7 35 / 35 8.7openai/gpt-5.6-luna 8.2anthropic/claude-haiku-5.5 high
42 Qwen3.5-4B 85.7 35 / 35 8.6openai/gpt-5.6-luna 8.2anthropic/claude-haiku-5.5 high
42 Qwen3.5-9B 85.7 35 / 35 8.7openai/gpt-5.6-luna 8.2anthropic/claude-haiku-5.5 high
42 Qwen3.6-35B-A3B 85.7 35 / 35 8.7openai/gpt-5.6-luna 8.1anthropic/claude-haiku-5.5 high
42 Spark-X2.6-4B 85.7 35 / 35 8.7openai/gpt-5.6-luna 8.3anthropic/claude-haiku-5.5 high
55 Cogito 14B V1 Preview Qwenfine-tune of Qwen 2.5 82.9 35 / 35 8.2openai/gpt-5.6-luna 7.6anthropic/claude-haiku-5.5 high
55 Exaone Deep 2.4B 82.9 35 / 35 7.0openai/gpt-5.6-luna 6.6anthropic/claude-haiku-5.5 high
55 Exaone3.5 2.4B Instruct 82.9 35 / 35 7.9openai/gpt-5.6-luna 7.2anthropic/claude-haiku-5.5 high
55 Gemma 3n E4B IT 82.9 35 / 35 7.7openai/gpt-5.6-luna 7.5anthropic/claude-haiku-5.5 high
55 GLM-4.5-Air 82.9 35 / 35 8.0openai/gpt-5.6-luna 7.3anthropic/claude-haiku-5.5 high
55 Granite4.1 30B 82.9 35 / 35 8.6openai/gpt-5.6-luna 8.0anthropic/claude-haiku-5.5 high
55 Lfm2.5 Thinking 1.2B 82.9 35 / 35 5.5openai/gpt-5.6-luna 5.5anthropic/claude-haiku-5.5 high
55 Medgemma 27B ITfine-tune of Gemma 3medical model 82.9 35 / 35 7.3openai/gpt-5.6-luna 7.6anthropic/claude-haiku-5.5 high
55 Ministral 3 14B Instruct (2512) 82.9 35 / 35 8.6openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 high
55 Ministral 3 8B Instruct 2512 82.9 35 / 35 8.2openai/gpt-5.6-luna 7.6anthropic/claude-haiku-5.5 high
55 Mistral Small 3.2 24B Instruct (2506) 82.9 35 / 35 8.4openai/gpt-5.6-luna 7.9anthropic/claude-haiku-5.5 high
55 Nemotron 3.5 Bolt E4B 82.9 35 / 35 8.3openai/gpt-5.6-luna 8.0anthropic/claude-haiku-5.5 high
55 Olmo 3.1 32B Instruct 82.9 35 / 35 8.3openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 high
55 Qwen2.5-Coder-14B-Instructcode model 82.9 35 / 35 8.1openai/gpt-5.6-luna 7.6anthropic/claude-haiku-5.5 high
55 Qwen2.5-Coder-7B-Instructcode model 82.9 35 / 35 8.0openai/gpt-5.6-luna 7.7anthropic/claude-haiku-5.5 high
55 Qwen3-4B-Instruct-2507 82.9 35 / 35 8.4openai/gpt-5.6-luna 7.9anthropic/claude-haiku-5.5 high
55 Qwen3-VL-30B-A3B-Instruct 82.9 35 / 35 8.6openai/gpt-5.6-luna 8.1anthropic/claude-haiku-5.5 high
55 Translategemma 12B ITfine-tune of Gemma 3translation model 82.9 35 / 35 8.3openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 high
73 Cogito 8B V1 Preview Llamafine-tune of Llama 3.1 80.0 35 / 35 8.0openai/gpt-5.6-luna 7.6anthropic/claude-haiku-5.5 high
73 Deepcoder 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bcode model 80.0 35 / 35 7.0openai/gpt-5.6-luna 7.0anthropic/claude-haiku-5.5 high
73 EXAONE-3.5-7.8B-Instruct 80.0 35 / 35 8.3openai/gpt-5.6-luna 7.6anthropic/claude-haiku-5.5 high
73 Falcon3 10B Instruct 80.0 35 / 35 8.1openai/gpt-5.6-luna 7.4anthropic/claude-haiku-5.5 high
73 Granite4.2 30B 80.0 35 / 35 8.3openai/gpt-5.6-luna 7.9anthropic/claude-haiku-5.5 high
73 Nemotron3 33B 80.0 35 / 35 7.7openai/gpt-5.6-luna 7.3anthropic/claude-haiku-5.5 high
73 Olmo-3-7B-Instruct 80.0 35 / 35 8.2openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 high
73 Qwen2.5-Coder-32B-Instructcode model 80.0 35 / 35 8.1openai/gpt-5.6-luna 7.4anthropic/claude-haiku-5.5 high
81 DeepSeek-R1-0528-Qwen3-8B 77.1 35 / 35 7.5openai/gpt-5.6-luna 6.9anthropic/claude-haiku-5.5 high
81 DeepSeek-R1-Distill-Qwen-7B 77.1 35 / 35 7.1openai/gpt-5.6-luna 6.7anthropic/claude-haiku-5.5 high
81 Devstral 24B Small 2505code model 77.1 35 / 35 7.7openai/gpt-5.6-luna 7.4anthropic/claude-haiku-5.5 high
81 Exaone3.5 32B Instruct 77.1 35 / 35 8.1openai/gpt-5.6-luna 7.4anthropic/claude-haiku-5.5 high
81 GLM-4.7-Flash 77.1 35 / 35 7.4openai/gpt-5.6-luna 6.6anthropic/claude-haiku-5.5 high
81 Lfm2 24B 77.1 35 / 35 7.5openai/gpt-5.6-luna 7.2anthropic/claude-haiku-5.5 high
81 Phi-3-mini-4k-instruct 77.1 35 / 35 7.4openai/gpt-5.6-luna 6.7anthropic/claude-haiku-5.5 high
81 Phi-4-mini-instruct 77.1 35 / 35 7.5openai/gpt-5.6-luna 6.7anthropic/claude-haiku-5.5 high
81 Phi3 14B Medium 128K Instruct 77.1 35 / 35 7.7openai/gpt-5.6-luna 6.9anthropic/claude-haiku-5.5 high
81 Qwen2.5-7B-Instruct 77.1 35 / 35 7.8openai/gpt-5.6-luna 7.0anthropic/claude-haiku-5.5 high
91 Command R7B Arabic 7B 02 2025 74.3 35 / 35 7.5openai/gpt-5.6-luna 7.1anthropic/claude-haiku-5.5 high
91 Deepcoder 14B Previewfine-tune of DeepSeek-R1-Distill-Qwen 14Bcode model 74.3 35 / 35 6.7openai/gpt-5.6-luna 6.2anthropic/claude-haiku-5.5 high
91 Granite4 Tiny H 74.3 35 / 35 7.6openai/gpt-5.6-luna 6.9anthropic/claude-haiku-5.5 high
91 Granite4.2 3B 74.3 35 / 35 7.1openai/gpt-5.6-luna 6.7anthropic/claude-haiku-5.5 high
91 Lfm3 1.7B 74.3 35 / 35 7.2openai/gpt-5.6-luna 7.0anthropic/claude-haiku-5.5 high
91 Ministral 3 3B Instruct 2512 74.3 35 / 35 7.5openai/gpt-5.6-luna 6.6anthropic/claude-haiku-5.5 high
91 Mistral Small 22B Instruct 2409 74.3 35 / 35 7.5openai/gpt-5.6-luna 7.4anthropic/claude-haiku-5.5 high
91 Qwen2.5-VL-7B-Instruct 74.3 35 / 35 7.3openai/gpt-5.6-luna 6.9anthropic/claude-haiku-5.5 high
91 Qwen3-4B-Thinking-2507 74.3 35 / 35 7.3openai/gpt-5.6-luna 6.9anthropic/claude-haiku-5.5 high
91 Qwen3.5 2B 74.3 35 / 35 7.3openai/gpt-5.6-luna 6.8anthropic/claude-haiku-5.5 high
101 Command R7B (12-2024) 71.4 35 / 35 6.7openai/gpt-5.6-luna 6.6anthropic/claude-haiku-5.5 high
101 Deepseek R1 1.5B Qwen Distill 71.4 35 / 35 6.1openai/gpt-5.6-luna 6.2anthropic/claude-haiku-5.5 high
101 dolphin-2.9.1-yi-1.5-34b 71.4 35 / 35 7.5openai/gpt-5.6-luna 6.9anthropic/claude-haiku-5.5 high
101 Falcon3 7B Instruct 71.4 35 / 35 7.3openai/gpt-5.6-luna 7.0anthropic/claude-haiku-5.5 high
101 Flint 40B A2B 71.4 35 / 35 7.4openai/gpt-5.6-luna 6.6anthropic/claude-haiku-5.5 high
101 gemma-3-4b-it 71.4 35 / 35 7.2openai/gpt-5.6-luna 6.9anthropic/claude-haiku-5.5 high
101 Marco O1 7Bfine-tune of Qwen 2 7B 71.4 35 / 35 6.8openai/gpt-5.6-luna 5.9anthropic/claude-haiku-5.5 high
101 Qwen2 Math 7B Instructmath model 71.4 35 / 35 6.5openai/gpt-5.6-luna 6.6anthropic/claude-haiku-5.5 high
101 Qwen2.5-3B-Instruct 71.4 35 / 35 6.9openai/gpt-5.6-luna 6.6anthropic/claude-haiku-5.5 high
101 Qwen2.5-VL-3B-Instruct 71.4 35 / 35 7.0openai/gpt-5.6-luna 6.4anthropic/claude-haiku-5.5 high
101 SmolLM3-3B 71.4 35 / 35 7.3openai/gpt-5.6-luna 6.6anthropic/claude-haiku-5.5 high
101 YI 34B Chat V1.5 71.4 35 / 35 7.6openai/gpt-5.6-luna 7.0anthropic/claude-haiku-5.5 high
113 Deepscaler 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bmath model 68.6 35 / 35 6.1openai/gpt-5.6-luna 6.0anthropic/claude-haiku-5.5 high
113 gemma-2-9b-it 68.6 35 / 35 7.4openai/gpt-5.6-luna 6.4anthropic/claude-haiku-5.5 high
113 Gemma3n E2B IT 68.6 35 / 35 6.4openai/gpt-5.6-luna 6.4anthropic/claude-haiku-5.5 high
113 Llama-3.2-3B-Instruct 68.6 35 / 35 6.9openai/gpt-5.6-luna 6.4anthropic/claude-haiku-5.5 high
113 Nemotron Cascade 2 30B A3B 68.6 35 / 35 6.7openai/gpt-5.6-luna 6.3anthropic/claude-haiku-5.5 high
113 Qwen3 1.7B 68.6 35 / 35 7.0openai/gpt-5.6-luna 6.4anthropic/claude-haiku-5.5 high
113 YI 9B Chat V1.5 68.6 35 / 35 6.6openai/gpt-5.6-luna 6.3anthropic/claude-haiku-5.5 high
120 DeepSeek-Coder-V2-Lite-Instructcode model 65.7 35 / 35 6.5openai/gpt-5.6-luna 6.2anthropic/claude-haiku-5.5 high
120 Gemma2 27B Instruct 65.7 35 / 35 7.3openai/gpt-5.6-luna 6.3anthropic/claude-haiku-5.5 high
120 granite-4.1-3b 65.7 35 / 35 6.7openai/gpt-5.6-luna 6.1anthropic/claude-haiku-5.5 high
120 Granite4.2 8B 65.7 35 / 35 6.8openai/gpt-5.6-luna 6.2anthropic/claude-haiku-5.5 high
120 Internlm2 7B Chat V2.5 65.7 35 / 35 6.1openai/gpt-5.6-luna 6.1anthropic/claude-haiku-5.5 high
120 Mathstral 7B V0.1fine-tune of Mistral 7Bmath model 65.7 35 / 35 6.1openai/gpt-5.6-luna 6.3anthropic/claude-haiku-5.5 high
120 medgemma-4b-itmedical model 65.7 35 / 35 6.8openai/gpt-5.6-luna 6.4anthropic/claude-haiku-5.5 high
120 Medgemma1.5 4B ITfine-tune of Gemma 3medical model 65.7 35 / 35 6.3openai/gpt-5.6-luna 6.1anthropic/claude-haiku-5.5 high
120 NVIDIA Nemotron 3 Nano 30B-A3B 65.7 35 / 35 7.1openai/gpt-5.6-luna 6.3anthropic/claude-haiku-5.5 high
120 Qwen2 Math 1.5B Instructmath model 65.7 35 / 35 5.9openai/gpt-5.6-luna 6.1anthropic/claude-haiku-5.5 high
120 Qwen2.5 Coder 1.5B Instructcode model 65.7 35 / 35 5.9openai/gpt-5.6-luna 5.6anthropic/claude-haiku-5.5 high
120 Qwen3 VL 2B Instruct 65.7 35 / 35 6.9openai/gpt-5.6-luna 6.2anthropic/claude-haiku-5.5 high
132 Granite3.3 8B 62.9 35 / 35 6.5openai/gpt-5.6-luna 6.2anthropic/claude-haiku-5.5 high
132 Granite4 3B 62.9 35 / 35 6.6openai/gpt-5.6-luna 6.1anthropic/claude-haiku-5.5 high
132 Granite4 Micro H 62.9 35 / 35 6.9openai/gpt-5.6-luna 6.2anthropic/claude-haiku-5.5 high
132 Lfm3 VL 2B 62.9 35 / 35 6.6openai/gpt-5.6-luna 6.1anthropic/claude-haiku-5.5 high
132 Olmo-3-7B-Think 62.9 35 / 35 6.1openai/gpt-5.6-luna 5.4anthropic/claude-haiku-5.5 high
132 Qwen2.5 1.5B Instruct 62.9 35 / 35 6.3openai/gpt-5.6-luna 5.7anthropic/claude-haiku-5.5 high
138 Gemma3 1B IT 60.0 35 / 35 5.7openai/gpt-5.6-luna 5.3anthropic/claude-haiku-5.5 high
138 Glm4 9B Chat 60.0 35 / 35 6.5openai/gpt-5.6-luna 5.9anthropic/claude-haiku-5.5 high
138 Llama-3.1-8B-Instruct 60.0 35 / 35 6.6openai/gpt-5.6-luna 5.9anthropic/claude-haiku-5.5 high
138 Olmo2 13B 1124 Instruct 60.0 35 / 35 6.5openai/gpt-5.6-luna 6.0anthropic/claude-haiku-5.5 high
138 Phi-3.5-mini-instruct 60.0 35 / 35 6.2openai/gpt-5.6-luna 5.6anthropic/claude-haiku-5.5 high
138 Qwen3 VL 2B Thinking 60.0 35 / 35 5.9openai/gpt-5.6-luna 5.4anthropic/claude-haiku-5.5 high
138 Tulu3 8Bfine-tune of Llama 3.1 60.0 35 / 35 6.1openai/gpt-5.6-luna 5.5anthropic/claude-haiku-5.5 high
145 Granite3.1 Dense 8B Instruct 57.1 35 / 35 6.0openai/gpt-5.6-luna 5.6anthropic/claude-haiku-5.5 high
145 Granite3.3 2B 57.1 35 / 35 6.4openai/gpt-5.6-luna 5.9anthropic/claude-haiku-5.5 high
145 Mistral Nemo 12B Instruct 2407 57.1 35 / 35 6.3openai/gpt-5.6-luna 5.7anthropic/claude-haiku-5.5 high
145 Qwen2.5-Coder-3B-Instructcode model 57.1 35 / 35 5.9openai/gpt-5.6-luna 5.7anthropic/claude-haiku-5.5 high
145 Smallthinker 3B Previewfine-tune of Qwen 2.5 3B 57.1 35 / 35 5.5openai/gpt-5.6-luna 5.0anthropic/claude-haiku-5.5 high
150 Aya Expanse 32B 54.3 35 / 35 6.1openai/gpt-5.6-luna 5.8anthropic/claude-haiku-5.5 high
150 Aya Expanse 8B 54.3 35 / 35 5.7openai/gpt-5.6-luna 5.4anthropic/claude-haiku-5.5 high
150 Granite3 Dense 8B Instruct 54.3 35 / 35 5.7openai/gpt-5.6-luna 5.4anthropic/claude-haiku-5.5 high
150 Hermes-3-Llama-3.1-8B 54.3 35 / 35 5.6openai/gpt-5.6-luna 5.5anthropic/claude-haiku-5.5 high
150 Phi3 3.8B Mini 128K Instruct 54.3 35 / 35 6.1openai/gpt-5.6-luna 5.3anthropic/claude-haiku-5.5 high
150 Sailor2 8B Chatfine-tune of Qwen 2.5 54.3 35 / 35 5.8openai/gpt-5.6-luna 5.5anthropic/claude-haiku-5.5 high
156 Falcon3 3B Instruct 51.4 35 / 35 5.7openai/gpt-5.6-luna 5.1anthropic/claude-haiku-5.5 high
156 Meta-Llama-3-8B-Instruct 51.4 35 / 35 6.0openai/gpt-5.6-luna 5.1anthropic/claude-haiku-5.5 high
156 Olmo2 7B 1124 Instruct 51.4 35 / 35 5.9openai/gpt-5.6-luna 5.1anthropic/claude-haiku-5.5 high
159 Codegeex4 9B Allcode model 48.6 35 / 35 5.1openai/gpt-5.6-luna 4.8anthropic/claude-haiku-5.5 high
159 Command R 35B 08 2024 48.6 35 / 35 5.4openai/gpt-5.6-luna 4.8anthropic/claude-haiku-5.5 high
159 Granite3 Dense 2B Instruct 48.6 35 / 35 5.5openai/gpt-5.6-luna 4.7anthropic/claude-haiku-5.5 high
159 Granite4 1B H 48.6 35 / 35 4.7openai/gpt-5.6-luna 4.1anthropic/claude-haiku-5.5 high
159 Qwen 14B Chat V1.5 48.6 35 / 35 5.4openai/gpt-5.6-luna 4.8anthropic/claude-haiku-5.5 high
164 Codegemma 7B Instruct V1.1code model 45.7 35 / 35 5.3openai/gpt-5.6-luna 4.9anthropic/claude-haiku-5.5 high
164 Codestral 22B V0.1code model 45.7 35 / 35 5.3openai/gpt-5.6-luna 4.9anthropic/claude-haiku-5.5 high
164 Cogito 3B V1 Preview Llamafine-tune of Llama 3.2 45.7 35 / 35 5.4openai/gpt-5.6-luna 5.0anthropic/claude-haiku-5.5 high
164 Granite3.1 Dense 2B Instruct 45.7 35 / 35 5.1openai/gpt-5.6-luna 4.6anthropic/claude-haiku-5.5 high
164 Granite3.2 8B Instruct 45.7 35 / 35 5.5openai/gpt-5.6-luna 4.9anthropic/claude-haiku-5.5 high
164 Llama2 13B Chat 45.7 35 / 35 4.8openai/gpt-5.6-luna 4.3anthropic/claude-haiku-5.5 high
164 Qwen 7B Chat V1.5 45.7 35 / 35 5.0openai/gpt-5.6-luna 4.4anthropic/claude-haiku-5.5 high
164 Qwen2-VL-7B-Instruct 45.7 35 / 35 4.7openai/gpt-5.6-luna 4.2anthropic/claude-haiku-5.5 high
164 Solar 10.7B Instruct V1 45.7 35 / 35 5.0openai/gpt-5.6-luna 4.5anthropic/claude-haiku-5.5 high
164 Stablelm Zephyr 3Bfine-tune of StableLM 3B 45.7 35 / 35 4.4openai/gpt-5.6-luna 3.9anthropic/claude-haiku-5.5 high
164 YI 6B Chat V1.5 45.7 35 / 35 4.8openai/gpt-5.6-luna 4.3anthropic/claude-haiku-5.5 high
175 Apertus-8B-Instruct-2509 42.9 35 / 35 5.1openai/gpt-5.6-luna 4.6anthropic/claude-haiku-5.5 high
175 Codellama 34B Instructcode model 42.9 35 / 35 4.7openai/gpt-5.6-luna 3.8anthropic/claude-haiku-5.5 high
175 Gemma2 2B Instruct 42.9 35 / 35 5.1openai/gpt-5.6-luna 4.3anthropic/claude-haiku-5.5 high
175 Granite3.2 2B Instruct 42.9 35 / 35 5.0openai/gpt-5.6-luna 4.3anthropic/claude-haiku-5.5 high
175 Mixtral 8x7B Instruct V0.1 42.9 35 / 35 4.9openai/gpt-5.6-luna 4.5anthropic/claude-haiku-5.5 high
180 Falcon3 1B Instruct 40.0 35 / 35 4.3openai/gpt-5.6-luna 4.1anthropic/claude-haiku-5.5 high
180 Gemma 7B Instruct V1.1 40.0 35 / 35 4.8openai/gpt-5.6-luna 4.3anthropic/claude-haiku-5.5 high
180 Granite Code 8B Instructcode model 40.0 35 / 35 4.5openai/gpt-5.6-luna 4.1anthropic/claude-haiku-5.5 high
180 Mistral-7B-Instruct-v0.2 40.0 35 / 35 4.3openai/gpt-5.6-luna 4.1anthropic/claude-haiku-5.5 high
184 Codellama 7B Instructcode model 37.1 35 / 35 4.2openai/gpt-5.6-luna 3.7anthropic/claude-haiku-5.5 high
184 Deepseek Llm 7B Chat 37.1 35 / 35 4.5openai/gpt-5.6-luna 4.1anthropic/claude-haiku-5.5 high
184 Granite Code 3B Instructcode model 37.1 35 / 35 3.9openai/gpt-5.6-luna 3.4anthropic/claude-haiku-5.5 high
184 Granite3 Moe 3B Instruct 37.1 35 / 35 4.4openai/gpt-5.6-luna 3.6anthropic/claude-haiku-5.5 high
184 Granite3.1 Moe 3B Instruct 37.1 35 / 35 4.5openai/gpt-5.6-luna 3.9anthropic/claude-haiku-5.5 high
184 Hermes3 3B Llama3.2fine-tune of Llama 3.2 37.1 35 / 35 4.4openai/gpt-5.6-luna 3.9anthropic/claude-haiku-5.5 high
184 Llama3.2 1B Instruct 37.1 35 / 35 4.1openai/gpt-5.6-luna 3.9anthropic/claude-haiku-5.5 high
184 Mistral 7B Instruct V0.3 37.1 35 / 35 3.9openai/gpt-5.6-luna 3.5anthropic/claude-haiku-5.5 high
184 Qwen3 0.6B 37.1 35 / 35 4.2openai/gpt-5.6-luna 3.9anthropic/claude-haiku-5.5 high
193 Aya 8B 23 34.3 35 / 35 3.5openai/gpt-5.6-luna 2.8anthropic/claude-haiku-5.5 high
193 Deepseek V2 16B Lite Chat 34.3 35 / 35 4.3openai/gpt-5.6-luna 4.0anthropic/claude-haiku-5.5 high
193 Magistral Small (2506) 34.3 35 / 35 2.9openai/gpt-5.6-luna 3.0anthropic/claude-haiku-5.5 high
193 Nemotron Mini 4B Instruct 34.3 35 / 35 3.7openai/gpt-5.6-luna 3.3anthropic/claude-haiku-5.5 high
193 Phi 2.7B Chat V2 34.3 35 / 35 3.1openai/gpt-5.6-luna 2.7anthropic/claude-haiku-5.5 high
198 Codellama 13B Instructcode model 31.4 35 / 35 3.9openai/gpt-5.6-luna 3.4anthropic/claude-haiku-5.5 high
198 Deepseek Coder 33B Instructcode model 31.4 35 / 35 3.6openai/gpt-5.6-luna 3.1anthropic/claude-haiku-5.5 high
198 Dolphin Phi 2.7B V2.6fine-tune of Phi-2 31.4 35 / 35 4.2openai/gpt-5.6-luna 3.7anthropic/claude-haiku-5.5 high
198 Granite3.1 Moe 1B Instruct 31.4 35 / 35 3.5openai/gpt-5.6-luna 3.2anthropic/claude-haiku-5.5 high
198 Internlm2 1.8B Chat V2.5 31.4 35 / 35 3.3openai/gpt-5.6-luna 3.1anthropic/claude-haiku-5.5 high
198 Qwen2 0.5B Instruct 31.4 35 / 35 3.4openai/gpt-5.6-luna 3.1anthropic/claude-haiku-5.5 high
198 YI Coder 9B Chatcode model 31.4 35 / 35 3.8openai/gpt-5.6-luna 3.6anthropic/claude-haiku-5.5 high
205 Qwen2 1.5B Instruct 28.6 35 / 35 3.4openai/gpt-5.6-luna 3.3anthropic/claude-haiku-5.5 high
205 Qwen2.5 0.5B Instruct 28.6 35 / 35 3.6openai/gpt-5.6-luna 3.3anthropic/claude-haiku-5.5 high
205 Smollm2 1.7B Instruct 28.6 35 / 35 2.8openai/gpt-5.6-luna 2.4anthropic/claude-haiku-5.5 high
208 Codeqwen 7B Chat V1.5code model 25.7 35 / 35 3.1openai/gpt-5.6-luna 2.9anthropic/claude-haiku-5.5 high
208 Gemma 2B Instruct V1.1 25.7 35 / 35 2.9openai/gpt-5.6-luna 2.4anthropic/claude-haiku-5.5 high
208 Llama-2-7b-chat-hf 25.7 35 / 35 3.6openai/gpt-5.6-luna 3.0anthropic/claude-haiku-5.5 high
211 Falcon2 11Bnot in the Overall ranking 22.9 35 / 35 2.9openai/gpt-5.6-luna 2.2anthropic/claude-haiku-5.5 high
211 Qwen2.5 Coder 0.5B Instructcode model 22.9 35 / 35 2.7openai/gpt-5.6-luna 2.1anthropic/claude-haiku-5.5 high
211 Stable Code 3B Instructcode model 22.9 35 / 35 2.9openai/gpt-5.6-luna 2.3anthropic/claude-haiku-5.5 high
211 Stablelm2 12B Chat 22.9 35 / 35 3.2openai/gpt-5.6-luna 2.8anthropic/claude-haiku-5.5 high
215 Granite4 350M H 20.0 35 / 35 2.1openai/gpt-5.6-luna 1.9anthropic/claude-haiku-5.5 high
215 Qwen 1.8B Chat V1.5 20.0 35 / 35 2.3openai/gpt-5.6-luna 2.3anthropic/claude-haiku-5.5 high
217 Codegemma 2B Code V1.1code model 17.1 35 / 35 2.1openai/gpt-5.6-luna 1.0anthropic/claude-haiku-5.5 high
217 Granite3 Moe 1B Instruct 17.1 35 / 35 2.4openai/gpt-5.6-luna 2.2anthropic/claude-haiku-5.5 high
217 YI Coder 1.5B Chatcode model 17.1 35 / 35 2.6openai/gpt-5.6-luna 2.1anthropic/claude-haiku-5.5 high
220 Qwen 0.5B Chat V1.5 14.3 35 / 35 2.0openai/gpt-5.6-luna 1.4anthropic/claude-haiku-5.5 high
221 Qwen 4B Chat V1.5 11.4 35 / 35 1.8openai/gpt-5.6-luna 1.3anthropic/claude-haiku-5.5 high
222 Deepseek Coder 6.7B Instructcode model 8.6 35 / 35 2.3openai/gpt-5.6-luna 1.6anthropic/claude-haiku-5.5 high
222 Sailor2 1B Chatfine-tune of Qwen 2.5 8.6 35 / 35 1.9openai/gpt-5.6-luna 1.6anthropic/claude-haiku-5.5 high
222 Smollm2 360M Instruct 8.6 35 / 35 1.5openai/gpt-5.6-luna 1.3anthropic/claude-haiku-5.5 high
222 Stablelm2 1.6B Chat 8.6 35 / 35 1.9openai/gpt-5.6-luna 1.6anthropic/claude-haiku-5.5 high
226 Deepseek Coder 1.3B Instructcode model 5.7 35 / 35 1.7openai/gpt-5.6-luna 1.1anthropic/claude-haiku-5.5 high
227 Opencoder 1.5B Instructcode model 2.9 35 / 35 1.7openai/gpt-5.6-luna 1.5anthropic/claude-haiku-5.5 high
227 Smollm2 135M Instruct 2.9 35 / 35 1.1openai/gpt-5.6-luna 0.7anthropic/claude-haiku-5.5 high

A dash means not scored, which is different from 0. Agreement compares the two judges: high means a mean gap under 1.5 points, medium under 3, low otherwise; n/a means fewer than 3 items were scored by both.

Not ranked in this category

The questions

In run r20261102-bv1-5246ab, 5 of the 35 questions in this category were answered correctly by every ranked model that has a score for them (33 ranked models in that run). Hard questions are the ones that half or fewer of the ranked models of 3B and over answered correctly. This category has 7.

The same ranking as JSON: reasoning.json.