All categories

Instruction following

Writing tasks with explicit rules, such as length, format, required words or the language of the reply. Code checks each rule.

This category counts toward Overall. The two judge models also score every item, and their scores are shown here separately. How the battery works.

Ranking

Models are ranked by their mean score in this category, from 0 to 100. Only models with at least 80% of the category's items scored, and none pending, are ranked here. The rule is the same as for Overall.

Instruction following: 228 ranked models. Click a column heading to sort.
Rank Model Mean (0 to 100) Items scored Judge A mean Judge B mean Agreement
1 Gemma 4 12B IT 100.0 30 / 30 9.5openai/gpt-5.6-luna 9.0anthropic/claude-haiku-5.5 high
1 gemma-4-26B-A4B-it 100.0 30 / 30 9.3openai/gpt-5.6-luna 9.0anthropic/claude-haiku-5.5 high
1 gemma-4-31B-it 100.0 30 / 30 9.5openai/gpt-5.6-luna 8.9anthropic/claude-haiku-5.5 high
1 gpt-oss-120b 100.0 30 / 30 9.0anthropic/claude-haiku-5.5 8.9deepseek/deepseek-v4.1-flash high
1 Muse Glimmer 30B 100.0 30 / 30 9.5openai/gpt-5.6-luna 8.9anthropic/claude-haiku-5.5 high
1 Nemotron 3.5 Bolt E4B 100.0 30 / 30 9.2openai/gpt-5.6-luna 8.9anthropic/claude-haiku-5.5 high
1 Qwen3.5-27B 100.0 30 / 30 9.4openai/gpt-5.6-luna 8.9anthropic/claude-haiku-5.5 high
8 Qwen3.8-27B 98.9 30 / 30 9.3openai/gpt-5.6-luna 8.7anthropic/claude-haiku-5.5 high
9 Llama 4 Scout 17B-16E Instruct 98.7 30 / 30 9.2openai/gpt-5.6-luna 8.7anthropic/claude-haiku-5.5 high
10 Qwen3-VL-8B-Instruct 98.5 30 / 30 9.4openai/gpt-5.6-luna 8.7anthropic/claude-haiku-5.5 high
11 Olmo-3-7B-Think 98.3 30 / 30 9.1openai/gpt-5.6-luna 8.7anthropic/claude-haiku-5.5 high
11 Qwen3.6-27B 98.3 30 / 30 9.2openai/gpt-5.6-luna 8.8anthropic/claude-haiku-5.5 high
13 Gemma2 27B Instruct 98.2 30 / 30 9.2openai/gpt-5.6-luna 8.6anthropic/claude-haiku-5.5 high
14 Nemotron Cascade 2 30B A3B 97.8 30 / 30 9.2openai/gpt-5.6-luna 8.5anthropic/claude-haiku-5.5 high
15 Llama-3.3-70B-Instruct 97.5 30 / 30 8.9openai/gpt-5.6-luna 8.3anthropic/claude-haiku-5.5 high
16 Gemma4 E4B IT 97.2 30 / 30 9.1openai/gpt-5.6-luna 8.7anthropic/claude-haiku-5.5 high
17 Olmo-3-7B-Instruct 96.7 30 / 30 9.1openai/gpt-5.6-luna 8.4anthropic/claude-haiku-5.5 high
18 Flint 40B A2B 96.6 30 / 30 9.1openai/gpt-5.6-luna 8.4anthropic/claude-haiku-5.5 high
18 Lfm2 24B 96.6 30 / 30 8.9openai/gpt-5.6-luna 8.3anthropic/claude-haiku-5.5 high
18 Qwen3-VL-4B-Instruct 96.6 30 / 30 9.0openai/gpt-5.6-luna 8.3anthropic/claude-haiku-5.5 high
18 Qwen3.6-35B-A3B 96.6 30 / 30 9.1openai/gpt-5.6-luna 8.3anthropic/claude-haiku-5.5 high
22 NVIDIA Nemotron 3 Nano 30B-A3B 96.5 30 / 30 9.1openai/gpt-5.6-luna 8.3anthropic/claude-haiku-5.5 high
23 Qwen3-32B 96.3 30 / 30 9.1openai/gpt-5.6-luna 8.1anthropic/claude-haiku-5.5 high
24 Qwen3-VL-32B-Instruct 96.1 30 / 30 8.9openai/gpt-5.6-luna 8.1anthropic/claude-haiku-5.5 high
25 gpt-oss-20b 95.8 30 / 30 8.3anthropic/claude-haiku-5.5 8.0deepseek/deepseek-v4.1-flash high
26 Gemma4 E2B IT 95.4 30 / 30 8.8openai/gpt-5.6-luna 8.3anthropic/claude-haiku-5.5 high
27 Olmo 3.1 32B Instruct 95.2 30 / 30 9.0openai/gpt-5.6-luna 8.2anthropic/claude-haiku-5.5 high
28 Qwen3-Coder-Nextcode model 95.2 30 / 30 9.1openai/gpt-5.6-luna 8.0anthropic/claude-haiku-5.5 high
29 QwQ 32B 95.1 30 / 30 9.1openai/gpt-5.6-luna 8.0anthropic/claude-haiku-5.5 high
30 Qwen2.5-Coder-32B-Instructcode model 94.8 30 / 30 8.8openai/gpt-5.6-luna 8.2anthropic/claude-haiku-5.5 high
31 Sage-A1.5-Q4_K_M 94.6 30 / 30 8.7openai/gpt-5.6-luna 7.9anthropic/claude-haiku-5.5 high
32 gemma-3-27b-it 94.5 30 / 30 8.8openai/gpt-5.6-luna 7.9anthropic/claude-haiku-5.5 high
33 GLM-4.5-Air 94.3 30 / 30 8.8openai/gpt-5.6-luna 8.1anthropic/claude-haiku-5.5 high
34 Qwen3-4B-Instruct-2507 94.2 30 / 30 8.9openai/gpt-5.6-luna 8.2anthropic/claude-haiku-5.5 high
35 Qwen2.5-32B-Instruct 94.1 30 / 30 8.8openai/gpt-5.6-luna 7.9anthropic/claude-haiku-5.5 high
36 Granite4.2 8B 94.1 30 / 30 8.7openai/gpt-5.6-luna 7.9anthropic/claude-haiku-5.5 high
36 Qwen3.5-9B 94.1 30 / 30 8.7openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 high
38 granite-4.1-3b 93.9 30 / 30 8.5openai/gpt-5.6-luna 7.7anthropic/claude-haiku-5.5 high
39 Granite4.1 30B 93.8 30 / 30 8.6openai/gpt-5.6-luna 7.9anthropic/claude-haiku-5.5 high
40 Nemotron 3.5 Lightning 30B A3B 93.7 30 / 30 8.7openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 high
41 Qwen3-14B 93.6 30 / 30 8.6openai/gpt-5.6-luna 8.1anthropic/claude-haiku-5.5 high
42 Qwen3.5-35B-A3B 93.5 30 / 30 8.8openai/gpt-5.6-luna 7.9anthropic/claude-haiku-5.5 high
43 Granite4 Small H 93.4 30 / 30 8.6openai/gpt-5.6-luna 7.9anthropic/claude-haiku-5.5 high
44 Laguna XS 2.1 93.3 30 / 30 8.6openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 high
45 Cogito 8B V1 Preview Llamafine-tune of Llama 3.1 93.1 30 / 30 8.6openai/gpt-5.6-luna 7.6anthropic/claude-haiku-5.5 high
45 Granite4.1 8B 93.1 30 / 30 8.8openai/gpt-5.6-luna 7.9anthropic/claude-haiku-5.5 high
47 Qwen3-Coder-30B-A3B-Instructcode model 92.9 30 / 30 8.8openai/gpt-5.6-luna 7.6anthropic/claude-haiku-5.5 high
48 Granite4.2 30B 92.8 30 / 30 8.7openai/gpt-5.6-luna 7.6anthropic/claude-haiku-5.5 high
49 gemma-2-9b-it 92.6 30 / 30 8.5openai/gpt-5.6-luna 7.6anthropic/claude-haiku-5.5 high
49 Llama-3.1-8B-Instruct 92.6 30 / 30 8.4openai/gpt-5.6-luna 7.9anthropic/claude-haiku-5.5 high
49 Nemotron3 33B 92.6 30 / 30 8.6openai/gpt-5.6-luna 7.5anthropic/claude-haiku-5.5 high
52 Qwen3-30B-A3B 92.3 30 / 30 8.6openai/gpt-5.6-luna 7.7anthropic/claude-haiku-5.5 high
53 Ornith 9B 91.9 30 / 30 8.6openai/gpt-5.6-luna 8.0anthropic/claude-haiku-5.5 high
54 Llama-3.2-3B-Instruct 91.7 30 / 30 8.3openai/gpt-5.6-luna 7.3anthropic/claude-haiku-5.5 high
55 Qwen2.5-14B-Instruct 91.3 30 / 30 8.4openai/gpt-5.6-luna 7.4anthropic/claude-haiku-5.5 high
56 Granite4.2 3B 91.2 30 / 30 8.4openai/gpt-5.6-luna 7.5anthropic/claude-haiku-5.5 high
57 Lfm2.5 8B A1B 91.1 30 / 30 7.2openai/gpt-5.6-luna 6.3anthropic/claude-haiku-5.5 high
58 Qwen3-4B 91.0 30 / 30 8.4openai/gpt-5.6-luna 7.2anthropic/claude-haiku-5.5 high
59 Qwen3.5-4B 90.9 30 / 30 8.4openai/gpt-5.6-luna 7.2anthropic/claude-haiku-5.5 high
60 Qwen3-30B-A3B-Instruct-2507 90.3 30 / 30 8.4openai/gpt-5.6-luna 7.3anthropic/claude-haiku-5.5 high
61 Granite4 Micro H 90.0 30 / 30 8.1openai/gpt-5.6-luna 7.4anthropic/claude-haiku-5.5 high
62 Qwen3 VL 2B Instruct 89.8 30 / 30 8.2openai/gpt-5.6-luna 7.4anthropic/claude-haiku-5.5 high
63 Exaone3.5 32B Instruct 89.8 30 / 30 8.2openai/gpt-5.6-luna 7.4anthropic/claude-haiku-5.5 high
64 Gemma3n E2B IT 89.7 30 / 30 8.3openai/gpt-5.6-luna 6.8anthropic/claude-haiku-5.5 medium
65 Medgemma 27B ITfine-tune of Gemma 3medical model 89.5 30 / 30 8.2openai/gpt-5.6-luna 7.2anthropic/claude-haiku-5.5 high
66 Cogito 14B V1 Preview Qwenfine-tune of Qwen 2.5 89.4 30 / 30 8.4openai/gpt-5.6-luna 7.0anthropic/claude-haiku-5.5 high
67 Meta-Llama-3-8B-Instruct 89.3 30 / 30 8.1openai/gpt-5.6-luna 7.0anthropic/claude-haiku-5.5 high
68 Devstral Small 2 24B Instruct (2512)code model 89.1 30 / 30 8.3openai/gpt-5.6-luna 7.2anthropic/claude-haiku-5.5 high
69 Qwen3-4B-Thinking-2507 88.6 30 / 30 7.7openai/gpt-5.6-luna 7.2anthropic/claude-haiku-5.5 high
70 Olmo2 13B 1124 Instruct 88.6 30 / 30 8.0openai/gpt-5.6-luna 6.7anthropic/claude-haiku-5.5 high
71 Qwen2.5-7B-Instruct 88.4 30 / 30 8.0openai/gpt-5.6-luna 6.9anthropic/claude-haiku-5.5 high
72 Qwen3-VL-30B-A3B-Instruct 88.4 30 / 30 8.4openai/gpt-5.6-luna 7.3anthropic/claude-haiku-5.5 high
73 gemma-3-12b-it 88.3 30 / 30 8.1openai/gpt-5.6-luna 6.9anthropic/claude-haiku-5.5 high
74 Lfm3 VL 2B 88.1 30 / 30 8.2openai/gpt-5.6-luna 7.0anthropic/claude-haiku-5.5 high
75 Qwen3-8B 87.7 30 / 30 8.0openai/gpt-5.6-luna 6.9anthropic/claude-haiku-5.5 high
76 Granite4 3B 87.3 30 / 30 8.1openai/gpt-5.6-luna 7.2anthropic/claude-haiku-5.5 high
77 Deepcoder 14B Previewfine-tune of DeepSeek-R1-Distill-Qwen 14Bcode model 86.8 30 / 30 7.9openai/gpt-5.6-luna 6.8anthropic/claude-haiku-5.5 high
78 Qwen3 1.7B 86.8 30 / 30 7.9openai/gpt-5.6-luna 6.7anthropic/claude-haiku-5.5 high
79 GLM-4.7-Flash 86.7 30 / 30 8.0openai/gpt-5.6-luna 6.9anthropic/claude-haiku-5.5 high
80 Translategemma 12B ITfine-tune of Gemma 3translation model 86.3 30 / 30 7.9openai/gpt-5.6-luna 7.1anthropic/claude-haiku-5.5 high
81 SmolLM3-3B 86.0 30 / 30 7.9openai/gpt-5.6-luna 6.7anthropic/claude-haiku-5.5 high
82 gemma-3-4b-it 85.9 30 / 30 7.5openai/gpt-5.6-luna 6.3anthropic/claude-haiku-5.5 high
83 Granite3.3 8B 85.7 30 / 30 7.9openai/gpt-5.6-luna 6.8anthropic/claude-haiku-5.5 high
84 Mistral Small 3.2 24B Instruct (2506) 85.4 30 / 30 8.0openai/gpt-5.6-luna 6.7anthropic/claude-haiku-5.5 high
85 Ornith-1.5-9B 85.2 30 / 30 7.9openai/gpt-5.6-luna 6.6anthropic/claude-haiku-5.5 high
86 Granite4 1B H 85.1 30 / 30 7.7openai/gpt-5.6-luna 6.3anthropic/claude-haiku-5.5 medium
87 Command R7B Arabic 7B 02 2025 84.8 30 / 30 7.8openai/gpt-5.6-luna 6.4anthropic/claude-haiku-5.5 high
87 Mistral Small 24B Instruct 2501 84.8 30 / 30 7.9openai/gpt-5.6-luna 6.9anthropic/claude-haiku-5.5 high
89 Falcon3 10B Instruct 84.8 30 / 30 7.6openai/gpt-5.6-luna 6.5anthropic/claude-haiku-5.5 high
90 Gemma2 2B Instruct 84.6 30 / 30 7.8openai/gpt-5.6-luna 6.6anthropic/claude-haiku-5.5 high
91 Olmo2 7B 1124 Instruct 84.4 30 / 30 7.6openai/gpt-5.6-luna 6.4anthropic/claude-haiku-5.5 high
92 DeepSeek-R1-0528-Qwen3-8B 84.3 30 / 30 8.0openai/gpt-5.6-luna 6.4anthropic/claude-haiku-5.5 medium
93 Devstral 24B Small 2505code model 84.2 30 / 30 7.8openai/gpt-5.6-luna 6.4anthropic/claude-haiku-5.5 medium
93 Mistral Small3.1 24B Instruct 2503 84.2 30 / 30 7.8openai/gpt-5.6-luna 6.7anthropic/claude-haiku-5.5 high
93 Qwen2.5-Coder-14B-Instructcode model 84.2 30 / 30 7.9openai/gpt-5.6-luna 6.7anthropic/claude-haiku-5.5 high
96 Phi-4-mini-instruct 84.1 30 / 30 7.5openai/gpt-5.6-luna 6.2anthropic/claude-haiku-5.5 high
97 Command R 35B 08 2024 84.1 30 / 30 7.5openai/gpt-5.6-luna 6.4anthropic/claude-haiku-5.5 high
98 Tulu3 8Bfine-tune of Llama 3.1 83.6 30 / 30 7.6openai/gpt-5.6-luna 6.3anthropic/claude-haiku-5.5 high
99 Gemma 3n E4B IT 83.4 30 / 30 7.6openai/gpt-5.6-luna 6.2anthropic/claude-haiku-5.5 high
100 Command R7B (12-2024) 82.9 30 / 30 7.6openai/gpt-5.6-luna 6.5anthropic/claude-haiku-5.5 high
101 Granite3.2 8B Instruct 82.7 30 / 30 7.7openai/gpt-5.6-luna 6.4anthropic/claude-haiku-5.5 high
101 Qwen2.5-3B-Instruct 82.7 30 / 30 7.5openai/gpt-5.6-luna 6.3anthropic/claude-haiku-5.5 high
103 Glm4 9B Chat 82.3 30 / 30 7.7openai/gpt-5.6-luna 6.5anthropic/claude-haiku-5.5 high
104 Deepseek R1 8B Llama Distill 82.3 30 / 30 7.6openai/gpt-5.6-luna 6.5anthropic/claude-haiku-5.5 high
105 Qwen3.5 2B 82.1 30 / 30 7.4openai/gpt-5.6-luna 6.0anthropic/claude-haiku-5.5 high
106 Cogito 3B V1 Preview Llamafine-tune of Llama 3.2 81.6 30 / 30 7.4openai/gpt-5.6-luna 5.9anthropic/claude-haiku-5.5 high
107 Aya Expanse 32B 81.3 30 / 30 7.4openai/gpt-5.6-luna 5.8anthropic/claude-haiku-5.5 medium
108 Ministral 3 14B Instruct (2512) 81.2 30 / 30 7.4openai/gpt-5.6-luna 5.9anthropic/claude-haiku-5.5 medium
109 DeepSeek-R1-Distill-Qwen-14B 80.8 30 / 30 7.6openai/gpt-5.6-luna 6.3anthropic/claude-haiku-5.5 high
109 EXAONE-3.5-7.8B-Instruct 80.8 30 / 30 7.3openai/gpt-5.6-luna 5.9anthropic/claude-haiku-5.5 medium
109 Lfm2.5 Thinking 1.2B 80.8 30 / 30 6.7openai/gpt-5.6-luna 5.7anthropic/claude-haiku-5.5 high
112 Mistral Nemo 12B Instruct 2407 80.7 30 / 30 7.2openai/gpt-5.6-luna 5.7anthropic/claude-haiku-5.5 high
112 Nanbeige_Nanbeige4.3-33B 80.7 30 / 30 7.9openai/gpt-5.6-luna 6.6anthropic/claude-haiku-5.5 high
114 Falcon3 7B Instruct 80.2 30 / 30 7.2openai/gpt-5.6-luna 5.9anthropic/claude-haiku-5.5 high
115 Granite4 Tiny H 79.7 30 / 30 7.3openai/gpt-5.6-luna 5.6anthropic/claude-haiku-5.5 medium
116 Aya Expanse 8B 79.5 30 / 30 7.2openai/gpt-5.6-luna 5.7anthropic/claude-haiku-5.5 medium
116 Smollm2 1.7B Instruct 79.5 30 / 30 7.2openai/gpt-5.6-luna 5.5anthropic/claude-haiku-5.5 medium
118 Granite4 350M H 79.3 30 / 30 6.6openai/gpt-5.6-luna 4.8anthropic/claude-haiku-5.5 medium
119 Mistral Small 22B Instruct 2409 78.9 30 / 30 7.2openai/gpt-5.6-luna 5.9anthropic/claude-haiku-5.5 medium
120 Qwen2.5-VL-7B-Instruct 78.8 30 / 30 7.3openai/gpt-5.6-luna 6.3anthropic/claude-haiku-5.5 high
121 Spark-X2.6-4B 78.7 30 / 30 7.5openai/gpt-5.6-luna 6.4anthropic/claude-haiku-5.5 high
122 medgemma-4b-itmedical model 78.6 30 / 30 7.1openai/gpt-5.6-luna 5.6anthropic/claude-haiku-5.5 medium
123 Codegemma 7B Instruct V1.1code model 78.1 30 / 30 7.1openai/gpt-5.6-luna 6.0anthropic/claude-haiku-5.5 high
124 DeepSeek-Coder-V2-Lite-Instructcode model 77.9 30 / 30 7.0openai/gpt-5.6-luna 5.6anthropic/claude-haiku-5.5 high
125 Lfm3 1.7B 77.9 30 / 30 7.0openai/gpt-5.6-luna 5.6anthropic/claude-haiku-5.5 high
126 Ministral 3 8B Instruct 2512 77.8 30 / 30 7.1openai/gpt-5.6-luna 5.7anthropic/claude-haiku-5.5 high
127 DeepSeek-R1-Distill-Qwen-7B 77.4 30 / 30 7.1openai/gpt-5.6-luna 5.9anthropic/claude-haiku-5.5 high
128 Qwen3 VL 2B Thinking 77.2 30 / 30 6.6openai/gpt-5.6-luna 5.6anthropic/claude-haiku-5.5 high
129 Llama3.2 1B Instruct 76.9 30 / 30 7.0openai/gpt-5.6-luna 5.5anthropic/claude-haiku-5.5 medium
129 Qwen 14B Chat V1.5 76.9 30 / 30 7.0openai/gpt-5.6-luna 5.8anthropic/claude-haiku-5.5 high
131 Falcon3 3B Instruct 76.8 30 / 30 6.9openai/gpt-5.6-luna 5.5anthropic/claude-haiku-5.5 high
132 YI 34B Chat V1.5 76.6 30 / 30 7.1openai/gpt-5.6-luna 5.5anthropic/claude-haiku-5.5 medium
133 Granite3 Dense 8B Instruct 76.4 30 / 30 6.8openai/gpt-5.6-luna 5.2anthropic/claude-haiku-5.5 medium
133 Hermes3 3B Llama3.2fine-tune of Llama 3.2 76.4 30 / 30 6.9openai/gpt-5.6-luna 5.6anthropic/claude-haiku-5.5 high
135 Granite3.3 2B 76.3 30 / 30 7.0openai/gpt-5.6-luna 5.4anthropic/claude-haiku-5.5 medium
136 Apertus-8B-Instruct-2509 76.3 30 / 30 7.0openai/gpt-5.6-luna 5.6anthropic/claude-haiku-5.5 high
137 Gemma 7B Instruct V1.1 76.0 30 / 30 6.9openai/gpt-5.6-luna 5.5anthropic/claude-haiku-5.5 high
138 DeepSeek-R1-Distill-Qwen-32B 75.7 30 / 30 6.9openai/gpt-5.6-luna 5.4anthropic/claude-haiku-5.5 medium
139 Granite3.2 2B Instruct 75.5 30 / 30 6.9openai/gpt-5.6-luna 5.4anthropic/claude-haiku-5.5 medium
140 Codegeex4 9B Allcode model 75.2 30 / 30 6.7openai/gpt-5.6-luna 5.2anthropic/claude-haiku-5.5 medium
141 Codellama 34B Instructcode model 75.1 30 / 30 6.9openai/gpt-5.6-luna 5.4anthropic/claude-haiku-5.5 medium
141 Mistral-7B-Instruct-v0.2 75.1 30 / 30 6.8openai/gpt-5.6-luna 5.4anthropic/claude-haiku-5.5 medium
143 Qwen2.5-VL-3B-Instruct 74.7 30 / 30 6.9openai/gpt-5.6-luna 5.5anthropic/claude-haiku-5.5 high
144 Aya 8B 23 74.7 30 / 30 6.7openai/gpt-5.6-luna 5.5anthropic/claude-haiku-5.5 high
145 Codestral 22B V0.1code model 74.4 30 / 30 6.8openai/gpt-5.6-luna 5.1anthropic/claude-haiku-5.5 medium
146 Granite3.1 Dense 8B Instruct 74.3 30 / 30 7.0openai/gpt-5.6-luna 5.6anthropic/claude-haiku-5.5 high
146 Qwen3 0.6B 74.3 30 / 30 6.3openai/gpt-5.6-luna 4.8anthropic/claude-haiku-5.5 medium
148 Codellama 13B Instructcode model 73.8 30 / 30 6.7openai/gpt-5.6-luna 5.3anthropic/claude-haiku-5.5 high
148 Hermes-3-Llama-3.1-8B 73.8 30 / 30 6.7openai/gpt-5.6-luna 5.6anthropic/claude-haiku-5.5 high
150 YI 9B Chat V1.5 73.7 30 / 30 6.8openai/gpt-5.6-luna 5.2anthropic/claude-haiku-5.5 medium
151 Exaone3.5 2.4B Instruct 73.6 30 / 30 6.5openai/gpt-5.6-luna 4.7anthropic/claude-haiku-5.5 medium
151 Granite3.1 Dense 2B Instruct 73.6 30 / 30 6.7openai/gpt-5.6-luna 5.3anthropic/claude-haiku-5.5 high
153 phi-4 73.2 30 / 30 6.7openai/gpt-5.6-luna 5.3anthropic/claude-haiku-5.5 high
154 dolphin-2.9.1-yi-1.5-34b 72.7 30 / 30 6.7openai/gpt-5.6-luna 5.5anthropic/claude-haiku-5.5 high
155 Qwen2.5-Coder-7B-Instructcode model 72.6 30 / 30 6.8openai/gpt-5.6-luna 5.5anthropic/claude-haiku-5.5 high
156 Nemotron Mini 4B Instruct 72.4 30 / 30 6.2openai/gpt-5.6-luna 4.6anthropic/claude-haiku-5.5 medium
157 Codeqwen 7B Chat V1.5code model 72.1 30 / 30 6.4openai/gpt-5.6-luna 4.9anthropic/claude-haiku-5.5 medium
157 Llama-2-7b-chat-hf 72.1 30 / 30 6.4openai/gpt-5.6-luna 5.2anthropic/claude-haiku-5.5 high
159 Gemma3 1B IT 71.9 30 / 30 6.6openai/gpt-5.6-luna 4.8anthropic/claude-haiku-5.5 medium
160 Qwen 7B Chat V1.5 71.9 30 / 30 6.4openai/gpt-5.6-luna 4.5anthropic/claude-haiku-5.5 medium
161 Phi3 14B Medium 128K Instruct 71.6 30 / 30 6.4openai/gpt-5.6-luna 4.7anthropic/claude-haiku-5.5 medium
162 Phi-3-mini-4k-instruct 70.7 30 / 30 6.5openai/gpt-5.6-luna 5.1anthropic/claude-haiku-5.5 medium
162 Qwen2-VL-7B-Instruct 70.7 30 / 30 6.6openai/gpt-5.6-luna 5.2anthropic/claude-haiku-5.5 high
164 YI 6B Chat V1.5 70.4 30 / 30 6.6openai/gpt-5.6-luna 4.8anthropic/claude-haiku-5.5 medium
165 Mistral 7B Instruct V0.3 70.3 30 / 30 6.4openai/gpt-5.6-luna 5.0anthropic/claude-haiku-5.5 high
166 Deepseek V2 16B Lite Chat 70.2 30 / 30 6.5openai/gpt-5.6-luna 4.9anthropic/claude-haiku-5.5 medium
167 Qwen2.5-Coder-3B-Instructcode model 70.0 30 / 30 6.4openai/gpt-5.6-luna 4.8anthropic/claude-haiku-5.5 medium
168 Solar 10.7B Instruct V1 69.9 30 / 30 6.2openai/gpt-5.6-luna 4.7anthropic/claude-haiku-5.5 medium
169 Dolphin Phi 2.7B V2.6fine-tune of Phi-2 69.8 30 / 30 6.5openai/gpt-5.6-luna 4.9anthropic/claude-haiku-5.5 medium
170 Internlm2 7B Chat V2.5 69.6 30 / 30 6.1openai/gpt-5.6-luna 4.7anthropic/claude-haiku-5.5 medium
171 Qwen2.5-VL-32B-Instruct 69.5 30 / 30 6.3openai/gpt-5.6-luna 4.8anthropic/claude-haiku-5.5 high
172 Granite3.1 Moe 3B Instruct 69.2 30 / 30 6.3openai/gpt-5.6-luna 5.0anthropic/claude-haiku-5.5 medium
173 Qwen2 Math 7B Instructmath model 68.7 30 / 30 6.1openai/gpt-5.6-luna 4.1anthropic/claude-haiku-5.5 medium
174 Deepcoder 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bcode model 68.6 30 / 30 5.6openai/gpt-5.6-luna 3.8anthropic/claude-haiku-5.5 medium
175 Codellama 7B Instructcode model 68.1 30 / 30 6.1openai/gpt-5.6-luna 4.6anthropic/claude-haiku-5.5 medium
176 Ministral 3 3B Instruct 2512 67.5 30 / 30 6.1openai/gpt-5.6-luna 4.3anthropic/claude-haiku-5.5 medium
177 Granite3 Moe 3B Instruct 67.2 30 / 30 6.0openai/gpt-5.6-luna 4.3anthropic/claude-haiku-5.5 medium
178 Deepscaler 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bmath model 66.9 30 / 30 5.2openai/gpt-5.6-luna 4.0anthropic/claude-haiku-5.5 high
179 Deepseek Llm 7B Chat 66.8 30 / 30 6.1openai/gpt-5.6-luna 4.4anthropic/claude-haiku-5.5 medium
180 Gemma 2B Instruct V1.1 66.6 30 / 30 6.1openai/gpt-5.6-luna 4.5anthropic/claude-haiku-5.5 medium
181 Deepseek R1 1.5B Qwen Distill 66.2 30 / 30 5.0openai/gpt-5.6-luna 3.7anthropic/claude-haiku-5.5 high
182 YI Coder 9B Chatcode model 65.3 30 / 30 5.8openai/gpt-5.6-luna 4.7anthropic/claude-haiku-5.5 high
183 Falcon3 1B Instruct 64.9 30 / 30 5.9openai/gpt-5.6-luna 4.2anthropic/claude-haiku-5.5 medium
184 Granite3 Dense 2B Instruct 63.4 30 / 30 5.8openai/gpt-5.6-luna 4.2anthropic/claude-haiku-5.5 medium
185 Mixtral 8x7B Instruct V0.1 62.6 30 / 30 5.7openai/gpt-5.6-luna 4.3anthropic/claude-haiku-5.5 high
186 Openthinker 7B V2fine-tune of Qwen 2.5 62.4 30 / 30 5.6openai/gpt-5.6-luna 4.5anthropic/claude-haiku-5.5 high
187 Qwen2 1.5B Instruct 61.6 30 / 30 5.7openai/gpt-5.6-luna 3.8anthropic/claude-haiku-5.5 medium
188 Qwen 4B Chat V1.5 60.4 30 / 30 5.2openai/gpt-5.6-luna 3.6anthropic/claude-haiku-5.5 medium
189 Phi 2.7B Chat V2 60.3 30 / 30 5.4openai/gpt-5.6-luna 3.9anthropic/claude-haiku-5.5 medium
190 Phi-3.5-mini-instruct 60.2 30 / 30 5.3openai/gpt-5.6-luna 3.5anthropic/claude-haiku-5.5 medium
190 Phi3 3.8B Mini 128K Instruct 60.2 30 / 30 5.4openai/gpt-5.6-luna 3.6anthropic/claude-haiku-5.5 medium
192 Stablelm Zephyr 3Bfine-tune of StableLM 3B 60.1 30 / 30 5.4openai/gpt-5.6-luna 4.0anthropic/claude-haiku-5.5 high
193 Smollm2 360M Instruct 59.9 30 / 30 5.2openai/gpt-5.6-luna 3.9anthropic/claude-haiku-5.5 high
194 Qwen2.5 1.5B Instruct 59.4 30 / 30 5.5openai/gpt-5.6-luna 4.0anthropic/claude-haiku-5.5 medium
195 Granite3 Moe 1B Instruct 59.1 30 / 30 4.9openai/gpt-5.6-luna 3.6anthropic/claude-haiku-5.5 high
196 Granite3.1 Moe 1B Instruct 59.0 30 / 30 5.3openai/gpt-5.6-luna 3.4anthropic/claude-haiku-5.5 medium
197 Qwen2.5 Coder 1.5B Instructcode model 58.8 30 / 30 5.4openai/gpt-5.6-luna 3.4anthropic/claude-haiku-5.5 medium
198 Falcon2 11Bnot in the Overall ranking 57.7 30 / 30 4.8openai/gpt-5.6-luna 2.9anthropic/claude-haiku-5.5 medium
199 Internlm2 1.8B Chat V2.5 56.6 30 / 30 5.2openai/gpt-5.6-luna 3.6anthropic/claude-haiku-5.5 medium
200 Stable Code 3B Instructcode model 55.8 30 / 30 4.7openai/gpt-5.6-luna 3.1anthropic/claude-haiku-5.5 medium
201 Qwen2.5 0.5B Instruct 54.9 30 / 30 4.8openai/gpt-5.6-luna 2.9anthropic/claude-haiku-5.5 medium
202 Granite Code 3B Instructcode model 54.2 30 / 30 4.9openai/gpt-5.6-luna 3.5anthropic/claude-haiku-5.5 medium
203 Deepseek Coder 6.7B Instructcode model 53.9 30 / 30 3.7openai/gpt-5.6-luna 2.4anthropic/claude-haiku-5.5 high
204 Granite Code 8B Instructcode model 53.7 30 / 30 4.8openai/gpt-5.6-luna 3.3anthropic/claude-haiku-5.5 medium
205 Mathstral 7B V0.1fine-tune of Mistral 7Bmath model 52.3 30 / 30 4.5openai/gpt-5.6-luna 3.0anthropic/claude-haiku-5.5 medium
206 Sailor2 8B Chatfine-tune of Qwen 2.5 52.2 30 / 30 4.6openai/gpt-5.6-luna 3.0anthropic/claude-haiku-5.5 medium
207 Medgemma1.5 4B ITfine-tune of Gemma 3medical model 52.2 30 / 30 4.1openai/gpt-5.6-luna 2.5anthropic/claude-haiku-5.5 medium
208 Opencoder 1.5B Instructcode model 50.8 30 / 30 4.3openai/gpt-5.6-luna 2.4anthropic/claude-haiku-5.5 medium
209 Qwen2 0.5B Instruct 50.2 30 / 30 4.2openai/gpt-5.6-luna 2.2anthropic/claude-haiku-5.5 medium
210 Qwen2.5 Coder 0.5B Instructcode model 49.5 30 / 30 4.1openai/gpt-5.6-luna 2.3anthropic/claude-haiku-5.5 medium
211 Llama2 13B Chat 47.8 30 / 30 4.6openai/gpt-5.6-luna 2.9anthropic/claude-haiku-5.5 medium
212 Stablelm2 12B Chat 47.0 30 / 30 4.3openai/gpt-5.6-luna 3.1anthropic/claude-haiku-5.5 high
213 Marco O1 7Bfine-tune of Qwen 2 7B 46.9 30 / 30 4.6openai/gpt-5.6-luna 3.6anthropic/claude-haiku-5.5 high
214 Deepseek Coder 33B Instructcode model 46.2 30 / 30 2.9openai/gpt-5.6-luna 1.7anthropic/claude-haiku-5.5 high
215 Phi4 Mini Reasoning 3.8B 44.3 30 / 30 3.6openai/gpt-5.6-luna 1.7anthropic/claude-haiku-5.5 medium
216 Qwen 0.5B Chat V1.5 40.8 30 / 30 3.5openai/gpt-5.6-luna 1.6anthropic/claude-haiku-5.5 medium
217 Deepseek Coder 1.3B Instructcode model 40.5 30 / 30 2.6openai/gpt-5.6-luna 1.0anthropic/claude-haiku-5.5 medium
218 Qwen 1.8B Chat V1.5 38.4 30 / 30 3.7openai/gpt-5.6-luna 2.3anthropic/claude-haiku-5.5 high
219 Smollm2 135M Instruct 36.1 30 / 30 3.0openai/gpt-5.6-luna 1.3anthropic/claude-haiku-5.5 medium
219 YI Coder 1.5B Chatcode model 36.1 30 / 30 3.1openai/gpt-5.6-luna 1.6anthropic/claude-haiku-5.5 medium
221 Magistral Small (2506) 34.8 30 / 30 1.1openai/gpt-5.6-luna 0.6anthropic/claude-haiku-5.5 high
222 Stablelm2 1.6B Chat 31.6 30 / 30 3.1openai/gpt-5.6-luna 1.6anthropic/claude-haiku-5.5 high
223 Sailor2 1B Chatfine-tune of Qwen 2.5 29.6 30 / 30 3.0openai/gpt-5.6-luna 1.5anthropic/claude-haiku-5.5 high
224 Codegemma 2B Code V1.1code model 29.5 30 / 30 1.6openai/gpt-5.6-luna 0.3anthropic/claude-haiku-5.5 high
225 Smallthinker 3B Previewfine-tune of Qwen 2.5 3B 27.8 30 / 30 2.7openai/gpt-5.6-luna 1.3anthropic/claude-haiku-5.5 high
226 Exaone Deep 7.8B 16.9 30 / 30 1.9openai/gpt-5.6-luna 1.0anthropic/claude-haiku-5.5 high
227 Qwen2 Math 1.5B Instructmath model 14.4 30 / 30 1.7openai/gpt-5.6-luna 0.7anthropic/claude-haiku-5.5 high
228 Exaone Deep 2.4B 13.9 30 / 30 1.6openai/gpt-5.6-luna 0.7anthropic/claude-haiku-5.5 high

A dash means not scored, which is different from 0. Agreement compares the two judges: high means a mean gap under 1.5 points, medium under 3, low otherwise; n/a means fewer than 3 items were scored by both.

Not ranked in this category

The questions

In run r20261102-bv1-5246ab, 0 of the 30 questions in this category were answered correctly by every ranked model that has a score for them (33 ranked models in that run). Hard questions are the ones that half or fewer of the ranked models of 3B and over answered correctly. This category has 6.

The same ranking as JSON: instruction.json.