All categories

Coding

Short programming tasks. The model's code is run against hidden unit tests in a sandbox with no network.

This category counts toward Overall. The two judge models also score every item, and their scores are shown here separately. How the battery works.

Ranking

Models are ranked by their mean score in this category, from 0 to 100. Only models with at least 80% of the category's items scored, and none pending, are ranked here. The rule is the same as for Overall.

Coding: 228 ranked models. Click a column heading to sort.
Rank Model Mean (0 to 100) Items scored Judge A mean Judge B mean Agreement
1 gpt-oss-120b 99.9 30 / 30 8.5anthropic/claude-haiku-5.5 9.0deepseek/deepseek-v4.1-flash high
2 Gemma 4 12B IT 98.2 30 / 30 8.8openai/gpt-5.6-luna 8.0anthropic/claude-haiku-5.5 high
3 Qwen3-Coder-Nextcode model 97.3 30 / 30 8.9openai/gpt-5.6-luna 8.2anthropic/claude-haiku-5.5 high
4 Qwen3.8-27B 96.9 30 / 30 8.9openai/gpt-5.6-luna 8.2anthropic/claude-haiku-5.5 high
5 gemma-4-31B-it 96.6 30 / 30 9.1openai/gpt-5.6-luna 8.4anthropic/claude-haiku-5.5 high
6 gemma-4-26B-A4B-it 96.6 30 / 30 8.9openai/gpt-5.6-luna 8.4anthropic/claude-haiku-5.5 high
7 gpt-oss-20b 95.1 30 / 30 8.2anthropic/claude-haiku-5.5 8.6deepseek/deepseek-v4.1-flash high
8 Qwen3.6-35B-A3B 94.1 30 / 30 8.6openai/gpt-5.6-luna 7.9anthropic/claude-haiku-5.5 high
9 Flint 40B A2B 93.4 30 / 30 8.7openai/gpt-5.6-luna 8.0anthropic/claude-haiku-5.5 high
10 Qwen3-30B-A3B-Instruct-2507 93.2 30 / 30 8.2openai/gpt-5.6-luna 7.6anthropic/claude-haiku-5.5 high
11 Qwen3.5-27B 93.1 30 / 30 8.4openai/gpt-5.6-luna 7.9anthropic/claude-haiku-5.5 high
12 Muse Glimmer 30B 92.4 30 / 30 8.6openai/gpt-5.6-luna 8.0anthropic/claude-haiku-5.5 high
13 Qwen3.6-27B 92.0 30 / 30 8.3openai/gpt-5.6-luna 7.5anthropic/claude-haiku-5.5 high
14 Qwen3-Coder-30B-A3B-Instructcode model 92.0 30 / 30 8.2openai/gpt-5.6-luna 7.5anthropic/claude-haiku-5.5 high
15 Nanbeige_Nanbeige4.3-33B 90.1 30 / 30 8.4openai/gpt-5.6-luna 7.7anthropic/claude-haiku-5.5 high
16 Gemma4 E4B IT 89.9 30 / 30 8.1openai/gpt-5.6-luna 7.6anthropic/claude-haiku-5.5 high
17 Nemotron 3.5 Lightning 30B A3B 89.0 30 / 30 7.9openai/gpt-5.6-luna 7.4anthropic/claude-haiku-5.5 high
18 Laguna XS 2.1 88.5 30 / 30 7.8openai/gpt-5.6-luna 7.3anthropic/claude-haiku-5.5 high
19 Granite4.2 30B 88.0 30 / 30 7.9openai/gpt-5.6-luna 7.2anthropic/claude-haiku-5.5 high
20 Qwen3-VL-30B-A3B-Instruct 87.8 30 / 30 8.1openai/gpt-5.6-luna 7.2anthropic/claude-haiku-5.5 high
21 Ministral 3 14B Instruct (2512) 87.5 30 / 30 7.9openai/gpt-5.6-luna 7.5anthropic/claude-haiku-5.5 high
22 GLM-4.5-Air 87.5 30 / 30 7.9openai/gpt-5.6-luna 6.9anthropic/claude-haiku-5.5 high
23 Qwen3.5-35B-A3B 87.1 30 / 30 8.0openai/gpt-5.6-luna 7.3anthropic/claude-haiku-5.5 high
24 phi-4 87.0 30 / 30 8.1openai/gpt-5.6-luna 7.3anthropic/claude-haiku-5.5 high
25 Qwen2.5-VL-32B-Instruct 86.6 30 / 30 7.7openai/gpt-5.6-luna 7.0anthropic/claude-haiku-5.5 high
26 Qwen3-30B-A3B 86.6 30 / 30 7.5openai/gpt-5.6-luna 6.6anthropic/claude-haiku-5.5 high
27 Qwen2.5-Coder-32B-Instructcode model 86.5 30 / 30 7.6openai/gpt-5.6-luna 7.1anthropic/claude-haiku-5.5 high
28 Qwen3-VL-32B-Instruct 86.5 30 / 30 7.8openai/gpt-5.6-luna 7.0anthropic/claude-haiku-5.5 high
29 Granite4 Small H 86.3 30 / 30 7.7openai/gpt-5.6-luna 6.8anthropic/claude-haiku-5.5 high
30 NVIDIA Nemotron 3 Nano 30B-A3B 86.0 30 / 30 8.0openai/gpt-5.6-luna 7.2anthropic/claude-haiku-5.5 high
31 Granite4.1 30B 86.0 30 / 30 7.7openai/gpt-5.6-luna 7.2anthropic/claude-haiku-5.5 high
32 gemma-3-27b-it 85.4 30 / 30 7.2openai/gpt-5.6-luna 6.4anthropic/claude-haiku-5.5 high
33 Granite4.2 8B 85.1 30 / 30 7.8openai/gpt-5.6-luna 7.1anthropic/claude-haiku-5.5 high
34 Olmo 3.1 32B Instruct 84.7 30 / 30 6.3openai/gpt-5.6-luna 7.1anthropic/claude-haiku-5.5 high
35 Sage-A1.5-Q4_K_M 84.6 30 / 30 7.6openai/gpt-5.6-luna 6.8anthropic/claude-haiku-5.5 high
36 Qwen3-32B 83.8 30 / 30 7.5openai/gpt-5.6-luna 6.8anthropic/claude-haiku-5.5 high
37 Nemotron3 33B 83.7 30 / 30 7.5openai/gpt-5.6-luna 7.1anthropic/claude-haiku-5.5 high
38 Devstral Small 2 24B Instruct (2512)code model 83.4 30 / 30 7.5openai/gpt-5.6-luna 6.6anthropic/claude-haiku-5.5 high
39 Mistral Small 24B Instruct 2501 82.8 30 / 30 7.5openai/gpt-5.6-luna 6.6anthropic/claude-haiku-5.5 high
40 Qwen2.5-Coder-14B-Instructcode model 82.7 30 / 30 7.0openai/gpt-5.6-luna 6.3anthropic/claude-haiku-5.5 high
41 Qwen2.5-32B-Instruct 82.0 30 / 30 7.2openai/gpt-5.6-luna 6.3anthropic/claude-haiku-5.5 high
42 gemma-3-12b-it 81.8 30 / 30 7.3openai/gpt-5.6-luna 6.8anthropic/claude-haiku-5.5 high
43 GLM-4.7-Flash 81.6 30 / 30 7.2openai/gpt-5.6-luna 6.5anthropic/claude-haiku-5.5 high
44 YI Coder 9B Chatcode model 81.3 30 / 30 5.9openai/gpt-5.6-luna 5.6anthropic/claude-haiku-5.5 high
45 Llama 4 Scout 17B-16E Instruct 81.0 30 / 30 7.1openai/gpt-5.6-luna 6.4anthropic/claude-haiku-5.5 high
46 Granite4 Tiny H 80.8 30 / 30 7.3openai/gpt-5.6-luna 6.4anthropic/claude-haiku-5.5 high
47 Llama-3.3-70B-Instruct 80.7 30 / 30 7.4openai/gpt-5.6-luna 6.6anthropic/claude-haiku-5.5 high
48 Gemma4 E2B IT 80.1 30 / 30 7.1openai/gpt-5.6-luna 6.3anthropic/claude-haiku-5.5 high
49 Translategemma 12B ITfine-tune of Gemma 3translation model 78.9 30 / 30 7.1openai/gpt-5.6-luna 6.2anthropic/claude-haiku-5.5 high
50 Cogito 14B V1 Preview Qwenfine-tune of Qwen 2.5 78.8 30 / 30 7.2openai/gpt-5.6-luna 6.5anthropic/claude-haiku-5.5 high
51 Ministral 3 8B Instruct 2512 78.4 30 / 30 7.2openai/gpt-5.6-luna 6.5anthropic/claude-haiku-5.5 high
52 Nemotron Cascade 2 30B A3B 77.9 30 / 30 7.1openai/gpt-5.6-luna 6.5anthropic/claude-haiku-5.5 high
53 Qwen3-VL-8B-Instruct 77.7 30 / 30 6.8openai/gpt-5.6-luna 5.6anthropic/claude-haiku-5.5 high
54 Qwen3.5-9B 77.4 30 / 30 7.3openai/gpt-5.6-luna 6.6anthropic/claude-haiku-5.5 high
55 Mistral Small 3.2 24B Instruct (2506) 77.2 30 / 30 6.8openai/gpt-5.6-luna 6.0anthropic/claude-haiku-5.5 high
56 Medgemma 27B ITfine-tune of Gemma 3medical model 77.0 30 / 30 6.9openai/gpt-5.6-luna 6.0anthropic/claude-haiku-5.5 high
57 Qwen2.5-14B-Instruct 76.5 30 / 30 7.2openai/gpt-5.6-luna 6.2anthropic/claude-haiku-5.5 high
58 Granite4.1 8B 76.5 30 / 30 6.7openai/gpt-5.6-luna 6.3anthropic/claude-haiku-5.5 high
59 Qwen3-14B 73.5 30 / 30 6.5openai/gpt-5.6-luna 5.8anthropic/claude-haiku-5.5 high
60 Mistral Small3.1 24B Instruct 2503 73.3 30 / 30 6.8openai/gpt-5.6-luna 6.0anthropic/claude-haiku-5.5 high
61 Nemotron 3.5 Bolt E4B 73.2 30 / 30 6.4openai/gpt-5.6-luna 6.1anthropic/claude-haiku-5.5 high
62 Codestral 22B V0.1code model 72.9 30 / 30 5.6openai/gpt-5.6-luna 5.8anthropic/claude-haiku-5.5 high
63 Lfm2 24B 72.9 30 / 30 6.7openai/gpt-5.6-luna 6.3anthropic/claude-haiku-5.5 high
64 DeepSeek-Coder-V2-Lite-Instructcode model 72.5 30 / 30 6.4openai/gpt-5.6-luna 5.8anthropic/claude-haiku-5.5 high
65 Qwen2.5-Coder-7B-Instructcode model 71.8 30 / 30 6.7openai/gpt-5.6-luna 5.8anthropic/claude-haiku-5.5 high
66 Exaone3.5 32B Instruct 71.5 30 / 30 6.6openai/gpt-5.6-luna 5.7anthropic/claude-haiku-5.5 high
67 Qwen3.5-4B 71.5 30 / 30 6.7openai/gpt-5.6-luna 5.8anthropic/claude-haiku-5.5 high
68 gemma-3-4b-it 71.4 30 / 30 6.5openai/gpt-5.6-luna 5.6anthropic/claude-haiku-5.5 high
69 Qwen3-8B 70.6 30 / 30 6.1openai/gpt-5.6-luna 5.6anthropic/claude-haiku-5.5 high
70 Devstral 24B Small 2505code model 70.4 30 / 30 6.5openai/gpt-5.6-luna 5.7anthropic/claude-haiku-5.5 high
71 Mistral Small 22B Instruct 2409 70.1 30 / 30 6.2openai/gpt-5.6-luna 5.3anthropic/claude-haiku-5.5 high
72 EXAONE-3.5-7.8B-Instruct 70.1 30 / 30 6.4openai/gpt-5.6-luna 5.6anthropic/claude-haiku-5.5 high
73 Qwen3-4B-Instruct-2507 69.7 30 / 30 6.3openai/gpt-5.6-luna 5.7anthropic/claude-haiku-5.5 high
74 Codeqwen 7B Chat V1.5code model 69.4 30 / 30 6.2openai/gpt-5.6-luna 5.3anthropic/claude-haiku-5.5 high
75 Qwen3-4B 69.0 30 / 30 6.2openai/gpt-5.6-luna 5.2anthropic/claude-haiku-5.5 high
76 Granite4 Micro H 68.8 30 / 30 6.2openai/gpt-5.6-luna 5.3anthropic/claude-haiku-5.5 high
77 Deepseek Coder 6.7B Instructcode model 68.5 30 / 30 6.1openai/gpt-5.6-luna 5.2anthropic/claude-haiku-5.5 high
78 Ornith-1.5-9B 67.9 30 / 30 6.2openai/gpt-5.6-luna 5.4anthropic/claude-haiku-5.5 high
79 Gemma2 27B Instruct 67.5 30 / 30 6.1openai/gpt-5.6-luna 5.3anthropic/claude-haiku-5.5 high
80 Command R7B Arabic 7B 02 2025 67.4 30 / 30 5.3openai/gpt-5.6-luna 4.4anthropic/claude-haiku-5.5 high
81 Deepseek Coder 33B Instructcode model 67.2 30 / 30 5.5openai/gpt-5.6-luna 5.2anthropic/claude-haiku-5.5 high
82 dolphin-2.9.1-yi-1.5-34b 65.7 30 / 30 5.6openai/gpt-5.6-luna 5.1anthropic/claude-haiku-5.5 high
83 Llama-3.1-8B-Instruct 65.6 30 / 30 6.2openai/gpt-5.6-luna 5.3anthropic/claude-haiku-5.5 high
84 Spark-X2.6-4B 65.6 30 / 30 6.4openai/gpt-5.6-luna 5.7anthropic/claude-haiku-5.5 high
85 Gemma 3n E4B IT 65.3 30 / 30 6.0openai/gpt-5.6-luna 5.1anthropic/claude-haiku-5.5 high
86 Exaone3.5 2.4B Instruct 64.7 30 / 30 6.3openai/gpt-5.6-luna 5.4anthropic/claude-haiku-5.5 high
87 Marco O1 7Bfine-tune of Qwen 2 7B 64.7 30 / 30 5.9openai/gpt-5.6-luna 5.1anthropic/claude-haiku-5.5 high
88 Aya Expanse 32B 64.7 30 / 30 6.2openai/gpt-5.6-luna 5.1anthropic/claude-haiku-5.5 high
89 Granite4 3B 64.7 30 / 30 6.3openai/gpt-5.6-luna 5.5anthropic/claude-haiku-5.5 high
90 Qwen2.5-VL-7B-Instruct 64.4 30 / 30 5.6openai/gpt-5.6-luna 4.6anthropic/claude-haiku-5.5 high
91 Falcon3 10B Instruct 64.3 30 / 30 5.6openai/gpt-5.6-luna 5.3anthropic/claude-haiku-5.5 high
92 Granite3.1 Dense 8B Instruct 64.2 30 / 30 6.0openai/gpt-5.6-luna 5.1anthropic/claude-haiku-5.5 high
93 Command R 35B 08 2024 63.9 30 / 30 5.9openai/gpt-5.6-luna 5.0anthropic/claude-haiku-5.5 high
94 Qwen2.5-7B-Instruct 63.2 30 / 30 5.9openai/gpt-5.6-luna 5.1anthropic/claude-haiku-5.5 high
95 Ornith 9B 63.0 30 / 30 6.2openai/gpt-5.6-luna 5.5anthropic/claude-haiku-5.5 high
96 granite-4.1-3b 62.6 30 / 30 5.4openai/gpt-5.6-luna 5.3anthropic/claude-haiku-5.5 high
97 Granite4.2 3B 61.6 30 / 30 6.0openai/gpt-5.6-luna 5.4anthropic/claude-haiku-5.5 high
98 Lfm3 1.7B 61.5 30 / 30 5.7openai/gpt-5.6-luna 5.2anthropic/claude-haiku-5.5 high
99 Phi3 14B Medium 128K Instruct 61.4 30 / 30 5.0openai/gpt-5.6-luna 5.1anthropic/claude-haiku-5.5 high
100 Gemma3n E2B IT 61.4 30 / 30 5.7openai/gpt-5.6-luna 4.7anthropic/claude-haiku-5.5 high
101 Qwen3-VL-4B-Instruct 61.2 30 / 30 5.8openai/gpt-5.6-luna 5.0anthropic/claude-haiku-5.5 high
102 Phi-4-mini-instruct 61.0 30 / 30 5.9openai/gpt-5.6-luna 5.2anthropic/claude-haiku-5.5 high
103 Ministral 3 3B Instruct 2512 60.6 30 / 30 5.5openai/gpt-5.6-luna 4.9anthropic/claude-haiku-5.5 high
104 QwQ 32B 60.6 30 / 30 5.6openai/gpt-5.6-luna 5.5anthropic/claude-haiku-5.5 high
105 Codegeex4 9B Allcode model 60.5 30 / 30 5.7openai/gpt-5.6-luna 5.1anthropic/claude-haiku-5.5 high
106 medgemma-4b-itmedical model 60.3 30 / 30 5.8openai/gpt-5.6-luna 4.8anthropic/claude-haiku-5.5 high
107 Qwen2.5-Coder-3B-Instructcode model 60.2 30 / 30 5.4openai/gpt-5.6-luna 4.6anthropic/claude-haiku-5.5 high
108 Meta-Llama-3-8B-Instruct 60.0 30 / 30 5.6openai/gpt-5.6-luna 4.7anthropic/claude-haiku-5.5 high
109 Glm4 9B Chat 60.0 30 / 30 5.7openai/gpt-5.6-luna 4.8anthropic/claude-haiku-5.5 high
110 Granite4 1B H 59.9 30 / 30 5.2openai/gpt-5.6-luna 4.9anthropic/claude-haiku-5.5 high
111 Command R7B (12-2024) 59.4 30 / 30 5.0openai/gpt-5.6-luna 4.3anthropic/claude-haiku-5.5 high
112 Granite3 Dense 8B Instruct 59.2 30 / 30 5.0openai/gpt-5.6-luna 4.2anthropic/claude-haiku-5.5 high
113 Falcon3 7B Instruct 58.7 30 / 30 5.6openai/gpt-5.6-luna 4.8anthropic/claude-haiku-5.5 high
114 Granite3.2 8B Instruct 58.5 30 / 30 5.8openai/gpt-5.6-luna 4.8anthropic/claude-haiku-5.5 high
115 Olmo-3-7B-Instruct 57.7 30 / 30 4.8openai/gpt-5.6-luna 5.0anthropic/claude-haiku-5.5 high
116 SmolLM3-3B 57.7 30 / 30 5.5openai/gpt-5.6-luna 4.9anthropic/claude-haiku-5.5 high
117 YI Coder 1.5B Chatcode model 57.7 30 / 30 4.7openai/gpt-5.6-luna 3.8anthropic/claude-haiku-5.5 high
118 YI 9B Chat V1.5 57.5 30 / 30 4.8openai/gpt-5.6-luna 4.3anthropic/claude-haiku-5.5 high
119 Opencoder 1.5B Instructcode model 57.3 30 / 30 4.7openai/gpt-5.6-luna 4.1anthropic/claude-haiku-5.5 high
120 Mistral Nemo 12B Instruct 2407 57.3 30 / 30 5.0openai/gpt-5.6-luna 4.8anthropic/claude-haiku-5.5 high
121 Medgemma1.5 4B ITfine-tune of Gemma 3medical model 56.9 30 / 30 4.9openai/gpt-5.6-luna 4.4anthropic/claude-haiku-5.5 high
122 Granite3.3 8B 56.6 30 / 30 5.2openai/gpt-5.6-luna 4.5anthropic/claude-haiku-5.5 high
123 Mathstral 7B V0.1fine-tune of Mistral 7Bmath model 56.6 30 / 30 4.5openai/gpt-5.6-luna 3.7anthropic/claude-haiku-5.5 high
124 Cogito 8B V1 Preview Llamafine-tune of Llama 3.1 56.1 30 / 30 5.3openai/gpt-5.6-luna 4.5anthropic/claude-haiku-5.5 high
125 Granite Code 8B Instructcode model 55.7 30 / 30 4.5openai/gpt-5.6-luna 3.8anthropic/claude-haiku-5.5 high
126 Codegemma 7B Instruct V1.1code model 55.5 30 / 30 5.4openai/gpt-5.6-luna 4.4anthropic/claude-haiku-5.5 high
127 Exaone Deep 7.8B 55.4 30 / 30 3.8openai/gpt-5.6-luna 3.8anthropic/claude-haiku-5.5 high
128 Deepseek Coder 1.3B Instructcode model 54.5 30 / 30 4.5openai/gpt-5.6-luna 3.7anthropic/claude-haiku-5.5 high
129 DeepSeek-R1-Distill-Qwen-32B 53.6 30 / 30 3.8openai/gpt-5.6-luna 3.3anthropic/claude-haiku-5.5 high
130 DeepSeek-R1-Distill-Qwen-14B 53.0 30 / 30 3.7openai/gpt-5.6-luna 3.3anthropic/claude-haiku-5.5 high
131 Llama-3.2-3B-Instruct 52.9 30 / 30 4.8openai/gpt-5.6-luna 4.5anthropic/claude-haiku-5.5 high
132 Deepseek R1 8B Llama Distill 52.8 30 / 30 3.9openai/gpt-5.6-luna 3.1anthropic/claude-haiku-5.5 high
133 Qwen2.5 Coder 1.5B Instructcode model 52.8 30 / 30 5.3openai/gpt-5.6-luna 4.4anthropic/claude-haiku-5.5 high
134 Tulu3 8Bfine-tune of Llama 3.1 52.7 30 / 30 4.7openai/gpt-5.6-luna 4.2anthropic/claude-haiku-5.5 high
135 Openthinker 7B V2fine-tune of Qwen 2.5 52.6 30 / 30 3.7openai/gpt-5.6-luna 2.5anthropic/claude-haiku-5.5 medium
136 Qwen3.5 2B 52.5 30 / 30 5.2openai/gpt-5.6-luna 4.4anthropic/claude-haiku-5.5 high
137 gemma-2-9b-it 51.8 30 / 30 4.7openai/gpt-5.6-luna 4.1anthropic/claude-haiku-5.5 high
138 YI 34B Chat V1.5 50.8 30 / 30 4.3openai/gpt-5.6-luna 4.2anthropic/claude-haiku-5.5 high
139 Granite3.3 2B 50.4 30 / 30 4.6openai/gpt-5.6-luna 3.8anthropic/claude-haiku-5.5 high
140 Aya Expanse 8B 50.0 30 / 30 4.4openai/gpt-5.6-luna 4.0anthropic/claude-haiku-5.5 high
141 Qwen2.5 1.5B Instruct 49.9 30 / 30 4.9openai/gpt-5.6-luna 4.1anthropic/claude-haiku-5.5 high
142 Qwen2-VL-7B-Instruct 49.7 30 / 30 4.7openai/gpt-5.6-luna 4.1anthropic/claude-haiku-5.5 high
143 Cogito 3B V1 Preview Llamafine-tune of Llama 3.2 49.3 30 / 30 4.7openai/gpt-5.6-luna 3.9anthropic/claude-haiku-5.5 high
144 Qwen3 VL 2B Instruct 49.2 30 / 30 5.3openai/gpt-5.6-luna 4.3anthropic/claude-haiku-5.5 high
145 Qwen2.5-3B-Instruct 48.7 30 / 30 4.6openai/gpt-5.6-luna 3.8anthropic/claude-haiku-5.5 high
146 Internlm2 7B Chat V2.5 48.7 30 / 30 4.2openai/gpt-5.6-luna 4.2anthropic/claude-haiku-5.5 high
147 Internlm2 1.8B Chat V2.5 48.5 30 / 30 4.9openai/gpt-5.6-luna 4.1anthropic/claude-haiku-5.5 high
148 Granite3.2 2B Instruct 48.4 30 / 30 5.1openai/gpt-5.6-luna 4.3anthropic/claude-haiku-5.5 high
149 Lfm2.5 8B A1B 47.3 30 / 30 3.4openai/gpt-5.6-luna 3.6anthropic/claude-haiku-5.5 high
150 Granite3.1 Dense 2B Instruct 46.6 30 / 30 4.8openai/gpt-5.6-luna 4.2anthropic/claude-haiku-5.5 high
151 Granite Code 3B Instructcode model 45.6 30 / 30 4.0openai/gpt-5.6-luna 3.0anthropic/claude-haiku-5.5 high
152 Qwen3 1.7B 45.5 30 / 30 4.2openai/gpt-5.6-luna 3.8anthropic/claude-haiku-5.5 high
153 Stablelm2 12B Chat 45.0 30 / 30 4.2openai/gpt-5.6-luna 3.4anthropic/claude-haiku-5.5 high
154 Qwen2.5-VL-3B-Instruct 42.6 30 / 30 4.2openai/gpt-5.6-luna 3.6anthropic/claude-haiku-5.5 high
155 Gemma3 1B IT 42.5 30 / 30 4.3openai/gpt-5.6-luna 3.5anthropic/claude-haiku-5.5 high
156 Gemma2 2B Instruct 42.4 30 / 30 4.2openai/gpt-5.6-luna 3.6anthropic/claude-haiku-5.5 high
157 Hermes-3-Llama-3.1-8B 42.3 30 / 30 3.9openai/gpt-5.6-luna 4.1anthropic/claude-haiku-5.5 high
158 Lfm3 VL 2B 42.2 30 / 30 4.4openai/gpt-5.6-luna 3.7anthropic/claude-haiku-5.5 high
159 Granite3 Dense 2B Instruct 42.2 30 / 30 4.4openai/gpt-5.6-luna 3.6anthropic/claude-haiku-5.5 high
160 DeepSeek-R1-0528-Qwen3-8B 42.1 30 / 30 3.5openai/gpt-5.6-luna 3.3anthropic/claude-haiku-5.5 high
161 Lfm2.5 Thinking 1.2B 42.1 30 / 30 3.5openai/gpt-5.6-luna 3.7anthropic/claude-haiku-5.5 high
162 Llama3.2 1B Instruct 41.7 30 / 30 4.5openai/gpt-5.6-luna 3.6anthropic/claude-haiku-5.5 high
163 Deepseek V2 16B Lite Chat 41.6 30 / 30 4.2openai/gpt-5.6-luna 3.4anthropic/claude-haiku-5.5 high
164 Mistral 7B Instruct V0.3 40.9 30 / 30 3.7openai/gpt-5.6-luna 3.0anthropic/claude-haiku-5.5 high
165 Falcon3 3B Instruct 40.9 30 / 30 4.1openai/gpt-5.6-luna 3.5anthropic/claude-haiku-5.5 high
166 Olmo2 13B 1124 Instruct 40.8 30 / 30 4.3openai/gpt-5.6-luna 3.1anthropic/claude-haiku-5.5 high
167 Olmo-3-7B-Think 40.0 30 / 30 3.8openai/gpt-5.6-luna 3.6anthropic/claude-haiku-5.5 high
168 Qwen2 Math 7B Instructmath model 39.1 30 / 30 3.8openai/gpt-5.6-luna 3.5anthropic/claude-haiku-5.5 high
169 Phi-3.5-mini-instruct 38.9 30 / 30 3.9openai/gpt-5.6-luna 3.6anthropic/claude-haiku-5.5 high
170 Codellama 34B Instructcode model 38.7 30 / 30 3.6openai/gpt-5.6-luna 3.5anthropic/claude-haiku-5.5 high
171 Phi3 3.8B Mini 128K Instruct 38.6 30 / 30 3.7openai/gpt-5.6-luna 3.6anthropic/claude-haiku-5.5 high
172 Granite3.1 Moe 1B Instruct 38.1 30 / 30 4.2openai/gpt-5.6-luna 3.5anthropic/claude-haiku-5.5 high
173 Deepseek Llm 7B Chat 37.4 30 / 30 3.8openai/gpt-5.6-luna 3.1anthropic/claude-haiku-5.5 high
174 DeepSeek-R1-Distill-Qwen-7B 37.0 30 / 30 2.7openai/gpt-5.6-luna 2.4anthropic/claude-haiku-5.5 high
175 Exaone Deep 2.4B 36.7 30 / 30 2.6openai/gpt-5.6-luna 2.7anthropic/claude-haiku-5.5 high
176 Codellama 7B Instructcode model 36.0 30 / 30 3.5openai/gpt-5.6-luna 2.9anthropic/claude-haiku-5.5 high
177 Granite4 350M H 36.0 30 / 30 3.4openai/gpt-5.6-luna 3.0anthropic/claude-haiku-5.5 high
178 Qwen2.5 Coder 0.5B Instructcode model 35.9 30 / 30 3.9openai/gpt-5.6-luna 3.0anthropic/claude-haiku-5.5 high
179 Smollm2 1.7B Instruct 35.3 30 / 30 3.7openai/gpt-5.6-luna 2.9anthropic/claude-haiku-5.5 high
180 Phi-3-mini-4k-instruct 35.1 30 / 30 3.4openai/gpt-5.6-luna 3.0anthropic/claude-haiku-5.5 high
181 Phi 2.7B Chat V2 34.7 30 / 30 3.1openai/gpt-5.6-luna 2.8anthropic/claude-haiku-5.5 high
182 Nemotron Mini 4B Instruct 34.6 30 / 30 3.6openai/gpt-5.6-luna 2.7anthropic/claude-haiku-5.5 high
183 Solar 10.7B Instruct V1 34.3 30 / 30 3.5openai/gpt-5.6-luna 2.8anthropic/claude-haiku-5.5 high
184 Deepcoder 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bcode model 34.2 30 / 30 2.4openai/gpt-5.6-luna 2.3anthropic/claude-haiku-5.5 high
185 Apertus-8B-Instruct-2509 33.0 30 / 30 3.4openai/gpt-5.6-luna 2.8anthropic/claude-haiku-5.5 high
186 Granite3.1 Moe 3B Instruct 32.6 30 / 30 3.6openai/gpt-5.6-luna 2.8anthropic/claude-haiku-5.5 high
187 Qwen3 0.6B 32.5 30 / 30 3.5openai/gpt-5.6-luna 2.7anthropic/claude-haiku-5.5 high
188 Phi4 Mini Reasoning 3.8B 31.8 30 / 30 2.3openai/gpt-5.6-luna 1.9anthropic/claude-haiku-5.5 high
189 Granite3 Moe 3B Instruct 31.6 30 / 30 3.4openai/gpt-5.6-luna 2.3anthropic/claude-haiku-5.5 high
190 Aya 8B 23 30.7 30 / 30 3.4openai/gpt-5.6-luna 2.5anthropic/claude-haiku-5.5 high
191 Qwen2 1.5B Instruct 30.6 30 / 30 3.3openai/gpt-5.6-luna 2.9anthropic/claude-haiku-5.5 high
192 Smallthinker 3B Previewfine-tune of Qwen 2.5 3B 29.8 30 / 30 2.1openai/gpt-5.6-luna 1.3anthropic/claude-haiku-5.5 high
193 Stablelm Zephyr 3Bfine-tune of StableLM 3B 29.3 30 / 30 3.4openai/gpt-5.6-luna 2.4anthropic/claude-haiku-5.5 high
194 Olmo2 7B 1124 Instruct 29.3 30 / 30 3.4openai/gpt-5.6-luna 2.8anthropic/claude-haiku-5.5 high
195 Dolphin Phi 2.7B V2.6fine-tune of Phi-2 28.6 30 / 30 2.3openai/gpt-5.6-luna 2.3anthropic/claude-haiku-5.5 high
196 Granite3 Moe 1B Instruct 27.6 30 / 30 3.0openai/gpt-5.6-luna 1.8anthropic/claude-haiku-5.5 high
197 Deepscaler 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bmath model 26.7 30 / 30 1.9openai/gpt-5.6-luna 1.7anthropic/claude-haiku-5.5 high
197 Qwen3-4B-Thinking-2507 26.7 30 / 30 2.0openai/gpt-5.6-luna 2.3anthropic/claude-haiku-5.5 high
199 Qwen2.5 0.5B Instruct 26.5 30 / 30 2.4openai/gpt-5.6-luna 2.1anthropic/claude-haiku-5.5 high
200 Gemma 7B Instruct V1.1 26.5 30 / 30 2.8openai/gpt-5.6-luna 2.4anthropic/claude-haiku-5.5 high
201 Stablelm2 1.6B Chat 26.1 30 / 30 2.4openai/gpt-5.6-luna 1.9anthropic/claude-haiku-5.5 high
202 Mistral-7B-Instruct-v0.2 25.3 30 / 30 2.7openai/gpt-5.6-luna 2.3anthropic/claude-haiku-5.5 high
203 Gemma 2B Instruct V1.1 25.2 30 / 30 3.0openai/gpt-5.6-luna 2.0anthropic/claude-haiku-5.5 high
204 Deepseek R1 1.5B Qwen Distill 24.4 30 / 30 1.8openai/gpt-5.6-luna 1.5anthropic/claude-haiku-5.5 high
205 Smollm2 360M Instruct 23.8 30 / 30 2.7openai/gpt-5.6-luna 2.0anthropic/claude-haiku-5.5 high
206 Codellama 13B Instructcode model 22.5 30 / 30 2.1openai/gpt-5.6-luna 1.9anthropic/claude-haiku-5.5 high
207 Mixtral 8x7B Instruct V0.1 20.1 30 / 30 2.0openai/gpt-5.6-luna 2.0anthropic/claude-haiku-5.5 high
208 Deepcoder 14B Previewfine-tune of DeepSeek-R1-Distill-Qwen 14Bcode model 20.0 30 / 30 1.4openai/gpt-5.6-luna 1.2anthropic/claude-haiku-5.5 high
209 YI 6B Chat V1.5 19.8 30 / 30 1.7openai/gpt-5.6-luna 1.8anthropic/claude-haiku-5.5 high
210 Sailor2 8B Chatfine-tune of Qwen 2.5 19.5 30 / 30 1.7openai/gpt-5.6-luna 1.7anthropic/claude-haiku-5.5 high
211 Falcon3 1B Instruct 18.7 30 / 30 1.8openai/gpt-5.6-luna 1.5anthropic/claude-haiku-5.5 high
212 Llama-2-7b-chat-hf 18.6 30 / 30 2.4openai/gpt-5.6-luna 1.7anthropic/claude-haiku-5.5 high
213 Qwen2 Math 1.5B Instructmath model 17.5 30 / 30 1.4openai/gpt-5.6-luna 1.0anthropic/claude-haiku-5.5 high
214 Falcon2 11Bnot in the Overall ranking 16.9 30 / 30 1.4openai/gpt-5.6-luna 1.0anthropic/claude-haiku-5.5 high
215 Smollm2 135M Instruct 14.1 30 / 30 1.6openai/gpt-5.6-luna 1.2anthropic/claude-haiku-5.5 high
216 Hermes3 3B Llama3.2fine-tune of Llama 3.2 11.3 30 / 30 1.1openai/gpt-5.6-luna 1.3anthropic/claude-haiku-5.5 high
217 Sailor2 1B Chatfine-tune of Qwen 2.5 10.5 30 / 30 1.9openai/gpt-5.6-luna 1.0anthropic/claude-haiku-5.5 high
218 Llama2 13B Chat 10.1 30 / 30 1.2openai/gpt-5.6-luna 1.0anthropic/claude-haiku-5.5 high
219 Magistral Small (2506) 10.0 30 / 30 0.7openai/gpt-5.6-luna 0.7anthropic/claude-haiku-5.5 high
220 Qwen2 0.5B Instruct 9.8 30 / 30 0.8openai/gpt-5.6-luna 0.9anthropic/claude-haiku-5.5 high
221 Qwen 14B Chat V1.5 9.7 30 / 30 1.0openai/gpt-5.6-luna 0.9anthropic/claude-haiku-5.5 high
222 Qwen 0.5B Chat V1.5 9.0 30 / 30 1.0openai/gpt-5.6-luna 0.6anthropic/claude-haiku-5.5 high
223 Stable Code 3B Instructcode model 6.5 30 / 30 0.8openai/gpt-5.6-luna 0.7anthropic/claude-haiku-5.5 high
224 Qwen 7B Chat V1.5 5.9 30 / 30 0.7openai/gpt-5.6-luna 0.6anthropic/claude-haiku-5.5 high
225 Codegemma 2B Code V1.1code model 3.9 30 / 30 0.3openai/gpt-5.6-luna 0.4anthropic/claude-haiku-5.5 high
226 Qwen 4B Chat V1.5 3.1 30 / 30 0.4openai/gpt-5.6-luna 0.3anthropic/claude-haiku-5.5 high
227 Qwen 1.8B Chat V1.5 2.4 30 / 30 0.3openai/gpt-5.6-luna 0.3anthropic/claude-haiku-5.5 high
228 Qwen3 VL 2B Thinking 0 30 / 30 0openai/gpt-5.6-luna 0anthropic/claude-haiku-5.5 high

A dash means not scored, which is different from 0. Agreement compares the two judges: high means a mean gap under 1.5 points, medium under 3, low otherwise; n/a means fewer than 3 items were scored by both.

Not ranked in this category

The questions

In run r20261102-bv1-5246ab, 2 of the 30 questions in this category were answered correctly by every ranked model that has a score for them (33 ranked models in that run). Hard questions are the ones that half or fewer of the ranked models of 3B and over answered correctly. This category has 16.

The same ranking as JSON: coding.json.