Coding
Short programming tasks. The model's code is run against hidden unit tests in a sandbox with no network.
This category counts toward Overall. The two judge models also score every item, and their scores are shown here separately. How the battery works.
Ranking
Models are ranked by their mean score in this category, from 0 to 100. Only models with at least 80% of the category's items scored, and none pending, are ranked here. The rule is the same as for Overall.
| Rank | Model | Mean (0 to 100) | Items scored | Judge A mean | Judge B mean | Agreement |
|---|---|---|---|---|---|---|
| 1 | gpt-oss-120b | 99.9 | 30 / 30 | 8.5anthropic/claude-haiku-5.5 | 9.0deepseek/deepseek-v4.1-flash | high |
| 2 | Gemma 4 12B IT | 98.2 | 30 / 30 | 8.8openai/gpt-5.6-luna | 8.0anthropic/claude-haiku-5.5 | high |
| 3 | Qwen3-Coder-Nextcode model | 97.3 | 30 / 30 | 8.9openai/gpt-5.6-luna | 8.2anthropic/claude-haiku-5.5 | high |
| 4 | Qwen3.8-27B | 96.9 | 30 / 30 | 8.9openai/gpt-5.6-luna | 8.2anthropic/claude-haiku-5.5 | high |
| 5 | gemma-4-31B-it | 96.6 | 30 / 30 | 9.1openai/gpt-5.6-luna | 8.4anthropic/claude-haiku-5.5 | high |
| 6 | gemma-4-26B-A4B-it | 96.6 | 30 / 30 | 8.9openai/gpt-5.6-luna | 8.4anthropic/claude-haiku-5.5 | high |
| 7 | gpt-oss-20b | 95.1 | 30 / 30 | 8.2anthropic/claude-haiku-5.5 | 8.6deepseek/deepseek-v4.1-flash | high |
| 8 | Qwen3.6-35B-A3B | 94.1 | 30 / 30 | 8.6openai/gpt-5.6-luna | 7.9anthropic/claude-haiku-5.5 | high |
| 9 | Flint 40B A2B | 93.4 | 30 / 30 | 8.7openai/gpt-5.6-luna | 8.0anthropic/claude-haiku-5.5 | high |
| 10 | Qwen3-30B-A3B-Instruct-2507 | 93.2 | 30 / 30 | 8.2openai/gpt-5.6-luna | 7.6anthropic/claude-haiku-5.5 | high |
| 11 | Qwen3.5-27B | 93.1 | 30 / 30 | 8.4openai/gpt-5.6-luna | 7.9anthropic/claude-haiku-5.5 | high |
| 12 | Muse Glimmer 30B | 92.4 | 30 / 30 | 8.6openai/gpt-5.6-luna | 8.0anthropic/claude-haiku-5.5 | high |
| 13 | Qwen3.6-27B | 92.0 | 30 / 30 | 8.3openai/gpt-5.6-luna | 7.5anthropic/claude-haiku-5.5 | high |
| 14 | Qwen3-Coder-30B-A3B-Instructcode model | 92.0 | 30 / 30 | 8.2openai/gpt-5.6-luna | 7.5anthropic/claude-haiku-5.5 | high |
| 15 | Nanbeige_Nanbeige4.3-33B | 90.1 | 30 / 30 | 8.4openai/gpt-5.6-luna | 7.7anthropic/claude-haiku-5.5 | high |
| 16 | Gemma4 E4B IT | 89.9 | 30 / 30 | 8.1openai/gpt-5.6-luna | 7.6anthropic/claude-haiku-5.5 | high |
| 17 | Nemotron 3.5 Lightning 30B A3B | 89.0 | 30 / 30 | 7.9openai/gpt-5.6-luna | 7.4anthropic/claude-haiku-5.5 | high |
| 18 | Laguna XS 2.1 | 88.5 | 30 / 30 | 7.8openai/gpt-5.6-luna | 7.3anthropic/claude-haiku-5.5 | high |
| 19 | Granite4.2 30B | 88.0 | 30 / 30 | 7.9openai/gpt-5.6-luna | 7.2anthropic/claude-haiku-5.5 | high |
| 20 | Qwen3-VL-30B-A3B-Instruct | 87.8 | 30 / 30 | 8.1openai/gpt-5.6-luna | 7.2anthropic/claude-haiku-5.5 | high |
| 21 | Ministral 3 14B Instruct (2512) | 87.5 | 30 / 30 | 7.9openai/gpt-5.6-luna | 7.5anthropic/claude-haiku-5.5 | high |
| 22 | GLM-4.5-Air | 87.5 | 30 / 30 | 7.9openai/gpt-5.6-luna | 6.9anthropic/claude-haiku-5.5 | high |
| 23 | Qwen3.5-35B-A3B | 87.1 | 30 / 30 | 8.0openai/gpt-5.6-luna | 7.3anthropic/claude-haiku-5.5 | high |
| 24 | phi-4 | 87.0 | 30 / 30 | 8.1openai/gpt-5.6-luna | 7.3anthropic/claude-haiku-5.5 | high |
| 25 | Qwen2.5-VL-32B-Instruct | 86.6 | 30 / 30 | 7.7openai/gpt-5.6-luna | 7.0anthropic/claude-haiku-5.5 | high |
| 26 | Qwen3-30B-A3B | 86.6 | 30 / 30 | 7.5openai/gpt-5.6-luna | 6.6anthropic/claude-haiku-5.5 | high |
| 27 | Qwen2.5-Coder-32B-Instructcode model | 86.5 | 30 / 30 | 7.6openai/gpt-5.6-luna | 7.1anthropic/claude-haiku-5.5 | high |
| 28 | Qwen3-VL-32B-Instruct | 86.5 | 30 / 30 | 7.8openai/gpt-5.6-luna | 7.0anthropic/claude-haiku-5.5 | high |
| 29 | Granite4 Small H | 86.3 | 30 / 30 | 7.7openai/gpt-5.6-luna | 6.8anthropic/claude-haiku-5.5 | high |
| 30 | NVIDIA Nemotron 3 Nano 30B-A3B | 86.0 | 30 / 30 | 8.0openai/gpt-5.6-luna | 7.2anthropic/claude-haiku-5.5 | high |
| 31 | Granite4.1 30B | 86.0 | 30 / 30 | 7.7openai/gpt-5.6-luna | 7.2anthropic/claude-haiku-5.5 | high |
| 32 | gemma-3-27b-it | 85.4 | 30 / 30 | 7.2openai/gpt-5.6-luna | 6.4anthropic/claude-haiku-5.5 | high |
| 33 | Granite4.2 8B | 85.1 | 30 / 30 | 7.8openai/gpt-5.6-luna | 7.1anthropic/claude-haiku-5.5 | high |
| 34 | Olmo 3.1 32B Instruct | 84.7 | 30 / 30 | 6.3openai/gpt-5.6-luna | 7.1anthropic/claude-haiku-5.5 | high |
| 35 | Sage-A1.5-Q4_K_M | 84.6 | 30 / 30 | 7.6openai/gpt-5.6-luna | 6.8anthropic/claude-haiku-5.5 | high |
| 36 | Qwen3-32B | 83.8 | 30 / 30 | 7.5openai/gpt-5.6-luna | 6.8anthropic/claude-haiku-5.5 | high |
| 37 | Nemotron3 33B | 83.7 | 30 / 30 | 7.5openai/gpt-5.6-luna | 7.1anthropic/claude-haiku-5.5 | high |
| 38 | Devstral Small 2 24B Instruct (2512)code model | 83.4 | 30 / 30 | 7.5openai/gpt-5.6-luna | 6.6anthropic/claude-haiku-5.5 | high |
| 39 | Mistral Small 24B Instruct 2501 | 82.8 | 30 / 30 | 7.5openai/gpt-5.6-luna | 6.6anthropic/claude-haiku-5.5 | high |
| 40 | Qwen2.5-Coder-14B-Instructcode model | 82.7 | 30 / 30 | 7.0openai/gpt-5.6-luna | 6.3anthropic/claude-haiku-5.5 | high |
| 41 | Qwen2.5-32B-Instruct | 82.0 | 30 / 30 | 7.2openai/gpt-5.6-luna | 6.3anthropic/claude-haiku-5.5 | high |
| 42 | gemma-3-12b-it | 81.8 | 30 / 30 | 7.3openai/gpt-5.6-luna | 6.8anthropic/claude-haiku-5.5 | high |
| 43 | GLM-4.7-Flash | 81.6 | 30 / 30 | 7.2openai/gpt-5.6-luna | 6.5anthropic/claude-haiku-5.5 | high |
| 44 | YI Coder 9B Chatcode model | 81.3 | 30 / 30 | 5.9openai/gpt-5.6-luna | 5.6anthropic/claude-haiku-5.5 | high |
| 45 | Llama 4 Scout 17B-16E Instruct | 81.0 | 30 / 30 | 7.1openai/gpt-5.6-luna | 6.4anthropic/claude-haiku-5.5 | high |
| 46 | Granite4 Tiny H | 80.8 | 30 / 30 | 7.3openai/gpt-5.6-luna | 6.4anthropic/claude-haiku-5.5 | high |
| 47 | Llama-3.3-70B-Instruct | 80.7 | 30 / 30 | 7.4openai/gpt-5.6-luna | 6.6anthropic/claude-haiku-5.5 | high |
| 48 | Gemma4 E2B IT | 80.1 | 30 / 30 | 7.1openai/gpt-5.6-luna | 6.3anthropic/claude-haiku-5.5 | high |
| 49 | Translategemma 12B ITfine-tune of Gemma 3translation model | 78.9 | 30 / 30 | 7.1openai/gpt-5.6-luna | 6.2anthropic/claude-haiku-5.5 | high |
| 50 | Cogito 14B V1 Preview Qwenfine-tune of Qwen 2.5 | 78.8 | 30 / 30 | 7.2openai/gpt-5.6-luna | 6.5anthropic/claude-haiku-5.5 | high |
| 51 | Ministral 3 8B Instruct 2512 | 78.4 | 30 / 30 | 7.2openai/gpt-5.6-luna | 6.5anthropic/claude-haiku-5.5 | high |
| 52 | Nemotron Cascade 2 30B A3B | 77.9 | 30 / 30 | 7.1openai/gpt-5.6-luna | 6.5anthropic/claude-haiku-5.5 | high |
| 53 | Qwen3-VL-8B-Instruct | 77.7 | 30 / 30 | 6.8openai/gpt-5.6-luna | 5.6anthropic/claude-haiku-5.5 | high |
| 54 | Qwen3.5-9B | 77.4 | 30 / 30 | 7.3openai/gpt-5.6-luna | 6.6anthropic/claude-haiku-5.5 | high |
| 55 | Mistral Small 3.2 24B Instruct (2506) | 77.2 | 30 / 30 | 6.8openai/gpt-5.6-luna | 6.0anthropic/claude-haiku-5.5 | high |
| 56 | Medgemma 27B ITfine-tune of Gemma 3medical model | 77.0 | 30 / 30 | 6.9openai/gpt-5.6-luna | 6.0anthropic/claude-haiku-5.5 | high |
| 57 | Qwen2.5-14B-Instruct | 76.5 | 30 / 30 | 7.2openai/gpt-5.6-luna | 6.2anthropic/claude-haiku-5.5 | high |
| 58 | Granite4.1 8B | 76.5 | 30 / 30 | 6.7openai/gpt-5.6-luna | 6.3anthropic/claude-haiku-5.5 | high |
| 59 | Qwen3-14B | 73.5 | 30 / 30 | 6.5openai/gpt-5.6-luna | 5.8anthropic/claude-haiku-5.5 | high |
| 60 | Mistral Small3.1 24B Instruct 2503 | 73.3 | 30 / 30 | 6.8openai/gpt-5.6-luna | 6.0anthropic/claude-haiku-5.5 | high |
| 61 | Nemotron 3.5 Bolt E4B | 73.2 | 30 / 30 | 6.4openai/gpt-5.6-luna | 6.1anthropic/claude-haiku-5.5 | high |
| 62 | Codestral 22B V0.1code model | 72.9 | 30 / 30 | 5.6openai/gpt-5.6-luna | 5.8anthropic/claude-haiku-5.5 | high |
| 63 | Lfm2 24B | 72.9 | 30 / 30 | 6.7openai/gpt-5.6-luna | 6.3anthropic/claude-haiku-5.5 | high |
| 64 | DeepSeek-Coder-V2-Lite-Instructcode model | 72.5 | 30 / 30 | 6.4openai/gpt-5.6-luna | 5.8anthropic/claude-haiku-5.5 | high |
| 65 | Qwen2.5-Coder-7B-Instructcode model | 71.8 | 30 / 30 | 6.7openai/gpt-5.6-luna | 5.8anthropic/claude-haiku-5.5 | high |
| 66 | Exaone3.5 32B Instruct | 71.5 | 30 / 30 | 6.6openai/gpt-5.6-luna | 5.7anthropic/claude-haiku-5.5 | high |
| 67 | Qwen3.5-4B | 71.5 | 30 / 30 | 6.7openai/gpt-5.6-luna | 5.8anthropic/claude-haiku-5.5 | high |
| 68 | gemma-3-4b-it | 71.4 | 30 / 30 | 6.5openai/gpt-5.6-luna | 5.6anthropic/claude-haiku-5.5 | high |
| 69 | Qwen3-8B | 70.6 | 30 / 30 | 6.1openai/gpt-5.6-luna | 5.6anthropic/claude-haiku-5.5 | high |
| 70 | Devstral 24B Small 2505code model | 70.4 | 30 / 30 | 6.5openai/gpt-5.6-luna | 5.7anthropic/claude-haiku-5.5 | high |
| 71 | Mistral Small 22B Instruct 2409 | 70.1 | 30 / 30 | 6.2openai/gpt-5.6-luna | 5.3anthropic/claude-haiku-5.5 | high |
| 72 | EXAONE-3.5-7.8B-Instruct | 70.1 | 30 / 30 | 6.4openai/gpt-5.6-luna | 5.6anthropic/claude-haiku-5.5 | high |
| 73 | Qwen3-4B-Instruct-2507 | 69.7 | 30 / 30 | 6.3openai/gpt-5.6-luna | 5.7anthropic/claude-haiku-5.5 | high |
| 74 | Codeqwen 7B Chat V1.5code model | 69.4 | 30 / 30 | 6.2openai/gpt-5.6-luna | 5.3anthropic/claude-haiku-5.5 | high |
| 75 | Qwen3-4B | 69.0 | 30 / 30 | 6.2openai/gpt-5.6-luna | 5.2anthropic/claude-haiku-5.5 | high |
| 76 | Granite4 Micro H | 68.8 | 30 / 30 | 6.2openai/gpt-5.6-luna | 5.3anthropic/claude-haiku-5.5 | high |
| 77 | Deepseek Coder 6.7B Instructcode model | 68.5 | 30 / 30 | 6.1openai/gpt-5.6-luna | 5.2anthropic/claude-haiku-5.5 | high |
| 78 | Ornith-1.5-9B | 67.9 | 30 / 30 | 6.2openai/gpt-5.6-luna | 5.4anthropic/claude-haiku-5.5 | high |
| 79 | Gemma2 27B Instruct | 67.5 | 30 / 30 | 6.1openai/gpt-5.6-luna | 5.3anthropic/claude-haiku-5.5 | high |
| 80 | Command R7B Arabic 7B 02 2025 | 67.4 | 30 / 30 | 5.3openai/gpt-5.6-luna | 4.4anthropic/claude-haiku-5.5 | high |
| 81 | Deepseek Coder 33B Instructcode model | 67.2 | 30 / 30 | 5.5openai/gpt-5.6-luna | 5.2anthropic/claude-haiku-5.5 | high |
| 82 | dolphin-2.9.1-yi-1.5-34b | 65.7 | 30 / 30 | 5.6openai/gpt-5.6-luna | 5.1anthropic/claude-haiku-5.5 | high |
| 83 | Llama-3.1-8B-Instruct | 65.6 | 30 / 30 | 6.2openai/gpt-5.6-luna | 5.3anthropic/claude-haiku-5.5 | high |
| 84 | Spark-X2.6-4B | 65.6 | 30 / 30 | 6.4openai/gpt-5.6-luna | 5.7anthropic/claude-haiku-5.5 | high |
| 85 | Gemma 3n E4B IT | 65.3 | 30 / 30 | 6.0openai/gpt-5.6-luna | 5.1anthropic/claude-haiku-5.5 | high |
| 86 | Exaone3.5 2.4B Instruct | 64.7 | 30 / 30 | 6.3openai/gpt-5.6-luna | 5.4anthropic/claude-haiku-5.5 | high |
| 87 | Marco O1 7Bfine-tune of Qwen 2 7B | 64.7 | 30 / 30 | 5.9openai/gpt-5.6-luna | 5.1anthropic/claude-haiku-5.5 | high |
| 88 | Aya Expanse 32B | 64.7 | 30 / 30 | 6.2openai/gpt-5.6-luna | 5.1anthropic/claude-haiku-5.5 | high |
| 89 | Granite4 3B | 64.7 | 30 / 30 | 6.3openai/gpt-5.6-luna | 5.5anthropic/claude-haiku-5.5 | high |
| 90 | Qwen2.5-VL-7B-Instruct | 64.4 | 30 / 30 | 5.6openai/gpt-5.6-luna | 4.6anthropic/claude-haiku-5.5 | high |
| 91 | Falcon3 10B Instruct | 64.3 | 30 / 30 | 5.6openai/gpt-5.6-luna | 5.3anthropic/claude-haiku-5.5 | high |
| 92 | Granite3.1 Dense 8B Instruct | 64.2 | 30 / 30 | 6.0openai/gpt-5.6-luna | 5.1anthropic/claude-haiku-5.5 | high |
| 93 | Command R 35B 08 2024 | 63.9 | 30 / 30 | 5.9openai/gpt-5.6-luna | 5.0anthropic/claude-haiku-5.5 | high |
| 94 | Qwen2.5-7B-Instruct | 63.2 | 30 / 30 | 5.9openai/gpt-5.6-luna | 5.1anthropic/claude-haiku-5.5 | high |
| 95 | Ornith 9B | 63.0 | 30 / 30 | 6.2openai/gpt-5.6-luna | 5.5anthropic/claude-haiku-5.5 | high |
| 96 | granite-4.1-3b | 62.6 | 30 / 30 | 5.4openai/gpt-5.6-luna | 5.3anthropic/claude-haiku-5.5 | high |
| 97 | Granite4.2 3B | 61.6 | 30 / 30 | 6.0openai/gpt-5.6-luna | 5.4anthropic/claude-haiku-5.5 | high |
| 98 | Lfm3 1.7B | 61.5 | 30 / 30 | 5.7openai/gpt-5.6-luna | 5.2anthropic/claude-haiku-5.5 | high |
| 99 | Phi3 14B Medium 128K Instruct | 61.4 | 30 / 30 | 5.0openai/gpt-5.6-luna | 5.1anthropic/claude-haiku-5.5 | high |
| 100 | Gemma3n E2B IT | 61.4 | 30 / 30 | 5.7openai/gpt-5.6-luna | 4.7anthropic/claude-haiku-5.5 | high |
| 101 | Qwen3-VL-4B-Instruct | 61.2 | 30 / 30 | 5.8openai/gpt-5.6-luna | 5.0anthropic/claude-haiku-5.5 | high |
| 102 | Phi-4-mini-instruct | 61.0 | 30 / 30 | 5.9openai/gpt-5.6-luna | 5.2anthropic/claude-haiku-5.5 | high |
| 103 | Ministral 3 3B Instruct 2512 | 60.6 | 30 / 30 | 5.5openai/gpt-5.6-luna | 4.9anthropic/claude-haiku-5.5 | high |
| 104 | QwQ 32B | 60.6 | 30 / 30 | 5.6openai/gpt-5.6-luna | 5.5anthropic/claude-haiku-5.5 | high |
| 105 | Codegeex4 9B Allcode model | 60.5 | 30 / 30 | 5.7openai/gpt-5.6-luna | 5.1anthropic/claude-haiku-5.5 | high |
| 106 | medgemma-4b-itmedical model | 60.3 | 30 / 30 | 5.8openai/gpt-5.6-luna | 4.8anthropic/claude-haiku-5.5 | high |
| 107 | Qwen2.5-Coder-3B-Instructcode model | 60.2 | 30 / 30 | 5.4openai/gpt-5.6-luna | 4.6anthropic/claude-haiku-5.5 | high |
| 108 | Meta-Llama-3-8B-Instruct | 60.0 | 30 / 30 | 5.6openai/gpt-5.6-luna | 4.7anthropic/claude-haiku-5.5 | high |
| 109 | Glm4 9B Chat | 60.0 | 30 / 30 | 5.7openai/gpt-5.6-luna | 4.8anthropic/claude-haiku-5.5 | high |
| 110 | Granite4 1B H | 59.9 | 30 / 30 | 5.2openai/gpt-5.6-luna | 4.9anthropic/claude-haiku-5.5 | high |
| 111 | Command R7B (12-2024) | 59.4 | 30 / 30 | 5.0openai/gpt-5.6-luna | 4.3anthropic/claude-haiku-5.5 | high |
| 112 | Granite3 Dense 8B Instruct | 59.2 | 30 / 30 | 5.0openai/gpt-5.6-luna | 4.2anthropic/claude-haiku-5.5 | high |
| 113 | Falcon3 7B Instruct | 58.7 | 30 / 30 | 5.6openai/gpt-5.6-luna | 4.8anthropic/claude-haiku-5.5 | high |
| 114 | Granite3.2 8B Instruct | 58.5 | 30 / 30 | 5.8openai/gpt-5.6-luna | 4.8anthropic/claude-haiku-5.5 | high |
| 115 | Olmo-3-7B-Instruct | 57.7 | 30 / 30 | 4.8openai/gpt-5.6-luna | 5.0anthropic/claude-haiku-5.5 | high |
| 116 | SmolLM3-3B | 57.7 | 30 / 30 | 5.5openai/gpt-5.6-luna | 4.9anthropic/claude-haiku-5.5 | high |
| 117 | YI Coder 1.5B Chatcode model | 57.7 | 30 / 30 | 4.7openai/gpt-5.6-luna | 3.8anthropic/claude-haiku-5.5 | high |
| 118 | YI 9B Chat V1.5 | 57.5 | 30 / 30 | 4.8openai/gpt-5.6-luna | 4.3anthropic/claude-haiku-5.5 | high |
| 119 | Opencoder 1.5B Instructcode model | 57.3 | 30 / 30 | 4.7openai/gpt-5.6-luna | 4.1anthropic/claude-haiku-5.5 | high |
| 120 | Mistral Nemo 12B Instruct 2407 | 57.3 | 30 / 30 | 5.0openai/gpt-5.6-luna | 4.8anthropic/claude-haiku-5.5 | high |
| 121 | Medgemma1.5 4B ITfine-tune of Gemma 3medical model | 56.9 | 30 / 30 | 4.9openai/gpt-5.6-luna | 4.4anthropic/claude-haiku-5.5 | high |
| 122 | Granite3.3 8B | 56.6 | 30 / 30 | 5.2openai/gpt-5.6-luna | 4.5anthropic/claude-haiku-5.5 | high |
| 123 | Mathstral 7B V0.1fine-tune of Mistral 7Bmath model | 56.6 | 30 / 30 | 4.5openai/gpt-5.6-luna | 3.7anthropic/claude-haiku-5.5 | high |
| 124 | Cogito 8B V1 Preview Llamafine-tune of Llama 3.1 | 56.1 | 30 / 30 | 5.3openai/gpt-5.6-luna | 4.5anthropic/claude-haiku-5.5 | high |
| 125 | Granite Code 8B Instructcode model | 55.7 | 30 / 30 | 4.5openai/gpt-5.6-luna | 3.8anthropic/claude-haiku-5.5 | high |
| 126 | Codegemma 7B Instruct V1.1code model | 55.5 | 30 / 30 | 5.4openai/gpt-5.6-luna | 4.4anthropic/claude-haiku-5.5 | high |
| 127 | Exaone Deep 7.8B | 55.4 | 30 / 30 | 3.8openai/gpt-5.6-luna | 3.8anthropic/claude-haiku-5.5 | high |
| 128 | Deepseek Coder 1.3B Instructcode model | 54.5 | 30 / 30 | 4.5openai/gpt-5.6-luna | 3.7anthropic/claude-haiku-5.5 | high |
| 129 | DeepSeek-R1-Distill-Qwen-32B | 53.6 | 30 / 30 | 3.8openai/gpt-5.6-luna | 3.3anthropic/claude-haiku-5.5 | high |
| 130 | DeepSeek-R1-Distill-Qwen-14B | 53.0 | 30 / 30 | 3.7openai/gpt-5.6-luna | 3.3anthropic/claude-haiku-5.5 | high |
| 131 | Llama-3.2-3B-Instruct | 52.9 | 30 / 30 | 4.8openai/gpt-5.6-luna | 4.5anthropic/claude-haiku-5.5 | high |
| 132 | Deepseek R1 8B Llama Distill | 52.8 | 30 / 30 | 3.9openai/gpt-5.6-luna | 3.1anthropic/claude-haiku-5.5 | high |
| 133 | Qwen2.5 Coder 1.5B Instructcode model | 52.8 | 30 / 30 | 5.3openai/gpt-5.6-luna | 4.4anthropic/claude-haiku-5.5 | high |
| 134 | Tulu3 8Bfine-tune of Llama 3.1 | 52.7 | 30 / 30 | 4.7openai/gpt-5.6-luna | 4.2anthropic/claude-haiku-5.5 | high |
| 135 | Openthinker 7B V2fine-tune of Qwen 2.5 | 52.6 | 30 / 30 | 3.7openai/gpt-5.6-luna | 2.5anthropic/claude-haiku-5.5 | medium |
| 136 | Qwen3.5 2B | 52.5 | 30 / 30 | 5.2openai/gpt-5.6-luna | 4.4anthropic/claude-haiku-5.5 | high |
| 137 | gemma-2-9b-it | 51.8 | 30 / 30 | 4.7openai/gpt-5.6-luna | 4.1anthropic/claude-haiku-5.5 | high |
| 138 | YI 34B Chat V1.5 | 50.8 | 30 / 30 | 4.3openai/gpt-5.6-luna | 4.2anthropic/claude-haiku-5.5 | high |
| 139 | Granite3.3 2B | 50.4 | 30 / 30 | 4.6openai/gpt-5.6-luna | 3.8anthropic/claude-haiku-5.5 | high |
| 140 | Aya Expanse 8B | 50.0 | 30 / 30 | 4.4openai/gpt-5.6-luna | 4.0anthropic/claude-haiku-5.5 | high |
| 141 | Qwen2.5 1.5B Instruct | 49.9 | 30 / 30 | 4.9openai/gpt-5.6-luna | 4.1anthropic/claude-haiku-5.5 | high |
| 142 | Qwen2-VL-7B-Instruct | 49.7 | 30 / 30 | 4.7openai/gpt-5.6-luna | 4.1anthropic/claude-haiku-5.5 | high |
| 143 | Cogito 3B V1 Preview Llamafine-tune of Llama 3.2 | 49.3 | 30 / 30 | 4.7openai/gpt-5.6-luna | 3.9anthropic/claude-haiku-5.5 | high |
| 144 | Qwen3 VL 2B Instruct | 49.2 | 30 / 30 | 5.3openai/gpt-5.6-luna | 4.3anthropic/claude-haiku-5.5 | high |
| 145 | Qwen2.5-3B-Instruct | 48.7 | 30 / 30 | 4.6openai/gpt-5.6-luna | 3.8anthropic/claude-haiku-5.5 | high |
| 146 | Internlm2 7B Chat V2.5 | 48.7 | 30 / 30 | 4.2openai/gpt-5.6-luna | 4.2anthropic/claude-haiku-5.5 | high |
| 147 | Internlm2 1.8B Chat V2.5 | 48.5 | 30 / 30 | 4.9openai/gpt-5.6-luna | 4.1anthropic/claude-haiku-5.5 | high |
| 148 | Granite3.2 2B Instruct | 48.4 | 30 / 30 | 5.1openai/gpt-5.6-luna | 4.3anthropic/claude-haiku-5.5 | high |
| 149 | Lfm2.5 8B A1B | 47.3 | 30 / 30 | 3.4openai/gpt-5.6-luna | 3.6anthropic/claude-haiku-5.5 | high |
| 150 | Granite3.1 Dense 2B Instruct | 46.6 | 30 / 30 | 4.8openai/gpt-5.6-luna | 4.2anthropic/claude-haiku-5.5 | high |
| 151 | Granite Code 3B Instructcode model | 45.6 | 30 / 30 | 4.0openai/gpt-5.6-luna | 3.0anthropic/claude-haiku-5.5 | high |
| 152 | Qwen3 1.7B | 45.5 | 30 / 30 | 4.2openai/gpt-5.6-luna | 3.8anthropic/claude-haiku-5.5 | high |
| 153 | Stablelm2 12B Chat | 45.0 | 30 / 30 | 4.2openai/gpt-5.6-luna | 3.4anthropic/claude-haiku-5.5 | high |
| 154 | Qwen2.5-VL-3B-Instruct | 42.6 | 30 / 30 | 4.2openai/gpt-5.6-luna | 3.6anthropic/claude-haiku-5.5 | high |
| 155 | Gemma3 1B IT | 42.5 | 30 / 30 | 4.3openai/gpt-5.6-luna | 3.5anthropic/claude-haiku-5.5 | high |
| 156 | Gemma2 2B Instruct | 42.4 | 30 / 30 | 4.2openai/gpt-5.6-luna | 3.6anthropic/claude-haiku-5.5 | high |
| 157 | Hermes-3-Llama-3.1-8B | 42.3 | 30 / 30 | 3.9openai/gpt-5.6-luna | 4.1anthropic/claude-haiku-5.5 | high |
| 158 | Lfm3 VL 2B | 42.2 | 30 / 30 | 4.4openai/gpt-5.6-luna | 3.7anthropic/claude-haiku-5.5 | high |
| 159 | Granite3 Dense 2B Instruct | 42.2 | 30 / 30 | 4.4openai/gpt-5.6-luna | 3.6anthropic/claude-haiku-5.5 | high |
| 160 | DeepSeek-R1-0528-Qwen3-8B | 42.1 | 30 / 30 | 3.5openai/gpt-5.6-luna | 3.3anthropic/claude-haiku-5.5 | high |
| 161 | Lfm2.5 Thinking 1.2B | 42.1 | 30 / 30 | 3.5openai/gpt-5.6-luna | 3.7anthropic/claude-haiku-5.5 | high |
| 162 | Llama3.2 1B Instruct | 41.7 | 30 / 30 | 4.5openai/gpt-5.6-luna | 3.6anthropic/claude-haiku-5.5 | high |
| 163 | Deepseek V2 16B Lite Chat | 41.6 | 30 / 30 | 4.2openai/gpt-5.6-luna | 3.4anthropic/claude-haiku-5.5 | high |
| 164 | Mistral 7B Instruct V0.3 | 40.9 | 30 / 30 | 3.7openai/gpt-5.6-luna | 3.0anthropic/claude-haiku-5.5 | high |
| 165 | Falcon3 3B Instruct | 40.9 | 30 / 30 | 4.1openai/gpt-5.6-luna | 3.5anthropic/claude-haiku-5.5 | high |
| 166 | Olmo2 13B 1124 Instruct | 40.8 | 30 / 30 | 4.3openai/gpt-5.6-luna | 3.1anthropic/claude-haiku-5.5 | high |
| 167 | Olmo-3-7B-Think | 40.0 | 30 / 30 | 3.8openai/gpt-5.6-luna | 3.6anthropic/claude-haiku-5.5 | high |
| 168 | Qwen2 Math 7B Instructmath model | 39.1 | 30 / 30 | 3.8openai/gpt-5.6-luna | 3.5anthropic/claude-haiku-5.5 | high |
| 169 | Phi-3.5-mini-instruct | 38.9 | 30 / 30 | 3.9openai/gpt-5.6-luna | 3.6anthropic/claude-haiku-5.5 | high |
| 170 | Codellama 34B Instructcode model | 38.7 | 30 / 30 | 3.6openai/gpt-5.6-luna | 3.5anthropic/claude-haiku-5.5 | high |
| 171 | Phi3 3.8B Mini 128K Instruct | 38.6 | 30 / 30 | 3.7openai/gpt-5.6-luna | 3.6anthropic/claude-haiku-5.5 | high |
| 172 | Granite3.1 Moe 1B Instruct | 38.1 | 30 / 30 | 4.2openai/gpt-5.6-luna | 3.5anthropic/claude-haiku-5.5 | high |
| 173 | Deepseek Llm 7B Chat | 37.4 | 30 / 30 | 3.8openai/gpt-5.6-luna | 3.1anthropic/claude-haiku-5.5 | high |
| 174 | DeepSeek-R1-Distill-Qwen-7B | 37.0 | 30 / 30 | 2.7openai/gpt-5.6-luna | 2.4anthropic/claude-haiku-5.5 | high |
| 175 | Exaone Deep 2.4B | 36.7 | 30 / 30 | 2.6openai/gpt-5.6-luna | 2.7anthropic/claude-haiku-5.5 | high |
| 176 | Codellama 7B Instructcode model | 36.0 | 30 / 30 | 3.5openai/gpt-5.6-luna | 2.9anthropic/claude-haiku-5.5 | high |
| 177 | Granite4 350M H | 36.0 | 30 / 30 | 3.4openai/gpt-5.6-luna | 3.0anthropic/claude-haiku-5.5 | high |
| 178 | Qwen2.5 Coder 0.5B Instructcode model | 35.9 | 30 / 30 | 3.9openai/gpt-5.6-luna | 3.0anthropic/claude-haiku-5.5 | high |
| 179 | Smollm2 1.7B Instruct | 35.3 | 30 / 30 | 3.7openai/gpt-5.6-luna | 2.9anthropic/claude-haiku-5.5 | high |
| 180 | Phi-3-mini-4k-instruct | 35.1 | 30 / 30 | 3.4openai/gpt-5.6-luna | 3.0anthropic/claude-haiku-5.5 | high |
| 181 | Phi 2.7B Chat V2 | 34.7 | 30 / 30 | 3.1openai/gpt-5.6-luna | 2.8anthropic/claude-haiku-5.5 | high |
| 182 | Nemotron Mini 4B Instruct | 34.6 | 30 / 30 | 3.6openai/gpt-5.6-luna | 2.7anthropic/claude-haiku-5.5 | high |
| 183 | Solar 10.7B Instruct V1 | 34.3 | 30 / 30 | 3.5openai/gpt-5.6-luna | 2.8anthropic/claude-haiku-5.5 | high |
| 184 | Deepcoder 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bcode model | 34.2 | 30 / 30 | 2.4openai/gpt-5.6-luna | 2.3anthropic/claude-haiku-5.5 | high |
| 185 | Apertus-8B-Instruct-2509 | 33.0 | 30 / 30 | 3.4openai/gpt-5.6-luna | 2.8anthropic/claude-haiku-5.5 | high |
| 186 | Granite3.1 Moe 3B Instruct | 32.6 | 30 / 30 | 3.6openai/gpt-5.6-luna | 2.8anthropic/claude-haiku-5.5 | high |
| 187 | Qwen3 0.6B | 32.5 | 30 / 30 | 3.5openai/gpt-5.6-luna | 2.7anthropic/claude-haiku-5.5 | high |
| 188 | Phi4 Mini Reasoning 3.8B | 31.8 | 30 / 30 | 2.3openai/gpt-5.6-luna | 1.9anthropic/claude-haiku-5.5 | high |
| 189 | Granite3 Moe 3B Instruct | 31.6 | 30 / 30 | 3.4openai/gpt-5.6-luna | 2.3anthropic/claude-haiku-5.5 | high |
| 190 | Aya 8B 23 | 30.7 | 30 / 30 | 3.4openai/gpt-5.6-luna | 2.5anthropic/claude-haiku-5.5 | high |
| 191 | Qwen2 1.5B Instruct | 30.6 | 30 / 30 | 3.3openai/gpt-5.6-luna | 2.9anthropic/claude-haiku-5.5 | high |
| 192 | Smallthinker 3B Previewfine-tune of Qwen 2.5 3B | 29.8 | 30 / 30 | 2.1openai/gpt-5.6-luna | 1.3anthropic/claude-haiku-5.5 | high |
| 193 | Stablelm Zephyr 3Bfine-tune of StableLM 3B | 29.3 | 30 / 30 | 3.4openai/gpt-5.6-luna | 2.4anthropic/claude-haiku-5.5 | high |
| 194 | Olmo2 7B 1124 Instruct | 29.3 | 30 / 30 | 3.4openai/gpt-5.6-luna | 2.8anthropic/claude-haiku-5.5 | high |
| 195 | Dolphin Phi 2.7B V2.6fine-tune of Phi-2 | 28.6 | 30 / 30 | 2.3openai/gpt-5.6-luna | 2.3anthropic/claude-haiku-5.5 | high |
| 196 | Granite3 Moe 1B Instruct | 27.6 | 30 / 30 | 3.0openai/gpt-5.6-luna | 1.8anthropic/claude-haiku-5.5 | high |
| 197 | Deepscaler 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bmath model | 26.7 | 30 / 30 | 1.9openai/gpt-5.6-luna | 1.7anthropic/claude-haiku-5.5 | high |
| 197 | Qwen3-4B-Thinking-2507 | 26.7 | 30 / 30 | 2.0openai/gpt-5.6-luna | 2.3anthropic/claude-haiku-5.5 | high |
| 199 | Qwen2.5 0.5B Instruct | 26.5 | 30 / 30 | 2.4openai/gpt-5.6-luna | 2.1anthropic/claude-haiku-5.5 | high |
| 200 | Gemma 7B Instruct V1.1 | 26.5 | 30 / 30 | 2.8openai/gpt-5.6-luna | 2.4anthropic/claude-haiku-5.5 | high |
| 201 | Stablelm2 1.6B Chat | 26.1 | 30 / 30 | 2.4openai/gpt-5.6-luna | 1.9anthropic/claude-haiku-5.5 | high |
| 202 | Mistral-7B-Instruct-v0.2 | 25.3 | 30 / 30 | 2.7openai/gpt-5.6-luna | 2.3anthropic/claude-haiku-5.5 | high |
| 203 | Gemma 2B Instruct V1.1 | 25.2 | 30 / 30 | 3.0openai/gpt-5.6-luna | 2.0anthropic/claude-haiku-5.5 | high |
| 204 | Deepseek R1 1.5B Qwen Distill | 24.4 | 30 / 30 | 1.8openai/gpt-5.6-luna | 1.5anthropic/claude-haiku-5.5 | high |
| 205 | Smollm2 360M Instruct | 23.8 | 30 / 30 | 2.7openai/gpt-5.6-luna | 2.0anthropic/claude-haiku-5.5 | high |
| 206 | Codellama 13B Instructcode model | 22.5 | 30 / 30 | 2.1openai/gpt-5.6-luna | 1.9anthropic/claude-haiku-5.5 | high |
| 207 | Mixtral 8x7B Instruct V0.1 | 20.1 | 30 / 30 | 2.0openai/gpt-5.6-luna | 2.0anthropic/claude-haiku-5.5 | high |
| 208 | Deepcoder 14B Previewfine-tune of DeepSeek-R1-Distill-Qwen 14Bcode model | 20.0 | 30 / 30 | 1.4openai/gpt-5.6-luna | 1.2anthropic/claude-haiku-5.5 | high |
| 209 | YI 6B Chat V1.5 | 19.8 | 30 / 30 | 1.7openai/gpt-5.6-luna | 1.8anthropic/claude-haiku-5.5 | high |
| 210 | Sailor2 8B Chatfine-tune of Qwen 2.5 | 19.5 | 30 / 30 | 1.7openai/gpt-5.6-luna | 1.7anthropic/claude-haiku-5.5 | high |
| 211 | Falcon3 1B Instruct | 18.7 | 30 / 30 | 1.8openai/gpt-5.6-luna | 1.5anthropic/claude-haiku-5.5 | high |
| 212 | Llama-2-7b-chat-hf | 18.6 | 30 / 30 | 2.4openai/gpt-5.6-luna | 1.7anthropic/claude-haiku-5.5 | high |
| 213 | Qwen2 Math 1.5B Instructmath model | 17.5 | 30 / 30 | 1.4openai/gpt-5.6-luna | 1.0anthropic/claude-haiku-5.5 | high |
| 214 | Falcon2 11Bnot in the Overall ranking | 16.9 | 30 / 30 | 1.4openai/gpt-5.6-luna | 1.0anthropic/claude-haiku-5.5 | high |
| 215 | Smollm2 135M Instruct | 14.1 | 30 / 30 | 1.6openai/gpt-5.6-luna | 1.2anthropic/claude-haiku-5.5 | high |
| 216 | Hermes3 3B Llama3.2fine-tune of Llama 3.2 | 11.3 | 30 / 30 | 1.1openai/gpt-5.6-luna | 1.3anthropic/claude-haiku-5.5 | high |
| 217 | Sailor2 1B Chatfine-tune of Qwen 2.5 | 10.5 | 30 / 30 | 1.9openai/gpt-5.6-luna | 1.0anthropic/claude-haiku-5.5 | high |
| 218 | Llama2 13B Chat | 10.1 | 30 / 30 | 1.2openai/gpt-5.6-luna | 1.0anthropic/claude-haiku-5.5 | high |
| 219 | Magistral Small (2506) | 10.0 | 30 / 30 | 0.7openai/gpt-5.6-luna | 0.7anthropic/claude-haiku-5.5 | high |
| 220 | Qwen2 0.5B Instruct | 9.8 | 30 / 30 | 0.8openai/gpt-5.6-luna | 0.9anthropic/claude-haiku-5.5 | high |
| 221 | Qwen 14B Chat V1.5 | 9.7 | 30 / 30 | 1.0openai/gpt-5.6-luna | 0.9anthropic/claude-haiku-5.5 | high |
| 222 | Qwen 0.5B Chat V1.5 | 9.0 | 30 / 30 | 1.0openai/gpt-5.6-luna | 0.6anthropic/claude-haiku-5.5 | high |
| 223 | Stable Code 3B Instructcode model | 6.5 | 30 / 30 | 0.8openai/gpt-5.6-luna | 0.7anthropic/claude-haiku-5.5 | high |
| 224 | Qwen 7B Chat V1.5 | 5.9 | 30 / 30 | 0.7openai/gpt-5.6-luna | 0.6anthropic/claude-haiku-5.5 | high |
| 225 | Codegemma 2B Code V1.1code model | 3.9 | 30 / 30 | 0.3openai/gpt-5.6-luna | 0.4anthropic/claude-haiku-5.5 | high |
| 226 | Qwen 4B Chat V1.5 | 3.1 | 30 / 30 | 0.4openai/gpt-5.6-luna | 0.3anthropic/claude-haiku-5.5 | high |
| 227 | Qwen 1.8B Chat V1.5 | 2.4 | 30 / 30 | 0.3openai/gpt-5.6-luna | 0.3anthropic/claude-haiku-5.5 | high |
| 228 | Qwen3 VL 2B Thinking | 0 | 30 / 30 | 0openai/gpt-5.6-luna | 0anthropic/claude-haiku-5.5 | high |
A dash means not scored, which is different from 0. Agreement compares the two judges: high means a mean gap under 1.5 points, medium under 3, low otherwise; n/a means fewer than 3 items were scored by both.
Not ranked in this category
- Apertus-8B-Instruct-2509: withheld from the ranking (see its page)
- Falcon3 10B Instruct: withheld from the ranking (see its page)
- Falcon3 1B Instruct: withheld from the ranking (see its page)
- Falcon3 3B Instruct: withheld from the ranking (see its page)
- Falcon3 7B Instruct: withheld from the ranking (see its page)
- Internlm2 1.8B Chat V2.5: withheld from the ranking (see its page)
- Phi-4-reasoning 14B: withheld from the ranking (see its page)
- Phi-4-reasoning-plus 14B: withheld from the ranking (see its page)
- Qwen2 Math 1.5B Instruct: withheld from the ranking (see its page)
- Qwen2 Math 7B Instruct: withheld from the ranking (see its page)
The questions
In run r20261102-bv1-5246ab, 2 of the 30 questions in this category were answered correctly by every ranked model that has a score for them (33 ranked models in that run). Hard questions are the ones that half or fewer of the ranked models of 3B and over answered correctly. This category has 16.
The same ranking as JSON: coding.json.