Instruction following
Writing tasks with explicit rules, such as length, format, required words or the language of the reply. Code checks each rule.
This category counts toward Overall. The two judge models also score every item, and their scores are shown here separately. How the battery works.
Ranking
Models are ranked by their mean score in this category, from 0 to 100. Only models with at least 80% of the category's items scored, and none pending, are ranked here. The rule is the same as for Overall.
| Rank | Model | Mean (0 to 100) | Items scored | Judge A mean | Judge B mean | Agreement |
|---|---|---|---|---|---|---|
| 1 | Gemma 4 12B IT | 100.0 | 30 / 30 | 9.5openai/gpt-5.6-luna | 9.0anthropic/claude-haiku-5.5 | high |
| 1 | gemma-4-26B-A4B-it | 100.0 | 30 / 30 | 9.3openai/gpt-5.6-luna | 9.0anthropic/claude-haiku-5.5 | high |
| 1 | gemma-4-31B-it | 100.0 | 30 / 30 | 9.5openai/gpt-5.6-luna | 8.9anthropic/claude-haiku-5.5 | high |
| 1 | gpt-oss-120b | 100.0 | 30 / 30 | 9.0anthropic/claude-haiku-5.5 | 8.9deepseek/deepseek-v4.1-flash | high |
| 1 | Muse Glimmer 30B | 100.0 | 30 / 30 | 9.5openai/gpt-5.6-luna | 8.9anthropic/claude-haiku-5.5 | high |
| 1 | Nemotron 3.5 Bolt E4B | 100.0 | 30 / 30 | 9.2openai/gpt-5.6-luna | 8.9anthropic/claude-haiku-5.5 | high |
| 1 | Qwen3.5-27B | 100.0 | 30 / 30 | 9.4openai/gpt-5.6-luna | 8.9anthropic/claude-haiku-5.5 | high |
| 8 | Qwen3.8-27B | 98.9 | 30 / 30 | 9.3openai/gpt-5.6-luna | 8.7anthropic/claude-haiku-5.5 | high |
| 9 | Llama 4 Scout 17B-16E Instruct | 98.7 | 30 / 30 | 9.2openai/gpt-5.6-luna | 8.7anthropic/claude-haiku-5.5 | high |
| 10 | Qwen3-VL-8B-Instruct | 98.5 | 30 / 30 | 9.4openai/gpt-5.6-luna | 8.7anthropic/claude-haiku-5.5 | high |
| 11 | Olmo-3-7B-Think | 98.3 | 30 / 30 | 9.1openai/gpt-5.6-luna | 8.7anthropic/claude-haiku-5.5 | high |
| 11 | Qwen3.6-27B | 98.3 | 30 / 30 | 9.2openai/gpt-5.6-luna | 8.8anthropic/claude-haiku-5.5 | high |
| 13 | Gemma2 27B Instruct | 98.2 | 30 / 30 | 9.2openai/gpt-5.6-luna | 8.6anthropic/claude-haiku-5.5 | high |
| 14 | Nemotron Cascade 2 30B A3B | 97.8 | 30 / 30 | 9.2openai/gpt-5.6-luna | 8.5anthropic/claude-haiku-5.5 | high |
| 15 | Llama-3.3-70B-Instruct | 97.5 | 30 / 30 | 8.9openai/gpt-5.6-luna | 8.3anthropic/claude-haiku-5.5 | high |
| 16 | Gemma4 E4B IT | 97.2 | 30 / 30 | 9.1openai/gpt-5.6-luna | 8.7anthropic/claude-haiku-5.5 | high |
| 17 | Olmo-3-7B-Instruct | 96.7 | 30 / 30 | 9.1openai/gpt-5.6-luna | 8.4anthropic/claude-haiku-5.5 | high |
| 18 | Flint 40B A2B | 96.6 | 30 / 30 | 9.1openai/gpt-5.6-luna | 8.4anthropic/claude-haiku-5.5 | high |
| 18 | Lfm2 24B | 96.6 | 30 / 30 | 8.9openai/gpt-5.6-luna | 8.3anthropic/claude-haiku-5.5 | high |
| 18 | Qwen3-VL-4B-Instruct | 96.6 | 30 / 30 | 9.0openai/gpt-5.6-luna | 8.3anthropic/claude-haiku-5.5 | high |
| 18 | Qwen3.6-35B-A3B | 96.6 | 30 / 30 | 9.1openai/gpt-5.6-luna | 8.3anthropic/claude-haiku-5.5 | high |
| 22 | NVIDIA Nemotron 3 Nano 30B-A3B | 96.5 | 30 / 30 | 9.1openai/gpt-5.6-luna | 8.3anthropic/claude-haiku-5.5 | high |
| 23 | Qwen3-32B | 96.3 | 30 / 30 | 9.1openai/gpt-5.6-luna | 8.1anthropic/claude-haiku-5.5 | high |
| 24 | Qwen3-VL-32B-Instruct | 96.1 | 30 / 30 | 8.9openai/gpt-5.6-luna | 8.1anthropic/claude-haiku-5.5 | high |
| 25 | gpt-oss-20b | 95.8 | 30 / 30 | 8.3anthropic/claude-haiku-5.5 | 8.0deepseek/deepseek-v4.1-flash | high |
| 26 | Gemma4 E2B IT | 95.4 | 30 / 30 | 8.8openai/gpt-5.6-luna | 8.3anthropic/claude-haiku-5.5 | high |
| 27 | Olmo 3.1 32B Instruct | 95.2 | 30 / 30 | 9.0openai/gpt-5.6-luna | 8.2anthropic/claude-haiku-5.5 | high |
| 28 | Qwen3-Coder-Nextcode model | 95.2 | 30 / 30 | 9.1openai/gpt-5.6-luna | 8.0anthropic/claude-haiku-5.5 | high |
| 29 | QwQ 32B | 95.1 | 30 / 30 | 9.1openai/gpt-5.6-luna | 8.0anthropic/claude-haiku-5.5 | high |
| 30 | Qwen2.5-Coder-32B-Instructcode model | 94.8 | 30 / 30 | 8.8openai/gpt-5.6-luna | 8.2anthropic/claude-haiku-5.5 | high |
| 31 | Sage-A1.5-Q4_K_M | 94.6 | 30 / 30 | 8.7openai/gpt-5.6-luna | 7.9anthropic/claude-haiku-5.5 | high |
| 32 | gemma-3-27b-it | 94.5 | 30 / 30 | 8.8openai/gpt-5.6-luna | 7.9anthropic/claude-haiku-5.5 | high |
| 33 | GLM-4.5-Air | 94.3 | 30 / 30 | 8.8openai/gpt-5.6-luna | 8.1anthropic/claude-haiku-5.5 | high |
| 34 | Qwen3-4B-Instruct-2507 | 94.2 | 30 / 30 | 8.9openai/gpt-5.6-luna | 8.2anthropic/claude-haiku-5.5 | high |
| 35 | Qwen2.5-32B-Instruct | 94.1 | 30 / 30 | 8.8openai/gpt-5.6-luna | 7.9anthropic/claude-haiku-5.5 | high |
| 36 | Granite4.2 8B | 94.1 | 30 / 30 | 8.7openai/gpt-5.6-luna | 7.9anthropic/claude-haiku-5.5 | high |
| 36 | Qwen3.5-9B | 94.1 | 30 / 30 | 8.7openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | high |
| 38 | granite-4.1-3b | 93.9 | 30 / 30 | 8.5openai/gpt-5.6-luna | 7.7anthropic/claude-haiku-5.5 | high |
| 39 | Granite4.1 30B | 93.8 | 30 / 30 | 8.6openai/gpt-5.6-luna | 7.9anthropic/claude-haiku-5.5 | high |
| 40 | Nemotron 3.5 Lightning 30B A3B | 93.7 | 30 / 30 | 8.7openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | high |
| 41 | Qwen3-14B | 93.6 | 30 / 30 | 8.6openai/gpt-5.6-luna | 8.1anthropic/claude-haiku-5.5 | high |
| 42 | Qwen3.5-35B-A3B | 93.5 | 30 / 30 | 8.8openai/gpt-5.6-luna | 7.9anthropic/claude-haiku-5.5 | high |
| 43 | Granite4 Small H | 93.4 | 30 / 30 | 8.6openai/gpt-5.6-luna | 7.9anthropic/claude-haiku-5.5 | high |
| 44 | Laguna XS 2.1 | 93.3 | 30 / 30 | 8.6openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | high |
| 45 | Cogito 8B V1 Preview Llamafine-tune of Llama 3.1 | 93.1 | 30 / 30 | 8.6openai/gpt-5.6-luna | 7.6anthropic/claude-haiku-5.5 | high |
| 45 | Granite4.1 8B | 93.1 | 30 / 30 | 8.8openai/gpt-5.6-luna | 7.9anthropic/claude-haiku-5.5 | high |
| 47 | Qwen3-Coder-30B-A3B-Instructcode model | 92.9 | 30 / 30 | 8.8openai/gpt-5.6-luna | 7.6anthropic/claude-haiku-5.5 | high |
| 48 | Granite4.2 30B | 92.8 | 30 / 30 | 8.7openai/gpt-5.6-luna | 7.6anthropic/claude-haiku-5.5 | high |
| 49 | gemma-2-9b-it | 92.6 | 30 / 30 | 8.5openai/gpt-5.6-luna | 7.6anthropic/claude-haiku-5.5 | high |
| 49 | Llama-3.1-8B-Instruct | 92.6 | 30 / 30 | 8.4openai/gpt-5.6-luna | 7.9anthropic/claude-haiku-5.5 | high |
| 49 | Nemotron3 33B | 92.6 | 30 / 30 | 8.6openai/gpt-5.6-luna | 7.5anthropic/claude-haiku-5.5 | high |
| 52 | Qwen3-30B-A3B | 92.3 | 30 / 30 | 8.6openai/gpt-5.6-luna | 7.7anthropic/claude-haiku-5.5 | high |
| 53 | Ornith 9B | 91.9 | 30 / 30 | 8.6openai/gpt-5.6-luna | 8.0anthropic/claude-haiku-5.5 | high |
| 54 | Llama-3.2-3B-Instruct | 91.7 | 30 / 30 | 8.3openai/gpt-5.6-luna | 7.3anthropic/claude-haiku-5.5 | high |
| 55 | Qwen2.5-14B-Instruct | 91.3 | 30 / 30 | 8.4openai/gpt-5.6-luna | 7.4anthropic/claude-haiku-5.5 | high |
| 56 | Granite4.2 3B | 91.2 | 30 / 30 | 8.4openai/gpt-5.6-luna | 7.5anthropic/claude-haiku-5.5 | high |
| 57 | Lfm2.5 8B A1B | 91.1 | 30 / 30 | 7.2openai/gpt-5.6-luna | 6.3anthropic/claude-haiku-5.5 | high |
| 58 | Qwen3-4B | 91.0 | 30 / 30 | 8.4openai/gpt-5.6-luna | 7.2anthropic/claude-haiku-5.5 | high |
| 59 | Qwen3.5-4B | 90.9 | 30 / 30 | 8.4openai/gpt-5.6-luna | 7.2anthropic/claude-haiku-5.5 | high |
| 60 | Qwen3-30B-A3B-Instruct-2507 | 90.3 | 30 / 30 | 8.4openai/gpt-5.6-luna | 7.3anthropic/claude-haiku-5.5 | high |
| 61 | Granite4 Micro H | 90.0 | 30 / 30 | 8.1openai/gpt-5.6-luna | 7.4anthropic/claude-haiku-5.5 | high |
| 62 | Qwen3 VL 2B Instruct | 89.8 | 30 / 30 | 8.2openai/gpt-5.6-luna | 7.4anthropic/claude-haiku-5.5 | high |
| 63 | Exaone3.5 32B Instruct | 89.8 | 30 / 30 | 8.2openai/gpt-5.6-luna | 7.4anthropic/claude-haiku-5.5 | high |
| 64 | Gemma3n E2B IT | 89.7 | 30 / 30 | 8.3openai/gpt-5.6-luna | 6.8anthropic/claude-haiku-5.5 | medium |
| 65 | Medgemma 27B ITfine-tune of Gemma 3medical model | 89.5 | 30 / 30 | 8.2openai/gpt-5.6-luna | 7.2anthropic/claude-haiku-5.5 | high |
| 66 | Cogito 14B V1 Preview Qwenfine-tune of Qwen 2.5 | 89.4 | 30 / 30 | 8.4openai/gpt-5.6-luna | 7.0anthropic/claude-haiku-5.5 | high |
| 67 | Meta-Llama-3-8B-Instruct | 89.3 | 30 / 30 | 8.1openai/gpt-5.6-luna | 7.0anthropic/claude-haiku-5.5 | high |
| 68 | Devstral Small 2 24B Instruct (2512)code model | 89.1 | 30 / 30 | 8.3openai/gpt-5.6-luna | 7.2anthropic/claude-haiku-5.5 | high |
| 69 | Qwen3-4B-Thinking-2507 | 88.6 | 30 / 30 | 7.7openai/gpt-5.6-luna | 7.2anthropic/claude-haiku-5.5 | high |
| 70 | Olmo2 13B 1124 Instruct | 88.6 | 30 / 30 | 8.0openai/gpt-5.6-luna | 6.7anthropic/claude-haiku-5.5 | high |
| 71 | Qwen2.5-7B-Instruct | 88.4 | 30 / 30 | 8.0openai/gpt-5.6-luna | 6.9anthropic/claude-haiku-5.5 | high |
| 72 | Qwen3-VL-30B-A3B-Instruct | 88.4 | 30 / 30 | 8.4openai/gpt-5.6-luna | 7.3anthropic/claude-haiku-5.5 | high |
| 73 | gemma-3-12b-it | 88.3 | 30 / 30 | 8.1openai/gpt-5.6-luna | 6.9anthropic/claude-haiku-5.5 | high |
| 74 | Lfm3 VL 2B | 88.1 | 30 / 30 | 8.2openai/gpt-5.6-luna | 7.0anthropic/claude-haiku-5.5 | high |
| 75 | Qwen3-8B | 87.7 | 30 / 30 | 8.0openai/gpt-5.6-luna | 6.9anthropic/claude-haiku-5.5 | high |
| 76 | Granite4 3B | 87.3 | 30 / 30 | 8.1openai/gpt-5.6-luna | 7.2anthropic/claude-haiku-5.5 | high |
| 77 | Deepcoder 14B Previewfine-tune of DeepSeek-R1-Distill-Qwen 14Bcode model | 86.8 | 30 / 30 | 7.9openai/gpt-5.6-luna | 6.8anthropic/claude-haiku-5.5 | high |
| 78 | Qwen3 1.7B | 86.8 | 30 / 30 | 7.9openai/gpt-5.6-luna | 6.7anthropic/claude-haiku-5.5 | high |
| 79 | GLM-4.7-Flash | 86.7 | 30 / 30 | 8.0openai/gpt-5.6-luna | 6.9anthropic/claude-haiku-5.5 | high |
| 80 | Translategemma 12B ITfine-tune of Gemma 3translation model | 86.3 | 30 / 30 | 7.9openai/gpt-5.6-luna | 7.1anthropic/claude-haiku-5.5 | high |
| 81 | SmolLM3-3B | 86.0 | 30 / 30 | 7.9openai/gpt-5.6-luna | 6.7anthropic/claude-haiku-5.5 | high |
| 82 | gemma-3-4b-it | 85.9 | 30 / 30 | 7.5openai/gpt-5.6-luna | 6.3anthropic/claude-haiku-5.5 | high |
| 83 | Granite3.3 8B | 85.7 | 30 / 30 | 7.9openai/gpt-5.6-luna | 6.8anthropic/claude-haiku-5.5 | high |
| 84 | Mistral Small 3.2 24B Instruct (2506) | 85.4 | 30 / 30 | 8.0openai/gpt-5.6-luna | 6.7anthropic/claude-haiku-5.5 | high |
| 85 | Ornith-1.5-9B | 85.2 | 30 / 30 | 7.9openai/gpt-5.6-luna | 6.6anthropic/claude-haiku-5.5 | high |
| 86 | Granite4 1B H | 85.1 | 30 / 30 | 7.7openai/gpt-5.6-luna | 6.3anthropic/claude-haiku-5.5 | medium |
| 87 | Command R7B Arabic 7B 02 2025 | 84.8 | 30 / 30 | 7.8openai/gpt-5.6-luna | 6.4anthropic/claude-haiku-5.5 | high |
| 87 | Mistral Small 24B Instruct 2501 | 84.8 | 30 / 30 | 7.9openai/gpt-5.6-luna | 6.9anthropic/claude-haiku-5.5 | high |
| 89 | Falcon3 10B Instruct | 84.8 | 30 / 30 | 7.6openai/gpt-5.6-luna | 6.5anthropic/claude-haiku-5.5 | high |
| 90 | Gemma2 2B Instruct | 84.6 | 30 / 30 | 7.8openai/gpt-5.6-luna | 6.6anthropic/claude-haiku-5.5 | high |
| 91 | Olmo2 7B 1124 Instruct | 84.4 | 30 / 30 | 7.6openai/gpt-5.6-luna | 6.4anthropic/claude-haiku-5.5 | high |
| 92 | DeepSeek-R1-0528-Qwen3-8B | 84.3 | 30 / 30 | 8.0openai/gpt-5.6-luna | 6.4anthropic/claude-haiku-5.5 | medium |
| 93 | Devstral 24B Small 2505code model | 84.2 | 30 / 30 | 7.8openai/gpt-5.6-luna | 6.4anthropic/claude-haiku-5.5 | medium |
| 93 | Mistral Small3.1 24B Instruct 2503 | 84.2 | 30 / 30 | 7.8openai/gpt-5.6-luna | 6.7anthropic/claude-haiku-5.5 | high |
| 93 | Qwen2.5-Coder-14B-Instructcode model | 84.2 | 30 / 30 | 7.9openai/gpt-5.6-luna | 6.7anthropic/claude-haiku-5.5 | high |
| 96 | Phi-4-mini-instruct | 84.1 | 30 / 30 | 7.5openai/gpt-5.6-luna | 6.2anthropic/claude-haiku-5.5 | high |
| 97 | Command R 35B 08 2024 | 84.1 | 30 / 30 | 7.5openai/gpt-5.6-luna | 6.4anthropic/claude-haiku-5.5 | high |
| 98 | Tulu3 8Bfine-tune of Llama 3.1 | 83.6 | 30 / 30 | 7.6openai/gpt-5.6-luna | 6.3anthropic/claude-haiku-5.5 | high |
| 99 | Gemma 3n E4B IT | 83.4 | 30 / 30 | 7.6openai/gpt-5.6-luna | 6.2anthropic/claude-haiku-5.5 | high |
| 100 | Command R7B (12-2024) | 82.9 | 30 / 30 | 7.6openai/gpt-5.6-luna | 6.5anthropic/claude-haiku-5.5 | high |
| 101 | Granite3.2 8B Instruct | 82.7 | 30 / 30 | 7.7openai/gpt-5.6-luna | 6.4anthropic/claude-haiku-5.5 | high |
| 101 | Qwen2.5-3B-Instruct | 82.7 | 30 / 30 | 7.5openai/gpt-5.6-luna | 6.3anthropic/claude-haiku-5.5 | high |
| 103 | Glm4 9B Chat | 82.3 | 30 / 30 | 7.7openai/gpt-5.6-luna | 6.5anthropic/claude-haiku-5.5 | high |
| 104 | Deepseek R1 8B Llama Distill | 82.3 | 30 / 30 | 7.6openai/gpt-5.6-luna | 6.5anthropic/claude-haiku-5.5 | high |
| 105 | Qwen3.5 2B | 82.1 | 30 / 30 | 7.4openai/gpt-5.6-luna | 6.0anthropic/claude-haiku-5.5 | high |
| 106 | Cogito 3B V1 Preview Llamafine-tune of Llama 3.2 | 81.6 | 30 / 30 | 7.4openai/gpt-5.6-luna | 5.9anthropic/claude-haiku-5.5 | high |
| 107 | Aya Expanse 32B | 81.3 | 30 / 30 | 7.4openai/gpt-5.6-luna | 5.8anthropic/claude-haiku-5.5 | medium |
| 108 | Ministral 3 14B Instruct (2512) | 81.2 | 30 / 30 | 7.4openai/gpt-5.6-luna | 5.9anthropic/claude-haiku-5.5 | medium |
| 109 | DeepSeek-R1-Distill-Qwen-14B | 80.8 | 30 / 30 | 7.6openai/gpt-5.6-luna | 6.3anthropic/claude-haiku-5.5 | high |
| 109 | EXAONE-3.5-7.8B-Instruct | 80.8 | 30 / 30 | 7.3openai/gpt-5.6-luna | 5.9anthropic/claude-haiku-5.5 | medium |
| 109 | Lfm2.5 Thinking 1.2B | 80.8 | 30 / 30 | 6.7openai/gpt-5.6-luna | 5.7anthropic/claude-haiku-5.5 | high |
| 112 | Mistral Nemo 12B Instruct 2407 | 80.7 | 30 / 30 | 7.2openai/gpt-5.6-luna | 5.7anthropic/claude-haiku-5.5 | high |
| 112 | Nanbeige_Nanbeige4.3-33B | 80.7 | 30 / 30 | 7.9openai/gpt-5.6-luna | 6.6anthropic/claude-haiku-5.5 | high |
| 114 | Falcon3 7B Instruct | 80.2 | 30 / 30 | 7.2openai/gpt-5.6-luna | 5.9anthropic/claude-haiku-5.5 | high |
| 115 | Granite4 Tiny H | 79.7 | 30 / 30 | 7.3openai/gpt-5.6-luna | 5.6anthropic/claude-haiku-5.5 | medium |
| 116 | Aya Expanse 8B | 79.5 | 30 / 30 | 7.2openai/gpt-5.6-luna | 5.7anthropic/claude-haiku-5.5 | medium |
| 116 | Smollm2 1.7B Instruct | 79.5 | 30 / 30 | 7.2openai/gpt-5.6-luna | 5.5anthropic/claude-haiku-5.5 | medium |
| 118 | Granite4 350M H | 79.3 | 30 / 30 | 6.6openai/gpt-5.6-luna | 4.8anthropic/claude-haiku-5.5 | medium |
| 119 | Mistral Small 22B Instruct 2409 | 78.9 | 30 / 30 | 7.2openai/gpt-5.6-luna | 5.9anthropic/claude-haiku-5.5 | medium |
| 120 | Qwen2.5-VL-7B-Instruct | 78.8 | 30 / 30 | 7.3openai/gpt-5.6-luna | 6.3anthropic/claude-haiku-5.5 | high |
| 121 | Spark-X2.6-4B | 78.7 | 30 / 30 | 7.5openai/gpt-5.6-luna | 6.4anthropic/claude-haiku-5.5 | high |
| 122 | medgemma-4b-itmedical model | 78.6 | 30 / 30 | 7.1openai/gpt-5.6-luna | 5.6anthropic/claude-haiku-5.5 | medium |
| 123 | Codegemma 7B Instruct V1.1code model | 78.1 | 30 / 30 | 7.1openai/gpt-5.6-luna | 6.0anthropic/claude-haiku-5.5 | high |
| 124 | DeepSeek-Coder-V2-Lite-Instructcode model | 77.9 | 30 / 30 | 7.0openai/gpt-5.6-luna | 5.6anthropic/claude-haiku-5.5 | high |
| 125 | Lfm3 1.7B | 77.9 | 30 / 30 | 7.0openai/gpt-5.6-luna | 5.6anthropic/claude-haiku-5.5 | high |
| 126 | Ministral 3 8B Instruct 2512 | 77.8 | 30 / 30 | 7.1openai/gpt-5.6-luna | 5.7anthropic/claude-haiku-5.5 | high |
| 127 | DeepSeek-R1-Distill-Qwen-7B | 77.4 | 30 / 30 | 7.1openai/gpt-5.6-luna | 5.9anthropic/claude-haiku-5.5 | high |
| 128 | Qwen3 VL 2B Thinking | 77.2 | 30 / 30 | 6.6openai/gpt-5.6-luna | 5.6anthropic/claude-haiku-5.5 | high |
| 129 | Llama3.2 1B Instruct | 76.9 | 30 / 30 | 7.0openai/gpt-5.6-luna | 5.5anthropic/claude-haiku-5.5 | medium |
| 129 | Qwen 14B Chat V1.5 | 76.9 | 30 / 30 | 7.0openai/gpt-5.6-luna | 5.8anthropic/claude-haiku-5.5 | high |
| 131 | Falcon3 3B Instruct | 76.8 | 30 / 30 | 6.9openai/gpt-5.6-luna | 5.5anthropic/claude-haiku-5.5 | high |
| 132 | YI 34B Chat V1.5 | 76.6 | 30 / 30 | 7.1openai/gpt-5.6-luna | 5.5anthropic/claude-haiku-5.5 | medium |
| 133 | Granite3 Dense 8B Instruct | 76.4 | 30 / 30 | 6.8openai/gpt-5.6-luna | 5.2anthropic/claude-haiku-5.5 | medium |
| 133 | Hermes3 3B Llama3.2fine-tune of Llama 3.2 | 76.4 | 30 / 30 | 6.9openai/gpt-5.6-luna | 5.6anthropic/claude-haiku-5.5 | high |
| 135 | Granite3.3 2B | 76.3 | 30 / 30 | 7.0openai/gpt-5.6-luna | 5.4anthropic/claude-haiku-5.5 | medium |
| 136 | Apertus-8B-Instruct-2509 | 76.3 | 30 / 30 | 7.0openai/gpt-5.6-luna | 5.6anthropic/claude-haiku-5.5 | high |
| 137 | Gemma 7B Instruct V1.1 | 76.0 | 30 / 30 | 6.9openai/gpt-5.6-luna | 5.5anthropic/claude-haiku-5.5 | high |
| 138 | DeepSeek-R1-Distill-Qwen-32B | 75.7 | 30 / 30 | 6.9openai/gpt-5.6-luna | 5.4anthropic/claude-haiku-5.5 | medium |
| 139 | Granite3.2 2B Instruct | 75.5 | 30 / 30 | 6.9openai/gpt-5.6-luna | 5.4anthropic/claude-haiku-5.5 | medium |
| 140 | Codegeex4 9B Allcode model | 75.2 | 30 / 30 | 6.7openai/gpt-5.6-luna | 5.2anthropic/claude-haiku-5.5 | medium |
| 141 | Codellama 34B Instructcode model | 75.1 | 30 / 30 | 6.9openai/gpt-5.6-luna | 5.4anthropic/claude-haiku-5.5 | medium |
| 141 | Mistral-7B-Instruct-v0.2 | 75.1 | 30 / 30 | 6.8openai/gpt-5.6-luna | 5.4anthropic/claude-haiku-5.5 | medium |
| 143 | Qwen2.5-VL-3B-Instruct | 74.7 | 30 / 30 | 6.9openai/gpt-5.6-luna | 5.5anthropic/claude-haiku-5.5 | high |
| 144 | Aya 8B 23 | 74.7 | 30 / 30 | 6.7openai/gpt-5.6-luna | 5.5anthropic/claude-haiku-5.5 | high |
| 145 | Codestral 22B V0.1code model | 74.4 | 30 / 30 | 6.8openai/gpt-5.6-luna | 5.1anthropic/claude-haiku-5.5 | medium |
| 146 | Granite3.1 Dense 8B Instruct | 74.3 | 30 / 30 | 7.0openai/gpt-5.6-luna | 5.6anthropic/claude-haiku-5.5 | high |
| 146 | Qwen3 0.6B | 74.3 | 30 / 30 | 6.3openai/gpt-5.6-luna | 4.8anthropic/claude-haiku-5.5 | medium |
| 148 | Codellama 13B Instructcode model | 73.8 | 30 / 30 | 6.7openai/gpt-5.6-luna | 5.3anthropic/claude-haiku-5.5 | high |
| 148 | Hermes-3-Llama-3.1-8B | 73.8 | 30 / 30 | 6.7openai/gpt-5.6-luna | 5.6anthropic/claude-haiku-5.5 | high |
| 150 | YI 9B Chat V1.5 | 73.7 | 30 / 30 | 6.8openai/gpt-5.6-luna | 5.2anthropic/claude-haiku-5.5 | medium |
| 151 | Exaone3.5 2.4B Instruct | 73.6 | 30 / 30 | 6.5openai/gpt-5.6-luna | 4.7anthropic/claude-haiku-5.5 | medium |
| 151 | Granite3.1 Dense 2B Instruct | 73.6 | 30 / 30 | 6.7openai/gpt-5.6-luna | 5.3anthropic/claude-haiku-5.5 | high |
| 153 | phi-4 | 73.2 | 30 / 30 | 6.7openai/gpt-5.6-luna | 5.3anthropic/claude-haiku-5.5 | high |
| 154 | dolphin-2.9.1-yi-1.5-34b | 72.7 | 30 / 30 | 6.7openai/gpt-5.6-luna | 5.5anthropic/claude-haiku-5.5 | high |
| 155 | Qwen2.5-Coder-7B-Instructcode model | 72.6 | 30 / 30 | 6.8openai/gpt-5.6-luna | 5.5anthropic/claude-haiku-5.5 | high |
| 156 | Nemotron Mini 4B Instruct | 72.4 | 30 / 30 | 6.2openai/gpt-5.6-luna | 4.6anthropic/claude-haiku-5.5 | medium |
| 157 | Codeqwen 7B Chat V1.5code model | 72.1 | 30 / 30 | 6.4openai/gpt-5.6-luna | 4.9anthropic/claude-haiku-5.5 | medium |
| 157 | Llama-2-7b-chat-hf | 72.1 | 30 / 30 | 6.4openai/gpt-5.6-luna | 5.2anthropic/claude-haiku-5.5 | high |
| 159 | Gemma3 1B IT | 71.9 | 30 / 30 | 6.6openai/gpt-5.6-luna | 4.8anthropic/claude-haiku-5.5 | medium |
| 160 | Qwen 7B Chat V1.5 | 71.9 | 30 / 30 | 6.4openai/gpt-5.6-luna | 4.5anthropic/claude-haiku-5.5 | medium |
| 161 | Phi3 14B Medium 128K Instruct | 71.6 | 30 / 30 | 6.4openai/gpt-5.6-luna | 4.7anthropic/claude-haiku-5.5 | medium |
| 162 | Phi-3-mini-4k-instruct | 70.7 | 30 / 30 | 6.5openai/gpt-5.6-luna | 5.1anthropic/claude-haiku-5.5 | medium |
| 162 | Qwen2-VL-7B-Instruct | 70.7 | 30 / 30 | 6.6openai/gpt-5.6-luna | 5.2anthropic/claude-haiku-5.5 | high |
| 164 | YI 6B Chat V1.5 | 70.4 | 30 / 30 | 6.6openai/gpt-5.6-luna | 4.8anthropic/claude-haiku-5.5 | medium |
| 165 | Mistral 7B Instruct V0.3 | 70.3 | 30 / 30 | 6.4openai/gpt-5.6-luna | 5.0anthropic/claude-haiku-5.5 | high |
| 166 | Deepseek V2 16B Lite Chat | 70.2 | 30 / 30 | 6.5openai/gpt-5.6-luna | 4.9anthropic/claude-haiku-5.5 | medium |
| 167 | Qwen2.5-Coder-3B-Instructcode model | 70.0 | 30 / 30 | 6.4openai/gpt-5.6-luna | 4.8anthropic/claude-haiku-5.5 | medium |
| 168 | Solar 10.7B Instruct V1 | 69.9 | 30 / 30 | 6.2openai/gpt-5.6-luna | 4.7anthropic/claude-haiku-5.5 | medium |
| 169 | Dolphin Phi 2.7B V2.6fine-tune of Phi-2 | 69.8 | 30 / 30 | 6.5openai/gpt-5.6-luna | 4.9anthropic/claude-haiku-5.5 | medium |
| 170 | Internlm2 7B Chat V2.5 | 69.6 | 30 / 30 | 6.1openai/gpt-5.6-luna | 4.7anthropic/claude-haiku-5.5 | medium |
| 171 | Qwen2.5-VL-32B-Instruct | 69.5 | 30 / 30 | 6.3openai/gpt-5.6-luna | 4.8anthropic/claude-haiku-5.5 | high |
| 172 | Granite3.1 Moe 3B Instruct | 69.2 | 30 / 30 | 6.3openai/gpt-5.6-luna | 5.0anthropic/claude-haiku-5.5 | medium |
| 173 | Qwen2 Math 7B Instructmath model | 68.7 | 30 / 30 | 6.1openai/gpt-5.6-luna | 4.1anthropic/claude-haiku-5.5 | medium |
| 174 | Deepcoder 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bcode model | 68.6 | 30 / 30 | 5.6openai/gpt-5.6-luna | 3.8anthropic/claude-haiku-5.5 | medium |
| 175 | Codellama 7B Instructcode model | 68.1 | 30 / 30 | 6.1openai/gpt-5.6-luna | 4.6anthropic/claude-haiku-5.5 | medium |
| 176 | Ministral 3 3B Instruct 2512 | 67.5 | 30 / 30 | 6.1openai/gpt-5.6-luna | 4.3anthropic/claude-haiku-5.5 | medium |
| 177 | Granite3 Moe 3B Instruct | 67.2 | 30 / 30 | 6.0openai/gpt-5.6-luna | 4.3anthropic/claude-haiku-5.5 | medium |
| 178 | Deepscaler 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bmath model | 66.9 | 30 / 30 | 5.2openai/gpt-5.6-luna | 4.0anthropic/claude-haiku-5.5 | high |
| 179 | Deepseek Llm 7B Chat | 66.8 | 30 / 30 | 6.1openai/gpt-5.6-luna | 4.4anthropic/claude-haiku-5.5 | medium |
| 180 | Gemma 2B Instruct V1.1 | 66.6 | 30 / 30 | 6.1openai/gpt-5.6-luna | 4.5anthropic/claude-haiku-5.5 | medium |
| 181 | Deepseek R1 1.5B Qwen Distill | 66.2 | 30 / 30 | 5.0openai/gpt-5.6-luna | 3.7anthropic/claude-haiku-5.5 | high |
| 182 | YI Coder 9B Chatcode model | 65.3 | 30 / 30 | 5.8openai/gpt-5.6-luna | 4.7anthropic/claude-haiku-5.5 | high |
| 183 | Falcon3 1B Instruct | 64.9 | 30 / 30 | 5.9openai/gpt-5.6-luna | 4.2anthropic/claude-haiku-5.5 | medium |
| 184 | Granite3 Dense 2B Instruct | 63.4 | 30 / 30 | 5.8openai/gpt-5.6-luna | 4.2anthropic/claude-haiku-5.5 | medium |
| 185 | Mixtral 8x7B Instruct V0.1 | 62.6 | 30 / 30 | 5.7openai/gpt-5.6-luna | 4.3anthropic/claude-haiku-5.5 | high |
| 186 | Openthinker 7B V2fine-tune of Qwen 2.5 | 62.4 | 30 / 30 | 5.6openai/gpt-5.6-luna | 4.5anthropic/claude-haiku-5.5 | high |
| 187 | Qwen2 1.5B Instruct | 61.6 | 30 / 30 | 5.7openai/gpt-5.6-luna | 3.8anthropic/claude-haiku-5.5 | medium |
| 188 | Qwen 4B Chat V1.5 | 60.4 | 30 / 30 | 5.2openai/gpt-5.6-luna | 3.6anthropic/claude-haiku-5.5 | medium |
| 189 | Phi 2.7B Chat V2 | 60.3 | 30 / 30 | 5.4openai/gpt-5.6-luna | 3.9anthropic/claude-haiku-5.5 | medium |
| 190 | Phi-3.5-mini-instruct | 60.2 | 30 / 30 | 5.3openai/gpt-5.6-luna | 3.5anthropic/claude-haiku-5.5 | medium |
| 190 | Phi3 3.8B Mini 128K Instruct | 60.2 | 30 / 30 | 5.4openai/gpt-5.6-luna | 3.6anthropic/claude-haiku-5.5 | medium |
| 192 | Stablelm Zephyr 3Bfine-tune of StableLM 3B | 60.1 | 30 / 30 | 5.4openai/gpt-5.6-luna | 4.0anthropic/claude-haiku-5.5 | high |
| 193 | Smollm2 360M Instruct | 59.9 | 30 / 30 | 5.2openai/gpt-5.6-luna | 3.9anthropic/claude-haiku-5.5 | high |
| 194 | Qwen2.5 1.5B Instruct | 59.4 | 30 / 30 | 5.5openai/gpt-5.6-luna | 4.0anthropic/claude-haiku-5.5 | medium |
| 195 | Granite3 Moe 1B Instruct | 59.1 | 30 / 30 | 4.9openai/gpt-5.6-luna | 3.6anthropic/claude-haiku-5.5 | high |
| 196 | Granite3.1 Moe 1B Instruct | 59.0 | 30 / 30 | 5.3openai/gpt-5.6-luna | 3.4anthropic/claude-haiku-5.5 | medium |
| 197 | Qwen2.5 Coder 1.5B Instructcode model | 58.8 | 30 / 30 | 5.4openai/gpt-5.6-luna | 3.4anthropic/claude-haiku-5.5 | medium |
| 198 | Falcon2 11Bnot in the Overall ranking | 57.7 | 30 / 30 | 4.8openai/gpt-5.6-luna | 2.9anthropic/claude-haiku-5.5 | medium |
| 199 | Internlm2 1.8B Chat V2.5 | 56.6 | 30 / 30 | 5.2openai/gpt-5.6-luna | 3.6anthropic/claude-haiku-5.5 | medium |
| 200 | Stable Code 3B Instructcode model | 55.8 | 30 / 30 | 4.7openai/gpt-5.6-luna | 3.1anthropic/claude-haiku-5.5 | medium |
| 201 | Qwen2.5 0.5B Instruct | 54.9 | 30 / 30 | 4.8openai/gpt-5.6-luna | 2.9anthropic/claude-haiku-5.5 | medium |
| 202 | Granite Code 3B Instructcode model | 54.2 | 30 / 30 | 4.9openai/gpt-5.6-luna | 3.5anthropic/claude-haiku-5.5 | medium |
| 203 | Deepseek Coder 6.7B Instructcode model | 53.9 | 30 / 30 | 3.7openai/gpt-5.6-luna | 2.4anthropic/claude-haiku-5.5 | high |
| 204 | Granite Code 8B Instructcode model | 53.7 | 30 / 30 | 4.8openai/gpt-5.6-luna | 3.3anthropic/claude-haiku-5.5 | medium |
| 205 | Mathstral 7B V0.1fine-tune of Mistral 7Bmath model | 52.3 | 30 / 30 | 4.5openai/gpt-5.6-luna | 3.0anthropic/claude-haiku-5.5 | medium |
| 206 | Sailor2 8B Chatfine-tune of Qwen 2.5 | 52.2 | 30 / 30 | 4.6openai/gpt-5.6-luna | 3.0anthropic/claude-haiku-5.5 | medium |
| 207 | Medgemma1.5 4B ITfine-tune of Gemma 3medical model | 52.2 | 30 / 30 | 4.1openai/gpt-5.6-luna | 2.5anthropic/claude-haiku-5.5 | medium |
| 208 | Opencoder 1.5B Instructcode model | 50.8 | 30 / 30 | 4.3openai/gpt-5.6-luna | 2.4anthropic/claude-haiku-5.5 | medium |
| 209 | Qwen2 0.5B Instruct | 50.2 | 30 / 30 | 4.2openai/gpt-5.6-luna | 2.2anthropic/claude-haiku-5.5 | medium |
| 210 | Qwen2.5 Coder 0.5B Instructcode model | 49.5 | 30 / 30 | 4.1openai/gpt-5.6-luna | 2.3anthropic/claude-haiku-5.5 | medium |
| 211 | Llama2 13B Chat | 47.8 | 30 / 30 | 4.6openai/gpt-5.6-luna | 2.9anthropic/claude-haiku-5.5 | medium |
| 212 | Stablelm2 12B Chat | 47.0 | 30 / 30 | 4.3openai/gpt-5.6-luna | 3.1anthropic/claude-haiku-5.5 | high |
| 213 | Marco O1 7Bfine-tune of Qwen 2 7B | 46.9 | 30 / 30 | 4.6openai/gpt-5.6-luna | 3.6anthropic/claude-haiku-5.5 | high |
| 214 | Deepseek Coder 33B Instructcode model | 46.2 | 30 / 30 | 2.9openai/gpt-5.6-luna | 1.7anthropic/claude-haiku-5.5 | high |
| 215 | Phi4 Mini Reasoning 3.8B | 44.3 | 30 / 30 | 3.6openai/gpt-5.6-luna | 1.7anthropic/claude-haiku-5.5 | medium |
| 216 | Qwen 0.5B Chat V1.5 | 40.8 | 30 / 30 | 3.5openai/gpt-5.6-luna | 1.6anthropic/claude-haiku-5.5 | medium |
| 217 | Deepseek Coder 1.3B Instructcode model | 40.5 | 30 / 30 | 2.6openai/gpt-5.6-luna | 1.0anthropic/claude-haiku-5.5 | medium |
| 218 | Qwen 1.8B Chat V1.5 | 38.4 | 30 / 30 | 3.7openai/gpt-5.6-luna | 2.3anthropic/claude-haiku-5.5 | high |
| 219 | Smollm2 135M Instruct | 36.1 | 30 / 30 | 3.0openai/gpt-5.6-luna | 1.3anthropic/claude-haiku-5.5 | medium |
| 219 | YI Coder 1.5B Chatcode model | 36.1 | 30 / 30 | 3.1openai/gpt-5.6-luna | 1.6anthropic/claude-haiku-5.5 | medium |
| 221 | Magistral Small (2506) | 34.8 | 30 / 30 | 1.1openai/gpt-5.6-luna | 0.6anthropic/claude-haiku-5.5 | high |
| 222 | Stablelm2 1.6B Chat | 31.6 | 30 / 30 | 3.1openai/gpt-5.6-luna | 1.6anthropic/claude-haiku-5.5 | high |
| 223 | Sailor2 1B Chatfine-tune of Qwen 2.5 | 29.6 | 30 / 30 | 3.0openai/gpt-5.6-luna | 1.5anthropic/claude-haiku-5.5 | high |
| 224 | Codegemma 2B Code V1.1code model | 29.5 | 30 / 30 | 1.6openai/gpt-5.6-luna | 0.3anthropic/claude-haiku-5.5 | high |
| 225 | Smallthinker 3B Previewfine-tune of Qwen 2.5 3B | 27.8 | 30 / 30 | 2.7openai/gpt-5.6-luna | 1.3anthropic/claude-haiku-5.5 | high |
| 226 | Exaone Deep 7.8B | 16.9 | 30 / 30 | 1.9openai/gpt-5.6-luna | 1.0anthropic/claude-haiku-5.5 | high |
| 227 | Qwen2 Math 1.5B Instructmath model | 14.4 | 30 / 30 | 1.7openai/gpt-5.6-luna | 0.7anthropic/claude-haiku-5.5 | high |
| 228 | Exaone Deep 2.4B | 13.9 | 30 / 30 | 1.6openai/gpt-5.6-luna | 0.7anthropic/claude-haiku-5.5 | high |
A dash means not scored, which is different from 0. Agreement compares the two judges: high means a mean gap under 1.5 points, medium under 3, low otherwise; n/a means fewer than 3 items were scored by both.
Not ranked in this category
- Apertus-8B-Instruct-2509: withheld from the ranking (see its page)
- Falcon3 10B Instruct: withheld from the ranking (see its page)
- Falcon3 1B Instruct: withheld from the ranking (see its page)
- Falcon3 3B Instruct: withheld from the ranking (see its page)
- Falcon3 7B Instruct: withheld from the ranking (see its page)
- Internlm2 1.8B Chat V2.5: withheld from the ranking (see its page)
- Phi-4-reasoning 14B: withheld from the ranking (see its page)
- Phi-4-reasoning-plus 14B: withheld from the ranking (see its page)
- Qwen2 Math 1.5B Instruct: withheld from the ranking (see its page)
- Qwen2 Math 7B Instruct: withheld from the ranking (see its page)
The questions
In run r20261102-bv1-5246ab, 0 of the 30 questions in this category were answered correctly by every ranked model that has a score for them (33 ranked models in that run). Hard questions are the ones that half or fewer of the ranked models of 3B and over answered correctly. This category has 6.
The same ranking as JSON: instruction.json.