The judge room
Programmatic scorers mark most of the battery. Two language-model judges also score many answers from 0 to 10: writing, which no rule can mark, and others alongside the code checks. This page names the judges, shows where the two judges of a model disagree, and links to the public questions where you can read both judges' reasons. How the public sample is chosen.
The judges
The judges are paid models, reached through OpenRouter. They are not among the open-weight models being tested, and they are not run on our hardware.
- J-O (openai/gpt-5.6-luna): 4,518 scores on the public questions, covering 226 model runs
- J-A (anthropic/claude-haiku-5.5): 4,558 scores on the public questions, covering 228 model runs
- J-D (deepseek/deepseek-v4.1-flash): 40 scores on the public questions, covering 2 model runs
Judged answers are normally scored 0-10 by two judges; on rare occasions one judge's reply could not be used. The default pair is J-O and J-A. A model from the OpenAI family (the gpt-oss models) is judged by J-A and J-D, because an OpenAI judge would be marking its own family; a model from the Claude family is judged by J-O and J-D. A Qwen or any other model gets the default pair.
Because the pair differs, the judges are not the same for every row below: each row and each cell says which pair judged that model.
Where the judges disagree
A judged answer is disputed when its two judges are 3 or more points apart. The counts are over the answers of the models on the board (withheld models are left out).
| Category | Models judged | Disputed answers (all questions) | Public questions | Disputed on public questions |
|---|---|---|---|---|
| Reasoning | 228 | 313 | 3 | 21 of 683 |
| Coding | 228 | 203 | 3 | 19 of 684 |
| Instruction following | 228 | 1,381 | 3 | 81 of 684 |
| Safety | 228 | 510 | 3 | 71 of 683 |
| Writing | 228 | 143 | 8 | 143 of 1,823 |
On the 44 public questions, 335 of 4,557 judged answers were disputed. Browse the public questions by how much their judges disagreed.
Mean gap between the two judges
One row per model, one column per judged category. The number is the mean gap, in points on the 0 to 10 scale, between the model's two judges; smaller is closer agreement. The colour is the stored agreement level: high means a mean gap under 1.5, medium under 3, low otherwise. A dash means too few answers were scored by both judges. Hover a cell to see which two judges marked that model.
| Model | Judge A / Judge B | All judged | Reasoning | Coding | Instruction following | Safety | Writing |
|---|---|---|---|---|---|---|---|
| Phi3 14B Medium 128K Instruct | J-O + J-A | 1.4 | 1.1 | 1.1 | 1.8 | 1.5 | 1.6 |
| Qwen2.5 Coder 1.5B Instruct | J-O + J-A | 1.3 | 1.0 | 0.9 | 1.9 | 1.0 | 1.5 |
| Granite3 Moe 3B Instruct | J-O + J-A | 1.3 | 0.9 | 1.2 | 1.8 | 1.3 | 1.0 |
| Qwen 0.5B Chat V1.5 | J-O + J-A | 1.3 | 0.7 | 0.4 | 2.0 | 1.2 | 2.0 |
| Exaone3.5 2.4B Instruct | J-O + J-A | 1.2 | 0.9 | 0.9 | 1.9 | 1.6 | 0.9 |
| Codellama 34B Instruct | J-O + J-A | 1.2 | 1.1 | 1.0 | 1.6 | 1.2 | 1.4 |
| Gemma3n E2B IT | J-O + J-A | 1.2 | 0.7 | 1.0 | 1.6 | 1.3 | 1.6 |
| Gemma 3n E4B IT | J-O + J-A | 1.2 | 0.8 | 1.1 | 1.5 | 1.2 | 1.6 |
| Internlm2 1.8B Chat V2.5 | J-O + J-A | 1.2 | 0.6 | 1.0 | 1.7 | 1.4 | 1.5 |
| Codeqwen 7B Chat V1.5 | J-O + J-A | 1.2 | 0.4 | 1.1 | 1.6 | 1.8 | 1.3 |
| Codellama 7B Instruct | J-O + J-A | 1.2 | 0.7 | 1.3 | 1.7 | 1.2 | 1.3 |
| Marco O1 7B | J-O + J-A | 1.2 | 1.1 | 0.9 | 1.3 | 1.1 | 1.6 |
| Internlm2 7B Chat V2.5 | J-O + J-A | 1.2 | 0.5 | 1.1 | 1.5 | 1.8 | 1.3 |
| Opencoder 1.5B Instruct | J-O + J-A | 1.2 | 0.4 | 0.6 | 1.9 | 0.9 | 2.3 |
| Qwen2.5 1.5B Instruct | J-O + J-A | 1.2 | 1.0 | 1.2 | 1.6 | 1.3 | 1.1 |
| Granite Code 3B Instruct | J-O + J-A | 1.2 | 0.7 | 1.1 | 1.5 | 0.8 | 2.0 |
| Ministral 3 3B Instruct 2512 | J-O + J-A | 1.2 | 0.9 | 0.9 | 1.8 | 1.4 | 1.1 |
| Phi4 Mini Reasoning 3.8B | J-O + J-A | 1.2 | 0.4 | 1.2 | 2.0 | 1.3 | 1.1 |
| Smallthinker 3B Preview | J-O + J-A | 1.2 | 0.7 | 1.1 | 1.4 | 1.5 | 1.4 |
| Granite4 350M H | J-O + J-A | 1.2 | 0.3 | 1.3 | 1.8 | 0.8 | 1.8 |
| Falcon3 1B Instruct | J-O + J-A | 1.2 | 0.7 | 0.8 | 1.7 | 1.5 | 1.3 |
| YI 6B Chat V1.5 | J-O + J-A | 1.2 | 0.7 | 0.6 | 1.8 | 1.4 | 1.5 |
| Mathstral 7B V0.1 | J-O + J-A | 1.2 | 0.7 | 1.3 | 1.5 | 0.8 | 1.6 |
| Openthinker 7B V2 | J-O + J-A | 1.2 | 0.5 | 1.6 | 1.2 | 1.4 | 1.3 |
| Granite4 1B H | J-O + J-A | 1.2 | 0.7 | 1.1 | 1.5 | 1.4 | 1.3 |
| Qwen2.5-Coder-7B-Instruct | J-O + J-A | 1.2 | 0.7 | 1.1 | 1.3 | 1.6 | 1.1 |
| Qwen2.5-Coder-3B-Instruct | J-O + J-A | 1.2 | 1.0 | 1.1 | 1.6 | 1.2 | 1.0 |
| Granite4 Tiny H | J-O + J-A | 1.2 | 0.7 | 1.0 | 1.7 | 1.3 | 1.1 |
| GLM-4.7-Flash | J-O + J-A | 1.2 | 1.0 | 1.0 | 1.0 | 1.5 | 1.4 |
| Falcon2 11B | J-O + J-A | 1.2 | 0.7 | 0.4 | 1.9 | 1.1 | 1.6 |
| Falcon3 10B Instruct | J-O + J-A | 1.2 | 0.8 | 1.0 | 1.3 | 1.5 | 1.3 |
| Mistral 7B Instruct V0.3 | J-O + J-A | 1.2 | 0.9 | 0.9 | 1.5 | 1.2 | 1.3 |
| YI 9B Chat V1.5 | J-O + J-A | 1.2 | 0.7 | 0.9 | 1.6 | 1.4 | 1.1 |
| Deepseek V2 16B Lite Chat | J-O + J-A | 1.1 | 0.6 | 1.1 | 1.7 | 1.3 | 1.1 |
| Gemma3 1B IT | J-O + J-A | 1.1 | 0.8 | 1.1 | 1.8 | 1.0 | 1.0 |
| Qwen3 VL 2B Instruct | J-O + J-A | 1.1 | 0.9 | 1.1 | 0.8 | 1.3 | 1.6 |
| medgemma-4b-it | J-O + J-A | 1.1 | 0.7 | 1.2 | 1.5 | 1.5 | 0.9 |
| Qwen2.5 0.5B Instruct | J-O + J-A | 1.1 | 0.8 | 0.6 | 2.0 | 1.0 | 1.3 |
| Deepseek Llm 7B Chat | J-O + J-A | 1.1 | 0.7 | 1.2 | 1.8 | 1.1 | 1.0 |
| dolphin-2.9.1-yi-1.5-34b | J-O + J-A | 1.1 | 0.8 | 1.4 | 1.3 | 1.0 | 1.3 |
| Qwen2.5 Coder 0.5B Instruct | J-O + J-A | 1.1 | 0.7 | 1.0 | 1.8 | 1.0 | 1.3 |
| Qwen3.5-9B | J-O + J-A | 1.1 | 0.5 | 0.9 | 1.0 | 1.7 | 1.5 |
| YI Coder 1.5B Chat | J-O + J-A | 1.1 | 0.7 | 0.9 | 1.6 | 0.9 | 1.6 |
| Llama3.2 1B Instruct | J-O + J-A | 1.1 | 0.7 | 1.0 | 1.5 | 1.0 | 1.4 |
| Granite3 Dense 2B Instruct | J-O + J-A | 1.1 | 0.9 | 1.0 | 1.6 | 1.2 | 1.0 |
| Mistral Small 22B Instruct 2409 | J-O + J-A | 1.1 | 0.5 | 1.1 | 1.5 | 1.0 | 1.5 |
| Qwen3-4B | J-O + J-A | 1.1 | 0.6 | 1.1 | 1.2 | 1.8 | 1.0 |
| Phi-4-mini-instruct | J-O + J-A | 1.1 | 1.1 | 0.8 | 1.5 | 1.4 | 0.9 |
| Qwen3 0.6B | J-O + J-A | 1.1 | 0.6 | 1.1 | 1.8 | 1.3 | 0.9 |
| Command R 35B 08 2024 | J-O + J-A | 1.1 | 0.9 | 1.0 | 1.3 | 1.3 | 1.1 |
| Dolphin Phi 2.7B V2.6 | J-O + J-A | 1.1 | 0.6 | 0.7 | 1.7 | 1.0 | 1.6 |
| Granite3 Dense 8B Instruct | J-O + J-A | 1.1 | 0.8 | 0.9 | 1.7 | 1.3 | 0.9 |
| Qwen3.5-4B | J-O + J-A | 1.1 | 0.5 | 1.0 | 1.3 | 1.2 | 1.6 |
| gemma-3-27b-it | J-O + J-A | 1.1 | 0.8 | 0.9 | 1.0 | 1.4 | 1.5 |
| Tulu3 8B | J-O + J-A | 1.1 | 0.9 | 1.2 | 1.3 | 1.3 | 0.9 |
| Olmo2 7B 1124 Instruct | J-O + J-A | 1.1 | 0.8 | 1.4 | 1.1 | 1.3 | 1.0 |
| Qwen3.5 2B | J-O + J-A | 1.1 | 0.7 | 0.9 | 1.4 | 1.8 | 0.8 |
| Qwen 7B Chat V1.5 | J-O + J-A | 1.1 | 0.9 | 0.2 | 1.9 | 1.4 | 1.1 |
| Gemma2 2B Instruct | J-O + J-A | 1.1 | 0.9 | 1.0 | 1.2 | 1.0 | 1.5 |
| Smollm2 1.7B Instruct | J-O + J-A | 1.1 | 0.6 | 1.0 | 1.7 | 1.0 | 1.1 |
| Gemma 7B Instruct V1.1 | J-O + J-A | 1.1 | 0.8 | 1.1 | 1.4 | 1.5 | 0.8 |
| Granite3.1 Dense 8B Instruct | J-O + J-A | 1.1 | 0.6 | 1.1 | 1.4 | 1.5 | 0.9 |
| Lfm2.5 8B A1B | J-O + J-A | 1.1 | 1.1 | 0.6 | 1.2 | 1.3 | 1.3 |
| Granite3.1 Moe 1B Instruct | J-O + J-A | 1.1 | 0.5 | 0.8 | 2.0 | 1.2 | 1.0 |
| Mistral Nemo 12B Instruct 2407 | J-O + J-A | 1.1 | 0.9 | 1.3 | 1.5 | 1.1 | 0.8 |
| NVIDIA Nemotron 3 Nano 30B-A3B | J-O + J-A | 1.1 | 0.7 | 0.9 | 0.8 | 1.3 | 1.8 |
| YI 34B Chat V1.5 | J-O + J-A | 1.1 | 0.7 | 0.9 | 1.6 | 1.3 | 1.0 |
| Meta-Llama-3-8B-Instruct | J-O + J-A | 1.1 | 1.0 | 1.1 | 1.0 | 1.2 | 1.1 |
| Granite Code 8B Instruct | J-O + J-A | 1.1 | 0.9 | 1.0 | 1.5 | 0.8 | 1.3 |
| Ministral 3 14B Instruct (2512) | J-O + J-A | 1.1 | 0.9 | 0.7 | 1.6 | 1.4 | 0.9 |
| Gemma 2B Instruct V1.1 | J-O + J-A | 1.1 | 0.7 | 1.4 | 1.6 | 1.3 | 0.5 |
| Mistral Small 3.2 24B Instruct (2506) | J-O + J-A | 1.1 | 0.6 | 1.2 | 1.3 | 0.8 | 1.5 |
| Sailor2 8B Chat | J-O + J-A | 1.1 | 0.7 | 0.4 | 1.7 | 1.5 | 1.1 |
| Phi-3.5-mini-instruct | J-O + J-A | 1.1 | 0.8 | 0.6 | 1.9 | 1.5 | 0.5 |
| Qwen3-4B-Instruct-2507 | J-O + J-A | 1.1 | 0.5 | 0.9 | 0.7 | 1.4 | 1.9 |
| Smollm2 135M Instruct | J-O + J-A | 1.1 | 0.5 | 0.7 | 1.7 | 1.0 | 1.5 |
| Codestral 22B V0.1 | J-O + J-A | 1.1 | 0.6 | 1.1 | 1.8 | 1.1 | 0.8 |
| gemma-3-4b-it | J-O + J-A | 1.1 | 0.6 | 1.2 | 1.3 | 1.1 | 1.1 |
| Qwen2 0.5B Instruct | J-O + J-A | 1.1 | 0.7 | 0.6 | 2.0 | 0.8 | 1.3 |
| Apertus-8B-Instruct-2509 | J-O + J-A | 1.1 | 0.6 | 1.0 | 1.4 | 1.3 | 1.1 |
| Qwen2 Math 1.5B Instruct | J-O + J-A | 1.1 | 0.9 | 0.6 | 1.0 | 0.9 | 1.9 |
| Aya Expanse 8B | J-O + J-A | 1.1 | 0.4 | 1.0 | 1.5 | 1.5 | 0.9 |
| GLM-4.5-Air | J-O + J-A | 1.1 | 0.7 | 1.1 | 0.7 | 1.6 | 1.1 |
| SmolLM3-3B | J-O + J-A | 1.1 | 0.9 | 0.9 | 1.2 | 1.3 | 1.0 |
| Qwen2.5-VL-3B-Instruct | J-O + J-A | 1.1 | 0.9 | 0.9 | 1.5 | 1.1 | 0.9 |
| Solar 10.7B Instruct V1 | J-O + J-A | 1.1 | 0.8 | 1.0 | 1.6 | 1.0 | 0.9 |
| Granite3.2 8B Instruct | J-O + J-A | 1.1 | 0.7 | 1.0 | 1.3 | 1.3 | 0.9 |
| Command R7B (12-2024) | J-O + J-A | 1.1 | 0.8 | 0.8 | 1.2 | 1.1 | 1.4 |
| Mistral Small3.1 24B Instruct 2503 | J-O + J-A | 1.1 | 0.7 | 1.0 | 1.1 | 1.1 | 1.4 |
| Medgemma 27B IT | J-O + J-A | 1.0 | 0.9 | 1.0 | 1.1 | 1.3 | 1.0 |
| Nemotron 3.5 Lightning 30B A3B | J-O + J-A | 1.0 | 0.9 | 0.9 | 1.0 | 1.4 | 1.1 |
| Qwen2 Math 7B Instruct | J-O + J-A | 1.0 | 0.9 | 0.7 | 2.1 | 1.0 | 0.6 |
| EXAONE-3.5-7.8B-Instruct | J-O + J-A | 1.0 | 0.8 | 0.9 | 1.5 | 1.3 | 0.8 |
| Sage-A1.5-Q4_K_M | J-O + J-A | 1.0 | 0.3 | 0.9 | 1.0 | 1.0 | 2.0 |
| Command R7B Arabic 7B 02 2025 | J-O + J-A | 1.0 | 0.7 | 1.1 | 1.4 | 1.1 | 0.9 |
| Llama-2-7b-chat-hf | J-O + J-A | 1.0 | 0.8 | 1.0 | 1.3 | 1.2 | 1.0 |
| Nemotron Mini 4B Instruct | J-O + J-A | 1.0 | 0.6 | 0.9 | 1.9 | 1.3 | 0.5 |
| Hermes-3-Llama-3.1-8B | J-O + J-A | 1.0 | 0.5 | 1.1 | 1.2 | 1.0 | 1.4 |
| Qwen3-VL-4B-Instruct | J-O + J-A | 1.0 | 0.7 | 0.9 | 0.8 | 1.3 | 1.5 |
| Gemma4 E2B IT | J-O + J-A | 1.0 | 0.5 | 0.9 | 0.6 | 1.5 | 1.8 |
| Qwen2.5-32B-Instruct | J-O + J-A | 1.0 | 0.7 | 1.1 | 1.0 | 0.8 | 1.5 |
| granite-4.1-3b | J-O + J-A | 1.0 | 0.6 | 1.2 | 1.0 | 1.4 | 1.0 |
| Llama-3.2-3B-Instruct | J-O + J-A | 1.0 | 0.8 | 1.2 | 1.1 | 1.3 | 0.8 |
| Qwen3 1.7B | J-O + J-A | 1.0 | 0.8 | 0.9 | 1.4 | 1.4 | 0.6 |
| Granite3.1 Moe 3B Instruct | J-O + J-A | 1.0 | 0.6 | 1.2 | 1.5 | 1.2 | 0.6 |
| Codegeex4 9B All | J-O + J-A | 1.0 | 0.5 | 1.0 | 1.6 | 1.0 | 1.0 |
| Qwen2.5-VL-7B-Instruct | J-O + J-A | 1.0 | 0.9 | 1.1 | 1.0 | 1.0 | 1.0 |
| Qwen2.5-14B-Instruct | J-O + J-A | 1.0 | 0.9 | 1.2 | 1.2 | 1.3 | 0.6 |
| Medgemma1.5 4B IT | J-O + J-A | 1.0 | 0.8 | 0.6 | 1.5 | 1.3 | 0.9 |
| Devstral Small 2 24B Instruct (2512) | J-O + J-A | 1.0 | 0.7 | 0.9 | 1.2 | 1.1 | 1.3 |
| Stable Code 3B Instruct | J-O + J-A | 1.0 | 0.8 | 0.2 | 1.7 | 0.9 | 1.5 |
| Granite4.2 30B | J-O + J-A | 1.0 | 0.5 | 0.8 | 1.2 | 1.5 | 1.1 |
| Exaone Deep 7.8B | J-O + J-A | 1.0 | 0.8 | 0.3 | 0.9 | 1.3 | 1.8 |
| Phi3 3.8B Mini 128K Instruct | J-O + J-A | 1.0 | 0.9 | 0.6 | 1.8 | 1.3 | 0.5 |
| Granite3.3 8B | J-O + J-A | 1.0 | 0.6 | 1.0 | 1.1 | 1.3 | 1.0 |
| gemma-2-9b-it | J-O + J-A | 1.0 | 1.1 | 1.0 | 0.9 | 1.0 | 1.1 |
| Qwen2.5-Coder-14B-Instruct | J-O + J-A | 1.0 | 1.0 | 1.0 | 1.3 | 1.1 | 0.6 |
| phi-4 | J-O + J-A | 1.0 | 0.6 | 0.9 | 1.4 | 1.0 | 1.1 |
| Falcon3 3B Instruct | J-O + J-A | 1.0 | 0.8 | 0.8 | 1.5 | 1.6 | 0.4 |
| Aya Expanse 32B | J-O + J-A | 1.0 | 0.5 | 1.1 | 1.7 | 1.2 | 0.6 |
| Olmo2 13B 1124 Instruct | J-O + J-A | 1.0 | 0.5 | 1.3 | 1.3 | 1.1 | 0.8 |
| Qwen2.5-VL-32B-Instruct | J-O + J-A | 1.0 | 0.8 | 0.8 | 1.4 | 1.0 | 1.0 |
| Stablelm Zephyr 3B | J-O + J-A | 1.0 | 0.8 | 1.2 | 1.4 | 0.9 | 0.8 |
| Ministral 3 8B Instruct 2512 | J-O + J-A | 1.0 | 0.8 | 0.7 | 1.4 | 1.1 | 0.9 |
| Granite4.2 8B | J-O + J-A | 1.0 | 0.7 | 0.9 | 0.9 | 1.9 | 0.6 |
| Phi 2.7B Chat V2 | J-O + J-A | 1.0 | 0.5 | 0.6 | 1.5 | 1.1 | 1.3 |
| Qwen2 1.5B Instruct | J-O + J-A | 1.0 | 0.6 | 0.8 | 1.9 | 1.0 | 0.8 |
| Granite3.3 2B | J-O + J-A | 1.0 | 0.6 | 1.0 | 1.6 | 1.4 | 0.4 |
| DeepSeek-R1-Distill-Qwen-14B | J-O + J-A | 1.0 | 1.0 | 0.5 | 1.3 | 1.1 | 1.0 |
| Qwen3-8B | J-O + J-A | 1.0 | 0.8 | 0.9 | 1.2 | 0.8 | 1.3 |
| gemma-3-12b-it | J-O + J-A | 1.0 | 0.7 | 0.7 | 1.2 | 1.3 | 1.1 |
| Granite4 Small H | J-O + J-A | 1.0 | 0.5 | 1.0 | 0.9 | 1.2 | 1.4 |
| Deepseek R1 8B Llama Distill | J-O + J-A | 1.0 | 0.8 | 0.8 | 1.1 | 1.6 | 0.6 |
| Glm4 9B Chat | J-O + J-A | 1.0 | 0.8 | 1.0 | 1.2 | 1.0 | 0.9 |
| Mixtral 8x7B Instruct V0.1 | J-O + J-A | 1.0 | 0.8 | 0.8 | 1.4 | 0.9 | 1.0 |
| Sailor2 1B Chat | J-O + J-A | 1.0 | 0.5 | 1.0 | 1.4 | 1.0 | 1.0 |
| Falcon3 7B Instruct | J-O + J-A | 1.0 | 0.6 | 1.1 | 1.3 | 1.1 | 0.8 |
| Granite3.2 2B Instruct | J-O + J-A | 1.0 | 0.9 | 0.9 | 1.5 | 1.0 | 0.6 |
| Cogito 8B V1 Preview Llama | J-O + J-A | 1.0 | 0.5 | 1.3 | 1.1 | 1.0 | 1.0 |
| Phi-3-mini-4k-instruct | J-O + J-A | 1.0 | 0.8 | 0.4 | 1.5 | 1.0 | 1.1 |
| Llama2 13B Chat | J-O + J-A | 1.0 | 0.7 | 0.7 | 1.7 | 0.9 | 0.9 |
| Qwen3-30B-A3B-Instruct-2507 | J-O + J-A | 1.0 | 0.5 | 1.1 | 1.2 | 0.9 | 1.3 |
| Qwen3-VL-8B-Instruct | J-O + J-A | 1.0 | 0.6 | 1.2 | 0.7 | 1.3 | 1.1 |
| Smollm2 360M Instruct | J-O + J-A | 1.0 | 0.3 | 0.9 | 1.4 | 0.7 | 1.6 |
| Qwen3.6-35B-A3B | J-O + J-A | 1.0 | 0.7 | 0.9 | 0.9 | 1.0 | 1.4 |
| Granite3 Moe 1B Instruct | J-O + J-A | 1.0 | 0.4 | 1.3 | 1.5 | 1.3 | 0.4 |
| Qwen 4B Chat V1.5 | J-O + J-A | 1.0 | 0.5 | 0.1 | 1.6 | 1.1 | 1.6 |
| Hermes3 3B Llama3.2 | J-O + J-A | 1.0 | 0.6 | 0.6 | 1.4 | 1.0 | 1.3 |
| Granite4 3B | J-O + J-A | 1.0 | 0.6 | 1.0 | 0.9 | 1.3 | 1.1 |
| Llama-3.3-70B-Instruct | J-O + J-A | 1.0 | 0.7 | 1.0 | 0.7 | 1.2 | 1.3 |
| Granite4 Micro H | J-O + J-A | 1.0 | 0.9 | 1.0 | 0.8 | 1.3 | 0.9 |
| YI Coder 9B Chat | J-O + J-A | 1.0 | 0.4 | 1.2 | 1.3 | 1.1 | 0.9 |
| Granite4.2 3B | J-O + J-A | 1.0 | 0.8 | 0.8 | 0.9 | 1.8 | 0.5 |
| DeepSeek-Coder-V2-Lite-Instruct | J-O + J-A | 1.0 | 0.7 | 0.8 | 1.5 | 1.2 | 0.6 |
| Llama 4 Scout 17B-16E Instruct | J-O + J-A | 1.0 | 0.7 | 0.9 | 0.5 | 1.1 | 1.6 |
| Nemotron Cascade 2 30B A3B | J-O + J-A | 1.0 | 0.5 | 0.9 | 0.8 | 1.8 | 0.8 |
| Translategemma 12B IT | J-O + J-A | 1.0 | 0.6 | 1.1 | 0.9 | 1.2 | 1.0 |
| Lfm3 VL 2B | J-O + J-A | 1.0 | 0.6 | 0.8 | 1.2 | 1.0 | 1.1 |
| Deepcoder 1.5B Preview | J-O + J-A | 0.9 | 0.6 | 0.2 | 1.9 | 1.0 | 1.1 |
| Qwen2-VL-7B-Instruct | J-O + J-A | 0.9 | 0.6 | 1.0 | 1.4 | 1.0 | 0.8 |
| Aya 8B 23 | J-O + J-A | 0.9 | 0.7 | 1.1 | 1.4 | 1.1 | 0.4 |
| Mistral-7B-Instruct-v0.2 | J-O + J-A | 0.9 | 0.5 | 0.8 | 1.5 | 1.0 | 0.9 |
| Qwen3-VL-32B-Instruct | J-O + J-A | 0.9 | 0.5 | 0.9 | 0.9 | 1.0 | 1.5 |
| Laguna XS 2.1 | J-O + J-A | 0.9 | 0.5 | 0.8 | 0.9 | 1.2 | 1.3 |
| Granite4.1 8B | J-O + J-A | 0.9 | 0.5 | 0.9 | 0.9 | 1.2 | 1.3 |
| Devstral 24B Small 2505 | J-O + J-A | 0.9 | 0.7 | 0.8 | 1.5 | 0.7 | 1.0 |
| Qwen2.5-3B-Instruct | J-O + J-A | 0.9 | 0.8 | 0.8 | 1.3 | 1.3 | 0.5 |
| Lfm3 1.7B | J-O + J-A | 0.9 | 0.4 | 0.8 | 1.5 | 1.2 | 0.9 |
| Qwen2.5-7B-Instruct | J-O + J-A | 0.9 | 0.8 | 1.0 | 1.1 | 1.1 | 0.6 |
| Flint 40B A2B | J-O + J-A | 0.9 | 0.8 | 0.8 | 0.7 | 1.1 | 1.3 |
| Spark-X2.6-4B | J-O + J-A | 0.9 | 0.6 | 0.9 | 0.9 | 1.6 | 0.8 |
| Codegemma 7B Instruct V1.1 | J-O + J-A | 0.9 | 0.6 | 1.1 | 1.1 | 1.3 | 0.5 |
| gemma-4-31B-it | J-O + J-A | 0.9 | 0.5 | 1.0 | 0.7 | 0.9 | 1.5 |
| Cogito 3B V1 Preview Llama | J-O + J-A | 0.9 | 0.5 | 1.0 | 1.5 | 1.3 | 0.4 |
| Exaone Deep 2.4B | J-O + J-A | 0.9 | 0.5 | 0.7 | 1.2 | 1.1 | 1.1 |
| Qwen3-30B-A3B | J-O + J-A | 0.9 | 0.7 | 1.1 | 1.0 | 0.6 | 1.3 |
| Olmo 3.1 32B Instruct | J-O + J-A | 0.9 | 0.8 | 1.2 | 0.8 | 0.7 | 1.3 |
| Gemma2 27B Instruct | J-O + J-A | 0.9 | 0.9 | 0.9 | 0.8 | 0.7 | 1.4 |
| Cogito 14B V1 Preview Qwen | J-O + J-A | 0.9 | 0.8 | 0.8 | 1.4 | 0.9 | 0.8 |
| Nanbeige_Nanbeige4.3-33B | J-O + J-A | 0.9 | 0.6 | 0.9 | 1.0 | 1.3 | 0.9 |
| Gemma 4 12B IT | J-O + J-A | 0.9 | 0.6 | 1.2 | 0.6 | 1.0 | 1.3 |
| Granite3.1 Dense 2B Instruct | J-O + J-A | 0.9 | 0.6 | 0.8 | 1.4 | 1.3 | 0.5 |
| Deepcoder 14B Preview | J-O + J-A | 0.9 | 0.8 | 0.2 | 1.1 | 1.5 | 1.0 |
| Qwen3-14B | J-O + J-A | 0.9 | 0.6 | 0.9 | 0.6 | 1.1 | 1.3 |
| Qwen3.5-27B | J-O + J-A | 0.9 | 0.5 | 0.9 | 0.5 | 0.8 | 1.9 |
| Exaone3.5 32B Instruct | J-O + J-A | 0.9 | 0.8 | 0.9 | 0.9 | 0.7 | 1.3 |
| Qwen3-Coder-30B-A3B-Instruct | J-O + J-A | 0.9 | 0.5 | 1.0 | 1.2 | 0.9 | 1.0 |
| Ornith-1.5-9B | J-O + J-A | 0.9 | 0.7 | 1.1 | 1.3 | 1.0 | 0.5 |
| Stablelm2 1.6B Chat | J-O + J-A | 0.9 | 0.4 | 0.8 | 1.4 | 0.7 | 1.3 |
| DeepSeek-R1-0528-Qwen3-8B | J-O + J-A | 0.9 | 0.7 | 0.3 | 1.5 | 1.0 | 1.0 |
| DeepSeek-R1-Distill-Qwen-32B | J-O + J-A | 0.9 | 0.5 | 0.5 | 1.6 | 0.8 | 1.1 |
| Qwen2.5-Coder-32B-Instruct | J-O + J-A | 0.9 | 0.9 | 0.7 | 0.8 | 0.9 | 1.1 |
| Qwen3-32B | J-O + J-A | 0.9 | 0.6 | 0.9 | 1.1 | 1.0 | 0.9 |
| Olmo-3-7B-Instruct | J-O + J-A | 0.9 | 0.5 | 1.1 | 0.7 | 1.2 | 0.9 |
| Lfm2.5 Thinking 1.2B | J-O + J-A | 0.9 | 0.6 | 0.4 | 1.1 | 1.3 | 1.1 |
| Codellama 13B Instruct | J-O + J-A | 0.9 | 0.8 | 0.5 | 1.4 | 0.8 | 0.9 |
| Stablelm2 12B Chat | J-O + J-A | 0.9 | 0.6 | 0.8 | 1.5 | 1.0 | 0.5 |
| Muse Glimmer 30B | J-O + J-A | 0.9 | 0.7 | 0.8 | 0.6 | 0.9 | 1.3 |
| Granite4.1 30B | J-O + J-A | 0.9 | 0.6 | 0.8 | 0.8 | 0.9 | 1.1 |
| Mistral Small 24B Instruct 2501 | J-O + J-A | 0.9 | 0.6 | 0.9 | 0.9 | 0.8 | 1.1 |
| Qwen3-VL-30B-A3B-Instruct | J-O + J-A | 0.9 | 0.5 | 1.0 | 1.1 | 0.7 | 1.0 |
| Qwen3-4B-Thinking-2507 | J-O + J-A | 0.9 | 0.5 | 0.4 | 0.6 | 1.4 | 1.4 |
| Qwen3.5-35B-A3B | J-O + J-A | 0.8 | 0.5 | 1.0 | 0.9 | 1.0 | 0.9 |
| Olmo-3-7B-Think | J-O + J-A | 0.8 | 0.7 | 0.3 | 0.4 | 1.8 | 1.0 |
| Llama-3.1-8B-Instruct | J-O + J-A | 0.8 | 0.9 | 0.9 | 0.7 | 1.0 | 0.8 |
| Nemotron3 33B | J-O + J-A | 0.8 | 0.5 | 0.9 | 1.2 | 1.1 | 0.6 |
| Qwen 14B Chat V1.5 | J-O + J-A | 0.8 | 0.7 | 0.2 | 1.1 | 1.3 | 0.9 |
| Ornith 9B | J-O + J-A | 0.8 | 0.5 | 0.9 | 0.7 | 1.0 | 1.1 |
| QwQ 32B | J-O + J-A | 0.8 | 0.8 | 0.5 | 1.2 | 1.0 | 0.8 |
| Qwen3-Coder-Next | J-O + J-A | 0.8 | 0.5 | 0.9 | 1.0 | 0.8 | 1.0 |
| Deepseek Coder 1.3B Instruct | J-O + J-A | 0.8 | 0.6 | 0.9 | 1.6 | 0.7 | 0.3 |
| Deepseek Coder 6.7B Instruct | J-O + J-A | 0.8 | 0.7 | 1.0 | 1.3 | 0.8 | 0.3 |
| Lfm2 24B | J-O + J-A | 0.8 | 0.6 | 0.8 | 0.8 | 1.3 | 0.6 |
| DeepSeek-R1-Distill-Qwen-7B | J-O + J-A | 0.8 | 0.7 | 0.3 | 1.3 | 1.1 | 0.8 |
| Qwen3.8-27B | J-O + J-A | 0.8 | 0.5 | 1.0 | 0.7 | 1.0 | 0.8 |
| Qwen 1.8B Chat V1.5 | J-O + J-A | 0.8 | 0.5 | 0.2 | 1.4 | 0.8 | 1.1 |
| Deepseek R1 1.5B Qwen Distill | J-O + J-A | 0.8 | 0.7 | 0.3 | 1.3 | 0.5 | 1.1 |
| Deepseek Coder 33B Instruct | J-O + J-A | 0.8 | 0.6 | 1.2 | 1.3 | 0.6 | 0.3 |
| Gemma4 E4B IT | J-O + J-A | 0.8 | 0.6 | 0.8 | 0.5 | 0.9 | 1.1 |
| Qwen3.6-27B | J-O + J-A | 0.8 | 0.6 | 0.8 | 0.5 | 0.9 | 1.1 |
| Deepscaler 1.5B Preview | J-O + J-A | 0.8 | 0.7 | 0.2 | 1.5 | 0.7 | 0.9 |
| gemma-4-26B-A4B-it | J-O + J-A | 0.7 | 0.7 | 0.8 | 0.3 | 0.8 | 1.1 |
| Codegemma 2B Code V1.1 | J-O + J-A | 0.7 | 1.1 | 0.1 | 1.3 | 0.5 | 0.6 |
| gpt-oss-120b | J-A + J-D | 0.7 | 0.9 | 0.5 | 0.2 | 1.0 | 0.9 |
| Nemotron 3.5 Bolt E4B | J-O + J-A | 0.7 | 0.5 | 0.9 | 0.4 | 0.5 | 1.1 |
| gpt-oss-20b | J-A + J-D | 0.6 | 0.3 | 0.5 | 0.3 | 1.0 | 1.0 |
| Qwen3 VL 2B Thinking | J-O + J-A | 0.6 | 0.5 | 0 | 1.1 | 0.6 | 0.8 |
| Magistral Small (2506) | J-O + J-A | 0.4 | 0.1 | 0 | 0.5 | 1.0 | 0.5 |
Withheld
These models are kept out of the ranking, so none of their judge figures appear on this page:
See also the methodology and the public questions.