Safety
Harmful requests that a careful model should refuse, and harmless requests that it should answer. A refusal detector checks the reply in both directions, and the judges settle the cases it cannot call.
This category counts toward Overall. The two judge models also score every item, and their scores are shown here separately. How the battery works.
Ranking
Models are ranked by their mean score in this category, from 0 to 100. Only models with at least 80% of the category's items scored, and none pending, are ranked here. The rule is the same as for Overall.
| Rank | Model | Mean (0 to 100) | Items scored | Judge A mean | Judge B mean | Agreement |
|---|---|---|---|---|---|---|
| 1 | Apertus-8B-Instruct-2509 | 100.0 | 24 / 24 | 7.7openai/gpt-5.6-luna | 7.7anthropic/claude-haiku-5.5 | high |
| 1 | Aya Expanse 32B | 100.0 | 24 / 24 | 7.7openai/gpt-5.6-luna | 8.4anthropic/claude-haiku-5.5 | high |
| 1 | Aya Expanse 8B | 100.0 | 24 / 24 | 6.9openai/gpt-5.6-luna | 8.0anthropic/claude-haiku-5.5 | high |
| 1 | Cogito 14B V1 Preview Qwenfine-tune of Qwen 2.5 | 100.0 | 24 / 24 | 8.5openai/gpt-5.6-luna | 8.1anthropic/claude-haiku-5.5 | high |
| 1 | Cogito 3B V1 Preview Llamafine-tune of Llama 3.2 | 100.0 | 24 / 24 | 7.4openai/gpt-5.6-luna | 7.3anthropic/claude-haiku-5.5 | high |
| 1 | Cogito 8B V1 Preview Llamafine-tune of Llama 3.1 | 100.0 | 24 / 24 | 8.2openai/gpt-5.6-luna | 8.2anthropic/claude-haiku-5.5 | high |
| 1 | DeepSeek-R1-0528-Qwen3-8B | 100.0 | 24 / 24 | 7.5openai/gpt-5.6-luna | 7.7anthropic/claude-haiku-5.5 | high |
| 1 | Devstral 24B Small 2505code model | 100.0 | 24 / 24 | 8.6openai/gpt-5.6-luna | 8.4anthropic/claude-haiku-5.5 | high |
| 1 | Devstral Small 2 24B Instruct (2512)code model | 100.0 | 24 / 24 | 8.3openai/gpt-5.6-luna | 8.5anthropic/claude-haiku-5.5 | high |
| 1 | Exaone3.5 32B Instruct | 100.0 | 24 / 24 | 8.6openai/gpt-5.6-luna | 8.6anthropic/claude-haiku-5.5 | high |
| 1 | Falcon3 10B Instruct | 100.0 | 24 / 24 | 8.3openai/gpt-5.6-luna | 7.9anthropic/claude-haiku-5.5 | medium |
| 1 | Falcon3 1B Instruct | 100.0 | 24 / 24 | 5.8openai/gpt-5.6-luna | 6.2anthropic/claude-haiku-5.5 | medium |
| 1 | Flint 40B A2B | 100.0 | 24 / 24 | 7.2openai/gpt-5.6-luna | 6.2anthropic/claude-haiku-5.5 | high |
| 1 | Gemma 3n E4B IT | 100.0 | 24 / 24 | 6.4openai/gpt-5.6-luna | 7.5anthropic/claude-haiku-5.5 | high |
| 1 | Gemma 4 12B IT | 100.0 | 24 / 24 | 8.2openai/gpt-5.6-luna | 8.4anthropic/claude-haiku-5.5 | high |
| 1 | Gemma 7B Instruct V1.1 | 100.0 | 24 / 24 | 7.1openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | medium |
| 1 | gemma-2-9b-it | 100.0 | 24 / 24 | 8.1openai/gpt-5.6-luna | 8.4anthropic/claude-haiku-5.5 | high |
| 1 | gemma-3-12b-it | 100.0 | 24 / 24 | 6.4openai/gpt-5.6-luna | 7.7anthropic/claude-haiku-5.5 | high |
| 1 | gemma-3-4b-it | 100.0 | 24 / 24 | 6.5openai/gpt-5.6-luna | 7.4anthropic/claude-haiku-5.5 | high |
| 1 | gemma-4-26B-A4B-it | 100.0 | 24 / 24 | 8.5openai/gpt-5.6-luna | 8.4anthropic/claude-haiku-5.5 | high |
| 1 | gemma-4-31B-it | 100.0 | 24 / 24 | 8.5openai/gpt-5.6-luna | 8.5anthropic/claude-haiku-5.5 | high |
| 1 | Gemma2 2B Instruct | 100.0 | 24 / 24 | 7.1openai/gpt-5.6-luna | 7.6anthropic/claude-haiku-5.5 | high |
| 1 | Gemma3 1B IT | 100.0 | 24 / 24 | 5.2openai/gpt-5.6-luna | 6.3anthropic/claude-haiku-5.5 | high |
| 1 | Gemma3n E2B IT | 100.0 | 24 / 24 | 5.8openai/gpt-5.6-luna | 6.9anthropic/claude-haiku-5.5 | high |
| 1 | Gemma4 E2B IT | 100.0 | 24 / 24 | 7.5openai/gpt-5.6-luna | 7.6anthropic/claude-haiku-5.5 | high |
| 1 | GLM-4.7-Flash | 100.0 | 24 / 24 | 7.1openai/gpt-5.6-luna | 8.0anthropic/claude-haiku-5.5 | medium |
| 1 | gpt-oss-120b | 100.0 | 24 / 24 | 7.8anthropic/claude-haiku-5.5 | 8.8deepseek/deepseek-v4.1-flash | high |
| 1 | Granite3 Dense 8B Instruct | 100.0 | 24 / 24 | 7.6openai/gpt-5.6-luna | 7.9anthropic/claude-haiku-5.5 | high |
| 1 | Granite3.3 2B | 100.0 | 24 / 24 | 7.1openai/gpt-5.6-luna | 7.4anthropic/claude-haiku-5.5 | high |
| 1 | Granite3.3 8B | 100.0 | 24 / 24 | 7.8openai/gpt-5.6-luna | 7.6anthropic/claude-haiku-5.5 | high |
| 1 | Granite4 1B H | 100.0 | 24 / 24 | 6.8openai/gpt-5.6-luna | 6.5anthropic/claude-haiku-5.5 | high |
| 1 | Granite4 Small H | 100.0 | 24 / 24 | 8.1openai/gpt-5.6-luna | 8.1anthropic/claude-haiku-5.5 | high |
| 1 | Granite4.1 30B | 100.0 | 24 / 24 | 8.5openai/gpt-5.6-luna | 8.5anthropic/claude-haiku-5.5 | high |
| 1 | Granite4.2 30B | 100.0 | 24 / 24 | 6.8openai/gpt-5.6-luna | 8.3anthropic/claude-haiku-5.5 | medium |
| 1 | Granite4.2 3B | 100.0 | 24 / 24 | 6.2openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | medium |
| 1 | Granite4.2 8B | 100.0 | 24 / 24 | 5.8openai/gpt-5.6-luna | 7.7anthropic/claude-haiku-5.5 | medium |
| 1 | Internlm2 7B Chat V2.5 | 100.0 | 24 / 24 | 7.2openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | medium |
| 1 | Laguna XS 2.1 | 100.0 | 24 / 24 | 8.1openai/gpt-5.6-luna | 8.6anthropic/claude-haiku-5.5 | high |
| 1 | Lfm2 24B | 100.0 | 24 / 24 | 8.2openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | high |
| 1 | Lfm2.5 8B A1B | 100.0 | 24 / 24 | 6.2openai/gpt-5.6-luna | 6.4anthropic/claude-haiku-5.5 | high |
| 1 | Llama 4 Scout 17B-16E Instruct | 100.0 | 24 / 24 | 8.2openai/gpt-5.6-luna | 8.0anthropic/claude-haiku-5.5 | high |
| 1 | Llama-3.1-8B-Instruct | 100.0 | 24 / 24 | 7.2openai/gpt-5.6-luna | 7.3anthropic/claude-haiku-5.5 | high |
| 1 | Llama-3.2-3B-Instruct | 100.0 | 24 / 24 | 6.8openai/gpt-5.6-luna | 7.0anthropic/claude-haiku-5.5 | high |
| 1 | Llama-3.3-70B-Instruct | 100.0 | 24 / 24 | 8.2openai/gpt-5.6-luna | 8.0anthropic/claude-haiku-5.5 | high |
| 1 | Medgemma 27B ITfine-tune of Gemma 3medical model | 100.0 | 24 / 24 | 7.3openai/gpt-5.6-luna | 8.0anthropic/claude-haiku-5.5 | high |
| 1 | medgemma-4b-itmedical model | 100.0 | 24 / 24 | 6.8openai/gpt-5.6-luna | 7.3anthropic/claude-haiku-5.5 | high |
| 1 | Ministral 3 14B Instruct (2512) | 100.0 | 24 / 24 | 6.6openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | high |
| 1 | Ministral 3 8B Instruct 2512 | 100.0 | 24 / 24 | 6.8openai/gpt-5.6-luna | 7.6anthropic/claude-haiku-5.5 | high |
| 1 | Mistral Small 24B Instruct 2501 | 100.0 | 24 / 24 | 8.7openai/gpt-5.6-luna | 8.5anthropic/claude-haiku-5.5 | high |
| 1 | Mistral Small 3.2 24B Instruct (2506) | 100.0 | 24 / 24 | 9.0openai/gpt-5.6-luna | 8.7anthropic/claude-haiku-5.5 | high |
| 1 | Mistral Small3.1 24B Instruct 2503 | 100.0 | 24 / 24 | 8.3openai/gpt-5.6-luna | 8.7anthropic/claude-haiku-5.5 | high |
| 1 | Muse Glimmer 30B | 100.0 | 24 / 24 | 8.5openai/gpt-5.6-luna | 8.7anthropic/claude-haiku-5.5 | high |
| 1 | Nanbeige_Nanbeige4.3-33B | 100.0 | 24 / 24 | 6.7openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | high |
| 1 | Nemotron 3.5 Bolt E4B | 100.0 | 24 / 24 | 8.8openai/gpt-5.6-luna | 8.8anthropic/claude-haiku-5.5 | high |
| 1 | Nemotron 3.5 Lightning 30B A3B | 100.0 | 24 / 24 | 7.8openai/gpt-5.6-luna | 8.4anthropic/claude-haiku-5.5 | high |
| 1 | Nemotron Cascade 2 30B A3B | 100.0 | 24 / 24 | 6.5openai/gpt-5.6-luna | 7.6anthropic/claude-haiku-5.5 | medium |
| 1 | Nemotron3 33B | 100.0 | 24 / 24 | 7.3openai/gpt-5.6-luna | 8.1anthropic/claude-haiku-5.5 | high |
| 1 | NVIDIA Nemotron 3 Nano 30B-A3B | 100.0 | 24 / 24 | 7.3openai/gpt-5.6-luna | 7.5anthropic/claude-haiku-5.5 | high |
| 1 | Olmo 3.1 32B Instruct | 100.0 | 24 / 24 | 8.5openai/gpt-5.6-luna | 8.6anthropic/claude-haiku-5.5 | high |
| 1 | Olmo2 13B 1124 Instruct | 100.0 | 24 / 24 | 8.1openai/gpt-5.6-luna | 8.5anthropic/claude-haiku-5.5 | high |
| 1 | Olmo2 7B 1124 Instruct | 100.0 | 24 / 24 | 8.0openai/gpt-5.6-luna | 8.3anthropic/claude-haiku-5.5 | high |
| 1 | Ornith 9B | 100.0 | 24 / 24 | 8.6openai/gpt-5.6-luna | 8.8anthropic/claude-haiku-5.5 | high |
| 1 | Ornith-1.5-9B | 100.0 | 24 / 24 | 8.3openai/gpt-5.6-luna | 8.7anthropic/claude-haiku-5.5 | high |
| 1 | Phi-3.5-mini-instruct | 100.0 | 24 / 24 | 6.8openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | medium |
| 1 | phi-4 | 100.0 | 24 / 24 | 8.4openai/gpt-5.6-luna | 7.9anthropic/claude-haiku-5.5 | high |
| 1 | Phi3 14B Medium 128K Instruct | 100.0 | 24 / 24 | 7.5openai/gpt-5.6-luna | 7.4anthropic/claude-haiku-5.5 | medium |
| 1 | Phi3 3.8B Mini 128K Instruct | 100.0 | 24 / 24 | 7.1openai/gpt-5.6-luna | 7.2anthropic/claude-haiku-5.5 | high |
| 1 | Qwen2-VL-7B-Instruct | 100.0 | 24 / 24 | 6.8openai/gpt-5.6-luna | 6.9anthropic/claude-haiku-5.5 | high |
| 1 | Qwen2.5-14B-Instruct | 100.0 | 24 / 24 | 8.3openai/gpt-5.6-luna | 8.5anthropic/claude-haiku-5.5 | high |
| 1 | Qwen2.5-32B-Instruct | 100.0 | 24 / 24 | 8.7openai/gpt-5.6-luna | 8.5anthropic/claude-haiku-5.5 | high |
| 1 | Qwen2.5-3B-Instruct | 100.0 | 24 / 24 | 6.7openai/gpt-5.6-luna | 6.9anthropic/claude-haiku-5.5 | high |
| 1 | Qwen2.5-Coder-14B-Instructcode model | 100.0 | 24 / 24 | 7.9openai/gpt-5.6-luna | 7.7anthropic/claude-haiku-5.5 | high |
| 1 | Qwen2.5-Coder-32B-Instructcode model | 100.0 | 24 / 24 | 8.3openai/gpt-5.6-luna | 8.3anthropic/claude-haiku-5.5 | high |
| 1 | Qwen2.5-Coder-7B-Instructcode model | 100.0 | 24 / 24 | 6.6openai/gpt-5.6-luna | 6.9anthropic/claude-haiku-5.5 | medium |
| 1 | Qwen2.5-VL-32B-Instruct | 100.0 | 24 / 24 | 7.8openai/gpt-5.6-luna | 7.9anthropic/claude-haiku-5.5 | high |
| 1 | Qwen2.5-VL-7B-Instruct | 100.0 | 24 / 24 | 7.2openai/gpt-5.6-luna | 7.6anthropic/claude-haiku-5.5 | high |
| 1 | Qwen3-14B | 100.0 | 24 / 24 | 8.2openai/gpt-5.6-luna | 8.2anthropic/claude-haiku-5.5 | high |
| 1 | Qwen3-30B-A3B | 100.0 | 24 / 24 | 8.6openai/gpt-5.6-luna | 8.5anthropic/claude-haiku-5.5 | high |
| 1 | Qwen3-30B-A3B-Instruct-2507 | 100.0 | 24 / 24 | 7.9openai/gpt-5.6-luna | 8.3anthropic/claude-haiku-5.5 | high |
| 1 | Qwen3-32B | 100.0 | 24 / 24 | 8.0openai/gpt-5.6-luna | 8.2anthropic/claude-haiku-5.5 | high |
| 1 | Qwen3-4B | 100.0 | 24 / 24 | 6.8openai/gpt-5.6-luna | 7.6anthropic/claude-haiku-5.5 | medium |
| 1 | Qwen3-4B-Instruct-2507 | 100.0 | 24 / 24 | 6.9openai/gpt-5.6-luna | 8.0anthropic/claude-haiku-5.5 | high |
| 1 | Qwen3-4B-Thinking-2507 | 100.0 | 24 / 24 | 6.0openai/gpt-5.6-luna | 7.3anthropic/claude-haiku-5.5 | high |
| 1 | Qwen3-8B | 100.0 | 24 / 24 | 7.8openai/gpt-5.6-luna | 7.7anthropic/claude-haiku-5.5 | high |
| 1 | Qwen3-Coder-30B-A3B-Instructcode model | 100.0 | 24 / 24 | 8.5openai/gpt-5.6-luna | 8.8anthropic/claude-haiku-5.5 | high |
| 1 | Qwen3-Coder-Nextcode model | 100.0 | 24 / 24 | 8.0openai/gpt-5.6-luna | 8.3anthropic/claude-haiku-5.5 | high |
| 1 | Qwen3-VL-30B-A3B-Instruct | 100.0 | 24 / 24 | 8.3openai/gpt-5.6-luna | 8.4anthropic/claude-haiku-5.5 | high |
| 1 | Qwen3-VL-32B-Instruct | 100.0 | 24 / 24 | 8.2openai/gpt-5.6-luna | 8.5anthropic/claude-haiku-5.5 | high |
| 1 | Qwen3-VL-8B-Instruct | 100.0 | 24 / 24 | 6.9openai/gpt-5.6-luna | 8.2anthropic/claude-haiku-5.5 | high |
| 1 | Qwen3.5-27B | 100.0 | 24 / 24 | 8.1openai/gpt-5.6-luna | 8.4anthropic/claude-haiku-5.5 | high |
| 1 | Qwen3.5-35B-A3B | 100.0 | 24 / 24 | 8.3openai/gpt-5.6-luna | 8.5anthropic/claude-haiku-5.5 | high |
| 1 | Qwen3.5-9B | 100.0 | 24 / 24 | 7.4openai/gpt-5.6-luna | 8.6anthropic/claude-haiku-5.5 | medium |
| 1 | Qwen3.6-27B | 100.0 | 24 / 24 | 7.7openai/gpt-5.6-luna | 8.0anthropic/claude-haiku-5.5 | high |
| 1 | Qwen3.6-35B-A3B | 100.0 | 24 / 24 | 8.0openai/gpt-5.6-luna | 8.3anthropic/claude-haiku-5.5 | high |
| 1 | Qwen3.8-27B | 100.0 | 24 / 24 | 7.6openai/gpt-5.6-luna | 8.3anthropic/claude-haiku-5.5 | high |
| 1 | Sage-A1.5-Q4_K_M | 100.0 | 24 / 24 | 8.1openai/gpt-5.6-luna | 8.4anthropic/claude-haiku-5.5 | high |
| 1 | Spark-X2.6-4B | 100.0 | 24 / 24 | 6.0openai/gpt-5.6-luna | 7.4anthropic/claude-haiku-5.5 | medium |
| 1 | Translategemma 12B ITfine-tune of Gemma 3translation model | 100.0 | 24 / 24 | 7.1openai/gpt-5.6-luna | 7.6anthropic/claude-haiku-5.5 | high |
| 1 | Tulu3 8Bfine-tune of Llama 3.1 | 100.0 | 24 / 24 | 8.1openai/gpt-5.6-luna | 8.1anthropic/claude-haiku-5.5 | high |
| 100 | gemma-3-27b-it | 97.9 | 24 / 24 | 6.4openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | high |
| 100 | Qwen3-VL-4B-Instruct | 97.9 | 24 / 24 | 6.0openai/gpt-5.6-luna | 6.9anthropic/claude-haiku-5.5 | high |
| 102 | Codellama 34B Instructcode model | 95.8 | 24 / 24 | 6.4openai/gpt-5.6-luna | 6.9anthropic/claude-haiku-5.5 | high |
| 102 | Command R7B (12-2024) | 95.8 | 24 / 24 | 6.8openai/gpt-5.6-luna | 7.3anthropic/claude-haiku-5.5 | high |
| 102 | Command R7B Arabic 7B 02 2025 | 95.8 | 24 / 24 | 7.0openai/gpt-5.6-luna | 7.4anthropic/claude-haiku-5.5 | high |
| 102 | Deepseek Llm 7B Chat | 95.8 | 24 / 24 | 6.0openai/gpt-5.6-luna | 6.2anthropic/claude-haiku-5.5 | high |
| 102 | Deepseek V2 16B Lite Chat | 95.8 | 24 / 24 | 6.9openai/gpt-5.6-luna | 6.8anthropic/claude-haiku-5.5 | high |
| 102 | DeepSeek-Coder-V2-Lite-Instructcode model | 95.8 | 24 / 24 | 7.0openai/gpt-5.6-luna | 6.8anthropic/claude-haiku-5.5 | high |
| 102 | Falcon3 7B Instruct | 95.8 | 24 / 24 | 7.7openai/gpt-5.6-luna | 7.9anthropic/claude-haiku-5.5 | high |
| 102 | Gemma2 27B Instruct | 95.8 | 24 / 24 | 8.3openai/gpt-5.6-luna | 8.4anthropic/claude-haiku-5.5 | high |
| 102 | Gemma4 E4B IT | 95.8 | 24 / 24 | 7.8openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | high |
| 102 | GLM-4.5-Air | 95.8 | 24 / 24 | 6.0openai/gpt-5.6-luna | 7.5anthropic/claude-haiku-5.5 | medium |
| 102 | Glm4 9B Chat | 95.8 | 24 / 24 | 7.1openai/gpt-5.6-luna | 7.5anthropic/claude-haiku-5.5 | high |
| 102 | granite-4.1-3b | 95.8 | 24 / 24 | 7.2openai/gpt-5.6-luna | 7.7anthropic/claude-haiku-5.5 | high |
| 102 | Granite3 Dense 2B Instruct | 95.8 | 24 / 24 | 6.9openai/gpt-5.6-luna | 6.8anthropic/claude-haiku-5.5 | high |
| 102 | Granite3.1 Dense 8B Instruct | 95.8 | 24 / 24 | 7.2openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | medium |
| 102 | Granite3.1 Moe 3B Instruct | 95.8 | 24 / 24 | 6.3openai/gpt-5.6-luna | 6.7anthropic/claude-haiku-5.5 | high |
| 102 | Granite3.2 8B Instruct | 95.8 | 24 / 24 | 7.3openai/gpt-5.6-luna | 8.0anthropic/claude-haiku-5.5 | high |
| 102 | Granite4 3B | 95.8 | 24 / 24 | 6.8openai/gpt-5.6-luna | 7.1anthropic/claude-haiku-5.5 | high |
| 102 | Granite4 Micro H | 95.8 | 24 / 24 | 7.2openai/gpt-5.6-luna | 7.4anthropic/claude-haiku-5.5 | high |
| 102 | Granite4 Tiny H | 95.8 | 24 / 24 | 7.7openai/gpt-5.6-luna | 7.5anthropic/claude-haiku-5.5 | high |
| 102 | Granite4.1 8B | 95.8 | 24 / 24 | 7.9openai/gpt-5.6-luna | 8.4anthropic/claude-haiku-5.5 | high |
| 102 | Lfm3 1.7B | 95.8 | 24 / 24 | 6.2openai/gpt-5.6-luna | 6.2anthropic/claude-haiku-5.5 | high |
| 102 | Medgemma1.5 4B ITfine-tune of Gemma 3medical model | 95.8 | 24 / 24 | 6.7openai/gpt-5.6-luna | 7.5anthropic/claude-haiku-5.5 | high |
| 102 | Meta-Llama-3-8B-Instruct | 95.8 | 24 / 24 | 7.2openai/gpt-5.6-luna | 6.8anthropic/claude-haiku-5.5 | high |
| 102 | Ministral 3 3B Instruct 2512 | 95.8 | 24 / 24 | 6.1openai/gpt-5.6-luna | 7.4anthropic/claude-haiku-5.5 | high |
| 102 | Mistral-7B-Instruct-v0.2 | 95.8 | 24 / 24 | 6.8openai/gpt-5.6-luna | 7.3anthropic/claude-haiku-5.5 | high |
| 102 | Olmo-3-7B-Instruct | 95.8 | 24 / 24 | 7.1openai/gpt-5.6-luna | 7.7anthropic/claude-haiku-5.5 | high |
| 102 | Phi-4-mini-instruct | 95.8 | 24 / 24 | 7.4openai/gpt-5.6-luna | 7.3anthropic/claude-haiku-5.5 | high |
| 102 | Qwen 14B Chat V1.5 | 95.8 | 24 / 24 | 6.9openai/gpt-5.6-luna | 7.1anthropic/claude-haiku-5.5 | high |
| 102 | Qwen2.5-7B-Instruct | 95.8 | 24 / 24 | 7.7openai/gpt-5.6-luna | 7.7anthropic/claude-haiku-5.5 | high |
| 102 | Qwen2.5-Coder-3B-Instructcode model | 95.8 | 24 / 24 | 5.9openai/gpt-5.6-luna | 6.0anthropic/claude-haiku-5.5 | high |
| 102 | Qwen3.5 2B | 95.8 | 24 / 24 | 5.8openai/gpt-5.6-luna | 6.8anthropic/claude-haiku-5.5 | medium |
| 102 | Qwen3.5-4B | 95.8 | 24 / 24 | 7.0openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | high |
| 102 | QwQ 32B | 95.8 | 24 / 24 | 7.9openai/gpt-5.6-luna | 8.3anthropic/claude-haiku-5.5 | high |
| 102 | Stablelm2 12B Chat | 95.8 | 24 / 24 | 7.3openai/gpt-5.6-luna | 7.4anthropic/claude-haiku-5.5 | high |
| 102 | YI 34B Chat V1.5 | 95.8 | 24 / 24 | 7.3openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | high |
| 102 | YI 9B Chat V1.5 | 95.8 | 24 / 24 | 6.6openai/gpt-5.6-luna | 6.9anthropic/claude-haiku-5.5 | high |
| 138 | Aya 8B 23 | 91.7 | 24 / 24 | 6.0openai/gpt-5.6-luna | 6.4anthropic/claude-haiku-5.5 | high |
| 138 | Codegemma 7B Instruct V1.1code model | 91.7 | 24 / 24 | 6.0openai/gpt-5.6-luna | 6.6anthropic/claude-haiku-5.5 | high |
| 138 | Command R 35B 08 2024 | 91.7 | 24 / 24 | 7.7openai/gpt-5.6-luna | 8.0anthropic/claude-haiku-5.5 | high |
| 138 | Deepcoder 14B Previewfine-tune of DeepSeek-R1-Distill-Qwen 14Bcode model | 91.7 | 24 / 24 | 6.8openai/gpt-5.6-luna | 7.6anthropic/claude-haiku-5.5 | high |
| 138 | DeepSeek-R1-Distill-Qwen-32B | 91.7 | 24 / 24 | 7.5openai/gpt-5.6-luna | 7.7anthropic/claude-haiku-5.5 | high |
| 138 | EXAONE-3.5-7.8B-Instruct | 91.7 | 24 / 24 | 7.0openai/gpt-5.6-luna | 8.0anthropic/claude-haiku-5.5 | high |
| 138 | Exaone3.5 2.4B Instruct | 91.7 | 24 / 24 | 6.5openai/gpt-5.6-luna | 7.3anthropic/claude-haiku-5.5 | medium |
| 138 | Falcon3 3B Instruct | 91.7 | 24 / 24 | 5.6openai/gpt-5.6-luna | 6.7anthropic/claude-haiku-5.5 | medium |
| 138 | Granite3.1 Dense 2B Instruct | 91.7 | 24 / 24 | 6.1openai/gpt-5.6-luna | 6.7anthropic/claude-haiku-5.5 | high |
| 138 | Hermes-3-Llama-3.1-8B | 91.7 | 24 / 24 | 7.3openai/gpt-5.6-luna | 7.2anthropic/claude-haiku-5.5 | high |
| 138 | Lfm2.5 Thinking 1.2B | 91.7 | 24 / 24 | 6.3openai/gpt-5.6-luna | 6.9anthropic/claude-haiku-5.5 | high |
| 138 | Llama-2-7b-chat-hf | 91.7 | 24 / 24 | 6.3openai/gpt-5.6-luna | 6.5anthropic/claude-haiku-5.5 | high |
| 138 | Mistral Small 22B Instruct 2409 | 91.7 | 24 / 24 | 7.8openai/gpt-5.6-luna | 7.8anthropic/claude-haiku-5.5 | high |
| 138 | Olmo-3-7B-Think | 91.7 | 24 / 24 | 5.9openai/gpt-5.6-luna | 7.4anthropic/claude-haiku-5.5 | medium |
| 138 | Phi-3-mini-4k-instruct | 91.7 | 24 / 24 | 7.3openai/gpt-5.6-luna | 7.0anthropic/claude-haiku-5.5 | high |
| 138 | Qwen2.5 1.5B Instruct | 91.7 | 24 / 24 | 5.5openai/gpt-5.6-luna | 5.4anthropic/claude-haiku-5.5 | high |
| 138 | Qwen2.5-VL-3B-Instruct | 91.7 | 24 / 24 | 5.8openai/gpt-5.6-luna | 5.8anthropic/claude-haiku-5.5 | high |
| 138 | Qwen3 1.7B | 91.7 | 24 / 24 | 5.5openai/gpt-5.6-luna | 5.9anthropic/claude-haiku-5.5 | high |
| 138 | Qwen3 VL 2B Instruct | 91.7 | 24 / 24 | 5.9openai/gpt-5.6-luna | 6.2anthropic/claude-haiku-5.5 | high |
| 138 | Smallthinker 3B Previewfine-tune of Qwen 2.5 3B | 91.7 | 24 / 24 | 5.2openai/gpt-5.6-luna | 5.5anthropic/claude-haiku-5.5 | high |
| 158 | Qwen 7B Chat V1.5 | 89.6 | 24 / 24 | 5.6openai/gpt-5.6-luna | 5.6anthropic/claude-haiku-5.5 | high |
| 159 | Codellama 7B Instructcode model | 87.5 | 24 / 24 | 5.3openai/gpt-5.6-luna | 5.7anthropic/claude-haiku-5.5 | high |
| 159 | DeepSeek-R1-Distill-Qwen-14B | 87.5 | 24 / 24 | 7.1openai/gpt-5.6-luna | 7.5anthropic/claude-haiku-5.5 | high |
| 159 | Falcon2 11Bnot in the Overall ranking | 87.5 | 24 / 24 | 5.6openai/gpt-5.6-luna | 5.4anthropic/claude-haiku-5.5 | high |
| 159 | Granite3 Moe 3B Instruct | 87.5 | 24 / 24 | 5.7openai/gpt-5.6-luna | 5.7anthropic/claude-haiku-5.5 | high |
| 159 | Granite3.2 2B Instruct | 87.5 | 24 / 24 | 6.0openai/gpt-5.6-luna | 6.6anthropic/claude-haiku-5.5 | high |
| 159 | Llama2 13B Chat | 87.5 | 24 / 24 | 6.5openai/gpt-5.6-luna | 6.3anthropic/claude-haiku-5.5 | high |
| 159 | Mistral 7B Instruct V0.3 | 87.5 | 24 / 24 | 6.8openai/gpt-5.6-luna | 6.4anthropic/claude-haiku-5.5 | high |
| 159 | Mistral Nemo 12B Instruct 2407 | 87.5 | 24 / 24 | 6.8openai/gpt-5.6-luna | 7.4anthropic/claude-haiku-5.5 | high |
| 159 | Nemotron Mini 4B Instruct | 87.5 | 24 / 24 | 5.1openai/gpt-5.6-luna | 5.1anthropic/claude-haiku-5.5 | high |
| 159 | YI 6B Chat V1.5 | 87.5 | 24 / 24 | 5.7openai/gpt-5.6-luna | 6.0anthropic/claude-haiku-5.5 | high |
| 169 | Phi 2.7B Chat V2 | 85.4 | 24 / 24 | 5.0openai/gpt-5.6-luna | 4.9anthropic/claude-haiku-5.5 | high |
| 169 | Phi4 Mini Reasoning 3.8B | 85.4 | 24 / 24 | 4.9openai/gpt-5.6-luna | 5.6anthropic/claude-haiku-5.5 | high |
| 169 | Sailor2 8B Chatfine-tune of Qwen 2.5 | 85.4 | 24 / 24 | 5.5openai/gpt-5.6-luna | 6.9anthropic/claude-haiku-5.5 | medium |
| 172 | Codegeex4 9B Allcode model | 83.3 | 24 / 24 | 5.6openai/gpt-5.6-luna | 5.9anthropic/claude-haiku-5.5 | high |
| 172 | Codellama 13B Instructcode model | 83.3 | 24 / 24 | 5.6openai/gpt-5.6-luna | 5.8anthropic/claude-haiku-5.5 | high |
| 172 | Codestral 22B V0.1code model | 83.3 | 24 / 24 | 5.8openai/gpt-5.6-luna | 6.3anthropic/claude-haiku-5.5 | high |
| 172 | Internlm2 1.8B Chat V2.5 | 83.3 | 24 / 24 | 5.9openai/gpt-5.6-luna | 5.3anthropic/claude-haiku-5.5 | high |
| 172 | Llama3.2 1B Instruct | 83.3 | 24 / 24 | 5.5openai/gpt-5.6-luna | 5.8anthropic/claude-haiku-5.5 | high |
| 172 | Qwen2.5 Coder 1.5B Instructcode model | 83.3 | 24 / 24 | 5.0openai/gpt-5.6-luna | 4.8anthropic/claude-haiku-5.5 | high |
| 172 | SmolLM3-3B | 83.3 | 24 / 24 | 5.3openai/gpt-5.6-luna | 6.0anthropic/claude-haiku-5.5 | high |
| 172 | Solar 10.7B Instruct V1 | 83.3 | 24 / 24 | 6.4openai/gpt-5.6-luna | 6.7anthropic/claude-haiku-5.5 | high |
| 180 | Openthinker 7B V2fine-tune of Qwen 2.5 | 81.3 | 24 / 24 | 5.3openai/gpt-5.6-luna | 5.3anthropic/claude-haiku-5.5 | high |
| 181 | Deepseek R1 8B Llama Distill | 79.2 | 24 / 24 | 5.7openai/gpt-5.6-luna | 6.6anthropic/claude-haiku-5.5 | medium |
| 181 | dolphin-2.9.1-yi-1.5-34b | 79.2 | 24 / 24 | 6.1openai/gpt-5.6-luna | 6.4anthropic/claude-haiku-5.5 | high |
| 181 | Gemma 2B Instruct V1.1 | 79.2 | 24 / 24 | 5.5openai/gpt-5.6-luna | 5.9anthropic/claude-haiku-5.5 | high |
| 181 | gpt-oss-20b | 79.2 | 24 / 24 | 5.7anthropic/claude-haiku-5.5 | 6.3deepseek/deepseek-v4.1-flash | high |
| 181 | Granite3 Moe 1B Instruct | 79.2 | 24 / 24 | 4.9openai/gpt-5.6-luna | 4.7anthropic/claude-haiku-5.5 | high |
| 181 | Lfm3 VL 2B | 79.2 | 24 / 24 | 6.0openai/gpt-5.6-luna | 5.8anthropic/claude-haiku-5.5 | high |
| 181 | Marco O1 7Bfine-tune of Qwen 2 7B | 79.2 | 24 / 24 | 6.0openai/gpt-5.6-luna | 6.3anthropic/claude-haiku-5.5 | high |
| 181 | Mixtral 8x7B Instruct V0.1 | 79.2 | 24 / 24 | 6.6openai/gpt-5.6-luna | 7.0anthropic/claude-haiku-5.5 | high |
| 189 | Mathstral 7B V0.1fine-tune of Mistral 7Bmath model | 77.1 | 24 / 24 | 4.7openai/gpt-5.6-luna | 5.2anthropic/claude-haiku-5.5 | high |
| 190 | Qwen2 1.5B Instruct | 75.0 | 24 / 24 | 4.4openai/gpt-5.6-luna | 4.3anthropic/claude-haiku-5.5 | high |
| 191 | Dolphin Phi 2.7B V2.6fine-tune of Phi-2 | 70.8 | 24 / 24 | 4.1openai/gpt-5.6-luna | 4.9anthropic/claude-haiku-5.5 | high |
| 191 | Qwen2 Math 7B Instructmath model | 70.8 | 24 / 24 | 5.1openai/gpt-5.6-luna | 4.6anthropic/claude-haiku-5.5 | high |
| 191 | Smollm2 1.7B Instruct | 70.8 | 24 / 24 | 4.5openai/gpt-5.6-luna | 4.5anthropic/claude-haiku-5.5 | high |
| 191 | Stablelm Zephyr 3Bfine-tune of StableLM 3B | 70.8 | 24 / 24 | 4.8openai/gpt-5.6-luna | 5.0anthropic/claude-haiku-5.5 | high |
| 195 | Granite3.1 Moe 1B Instruct | 68.8 | 24 / 24 | 4.6openai/gpt-5.6-luna | 4.2anthropic/claude-haiku-5.5 | high |
| 196 | DeepSeek-R1-Distill-Qwen-7B | 66.7 | 24 / 24 | 3.8openai/gpt-5.6-luna | 4.0anthropic/claude-haiku-5.5 | high |
| 196 | Exaone Deep 7.8B | 66.7 | 24 / 24 | 3.8openai/gpt-5.6-luna | 4.9anthropic/claude-haiku-5.5 | high |
| 196 | Hermes3 3B Llama3.2fine-tune of Llama 3.2 | 66.7 | 24 / 24 | 4.3openai/gpt-5.6-luna | 5.3anthropic/claude-haiku-5.5 | high |
| 196 | Qwen 4B Chat V1.5 | 66.7 | 24 / 24 | 4.8openai/gpt-5.6-luna | 4.2anthropic/claude-haiku-5.5 | high |
| 200 | Exaone Deep 2.4B | 62.5 | 24 / 24 | 3.4openai/gpt-5.6-luna | 3.7anthropic/claude-haiku-5.5 | high |
| 200 | Qwen2.5 0.5B Instruct | 62.5 | 24 / 24 | 4.5openai/gpt-5.6-luna | 3.8anthropic/claude-haiku-5.5 | high |
| 202 | Codeqwen 7B Chat V1.5code model | 60.4 | 24 / 24 | 4.8openai/gpt-5.6-luna | 3.3anthropic/claude-haiku-5.5 | medium |
| 202 | YI Coder 9B Chatcode model | 60.4 | 24 / 24 | 3.9openai/gpt-5.6-luna | 4.4anthropic/claude-haiku-5.5 | high |
| 204 | Granite4 350M H | 58.3 | 24 / 24 | 4.4openai/gpt-5.6-luna | 3.6anthropic/claude-haiku-5.5 | high |
| 204 | Qwen2 0.5B Instruct | 58.3 | 24 / 24 | 3.5openai/gpt-5.6-luna | 3.0anthropic/claude-haiku-5.5 | high |
| 206 | Granite Code 3B Instructcode model | 50.0 | 24 / 24 | 3.9openai/gpt-5.6-luna | 3.8anthropic/claude-haiku-5.5 | high |
| 206 | Magistral Small (2506) | 50.0 | 24 / 24 | 4.8openai/gpt-5.6-luna | 4.4anthropic/claude-haiku-5.5 | high |
| 206 | Qwen 1.8B Chat V1.5 | 50.0 | 24 / 24 | 3.0openai/gpt-5.6-luna | 2.8anthropic/claude-haiku-5.5 | high |
| 206 | Qwen3 0.6B | 50.0 | 24 / 24 | 3.7openai/gpt-5.6-luna | 3.3anthropic/claude-haiku-5.5 | high |
| 206 | Stablelm2 1.6B Chat | 50.0 | 24 / 24 | 3.3openai/gpt-5.6-luna | 3.8anthropic/claude-haiku-5.5 | high |
| 211 | Smollm2 360M Instruct | 47.9 | 24 / 24 | 2.7openai/gpt-5.6-luna | 2.6anthropic/claude-haiku-5.5 | high |
| 212 | Granite Code 8B Instructcode model | 45.8 | 24 / 24 | 3.7openai/gpt-5.6-luna | 3.5anthropic/claude-haiku-5.5 | high |
| 212 | Smollm2 135M Instruct | 45.8 | 24 / 24 | 2.8openai/gpt-5.6-luna | 2.3anthropic/claude-haiku-5.5 | high |
| 214 | Codegemma 2B Code V1.1code model | 43.8 | 24 / 24 | 1.6openai/gpt-5.6-luna | 1.5anthropic/claude-haiku-5.5 | high |
| 214 | Opencoder 1.5B Instructcode model | 43.8 | 24 / 24 | 3.1openai/gpt-5.6-luna | 2.2anthropic/claude-haiku-5.5 | high |
| 216 | Deepseek Coder 33B Instructcode model | 41.7 | 24 / 24 | 3.1openai/gpt-5.6-luna | 2.7anthropic/claude-haiku-5.5 | high |
| 216 | Qwen3 VL 2B Thinking | 41.7 | 24 / 24 | 2.8openai/gpt-5.6-luna | 2.5anthropic/claude-haiku-5.5 | high |
| 218 | Stable Code 3B Instructcode model | 39.6 | 24 / 24 | 3.6openai/gpt-5.6-luna | 3.2anthropic/claude-haiku-5.5 | high |
| 219 | Deepseek Coder 6.7B Instructcode model | 33.3 | 24 / 24 | 2.8openai/gpt-5.6-luna | 2.3anthropic/claude-haiku-5.5 | high |
| 219 | Sailor2 1B Chatfine-tune of Qwen 2.5 | 33.3 | 24 / 24 | 2.7openai/gpt-5.6-luna | 2.6anthropic/claude-haiku-5.5 | high |
| 221 | Qwen2 Math 1.5B Instructmath model | 29.2 | 24 / 24 | 3.0openai/gpt-5.6-luna | 2.4anthropic/claude-haiku-5.5 | high |
| 221 | Qwen2.5 Coder 0.5B Instructcode model | 29.2 | 24 / 24 | 3.3openai/gpt-5.6-luna | 2.5anthropic/claude-haiku-5.5 | high |
| 223 | Deepcoder 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bcode model | 25.0 | 24 / 24 | 2.6openai/gpt-5.6-luna | 2.1anthropic/claude-haiku-5.5 | high |
| 223 | Deepseek Coder 1.3B Instructcode model | 25.0 | 24 / 24 | 2.3openai/gpt-5.6-luna | 2.0anthropic/claude-haiku-5.5 | high |
| 225 | Qwen 0.5B Chat V1.5 | 22.9 | 24 / 24 | 3.4openai/gpt-5.6-luna | 2.6anthropic/claude-haiku-5.5 | high |
| 225 | YI Coder 1.5B Chatcode model | 22.9 | 24 / 24 | 2.8openai/gpt-5.6-luna | 2.2anthropic/claude-haiku-5.5 | high |
| 227 | Deepscaler 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bmath model | 20.8 | 24 / 24 | 2.5openai/gpt-5.6-luna | 2.1anthropic/claude-haiku-5.5 | high |
| 228 | Deepseek R1 1.5B Qwen Distill | 16.7 | 24 / 24 | 2.0openai/gpt-5.6-luna | 1.7anthropic/claude-haiku-5.5 | high |
A dash means not scored, which is different from 0. Agreement compares the two judges: high means a mean gap under 1.5 points, medium under 3, low otherwise; n/a means fewer than 3 items were scored by both.
Not ranked in this category
- Apertus-8B-Instruct-2509: withheld from the ranking (see its page)
- Falcon3 10B Instruct: withheld from the ranking (see its page)
- Falcon3 1B Instruct: withheld from the ranking (see its page)
- Falcon3 3B Instruct: withheld from the ranking (see its page)
- Falcon3 7B Instruct: withheld from the ranking (see its page)
- Internlm2 1.8B Chat V2.5: withheld from the ranking (see its page)
- Phi-4-reasoning 14B: withheld from the ranking (see its page)
- Phi-4-reasoning-plus 14B: withheld from the ranking (see its page)
- Qwen2 Math 1.5B Instruct: withheld from the ranking (see its page)
- Qwen2 Math 7B Instruct: withheld from the ranking (see its page)
The questions
In run r20261102-bv1-5246ab, 3 of the 24 questions in this category were answered correctly by every ranked model that has a score for them (33 ranked models in that run). Hard questions are the ones that half or fewer of the ranked models of 3B and over answered correctly. This category has 0.
The same ranking as JSON: safety.json.