All categories

Safety

Harmful requests that a careful model should refuse, and harmless requests that it should answer. A refusal detector checks the reply in both directions, and the judges settle the cases it cannot call.

This category counts toward Overall. The two judge models also score every item, and their scores are shown here separately. How the battery works.

Ranking

Models are ranked by their mean score in this category, from 0 to 100. Only models with at least 80% of the category's items scored, and none pending, are ranked here. The rule is the same as for Overall.

Safety: 228 ranked models. Click a column heading to sort.
Rank Model Mean (0 to 100) Items scored Judge A mean Judge B mean Agreement
1 Apertus-8B-Instruct-2509 100.0 24 / 24 7.7openai/gpt-5.6-luna 7.7anthropic/claude-haiku-5.5 high
1 Aya Expanse 32B 100.0 24 / 24 7.7openai/gpt-5.6-luna 8.4anthropic/claude-haiku-5.5 high
1 Aya Expanse 8B 100.0 24 / 24 6.9openai/gpt-5.6-luna 8.0anthropic/claude-haiku-5.5 high
1 Cogito 14B V1 Preview Qwenfine-tune of Qwen 2.5 100.0 24 / 24 8.5openai/gpt-5.6-luna 8.1anthropic/claude-haiku-5.5 high
1 Cogito 3B V1 Preview Llamafine-tune of Llama 3.2 100.0 24 / 24 7.4openai/gpt-5.6-luna 7.3anthropic/claude-haiku-5.5 high
1 Cogito 8B V1 Preview Llamafine-tune of Llama 3.1 100.0 24 / 24 8.2openai/gpt-5.6-luna 8.2anthropic/claude-haiku-5.5 high
1 DeepSeek-R1-0528-Qwen3-8B 100.0 24 / 24 7.5openai/gpt-5.6-luna 7.7anthropic/claude-haiku-5.5 high
1 Devstral 24B Small 2505code model 100.0 24 / 24 8.6openai/gpt-5.6-luna 8.4anthropic/claude-haiku-5.5 high
1 Devstral Small 2 24B Instruct (2512)code model 100.0 24 / 24 8.3openai/gpt-5.6-luna 8.5anthropic/claude-haiku-5.5 high
1 Exaone3.5 32B Instruct 100.0 24 / 24 8.6openai/gpt-5.6-luna 8.6anthropic/claude-haiku-5.5 high
1 Falcon3 10B Instruct 100.0 24 / 24 8.3openai/gpt-5.6-luna 7.9anthropic/claude-haiku-5.5 medium
1 Falcon3 1B Instruct 100.0 24 / 24 5.8openai/gpt-5.6-luna 6.2anthropic/claude-haiku-5.5 medium
1 Flint 40B A2B 100.0 24 / 24 7.2openai/gpt-5.6-luna 6.2anthropic/claude-haiku-5.5 high
1 Gemma 3n E4B IT 100.0 24 / 24 6.4openai/gpt-5.6-luna 7.5anthropic/claude-haiku-5.5 high
1 Gemma 4 12B IT 100.0 24 / 24 8.2openai/gpt-5.6-luna 8.4anthropic/claude-haiku-5.5 high
1 Gemma 7B Instruct V1.1 100.0 24 / 24 7.1openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 medium
1 gemma-2-9b-it 100.0 24 / 24 8.1openai/gpt-5.6-luna 8.4anthropic/claude-haiku-5.5 high
1 gemma-3-12b-it 100.0 24 / 24 6.4openai/gpt-5.6-luna 7.7anthropic/claude-haiku-5.5 high
1 gemma-3-4b-it 100.0 24 / 24 6.5openai/gpt-5.6-luna 7.4anthropic/claude-haiku-5.5 high
1 gemma-4-26B-A4B-it 100.0 24 / 24 8.5openai/gpt-5.6-luna 8.4anthropic/claude-haiku-5.5 high
1 gemma-4-31B-it 100.0 24 / 24 8.5openai/gpt-5.6-luna 8.5anthropic/claude-haiku-5.5 high
1 Gemma2 2B Instruct 100.0 24 / 24 7.1openai/gpt-5.6-luna 7.6anthropic/claude-haiku-5.5 high
1 Gemma3 1B IT 100.0 24 / 24 5.2openai/gpt-5.6-luna 6.3anthropic/claude-haiku-5.5 high
1 Gemma3n E2B IT 100.0 24 / 24 5.8openai/gpt-5.6-luna 6.9anthropic/claude-haiku-5.5 high
1 Gemma4 E2B IT 100.0 24 / 24 7.5openai/gpt-5.6-luna 7.6anthropic/claude-haiku-5.5 high
1 GLM-4.7-Flash 100.0 24 / 24 7.1openai/gpt-5.6-luna 8.0anthropic/claude-haiku-5.5 medium
1 gpt-oss-120b 100.0 24 / 24 7.8anthropic/claude-haiku-5.5 8.8deepseek/deepseek-v4.1-flash high
1 Granite3 Dense 8B Instruct 100.0 24 / 24 7.6openai/gpt-5.6-luna 7.9anthropic/claude-haiku-5.5 high
1 Granite3.3 2B 100.0 24 / 24 7.1openai/gpt-5.6-luna 7.4anthropic/claude-haiku-5.5 high
1 Granite3.3 8B 100.0 24 / 24 7.8openai/gpt-5.6-luna 7.6anthropic/claude-haiku-5.5 high
1 Granite4 1B H 100.0 24 / 24 6.8openai/gpt-5.6-luna 6.5anthropic/claude-haiku-5.5 high
1 Granite4 Small H 100.0 24 / 24 8.1openai/gpt-5.6-luna 8.1anthropic/claude-haiku-5.5 high
1 Granite4.1 30B 100.0 24 / 24 8.5openai/gpt-5.6-luna 8.5anthropic/claude-haiku-5.5 high
1 Granite4.2 30B 100.0 24 / 24 6.8openai/gpt-5.6-luna 8.3anthropic/claude-haiku-5.5 medium
1 Granite4.2 3B 100.0 24 / 24 6.2openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 medium
1 Granite4.2 8B 100.0 24 / 24 5.8openai/gpt-5.6-luna 7.7anthropic/claude-haiku-5.5 medium
1 Internlm2 7B Chat V2.5 100.0 24 / 24 7.2openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 medium
1 Laguna XS 2.1 100.0 24 / 24 8.1openai/gpt-5.6-luna 8.6anthropic/claude-haiku-5.5 high
1 Lfm2 24B 100.0 24 / 24 8.2openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 high
1 Lfm2.5 8B A1B 100.0 24 / 24 6.2openai/gpt-5.6-luna 6.4anthropic/claude-haiku-5.5 high
1 Llama 4 Scout 17B-16E Instruct 100.0 24 / 24 8.2openai/gpt-5.6-luna 8.0anthropic/claude-haiku-5.5 high
1 Llama-3.1-8B-Instruct 100.0 24 / 24 7.2openai/gpt-5.6-luna 7.3anthropic/claude-haiku-5.5 high
1 Llama-3.2-3B-Instruct 100.0 24 / 24 6.8openai/gpt-5.6-luna 7.0anthropic/claude-haiku-5.5 high
1 Llama-3.3-70B-Instruct 100.0 24 / 24 8.2openai/gpt-5.6-luna 8.0anthropic/claude-haiku-5.5 high
1 Medgemma 27B ITfine-tune of Gemma 3medical model 100.0 24 / 24 7.3openai/gpt-5.6-luna 8.0anthropic/claude-haiku-5.5 high
1 medgemma-4b-itmedical model 100.0 24 / 24 6.8openai/gpt-5.6-luna 7.3anthropic/claude-haiku-5.5 high
1 Ministral 3 14B Instruct (2512) 100.0 24 / 24 6.6openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 high
1 Ministral 3 8B Instruct 2512 100.0 24 / 24 6.8openai/gpt-5.6-luna 7.6anthropic/claude-haiku-5.5 high
1 Mistral Small 24B Instruct 2501 100.0 24 / 24 8.7openai/gpt-5.6-luna 8.5anthropic/claude-haiku-5.5 high
1 Mistral Small 3.2 24B Instruct (2506) 100.0 24 / 24 9.0openai/gpt-5.6-luna 8.7anthropic/claude-haiku-5.5 high
1 Mistral Small3.1 24B Instruct 2503 100.0 24 / 24 8.3openai/gpt-5.6-luna 8.7anthropic/claude-haiku-5.5 high
1 Muse Glimmer 30B 100.0 24 / 24 8.5openai/gpt-5.6-luna 8.7anthropic/claude-haiku-5.5 high
1 Nanbeige_Nanbeige4.3-33B 100.0 24 / 24 6.7openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 high
1 Nemotron 3.5 Bolt E4B 100.0 24 / 24 8.8openai/gpt-5.6-luna 8.8anthropic/claude-haiku-5.5 high
1 Nemotron 3.5 Lightning 30B A3B 100.0 24 / 24 7.8openai/gpt-5.6-luna 8.4anthropic/claude-haiku-5.5 high
1 Nemotron Cascade 2 30B A3B 100.0 24 / 24 6.5openai/gpt-5.6-luna 7.6anthropic/claude-haiku-5.5 medium
1 Nemotron3 33B 100.0 24 / 24 7.3openai/gpt-5.6-luna 8.1anthropic/claude-haiku-5.5 high
1 NVIDIA Nemotron 3 Nano 30B-A3B 100.0 24 / 24 7.3openai/gpt-5.6-luna 7.5anthropic/claude-haiku-5.5 high
1 Olmo 3.1 32B Instruct 100.0 24 / 24 8.5openai/gpt-5.6-luna 8.6anthropic/claude-haiku-5.5 high
1 Olmo2 13B 1124 Instruct 100.0 24 / 24 8.1openai/gpt-5.6-luna 8.5anthropic/claude-haiku-5.5 high
1 Olmo2 7B 1124 Instruct 100.0 24 / 24 8.0openai/gpt-5.6-luna 8.3anthropic/claude-haiku-5.5 high
1 Ornith 9B 100.0 24 / 24 8.6openai/gpt-5.6-luna 8.8anthropic/claude-haiku-5.5 high
1 Ornith-1.5-9B 100.0 24 / 24 8.3openai/gpt-5.6-luna 8.7anthropic/claude-haiku-5.5 high
1 Phi-3.5-mini-instruct 100.0 24 / 24 6.8openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 medium
1 phi-4 100.0 24 / 24 8.4openai/gpt-5.6-luna 7.9anthropic/claude-haiku-5.5 high
1 Phi3 14B Medium 128K Instruct 100.0 24 / 24 7.5openai/gpt-5.6-luna 7.4anthropic/claude-haiku-5.5 medium
1 Phi3 3.8B Mini 128K Instruct 100.0 24 / 24 7.1openai/gpt-5.6-luna 7.2anthropic/claude-haiku-5.5 high
1 Qwen2-VL-7B-Instruct 100.0 24 / 24 6.8openai/gpt-5.6-luna 6.9anthropic/claude-haiku-5.5 high
1 Qwen2.5-14B-Instruct 100.0 24 / 24 8.3openai/gpt-5.6-luna 8.5anthropic/claude-haiku-5.5 high
1 Qwen2.5-32B-Instruct 100.0 24 / 24 8.7openai/gpt-5.6-luna 8.5anthropic/claude-haiku-5.5 high
1 Qwen2.5-3B-Instruct 100.0 24 / 24 6.7openai/gpt-5.6-luna 6.9anthropic/claude-haiku-5.5 high
1 Qwen2.5-Coder-14B-Instructcode model 100.0 24 / 24 7.9openai/gpt-5.6-luna 7.7anthropic/claude-haiku-5.5 high
1 Qwen2.5-Coder-32B-Instructcode model 100.0 24 / 24 8.3openai/gpt-5.6-luna 8.3anthropic/claude-haiku-5.5 high
1 Qwen2.5-Coder-7B-Instructcode model 100.0 24 / 24 6.6openai/gpt-5.6-luna 6.9anthropic/claude-haiku-5.5 medium
1 Qwen2.5-VL-32B-Instruct 100.0 24 / 24 7.8openai/gpt-5.6-luna 7.9anthropic/claude-haiku-5.5 high
1 Qwen2.5-VL-7B-Instruct 100.0 24 / 24 7.2openai/gpt-5.6-luna 7.6anthropic/claude-haiku-5.5 high
1 Qwen3-14B 100.0 24 / 24 8.2openai/gpt-5.6-luna 8.2anthropic/claude-haiku-5.5 high
1 Qwen3-30B-A3B 100.0 24 / 24 8.6openai/gpt-5.6-luna 8.5anthropic/claude-haiku-5.5 high
1 Qwen3-30B-A3B-Instruct-2507 100.0 24 / 24 7.9openai/gpt-5.6-luna 8.3anthropic/claude-haiku-5.5 high
1 Qwen3-32B 100.0 24 / 24 8.0openai/gpt-5.6-luna 8.2anthropic/claude-haiku-5.5 high
1 Qwen3-4B 100.0 24 / 24 6.8openai/gpt-5.6-luna 7.6anthropic/claude-haiku-5.5 medium
1 Qwen3-4B-Instruct-2507 100.0 24 / 24 6.9openai/gpt-5.6-luna 8.0anthropic/claude-haiku-5.5 high
1 Qwen3-4B-Thinking-2507 100.0 24 / 24 6.0openai/gpt-5.6-luna 7.3anthropic/claude-haiku-5.5 high
1 Qwen3-8B 100.0 24 / 24 7.8openai/gpt-5.6-luna 7.7anthropic/claude-haiku-5.5 high
1 Qwen3-Coder-30B-A3B-Instructcode model 100.0 24 / 24 8.5openai/gpt-5.6-luna 8.8anthropic/claude-haiku-5.5 high
1 Qwen3-Coder-Nextcode model 100.0 24 / 24 8.0openai/gpt-5.6-luna 8.3anthropic/claude-haiku-5.5 high
1 Qwen3-VL-30B-A3B-Instruct 100.0 24 / 24 8.3openai/gpt-5.6-luna 8.4anthropic/claude-haiku-5.5 high
1 Qwen3-VL-32B-Instruct 100.0 24 / 24 8.2openai/gpt-5.6-luna 8.5anthropic/claude-haiku-5.5 high
1 Qwen3-VL-8B-Instruct 100.0 24 / 24 6.9openai/gpt-5.6-luna 8.2anthropic/claude-haiku-5.5 high
1 Qwen3.5-27B 100.0 24 / 24 8.1openai/gpt-5.6-luna 8.4anthropic/claude-haiku-5.5 high
1 Qwen3.5-35B-A3B 100.0 24 / 24 8.3openai/gpt-5.6-luna 8.5anthropic/claude-haiku-5.5 high
1 Qwen3.5-9B 100.0 24 / 24 7.4openai/gpt-5.6-luna 8.6anthropic/claude-haiku-5.5 medium
1 Qwen3.6-27B 100.0 24 / 24 7.7openai/gpt-5.6-luna 8.0anthropic/claude-haiku-5.5 high
1 Qwen3.6-35B-A3B 100.0 24 / 24 8.0openai/gpt-5.6-luna 8.3anthropic/claude-haiku-5.5 high
1 Qwen3.8-27B 100.0 24 / 24 7.6openai/gpt-5.6-luna 8.3anthropic/claude-haiku-5.5 high
1 Sage-A1.5-Q4_K_M 100.0 24 / 24 8.1openai/gpt-5.6-luna 8.4anthropic/claude-haiku-5.5 high
1 Spark-X2.6-4B 100.0 24 / 24 6.0openai/gpt-5.6-luna 7.4anthropic/claude-haiku-5.5 medium
1 Translategemma 12B ITfine-tune of Gemma 3translation model 100.0 24 / 24 7.1openai/gpt-5.6-luna 7.6anthropic/claude-haiku-5.5 high
1 Tulu3 8Bfine-tune of Llama 3.1 100.0 24 / 24 8.1openai/gpt-5.6-luna 8.1anthropic/claude-haiku-5.5 high
100 gemma-3-27b-it 97.9 24 / 24 6.4openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 high
100 Qwen3-VL-4B-Instruct 97.9 24 / 24 6.0openai/gpt-5.6-luna 6.9anthropic/claude-haiku-5.5 high
102 Codellama 34B Instructcode model 95.8 24 / 24 6.4openai/gpt-5.6-luna 6.9anthropic/claude-haiku-5.5 high
102 Command R7B (12-2024) 95.8 24 / 24 6.8openai/gpt-5.6-luna 7.3anthropic/claude-haiku-5.5 high
102 Command R7B Arabic 7B 02 2025 95.8 24 / 24 7.0openai/gpt-5.6-luna 7.4anthropic/claude-haiku-5.5 high
102 Deepseek Llm 7B Chat 95.8 24 / 24 6.0openai/gpt-5.6-luna 6.2anthropic/claude-haiku-5.5 high
102 Deepseek V2 16B Lite Chat 95.8 24 / 24 6.9openai/gpt-5.6-luna 6.8anthropic/claude-haiku-5.5 high
102 DeepSeek-Coder-V2-Lite-Instructcode model 95.8 24 / 24 7.0openai/gpt-5.6-luna 6.8anthropic/claude-haiku-5.5 high
102 Falcon3 7B Instruct 95.8 24 / 24 7.7openai/gpt-5.6-luna 7.9anthropic/claude-haiku-5.5 high
102 Gemma2 27B Instruct 95.8 24 / 24 8.3openai/gpt-5.6-luna 8.4anthropic/claude-haiku-5.5 high
102 Gemma4 E4B IT 95.8 24 / 24 7.8openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 high
102 GLM-4.5-Air 95.8 24 / 24 6.0openai/gpt-5.6-luna 7.5anthropic/claude-haiku-5.5 medium
102 Glm4 9B Chat 95.8 24 / 24 7.1openai/gpt-5.6-luna 7.5anthropic/claude-haiku-5.5 high
102 granite-4.1-3b 95.8 24 / 24 7.2openai/gpt-5.6-luna 7.7anthropic/claude-haiku-5.5 high
102 Granite3 Dense 2B Instruct 95.8 24 / 24 6.9openai/gpt-5.6-luna 6.8anthropic/claude-haiku-5.5 high
102 Granite3.1 Dense 8B Instruct 95.8 24 / 24 7.2openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 medium
102 Granite3.1 Moe 3B Instruct 95.8 24 / 24 6.3openai/gpt-5.6-luna 6.7anthropic/claude-haiku-5.5 high
102 Granite3.2 8B Instruct 95.8 24 / 24 7.3openai/gpt-5.6-luna 8.0anthropic/claude-haiku-5.5 high
102 Granite4 3B 95.8 24 / 24 6.8openai/gpt-5.6-luna 7.1anthropic/claude-haiku-5.5 high
102 Granite4 Micro H 95.8 24 / 24 7.2openai/gpt-5.6-luna 7.4anthropic/claude-haiku-5.5 high
102 Granite4 Tiny H 95.8 24 / 24 7.7openai/gpt-5.6-luna 7.5anthropic/claude-haiku-5.5 high
102 Granite4.1 8B 95.8 24 / 24 7.9openai/gpt-5.6-luna 8.4anthropic/claude-haiku-5.5 high
102 Lfm3 1.7B 95.8 24 / 24 6.2openai/gpt-5.6-luna 6.2anthropic/claude-haiku-5.5 high
102 Medgemma1.5 4B ITfine-tune of Gemma 3medical model 95.8 24 / 24 6.7openai/gpt-5.6-luna 7.5anthropic/claude-haiku-5.5 high
102 Meta-Llama-3-8B-Instruct 95.8 24 / 24 7.2openai/gpt-5.6-luna 6.8anthropic/claude-haiku-5.5 high
102 Ministral 3 3B Instruct 2512 95.8 24 / 24 6.1openai/gpt-5.6-luna 7.4anthropic/claude-haiku-5.5 high
102 Mistral-7B-Instruct-v0.2 95.8 24 / 24 6.8openai/gpt-5.6-luna 7.3anthropic/claude-haiku-5.5 high
102 Olmo-3-7B-Instruct 95.8 24 / 24 7.1openai/gpt-5.6-luna 7.7anthropic/claude-haiku-5.5 high
102 Phi-4-mini-instruct 95.8 24 / 24 7.4openai/gpt-5.6-luna 7.3anthropic/claude-haiku-5.5 high
102 Qwen 14B Chat V1.5 95.8 24 / 24 6.9openai/gpt-5.6-luna 7.1anthropic/claude-haiku-5.5 high
102 Qwen2.5-7B-Instruct 95.8 24 / 24 7.7openai/gpt-5.6-luna 7.7anthropic/claude-haiku-5.5 high
102 Qwen2.5-Coder-3B-Instructcode model 95.8 24 / 24 5.9openai/gpt-5.6-luna 6.0anthropic/claude-haiku-5.5 high
102 Qwen3.5 2B 95.8 24 / 24 5.8openai/gpt-5.6-luna 6.8anthropic/claude-haiku-5.5 medium
102 Qwen3.5-4B 95.8 24 / 24 7.0openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 high
102 QwQ 32B 95.8 24 / 24 7.9openai/gpt-5.6-luna 8.3anthropic/claude-haiku-5.5 high
102 Stablelm2 12B Chat 95.8 24 / 24 7.3openai/gpt-5.6-luna 7.4anthropic/claude-haiku-5.5 high
102 YI 34B Chat V1.5 95.8 24 / 24 7.3openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 high
102 YI 9B Chat V1.5 95.8 24 / 24 6.6openai/gpt-5.6-luna 6.9anthropic/claude-haiku-5.5 high
138 Aya 8B 23 91.7 24 / 24 6.0openai/gpt-5.6-luna 6.4anthropic/claude-haiku-5.5 high
138 Codegemma 7B Instruct V1.1code model 91.7 24 / 24 6.0openai/gpt-5.6-luna 6.6anthropic/claude-haiku-5.5 high
138 Command R 35B 08 2024 91.7 24 / 24 7.7openai/gpt-5.6-luna 8.0anthropic/claude-haiku-5.5 high
138 Deepcoder 14B Previewfine-tune of DeepSeek-R1-Distill-Qwen 14Bcode model 91.7 24 / 24 6.8openai/gpt-5.6-luna 7.6anthropic/claude-haiku-5.5 high
138 DeepSeek-R1-Distill-Qwen-32B 91.7 24 / 24 7.5openai/gpt-5.6-luna 7.7anthropic/claude-haiku-5.5 high
138 EXAONE-3.5-7.8B-Instruct 91.7 24 / 24 7.0openai/gpt-5.6-luna 8.0anthropic/claude-haiku-5.5 high
138 Exaone3.5 2.4B Instruct 91.7 24 / 24 6.5openai/gpt-5.6-luna 7.3anthropic/claude-haiku-5.5 medium
138 Falcon3 3B Instruct 91.7 24 / 24 5.6openai/gpt-5.6-luna 6.7anthropic/claude-haiku-5.5 medium
138 Granite3.1 Dense 2B Instruct 91.7 24 / 24 6.1openai/gpt-5.6-luna 6.7anthropic/claude-haiku-5.5 high
138 Hermes-3-Llama-3.1-8B 91.7 24 / 24 7.3openai/gpt-5.6-luna 7.2anthropic/claude-haiku-5.5 high
138 Lfm2.5 Thinking 1.2B 91.7 24 / 24 6.3openai/gpt-5.6-luna 6.9anthropic/claude-haiku-5.5 high
138 Llama-2-7b-chat-hf 91.7 24 / 24 6.3openai/gpt-5.6-luna 6.5anthropic/claude-haiku-5.5 high
138 Mistral Small 22B Instruct 2409 91.7 24 / 24 7.8openai/gpt-5.6-luna 7.8anthropic/claude-haiku-5.5 high
138 Olmo-3-7B-Think 91.7 24 / 24 5.9openai/gpt-5.6-luna 7.4anthropic/claude-haiku-5.5 medium
138 Phi-3-mini-4k-instruct 91.7 24 / 24 7.3openai/gpt-5.6-luna 7.0anthropic/claude-haiku-5.5 high
138 Qwen2.5 1.5B Instruct 91.7 24 / 24 5.5openai/gpt-5.6-luna 5.4anthropic/claude-haiku-5.5 high
138 Qwen2.5-VL-3B-Instruct 91.7 24 / 24 5.8openai/gpt-5.6-luna 5.8anthropic/claude-haiku-5.5 high
138 Qwen3 1.7B 91.7 24 / 24 5.5openai/gpt-5.6-luna 5.9anthropic/claude-haiku-5.5 high
138 Qwen3 VL 2B Instruct 91.7 24 / 24 5.9openai/gpt-5.6-luna 6.2anthropic/claude-haiku-5.5 high
138 Smallthinker 3B Previewfine-tune of Qwen 2.5 3B 91.7 24 / 24 5.2openai/gpt-5.6-luna 5.5anthropic/claude-haiku-5.5 high
158 Qwen 7B Chat V1.5 89.6 24 / 24 5.6openai/gpt-5.6-luna 5.6anthropic/claude-haiku-5.5 high
159 Codellama 7B Instructcode model 87.5 24 / 24 5.3openai/gpt-5.6-luna 5.7anthropic/claude-haiku-5.5 high
159 DeepSeek-R1-Distill-Qwen-14B 87.5 24 / 24 7.1openai/gpt-5.6-luna 7.5anthropic/claude-haiku-5.5 high
159 Falcon2 11Bnot in the Overall ranking 87.5 24 / 24 5.6openai/gpt-5.6-luna 5.4anthropic/claude-haiku-5.5 high
159 Granite3 Moe 3B Instruct 87.5 24 / 24 5.7openai/gpt-5.6-luna 5.7anthropic/claude-haiku-5.5 high
159 Granite3.2 2B Instruct 87.5 24 / 24 6.0openai/gpt-5.6-luna 6.6anthropic/claude-haiku-5.5 high
159 Llama2 13B Chat 87.5 24 / 24 6.5openai/gpt-5.6-luna 6.3anthropic/claude-haiku-5.5 high
159 Mistral 7B Instruct V0.3 87.5 24 / 24 6.8openai/gpt-5.6-luna 6.4anthropic/claude-haiku-5.5 high
159 Mistral Nemo 12B Instruct 2407 87.5 24 / 24 6.8openai/gpt-5.6-luna 7.4anthropic/claude-haiku-5.5 high
159 Nemotron Mini 4B Instruct 87.5 24 / 24 5.1openai/gpt-5.6-luna 5.1anthropic/claude-haiku-5.5 high
159 YI 6B Chat V1.5 87.5 24 / 24 5.7openai/gpt-5.6-luna 6.0anthropic/claude-haiku-5.5 high
169 Phi 2.7B Chat V2 85.4 24 / 24 5.0openai/gpt-5.6-luna 4.9anthropic/claude-haiku-5.5 high
169 Phi4 Mini Reasoning 3.8B 85.4 24 / 24 4.9openai/gpt-5.6-luna 5.6anthropic/claude-haiku-5.5 high
169 Sailor2 8B Chatfine-tune of Qwen 2.5 85.4 24 / 24 5.5openai/gpt-5.6-luna 6.9anthropic/claude-haiku-5.5 medium
172 Codegeex4 9B Allcode model 83.3 24 / 24 5.6openai/gpt-5.6-luna 5.9anthropic/claude-haiku-5.5 high
172 Codellama 13B Instructcode model 83.3 24 / 24 5.6openai/gpt-5.6-luna 5.8anthropic/claude-haiku-5.5 high
172 Codestral 22B V0.1code model 83.3 24 / 24 5.8openai/gpt-5.6-luna 6.3anthropic/claude-haiku-5.5 high
172 Internlm2 1.8B Chat V2.5 83.3 24 / 24 5.9openai/gpt-5.6-luna 5.3anthropic/claude-haiku-5.5 high
172 Llama3.2 1B Instruct 83.3 24 / 24 5.5openai/gpt-5.6-luna 5.8anthropic/claude-haiku-5.5 high
172 Qwen2.5 Coder 1.5B Instructcode model 83.3 24 / 24 5.0openai/gpt-5.6-luna 4.8anthropic/claude-haiku-5.5 high
172 SmolLM3-3B 83.3 24 / 24 5.3openai/gpt-5.6-luna 6.0anthropic/claude-haiku-5.5 high
172 Solar 10.7B Instruct V1 83.3 24 / 24 6.4openai/gpt-5.6-luna 6.7anthropic/claude-haiku-5.5 high
180 Openthinker 7B V2fine-tune of Qwen 2.5 81.3 24 / 24 5.3openai/gpt-5.6-luna 5.3anthropic/claude-haiku-5.5 high
181 Deepseek R1 8B Llama Distill 79.2 24 / 24 5.7openai/gpt-5.6-luna 6.6anthropic/claude-haiku-5.5 medium
181 dolphin-2.9.1-yi-1.5-34b 79.2 24 / 24 6.1openai/gpt-5.6-luna 6.4anthropic/claude-haiku-5.5 high
181 Gemma 2B Instruct V1.1 79.2 24 / 24 5.5openai/gpt-5.6-luna 5.9anthropic/claude-haiku-5.5 high
181 gpt-oss-20b 79.2 24 / 24 5.7anthropic/claude-haiku-5.5 6.3deepseek/deepseek-v4.1-flash high
181 Granite3 Moe 1B Instruct 79.2 24 / 24 4.9openai/gpt-5.6-luna 4.7anthropic/claude-haiku-5.5 high
181 Lfm3 VL 2B 79.2 24 / 24 6.0openai/gpt-5.6-luna 5.8anthropic/claude-haiku-5.5 high
181 Marco O1 7Bfine-tune of Qwen 2 7B 79.2 24 / 24 6.0openai/gpt-5.6-luna 6.3anthropic/claude-haiku-5.5 high
181 Mixtral 8x7B Instruct V0.1 79.2 24 / 24 6.6openai/gpt-5.6-luna 7.0anthropic/claude-haiku-5.5 high
189 Mathstral 7B V0.1fine-tune of Mistral 7Bmath model 77.1 24 / 24 4.7openai/gpt-5.6-luna 5.2anthropic/claude-haiku-5.5 high
190 Qwen2 1.5B Instruct 75.0 24 / 24 4.4openai/gpt-5.6-luna 4.3anthropic/claude-haiku-5.5 high
191 Dolphin Phi 2.7B V2.6fine-tune of Phi-2 70.8 24 / 24 4.1openai/gpt-5.6-luna 4.9anthropic/claude-haiku-5.5 high
191 Qwen2 Math 7B Instructmath model 70.8 24 / 24 5.1openai/gpt-5.6-luna 4.6anthropic/claude-haiku-5.5 high
191 Smollm2 1.7B Instruct 70.8 24 / 24 4.5openai/gpt-5.6-luna 4.5anthropic/claude-haiku-5.5 high
191 Stablelm Zephyr 3Bfine-tune of StableLM 3B 70.8 24 / 24 4.8openai/gpt-5.6-luna 5.0anthropic/claude-haiku-5.5 high
195 Granite3.1 Moe 1B Instruct 68.8 24 / 24 4.6openai/gpt-5.6-luna 4.2anthropic/claude-haiku-5.5 high
196 DeepSeek-R1-Distill-Qwen-7B 66.7 24 / 24 3.8openai/gpt-5.6-luna 4.0anthropic/claude-haiku-5.5 high
196 Exaone Deep 7.8B 66.7 24 / 24 3.8openai/gpt-5.6-luna 4.9anthropic/claude-haiku-5.5 high
196 Hermes3 3B Llama3.2fine-tune of Llama 3.2 66.7 24 / 24 4.3openai/gpt-5.6-luna 5.3anthropic/claude-haiku-5.5 high
196 Qwen 4B Chat V1.5 66.7 24 / 24 4.8openai/gpt-5.6-luna 4.2anthropic/claude-haiku-5.5 high
200 Exaone Deep 2.4B 62.5 24 / 24 3.4openai/gpt-5.6-luna 3.7anthropic/claude-haiku-5.5 high
200 Qwen2.5 0.5B Instruct 62.5 24 / 24 4.5openai/gpt-5.6-luna 3.8anthropic/claude-haiku-5.5 high
202 Codeqwen 7B Chat V1.5code model 60.4 24 / 24 4.8openai/gpt-5.6-luna 3.3anthropic/claude-haiku-5.5 medium
202 YI Coder 9B Chatcode model 60.4 24 / 24 3.9openai/gpt-5.6-luna 4.4anthropic/claude-haiku-5.5 high
204 Granite4 350M H 58.3 24 / 24 4.4openai/gpt-5.6-luna 3.6anthropic/claude-haiku-5.5 high
204 Qwen2 0.5B Instruct 58.3 24 / 24 3.5openai/gpt-5.6-luna 3.0anthropic/claude-haiku-5.5 high
206 Granite Code 3B Instructcode model 50.0 24 / 24 3.9openai/gpt-5.6-luna 3.8anthropic/claude-haiku-5.5 high
206 Magistral Small (2506) 50.0 24 / 24 4.8openai/gpt-5.6-luna 4.4anthropic/claude-haiku-5.5 high
206 Qwen 1.8B Chat V1.5 50.0 24 / 24 3.0openai/gpt-5.6-luna 2.8anthropic/claude-haiku-5.5 high
206 Qwen3 0.6B 50.0 24 / 24 3.7openai/gpt-5.6-luna 3.3anthropic/claude-haiku-5.5 high
206 Stablelm2 1.6B Chat 50.0 24 / 24 3.3openai/gpt-5.6-luna 3.8anthropic/claude-haiku-5.5 high
211 Smollm2 360M Instruct 47.9 24 / 24 2.7openai/gpt-5.6-luna 2.6anthropic/claude-haiku-5.5 high
212 Granite Code 8B Instructcode model 45.8 24 / 24 3.7openai/gpt-5.6-luna 3.5anthropic/claude-haiku-5.5 high
212 Smollm2 135M Instruct 45.8 24 / 24 2.8openai/gpt-5.6-luna 2.3anthropic/claude-haiku-5.5 high
214 Codegemma 2B Code V1.1code model 43.8 24 / 24 1.6openai/gpt-5.6-luna 1.5anthropic/claude-haiku-5.5 high
214 Opencoder 1.5B Instructcode model 43.8 24 / 24 3.1openai/gpt-5.6-luna 2.2anthropic/claude-haiku-5.5 high
216 Deepseek Coder 33B Instructcode model 41.7 24 / 24 3.1openai/gpt-5.6-luna 2.7anthropic/claude-haiku-5.5 high
216 Qwen3 VL 2B Thinking 41.7 24 / 24 2.8openai/gpt-5.6-luna 2.5anthropic/claude-haiku-5.5 high
218 Stable Code 3B Instructcode model 39.6 24 / 24 3.6openai/gpt-5.6-luna 3.2anthropic/claude-haiku-5.5 high
219 Deepseek Coder 6.7B Instructcode model 33.3 24 / 24 2.8openai/gpt-5.6-luna 2.3anthropic/claude-haiku-5.5 high
219 Sailor2 1B Chatfine-tune of Qwen 2.5 33.3 24 / 24 2.7openai/gpt-5.6-luna 2.6anthropic/claude-haiku-5.5 high
221 Qwen2 Math 1.5B Instructmath model 29.2 24 / 24 3.0openai/gpt-5.6-luna 2.4anthropic/claude-haiku-5.5 high
221 Qwen2.5 Coder 0.5B Instructcode model 29.2 24 / 24 3.3openai/gpt-5.6-luna 2.5anthropic/claude-haiku-5.5 high
223 Deepcoder 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bcode model 25.0 24 / 24 2.6openai/gpt-5.6-luna 2.1anthropic/claude-haiku-5.5 high
223 Deepseek Coder 1.3B Instructcode model 25.0 24 / 24 2.3openai/gpt-5.6-luna 2.0anthropic/claude-haiku-5.5 high
225 Qwen 0.5B Chat V1.5 22.9 24 / 24 3.4openai/gpt-5.6-luna 2.6anthropic/claude-haiku-5.5 high
225 YI Coder 1.5B Chatcode model 22.9 24 / 24 2.8openai/gpt-5.6-luna 2.2anthropic/claude-haiku-5.5 high
227 Deepscaler 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bmath model 20.8 24 / 24 2.5openai/gpt-5.6-luna 2.1anthropic/claude-haiku-5.5 high
228 Deepseek R1 1.5B Qwen Distill 16.7 24 / 24 2.0openai/gpt-5.6-luna 1.7anthropic/claude-haiku-5.5 high

A dash means not scored, which is different from 0. Agreement compares the two judges: high means a mean gap under 1.5 points, medium under 3, low otherwise; n/a means fewer than 3 items were scored by both.

Not ranked in this category

The questions

In run r20261102-bv1-5246ab, 3 of the 24 questions in this category were answered correctly by every ranked model that has a score for them (33 ranked models in that run). Hard questions are the ones that half or fewer of the ranked models of 3B and over answered correctly. This category has 0.

The same ranking as JSON: safety.json.