All categories

Math

Maths questions with a single numeric answer. The model gives its final answer on the last line, and code compares the number.

This category counts toward Overall. How the battery works.

Ranking

Models are ranked by their mean score in this category, from 0 to 100. Only models with at least 80% of the category's items scored, and none pending, are ranked here. The rule is the same as for Overall.

Math: 228 ranked models. Click a column heading to sort.
Rank Model Mean (0 to 100) Items scored
1 Flint 40B A2B 100.0 25 / 25
1 Gemma 4 12B IT 100.0 25 / 25
1 gemma-4-26B-A4B-it 100.0 25 / 25
1 gemma-4-31B-it 100.0 25 / 25
1 Muse Glimmer 30B 100.0 25 / 25
6 Deepseek R1 8B Llama Distill 96.0 25 / 25
6 DeepSeek-R1-Distill-Qwen-7B 96.0 25 / 25
6 gemma-3-12b-it 96.0 25 / 25
6 gemma-3-27b-it 96.0 25 / 25
6 Gemma4 E4B IT 96.0 25 / 25
6 GLM-4.5-Air 96.0 25 / 25
6 gpt-oss-120b 96.0 25 / 25
6 gpt-oss-20b 96.0 25 / 25
6 Granite4.1 30B 96.0 25 / 25
6 Lfm2 24B 96.0 25 / 25
6 Mistral Small 3.2 24B Instruct (2506) 96.0 25 / 25
6 Nemotron 3.5 Bolt E4B 96.0 25 / 25
6 Olmo-3-7B-Instruct 96.0 25 / 25
6 Phi4 Mini Reasoning 3.8B 96.0 25 / 25
6 Qwen3-30B-A3B-Instruct-2507 96.0 25 / 25
21 Deepscaler 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bmath model 92.0 25 / 25
21 DeepSeek-R1-Distill-Qwen-32B 92.0 25 / 25
21 Gemma 3n E4B IT 92.0 25 / 25
21 GLM-4.7-Flash 92.0 25 / 25
21 Nemotron 3.5 Lightning 30B A3B 92.0 25 / 25
21 Qwen2.5-32B-Instruct 92.0 25 / 25
21 Qwen3-Coder-30B-A3B-Instructcode model 92.0 25 / 25
21 Qwen3-Coder-Nextcode model 92.0 25 / 25
21 Qwen3-VL-4B-Instruct 92.0 25 / 25
21 Qwen3-VL-8B-Instruct 92.0 25 / 25
21 Qwen3.5-35B-A3B 92.0 25 / 25
21 Qwen3.6-35B-A3B 92.0 25 / 25
21 Translategemma 12B ITfine-tune of Gemma 3translation model 92.0 25 / 25
34 Deepcoder 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bcode model 88.0 25 / 25
34 DeepSeek-R1-Distill-Qwen-14B 88.0 25 / 25
34 Exaone Deep 7.8B 88.0 25 / 25
34 Gemma4 E2B IT 88.0 25 / 25
34 Granite4 Small H 88.0 25 / 25
34 Granite4.1 8B 88.0 25 / 25
34 Laguna XS 2.1 88.0 25 / 25
34 Lfm2.5 8B A1B 88.0 25 / 25
34 Ministral 3 14B Instruct (2512) 88.0 25 / 25
34 Ministral 3 3B Instruct 2512 88.0 25 / 25
34 Openthinker 7B V2fine-tune of Qwen 2.5 88.0 25 / 25
34 Qwen3-14B 88.0 25 / 25
34 Qwen3-32B 88.0 25 / 25
34 Qwen3-4B 88.0 25 / 25
34 Qwen3-8B 88.0 25 / 25
34 Qwen3-VL-30B-A3B-Instruct 88.0 25 / 25
34 Qwen3-VL-32B-Instruct 88.0 25 / 25
34 Qwen3.5-4B 88.0 25 / 25
34 Qwen3.6-27B 88.0 25 / 25
34 QwQ 32B 88.0 25 / 25
34 Sage-A1.5-Q4_K_M 88.0 25 / 25
55 Cogito 14B V1 Preview Qwenfine-tune of Qwen 2.5 84.0 25 / 25
55 gemma-3-4b-it 84.0 25 / 25
55 Llama 4 Scout 17B-16E Instruct 84.0 25 / 25
55 Llama-3.3-70B-Instruct 84.0 25 / 25
55 Medgemma 27B ITfine-tune of Gemma 3medical model 84.0 25 / 25
55 Mistral Small 24B Instruct 2501 84.0 25 / 25
55 Nemotron3 33B 84.0 25 / 25
55 NVIDIA Nemotron 3 Nano 30B-A3B 84.0 25 / 25
55 Olmo 3.1 32B Instruct 84.0 25 / 25
55 Ornith 9B 84.0 25 / 25
55 Ornith-1.5-9B 84.0 25 / 25
55 phi-4 84.0 25 / 25
55 Qwen2.5-14B-Instruct 84.0 25 / 25
55 Qwen2.5-VL-32B-Instruct 84.0 25 / 25
55 Qwen3-30B-A3B 84.0 25 / 25
55 Qwen3.5-27B 84.0 25 / 25
55 Qwen3.5-9B 84.0 25 / 25
55 Spark-X2.6-4B 84.0 25 / 25
73 granite-4.1-3b 80.0 25 / 25
73 Granite4.2 30B 80.0 25 / 25
73 Lfm2.5 Thinking 1.2B 80.0 25 / 25
73 Ministral 3 8B Instruct 2512 80.0 25 / 25
73 Nanbeige_Nanbeige4.3-33B 80.0 25 / 25
73 Nemotron Cascade 2 30B A3B 80.0 25 / 25
73 Qwen3-4B-Instruct-2507 80.0 25 / 25
73 Qwen3.5 2B 80.0 25 / 25
73 Qwen3.8-27B 80.0 25 / 25
82 Deepseek R1 1.5B Qwen Distill 76.0 25 / 25
82 Devstral 24B Small 2505code model 76.0 25 / 25
82 Devstral Small 2 24B Instruct (2512)code model 76.0 25 / 25
82 Exaone3.5 2.4B Instruct 76.0 25 / 25
82 Exaone3.5 32B Instruct 76.0 25 / 25
82 Falcon3 7B Instruct 76.0 25 / 25
82 Gemma3n E2B IT 76.0 25 / 25
82 Granite3.3 8B 76.0 25 / 25
82 Lfm3 1.7B 76.0 25 / 25
82 Mistral Small3.1 24B Instruct 2503 76.0 25 / 25
82 Phi-4-mini-instruct 76.0 25 / 25
82 Qwen2 Math 7B Instructmath model 76.0 25 / 25
82 Qwen2.5-7B-Instruct 76.0 25 / 25
82 Qwen3-4B-Thinking-2507 76.0 25 / 25
96 Command R7B (12-2024) 72.0 25 / 25
96 Exaone Deep 2.4B 72.0 25 / 25
96 EXAONE-3.5-7.8B-Instruct 72.0 25 / 25
96 medgemma-4b-itmedical model 72.0 25 / 25
96 Mistral Small 22B Instruct 2409 72.0 25 / 25
96 Qwen2 Math 1.5B Instructmath model 72.0 25 / 25
96 Qwen2.5-3B-Instruct 72.0 25 / 25
96 Qwen2.5-Coder-14B-Instructcode model 72.0 25 / 25
96 Qwen2.5-Coder-7B-Instructcode model 72.0 25 / 25
96 Qwen3 1.7B 72.0 25 / 25
96 SmolLM3-3B 72.0 25 / 25
107 Command R7B Arabic 7B 02 2025 68.0 25 / 25
107 Gemma2 27B Instruct 68.0 25 / 25
107 Granite4 Tiny H 68.0 25 / 25
107 Granite4.2 8B 68.0 25 / 25
107 Mathstral 7B V0.1fine-tune of Mistral 7Bmath model 68.0 25 / 25
107 Qwen2.5-VL-7B-Instruct 68.0 25 / 25
107 Smallthinker 3B Previewfine-tune of Qwen 2.5 3B 68.0 25 / 25
114 Cogito 8B V1 Preview Llamafine-tune of Llama 3.1 64.0 25 / 25
114 Gemma3 1B IT 64.0 25 / 25
114 Marco O1 7Bfine-tune of Qwen 2 7B 64.0 25 / 25
114 Qwen2-VL-7B-Instruct 64.0 25 / 25
114 Qwen2.5-Coder-32B-Instructcode model 64.0 25 / 25
114 Qwen2.5-VL-3B-Instruct 64.0 25 / 25
114 Qwen3 VL 2B Instruct 64.0 25 / 25
114 YI 9B Chat V1.5 64.0 25 / 25
122 Falcon3 10B Instruct 60.0 25 / 25
122 gemma-2-9b-it 60.0 25 / 25
122 Glm4 9B Chat 60.0 25 / 25
122 Granite4 Micro H 60.0 25 / 25
122 Internlm2 7B Chat V2.5 60.0 25 / 25
122 Lfm3 VL 2B 60.0 25 / 25
122 YI 34B Chat V1.5 60.0 25 / 25
129 Deepcoder 14B Previewfine-tune of DeepSeek-R1-Distill-Qwen 14Bcode model 56.0 25 / 25
129 DeepSeek-Coder-V2-Lite-Instructcode model 56.0 25 / 25
129 dolphin-2.9.1-yi-1.5-34b 56.0 25 / 25
129 Granite3.2 8B Instruct 56.0 25 / 25
129 Granite4 3B 56.0 25 / 25
129 Granite4.2 3B 56.0 25 / 25
129 Llama-3.1-8B-Instruct 56.0 25 / 25
129 Mistral Nemo 12B Instruct 2407 56.0 25 / 25
129 Phi-3-mini-4k-instruct 56.0 25 / 25
129 Phi3 14B Medium 128K Instruct 56.0 25 / 25
139 Falcon3 3B Instruct 52.0 25 / 25
139 Granite3.1 Dense 8B Instruct 52.0 25 / 25
139 Granite3.2 2B Instruct 52.0 25 / 25
139 Granite3.3 2B 52.0 25 / 25
139 Granite4 1B H 52.0 25 / 25
139 Hermes-3-Llama-3.1-8B 52.0 25 / 25
139 Llama-3.2-3B-Instruct 52.0 25 / 25
139 Olmo2 13B 1124 Instruct 52.0 25 / 25
139 Olmo2 7B 1124 Instruct 52.0 25 / 25
139 Qwen2.5 1.5B Instruct 52.0 25 / 25
139 Qwen3 VL 2B Thinking 52.0 25 / 25
139 Sailor2 8B Chatfine-tune of Qwen 2.5 52.0 25 / 25
139 Tulu3 8Bfine-tune of Llama 3.1 52.0 25 / 25
152 Aya Expanse 32B 48.0 25 / 25
152 Codestral 22B V0.1code model 48.0 25 / 25
152 DeepSeek-R1-0528-Qwen3-8B 48.0 25 / 25
152 Magistral Small (2506) 48.0 25 / 25
152 Medgemma1.5 4B ITfine-tune of Gemma 3medical model 48.0 25 / 25
152 Olmo-3-7B-Think 48.0 25 / 25
152 Phi-3.5-mini-instruct 48.0 25 / 25
152 YI 6B Chat V1.5 48.0 25 / 25
160 Gemma2 2B Instruct 44.0 25 / 25
160 Granite3 Dense 8B Instruct 44.0 25 / 25
160 Qwen2.5 Coder 1.5B Instructcode model 44.0 25 / 25
160 Qwen2.5-Coder-3B-Instructcode model 44.0 25 / 25
164 Aya Expanse 8B 40.0 25 / 25
164 Cogito 3B V1 Preview Llamafine-tune of Llama 3.2 40.0 25 / 25
164 Deepseek V2 16B Lite Chat 40.0 25 / 25
164 Dolphin Phi 2.7B V2.6fine-tune of Phi-2 40.0 25 / 25
164 Llama3.2 1B Instruct 40.0 25 / 25
164 Phi3 3.8B Mini 128K Instruct 40.0 25 / 25
164 Qwen 14B Chat V1.5 40.0 25 / 25
164 Qwen3 0.6B 40.0 25 / 25
172 Codegeex4 9B Allcode model 36.0 25 / 25
172 Granite3.1 Dense 2B Instruct 36.0 25 / 25
172 Meta-Llama-3-8B-Instruct 36.0 25 / 25
172 Qwen2.5 0.5B Instruct 36.0 25 / 25
176 Apertus-8B-Instruct-2509 32.0 25 / 25
176 Command R 35B 08 2024 32.0 25 / 25
176 Granite3.1 Moe 3B Instruct 32.0 25 / 25
176 Qwen2 1.5B Instruct 32.0 25 / 25
180 Granite Code 8B Instructcode model 28.0 25 / 25
180 Hermes3 3B Llama3.2fine-tune of Llama 3.2 28.0 25 / 25
180 Internlm2 1.8B Chat V2.5 28.0 25 / 25
183 Codegemma 7B Instruct V1.1code model 24.0 25 / 25
183 Deepseek Llm 7B Chat 24.0 25 / 25
183 Gemma 7B Instruct V1.1 24.0 25 / 25
183 Granite Code 3B Instructcode model 24.0 25 / 25
183 Granite3 Dense 2B Instruct 24.0 25 / 25
183 Mixtral 8x7B Instruct V0.1 24.0 25 / 25
189 Falcon3 1B Instruct 20.0 25 / 25
189 Gemma 2B Instruct V1.1 20.0 25 / 25
189 Granite3 Moe 3B Instruct 20.0 25 / 25
192 Codellama 34B Instructcode model 16.0 25 / 25
192 Falcon2 11Bnot in the Overall ranking 16.0 25 / 25
192 Mistral 7B Instruct V0.3 16.0 25 / 25
192 Qwen 7B Chat V1.5 16.0 25 / 25
192 Solar 10.7B Instruct V1 16.0 25 / 25
192 Stablelm Zephyr 3Bfine-tune of StableLM 3B 16.0 25 / 25
192 YI Coder 9B Chatcode model 16.0 25 / 25
199 Codellama 13B Instructcode model 12.0 25 / 25
199 Deepseek Coder 33B Instructcode model 12.0 25 / 25
199 Granite3.1 Moe 1B Instruct 12.0 25 / 25
199 Mistral-7B-Instruct-v0.2 12.0 25 / 25
199 Nemotron Mini 4B Instruct 12.0 25 / 25
204 Codeqwen 7B Chat V1.5code model 8.0 25 / 25
204 Granite3 Moe 1B Instruct 8.0 25 / 25
204 Llama2 13B Chat 8.0 25 / 25
204 Qwen2 0.5B Instruct 8.0 25 / 25
204 Qwen2.5 Coder 0.5B Instructcode model 8.0 25 / 25
204 Smollm2 1.7B Instruct 8.0 25 / 25
204 Smollm2 360M Instruct 8.0 25 / 25
204 Stablelm2 12B Chat 8.0 25 / 25
212 Codellama 7B Instructcode model 4.0 25 / 25
212 Llama-2-7b-chat-hf 4.0 25 / 25
212 Qwen 1.8B Chat V1.5 4.0 25 / 25
212 Qwen 4B Chat V1.5 4.0 25 / 25
212 Stablelm2 1.6B Chat 4.0 25 / 25
217 Aya 8B 23 0 25 / 25
217 Codegemma 2B Code V1.1code model 0 25 / 25
217 Deepseek Coder 1.3B Instructcode model 0 25 / 25
217 Deepseek Coder 6.7B Instructcode model 0 25 / 25
217 Granite4 350M H 0 25 / 25
217 Opencoder 1.5B Instructcode model 0 25 / 25
217 Phi 2.7B Chat V2 0 25 / 25
217 Qwen 0.5B Chat V1.5 0 25 / 25
217 Sailor2 1B Chatfine-tune of Qwen 2.5 0 25 / 25
217 Smollm2 135M Instruct 0 25 / 25
217 Stable Code 3B Instructcode model 0 25 / 25
217 YI Coder 1.5B Chatcode model 0 25 / 25

A dash means not scored, which is different from 0. Agreement compares the two judges: high means a mean gap under 1.5 points, medium under 3, low otherwise; n/a means fewer than 3 items were scored by both.

Not ranked in this category

The questions

In run r20261102-bv1-5246ab, 0 of the 25 questions in this category were answered correctly by every ranked model that has a score for them (33 ranked models in that run). Hard questions are the ones that half or fewer of the ranked models of 3B and over answered correctly. This category has 4.

The same ranking as JSON: math.json.