All categories

Tool use

The model is given tool definitions. It must reply with one JSON tool call that has the right tool and arguments, or with no call when none is needed.

This category counts toward Overall. How the battery works.

Ranking

Models are ranked by their mean score in this category, from 0 to 100. Only models with at least 80% of the category's items scored, and none pending, are ranked here. The rule is the same as for Overall.

Tool use: 228 ranked models. Click a column heading to sort.
Rank Model Mean (0 to 100) Items scored
1 Aya Expanse 8B 100.0 20 / 20
1 Cogito 14B V1 Preview Qwenfine-tune of Qwen 2.5 100.0 20 / 20
1 Cogito 8B V1 Preview Llamafine-tune of Llama 3.1 100.0 20 / 20
1 Deepcoder 14B Previewfine-tune of DeepSeek-R1-Distill-Qwen 14Bcode model 100.0 20 / 20
1 Deepseek R1 8B Llama Distill 100.0 20 / 20
1 DeepSeek-R1-Distill-Qwen-14B 100.0 20 / 20
1 DeepSeek-R1-Distill-Qwen-7B 100.0 20 / 20
1 Devstral 24B Small 2505code model 100.0 20 / 20
1 Devstral Small 2 24B Instruct (2512)code model 100.0 20 / 20
1 EXAONE-3.5-7.8B-Instruct 100.0 20 / 20
1 Exaone3.5 32B Instruct 100.0 20 / 20
1 Gemma 4 12B IT 100.0 20 / 20
1 gemma-3-12b-it 100.0 20 / 20
1 gemma-3-27b-it 100.0 20 / 20
1 gemma-4-26B-A4B-it 100.0 20 / 20
1 gemma-4-31B-it 100.0 20 / 20
1 Gemma2 27B Instruct 100.0 20 / 20
1 Gemma4 E4B IT 100.0 20 / 20
1 GLM-4.5-Air 100.0 20 / 20
1 gpt-oss-120b 100.0 20 / 20
1 Granite3.2 8B Instruct 100.0 20 / 20
1 Granite4 Micro H 100.0 20 / 20
1 Granite4 Small H 100.0 20 / 20
1 Granite4.1 30B 100.0 20 / 20
1 Granite4.1 8B 100.0 20 / 20
1 Granite4.2 30B 100.0 20 / 20
1 Llama-3.1-8B-Instruct 100.0 20 / 20
1 Llama-3.3-70B-Instruct 100.0 20 / 20
1 Magistral Small (2506) 100.0 20 / 20
1 Ministral 3 14B Instruct (2512) 100.0 20 / 20
1 Ministral 3 8B Instruct 2512 100.0 20 / 20
1 Mistral Small 24B Instruct 2501 100.0 20 / 20
1 Mistral Small 3.2 24B Instruct (2506) 100.0 20 / 20
1 Mistral Small3.1 24B Instruct 2503 100.0 20 / 20
1 Muse Glimmer 30B 100.0 20 / 20
1 Nemotron Cascade 2 30B A3B 100.0 20 / 20
1 Ornith 9B 100.0 20 / 20
1 Qwen2-VL-7B-Instruct 100.0 20 / 20
1 Qwen2.5-32B-Instruct 100.0 20 / 20
1 Qwen2.5-7B-Instruct 100.0 20 / 20
1 Qwen2.5-Coder-14B-Instructcode model 100.0 20 / 20
1 Qwen2.5-Coder-32B-Instructcode model 100.0 20 / 20
1 Qwen2.5-Coder-7B-Instructcode model 100.0 20 / 20
1 Qwen2.5-VL-32B-Instruct 100.0 20 / 20
1 Qwen3 1.7B 100.0 20 / 20
1 Qwen3 VL 2B Instruct 100.0 20 / 20
1 Qwen3-14B 100.0 20 / 20
1 Qwen3-30B-A3B 100.0 20 / 20
1 Qwen3-30B-A3B-Instruct-2507 100.0 20 / 20
1 Qwen3-32B 100.0 20 / 20
1 Qwen3-4B 100.0 20 / 20
1 Qwen3-4B-Instruct-2507 100.0 20 / 20
1 Qwen3-4B-Thinking-2507 100.0 20 / 20
1 Qwen3-8B 100.0 20 / 20
1 Qwen3-Coder-30B-A3B-Instructcode model 100.0 20 / 20
1 Qwen3-Coder-Nextcode model 100.0 20 / 20
1 Qwen3-VL-30B-A3B-Instruct 100.0 20 / 20
1 Qwen3-VL-32B-Instruct 100.0 20 / 20
1 Qwen3-VL-4B-Instruct 100.0 20 / 20
1 Qwen3-VL-8B-Instruct 100.0 20 / 20
1 Qwen3.5-27B 100.0 20 / 20
1 Qwen3.5-9B 100.0 20 / 20
1 Qwen3.6-27B 100.0 20 / 20
1 Qwen3.6-35B-A3B 100.0 20 / 20
1 Qwen3.8-27B 100.0 20 / 20
1 QwQ 32B 100.0 20 / 20
1 Sage-A1.5-Q4_K_M 100.0 20 / 20
1 Translategemma 12B ITfine-tune of Gemma 3translation model 100.0 20 / 20
1 YI 34B Chat V1.5 100.0 20 / 20
70 gpt-oss-20b 97.5 20 / 20
70 Granite3.3 8B 97.5 20 / 20
70 Lfm2.5 Thinking 1.2B 97.5 20 / 20
70 Nemotron3 33B 97.5 20 / 20
74 Aya 8B 23 95.0 20 / 20
74 Aya Expanse 32B 95.0 20 / 20
74 Codegemma 7B Instruct V1.1code model 95.0 20 / 20
74 Codeqwen 7B Chat V1.5code model 95.0 20 / 20
74 DeepSeek-Coder-V2-Lite-Instructcode model 95.0 20 / 20
74 DeepSeek-R1-0528-Qwen3-8B 95.0 20 / 20
74 DeepSeek-R1-Distill-Qwen-32B 95.0 20 / 20
74 Exaone3.5 2.4B Instruct 95.0 20 / 20
74 Gemma 3n E4B IT 95.0 20 / 20
74 gemma-2-9b-it 95.0 20 / 20
74 Gemma4 E2B IT 95.0 20 / 20
74 Granite3 Dense 8B Instruct 95.0 20 / 20
74 Granite3.1 Dense 8B Instruct 95.0 20 / 20
74 Granite3.2 2B Instruct 95.0 20 / 20
74 Granite4.2 3B 95.0 20 / 20
74 Lfm3 VL 2B 95.0 20 / 20
74 Medgemma 27B ITfine-tune of Gemma 3medical model 95.0 20 / 20
74 Ministral 3 3B Instruct 2512 95.0 20 / 20
74 Mistral Small 22B Instruct 2409 95.0 20 / 20
74 Nanbeige_Nanbeige4.3-33B 95.0 20 / 20
74 Nemotron 3.5 Lightning 30B A3B 95.0 20 / 20
74 Olmo 3.1 32B Instruct 95.0 20 / 20
74 Olmo-3-7B-Instruct 95.0 20 / 20
74 Ornith-1.5-9B 95.0 20 / 20
74 Phi-4-mini-instruct 95.0 20 / 20
74 Phi3 3.8B Mini 128K Instruct 95.0 20 / 20
74 Qwen2.5-14B-Instruct 95.0 20 / 20
74 Qwen2.5-Coder-3B-Instructcode model 95.0 20 / 20
74 Qwen2.5-VL-7B-Instruct 95.0 20 / 20
74 Qwen3.5-35B-A3B 95.0 20 / 20
74 Solar 10.7B Instruct V1 95.0 20 / 20
74 YI Coder 9B Chatcode model 95.0 20 / 20
106 Codellama 34B Instructcode model 92.5 20 / 20
106 Cogito 3B V1 Preview Llamafine-tune of Llama 3.2 92.5 20 / 20
106 Llama 4 Scout 17B-16E Instruct 92.5 20 / 20
106 Llama-3.2-3B-Instruct 92.5 20 / 20
106 Medgemma1.5 4B ITfine-tune of Gemma 3medical model 92.5 20 / 20
106 Qwen2.5 1.5B Instruct 92.5 20 / 20
106 YI 6B Chat V1.5 92.5 20 / 20
113 Command R 35B 08 2024 90.0 20 / 20
113 Falcon3 10B Instruct 90.0 20 / 20
113 Granite3.3 2B 90.0 20 / 20
113 Granite4 3B 90.0 20 / 20
113 Marco O1 7Bfine-tune of Qwen 2 7B 90.0 20 / 20
113 Mistral 7B Instruct V0.3 90.0 20 / 20
113 Nemotron 3.5 Bolt E4B 90.0 20 / 20
113 NVIDIA Nemotron 3 Nano 30B-A3B 90.0 20 / 20
113 Olmo2 7B 1124 Instruct 90.0 20 / 20
113 Qwen2 Math 7B Instructmath model 90.0 20 / 20
113 Qwen2.5-3B-Instruct 90.0 20 / 20
113 Tulu3 8Bfine-tune of Llama 3.1 90.0 20 / 20
125 Deepseek V2 16B Lite Chat 87.5 20 / 20
125 Falcon3 7B Instruct 87.5 20 / 20
125 Gemma3n E2B IT 87.5 20 / 20
125 Phi-3.5-mini-instruct 87.5 20 / 20
125 Phi3 14B Medium 128K Instruct 87.5 20 / 20
125 YI 9B Chat V1.5 87.5 20 / 20
131 Codellama 7B Instructcode model 85.0 20 / 20
131 Codestral 22B V0.1code model 85.0 20 / 20
131 Command R7B Arabic 7B 02 2025 85.0 20 / 20
131 dolphin-2.9.1-yi-1.5-34b 85.0 20 / 20
131 gemma-3-4b-it 85.0 20 / 20
131 GLM-4.7-Flash 85.0 20 / 20
131 granite-4.1-3b 85.0 20 / 20
131 Granite3.1 Moe 3B Instruct 85.0 20 / 20
131 Hermes-3-Llama-3.1-8B 85.0 20 / 20
131 Internlm2 7B Chat V2.5 85.0 20 / 20
131 Lfm2 24B 85.0 20 / 20
131 medgemma-4b-itmedical model 85.0 20 / 20
131 Meta-Llama-3-8B-Instruct 85.0 20 / 20
131 Mistral Nemo 12B Instruct 2407 85.0 20 / 20
131 Mixtral 8x7B Instruct V0.1 85.0 20 / 20
131 Nemotron Mini 4B Instruct 85.0 20 / 20
131 Qwen3.5-4B 85.0 20 / 20
131 Spark-X2.6-4B 85.0 20 / 20
149 Codellama 13B Instructcode model 82.5 20 / 20
149 Granite3 Dense 2B Instruct 82.5 20 / 20
149 Qwen2.5 Coder 1.5B Instructcode model 82.5 20 / 20
149 Qwen3 0.6B 82.5 20 / 20
149 Qwen3 VL 2B Thinking 82.5 20 / 20
149 SmolLM3-3B 82.5 20 / 20
155 Apertus-8B-Instruct-2509 80.0 20 / 20
155 Codegeex4 9B Allcode model 80.0 20 / 20
155 Exaone Deep 7.8B 80.0 20 / 20
155 Falcon3 3B Instruct 80.0 20 / 20
155 Flint 40B A2B 80.0 20 / 20
155 Gemma 7B Instruct V1.1 80.0 20 / 20
155 Granite3.1 Dense 2B Instruct 80.0 20 / 20
155 Granite4 1B H 80.0 20 / 20
155 Granite4 350M H 80.0 20 / 20
155 Granite4 Tiny H 80.0 20 / 20
155 Granite4.2 8B 80.0 20 / 20
155 Mistral-7B-Instruct-v0.2 80.0 20 / 20
155 Olmo-3-7B-Think 80.0 20 / 20
155 Phi-3-mini-4k-instruct 80.0 20 / 20
155 Qwen2 1.5B Instruct 80.0 20 / 20
155 Qwen2.5-VL-3B-Instruct 80.0 20 / 20
155 Smallthinker 3B Previewfine-tune of Qwen 2.5 3B 80.0 20 / 20
172 Deepseek Llm 7B Chat 77.5 20 / 20
172 Gemma2 2B Instruct 77.5 20 / 20
172 Opencoder 1.5B Instructcode model 77.5 20 / 20
172 Qwen2 0.5B Instruct 77.5 20 / 20
176 Llama2 13B Chat 75.0 20 / 20
176 Olmo2 13B 1124 Instruct 75.0 20 / 20
176 Openthinker 7B V2fine-tune of Qwen 2.5 75.0 20 / 20
179 Exaone Deep 2.4B 72.5 20 / 20
179 Llama3.2 1B Instruct 72.5 20 / 20
181 Deepseek Coder 33B Instructcode model 70.0 20 / 20
181 Internlm2 1.8B Chat V2.5 70.0 20 / 20
181 Laguna XS 2.1 70.0 20 / 20
181 Mathstral 7B V0.1fine-tune of Mistral 7Bmath model 70.0 20 / 20
181 phi-4 70.0 20 / 20
181 Qwen3.5 2B 70.0 20 / 20
181 Smollm2 1.7B Instruct 70.0 20 / 20
188 Deepcoder 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bcode model 67.5 20 / 20
188 Stablelm2 12B Chat 67.5 20 / 20
190 Deepscaler 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bmath model 65.0 20 / 20
190 Dolphin Phi 2.7B V2.6fine-tune of Phi-2 65.0 20 / 20
190 Llama-2-7b-chat-hf 65.0 20 / 20
190 Qwen 14B Chat V1.5 65.0 20 / 20
190 Qwen2.5 Coder 0.5B Instructcode model 65.0 20 / 20
190 Stablelm Zephyr 3Bfine-tune of StableLM 3B 65.0 20 / 20
196 Deepseek R1 1.5B Qwen Distill 62.5 20 / 20
196 Granite3.1 Moe 1B Instruct 62.5 20 / 20
196 Qwen2.5 0.5B Instruct 62.5 20 / 20
199 Glm4 9B Chat 60.0 20 / 20
199 Granite3 Moe 1B Instruct 60.0 20 / 20
199 Hermes3 3B Llama3.2fine-tune of Llama 3.2 60.0 20 / 20
202 Deepseek Coder 6.7B Instructcode model 55.0 20 / 20
202 Phi4 Mini Reasoning 3.8B 55.0 20 / 20
204 Granite3 Moe 3B Instruct 45.0 20 / 20
205 Phi 2.7B Chat V2 42.5 20 / 20
205 Stablelm2 1.6B Chat 42.5 20 / 20
207 Deepseek Coder 1.3B Instructcode model 40.0 20 / 20
207 Falcon3 1B Instruct 40.0 20 / 20
207 Lfm3 1.7B 40.0 20 / 20
210 Gemma 2B Instruct V1.1 35.0 20 / 20
210 Gemma3 1B IT 35.0 20 / 20
210 Qwen 7B Chat V1.5 35.0 20 / 20
210 Sailor2 8B Chatfine-tune of Qwen 2.5 35.0 20 / 20
210 YI Coder 1.5B Chatcode model 35.0 20 / 20
215 Lfm2.5 8B A1B 30.0 20 / 20
216 Sailor2 1B Chatfine-tune of Qwen 2.5 27.5 20 / 20
217 Command R7B (12-2024) 25.0 20 / 20
217 Smollm2 135M Instruct 25.0 20 / 20
217 Smollm2 360M Instruct 25.0 20 / 20
220 Falcon2 11Bnot in the Overall ranking 20.0 20 / 20
220 Qwen 1.8B Chat V1.5 20.0 20 / 20
222 Qwen2 Math 1.5B Instructmath model 17.5 20 / 20
223 Granite Code 3B Instructcode model 15.0 20 / 20
223 Qwen 0.5B Chat V1.5 15.0 20 / 20
223 Qwen 4B Chat V1.5 15.0 20 / 20
223 Stable Code 3B Instructcode model 15.0 20 / 20
227 Codegemma 2B Code V1.1code model 10.0 20 / 20
227 Granite Code 8B Instructcode model 10.0 20 / 20

A dash means not scored, which is different from 0. Agreement compares the two judges: high means a mean gap under 1.5 points, medium under 3, low otherwise; n/a means fewer than 3 items were scored by both.

Not ranked in this category

The questions

In run r20261102-bv1-5246ab, 2 of the 20 questions in this category were answered correctly by every ranked model that has a score for them (33 ranked models in that run). Hard questions are the ones that half or fewer of the ranked models of 3B and over answered correctly. This category has 0.

The same ranking as JSON: tool_use.json.