All categories

Multilingual

Questions asked in languages other than English, with a numeric answer. Code also checks that the reply is written in the requested language.

This category counts toward Overall. How the battery works.

Ranking

Models are ranked by their mean score in this category, from 0 to 100. Only models with at least 80% of the category's items scored, and none pending, are ranked here. The rule is the same as for Overall.

Multilingual: 228 ranked models. Click a column heading to sort.
Rank Model Mean (0 to 100) Items scored
1 Devstral Small 2 24B Instruct (2512)code model 100.0 20 / 20
1 Gemma 4 12B IT 100.0 20 / 20
1 gemma-2-9b-it 100.0 20 / 20
1 gemma-3-12b-it 100.0 20 / 20
1 gemma-3-27b-it 100.0 20 / 20
1 gemma-3-4b-it 100.0 20 / 20
1 gemma-4-26B-A4B-it 100.0 20 / 20
1 gemma-4-31B-it 100.0 20 / 20
1 Gemma4 E2B IT 100.0 20 / 20
1 Gemma4 E4B IT 100.0 20 / 20
1 GLM-4.7-Flash 100.0 20 / 20
1 gpt-oss-120b 100.0 20 / 20
1 gpt-oss-20b 100.0 20 / 20
1 Granite4 Small H 100.0 20 / 20
1 Granite4.1 30B 100.0 20 / 20
1 Granite4.2 30B 100.0 20 / 20
1 Llama 4 Scout 17B-16E Instruct 100.0 20 / 20
1 Llama-3.3-70B-Instruct 100.0 20 / 20
1 Mistral Small 3.2 24B Instruct (2506) 100.0 20 / 20
1 Mistral Small3.1 24B Instruct 2503 100.0 20 / 20
1 Muse Glimmer 30B 100.0 20 / 20
1 Qwen3-30B-A3B 100.0 20 / 20
1 Qwen3-32B 100.0 20 / 20
1 Qwen3-4B-Instruct-2507 100.0 20 / 20
1 Qwen3-8B 100.0 20 / 20
1 Qwen3-Coder-30B-A3B-Instructcode model 100.0 20 / 20
1 Qwen3-Coder-Nextcode model 100.0 20 / 20
1 Qwen3-VL-30B-A3B-Instruct 100.0 20 / 20
1 Qwen3-VL-8B-Instruct 100.0 20 / 20
1 Qwen3.5-27B 100.0 20 / 20
1 Qwen3.5-35B-A3B 100.0 20 / 20
1 Qwen3.6-27B 100.0 20 / 20
1 Qwen3.6-35B-A3B 100.0 20 / 20
1 Qwen3.8-27B 100.0 20 / 20
1 QwQ 32B 100.0 20 / 20
1 Sage-A1.5-Q4_K_M 100.0 20 / 20
1 Translategemma 12B ITfine-tune of Gemma 3translation model 100.0 20 / 20
38 Devstral 24B Small 2505code model 98.3 20 / 20
38 Laguna XS 2.1 98.3 20 / 20
38 Ministral 3 14B Instruct (2512) 98.3 20 / 20
38 Nemotron 3.5 Bolt E4B 98.3 20 / 20
38 Qwen3-30B-A3B-Instruct-2507 98.3 20 / 20
38 Qwen3-VL-32B-Instruct 98.3 20 / 20
38 Qwen3.5-4B 98.3 20 / 20
38 Qwen3.5-9B 98.3 20 / 20
38 Spark-X2.6-4B 98.3 20 / 20
47 Mistral Small 24B Instruct 2501 97.5 20 / 20
48 Command R7B (12-2024) 96.7 20 / 20
48 DeepSeek-R1-Distill-Qwen-32B 96.7 20 / 20
48 Nemotron 3.5 Lightning 30B A3B 96.7 20 / 20
48 Qwen3-VL-4B-Instruct 96.7 20 / 20
52 DeepSeek-R1-0528-Qwen3-8B 95.0 20 / 20
52 Gemma2 27B Instruct 95.0 20 / 20
52 GLM-4.5-Air 95.0 20 / 20
52 Medgemma 27B ITfine-tune of Gemma 3medical model 95.0 20 / 20
52 Qwen2.5-Coder-32B-Instructcode model 95.0 20 / 20
57 Mistral Small 22B Instruct 2409 94.2 20 / 20
58 Aya Expanse 32B 93.3 20 / 20
58 Gemma 3n E4B IT 93.3 20 / 20
58 Granite4.1 8B 93.3 20 / 20
58 phi-4 93.3 20 / 20
58 Qwen2.5-32B-Instruct 93.3 20 / 20
63 Ministral 3 3B Instruct 2512 92.5 20 / 20
64 Cogito 14B V1 Preview Qwenfine-tune of Qwen 2.5 91.7 20 / 20
64 Flint 40B A2B 91.7 20 / 20
64 Lfm2 24B 91.7 20 / 20
64 Lfm3 VL 2B 91.7 20 / 20
64 Qwen2.5-14B-Instruct 91.7 20 / 20
64 Qwen2.5-7B-Instruct 91.7 20 / 20
64 Qwen3-4B-Thinking-2507 91.7 20 / 20
71 Command R7B Arabic 7B 02 2025 90.8 20 / 20
72 Nanbeige_Nanbeige4.3-33B 90.0 20 / 20
72 Olmo 3.1 32B Instruct 90.0 20 / 20
72 Olmo-3-7B-Think 90.0 20 / 20
72 Qwen2.5-Coder-7B-Instructcode model 90.0 20 / 20
72 Qwen2.5-VL-32B-Instruct 90.0 20 / 20
72 Qwen3-14B 90.0 20 / 20
78 DeepSeek-R1-Distill-Qwen-14B 89.2 20 / 20
79 EXAONE-3.5-7.8B-Instruct 88.3 20 / 20
79 Granite4.2 8B 88.3 20 / 20
79 Ministral 3 8B Instruct 2512 88.3 20 / 20
79 Ornith-1.5-9B 88.3 20 / 20
79 Qwen3-4B 88.3 20 / 20
84 Exaone3.5 32B Instruct 87.5 20 / 20
84 Openthinker 7B V2fine-tune of Qwen 2.5 87.5 20 / 20
84 Qwen3 1.7B 87.5 20 / 20
87 DeepSeek-Coder-V2-Lite-Instructcode model 86.7 20 / 20
87 granite-4.1-3b 86.7 20 / 20
87 Olmo-3-7B-Instruct 86.7 20 / 20
90 Qwen2.5-Coder-14B-Instructcode model 85.8 20 / 20
91 Glm4 9B Chat 85.0 20 / 20
91 Mistral Nemo 12B Instruct 2407 85.0 20 / 20
91 Ornith 9B 85.0 20 / 20
94 Command R 35B 08 2024 84.2 20 / 20
95 Codestral 22B V0.1code model 83.3 20 / 20
95 Falcon3 10B Instruct 83.3 20 / 20
95 Granite4 Micro H 83.3 20 / 20
95 Lfm3 1.7B 83.3 20 / 20
95 Qwen2.5-VL-7B-Instruct 83.3 20 / 20
95 YI 34B Chat V1.5 83.3 20 / 20
101 Cogito 8B V1 Preview Llamafine-tune of Llama 3.1 81.7 20 / 20
101 Granite3.2 8B Instruct 81.7 20 / 20
101 Lfm2.5 8B A1B 81.7 20 / 20
101 Llama-3.1-8B-Instruct 81.7 20 / 20
101 Llama-3.2-3B-Instruct 81.7 20 / 20
101 Meta-Llama-3-8B-Instruct 81.7 20 / 20
101 Phi-4-mini-instruct 81.7 20 / 20
101 Sailor2 8B Chatfine-tune of Qwen 2.5 81.7 20 / 20
109 DeepSeek-R1-Distill-Qwen-7B 80.8 20 / 20
110 Gemma2 2B Instruct 80.0 20 / 20
110 Gemma3n E2B IT 80.0 20 / 20
110 Granite4 Tiny H 80.0 20 / 20
110 Mathstral 7B V0.1fine-tune of Mistral 7Bmath model 80.0 20 / 20
110 Olmo2 13B 1124 Instruct 80.0 20 / 20
115 Granite4 3B 78.3 20 / 20
115 Nemotron Cascade 2 30B A3B 78.3 20 / 20
115 Qwen2.5-Coder-3B-Instructcode model 78.3 20 / 20
118 medgemma-4b-itmedical model 77.5 20 / 20
118 Qwen2 Math 7B Instructmath model 77.5 20 / 20
120 Cogito 3B V1 Preview Llamafine-tune of Llama 3.2 76.7 20 / 20
120 Smallthinker 3B Previewfine-tune of Qwen 2.5 3B 76.7 20 / 20
122 Qwen3 VL 2B Instruct 75.8 20 / 20
123 Apertus-8B-Instruct-2509 75.0 20 / 20
123 Deepcoder 14B Previewfine-tune of DeepSeek-R1-Distill-Qwen 14Bcode model 75.0 20 / 20
123 Granite4.2 3B 75.0 20 / 20
123 NVIDIA Nemotron 3 Nano 30B-A3B 75.0 20 / 20
123 Phi4 Mini Reasoning 3.8B 75.0 20 / 20
123 Qwen2.5-3B-Instruct 75.0 20 / 20
123 SmolLM3-3B 75.0 20 / 20
123 YI 9B Chat V1.5 75.0 20 / 20
131 Medgemma1.5 4B ITfine-tune of Gemma 3medical model 74.2 20 / 20
132 Aya Expanse 8B 72.5 20 / 20
132 Granite3.3 8B 72.5 20 / 20
134 Deepseek R1 8B Llama Distill 71.7 20 / 20
134 Hermes-3-Llama-3.1-8B 71.7 20 / 20
134 Lfm2.5 Thinking 1.2B 71.7 20 / 20
137 Codegemma 7B Instruct V1.1code model 70.0 20 / 20
137 Magistral Small (2506) 70.0 20 / 20
139 Marco O1 7Bfine-tune of Qwen 2 7B 69.2 20 / 20
139 Phi3 14B Medium 128K Instruct 69.2 20 / 20
141 Falcon3 7B Instruct 67.5 20 / 20
141 Tulu3 8Bfine-tune of Llama 3.1 67.5 20 / 20
143 dolphin-2.9.1-yi-1.5-34b 66.7 20 / 20
143 Qwen2-VL-7B-Instruct 66.7 20 / 20
145 Deepcoder 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bcode model 65.8 20 / 20
146 Codegeex4 9B Allcode model 63.3 20 / 20
146 Qwen3 VL 2B Thinking 63.3 20 / 20
146 Qwen3.5 2B 63.3 20 / 20
149 Exaone3.5 2.4B Instruct 62.5 20 / 20
150 Granite3.1 Dense 8B Instruct 61.7 20 / 20
151 Olmo2 7B 1124 Instruct 60.0 20 / 20
151 Qwen2.5-VL-3B-Instruct 60.0 20 / 20
153 Solar 10.7B Instruct V1 58.3 20 / 20
154 Granite3.3 2B 57.5 20 / 20
155 Deepscaler 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bmath model 56.7 20 / 20
155 Gemma 7B Instruct V1.1 56.7 20 / 20
157 Llama3.2 1B Instruct 55.0 20 / 20
157 Nemotron3 33B 55.0 20 / 20
159 YI Coder 9B Chatcode model 54.2 20 / 20
160 Gemma3 1B IT 52.5 20 / 20
160 Granite3 Dense 8B Instruct 52.5 20 / 20
160 Phi-3-mini-4k-instruct 52.5 20 / 20
160 Qwen 14B Chat V1.5 52.5 20 / 20
164 YI 6B Chat V1.5 51.7 20 / 20
165 Deepseek V2 16B Lite Chat 50.8 20 / 20
165 Exaone Deep 7.8B 50.8 20 / 20
165 Falcon3 3B Instruct 50.8 20 / 20
168 Codellama 34B Instructcode model 50.0 20 / 20
169 Granite3 Dense 2B Instruct 48.3 20 / 20
169 Qwen2 1.5B Instruct 48.3 20 / 20
169 Qwen2.5 1.5B Instruct 48.3 20 / 20
169 Qwen3 0.6B 48.3 20 / 20
173 Deepseek R1 1.5B Qwen Distill 47.5 20 / 20
173 Nemotron Mini 4B Instruct 47.5 20 / 20
175 Phi-3.5-mini-instruct 46.7 20 / 20
176 Deepseek Llm 7B Chat 45.8 20 / 20
177 Granite3.1 Dense 2B Instruct 45.0 20 / 20
177 Internlm2 7B Chat V2.5 45.0 20 / 20
177 Qwen 7B Chat V1.5 45.0 20 / 20
180 Granite3.2 2B Instruct 43.3 20 / 20
180 Granite4 1B H 43.3 20 / 20
180 Granite4 350M H 43.3 20 / 20
180 Hermes3 3B Llama3.2fine-tune of Llama 3.2 43.3 20 / 20
184 Granite Code 3B Instructcode model 42.5 20 / 20
184 Mistral-7B-Instruct-v0.2 42.5 20 / 20
186 Granite Code 8B Instructcode model 41.7 20 / 20
186 Granite3.1 Moe 3B Instruct 41.7 20 / 20
186 Mistral 7B Instruct V0.3 41.7 20 / 20
186 Mixtral 8x7B Instruct V0.1 41.7 20 / 20
190 Aya 8B 23 39.2 20 / 20
191 Exaone Deep 2.4B 38.3 20 / 20
192 Gemma 2B Instruct V1.1 37.5 20 / 20
192 Qwen 4B Chat V1.5 37.5 20 / 20
192 Qwen2 Math 1.5B Instructmath model 37.5 20 / 20
195 Granite3 Moe 3B Instruct 36.7 20 / 20
196 Codeqwen 7B Chat V1.5code model 35.0 20 / 20
196 Smollm2 1.7B Instruct 35.0 20 / 20
198 Falcon3 1B Instruct 34.2 20 / 20
198 Stablelm Zephyr 3Bfine-tune of StableLM 3B 34.2 20 / 20
200 Codellama 13B Instructcode model 33.3 20 / 20
200 Llama-2-7b-chat-hf 33.3 20 / 20
202 Llama2 13B Chat 32.5 20 / 20
203 Phi3 3.8B Mini 128K Instruct 31.7 20 / 20
203 Qwen2.5 Coder 0.5B Instructcode model 31.7 20 / 20
205 Qwen2.5 Coder 1.5B Instructcode model 30.8 20 / 20
206 Qwen 0.5B Chat V1.5 30.0 20 / 20
207 Internlm2 1.8B Chat V2.5 29.2 20 / 20
208 Qwen 1.8B Chat V1.5 27.5 20 / 20
209 Stable Code 3B Instructcode model 26.7 20 / 20
210 Deepseek Coder 33B Instructcode model 25.8 20 / 20
211 Sailor2 1B Chatfine-tune of Qwen 2.5 25.0 20 / 20
212 Granite3.1 Moe 1B Instruct 24.2 20 / 20
212 Opencoder 1.5B Instructcode model 24.2 20 / 20
214 Deepseek Coder 1.3B Instructcode model 23.3 20 / 20
214 Qwen2 0.5B Instruct 23.3 20 / 20
214 Qwen2.5 0.5B Instruct 23.3 20 / 20
217 Falcon2 11Bnot in the Overall ranking 22.5 20 / 20
218 Codellama 7B Instructcode model 21.7 20 / 20
218 Deepseek Coder 6.7B Instructcode model 21.7 20 / 20
218 Smollm2 360M Instruct 21.7 20 / 20
221 Granite3 Moe 1B Instruct 20.8 20 / 20
222 Stablelm2 12B Chat 20.0 20 / 20
223 Phi 2.7B Chat V2 18.3 20 / 20
224 Codegemma 2B Code V1.1code model 16.7 20 / 20
224 Stablelm2 1.6B Chat 16.7 20 / 20
226 Dolphin Phi 2.7B V2.6fine-tune of Phi-2 15.0 20 / 20
226 Smollm2 135M Instruct 15.0 20 / 20
228 YI Coder 1.5B Chatcode model 10.8 20 / 20

A dash means not scored, which is different from 0. Agreement compares the two judges: high means a mean gap under 1.5 points, medium under 3, low otherwise; n/a means fewer than 3 items were scored by both.

Not ranked in this category

The questions

In run r20261102-bv1-5246ab, 0 of the 20 questions in this category were answered correctly by every ranked model that has a score for them (33 ranked models in that run). Hard questions are the ones that half or fewer of the ranked models of 3B and over answered correctly. This category has 0.

The same ranking as JSON: multilingual.json.