Multilingual
Questions asked in languages other than English, with a numeric answer. Code also checks that the reply is written in the requested language.
This category counts toward Overall. How the battery works.
Ranking
Models are ranked by their mean score in this category, from 0 to 100. Only models with at least 80% of the category's items scored, and none pending, are ranked here. The rule is the same as for Overall.
| Rank | Model | Mean (0 to 100) | Items scored |
|---|---|---|---|
| 1 | Devstral Small 2 24B Instruct (2512)code model | 100.0 | 20 / 20 |
| 1 | Gemma 4 12B IT | 100.0 | 20 / 20 |
| 1 | gemma-2-9b-it | 100.0 | 20 / 20 |
| 1 | gemma-3-12b-it | 100.0 | 20 / 20 |
| 1 | gemma-3-27b-it | 100.0 | 20 / 20 |
| 1 | gemma-3-4b-it | 100.0 | 20 / 20 |
| 1 | gemma-4-26B-A4B-it | 100.0 | 20 / 20 |
| 1 | gemma-4-31B-it | 100.0 | 20 / 20 |
| 1 | Gemma4 E2B IT | 100.0 | 20 / 20 |
| 1 | Gemma4 E4B IT | 100.0 | 20 / 20 |
| 1 | GLM-4.7-Flash | 100.0 | 20 / 20 |
| 1 | gpt-oss-120b | 100.0 | 20 / 20 |
| 1 | gpt-oss-20b | 100.0 | 20 / 20 |
| 1 | Granite4 Small H | 100.0 | 20 / 20 |
| 1 | Granite4.1 30B | 100.0 | 20 / 20 |
| 1 | Granite4.2 30B | 100.0 | 20 / 20 |
| 1 | Llama 4 Scout 17B-16E Instruct | 100.0 | 20 / 20 |
| 1 | Llama-3.3-70B-Instruct | 100.0 | 20 / 20 |
| 1 | Mistral Small 3.2 24B Instruct (2506) | 100.0 | 20 / 20 |
| 1 | Mistral Small3.1 24B Instruct 2503 | 100.0 | 20 / 20 |
| 1 | Muse Glimmer 30B | 100.0 | 20 / 20 |
| 1 | Qwen3-30B-A3B | 100.0 | 20 / 20 |
| 1 | Qwen3-32B | 100.0 | 20 / 20 |
| 1 | Qwen3-4B-Instruct-2507 | 100.0 | 20 / 20 |
| 1 | Qwen3-8B | 100.0 | 20 / 20 |
| 1 | Qwen3-Coder-30B-A3B-Instructcode model | 100.0 | 20 / 20 |
| 1 | Qwen3-Coder-Nextcode model | 100.0 | 20 / 20 |
| 1 | Qwen3-VL-30B-A3B-Instruct | 100.0 | 20 / 20 |
| 1 | Qwen3-VL-8B-Instruct | 100.0 | 20 / 20 |
| 1 | Qwen3.5-27B | 100.0 | 20 / 20 |
| 1 | Qwen3.5-35B-A3B | 100.0 | 20 / 20 |
| 1 | Qwen3.6-27B | 100.0 | 20 / 20 |
| 1 | Qwen3.6-35B-A3B | 100.0 | 20 / 20 |
| 1 | Qwen3.8-27B | 100.0 | 20 / 20 |
| 1 | QwQ 32B | 100.0 | 20 / 20 |
| 1 | Sage-A1.5-Q4_K_M | 100.0 | 20 / 20 |
| 1 | Translategemma 12B ITfine-tune of Gemma 3translation model | 100.0 | 20 / 20 |
| 38 | Devstral 24B Small 2505code model | 98.3 | 20 / 20 |
| 38 | Laguna XS 2.1 | 98.3 | 20 / 20 |
| 38 | Ministral 3 14B Instruct (2512) | 98.3 | 20 / 20 |
| 38 | Nemotron 3.5 Bolt E4B | 98.3 | 20 / 20 |
| 38 | Qwen3-30B-A3B-Instruct-2507 | 98.3 | 20 / 20 |
| 38 | Qwen3-VL-32B-Instruct | 98.3 | 20 / 20 |
| 38 | Qwen3.5-4B | 98.3 | 20 / 20 |
| 38 | Qwen3.5-9B | 98.3 | 20 / 20 |
| 38 | Spark-X2.6-4B | 98.3 | 20 / 20 |
| 47 | Mistral Small 24B Instruct 2501 | 97.5 | 20 / 20 |
| 48 | Command R7B (12-2024) | 96.7 | 20 / 20 |
| 48 | DeepSeek-R1-Distill-Qwen-32B | 96.7 | 20 / 20 |
| 48 | Nemotron 3.5 Lightning 30B A3B | 96.7 | 20 / 20 |
| 48 | Qwen3-VL-4B-Instruct | 96.7 | 20 / 20 |
| 52 | DeepSeek-R1-0528-Qwen3-8B | 95.0 | 20 / 20 |
| 52 | Gemma2 27B Instruct | 95.0 | 20 / 20 |
| 52 | GLM-4.5-Air | 95.0 | 20 / 20 |
| 52 | Medgemma 27B ITfine-tune of Gemma 3medical model | 95.0 | 20 / 20 |
| 52 | Qwen2.5-Coder-32B-Instructcode model | 95.0 | 20 / 20 |
| 57 | Mistral Small 22B Instruct 2409 | 94.2 | 20 / 20 |
| 58 | Aya Expanse 32B | 93.3 | 20 / 20 |
| 58 | Gemma 3n E4B IT | 93.3 | 20 / 20 |
| 58 | Granite4.1 8B | 93.3 | 20 / 20 |
| 58 | phi-4 | 93.3 | 20 / 20 |
| 58 | Qwen2.5-32B-Instruct | 93.3 | 20 / 20 |
| 63 | Ministral 3 3B Instruct 2512 | 92.5 | 20 / 20 |
| 64 | Cogito 14B V1 Preview Qwenfine-tune of Qwen 2.5 | 91.7 | 20 / 20 |
| 64 | Flint 40B A2B | 91.7 | 20 / 20 |
| 64 | Lfm2 24B | 91.7 | 20 / 20 |
| 64 | Lfm3 VL 2B | 91.7 | 20 / 20 |
| 64 | Qwen2.5-14B-Instruct | 91.7 | 20 / 20 |
| 64 | Qwen2.5-7B-Instruct | 91.7 | 20 / 20 |
| 64 | Qwen3-4B-Thinking-2507 | 91.7 | 20 / 20 |
| 71 | Command R7B Arabic 7B 02 2025 | 90.8 | 20 / 20 |
| 72 | Nanbeige_Nanbeige4.3-33B | 90.0 | 20 / 20 |
| 72 | Olmo 3.1 32B Instruct | 90.0 | 20 / 20 |
| 72 | Olmo-3-7B-Think | 90.0 | 20 / 20 |
| 72 | Qwen2.5-Coder-7B-Instructcode model | 90.0 | 20 / 20 |
| 72 | Qwen2.5-VL-32B-Instruct | 90.0 | 20 / 20 |
| 72 | Qwen3-14B | 90.0 | 20 / 20 |
| 78 | DeepSeek-R1-Distill-Qwen-14B | 89.2 | 20 / 20 |
| 79 | EXAONE-3.5-7.8B-Instruct | 88.3 | 20 / 20 |
| 79 | Granite4.2 8B | 88.3 | 20 / 20 |
| 79 | Ministral 3 8B Instruct 2512 | 88.3 | 20 / 20 |
| 79 | Ornith-1.5-9B | 88.3 | 20 / 20 |
| 79 | Qwen3-4B | 88.3 | 20 / 20 |
| 84 | Exaone3.5 32B Instruct | 87.5 | 20 / 20 |
| 84 | Openthinker 7B V2fine-tune of Qwen 2.5 | 87.5 | 20 / 20 |
| 84 | Qwen3 1.7B | 87.5 | 20 / 20 |
| 87 | DeepSeek-Coder-V2-Lite-Instructcode model | 86.7 | 20 / 20 |
| 87 | granite-4.1-3b | 86.7 | 20 / 20 |
| 87 | Olmo-3-7B-Instruct | 86.7 | 20 / 20 |
| 90 | Qwen2.5-Coder-14B-Instructcode model | 85.8 | 20 / 20 |
| 91 | Glm4 9B Chat | 85.0 | 20 / 20 |
| 91 | Mistral Nemo 12B Instruct 2407 | 85.0 | 20 / 20 |
| 91 | Ornith 9B | 85.0 | 20 / 20 |
| 94 | Command R 35B 08 2024 | 84.2 | 20 / 20 |
| 95 | Codestral 22B V0.1code model | 83.3 | 20 / 20 |
| 95 | Falcon3 10B Instruct | 83.3 | 20 / 20 |
| 95 | Granite4 Micro H | 83.3 | 20 / 20 |
| 95 | Lfm3 1.7B | 83.3 | 20 / 20 |
| 95 | Qwen2.5-VL-7B-Instruct | 83.3 | 20 / 20 |
| 95 | YI 34B Chat V1.5 | 83.3 | 20 / 20 |
| 101 | Cogito 8B V1 Preview Llamafine-tune of Llama 3.1 | 81.7 | 20 / 20 |
| 101 | Granite3.2 8B Instruct | 81.7 | 20 / 20 |
| 101 | Lfm2.5 8B A1B | 81.7 | 20 / 20 |
| 101 | Llama-3.1-8B-Instruct | 81.7 | 20 / 20 |
| 101 | Llama-3.2-3B-Instruct | 81.7 | 20 / 20 |
| 101 | Meta-Llama-3-8B-Instruct | 81.7 | 20 / 20 |
| 101 | Phi-4-mini-instruct | 81.7 | 20 / 20 |
| 101 | Sailor2 8B Chatfine-tune of Qwen 2.5 | 81.7 | 20 / 20 |
| 109 | DeepSeek-R1-Distill-Qwen-7B | 80.8 | 20 / 20 |
| 110 | Gemma2 2B Instruct | 80.0 | 20 / 20 |
| 110 | Gemma3n E2B IT | 80.0 | 20 / 20 |
| 110 | Granite4 Tiny H | 80.0 | 20 / 20 |
| 110 | Mathstral 7B V0.1fine-tune of Mistral 7Bmath model | 80.0 | 20 / 20 |
| 110 | Olmo2 13B 1124 Instruct | 80.0 | 20 / 20 |
| 115 | Granite4 3B | 78.3 | 20 / 20 |
| 115 | Nemotron Cascade 2 30B A3B | 78.3 | 20 / 20 |
| 115 | Qwen2.5-Coder-3B-Instructcode model | 78.3 | 20 / 20 |
| 118 | medgemma-4b-itmedical model | 77.5 | 20 / 20 |
| 118 | Qwen2 Math 7B Instructmath model | 77.5 | 20 / 20 |
| 120 | Cogito 3B V1 Preview Llamafine-tune of Llama 3.2 | 76.7 | 20 / 20 |
| 120 | Smallthinker 3B Previewfine-tune of Qwen 2.5 3B | 76.7 | 20 / 20 |
| 122 | Qwen3 VL 2B Instruct | 75.8 | 20 / 20 |
| 123 | Apertus-8B-Instruct-2509 | 75.0 | 20 / 20 |
| 123 | Deepcoder 14B Previewfine-tune of DeepSeek-R1-Distill-Qwen 14Bcode model | 75.0 | 20 / 20 |
| 123 | Granite4.2 3B | 75.0 | 20 / 20 |
| 123 | NVIDIA Nemotron 3 Nano 30B-A3B | 75.0 | 20 / 20 |
| 123 | Phi4 Mini Reasoning 3.8B | 75.0 | 20 / 20 |
| 123 | Qwen2.5-3B-Instruct | 75.0 | 20 / 20 |
| 123 | SmolLM3-3B | 75.0 | 20 / 20 |
| 123 | YI 9B Chat V1.5 | 75.0 | 20 / 20 |
| 131 | Medgemma1.5 4B ITfine-tune of Gemma 3medical model | 74.2 | 20 / 20 |
| 132 | Aya Expanse 8B | 72.5 | 20 / 20 |
| 132 | Granite3.3 8B | 72.5 | 20 / 20 |
| 134 | Deepseek R1 8B Llama Distill | 71.7 | 20 / 20 |
| 134 | Hermes-3-Llama-3.1-8B | 71.7 | 20 / 20 |
| 134 | Lfm2.5 Thinking 1.2B | 71.7 | 20 / 20 |
| 137 | Codegemma 7B Instruct V1.1code model | 70.0 | 20 / 20 |
| 137 | Magistral Small (2506) | 70.0 | 20 / 20 |
| 139 | Marco O1 7Bfine-tune of Qwen 2 7B | 69.2 | 20 / 20 |
| 139 | Phi3 14B Medium 128K Instruct | 69.2 | 20 / 20 |
| 141 | Falcon3 7B Instruct | 67.5 | 20 / 20 |
| 141 | Tulu3 8Bfine-tune of Llama 3.1 | 67.5 | 20 / 20 |
| 143 | dolphin-2.9.1-yi-1.5-34b | 66.7 | 20 / 20 |
| 143 | Qwen2-VL-7B-Instruct | 66.7 | 20 / 20 |
| 145 | Deepcoder 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bcode model | 65.8 | 20 / 20 |
| 146 | Codegeex4 9B Allcode model | 63.3 | 20 / 20 |
| 146 | Qwen3 VL 2B Thinking | 63.3 | 20 / 20 |
| 146 | Qwen3.5 2B | 63.3 | 20 / 20 |
| 149 | Exaone3.5 2.4B Instruct | 62.5 | 20 / 20 |
| 150 | Granite3.1 Dense 8B Instruct | 61.7 | 20 / 20 |
| 151 | Olmo2 7B 1124 Instruct | 60.0 | 20 / 20 |
| 151 | Qwen2.5-VL-3B-Instruct | 60.0 | 20 / 20 |
| 153 | Solar 10.7B Instruct V1 | 58.3 | 20 / 20 |
| 154 | Granite3.3 2B | 57.5 | 20 / 20 |
| 155 | Deepscaler 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bmath model | 56.7 | 20 / 20 |
| 155 | Gemma 7B Instruct V1.1 | 56.7 | 20 / 20 |
| 157 | Llama3.2 1B Instruct | 55.0 | 20 / 20 |
| 157 | Nemotron3 33B | 55.0 | 20 / 20 |
| 159 | YI Coder 9B Chatcode model | 54.2 | 20 / 20 |
| 160 | Gemma3 1B IT | 52.5 | 20 / 20 |
| 160 | Granite3 Dense 8B Instruct | 52.5 | 20 / 20 |
| 160 | Phi-3-mini-4k-instruct | 52.5 | 20 / 20 |
| 160 | Qwen 14B Chat V1.5 | 52.5 | 20 / 20 |
| 164 | YI 6B Chat V1.5 | 51.7 | 20 / 20 |
| 165 | Deepseek V2 16B Lite Chat | 50.8 | 20 / 20 |
| 165 | Exaone Deep 7.8B | 50.8 | 20 / 20 |
| 165 | Falcon3 3B Instruct | 50.8 | 20 / 20 |
| 168 | Codellama 34B Instructcode model | 50.0 | 20 / 20 |
| 169 | Granite3 Dense 2B Instruct | 48.3 | 20 / 20 |
| 169 | Qwen2 1.5B Instruct | 48.3 | 20 / 20 |
| 169 | Qwen2.5 1.5B Instruct | 48.3 | 20 / 20 |
| 169 | Qwen3 0.6B | 48.3 | 20 / 20 |
| 173 | Deepseek R1 1.5B Qwen Distill | 47.5 | 20 / 20 |
| 173 | Nemotron Mini 4B Instruct | 47.5 | 20 / 20 |
| 175 | Phi-3.5-mini-instruct | 46.7 | 20 / 20 |
| 176 | Deepseek Llm 7B Chat | 45.8 | 20 / 20 |
| 177 | Granite3.1 Dense 2B Instruct | 45.0 | 20 / 20 |
| 177 | Internlm2 7B Chat V2.5 | 45.0 | 20 / 20 |
| 177 | Qwen 7B Chat V1.5 | 45.0 | 20 / 20 |
| 180 | Granite3.2 2B Instruct | 43.3 | 20 / 20 |
| 180 | Granite4 1B H | 43.3 | 20 / 20 |
| 180 | Granite4 350M H | 43.3 | 20 / 20 |
| 180 | Hermes3 3B Llama3.2fine-tune of Llama 3.2 | 43.3 | 20 / 20 |
| 184 | Granite Code 3B Instructcode model | 42.5 | 20 / 20 |
| 184 | Mistral-7B-Instruct-v0.2 | 42.5 | 20 / 20 |
| 186 | Granite Code 8B Instructcode model | 41.7 | 20 / 20 |
| 186 | Granite3.1 Moe 3B Instruct | 41.7 | 20 / 20 |
| 186 | Mistral 7B Instruct V0.3 | 41.7 | 20 / 20 |
| 186 | Mixtral 8x7B Instruct V0.1 | 41.7 | 20 / 20 |
| 190 | Aya 8B 23 | 39.2 | 20 / 20 |
| 191 | Exaone Deep 2.4B | 38.3 | 20 / 20 |
| 192 | Gemma 2B Instruct V1.1 | 37.5 | 20 / 20 |
| 192 | Qwen 4B Chat V1.5 | 37.5 | 20 / 20 |
| 192 | Qwen2 Math 1.5B Instructmath model | 37.5 | 20 / 20 |
| 195 | Granite3 Moe 3B Instruct | 36.7 | 20 / 20 |
| 196 | Codeqwen 7B Chat V1.5code model | 35.0 | 20 / 20 |
| 196 | Smollm2 1.7B Instruct | 35.0 | 20 / 20 |
| 198 | Falcon3 1B Instruct | 34.2 | 20 / 20 |
| 198 | Stablelm Zephyr 3Bfine-tune of StableLM 3B | 34.2 | 20 / 20 |
| 200 | Codellama 13B Instructcode model | 33.3 | 20 / 20 |
| 200 | Llama-2-7b-chat-hf | 33.3 | 20 / 20 |
| 202 | Llama2 13B Chat | 32.5 | 20 / 20 |
| 203 | Phi3 3.8B Mini 128K Instruct | 31.7 | 20 / 20 |
| 203 | Qwen2.5 Coder 0.5B Instructcode model | 31.7 | 20 / 20 |
| 205 | Qwen2.5 Coder 1.5B Instructcode model | 30.8 | 20 / 20 |
| 206 | Qwen 0.5B Chat V1.5 | 30.0 | 20 / 20 |
| 207 | Internlm2 1.8B Chat V2.5 | 29.2 | 20 / 20 |
| 208 | Qwen 1.8B Chat V1.5 | 27.5 | 20 / 20 |
| 209 | Stable Code 3B Instructcode model | 26.7 | 20 / 20 |
| 210 | Deepseek Coder 33B Instructcode model | 25.8 | 20 / 20 |
| 211 | Sailor2 1B Chatfine-tune of Qwen 2.5 | 25.0 | 20 / 20 |
| 212 | Granite3.1 Moe 1B Instruct | 24.2 | 20 / 20 |
| 212 | Opencoder 1.5B Instructcode model | 24.2 | 20 / 20 |
| 214 | Deepseek Coder 1.3B Instructcode model | 23.3 | 20 / 20 |
| 214 | Qwen2 0.5B Instruct | 23.3 | 20 / 20 |
| 214 | Qwen2.5 0.5B Instruct | 23.3 | 20 / 20 |
| 217 | Falcon2 11Bnot in the Overall ranking | 22.5 | 20 / 20 |
| 218 | Codellama 7B Instructcode model | 21.7 | 20 / 20 |
| 218 | Deepseek Coder 6.7B Instructcode model | 21.7 | 20 / 20 |
| 218 | Smollm2 360M Instruct | 21.7 | 20 / 20 |
| 221 | Granite3 Moe 1B Instruct | 20.8 | 20 / 20 |
| 222 | Stablelm2 12B Chat | 20.0 | 20 / 20 |
| 223 | Phi 2.7B Chat V2 | 18.3 | 20 / 20 |
| 224 | Codegemma 2B Code V1.1code model | 16.7 | 20 / 20 |
| 224 | Stablelm2 1.6B Chat | 16.7 | 20 / 20 |
| 226 | Dolphin Phi 2.7B V2.6fine-tune of Phi-2 | 15.0 | 20 / 20 |
| 226 | Smollm2 135M Instruct | 15.0 | 20 / 20 |
| 228 | YI Coder 1.5B Chatcode model | 10.8 | 20 / 20 |
A dash means not scored, which is different from 0. Agreement compares the two judges: high means a mean gap under 1.5 points, medium under 3, low otherwise; n/a means fewer than 3 items were scored by both.
Not ranked in this category
- Apertus-8B-Instruct-2509: withheld from the ranking (see its page)
- Falcon3 10B Instruct: withheld from the ranking (see its page)
- Falcon3 1B Instruct: withheld from the ranking (see its page)
- Falcon3 3B Instruct: withheld from the ranking (see its page)
- Falcon3 7B Instruct: withheld from the ranking (see its page)
- Internlm2 1.8B Chat V2.5: withheld from the ranking (see its page)
- Phi-4-reasoning 14B: withheld from the ranking (see its page)
- Phi-4-reasoning-plus 14B: withheld from the ranking (see its page)
- Qwen2 Math 1.5B Instruct: withheld from the ranking (see its page)
- Qwen2 Math 7B Instruct: withheld from the ranking (see its page)
The questions
In run r20261102-bv1-5246ab, 0 of the 20 questions in this category were answered correctly by every ranked model that has a score for them (33 ranked models in that run). Hard questions are the ones that half or fewer of the ranked models of 3B and over answered correctly. This category has 0.
The same ranking as JSON: multilingual.json.