All categories

Knowledge & honesty

Factual questions with one exact answer, some of them multiple choice. The final answer is matched exactly.

This category counts toward Overall. How the battery works.

Ranking

Models are ranked by their mean score in this category, from 0 to 100. Only models with at least 80% of the category's items scored, and none pending, are ranked here. The rule is the same as for Overall.

Knowledge & honesty: 228 ranked models. Click a column heading to sort.
Rank Model Mean (0 to 100) Items scored
1 Aya Expanse 32B 100.0 30 / 30
1 Deepcoder 14B Previewfine-tune of DeepSeek-R1-Distill-Qwen 14Bcode model 100.0 30 / 30
1 DeepSeek-R1-Distill-Qwen-14B 100.0 30 / 30
1 DeepSeek-R1-Distill-Qwen-32B 100.0 30 / 30
1 Devstral 24B Small 2505code model 100.0 30 / 30
1 Devstral Small 2 24B Instruct (2512)code model 100.0 30 / 30
1 EXAONE-3.5-7.8B-Instruct 100.0 30 / 30
1 Gemma 4 12B IT 100.0 30 / 30
1 gemma-4-26B-A4B-it 100.0 30 / 30
1 gemma-4-31B-it 100.0 30 / 30
1 Gemma4 E4B IT 100.0 30 / 30
1 GLM-4.5-Air 100.0 30 / 30
1 GLM-4.7-Flash 100.0 30 / 30
1 gpt-oss-120b 100.0 30 / 30
1 gpt-oss-20b 100.0 30 / 30
1 Granite3.1 Dense 8B Instruct 100.0 30 / 30
1 Granite4 Small H 100.0 30 / 30
1 Granite4.1 30B 100.0 30 / 30
1 Granite4.2 30B 100.0 30 / 30
1 Granite4.2 8B 100.0 30 / 30
1 Lfm2 24B 100.0 30 / 30
1 Llama 4 Scout 17B-16E Instruct 100.0 30 / 30
1 Llama-3.3-70B-Instruct 100.0 30 / 30
1 Medgemma 27B ITfine-tune of Gemma 3medical model 100.0 30 / 30
1 Mistral Small 24B Instruct 2501 100.0 30 / 30
1 Mistral Small 3.2 24B Instruct (2506) 100.0 30 / 30
1 Mistral Small3.1 24B Instruct 2503 100.0 30 / 30
1 Muse Glimmer 30B 100.0 30 / 30
1 Nanbeige_Nanbeige4.3-33B 100.0 30 / 30
1 Nemotron 3.5 Bolt E4B 100.0 30 / 30
1 Nemotron 3.5 Lightning 30B A3B 100.0 30 / 30
1 Nemotron3 33B 100.0 30 / 30
1 Olmo-3-7B-Think 100.0 30 / 30
1 Openthinker 7B V2fine-tune of Qwen 2.5 100.0 30 / 30
1 Ornith 9B 100.0 30 / 30
1 Ornith-1.5-9B 100.0 30 / 30
1 Qwen2.5-14B-Instruct 100.0 30 / 30
1 Qwen2.5-32B-Instruct 100.0 30 / 30
1 Qwen2.5-7B-Instruct 100.0 30 / 30
1 Qwen2.5-Coder-32B-Instructcode model 100.0 30 / 30
1 Qwen2.5-VL-32B-Instruct 100.0 30 / 30
1 Qwen2.5-VL-3B-Instruct 100.0 30 / 30
1 Qwen3-14B 100.0 30 / 30
1 Qwen3-30B-A3B 100.0 30 / 30
1 Qwen3-30B-A3B-Instruct-2507 100.0 30 / 30
1 Qwen3-32B 100.0 30 / 30
1 Qwen3-4B 100.0 30 / 30
1 Qwen3-4B-Thinking-2507 100.0 30 / 30
1 Qwen3-8B 100.0 30 / 30
1 Qwen3-Coder-30B-A3B-Instructcode model 100.0 30 / 30
1 Qwen3-Coder-Nextcode model 100.0 30 / 30
1 Qwen3-VL-30B-A3B-Instruct 100.0 30 / 30
1 Qwen3-VL-32B-Instruct 100.0 30 / 30
1 Qwen3-VL-8B-Instruct 100.0 30 / 30
1 Qwen3.5-27B 100.0 30 / 30
1 Qwen3.5-35B-A3B 100.0 30 / 30
1 Qwen3.5-4B 100.0 30 / 30
1 Qwen3.5-9B 100.0 30 / 30
1 Qwen3.6-27B 100.0 30 / 30
1 Qwen3.6-35B-A3B 100.0 30 / 30
1 Qwen3.8-27B 100.0 30 / 30
1 QwQ 32B 100.0 30 / 30
1 Sage-A1.5-Q4_K_M 100.0 30 / 30
1 Translategemma 12B ITfine-tune of Gemma 3translation model 100.0 30 / 30
1 YI 34B Chat V1.5 100.0 30 / 30
66 Codestral 22B V0.1code model 96.7 30 / 30
66 Cogito 14B V1 Preview Qwenfine-tune of Qwen 2.5 96.7 30 / 30
66 Cogito 8B V1 Preview Llamafine-tune of Llama 3.1 96.7 30 / 30
66 DeepSeek-R1-0528-Qwen3-8B 96.7 30 / 30
66 Exaone3.5 32B Instruct 96.7 30 / 30
66 Falcon3 10B Instruct 96.7 30 / 30
66 Gemma 3n E4B IT 96.7 30 / 30
66 gemma-3-12b-it 96.7 30 / 30
66 gemma-3-27b-it 96.7 30 / 30
66 Gemma2 27B Instruct 96.7 30 / 30
66 Gemma4 E2B IT 96.7 30 / 30
66 Granite3.2 8B Instruct 96.7 30 / 30
66 Granite3.3 8B 96.7 30 / 30
66 Granite4.1 8B 96.7 30 / 30
66 Laguna XS 2.1 96.7 30 / 30
66 Lfm2.5 8B A1B 96.7 30 / 30
66 Ministral 3 14B Instruct (2512) 96.7 30 / 30
66 Ministral 3 8B Instruct 2512 96.7 30 / 30
66 Mistral Nemo 12B Instruct 2407 96.7 30 / 30
66 Mistral Small 22B Instruct 2409 96.7 30 / 30
66 NVIDIA Nemotron 3 Nano 30B-A3B 96.7 30 / 30
66 Olmo 3.1 32B Instruct 96.7 30 / 30
66 Olmo2 13B 1124 Instruct 96.7 30 / 30
66 Phi-3-mini-4k-instruct 96.7 30 / 30
66 phi-4 96.7 30 / 30
66 Phi-4-mini-instruct 96.7 30 / 30
66 Phi3 3.8B Mini 128K Instruct 96.7 30 / 30
66 Qwen2.5-VL-7B-Instruct 96.7 30 / 30
66 Qwen3-4B-Instruct-2507 96.7 30 / 30
66 SmolLM3-3B 96.7 30 / 30
96 Codegeex4 9B Allcode model 93.3 30 / 30
96 Command R 35B 08 2024 93.3 30 / 30
96 Command R7B Arabic 7B 02 2025 93.3 30 / 30
96 dolphin-2.9.1-yi-1.5-34b 93.3 30 / 30
96 Falcon3 7B Instruct 93.3 30 / 30
96 Granite3 Dense 8B Instruct 93.3 30 / 30
96 Hermes-3-Llama-3.1-8B 93.3 30 / 30
96 Ministral 3 3B Instruct 2512 93.3 30 / 30
96 Mistral 7B Instruct V0.3 93.3 30 / 30
96 Olmo-3-7B-Instruct 93.3 30 / 30
96 Olmo2 7B 1124 Instruct 93.3 30 / 30
96 Qwen 14B Chat V1.5 93.3 30 / 30
96 Qwen2.5 1.5B Instruct 93.3 30 / 30
96 Qwen2.5-Coder-14B-Instructcode model 93.3 30 / 30
96 Qwen2.5-Coder-7B-Instructcode model 93.3 30 / 30
96 Qwen3 VL 2B Instruct 93.3 30 / 30
112 Aya Expanse 8B 90.0 30 / 30
112 Deepseek R1 8B Llama Distill 90.0 30 / 30
112 Exaone3.5 2.4B Instruct 90.0 30 / 30
112 Flint 40B A2B 90.0 30 / 30
112 gemma-2-9b-it 90.0 30 / 30
112 Glm4 9B Chat 90.0 30 / 30
112 granite-4.1-3b 90.0 30 / 30
112 Granite3.3 2B 90.0 30 / 30
112 Granite4 Micro H 90.0 30 / 30
112 Granite4.2 3B 90.0 30 / 30
112 Lfm3 1.7B 90.0 30 / 30
112 Llama-3.1-8B-Instruct 90.0 30 / 30
112 Meta-Llama-3-8B-Instruct 90.0 30 / 30
112 Mistral-7B-Instruct-v0.2 90.0 30 / 30
112 Nemotron Cascade 2 30B A3B 90.0 30 / 30
112 Phi-3.5-mini-instruct 90.0 30 / 30
112 Phi3 14B Medium 128K Instruct 90.0 30 / 30
112 Phi4 Mini Reasoning 3.8B 90.0 30 / 30
112 Qwen2.5-3B-Instruct 90.0 30 / 30
112 Qwen3-VL-4B-Instruct 90.0 30 / 30
112 Spark-X2.6-4B 90.0 30 / 30
112 Tulu3 8Bfine-tune of Llama 3.1 90.0 30 / 30
112 YI 9B Chat V1.5 90.0 30 / 30
135 Cogito 3B V1 Preview Llamafine-tune of Llama 3.2 86.7 30 / 30
135 Command R7B (12-2024) 86.7 30 / 30
135 Exaone Deep 7.8B 86.7 30 / 30
135 Gemma2 2B Instruct 86.7 30 / 30
135 Granite3 Dense 2B Instruct 86.7 30 / 30
135 Granite4 Tiny H 86.7 30 / 30
135 Lfm3 VL 2B 86.7 30 / 30
135 Llama-3.2-3B-Instruct 86.7 30 / 30
135 Mathstral 7B V0.1fine-tune of Mistral 7Bmath model 86.7 30 / 30
135 Medgemma1.5 4B ITfine-tune of Gemma 3medical model 86.7 30 / 30
135 Qwen2-VL-7B-Instruct 86.7 30 / 30
135 Solar 10.7B Instruct V1 86.7 30 / 30
147 Apertus-8B-Instruct-2509 83.3 30 / 30
147 Codellama 13B Instructcode model 83.3 30 / 30
147 DeepSeek-Coder-V2-Lite-Instructcode model 83.3 30 / 30
147 DeepSeek-R1-Distill-Qwen-7B 83.3 30 / 30
147 Gemma 7B Instruct V1.1 83.3 30 / 30
147 Gemma3n E2B IT 83.3 30 / 30
147 Granite3.1 Moe 3B Instruct 83.3 30 / 30
147 Lfm2.5 Thinking 1.2B 83.3 30 / 30
147 Mixtral 8x7B Instruct V0.1 83.3 30 / 30
147 Nemotron Mini 4B Instruct 83.3 30 / 30
147 Phi 2.7B Chat V2 83.3 30 / 30
147 YI Coder 9B Chatcode model 83.3 30 / 30
159 gemma-3-4b-it 80.0 30 / 30
159 Granite3.2 2B Instruct 80.0 30 / 30
159 medgemma-4b-itmedical model 80.0 30 / 30
159 Qwen2.5-Coder-3B-Instructcode model 80.0 30 / 30
159 Qwen3.5 2B 80.0 30 / 30
159 Sailor2 8B Chatfine-tune of Qwen 2.5 80.0 30 / 30
159 Smallthinker 3B Previewfine-tune of Qwen 2.5 3B 80.0 30 / 30
166 Aya 8B 23 76.7 30 / 30
166 Codellama 7B Instructcode model 76.7 30 / 30
166 Deepseek Llm 7B Chat 76.7 30 / 30
166 Granite3 Moe 3B Instruct 76.7 30 / 30
166 Granite3.1 Dense 2B Instruct 76.7 30 / 30
166 Granite4 1B H 76.7 30 / 30
166 Granite4 350M H 76.7 30 / 30
166 Granite4 3B 76.7 30 / 30
166 Llama3.2 1B Instruct 76.7 30 / 30
166 Marco O1 7Bfine-tune of Qwen 2 7B 76.7 30 / 30
166 Qwen 7B Chat V1.5 76.7 30 / 30
166 Qwen3 1.7B 76.7 30 / 30
178 Codellama 34B Instructcode model 73.3 30 / 30
178 Falcon3 3B Instruct 73.3 30 / 30
178 Llama2 13B Chat 73.3 30 / 30
178 Qwen 4B Chat V1.5 73.3 30 / 30
178 Qwen3 VL 2B Thinking 73.3 30 / 30
178 YI 6B Chat V1.5 73.3 30 / 30
184 Codegemma 7B Instruct V1.1code model 70.0 30 / 30
184 Deepseek V2 16B Lite Chat 70.0 30 / 30
184 Dolphin Phi 2.7B V2.6fine-tune of Phi-2 70.0 30 / 30
184 Falcon2 11Bnot in the Overall ranking 70.0 30 / 30
184 Falcon3 1B Instruct 70.0 30 / 30
184 Internlm2 1.8B Chat V2.5 70.0 30 / 30
184 Stablelm2 12B Chat 70.0 30 / 30
191 Llama-2-7b-chat-hf 66.7 30 / 30
191 Qwen2 Math 7B Instructmath model 66.7 30 / 30
193 Exaone Deep 2.4B 63.3 30 / 30
193 Gemma3 1B IT 63.3 30 / 30
193 Granite Code 8B Instructcode model 63.3 30 / 30
193 Granite3.1 Moe 1B Instruct 63.3 30 / 30
193 Hermes3 3B Llama3.2fine-tune of Llama 3.2 63.3 30 / 30
193 Qwen2 1.5B Instruct 63.3 30 / 30
193 Smollm2 1.7B Instruct 63.3 30 / 30
200 Gemma 2B Instruct V1.1 60.0 30 / 30
200 Internlm2 7B Chat V2.5 60.0 30 / 30
200 Qwen 1.8B Chat V1.5 60.0 30 / 30
200 Qwen2.5 Coder 1.5B Instructcode model 60.0 30 / 30
200 Sailor2 1B Chatfine-tune of Qwen 2.5 60.0 30 / 30
205 Deepcoder 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bcode model 56.7 30 / 30
205 Granite Code 3B Instructcode model 56.7 30 / 30
205 Magistral Small (2506) 56.7 30 / 30
205 Qwen2.5 0.5B Instruct 56.7 30 / 30
209 Deepseek R1 1.5B Qwen Distill 53.3 30 / 30
209 YI Coder 1.5B Chatcode model 53.3 30 / 30
211 Deepscaler 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bmath model 50.0 30 / 30
211 Stablelm Zephyr 3Bfine-tune of StableLM 3B 50.0 30 / 30
213 Smollm2 360M Instruct 46.7 30 / 30
214 Qwen2 0.5B Instruct 43.3 30 / 30
215 Codeqwen 7B Chat V1.5code model 40.0 30 / 30
215 Stablelm2 1.6B Chat 40.0 30 / 30
217 Granite3 Moe 1B Instruct 36.7 30 / 30
217 Smollm2 135M Instruct 36.7 30 / 30
219 Qwen2 Math 1.5B Instructmath model 33.3 30 / 30
219 Stable Code 3B Instructcode model 33.3 30 / 30
221 Opencoder 1.5B Instructcode model 20.0 30 / 30
221 Qwen2.5 Coder 0.5B Instructcode model 20.0 30 / 30
221 Qwen3 0.6B 20.0 30 / 30
224 Codegemma 2B Code V1.1code model 16.7 30 / 30
225 Qwen 0.5B Chat V1.5 13.3 30 / 30
226 Deepseek Coder 6.7B Instructcode model 10.0 30 / 30
227 Deepseek Coder 33B Instructcode model 6.7 30 / 30
228 Deepseek Coder 1.3B Instructcode model 3.3 30 / 30

A dash means not scored, which is different from 0. Agreement compares the two judges: high means a mean gap under 1.5 points, medium under 3, low otherwise; n/a means fewer than 3 items were scored by both.

Not ranked in this category

The questions

In run r20261102-bv1-5246ab, 1 of the 30 questions in this category was answered correctly by every ranked model that has a score for it (33 ranked models in that run). Hard questions are the ones that half or fewer of the ranked models of 3B and over answered correctly. This category has 0.

The same ranking as JSON: knowledge.json.