All categories

Long context

A fact hidden in a long, made-up document. The model must find it, sometimes among look-alike facts or by combining two facts.

This category counts toward Overall. How the battery works.

Ranking

Models are ranked by their mean score in this category, from 0 to 100. Only models with at least 80% of the category's items scored, and none pending, are ranked here. The rule is the same as for Overall.

Long context: 227 ranked models. Click a column heading to sort.
Rank Model Mean (0 to 100) Items scored
1 DeepSeek-R1-Distill-Qwen-32B 100.0 18 / 18
1 gemma-3-27b-it 100.0 18 / 18
1 gemma-4-31B-it 100.0 18 / 18
1 Gemma4 E2B IT 100.0 18 / 18
1 Ministral 3 14B Instruct (2512) 100.0 18 / 18
1 Qwen3.5-27B 100.0 18 / 18
1 Qwen3.5-35B-A3B 100.0 18 / 18
1 QwQ 32B 100.0 18 / 18
9 Devstral 24B Small 2505code model 94.4 18 / 18
9 Devstral Small 2 24B Instruct (2512)code model 94.4 18 / 18
9 Gemma 4 12B IT 94.4 18 / 18
9 Gemma4 E4B IT 94.4 18 / 18
9 gpt-oss-120b 94.4 18 / 18
9 gpt-oss-20b 94.4 18 / 18
9 Granite4.1 8B 94.4 18 / 18
9 Granite4.2 30B 94.4 18 / 18
9 Laguna XS 2.1 94.4 18 / 18
9 Llama 4 Scout 17B-16E Instruct 94.4 18 / 18
9 Ministral 3 8B Instruct 2512 94.4 18 / 18
9 Mistral Small 24B Instruct 2501 94.4 18 / 18
9 Mistral Small 3.2 24B Instruct (2506) 94.4 18 / 18
9 Muse Glimmer 30B 94.4 18 / 18
9 Ornith 9B 94.4 18 / 18
9 Qwen2.5-14B-Instruct 94.4 18 / 18
9 Qwen2.5-32B-Instruct 94.4 18 / 18
9 Qwen2.5-Coder-32B-Instructcode model 94.4 18 / 18
9 Qwen3-4B-Thinking-2507 94.4 18 / 18
9 Qwen3-Coder-30B-A3B-Instructcode model 94.4 18 / 18
9 Qwen3-Coder-Nextcode model 94.4 18 / 18
9 Qwen3-VL-32B-Instruct 94.4 18 / 18
9 Qwen3-VL-8B-Instruct 94.4 18 / 18
9 Qwen3.8-27B 94.4 18 / 18
9 Sage-A1.5-Q4_K_M 94.4 18 / 18
34 Cogito 14B V1 Preview Qwenfine-tune of Qwen 2.5 88.9 18 / 18
34 Command R 35B 08 2024 88.9 18 / 18
34 gemma-4-26B-A4B-it 88.9 18 / 18
34 Granite4 Small H 88.9 18 / 18
34 Llama-3.3-70B-Instruct 88.9 18 / 18
34 Medgemma 27B ITfine-tune of Gemma 3medical model 88.9 18 / 18
34 Mistral Small 22B Instruct 2409 88.9 18 / 18
34 Mistral Small3.1 24B Instruct 2503 88.9 18 / 18
34 Nanbeige_Nanbeige4.3-33B 88.9 18 / 18
34 Olmo-3-7B-Think 88.9 18 / 18
34 Ornith-1.5-9B 88.9 18 / 18
34 Phi3 14B Medium 128K Instruct 88.9 18 / 18
34 Qwen2.5-Coder-14B-Instructcode model 88.9 18 / 18
34 Qwen2.5-VL-32B-Instruct 88.9 18 / 18
34 Qwen3-14B 88.9 18 / 18
34 Qwen3-30B-A3B-Instruct-2507 88.9 18 / 18
34 Qwen3-32B 88.9 18 / 18
34 Qwen3-4B-Instruct-2507 88.9 18 / 18
34 Qwen3-VL-30B-A3B-Instruct 88.9 18 / 18
34 Qwen3.5-9B 88.9 18 / 18
34 Qwen3.6-27B 88.9 18 / 18
55 Codestral 22B V0.1code model 83.3 18 / 18
55 Cogito 8B V1 Preview Llamafine-tune of Llama 3.1 83.3 18 / 18
55 Command R7B (12-2024) 83.3 18 / 18
55 gemma-3-12b-it 83.3 18 / 18
55 Gemma2 27B Instruct 83.3 18 / 18
55 GLM-4.5-Air 83.3 18 / 18
55 Llama-3.1-8B-Instruct 83.3 18 / 18
55 Meta-Llama-3-8B-Instruct 83.3 18 / 18
55 Nemotron 3.5 Lightning 30B A3B 83.3 18 / 18
55 Nemotron3 33B 83.3 18 / 18
55 Qwen3-30B-A3B 83.3 18 / 18
55 Qwen3-8B 83.3 18 / 18
55 Qwen3-VL-4B-Instruct 83.3 18 / 18
55 Qwen3.5-4B 83.3 18 / 18
55 Qwen3.6-35B-A3B 83.3 18 / 18
70 Codegeex4 9B Allcode model 77.8 18 / 18
70 Command R7B Arabic 7B 02 2025 77.8 18 / 18
70 DeepSeek-Coder-V2-Lite-Instructcode model 77.8 18 / 18
70 DeepSeek-R1-0528-Qwen3-8B 77.8 18 / 18
70 DeepSeek-R1-Distill-Qwen-14B 77.8 18 / 18
70 Flint 40B A2B 77.8 18 / 18
70 GLM-4.7-Flash 77.8 18 / 18
70 Mistral Nemo 12B Instruct 2407 77.8 18 / 18
70 Nemotron Cascade 2 30B A3B 77.8 18 / 18
70 Olmo 3.1 32B Instruct 77.8 18 / 18
70 phi-4 77.8 18 / 18
70 Qwen2.5-7B-Instruct 77.8 18 / 18
70 Qwen2.5-Coder-7B-Instructcode model 77.8 18 / 18
70 Qwen2.5-VL-7B-Instruct 77.8 18 / 18
70 Qwen3-4B 77.8 18 / 18
70 Qwen3.5 2B 77.8 18 / 18
70 SmolLM3-3B 77.8 18 / 18
70 Spark-X2.6-4B 77.8 18 / 18
88 Aya Expanse 32B 72.2 18 / 18
88 Deepseek R1 8B Llama Distill 72.2 18 / 18
88 Falcon3 10B Instruct 72.2 18 / 18
88 Gemma3n E2B IT 72.2 18 / 18
88 Granite4.1 30B 72.2 18 / 18
88 Internlm2 7B Chat V2.5 72.2 18 / 18
88 Marco O1 7Bfine-tune of Qwen 2 7B 72.2 18 / 18
88 Medgemma1.5 4B ITfine-tune of Gemma 3medical model 72.2 18 / 18
88 Mistral-7B-Instruct-v0.2 72.2 18 / 18
88 Mixtral 8x7B Instruct V0.1 72.2 18 / 18
88 Nemotron 3.5 Bolt E4B 72.2 18 / 18
88 Phi-3.5-mini-instruct 72.2 18 / 18
88 Qwen2.5-Coder-3B-Instructcode model 72.2 18 / 18
101 Aya Expanse 8B 66.7 18 / 18
101 Codellama 34B Instructcode model 66.7 18 / 18
101 Deepcoder 14B Previewfine-tune of DeepSeek-R1-Distill-Qwen 14Bcode model 66.7 18 / 18
101 dolphin-2.9.1-yi-1.5-34b 66.7 18 / 18
101 Exaone3.5 32B Instruct 66.7 18 / 18
101 Falcon3 3B Instruct 66.7 18 / 18
101 Gemma 3n E4B IT 66.7 18 / 18
101 Glm4 9B Chat 66.7 18 / 18
101 granite-4.1-3b 66.7 18 / 18
101 Granite3.3 8B 66.7 18 / 18
101 Granite4 Tiny H 66.7 18 / 18
101 Lfm2 24B 66.7 18 / 18
101 Phi-4-mini-instruct 66.7 18 / 18
101 Phi4 Mini Reasoning 3.8B 66.7 18 / 18
101 Qwen2-VL-7B-Instruct 66.7 18 / 18
101 Qwen2.5 1.5B Instruct 66.7 18 / 18
101 Qwen2.5 Coder 1.5B Instructcode model 66.7 18 / 18
101 Qwen2.5-VL-3B-Instruct 66.7 18 / 18
101 Qwen3 VL 2B Instruct 66.7 18 / 18
101 Translategemma 12B ITfine-tune of Gemma 3translation model 66.7 18 / 18
121 Aya 8B 23 61.1 18 / 18
121 Codellama 7B Instructcode model 61.1 18 / 18
121 Cogito 3B V1 Preview Llamafine-tune of Llama 3.2 61.1 18 / 18
121 Deepseek Coder 6.7B Instructcode model 61.1 18 / 18
121 Exaone Deep 7.8B 61.1 18 / 18
121 Gemma 7B Instruct V1.1 61.1 18 / 18
121 gemma-2-9b-it 61.1 18 / 18
121 gemma-3-4b-it 61.1 18 / 18
121 Granite3.1 Moe 3B Instruct 61.1 18 / 18
121 Granite4.2 3B 61.1 18 / 18
121 Lfm2.5 8B A1B 61.1 18 / 18
121 Ministral 3 3B Instruct 2512 61.1 18 / 18
121 Mistral 7B Instruct V0.3 61.1 18 / 18
121 NVIDIA Nemotron 3 Nano 30B-A3B 61.1 18 / 18
121 Qwen2.5-3B-Instruct 61.1 18 / 18
121 Tulu3 8Bfine-tune of Llama 3.1 61.1 18 / 18
137 EXAONE-3.5-7.8B-Instruct 55.6 18 / 18
137 Falcon3 7B Instruct 55.6 18 / 18
137 Gemma2 2B Instruct 55.6 18 / 18
137 Granite3.1 Dense 8B Instruct 55.6 18 / 18
137 Granite4 1B H 55.6 18 / 18
137 Granite4 3B 55.6 18 / 18
137 Granite4.2 8B 55.6 18 / 18
137 Llama-3.2-3B-Instruct 55.6 18 / 18
137 Mathstral 7B V0.1fine-tune of Mistral 7Bmath model 55.6 18 / 18
137 Openthinker 7B V2fine-tune of Qwen 2.5 55.6 18 / 18
137 Qwen3 1.7B 55.6 18 / 18
137 Sailor2 8B Chatfine-tune of Qwen 2.5 55.6 18 / 18
149 Apertus-8B-Instruct-2509 50.0 18 / 18
149 Codellama 13B Instructcode model 50.0 18 / 18
149 Exaone Deep 2.4B 50.0 18 / 18
149 Exaone3.5 2.4B Instruct 50.0 18 / 18
149 Granite3.1 Dense 2B Instruct 50.0 18 / 18
149 Granite3.2 2B Instruct 50.0 18 / 18
149 Granite3.2 8B Instruct 50.0 18 / 18
149 Hermes3 3B Llama3.2fine-tune of Llama 3.2 50.0 18 / 18
149 Internlm2 1.8B Chat V2.5 50.0 18 / 18
149 medgemma-4b-itmedical model 50.0 18 / 18
149 Phi3 3.8B Mini 128K Instruct 50.0 18 / 18
160 DeepSeek-R1-Distill-Qwen-7B 44.4 18 / 18
160 Granite3.3 2B 44.4 18 / 18
160 Hermes-3-Llama-3.1-8B 44.4 18 / 18
160 Lfm3 VL 2B 44.4 18 / 18
160 Magistral Small (2506) 44.4 18 / 18
160 Olmo-3-7B-Instruct 44.4 18 / 18
160 YI Coder 9B Chatcode model 44.4 18 / 18
167 Codeqwen 7B Chat V1.5code model 38.9 18 / 18
167 Granite4 Micro H 38.9 18 / 18
167 Lfm2.5 Thinking 1.2B 38.9 18 / 18
167 Llama3.2 1B Instruct 38.9 18 / 18
167 Qwen2.5 0.5B Instruct 38.9 18 / 18
167 Smallthinker 3B Previewfine-tune of Qwen 2.5 3B 38.9 18 / 18
167 Solar 10.7B Instruct V1 38.9 18 / 18
167 YI 34B Chat V1.5 38.9 18 / 18
175 Deepseek V2 16B Lite Chat 33.3 18 / 18
175 Granite3 Dense 2B Instruct 33.3 18 / 18
175 Granite3 Dense 8B Instruct 33.3 18 / 18
175 Granite3.1 Moe 1B Instruct 33.3 18 / 18
175 Lfm3 1.7B 33.3 18 / 18
175 Olmo2 13B 1124 Instruct 33.3 18 / 18
175 Qwen2 Math 7B Instructmath model 33.3 18 / 18
175 Qwen2.5 Coder 0.5B Instructcode model 33.3 18 / 18
175 Qwen3 VL 2B Thinking 33.3 18 / 18
175 Smollm2 1.7B Instruct 33.3 18 / 18
175 YI 6B Chat V1.5 33.3 18 / 18
186 Deepseek Llm 7B Chat 27.8 18 / 18
186 Gemma 2B Instruct V1.1 27.8 18 / 18
186 Granite Code 8B Instructcode model 27.8 18 / 18
186 Granite3 Moe 3B Instruct 27.8 18 / 18
186 Llama2 13B Chat 27.8 18 / 18
186 Olmo2 7B 1124 Instruct 27.8 18 / 18
186 Qwen2 1.5B Instruct 27.8 18 / 18
193 Deepscaler 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bmath model 22.2 18 / 18
193 Deepseek R1 1.5B Qwen Distill 22.2 18 / 18
193 Gemma3 1B IT 22.2 18 / 18
193 Qwen2 0.5B Instruct 22.2 18 / 18
193 Stablelm Zephyr 3Bfine-tune of StableLM 3B 22.2 18 / 18
193 YI 9B Chat V1.5 22.2 18 / 18
199 Deepcoder 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bcode model 16.7 18 / 18
199 Granite Code 3B Instructcode model 16.7 18 / 18
199 Granite4 350M H 16.7 18 / 18
199 Llama-2-7b-chat-hf 16.7 18 / 18
199 Sailor2 1B Chatfine-tune of Qwen 2.5 16.7 18 / 18
204 Deepseek Coder 1.3B Instructcode model 11.1 18 / 18
204 Falcon3 1B Instruct 11.1 18 / 18
204 Granite3 Moe 1B Instruct 11.1 18 / 18
204 Nemotron Mini 4B Instruct 11.1 18 / 18
204 Opencoder 1.5B Instructcode model 11.1 18 / 18
204 Phi-3-mini-4k-instruct 11.1 18 / 18
204 Qwen 14B Chat V1.5 11.1 18 / 18
204 Qwen 7B Chat V1.5 11.1 18 / 18
204 Qwen2 Math 1.5B Instructmath model 11.1 18 / 18
204 Qwen3 0.6B 11.1 18 / 18
204 Smollm2 360M Instruct 11.1 18 / 18
215 Codegemma 7B Instruct V1.1code model 5.6 18 / 18
215 Dolphin Phi 2.7B V2.6fine-tune of Phi-2 5.6 18 / 18
215 Smollm2 135M Instruct 5.6 18 / 18
215 Stable Code 3B Instructcode model 5.6 18 / 18
215 Stablelm2 12B Chat 5.6 18 / 18
215 YI Coder 1.5B Chatcode model 5.6 18 / 18
221 Codegemma 2B Code V1.1code model 0 18 / 18
221 Deepseek Coder 33B Instructcode model 0 18 / 18
221 Phi 2.7B Chat V2 0 18 / 18
221 Qwen 0.5B Chat V1.5 0 18 / 18
221 Qwen 1.8B Chat V1.5 0 18 / 18
221 Qwen 4B Chat V1.5 0 18 / 18
221 Stablelm2 1.6B Chat 0 18 / 18

A dash means not scored, which is different from 0. Agreement compares the two judges: high means a mean gap under 1.5 points, medium under 3, low otherwise; n/a means fewer than 3 items were scored by both.

Not ranked in this category

The questions

In run r20261102-bv1-5246ab, 0 of the 18 questions in this category were answered correctly by every ranked model that has a score for them (33 ranked models in that run). Hard questions are the ones that half or fewer of the ranked models of 3B and over answered correctly. This category has 2.

The same ranking as JSON: long_context.json.