Long context
A fact hidden in a long, made-up document. The model must find it, sometimes among look-alike facts or by combining two facts.
This category counts toward Overall. How the battery works.
Ranking
Models are ranked by their mean score in this category, from 0 to 100. Only models with at least 80% of the category's items scored, and none pending, are ranked here. The rule is the same as for Overall.
| Rank | Model | Mean (0 to 100) | Items scored |
|---|---|---|---|
| 1 | DeepSeek-R1-Distill-Qwen-32B | 100.0 | 18 / 18 |
| 1 | gemma-3-27b-it | 100.0 | 18 / 18 |
| 1 | gemma-4-31B-it | 100.0 | 18 / 18 |
| 1 | Gemma4 E2B IT | 100.0 | 18 / 18 |
| 1 | Ministral 3 14B Instruct (2512) | 100.0 | 18 / 18 |
| 1 | Qwen3.5-27B | 100.0 | 18 / 18 |
| 1 | Qwen3.5-35B-A3B | 100.0 | 18 / 18 |
| 1 | QwQ 32B | 100.0 | 18 / 18 |
| 9 | Devstral 24B Small 2505code model | 94.4 | 18 / 18 |
| 9 | Devstral Small 2 24B Instruct (2512)code model | 94.4 | 18 / 18 |
| 9 | Gemma 4 12B IT | 94.4 | 18 / 18 |
| 9 | Gemma4 E4B IT | 94.4 | 18 / 18 |
| 9 | gpt-oss-120b | 94.4 | 18 / 18 |
| 9 | gpt-oss-20b | 94.4 | 18 / 18 |
| 9 | Granite4.1 8B | 94.4 | 18 / 18 |
| 9 | Granite4.2 30B | 94.4 | 18 / 18 |
| 9 | Laguna XS 2.1 | 94.4 | 18 / 18 |
| 9 | Llama 4 Scout 17B-16E Instruct | 94.4 | 18 / 18 |
| 9 | Ministral 3 8B Instruct 2512 | 94.4 | 18 / 18 |
| 9 | Mistral Small 24B Instruct 2501 | 94.4 | 18 / 18 |
| 9 | Mistral Small 3.2 24B Instruct (2506) | 94.4 | 18 / 18 |
| 9 | Muse Glimmer 30B | 94.4 | 18 / 18 |
| 9 | Ornith 9B | 94.4 | 18 / 18 |
| 9 | Qwen2.5-14B-Instruct | 94.4 | 18 / 18 |
| 9 | Qwen2.5-32B-Instruct | 94.4 | 18 / 18 |
| 9 | Qwen2.5-Coder-32B-Instructcode model | 94.4 | 18 / 18 |
| 9 | Qwen3-4B-Thinking-2507 | 94.4 | 18 / 18 |
| 9 | Qwen3-Coder-30B-A3B-Instructcode model | 94.4 | 18 / 18 |
| 9 | Qwen3-Coder-Nextcode model | 94.4 | 18 / 18 |
| 9 | Qwen3-VL-32B-Instruct | 94.4 | 18 / 18 |
| 9 | Qwen3-VL-8B-Instruct | 94.4 | 18 / 18 |
| 9 | Qwen3.8-27B | 94.4 | 18 / 18 |
| 9 | Sage-A1.5-Q4_K_M | 94.4 | 18 / 18 |
| 34 | Cogito 14B V1 Preview Qwenfine-tune of Qwen 2.5 | 88.9 | 18 / 18 |
| 34 | Command R 35B 08 2024 | 88.9 | 18 / 18 |
| 34 | gemma-4-26B-A4B-it | 88.9 | 18 / 18 |
| 34 | Granite4 Small H | 88.9 | 18 / 18 |
| 34 | Llama-3.3-70B-Instruct | 88.9 | 18 / 18 |
| 34 | Medgemma 27B ITfine-tune of Gemma 3medical model | 88.9 | 18 / 18 |
| 34 | Mistral Small 22B Instruct 2409 | 88.9 | 18 / 18 |
| 34 | Mistral Small3.1 24B Instruct 2503 | 88.9 | 18 / 18 |
| 34 | Nanbeige_Nanbeige4.3-33B | 88.9 | 18 / 18 |
| 34 | Olmo-3-7B-Think | 88.9 | 18 / 18 |
| 34 | Ornith-1.5-9B | 88.9 | 18 / 18 |
| 34 | Phi3 14B Medium 128K Instruct | 88.9 | 18 / 18 |
| 34 | Qwen2.5-Coder-14B-Instructcode model | 88.9 | 18 / 18 |
| 34 | Qwen2.5-VL-32B-Instruct | 88.9 | 18 / 18 |
| 34 | Qwen3-14B | 88.9 | 18 / 18 |
| 34 | Qwen3-30B-A3B-Instruct-2507 | 88.9 | 18 / 18 |
| 34 | Qwen3-32B | 88.9 | 18 / 18 |
| 34 | Qwen3-4B-Instruct-2507 | 88.9 | 18 / 18 |
| 34 | Qwen3-VL-30B-A3B-Instruct | 88.9 | 18 / 18 |
| 34 | Qwen3.5-9B | 88.9 | 18 / 18 |
| 34 | Qwen3.6-27B | 88.9 | 18 / 18 |
| 55 | Codestral 22B V0.1code model | 83.3 | 18 / 18 |
| 55 | Cogito 8B V1 Preview Llamafine-tune of Llama 3.1 | 83.3 | 18 / 18 |
| 55 | Command R7B (12-2024) | 83.3 | 18 / 18 |
| 55 | gemma-3-12b-it | 83.3 | 18 / 18 |
| 55 | Gemma2 27B Instruct | 83.3 | 18 / 18 |
| 55 | GLM-4.5-Air | 83.3 | 18 / 18 |
| 55 | Llama-3.1-8B-Instruct | 83.3 | 18 / 18 |
| 55 | Meta-Llama-3-8B-Instruct | 83.3 | 18 / 18 |
| 55 | Nemotron 3.5 Lightning 30B A3B | 83.3 | 18 / 18 |
| 55 | Nemotron3 33B | 83.3 | 18 / 18 |
| 55 | Qwen3-30B-A3B | 83.3 | 18 / 18 |
| 55 | Qwen3-8B | 83.3 | 18 / 18 |
| 55 | Qwen3-VL-4B-Instruct | 83.3 | 18 / 18 |
| 55 | Qwen3.5-4B | 83.3 | 18 / 18 |
| 55 | Qwen3.6-35B-A3B | 83.3 | 18 / 18 |
| 70 | Codegeex4 9B Allcode model | 77.8 | 18 / 18 |
| 70 | Command R7B Arabic 7B 02 2025 | 77.8 | 18 / 18 |
| 70 | DeepSeek-Coder-V2-Lite-Instructcode model | 77.8 | 18 / 18 |
| 70 | DeepSeek-R1-0528-Qwen3-8B | 77.8 | 18 / 18 |
| 70 | DeepSeek-R1-Distill-Qwen-14B | 77.8 | 18 / 18 |
| 70 | Flint 40B A2B | 77.8 | 18 / 18 |
| 70 | GLM-4.7-Flash | 77.8 | 18 / 18 |
| 70 | Mistral Nemo 12B Instruct 2407 | 77.8 | 18 / 18 |
| 70 | Nemotron Cascade 2 30B A3B | 77.8 | 18 / 18 |
| 70 | Olmo 3.1 32B Instruct | 77.8 | 18 / 18 |
| 70 | phi-4 | 77.8 | 18 / 18 |
| 70 | Qwen2.5-7B-Instruct | 77.8 | 18 / 18 |
| 70 | Qwen2.5-Coder-7B-Instructcode model | 77.8 | 18 / 18 |
| 70 | Qwen2.5-VL-7B-Instruct | 77.8 | 18 / 18 |
| 70 | Qwen3-4B | 77.8 | 18 / 18 |
| 70 | Qwen3.5 2B | 77.8 | 18 / 18 |
| 70 | SmolLM3-3B | 77.8 | 18 / 18 |
| 70 | Spark-X2.6-4B | 77.8 | 18 / 18 |
| 88 | Aya Expanse 32B | 72.2 | 18 / 18 |
| 88 | Deepseek R1 8B Llama Distill | 72.2 | 18 / 18 |
| 88 | Falcon3 10B Instruct | 72.2 | 18 / 18 |
| 88 | Gemma3n E2B IT | 72.2 | 18 / 18 |
| 88 | Granite4.1 30B | 72.2 | 18 / 18 |
| 88 | Internlm2 7B Chat V2.5 | 72.2 | 18 / 18 |
| 88 | Marco O1 7Bfine-tune of Qwen 2 7B | 72.2 | 18 / 18 |
| 88 | Medgemma1.5 4B ITfine-tune of Gemma 3medical model | 72.2 | 18 / 18 |
| 88 | Mistral-7B-Instruct-v0.2 | 72.2 | 18 / 18 |
| 88 | Mixtral 8x7B Instruct V0.1 | 72.2 | 18 / 18 |
| 88 | Nemotron 3.5 Bolt E4B | 72.2 | 18 / 18 |
| 88 | Phi-3.5-mini-instruct | 72.2 | 18 / 18 |
| 88 | Qwen2.5-Coder-3B-Instructcode model | 72.2 | 18 / 18 |
| 101 | Aya Expanse 8B | 66.7 | 18 / 18 |
| 101 | Codellama 34B Instructcode model | 66.7 | 18 / 18 |
| 101 | Deepcoder 14B Previewfine-tune of DeepSeek-R1-Distill-Qwen 14Bcode model | 66.7 | 18 / 18 |
| 101 | dolphin-2.9.1-yi-1.5-34b | 66.7 | 18 / 18 |
| 101 | Exaone3.5 32B Instruct | 66.7 | 18 / 18 |
| 101 | Falcon3 3B Instruct | 66.7 | 18 / 18 |
| 101 | Gemma 3n E4B IT | 66.7 | 18 / 18 |
| 101 | Glm4 9B Chat | 66.7 | 18 / 18 |
| 101 | granite-4.1-3b | 66.7 | 18 / 18 |
| 101 | Granite3.3 8B | 66.7 | 18 / 18 |
| 101 | Granite4 Tiny H | 66.7 | 18 / 18 |
| 101 | Lfm2 24B | 66.7 | 18 / 18 |
| 101 | Phi-4-mini-instruct | 66.7 | 18 / 18 |
| 101 | Phi4 Mini Reasoning 3.8B | 66.7 | 18 / 18 |
| 101 | Qwen2-VL-7B-Instruct | 66.7 | 18 / 18 |
| 101 | Qwen2.5 1.5B Instruct | 66.7 | 18 / 18 |
| 101 | Qwen2.5 Coder 1.5B Instructcode model | 66.7 | 18 / 18 |
| 101 | Qwen2.5-VL-3B-Instruct | 66.7 | 18 / 18 |
| 101 | Qwen3 VL 2B Instruct | 66.7 | 18 / 18 |
| 101 | Translategemma 12B ITfine-tune of Gemma 3translation model | 66.7 | 18 / 18 |
| 121 | Aya 8B 23 | 61.1 | 18 / 18 |
| 121 | Codellama 7B Instructcode model | 61.1 | 18 / 18 |
| 121 | Cogito 3B V1 Preview Llamafine-tune of Llama 3.2 | 61.1 | 18 / 18 |
| 121 | Deepseek Coder 6.7B Instructcode model | 61.1 | 18 / 18 |
| 121 | Exaone Deep 7.8B | 61.1 | 18 / 18 |
| 121 | Gemma 7B Instruct V1.1 | 61.1 | 18 / 18 |
| 121 | gemma-2-9b-it | 61.1 | 18 / 18 |
| 121 | gemma-3-4b-it | 61.1 | 18 / 18 |
| 121 | Granite3.1 Moe 3B Instruct | 61.1 | 18 / 18 |
| 121 | Granite4.2 3B | 61.1 | 18 / 18 |
| 121 | Lfm2.5 8B A1B | 61.1 | 18 / 18 |
| 121 | Ministral 3 3B Instruct 2512 | 61.1 | 18 / 18 |
| 121 | Mistral 7B Instruct V0.3 | 61.1 | 18 / 18 |
| 121 | NVIDIA Nemotron 3 Nano 30B-A3B | 61.1 | 18 / 18 |
| 121 | Qwen2.5-3B-Instruct | 61.1 | 18 / 18 |
| 121 | Tulu3 8Bfine-tune of Llama 3.1 | 61.1 | 18 / 18 |
| 137 | EXAONE-3.5-7.8B-Instruct | 55.6 | 18 / 18 |
| 137 | Falcon3 7B Instruct | 55.6 | 18 / 18 |
| 137 | Gemma2 2B Instruct | 55.6 | 18 / 18 |
| 137 | Granite3.1 Dense 8B Instruct | 55.6 | 18 / 18 |
| 137 | Granite4 1B H | 55.6 | 18 / 18 |
| 137 | Granite4 3B | 55.6 | 18 / 18 |
| 137 | Granite4.2 8B | 55.6 | 18 / 18 |
| 137 | Llama-3.2-3B-Instruct | 55.6 | 18 / 18 |
| 137 | Mathstral 7B V0.1fine-tune of Mistral 7Bmath model | 55.6 | 18 / 18 |
| 137 | Openthinker 7B V2fine-tune of Qwen 2.5 | 55.6 | 18 / 18 |
| 137 | Qwen3 1.7B | 55.6 | 18 / 18 |
| 137 | Sailor2 8B Chatfine-tune of Qwen 2.5 | 55.6 | 18 / 18 |
| 149 | Apertus-8B-Instruct-2509 | 50.0 | 18 / 18 |
| 149 | Codellama 13B Instructcode model | 50.0 | 18 / 18 |
| 149 | Exaone Deep 2.4B | 50.0 | 18 / 18 |
| 149 | Exaone3.5 2.4B Instruct | 50.0 | 18 / 18 |
| 149 | Granite3.1 Dense 2B Instruct | 50.0 | 18 / 18 |
| 149 | Granite3.2 2B Instruct | 50.0 | 18 / 18 |
| 149 | Granite3.2 8B Instruct | 50.0 | 18 / 18 |
| 149 | Hermes3 3B Llama3.2fine-tune of Llama 3.2 | 50.0 | 18 / 18 |
| 149 | Internlm2 1.8B Chat V2.5 | 50.0 | 18 / 18 |
| 149 | medgemma-4b-itmedical model | 50.0 | 18 / 18 |
| 149 | Phi3 3.8B Mini 128K Instruct | 50.0 | 18 / 18 |
| 160 | DeepSeek-R1-Distill-Qwen-7B | 44.4 | 18 / 18 |
| 160 | Granite3.3 2B | 44.4 | 18 / 18 |
| 160 | Hermes-3-Llama-3.1-8B | 44.4 | 18 / 18 |
| 160 | Lfm3 VL 2B | 44.4 | 18 / 18 |
| 160 | Magistral Small (2506) | 44.4 | 18 / 18 |
| 160 | Olmo-3-7B-Instruct | 44.4 | 18 / 18 |
| 160 | YI Coder 9B Chatcode model | 44.4 | 18 / 18 |
| 167 | Codeqwen 7B Chat V1.5code model | 38.9 | 18 / 18 |
| 167 | Granite4 Micro H | 38.9 | 18 / 18 |
| 167 | Lfm2.5 Thinking 1.2B | 38.9 | 18 / 18 |
| 167 | Llama3.2 1B Instruct | 38.9 | 18 / 18 |
| 167 | Qwen2.5 0.5B Instruct | 38.9 | 18 / 18 |
| 167 | Smallthinker 3B Previewfine-tune of Qwen 2.5 3B | 38.9 | 18 / 18 |
| 167 | Solar 10.7B Instruct V1 | 38.9 | 18 / 18 |
| 167 | YI 34B Chat V1.5 | 38.9 | 18 / 18 |
| 175 | Deepseek V2 16B Lite Chat | 33.3 | 18 / 18 |
| 175 | Granite3 Dense 2B Instruct | 33.3 | 18 / 18 |
| 175 | Granite3 Dense 8B Instruct | 33.3 | 18 / 18 |
| 175 | Granite3.1 Moe 1B Instruct | 33.3 | 18 / 18 |
| 175 | Lfm3 1.7B | 33.3 | 18 / 18 |
| 175 | Olmo2 13B 1124 Instruct | 33.3 | 18 / 18 |
| 175 | Qwen2 Math 7B Instructmath model | 33.3 | 18 / 18 |
| 175 | Qwen2.5 Coder 0.5B Instructcode model | 33.3 | 18 / 18 |
| 175 | Qwen3 VL 2B Thinking | 33.3 | 18 / 18 |
| 175 | Smollm2 1.7B Instruct | 33.3 | 18 / 18 |
| 175 | YI 6B Chat V1.5 | 33.3 | 18 / 18 |
| 186 | Deepseek Llm 7B Chat | 27.8 | 18 / 18 |
| 186 | Gemma 2B Instruct V1.1 | 27.8 | 18 / 18 |
| 186 | Granite Code 8B Instructcode model | 27.8 | 18 / 18 |
| 186 | Granite3 Moe 3B Instruct | 27.8 | 18 / 18 |
| 186 | Llama2 13B Chat | 27.8 | 18 / 18 |
| 186 | Olmo2 7B 1124 Instruct | 27.8 | 18 / 18 |
| 186 | Qwen2 1.5B Instruct | 27.8 | 18 / 18 |
| 193 | Deepscaler 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bmath model | 22.2 | 18 / 18 |
| 193 | Deepseek R1 1.5B Qwen Distill | 22.2 | 18 / 18 |
| 193 | Gemma3 1B IT | 22.2 | 18 / 18 |
| 193 | Qwen2 0.5B Instruct | 22.2 | 18 / 18 |
| 193 | Stablelm Zephyr 3Bfine-tune of StableLM 3B | 22.2 | 18 / 18 |
| 193 | YI 9B Chat V1.5 | 22.2 | 18 / 18 |
| 199 | Deepcoder 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bcode model | 16.7 | 18 / 18 |
| 199 | Granite Code 3B Instructcode model | 16.7 | 18 / 18 |
| 199 | Granite4 350M H | 16.7 | 18 / 18 |
| 199 | Llama-2-7b-chat-hf | 16.7 | 18 / 18 |
| 199 | Sailor2 1B Chatfine-tune of Qwen 2.5 | 16.7 | 18 / 18 |
| 204 | Deepseek Coder 1.3B Instructcode model | 11.1 | 18 / 18 |
| 204 | Falcon3 1B Instruct | 11.1 | 18 / 18 |
| 204 | Granite3 Moe 1B Instruct | 11.1 | 18 / 18 |
| 204 | Nemotron Mini 4B Instruct | 11.1 | 18 / 18 |
| 204 | Opencoder 1.5B Instructcode model | 11.1 | 18 / 18 |
| 204 | Phi-3-mini-4k-instruct | 11.1 | 18 / 18 |
| 204 | Qwen 14B Chat V1.5 | 11.1 | 18 / 18 |
| 204 | Qwen 7B Chat V1.5 | 11.1 | 18 / 18 |
| 204 | Qwen2 Math 1.5B Instructmath model | 11.1 | 18 / 18 |
| 204 | Qwen3 0.6B | 11.1 | 18 / 18 |
| 204 | Smollm2 360M Instruct | 11.1 | 18 / 18 |
| 215 | Codegemma 7B Instruct V1.1code model | 5.6 | 18 / 18 |
| 215 | Dolphin Phi 2.7B V2.6fine-tune of Phi-2 | 5.6 | 18 / 18 |
| 215 | Smollm2 135M Instruct | 5.6 | 18 / 18 |
| 215 | Stable Code 3B Instructcode model | 5.6 | 18 / 18 |
| 215 | Stablelm2 12B Chat | 5.6 | 18 / 18 |
| 215 | YI Coder 1.5B Chatcode model | 5.6 | 18 / 18 |
| 221 | Codegemma 2B Code V1.1code model | 0 | 18 / 18 |
| 221 | Deepseek Coder 33B Instructcode model | 0 | 18 / 18 |
| 221 | Phi 2.7B Chat V2 | 0 | 18 / 18 |
| 221 | Qwen 0.5B Chat V1.5 | 0 | 18 / 18 |
| 221 | Qwen 1.8B Chat V1.5 | 0 | 18 / 18 |
| 221 | Qwen 4B Chat V1.5 | 0 | 18 / 18 |
| 221 | Stablelm2 1.6B Chat | 0 | 18 / 18 |
A dash means not scored, which is different from 0. Agreement compares the two judges: high means a mean gap under 1.5 points, medium under 3, low otherwise; n/a means fewer than 3 items were scored by both.
Not ranked in this category
- Apertus-8B-Instruct-2509: withheld from the ranking (see its page)
- Falcon2 11B: partial: 12/18 scored, 80% needed
- Falcon3 10B Instruct: withheld from the ranking (see its page)
- Falcon3 1B Instruct: withheld from the ranking (see its page)
- Falcon3 3B Instruct: withheld from the ranking (see its page)
- Falcon3 7B Instruct: withheld from the ranking (see its page)
- Internlm2 1.8B Chat V2.5: withheld from the ranking (see its page)
- Phi-4-reasoning 14B: withheld from the ranking (see its page)
- Phi-4-reasoning-plus 14B: withheld from the ranking (see its page)
- Qwen2 Math 1.5B Instruct: withheld from the ranking (see its page)
- Qwen2 Math 7B Instruct: withheld from the ranking (see its page)
The questions
In run r20261102-bv1-5246ab, 0 of the 18 questions in this category were answered correctly by every ranked model that has a score for them (33 ranked models in that run). Hard questions are the ones that half or fewer of the ranked models of 3B and over answered correctly. This category has 2.
The same ranking as JSON: long_context.json.