Trust / Gotcha questions

Gotcha questions

A gotcha is a well-known puzzle or trap with one right answer that a model can easily get wrong by pattern-matching. The battery holds 15 of them. 3 are public and shown below; the other 12 stay private and appear by id only.

Who fell for which

One row per model, best first. A green check is a correct answer, a red cross is a wrong one, a dash is no usable score. The bottom row counts the models that got each question wrong.
Modelgotchas-001gotchas-002gotchas-003gotchas-004gotchas-005gotchas-006gotchas-007gotchas-008gotchas-009gotchas-010gotchas-011gotchas-012gotchas-013gotchas-014gotchas-015
gemma-4-26B-A4B-it ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓
gpt-oss-120b ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓
gpt-oss-20b ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓
Muse Glimmer 30B ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓
Nemotron 3.5 Lightning 30B A3B ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓
Olmo 3.1 32B Instruct ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓
Ornith-1.5-9B ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓
Qwen3-4B ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓
Qwen3-Coder-Next ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓
QwQ 32B ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓
DeepSeek-R1-Distill-Qwen-14B ✗✓✓✓✓✓✓✓✓✓✓✓✓✓✓
DeepSeek-R1-Distill-Qwen-32B ✓✓✓✗✓✓✓✓✓✓✓✓✓✓✓
Exaone Deep 2.4B ✓✓✗✓✓✓✓✓✓✓✓✓✓✓✓
Exaone Deep 7.8B ✓✓✓✓✓✓✗✓✓✓✓✓✓✓✓
Gemma 4 12B IT ✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓
gemma-3-27b-it ✗✓✓✓✓✓✓✓✓✓✓✓✓✓✓
gemma-4-31B-it ✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓
Gemma4 E2B IT ✓✓✗✓✓✓✓✓✓✓✓✓✓✓✓
Gemma4 E4B IT ✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓
GLM-4.5-Air ✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓
Granite4.2 30B ✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓
Laguna XS 2.1 ✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓
Lfm3 1.7B ✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓
Llama 4 Scout 17B-16E Instruct ✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓
Llama-3.3-70B-Instruct ✓✓✓✗✓✓✓✓✓✓✓✓✓✓✓
Mistral Small 3.2 24B Instruct (2506) ✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓
Nemotron 3.5 Bolt E4B ✓✓✓✓✓✓✓✓✗✓✓✓✓✓✓
NVIDIA Nemotron 3 Nano 30B-A3B ✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓
Olmo-3-7B-Instruct ✓✓✓✓✓✓✓✓✗✓✓✓✓✓✓
Ornith 9B ✓✓✓✓✓✗✓✓✓✓✓✓✓✓✓
Phi4 Mini Reasoning 3.8B ✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓
Qwen3-30B-A3B ✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓
Qwen3-4B-Instruct-2507 ✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓
Qwen3-4B-Thinking-2507 ✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓
Qwen3-VL-32B-Instruct ✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓
Qwen3-VL-8B-Instruct ✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓
Qwen3.5-27B ✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓
Qwen3.5-35B-A3B ✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓
Qwen3.5-9B ✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓
Qwen3.6-27B ✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓
Qwen3.6-35B-A3B ✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓
Qwen3.8-27B ✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓
Sage-A1.5-Q4_K_M ✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓
Deepseek R1 8B Llama Distill ✗✓✗✓✓✓✓✓✓✓✓✓✓✓✓
DeepSeek-R1-0528-Qwen3-8B ✓✗✓✓✓✓✓✓✓✗✓✓✓✓✓
Falcon3 10B Instruct ✓✓✓✓✓✓✗✓✓✗✓✓✓✓✓
Flint 40B A2B ✓✓✓✓✓✓✓✓✗✗✓✓✓✓✓
Granite4.1 30B ✗✓✓✓✗✓✓✓✓✓✓✓✓✓✓
Granite4.2 3B ✓✓✓✓✓✓✓✓✗✗✓✓✓✓✓
Medgemma 27B IT ✓✓✓✓✓✓✗✓✓✗✓✓✓✓✓
Ministral 3 14B Instruct (2512) ✓✓✓✓✓✓✓✓✗✗✓✓✓✓✓
Mistral Nemo 12B Instruct 2407 ✓✓✓✓✓✓✗✓✓✗✓✓✓✓✓
Mistral Small 24B Instruct 2501 ✓✓✗✓✗✓✓✓✓✓✓✓✓✓✓
Nemotron3 33B ✓✓✓✓✓✓✗✓✓✗✓✓✓✓✓
Openthinker 7B V2 ✓✗✓✓✓✓✓✓✓✓✓✓✓✓✗
phi-4 ✓✓✓✓✓✓✗✓✓✗✓✓✓✓✓
Qwen2.5-14B-Instruct ✗✗✓✓✓✓✓✓✓✓✓✓✓✓✓
Qwen2.5-VL-32B-Instruct ✓✓✓✓✗✓✓✓✓✗✓✓✓✓✓
Qwen3-30B-A3B-Instruct-2507 ✓✓✓✓✓✗✓✓✓✗✓✓✓✓✓
Qwen3-32B ✓✓✓✓✓✗✓✓✓✗✓✓✓✓✓
Qwen3-8B ✓✓✓✓✓✗✓✓✓✗✓✓✓✓✓
Qwen3-VL-4B-Instruct ✓✓✓✓✓✓✓✓✗✗✓✓✓✓✓
Qwen3.5-4B ✓✓✓✓✓✗✓✓✓✗✓✓✓✓✓
Translategemma 12B IT ✓✓✓✓✓✓✗✓✓✗✓✓✓✓✓
Aya Expanse 8B ✓✓✓✓✓✓✗✓✗✗✓✓✓✓✓
Cogito 14B V1 Preview Qwen ✗✓✓✓✓✓✗✓✓✗✓✓✓✓✓
Command R7B (12-2024) ✗✓✓✓✓✗✗✓✓✓✓✓✓✓✓
Deepcoder 1.5B Preview ✓✓✗✓✓✓✗✓✓✗✓✓✓✓✓
DeepSeek-R1-Distill-Qwen-7B ✓✗✗✓✓✓✓✓✓✗✓✓✓✓✓
Devstral 24B Small 2505 ✗✓✓✓✗✓✓✓✓✗✓✓✓✓✓
Devstral Small 2 24B Instruct (2512) ✓✓✓✓✗✓✗✓✓✗✓✓✓✓✓
EXAONE-3.5-7.8B-Instruct ✗✓✗✓✓✓✗✓✓✓✓✓✓✓✓
Exaone3.5 32B Instruct ✓✓✓✓✓✓✗✓✗✗✓✓✓✓✓
Falcon3 7B Instruct ✗✓✓✓✓✓✗✓✓✗✓✓✓✓✓
Gemma2 27B Instruct ✗✓✓✓✗✓✗✓✓✓✓✓✓✓✓
Granite4 3B ✗✓✓✓✓✓✓✓✗✗✓✓✓✓✓
Granite4 Small H ✗✓✗✓✓✓✓✓✓✗✓✓✓✓✓
Granite4.1 8B ✗✓✗✓✓✓✗✓✓✓✓✓✓✓✓
Lfm2 24B ✓✓✗✓✓✗✗✓✓✓✓✓✓✓✓
Lfm2.5 8B A1B ✓✓✗✓✓✓✓✓✗✗✓✓✓✓✓
Lfm2.5 Thinking 1.2B ✓✓✓✗✗✓✗✓✓✓✓✓✓✓✓
Lfm3 VL 2B ✓✓✗✓✓✓✓✓✗✗✓✓✓✓✓
Ministral 3 8B Instruct 2512 ✓✓✓✓✓✗✓✓✗✗✓✓✓✓✓
Nanbeige_Nanbeige4.3-33B ✓✓✗✓✓✗✓✓✓✗✓✓✓✓✓
Nemotron Cascade 2 30B A3B ✓✓✓✓✓✓✗✓✗✗✓✓✓✓✓
Olmo-3-7B-Think ✓✓✓✓✓✗✗✓✓✗✓✓✓✓✓
Phi-3.5-mini-instruct ✗✓✓✓✓✓✗✓✓✓✓✓✗✓✓
Phi-4-mini-instruct ✗✓✓✓✓✓✗✓✓✗✓✓✓✓✓
Qwen2.5-32B-Instruct ✗✓✓✓✗✗✓✓✓✓✓✓✓✓✓
Qwen2.5-Coder-32B-Instruct ✓✓✗✓✓✓✓✓✗✗✓✓✓✓✓
Qwen2.5-Coder-7B-Instruct ✓✓✓✓✓✓✗✓✗✗✓✓✓✓✓
Qwen3-14B ✓✓✓✓✓✗✗✓✓✗✓✓✓✓✓
Qwen3-Coder-30B-A3B-Instruct ✓✓✗✓✓✗✓✓✓✗✓✓✓✓✓
Qwen3-VL-30B-A3B-Instruct ✓✓✗✓✓✗✓✓✓✗✓✓✓✓✓
YI 34B Chat V1.5 ✓✓✗✓✗✓✗✓✓✓✓✓✓✓✓
Cogito 8B V1 Preview Llama ✓✓✗✓✓✓✗✓✗✓✓✗✓✓✓
Deepseek R1 1.5B Qwen Distill ✓✗✗✓✓✓✗✓✓✗✓✓✓✓✓
Gemma 3n E4B IT ✓✓✗✓✓✓✗✓✗✗✓✓✓✓✓
gemma-2-9b-it ✓✓✓✓✗✓✗✓✗✗✓✓✓✓✓
gemma-3-4b-it ✓✓✗✓✓✓✗✓✗✗✓✓✓✓✓
Glm4 9B Chat ✗✓✓✓✓✓✗✓✓✗✓✓✓✓✗
granite-4.1-3b ✗✓✓✓✓✓✗✓✗✗✓✓✓✓✓
Granite3 Dense 8B Instruct ✗✓✓✓✗✓✗✓✗✓✓✓✓✓✓
Granite3.2 8B Instruct ✗✓✗✓✓✓✗✓✓✗✓✓✓✓✓
Granite3.3 8B ✗✓✓✓✓✗✗✓✓✗✓✓✓✓✓
Granite4 Micro H ✗✓✓✓✓✗✗✓✗✓✓✓✓✓✓
Medgemma1.5 4B IT ✓✓✓✓✓✓✗✓✗✗✓✓✓✓✗
Mixtral 8x7B Instruct V0.1 ✓✓✓✓✗✓✗✓✓✗✗✓✓✓✓
Phi3 14B Medium 128K Instruct ✗✓✓✓✓✓✗✓✗✗✓✓✓✓✓
Qwen2.5-Coder-14B-Instruct ✗✓✓✓✓✓✗✓✗✗✓✓✓✓✓
Qwen3 1.7B ✗✓✓✗✓✓✗✓✓✗✓✓✓✓✓
Spark-X2.6-4B ✓✓✓✓✗✓✗✓✗✓✓✓✓✓✗
Aya Expanse 32B ✗✓✗✓✓✓✗✓✗✗✓✓✓✓✓
Codellama 13B Instruct ✓✓✓✓✗✓✓✓✗✗✗✗✓✓✓
Command R7B Arabic 7B 02 2025 ✗✓✓✓✗✓✓✓✗✓✓✓✓✗✗
Deepscaler 1.5B Preview ✓✓✗✓✗✗✗✓✓✗✓✓✓✓✓
dolphin-2.9.1-yi-1.5-34b ✗✗✗✓✗✓✗✓✓✓✓✓✓✓✓
Exaone3.5 2.4B Instruct ✓✓✗✓✗✓✗✓✓✗✓✓✓✗✓
Falcon3 3B Instruct ✗✓✓✓✗✓✗✓✓✗✓✓✗✓✓
gemma-3-12b-it ✗✗✗✓✓✓✗✓✓✗✓✓✓✓✓
Gemma2 2B Instruct ✗✗✓✓✓✓✗✓✗✗✓✓✓✓✓
GLM-4.7-Flash ✓✓✗✓✓✗✗✓✗✗✓✓✓✓✓
Granite Code 8B Instruct ✓✓✓✓✗✓✓✓✓✗✓✗✓✗✗
Granite3.1 Dense 8B Instruct ✗✓✗✓✗✓✗✓✓✗✓✓✓✓✓
Granite3.1 Moe 3B Instruct ✓✓✗✓✗✓✓✗✓✓✓✓✗✓✗
Granite4.2 8B ✗✗✗✓✓✗✓✓✓✗✓✓✓✓✓
Hermes-3-Llama-3.1-8B ✓✓✗✓✗✓✗✓✗✗✓✓✓✓✓
Llama-3.1-8B-Instruct ✓✓✓✓✗✗✗✓✗✗✓✓✓✓✓
Marco O1 7B ✗✓✗✓✗✓✓✗✗✓✓✓✓✓✓
medgemma-4b-it ✗✓✗✓✓✓✗✗✗✓✓✓✓✓✓
Mistral Small 22B Instruct 2409 ✓✓✗✓✓✓✗✓✗✓✓✓✓✗✗
Mistral Small3.1 24B Instruct 2503 ✗✓✗✓✗✓✗✓✓✗✓✓✓✓✓
Qwen 7B Chat V1.5 ✓✓✗✓✓✓✓✓✗✓✓✗✗✗✓
Qwen2.5-7B-Instruct ✓✓✗✓✓✓✗✓✗✗✓✓✗✓✓
Qwen3.5 2B ✓✗✓✓✗✓✗✓✗✗✓✓✓✓✓
Sailor2 8B Chat ✗✗✓✓✓✓✗✓✗✗✓✓✓✓✓
Codestral 22B V0.1 ✗✓✓✓✗✓✗✗✗✗✓✓✓✓✓
Deepcoder 14B Preview ✗✓✗✗✗✓✓✓✗✗✓✓✓✓✓
Gemma 7B Instruct V1.1 ✓✗✓✓✗✓✗✗✓✓✓✗✗✓✓
Granite4 Tiny H ✗✓✗✓✓✗✗✓✗✗✓✓✓✓✓
Llama-3.2-3B-Instruct ✗✓✓✗✗✓✗✓✗✓✓✓✓✓✗
Mathstral 7B V0.1 ✗✓✓✓✗✓✗✗✗✗✓✓✓✓✓
Meta-Llama-3-8B-Instruct ✗✓✗✓✗✓✗✓✗✗✓✓✓✓✓
Mistral-7B-Instruct-v0.2 ✓✓✓✓✗✗✗✓✗✓✓✗✗✓✓
Phi-3-mini-4k-instruct ✗✓✓✓✗✓✗✓✗✗✓✓✗✓✓
Qwen 14B Chat V1.5 ✗✓✗✓✗✗✗✓✗✓✓✓✓✓✓
Qwen2 1.5B Instruct ✓✓✗✓✓✓✗✗✗✓✓✓✓✗✗
Qwen2 Math 7B Instruct ✗✗✓✓✗✓✗✓✗✓✓✓✓✗✓
Qwen2-VL-7B-Instruct ✗✓✗✓✗✓✗✓✓✓✓✗✗✓✓
Qwen2.5 1.5B Instruct ✗✓✗✓✓✓✗✗✗✗✓✓✓✓✓
Qwen2.5-Coder-3B-Instruct ✓✓✗✓✓✓✗✗✗✗✓✓✓✓✗
Qwen2.5-VL-3B-Instruct ✗✓✗✓✗✓✗✓✗✓✓✓✗✓✓
Qwen2.5-VL-7B-Instruct ✗✓✗✓✗✓✗✓✗✗✓✓✓✓✓
Qwen3 VL 2B Instruct ✓✗✓✓✗✗✗✓✓✗✓✓✓✓✗
Smallthinker 3B Preview ✓✗✗✓✗✗✓✗✓✓✓✓✓✓✗
SmolLM3-3B ✗✓✗✓✗✓✗✓✓✗✓✓✓✓✗
YI 6B Chat V1.5 ✗✓✗✓✗✓✓✓✗✓✗✓✗✓✓
YI 9B Chat V1.5 ✗✓✗✓✗✓✗✓✓✗✓✓✗✓✓
Cogito 3B V1 Preview Llama ✗✓✗✗✗✓✗✓✗✓✓✓✓✗✓
Deepseek Coder 33B Instruct ✗✓✗✓✓✗✓✗✗✓✗✓✓✗✓
Dolphin Phi 2.7B V2.6 ✓✓✓✓✗✓✗✗✗✗✓✗✓✗✓
Gemma3 1B IT ✗✓✓✓✓✓✗✗✗✗✓✗✗✓✓
Gemma3n E2B IT ✗✗✗✓✓✓✗✓✓✗✓✗✓✓✗
Granite3 Moe 3B Instruct ✗✓✗✓✗✓✓✗✓✗✓✗✗✓✓
Granite4 1B H ✗✓✗✗✓✓✗✓✗✗✓✓✗✓✓
Llama2 13B Chat ✓✓✓✗✗✓✗✗✗✓✓✗✓✗✓
Ministral 3 3B Instruct 2512 ✗✗✗✓✗✗✓✓✗✗✓✓✓✓✓
Mistral 7B Instruct V0.3 ✗✓✗✓✗✓✗✓✗✗✓✓✗✓✓
Olmo2 13B 1124 Instruct ✗✓✗✓✓✓✗✓✗✓✓✓✗✗✗
Phi3 3.8B Mini 128K Instruct ✗✓✗✓✓✗✗✓✗✗✓✓✗✓✓
Qwen2.5 Coder 1.5B Instruct ✗✓✗✓✓✓✗✗✗✗✓✓✓✓✗
Qwen2.5-3B-Instruct ✗✗✗✓✗✓✗✓✗✗✓✓✓✓✓
Qwen3 VL 2B Thinking ✗✓✓✓✗✗✗✓✗✗✓✓✓✗✓
YI Coder 9B Chat ✗✗✗✓✗✓✗✓✓✓✓✗✓✓✗
Apertus-8B-Instruct-2509 ✗✗✗✓✗✓✗✓✗✗✓✓✓✗✓
Codegemma 7B Instruct V1.1 ✗✗✗✓✗✓✗✗✓✓✓✗✓✓✗
Command R 35B 08 2024 ✗✓✗✓✗✓✗✓✗✗✓✓✓✗✗
DeepSeek-Coder-V2-Lite-Instruct ✗✗✗✓✗✓✗✓✓✗✓✓✗✗✓
Granite3.2 2B Instruct ✓✓✗✓✓✓✗✗✗✗✓✓✗✗✗
Granite3.3 2B ✗✗✗✓✗✓✗✗✓✗✓✓✓✗✓
Internlm2 7B Chat V2.5 ✗✓✓✗✗✓✗✗✗✓✓✓✓✗✗
Nemotron Mini 4B Instruct ✓✓✗✓✗✓✗✗✓✓✓✗✗✗✗
Olmo2 7B 1124 Instruct ✗✗✗✓✓✗✗✗✗✓✓✓✓✗✓
Smollm2 360M Instruct ✓✗✗✓✗✓✗✓✗✓✓✗✗✗✓
Tulu3 8B ✗✓✗✗✗✓✗✓✓✗✓✓✓✗✗
Aya 8B 23 ✓✗✗✓✗✓✗✓✗✓✓✗✗✗✗
Codellama 7B Instruct ✗✓✗✗✗✓✗✓✗✗✓✗✓✗✓
Codeqwen 7B Chat V1.5 ✗✓✓✗✗✓✗✗✓✗✗✗✓✓✗
Deepseek Llm 7B Chat ✗✓✗✓✗✓✗✗✓✓✓✗✗✗✗
Deepseek V2 16B Lite Chat ✗✓✗✓✗✓✗✗✓✗✓✗✓✗✗
Granite Code 3B Instruct ✓✓✗✓✗✓✗✗✓✗✓✗✗✗✗
Granite3 Dense 2B Instruct ✗✓✗✓✗✗✗✓✓✗✓✗✗✗✓
Granite3.1 Dense 2B Instruct ✓✗✗✓✓✓✗✗✗✗✓✓✗✗✗
Granite3.1 Moe 1B Instruct ✗✗✓✓✗✓✗✓✗✗✓✗✗✓✗
Phi 2.7B Chat V2 ✗✓✓✗✗✗✗✓✓✓✗✗✗✗✓
Qwen 1.8B Chat V1.5 ✗✗✗✓✓✓✓✗✓✓✗✗✗✗✗
Qwen2 Math 1.5B Instruct ✗✗✗✓✓✓✗✗✗✗✓✓✓✗✗
Qwen2.5 0.5B Instruct ✗✗✗✓✗✓✗✗✓✗✓✓✗✓✗
Smollm2 1.7B Instruct ✓✗✗✗✓✓✗✗✓✗✗✗✓✗✓
Stable Code 3B Instruct ✗✓✗✓✗✗✗✗✓✓✗✗✗✓✓
Codegeex4 9B All ✗✓✗✗✗✓✗✓✗✗✗✓✓✗✗
Codellama 34B Instruct ✗✓✗✓✗✓✗✗✗✓✓✗✗✗✗
Deepseek Coder 6.7B Instruct ✗✓✓✓✗✓✓✗✗✗✗✗✗✗✗
Falcon2 11B ✗✗✓✗✗✗✗✗✓✓✓✗✓✗✗
Falcon3 1B Instruct ✗✗✗✓✗✓✗✗✗✗✓✗✓✓✗
Granite3 Moe 1B Instruct ✗✗✗✓✓✗✓✗✓✗✓✗✗✗✗
Hermes3 3B Llama3.2 ✓✗✗✗✗✓✗✓✗✓✓✗✗✗✗
Internlm2 1.8B Chat V2.5 ✗✗✗✓✓✗✗✓✗✓✓✗✗✗✗
Llama-2-7b-chat-hf ✗✗✗✓✗✓✗✗✗✗✗✗✓✓✓
Qwen2.5 Coder 0.5B Instruct ✗✓✗✓✗✓✗✗✗✗✓✗✗✓✗
Qwen3 0.6B ✗✗✗✓✗✓✗✗✗✓✓✗✓✗✗
Solar 10.7B Instruct V1 ✗✓✗✓✗✗✗✗✓✓✓✗✗✗✗
Stablelm2 12B Chat ✓✓✗✓✗✓✗✗✓✗✗✗✗✗✗
YI Coder 1.5B Chat ✗✓✗✓✗✓✗✗✓✓✗✗✗✗✗
Gemma 2B Instruct V1.1 ✗✗✓✓✗✓✗✗✗✗✗✗✗✗✓
Magistral Small (2506) ✗✗✓✓✓✗✗✓✗✗✗✗✗✗✗
Opencoder 1.5B Instruct ✓✓✓✗✗✗✗✓✗✗✗✗✗✗✗
Qwen 0.5B Chat V1.5 ✓✗✓✓✗✗✓✗✗✗✗✗✗✗✗
Qwen2 0.5B Instruct ✗✗✗✗✗✓✓✓✗✗✗✗✓✗✗
Granite4 350M H ✗✓✗✗✗✗✗✗✗✗✗✓✗✗✓
Qwen 4B Chat V1.5 ✗✗✗✓✗✓✗✗✗✓✗✗✗✗✗
Stablelm Zephyr 3B ✗✗✗✗✗✓✗✗✗✗✓✗✗✓✗
Llama3.2 1B Instruct ✗✗✗✗✗✗✗✗✓✗✓✗✗✗✗
Stablelm2 1.6B Chat ✗✗✗✗✗✗✗✗✓✗✗✗✗✓✗
Codegemma 2B Code V1.1 ✗✗✗✗✗✗✗✗✓✗✗✗✗✗✗
Deepseek Coder 1.3B Instruct ✗✗✗✗✗✗✓✗✗✗✗✗✗✗✗
Sailor2 1B Chat ✗✗✗✗✗✓✗✗✗✗✗✗✗✗✗
Smollm2 135M Instruct ✗✗✗✗✗✗✓✗✗✗✗✗✗✗✗
Fell for it111/22851/228108/22828/22899/22846/228139/22854/228100/228150/22826/22854/22854/22857/22859/228

The 15 questions

Withheld

These models are left out of every figure on this page. Why each was withheld is on the leaderboard.