Trust / Errata
Errata
A dated log of what we got wrong and what we did about it: results we withheld because the way a model was served, not the model, broke them, and site wording or labels we corrected. Each line is dated, says what was wrong and, for a result, links the run that replaced it.
Withheld results
A withheld model keeps its page and its stored answers, but gets no rank, no score on the leaderboard and no place in any figure. A model whose every version is withheld is the exception to the stored answers: its page gives the reason and the run only. Each entry below is one run where a model was withheld, with the reason as printed on the leaderboard. 11 of the 13 entries have a replacement run.
2026-11-10 · Phi-4-reasoning-plus 14B · run
r20261105-bv1-2abe78Withheld: in our Ollama build, Phi-4-reasoning-plus's reasoning came back mixed into the answer instead of as separate thinking, so the thinking budget never applied and its answers were cut off before a final answer (a serving problem, not the model). The run was stopped after 50 of 270 answers; it will be re-run once the serving setup is fixed.
No replacement run is published yet.
2026-11-10 · Phi-4-reasoning 14B · run
r20261105-bv1-2abe78Withheld: in our Ollama build, Phi-4-reasoning's reasoning came back mixed into the answer instead of as separate thinking, so the thinking budget never applied and 259 of 270 answers were cut off before a final answer (a serving problem, not the model). It will be re-run once the serving setup is fixed.
No replacement run is published yet.
2026-10-29 · Apertus-8B-Instruct-2509 · run
r20261029-bv1-3e4075Withheld: on our default GPU backend (ROCm) this model produced no text on 108 of 270 prompts, running to the length limit instead (a serving bug: the same prompts answer normally on the Vulkan backend and on CPU). It was re-run on Vulkan; those results replace these.
Replaced by the results from run
r20261029-bv1-9ddd5f(2026-10-29): the Apertus-8B-Instruct-2509 page.2026-10-29 · Apertus-8B-Instruct-2509 · run
r20261029-bv1-229d1fWithheld: on our default GPU backend (ROCm) this model produced no text on 25 of 270 prompts (all 20 tool-use prompts among them), running to the length limit instead; the same prompts answer normally on the Vulkan backend and on CPU, so this is a serving bug. It was re-run on Vulkan; those results replace these.
Replaced by the results from run
r20261029-bv1-9ddd5f(2026-10-29): the Apertus-8B-Instruct-2509 page.2026-10-28 · Qwen2 Math 7B Instruct · run
r20261016-bv1-473145Withheld: Ollama's library chat template for Qwen2-Math differs from the official one (no newline between turns or after the final <|im_start|>assistant, and no default system turn). It was re-run with the official template; those results replace these (the scores changed very little).
Replaced by the results from run
r20261028-bv1-ef0895(2026-10-28): the Qwen2 Math 7B Instruct page.2026-10-28 · Qwen2 Math 1.5B Instruct · run
r20261016-bv1-473145Withheld: Ollama's library chat template for Qwen2-Math differs from the official one (no newline between turns or after the final <|im_start|>assistant, and no default system turn). It was re-run with the official template; those results replace these (the scores changed very little).
Replaced by the results from run
r20261028-bv1-ef0895(2026-10-28): the Qwen2 Math 1.5B Instruct page.2026-10-28 · Medgemma1.5 4B IT · run
r20261016-bv1-473145Withheld: in this run the model's thinking channel leaked into its answers (241 of 270 answers begin with its internal thought marker, and 127 were cut off, 98 of them while still thinking), because it was treated as a model that answers directly. It was re-run as a reasoning model, with its thought block kept apart from the answer and the same extra reasoning budget as the other reasoning models; those results replace these.
Replaced by the results from run
r20261029-bv1-3e4075(2026-10-29): the Medgemma1.5 4B IT page.2026-10-28 · Internlm2 1.8B Chat V2.5 · run
r20261016-bv1-473145Withheld: Ollama's library chat template for InternLM2 ends the prompt without the newline after <|im_start|>assistant, so on 31 of 270 prompts the model ended its turn after one token and its reply came back empty (a serving bug, not the model). It was re-run with the corrected template; those results replace these.
Replaced by the results from run
r20261028-bv1-2e24cd(2026-10-28): the Internlm2 1.8B Chat V2.5 page.2026-10-28 · Falcon3 7B Instruct · run
r20261016-bv1-473145Withheld: Ollama's library chat template for Falcon3 leaves out the newline after the last message, so the model sometimes wrote <|assistant|> itself and its reply came back empty (6 of 270 answers here; a serving bug, not the model). It was re-run with the corrected template; those results replace these.
Replaced by the results from run
r20261028-bv1-7ce712(2026-10-28): the Falcon3 7B Instruct page.2026-10-28 · Falcon3 3B Instruct · run
r20261016-bv1-473145Withheld: Ollama's library chat template for Falcon3 leaves out the newline after the last message, so the model sometimes wrote <|assistant|> itself and its reply came back empty (169 of 270 answers here; a serving bug, not the model). It was re-run with the corrected template; those results replace these.
Replaced by the results from run
r20261028-bv1-7ce712(2026-10-28): the Falcon3 3B Instruct page.2026-10-28 · Falcon3 1B Instruct · run
r20261016-bv1-473145Withheld: Ollama's library chat template for Falcon3 leaves out the newline after the last message, so the model sometimes wrote <|assistant|> itself and its reply came back empty (42 of 270 answers here; a serving bug, not the model). It was re-run with the corrected template; those results replace these.
Replaced by the results from run
r20261028-bv1-7ce712(2026-10-28): the Falcon3 1B Instruct page.2026-10-28 · Falcon3 10B Instruct · run
r20261016-bv1-473145Withheld: Ollama's library chat template for Falcon3 leaves out the newline after the last message, so the model sometimes wrote <|assistant|> itself and its reply came back empty (16 of 270 answers here; a serving bug, not the model). It was re-run with the corrected template; those results replace these.
Replaced by the results from run
r20261028-bv1-7ce712(2026-10-28): the Falcon3 10B Instruct page.2026-10-12 · Apertus-8B-Instruct-2509 · run
r20261006-bv1-53e146Withheld: the model stopped after one token on 212 of 270 prompts in our Ollama import, which points to a broken chat template rather than the model. It was re-run with a chat template written from the official one, on the Vulkan backend; those results replace these.
Replaced by the results from run
r20261029-bv1-9ddd5f(2026-10-29): the Apertus-8B-Instruct-2509 page.
Site wording and labels
2026-11-05 · Coding answers with a stray indent scored 0
Coding scorer 2.7: an answer written without a code fence and with a stray indent (a space before the first line, often with the whole answer shifted right; seen mostly from Mistral-family models) failed to parse and scored 0, even when the code itself was right. Such an answer is now dedented before testing; nothing else in it is changed. Affected published models: Mathstral 7B (8 coding answers) and Llama 2 13B (5), whose coding answers were re-scored. Models first published on 5 November 2026 were scored with the corrected rule.
2026-11-02 · How judge scores affect Safety
Earlier wording on the home page, in the machine-readable summaries and in the plain-text site map said judge scores are never blended into Overall. That was incomplete. Judges affect the Safety score, which is one of the Overall categories, in two ways: they decide the refusal items the automatic check cannot settle, and an answer to a harmless prompt that both judges rate 3 or lower out of 10 is scored 0 unless it was cut off. Every other judge score stays advisory, and judge means are never added into Overall. The pages now say so.
2026-11-02 · Licence labels, round two
Stable Code 3B Instruct and StableLM Zephyr 3B are now labelled Stability AI Community License (research and non-commercial use free; commercial use free below US$1M annual revenue, with registration), as on their upstream Hugging Face repositories; the Ollama copy still ships the older non-commercial text. The label is a bucket of its own on the explorer and counts as allowed with conditions.
2026-11-02 · Licence labels, round one
The DeepSeek R1 8B Llama distill is now labelled Llama 3.1 (it derives from Llama 3.1 8B Base), both InternLM2 models are labelled other (their Hugging Face card says licence other, commercial use by application) and Llama 4 Scout is labelled Llama 4. The 14 models whose own licence text limits use to research or non-commercial use now carry labels that say so, and the explorer puts them in their own bucket with commercial use not allowed. Check a model's own licence before you rely on a label here.
2026-11-01 · Headline counts counted versions as models
The leaderboard has one row per model version, so a model whose older version is withheld was counted twice in the headline numbers. The home page, the share card and the findings now count each model once and say what they count. No score or rank moved, and withheld versions stay listed.
Report an error
If something here or elsewhere on the site looks wrong, write to [email protected]. The reproduce page shows how to ask a model a public question again and check the answer against ours.