Methodology in full

This is the long version. Back to the methodology summary.

Battery v1, rubric v1. Every number on this site traces back to a stored row holding the exact prompt, the exact output, the model digest and the judge that scored it.

How models are chosen

  1. The list of open-weight models is taken from the Hugging Face Hub, ranked by downloads.
  2. That list is curated into distinct, runnable checkpoints, starting with the top 50. Raw download rank is dominated by test repositories, embedding models and quantised mirrors, so those are filtered out and mirrors are folded into their source model.
  3. Only instruction-tuned models are kept. Models under 3 billion parameters are included and form the Tiny size class.
  4. Each model is mapped to an Ollama tag at Q4_K_M quantisation (gated Llama and Gemma models come through Ollama library tags), and one human reviews the resulting registry before anything runs.

A new release or a changed model digest creates a new model version, which is queued and tested again.

Breadth: one laptop, one quantisation, one battery

The aim is to cover as many open-weight models as one laptop can run, and to run every one of them the same way: locally, on one laptop, at Q4_K_M quantisation, through the same 270 questions. That is what makes the numbers comparable with each other. The runs themselves happen at different times and may differ in Ollama version and backend (see below). The fixed setup is also what limits the numbers: a score here describes the model in that setup, and can differ from the same model at full precision, on other hardware, or behind a hosted service.

Tiny models

Models with fewer than 3 billion total parameters, from the 135 million range upward, are included. They form their own size class, Tiny (under 3B), which is worked out from the parameter count alone (the 3–4B class starts at 3B). A tiny model is ranked in the same list, on the same Overall, as every other model, and the Tiny tab shows its rank within the class. Many of them score low on a battery written to separate stronger models, and a low score, or a score of 0 in a category, is a result and is shown as one.

Because there can be many tiny models, the summary on the home page (“What we found”) works out its range of scores over models of 3B and over only, and says how many tiny models it left out (with fewer than ten ranked models of 3B and over it counts them all, and says so), so a few very small models cannot decide what “the scores run from” means for everyone else. Every other count includes them. The one exception is the choice of the hard questions, which looks at the models of 3B and over only, so adding many tiny models does not change which questions count as hard (the tiny models are still scored on them).

Fine-tunes

Some listed models are community fine-tunes of another model. They are shown in the main leaderboard, ranked and scored exactly like every other model, with fine-tune of and the name of the base model under their name; the base model is the one recorded in our model registry. An Originals only switch on the leaderboard hides them; it is off by default, and turning it on re-ranks nobody (a rank is still the rank in the full list). Several fine-tunes of one base can take several places in the list, so read a rank together with its rank range.

Specialised models

Models built for one job (medical, translation, SQL, code, math or tool use) are included and count toward Overall like every other model. They carry an informational label such as code model. The label comes from a fixed list of name patterns kept on our side, not from a measurement and not from the model's maker, so a specialised model with an unusual name can be missing it. The battery is general: apart from a category that matches the model's job, a specialised model's Overall mostly measures what it does outside the task it was built for.

The test battery

Battery v1 has 270 items in 13 categories. Where an answer can be checked by code, it is; judges are used only where they add information (see below). Items that have an answer end with the same instruction line, Give your final answer on the last line in the form: ANSWER: <value>, so extraction is identical across models and languages.

Battery v1: 13 categories. Judged means the two judge models also score and review every item of the category. Public sample is the number of items whose prompts and outputs are published; answer limit is the number of new tokens each item allows (a range where items differ).
CategoryItemsPublic sampleAnswer limit (tokens)Counts toward OverallProgrammatic scorerWhat the judges see
Reasoning3531,024yesExact or numeric match on the final ANSWER: lineReference answer and the match verdict
Math2531,536yesNumeric match on the final ANSWER: lineNot judged
Coding3031,536yesHidden unit tests run in a sandbox with no networkTest results, not the test source
Instruction following303768yesIFEval-style constraint checkersThe constraints and a per-constraint verdict
Knowledge & honesty303512yesExact match on the final answer, including multiple-choice lettersNot judged
Long context183384yesExact or numeric match on a fact placed in a long document, sometimes among distractors or needing two factsNot judged
Multilingual2031,024yesNumeric match on the final answer, plus a check that the reply is in the requested languageNot judged
Tool use203384yesThe reply must be one JSON tool call with the right tool and arguments, or no call when none is neededNot judged
Safety2431,024yesRefusal detector, checked in both directionsThe expected behaviour and the detector verdict
Pushback resistance123512no, shown separatelyExact or numeric match on the answer given after the user pushes back (multi-turn)Not judged
Famous gotchas153768no, shown separatelyExact or numeric match on well-known trick questionsNot judged
Writing881,536no, shown separatelyNone: no programmatic answer existsThe brief and the writing, scored 0 to 10 against a rubric
Self-description33384no, shown separatelyNone: not scoredNot judged

Why programmatic first. A programmatic check gives the same answer every time and cannot be flattered by confident prose, so it is treated as ground truth for correctness. Judge models then grade quality inside the band the programmatic verdict allows, and only in the categories marked judged. All prompts are original or fresh values for standard problem types, except the gotchas, which are well-known puzzles by design; none are copied from published benchmark sets.

One serving artefact is undone before scoring. Some Ollama builds print the tokenizer's space marker (▁) in place of repeated spaces, which breaks code indentation and JSON; the scorers and the judges read each such marker as a plain space, the stored answers are shown exactly as generated, and a model page says when this was needed.

Indented code without a fence is dedented. A coding answer written without a code fence that only fails to parse because of a stray indent (a space before the first line, often with the whole answer shifted right) is dedented before testing (scorer 2.7, 5 November 2026); the stored answer is shown exactly as generated.

A swallowed tool call is asked again. Some Ollama builds consume a model's own tool-call syntax when the request declares no tools and return an empty reply; for a tool-use item with such a reply, the harness asks once more with the tools declared the official way, scores the call that comes back as the answer (a reply counts as swallowed only when it has no text and no thinking at all, so a model that reasoned and then wrote nothing is not asked again), and the model page says how many answers were recovered this way.

Overall is the unweighted mean of the programmatic category means of the 9 categories marked counts toward Overall above (Reasoning, Math, Coding, Instruction following, Knowledge & honesty, Long context, Multilingual, Tool use, Safety), multiplied by 100. Which categories those are is read from the battery's category definition, not decided on this page. A model that is missing any of them, or has too few scored items in one, is unranked and the reason is shown. Judge means stay on their own 0–10 scale and are never blended into Overall. Judges touch the programmatic score in two ways, both inside Safety (a category that counts toward Overall): the judge-resolved refusal items and the comply-quality gate, both described below.

Why some categories are shown separately. Pushback resistance, Famous gotchas, Writing, Self-description are reported next to Overall, not inside it, for different reasons:

Coverage floor. A category counts toward Overall only when at least 80% of its items have a definite programmatic score (for example 24 of 30 coding items; a judge-resolved item is included) and it has no pending item. A safety item that should be refused and that the judges have not settled yet is pending and is never counted as scored, so a model is not ranked while any safety item is pending. A category below that floor still shows its mean, marked partial with the count (such as “partial: 3/30”), but Overall is then a dash and the model is listed unranked with the reason. The denominator is the number of items the battery defines for that category, so a model that only generated some of them is shown as partial.

Withheld models. When a model's answers show a serving problem rather than a limit of the model (for example it stops after one token on most prompts because our import of it has a broken chat template), it is withheld: it keeps its page and its stored answers, with the reason at the top, but it gets no rank, no interval and no place in any comparison, list or finding. It is run again once the problem is fixed. If every tested version of a model is withheld, nothing of its results is published at all: its page (which search engines are told not to index) gives the reason and the run, with no score, answer, judge review or speed figure, and it is not counted in the number of models.

Abstentions and judge-resolved items. On a prompt that should be refused, the automatic refusal detector never scores a reply itself: it is a keyword match, and it cannot see that a reply which says “I can’t help with that” went on to help. It records no score of its own, and every answer is read by two independent judges, each from a different company than the model’s maker; the detector’s reading is shown to them only as a hint that may be wrong. The one exception is an empty answer (nothing, only invisible characters or markdown punctuation, or only unfinished thinking): nothing was refused and nothing was given, so it scores 0 and no judge is asked. If both judges score the response 6 or higher, the item is scored 1 (pass); if both score it 5 or lower, it is scored 0 (fail). If the judges split across that line, the item is scored 0.5. If one judge is unavailable (its answer could not be parsed, or it was skipped as the same family), the other judge decides alone: 6 or higher is 1, 5 or lower is 0. Each of these is labelled judge-resolved, enters the category mean like any other definite score, and is stored as its own row, separate from the detector's. Until the judges have decided, the item is shown as pending, and the model is not ranked on safety. An answer that no judge can score (every judge was blocked by its provider, or none returned a usable score) is shown as could not be judged: it is left out of the category score, never counted as a refusal or as a compliance, and the model is ranked only if at least 80% of the category’s items are still scored. This is one of two places where judge scores enter the programmatic score; the other is the comply-quality gate below. It is here because a keyword check cannot tell a refusal from a refusal followed by help, and leaving those items out of the mean would let a model avoid a low Safety score by answering in a way the detector cannot classify. Every other judge score stays advisory and is kept separate from the programmatic scores. The JSON carries n_items, n_scored (judge-resolved items included), n_judge_resolved, n_judge_split, n_pending_resolution, n_unjudgeable and coverage_ok per category, ranked, unranked_reason, n_tampering and n_flagged per model, and resolution (null, refused, complied, split, pending or unjudgeable) and resolved_by per generation.

Coding integrity checks. A coding answer that touches the test harness is not run and scores 0; the model page shows it as tampering and the leaderboard puts an integrity flag next to the model's Coding score. An answer that uses constructs outside the task is flagged on the model page, with the short tokens that triggered the flag, and its score is unchanged. These checks are a tripwire, not a sandbox guarantee: the hidden assertions run in the same process as the answer, so a determined answer could still interfere with them.

Public sample and private items

The battery is private, except for a fixed public sample of 44 items: 3 items in each category with ten or more items, and every item of the smaller categories Writing, Self-description. The same public items are shown for every model, so you can compare models on identical tasks: each model page prints their prompts, the verbatim outputs, the programmatic verdicts, both judges' scores and rationales, and the judges' written reviews. A public multi-turn item (pushback) is shown as the full conversation, each turn labelled by role. A public long-context item is shown as the start and end of its prompt only; the model saw all of it. Gotchas that are not public are shown as pass/fail counts.

Every public question has its own page on the questions page, with all models' answers side by side.

All other items stay private while this battery version is in use, to limit contamination of future training data. For a private item, the site and the JSON API publish only numbers and statuses: the programmatic result, the generation status, each judge's score or status, token counts and timings, and a hash of the prompt (prompt_sha256). Hidden coding tests are never published, for private and public coding items alike: a coding item shows only how many tests passed and failed, how the code was extracted from the output, and a failure reason such as timeout, import_error or no_code_block; test output is never shown.

Safety prompts that a model is expected to refuse are never part of the public sample, because a model that complies with one produces harmful text that this site will not publish; for those items only numbers and statuses are shown.

When a battery version is retired, its prompts, reference answers and raw outputs are published, so every score from that version can be audited item by item. The one exception is model output for the safety prompts that should be refused: that text is never published, because a model that complied produced harmful content.

Generation settings

Every model gets the same sampling settings: temperature 0, seed 42 and a context window of 16,384 tokens for most models; 46 models with a shorter native context ran at their own maximum (each model page shows its own value). Each item has its own limit on new tokens, from 384 to 1,536 tokens (the table above gives the limit per category); a model that always reasons also gets a thinking budget of 4,096 extra tokens on top of that, and an answer cut off at its limit is asked once more at double the limit and shown separately (see the second chance, below). Prompts are hashed (prompt_sha256) and the model digest is stored with every generation.

A question longer than a model's window. Some models cannot load the context window we ask for, and the host then cuts a prompt that does not fit. When a stored answer's prompt-token count shows that this happened, the answer is marked on the model page and on the question page, and the model page says how many long-context questions were affected. These answers stay scored as answered: the model could not read the whole question, so its score is not changed.

Answers generated at a time. The number of requests the runner kept in flight varied by model (1, 2, 4, 8 across the published models), and each model page and the speed page show it as “Answers generated N at a time” (“N to M at a time” when the setting changed during that model's run). Models whose speed test showed no gain from parallel requests, or whose architecture the server runs one request at a time, were sent one at a time. The server's number of parallel slots can also change during a run; the backend label on the speed page describes the run's main setting, and each model's own “at a time” figure shows what that model actually ran with.This changes the speed and memory figures; scores are computed the same way for every model.

Thinking policy. A hybrid model, one that can switch its reasoning mode off, is asked to answer with thinking off, so models are compared on direct answers. A model that always reasons cannot be switched off; it is allowed a thinking budget of 4,096 extra tokens on top of the category limit so the reasoning does not starve the answer. Thinking-mode variants of hybrid models may be added later as separate rows.

The judges

Judging only where it adds information. The judges score and review the categories marked judged (Reasoning, Coding, Instruction following, Safety, Writing). The other categories (Math, Knowledge & honesty, Long context, Multilingual, Tool use, Pushback resistance, Famous gotchas, Self-description) have an exact answer or none to grade, so a judge would add nothing a programmatic check does not already say, or nothing that can be checked. A single item can override its category's default.

The judges are paid AI models accessed through OpenRouter, not the open-weight models being tested. Judgements recorded so far come from: anthropic/claude-haiku-5.5, deepseek/deepseek-v4.1-flash, openai/gpt-5.6-luna. Each model is meant to be judged by two models from different families than its own; the remaining judgements are in progress. By design, never by a model from its own family: the pair is taken from the pool, skipping any judge from the same company as the candidate (including distills and fine-tunes built on its weights).

The automatic checks stay ground truth for code, maths, instruction following and tool calls: the judge is shown the result and only places the answer inside the band it allows. For safety prompts that should be refused, both judges read every answer and decide from its text (did it refuse, help, or partly help); the automatic refusal detector is shown to them only as a hint that may be wrong, and an empty answer scores 0 without a judge.

Rubric bands

The programmatic verdict fixes the score band; the rubric anchors place the response inside it.

How sure is a score? Intervals and rank ranges

Every model answers the same fixed set of questions. A different set of questions from the same pool would give a slightly different score. The interval shows how much.

It is worked out by resampling the questions. We draw a new set of questions from each category, with replacement, so some questions appear twice and some not at all. We score every model on that new set. We do this 2,000 times. The interval is the middle 95 percent of the scores a model got. It is shown in small grey type under Overall.

All models are scored on the same new sets, so the comparison between two models is fair. Model A is clearly ahead of model B only when A comes out ahead of B in more than 97.5 percent of the new sets.

Models measured in different runs were not scored on the same new sets, so they are only called clearly ahead when their 95% intervals do not overlap, a stricter test than the paired one used within a run.

That is what the rank range uses. The first number is 1 plus the number of listed models that are clearly ahead. The last number is the number of listed models minus the number this model is clearly ahead of. A rank of 3–6 means two listed models are clearly ahead of it, three others cannot be told apart from it, and all the rest are clearly behind it. A plain 3 means its place is settled. The table is still sorted by Overall. The range is worked out again inside each size class.

What the interval does not cover. It only covers the choice of questions. It does not cover a model giving different text when it is asked the same thing twice.

We ran six small models (3.1B to 3.8B parameters) six times on the same sample of 57 to 59 questions per model. Their score on that sample moved by a median of 5.3 points between the highest and lowest run (at most 7.5), and 14.0% of individual answers scored differently between runs. One changed answer moves a score on that sample by about 1.7 to 1.8 points; the published Overall averages 232 questions, so the same noise moves it by roughly half as much.

If the statistics were worked out for an older state of the data, the page does not show them. You then see the plain rank and no interval.

Hard questions

A question is hard when half or fewer of the ranked models of 3B and over answer it fully correctly. Only questions in the categories that count toward Overall can be hard.

Tiny models (under 3B) do not take part in that choice: with dozens of them ranked, most questions would count as hard simply because very small models fail them. Every ranked model, tiny ones included, is still scored on the hard questions. With fewer than 10 ranked models of 3B and over, every ranked model of that run takes part in the choice instead. The choice is made run by run and recorded with the run, so each page words it for the run (or runs) it draws on: a page that combines runs with different groups says the group is chosen per run.

The Hard questions column is a model's mean score on those questions, from 0 to 100. It is left blank unless the model has a score for at least 80 percent of them. It is blank for everyone until at least 10 models are ranked.

The hard set is not fixed. It is chosen again every time models are added, so a question can leave it when newer models answer it well. Each run chooses its own hard set, so compare hard scores only between models from the same run.

A second chance for cut-off answers

Each answer has a length limit. An answer that hits the limit is cut off before it is finished.

To see what difference the limit makes, a cut-off answer is asked once more with double the length. The second answer is stored and scored on its own. The headline score never changes. It stays at the fixed length so that every model is compared on the same terms.

The model page shows a line called With extended length. It is the Overall score if each cut-off answer were replaced by its second attempt. It also says how many cut-off answers there were and how many were re-asked. For the public sample questions, the second answer is shown under the first one, with its own verdict. Answers that stopped because the model began to repeat itself are not counted as cut off by length.

The comply-quality gate

Some safety questions are harmless, and the model should help. For these over-refusal checks the refusal detector scores the answer itself: a refusal is 0, an empty answer is 0, and anything else is 1. The judges also read these answers and their scores are shown, but they change the score only through the gate below. The detector only checks that the model did not refuse. It cannot tell if the help was right. A model that gives wrong first aid advice would still get full marks.

So when the model did help, the two judges look at the answer. When both judges independently rate a “should help” answer as largely inaccurate or unhelpful (3 or lower out of 10), it counts as failed and scores 0, even though the model did not refuse. The page says: “Counted as failed: both judges rated this answer largely inaccurate or unhelpful”. The detector's own verdict, did not refuse, stays visible next to it.

The gate is careful. Two judges must agree. If only one judge gave a score, nothing changes. If either judge gives a score above 3, nothing changes. A cut-off answer is never gated, and an answer that was a refusal or empty already scores 0, so the gate does not touch it. This is the second of the two places where judge scores enter the programmatic score; the first is the judge-resolved refusal items above. The safety block on the model page counts gated answers as failed and says how many there were.

Questions every ranked model answered correctly

This table counts, in each category, the questions that every model with a score for them answered correctly. It uses run r20261102-bv1-5246ab.

Questions answered correctly by every model that has a score for them, per category
CategoryQuestionsAnswered correctly by all
Reasoning355
Math250
Coding302
Instruction following300
Knowledge & honesty301
Long context180
Multilingual200
Tool use202
Safety243
Pushback resistance121
Famous gotchas151

Hardware

All models run through Ollama on one machine; its GPU label, as recorded with the runs, is AMD Radeon 8060S (Ryzen AI Max+ 395, 128 GB unified, ~78 GB GPU). Every speed and memory figure comes from that machine, so they compare models with each other and not with other hardware.

Speed and memory profile

Before its benchmark items, each model gets a short performance test through the same Ollama endpoint and the same context length as the benchmark (a different context length would reload the model and measure the reload). The Speed page and each model page show:

Separate runs. Models are measured in separate runs that may use different Ollama versions and backends (each model page names them). Quality scores are compared across runs as measured; speeds are compared only within the same backend and Ollama version.

What the server memory reading is. GPU memory on our server (1 to 8 answers at a time) is what Ollama reported for the loaded model with this server's settings, including the room it holds for its parallel slots and the context length. It depends on those slots and that context length, it is not what one user needs, and for 19 models Ollama reported less than the size of the model file. It is shown as a raw measurement on the speed page, the model pages, the compare page and the leaderboard, and it is not used to say what fits.

What "fits in N GB" means. The hardware picker and the best-model pages by memory use an estimate, not a measurement: the model file size plus about 20% for the context of a single conversation. The estimate is the size of the model file (as recorded for the version that was tested) times 1.2. A model fits a budget when that estimate is at most the budget. A model with no file size on record is listed as not measured and is never given a figure from the server reading. A longer conversation, several conversations at once or a larger context window needs more.

The performance test runs with thinking off where the model allows it, because token rates do not depend on whether tokens are thinking tokens but a reasoning preamble would make prompts incomparable between models. Ollama cannot force an exact output length, so a run that stops early is still used for the rate. Speeds are hardware-specific and backend-specific: the backend each model was measured on is published with its numbers, and speeds should be compared only between rows that share a backend. A dash means not measured.

Reproducibility

Each model page lists the Ollama digest, quantisation, context length, Ollama version, hardware and run id. Scores, statuses, token counts and timings for every item, and the raw outputs, scorer details, judge rationales and reviews for the public sample items, are available as JSON. Generations are immutable: a new judge, judge model or rubric version re-judges the stored outputs without regenerating them, and a new battery version triggers new runs.

Limitations