Methodology

Battery v1, rubric v1. Every number traces to a stored row holding the prompt, output, model digest and judge scores. The full methodology has every detail.

Judges. The judges are paid AI models accessed through OpenRouter, not the open-weight models being tested. Recorded so far: anthropic/claude-haiku-5.5, deepseek/deepseek-v4.1-flash, openai/gpt-5.6-luna. Each model is judged by two of them, never by a model from its own family.

The test battery

270 items in 13 categories. Code checks what it can. Judges decide refusal items and fail finished harmless answers both rate 3/10 or lower; the rest is advisory. Overall is the mean of 9 categories. Full detail

Public sample

44 items are public. All other prompts stay private to limit contamination. Full detail

Generation settings

Every model gets temperature 0, seed 42 and a fixed token limit (more for thinking models). Full detail

Speed and memory

GPU memory on our server (1 to 8 answers at a time) is what Ollama reported for the loaded model, not what one user needs. Compare speeds only within one backend. Full detail

Fine-tunes

They are ranked like any other model, with the base model named. Full detail

Specialised models

They count toward Overall; the label comes from name patterns. Full detail

Hard questions

Hard means half or fewer of the ranked models of 3B and over answer it fully correctly. Full detail

Intervals

An interval shows how much a score would move with a different set of questions. We ran six small models (3.1B to 3.8B parameters) six times on the same sample of 57 to 59 questions per model. Their score on that sample moved by a median of 5.3 points between the highest and lowest run (at most 7.5), and 14.0% of individual answers scored differently between runs. One changed answer moves a score on that sample by about 1.7 to 1.8 points; the published Overall averages 232 questions, so the same noise moves it by roughly half as much. Full detail

Cut-off answers

They are asked again at double the length; the headline score never changes. Full detail

Limitations

The battery is small, close models cannot be separated, and judges are fallible. Full detail

Sections of the full page