Methodology
Battery v1, rubric v1. Every number traces to a stored row holding the prompt, output, model digest and judge scores. The full methodology has every detail.
Judges. The judges are paid AI models accessed through OpenRouter, not the open-weight models being tested. Recorded so far: anthropic/claude-haiku-5.5, deepseek/deepseek-v4.1-flash, openai/gpt-5.6-luna. Each model is judged by two of them, never by a model from its own family.
The test battery
270 items in 13 categories. Code checks what it can. Judges decide refusal items and fail finished harmless answers both rate 3/10 or lower; the rest is advisory. Overall is the mean of 9 categories. Full detail
Public sample
44 items are public. All other prompts stay private to limit contamination. Full detail
Generation settings
Every model gets temperature 0, seed 42 and a fixed token limit (more for thinking models). Full detail
Speed and memory
GPU memory on our server (1 to 8 answers at a time) is what Ollama reported for the loaded model, not what one user needs. Compare speeds only within one backend. Full detail
Fine-tunes
They are ranked like any other model, with the base model named. Full detail
Specialised models
They count toward Overall; the label comes from name patterns. Full detail
Hard questions
Hard means half or fewer of the ranked models of 3B and over answer it fully correctly. Full detail
Intervals
An interval shows how much a score would move with a different set of questions. We ran six small models (3.1B to 3.8B parameters) six times on the same sample of 57 to 59 questions per model. Their score on that sample moved by a median of 5.3 points between the highest and lowest run (at most 7.5), and 14.0% of individual answers scored differently between runs. One changed answer moves a score on that sample by about 1.7 to 1.8 points; the published Overall averages 232 questions, so the same noise moves it by roughly half as much. Full detail
Cut-off answers
They are asked again at double the length; the headline score never changes. Full detail
Limitations
The battery is small, close models cannot be separated, and judges are fallible. Full detail