About OpenWeights Bench
OpenWeights Bench is an independent benchmark and review site for open-weight AI models. It takes the most-downloaded open-weight models, runs each one locally on one laptop through a fixed battery of 270 items in 13 categories (reasoning, math, coding, instruction following, knowledge & honesty, long context, multilingual, tool use, safety, pushback resistance, famous gotchas, writing, self-description), measures its speed and memory use, and publishes what happened: the scores, the speed profile, the raw outputs of a public sample, and the written reviews of two judge models.
Principles
- Show the evidence. Every score links back to stored prompts, outputs and verdicts. Raw results are available as JSON.
- Programmatic checks first. Where an answer can be checked by code (code, maths, instructions, tool calls), it is, and the check counts as ground truth. Safety is the exception: for a prompt that should be refused, every answer is read by two independent judges from other companies than the model’s maker, and the automatic refusal check is shown to them only as a hint. An empty answer scores 0. On a harmless prompt the model should answer, the check still scores the answer, but an answer that both judges rate 3 or lower out of 10 is scored 0 unless it was cut off. Every other judge score is advisory.
- Judges are shown, not hidden. Two judge models, each from a different company than the model being tested, score the items where a judge adds information. Both scores are displayed and disagreements are flagged.
- A missing score is not a zero. Anything that was not scored, or could not be parsed, is shown as such.
- The judges are paid models, and we say so. They are accessed through OpenRouter (openai/gpt-5.6-luna, anthropic/claude-haiku-5.5, deepseek/deepseek-v4.1-flash). They are never the model under test or a model from its family.
What this site is not
It is not a leaderboard of overall model quality. The battery is small, runs at one quantisation and one seed, and judge models are fallible. Treat the results as one set of measurements and read the limitations.
Licence
The results and scores on this site are licensed under CC BY 4.0: you may copy, share and reuse them, including commercially, as long as you credit OpenWeights Bench (openweightsbench.com). This covers our results and scores only. It does not cover the judges’ reviews, the models’ answers, the question texts or the models themselves; each model has its own licence terms.
Contact
Questions or corrections: [email protected].
Updates
New models and new versions of existing models are tested as they appear, and the battery and rubric versions are shown on every run. Results from different battery versions are not directly comparable.