Trust / Reproducibility
Reproducibility
Every model is run once at temperature 0 with a fixed seed, and even so a second run does not give exactly the same answers. This page shows how far a score moved when we repeated the battery.
All repeated models together
We ran six small models (3.1B to 3.8B parameters) six times on the same sample of 57 to 59 questions per model. Their score on that sample moved by a median of 5.3 points between the highest and lowest run (at most 7.5), and 14.0% of individual answers scored differently between runs. One changed answer moves a score on that sample by about 1.7 to 1.8 points; the published Overall averages 232 questions, so the same noise moves it by roughly half as much.
- 5.3median spread between the highest and lowest run (points)
- 7.5largest spread of any model (points)
- 14.0%of answers scored differently between runs
- 61.4%of answers worded differently between runs
How the leaderboard turns this into an interval is described in the methodology.
Model by model
| Model | Questions | Runs | Lowest | Highest | Spread | SD | Scored differently | Worded differently |
|---|---|---|---|---|---|---|---|---|
| Qwen2.5-3B-Instruct3.1B parameters | 58 | 6 | 70.8 | 78.4 | 7.5 | 3.0 | 17.2% | 70.7% |
| Llama-3.2-3B-Instruct3.2B parameters | 59 | 6 | 73.6 | 80.0 | 6.4 | 2.2 | 13.6% | 54.2% |
| SmolLM3-3B3.1B parameters | 57 | 6 | 75.1 | 81.4 | 6.3 | 2.1 | 15.8% | 61.4% |
| Qwen2.5-Coder-3B-Instruct3.1B parameters | 59 | 6 | 67.0 | 71.3 | 4.3 | 1.6 | 13.6% | 55.9% |
| Qwen2.5-VL-3B-Instruct3.8B parameters | 58 | 6 | 66.8 | 70.7 | 4.0 | 1.4 | 15.5% | 58.6% |
| granite-4.1-3b3.4B parameters | 59 | 6 | 79.8 | 83.5 | 3.6 | 1.2 | 8.5% | 67.8% |
Withheld
These models are left out of every figure on this page. Why each was withheld is on the leaderboard.