Trust / Reproducibility

Reproducibility

Every model is run once at temperature 0 with a fixed seed, and even so a second run does not give exactly the same answers. This page shows how far a score moved when we repeated the battery.

All repeated models together

We ran six small models (3.1B to 3.8B parameters) six times on the same sample of 57 to 59 questions per model. Their score on that sample moved by a median of 5.3 points between the highest and lowest run (at most 7.5), and 14.0% of individual answers scored differently between runs. One changed answer moves a score on that sample by about 1.7 to 1.8 points; the published Overall averages 232 questions, so the same noise moves it by roughly half as much.

How the leaderboard turns this into an interval is described in the methodology.

Model by model

Each model was run several times on the same questions (the Runs column gives the count). Spread is the highest run minus the lowest run on those questions, in points. Click a column heading to sort.
Model Questions Runs Lowest Highest Spread SD Scored differently Worded differently
Qwen2.5-3B-Instruct3.1B parameters 58 6 70.8 78.4 7.5 3.0 17.2% 70.7%
Llama-3.2-3B-Instruct3.2B parameters 59 6 73.6 80.0 6.4 2.2 13.6% 54.2%
SmolLM3-3B3.1B parameters 57 6 75.1 81.4 6.3 2.1 15.8% 61.4%
Qwen2.5-Coder-3B-Instruct3.1B parameters 59 6 67.0 71.3 4.3 1.6 13.6% 55.9%
Qwen2.5-VL-3B-Instruct3.8B parameters 58 6 66.8 70.7 4.0 1.4 15.5% 58.6%
granite-4.1-3b3.4B parameters 59 6 79.8 83.5 3.6 1.2 8.5% 67.8%

Withheld

These models are left out of every figure on this page. Why each was withheld is on the leaderboard.