Trust

A score is a measurement, and a measurement has limits. These pages put the limits next to the numbers: how much a score moves when the battery is run again, how often an answer was cut off, which inference backend produced each run, and which trick questions models fell for.

Reproducibility

How much a model's score moves when the same battery is run again.

Truncation

Answers that hit the length limit, and what a second chance changes.

Backends

Which inference backend produced each run, and why speeds compare only within one.

Gotcha questions

Every trick question in the battery and which models fell for it.

Errata

What we withheld or corrected, when, and the run that replaced it.

Reproduce an answer

The tag, digest, options and prompt hash behind one stored answer, and what cannot be repeated.

The judges

Where the two judge models agree and where they split.

Methodology

How the battery is built, scored and judged, and what it cannot tell you.