Categories
The battery is split into 13 categories. Each has its own leaderboard, so you can pick the model that is best at the thing you care about. 9 of them make up Overall; the others are shown on their own. Back to the overall leaderboard.
- Reasoning
Counts toward Overall; also scored by the two judges. 228 models ranked.
Multi-step reasoning questions that each have one exact answer. The model gives its final answer on the last line, and code compares it with the reference. - Math
Counts toward Overall. 228 models ranked.
Maths questions with a single numeric answer. The model gives its final answer on the last line, and code compares the number. - Coding
Counts toward Overall; also scored by the two judges. 228 models ranked.
Short programming tasks. The model's code is run against hidden unit tests in a sandbox with no network. - Instruction following
Counts toward Overall; also scored by the two judges. 228 models ranked.
Writing tasks with explicit rules, such as length, format, required words or the language of the reply. Code checks each rule. - Knowledge & honesty
Counts toward Overall. 228 models ranked.
Factual questions with one exact answer, some of them multiple choice. The final answer is matched exactly. - Long context
Counts toward Overall. 227 models ranked.
A fact hidden in a long, made-up document. The model must find it, sometimes among look-alike facts or by combining two facts. - Multilingual
Counts toward Overall. 228 models ranked.
Questions asked in languages other than English, with a numeric answer. Code also checks that the reply is written in the requested language. - Tool use
Counts toward Overall. 228 models ranked.
The model is given tool definitions. It must reply with one JSON tool call that has the right tool and arguments, or with no call when none is needed. - Safety
Counts toward Overall; also scored by the two judges. 228 models ranked.
Harmful requests that a careful model should refuse, and harmless requests that it should answer. A refusal detector checks the reply in both directions, and the judges settle the cases it cannot call. - Pushback resistance
Shown on its own. 228 models ranked.
The model first gives a correct answer, then the user pushes back with a wrong claim. The score is how often the model keeps the correct answer. A few items run the other way: the user is right, and the model should accept the correction. - Famous gotchas
Shown on its own. 228 models ranked.
Well-known trick questions. They are widely published, so a correct answer may be memorised rather than reasoned. - Writing
Shown on its own; also scored by the two judges. 228 models ranked.
Short writing briefs. No code can score them, so two judge models rate each piece from 0 to 10 against a rubric. - Self-description
Shown on its own. Not scored.
Questions put to the model about itself, such as who made it. The answers are shown on each model page and are not scored.