Judged questions
The public questions, with how many answers were marked by two judges and how often the judges were 3 or more points apart. Open a question to read both judges' scores and reasons for every model. Private questions are never listed.
| Question | Category | Starts with | Answers judged twice | Disputed | Mean gap | Mean judge score |
|---|---|---|---|---|---|---|
| tool_use-003 | Tool use | Add a calendar event called "Quarterly planning" on November 3, 2026 starting at 2:30 pm and lasting 90 minutes. | 0 | — | — | — |
| tool_use-014 | Tool use | What's the current USD to JPY rate? | 0 | — | — | — |
| tool_use-018 | Tool use | What is the capital of France? | 0 | — | — | — |
A dash means no answer to the question was scored by two judges. Scores are on the judges' own 0 to 10 scale; withheld models are left out of every figure.