Trust / Gotcha questions
Gotcha questions
A gotcha is a well-known puzzle or trap with one right answer that a model can easily get wrong by pattern-matching. The battery holds 15 of them. 3 are public and shown below; the other 12 stay private and appear by id only.
Who fell for which
The 15 questions
- gotchas-001: How many times does the letter 'r' appear in the word 'strawberry'? Give your final answer on the last line in the form: ANSWER: <value> (111 of 228 models got it wrong)
- gotchas-002: prompt private (51 of 228 models got it wrong)
- gotchas-003: prompt private (108 of 228 models got it wrong)
- gotchas-004: prompt private (28 of 228 models got it wrong)
- gotchas-005: prompt private (99 of 228 models got it wrong)
- gotchas-006: prompt private (46 of 228 models got it wrong)
- gotchas-007: prompt private (139 of 228 models got it wrong)
- gotchas-008: prompt private (54 of 228 models got it wrong)
- gotchas-009: prompt private (100 of 228 models got it wrong)
- gotchas-010: You pick door 1 of 3 doors; one hides a car, the other two hide goats. The host, who does NOT know where the car is, opens door 3 at random… (150 of 228 models got it wrong)
- gotchas-011: prompt private (26 of 228 models got it wrong)
- gotchas-012: prompt private (54 of 228 models got it wrong)
- gotchas-013: A doctor gives you 3 pills and tells you to take one every 30 minutes, starting now. How many minutes after taking the first pill do you tak… (54 of 228 models got it wrong)
- gotchas-014: prompt private (57 of 228 models got it wrong)
- gotchas-015: prompt private (59 of 228 models got it wrong)
Withheld
These models are left out of every figure on this page. Why each was withheld is on the leaderboard.