← Judged questions

reasoning-004: both judges

A public question in Reasoning, battery v1. It starts: “A movie starts at 19:45 (24-hour clock) and runs for 2 hours 20 minutes. What time does it end? Answer in 24-hour HH:MM format. Give your fi…” Read the question and every model's answer.

228 models judged; 8 disputed (judges 3 or more points apart). The gap is the distance between the two judges' scores. Models are ordered by that gap, widest first.

Codellama 34B Instruct gap 4, disputed

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 10/10

The response gives the correct end time, 22:05, exactly matching the reference answer and required HH:MM format. It also places the answer on the final line as requested.

The response’s strength is its exact compliance with the requested output form: it supplies only the required ANSWER line and gives the correct time, 22:05. Its brevity is appropriate because the task asks for a direct final answer and does not require a shown calculation. There is no substantive weakness in the submitted response; it matches the reference and satisfies the formatting constraint without introducing irrelevant material.

Judge B: J-A (anthropic/claude-haiku-5.5) 6/10

The response gives only the line 'ANSWER: 22:05', which matches the reference and the programmatic verdict of correct. It shows no supporting arithmetic, so the final answer is right but the argument that justifies it is absent.

The response reaches the correct end time of 22:05 for a 19:45 start with a 2 hour 20 minute runtime, and it uses the required ANSWER line in the requested HH:MM format. Its main weakness is that it offers no working at all. Nothing shows how the 2 hours and 20 minutes were added to 19:45, so a reader cannot verify the step from the response alone. The answer is correct, but the response gives no reasoning that supports it.

Command R 35B 08 2024 gap 4, disputed

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 10/10

The response gives the correct end time, 22:05, and places it on the final line in the required “ANSWER: <value>” format. It is concise and fully addresses the task.

The response correctly computes the ending time by adding 2 hours and 20 minutes to 19:45, producing 22:05. It also follows the required final-line format exactly, making the answer easy to verify. Its only minor limitation is that it does not show the intermediate addition, but the task requests a formatted answer rather than an explanation, so this omission does not affect fulfillment. The response is direct, relevant, and complete for the stated question.

Judge B: J-A (anthropic/claude-haiku-5.5) 6/10

Verdict is correct and the final answer 22:05 matches the reference, which is 19:45 plus 2h20m. The response gives no supporting arithmetic, so the argument that justifies the answer is missing. Format follows the required ANSWER line.

The response states the end time as 22:05 and gives the required ANSWER line in the requested HH:MM format, matching the reference. Its strength is that the final value is exactly right and presented cleanly without padding. Its weakness is that it shows no working: it never breaks down the 2 hours 20 minutes added to 19:45, so a reader cannot see how the result was reached. The answer is correct, but the justification is absent, which caps the quality of the response even though the outcome is fully accurate.

Hermes3 3B Llama3.2 gap 4, disputed

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 10/10

The response gives the correct end time, 22:05, in the required 24-hour HH:MM format and places it on the final line with the requested ANSWER label.

The response directly supplies the required answer and matches the reference exactly. Its concise format is appropriate because the task requests a simple time calculation and explicitly specifies the final-answer form. The only minor limitation is that it does not show the intermediate addition of 2 hours 20 minutes to 19:45, so the arithmetic is not demonstrated; however, no working was required, and the stated result is unambiguous and correctly formatted.

Judge B: J-A (anthropic/claude-haiku-5.5) 6/10

The response gives only the line 'ANSWER: 22:05', which matches the reference and the programmatic verdict of correct. It shows no supporting arithmetic, so the argument that would justify the answer is absent. Format is correct.

The response reaches the correct end time of 22:05 and places it on the required final line in the requested ANSWER format. Its main strength is that the final value is accurate and cleanly formatted. Its weakness is that it provides no working, such as adding 2 hours to 19:45 to reach 21:45 and then adding 20 minutes. The answer is therefore correct but unsupported, so a reader cannot verify how it was derived from the prompt without redoing the calculation themselves.

Phi 2.7B Chat V2 gap 4, disputed

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 10/10

The response gives the correct computed end time, 22:05, exactly matching the reference and the required ANSWER format. It is concise and contains no irrelevant or erroneous content.

The response correctly identifies the ending time as 22:05 and places it in the requested ANSWER format. Its brevity is appropriate because the task asks only for the final value, and the result matches the programmatic verdict. The only minor presentation issue is a leading space before the answer line, which does not affect readability or compliance. No substantive weakness is present in the calculation or final formatting.

Judge B: J-A (anthropic/claude-haiku-5.5) 6/10

The response gives only the final line 'ANSWER: 22:05', which matches the reference and the programmatic verdict of correct. It provides no supporting arithmetic, so the argument that justifies the answer is absent, placing it at the lower end of the correct band.

The response reaches the correct end time of 22:05 and uses the requested ANSWER line format. Its strength is that the final value is accurate and stated in the required 24-hour HH:MM form. Its weakness is that it shows no working: it does not indicate how the 2 hours 20 minutes was added to 19:45, so the reader cannot verify the derivation from the response itself. The answer is correct but unsupported by any visible reasoning.

Phi-3.5-mini-instruct gap 4, disputed

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 10/10

The response correctly adds 2 hours 20 minutes to 19:45, reaching 22:05, and places the required ANSWER line last in the requested 24-hour format.

The response directly addresses the time calculation and gives the correct ending time, 22:05, in the required HH:MM format. It also follows the instruction to place the final answer on the last line. Its only minor limitation is that it does not show a separate intermediate addition step, although the stated conclusion is fully sufficient for this straightforward arithmetic task. The response is concise, relevant, and internally consistent.

Judge B: J-A (anthropic/claude-haiku-5.5) 6/10

The final answer 22:05 matches the reference and the programmatic verdict is correct. The response gives the correct end time and the ANSWER line in the required format, but it offers no working. Nothing in the text shows how 19:45 plus 2h20m was computed, so the argument does not justify the answer.

The response reaches the correct end time of 22:05 and places it on the required final ANSWER line in HH:MM format, which satisfies the task's output constraint. Its main weakness is that it provides no supporting arithmetic. It does not show how the 2 hours 20 minutes was added to 19:45, so a reader cannot verify the result from the response itself. The reply is brief and direct, with no extraneous content or hedging, but the justification the rubric asks for is absent.

Qwen3 VL 2B Thinking gap 4, disputed

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 10/10

The response gives the correct end time, 22:05, and follows the required final-answer format exactly. Its concise presentation is appropriate because the task only requests the result.

The response’s main strength is that it provides the correct time and uses the specified “ANSWER: <value>” format without extraneous text. Its only minor weakness is that it does not show the intermediate addition of two hours and twenty minutes to 19:45, so the calculation is implicit rather than demonstrated. However, the task requests a formatted answer rather than an explanation, making the concise response fully effective for this prompt.

Judge B: J-A (anthropic/claude-haiku-5.5) 6/10

The response gives only the line 'ANSWER: 22:05', which matches the reference and the programmatic verdict of correct. No supporting arithmetic is shown, so the argument that justifies the answer is missing, placing it at the lower end of the correct band.

The response reaches the correct end time of 22:05 and presents it on the required final ANSWER line in 24-hour HH:MM format, which satisfies the task's output constraint. Its main weakness is that it offers no working at all: there is no visible step adding 2 hours 20 minutes to 19:45, so the reader cannot see how the answer was derived. The response is terse, which is acceptable when correct, but here the justification is absent rather than concise. Under the reasoning rubric, a right answer without a supporting argument is limited to the lower correct band.

Tulu3 8B gap 4, disputed

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 10/10

The response gives the correct end time, 22:05, in the requested 24-hour HH:MM format and places it on the final line with the required ANSWER label.

The response correctly adds 2 hours and 20 minutes to the 19:45 start time, producing 22:05. Its sole answer line follows the requested format exactly, including the ANSWER label and 24-hour time notation. The response is appropriately concise for a straightforward arithmetic task. A minor limitation is that it does not show the intermediate addition, but the task only required the final answer, so this omission does not affect correctness or compliance.

Judge B: J-A (anthropic/claude-haiku-5.5) 6/10

The final answer 22:05 is correct per the programmatic verdict, matching 19:45 plus 2h20m. However, the response contains only the ANSWER line with no worked argument showing the addition of 2 hours and 20 minutes, so the supporting reasoning is absent.

The response gives the correct end time, 22:05, on the required ANSWER line and uses the requested 24-hour HH:MM format. It is terse and contains no extraneous text, which fits the instruction. Its weakness is that it shows no working: nothing demonstrates how the 2 hours 20 minutes was added to 19:45, such as adding 2 hours to reach 21:45 and then 20 minutes. The correct result is therefore asserted rather than justified, and a reader cannot verify the arithmetic from the response itself.

Granite3 Moe 3B Instruct gap 3, disputed

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 10/10

The response gives the correct end time, 22:05, exactly matching the reference and required final-answer format. It is concise and contains no irrelevant or incorrect material.

The response’s strength is that it supplies the correct time, “22:05,” in the requested 24-hour format and places it on the final answer line. It is also appropriately concise for a straightforward time-addition task. Its only minor limitation is that it does not show the intermediate addition of two hours and twenty minutes, but the prompt asked for the final answer rather than an explanation, so this omission does not affect task completion.

Judge B: J-A (anthropic/claude-haiku-5.5) 7/10

The response gives only the final line 'ANSWER: 22:05', which matches the reference and the programmatic verdict of correct. It shows no supporting arithmetic, so the argument that would justify the answer is absent, placing it at the lower end of the correct band.

The response reports the correct end time of 22:05 in the requested HH:MM format on the required final line. Its strength is accuracy and strict compliance with the answer format. Its weakness is that it provides no working, so the reader cannot see how the 2 hours 20 minutes was added to 19:45 or verify the carry across the hour boundary. The task was simple and the answer is right, but the response offers no justification beyond the bare value.

Codellama 13B Instruct gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 10/10

The response correctly adds 2 hours 20 minutes to 19:45, obtaining 22:05. It follows the required 24-hour HH:MM format and places the required ANSWER line last.

The response clearly identifies the start time and duration, then performs the addition correctly to reach 22:05. It also satisfies the required final-line format exactly, making the answer easy to verify. A minor limitation is that it does not separately explain the minute calculation or any hour carry, but the displayed equation is sufficient for this straightforward task. No substantive reasoning or formatting error is present.

Judge B: J-A (anthropic/claude-haiku-5.5) 8/10

Final answer 22:05 matches the reference and the programmatic verdict is correct. The argument is a single line restating the addition, 19:45 + 2:20 = 22:05, with no intermediate steps shown, so the justification is minimal but adequate. Answer is on the required final ANSWER line in HH:MM format.

The response reaches the correct end time of 22:05 and presents it on the required final line in the requested 24-hour HH:MM format. Its argument is a single equation that adds the 2 hours 20 minutes to 19:45 without showing the carry or the minute-by-minute steps, so a reader must verify the addition independently. The response is terse and free of padding, which suits a task of this size. The main weakness is that the justification is compressed to one line, leaving the minute carry unexplained even though the result is correct.

Command R7B Arabic 7B 02 2025 gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 8/10

The response correctly adds 2 hours 20 minutes to 19:45, obtaining 22:05. The arithmetic and intermediate steps are clear and fully support the correct result. However, it ends with “Final Answer: 22:05” rather than the explicitly required “ANSWER: 22:05” format.

The response clearly identifies the starting time, separates the duration into hours and minutes, and performs both additions correctly, reaching 22:05. Its step-by-step arithmetic is easy to verify and directly addresses the task. The main weakness is formatting: the task required the final line to use the exact form “ANSWER: <value>,” whereas the response instead uses “Final Answer: 22:05.” The numerical answer remains correct, but the requested final-answer label was not followed exactly.

Judge B: J-A (anthropic/claude-haiku-5.5) 10/10

The response converts 19:45 correctly, adds 2 hours to reach 21:45, then adds 20 minutes to reach 22:05. Each step follows from the last with no padding, and the final answer matches the reference and the correct verdict. The ANSWER line format is not present verbatim, but the value 22:05 is stated as the final answer.

The response works through the problem in a clear, stepwise manner, splitting the duration into hours and minutes and adding them sequentially to the start time. Each intermediate result (21:45, then 22:05) is shown and correct, and the final value matches the reference answer. Its main weakness is formatting: it labels the result as 'Final Answer' rather than using the requested 'ANSWER:' line, and the opening step about converting to 24-hour format is unnecessary given the input is already in that format. The reasoning itself is sound and minimal enough for the task.

Flint 40B A2B gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 9/10

The response correctly adds 2 hours 20 minutes to 19:45, obtaining 22:05, and places the required “ANSWER: 22:05” on the last line. The arithmetic and requested format are correct, with only unnecessary meta-text before the answer.

The response’s strength is that it performs the time addition correctly and gives the exact required final-answer line, including the proper 24-hour format. Its weakness is the inclusion of internal-looking planning text and a stray closing thinking tag before the concise explanation, which is unnecessary for this simple task and slightly reduces output cleanliness. The final line nevertheless satisfies the explicit placement and formatting requirement.

Judge B: J-A (anthropic/claude-haiku-5.5) 7/10

Final answer 22:05 matches the verdict of correct. The argument is a single line of addition (19:45 + 2:20 = 22:05) with no worked steps shown. The visible text contains a stray closing think tag and leaked planning notes about the answer format, which is a flaw in presentation.

The response reaches the correct end time, 22:05, and gives it in the requested 24-hour HH:MM format on the final line with the required ANSWER prefix. Its supporting work is thin: it states the addition result in one line without showing how the minutes and hours combine, so the justification is brief. The output also leaks internal planning notes about the answer format and includes a stray closing think tag, which clutters an otherwise correct reply.

Gemma 3n E4B IT gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

The response correctly computes 19:45 plus 2 hours 20 minutes as 22:05, including proper normalization of 65 minutes. However, it does not provide the required final line in the exact form “ANSWER: <value>”; instead, it uses prose and LaTeX boxing.

The response shows the time addition explicitly and correctly converts the intermediate 21:65 into 22:05, so the mathematical result is reliable. Its main weakness is instruction compliance: the requested final line beginning exactly with “ANSWER:” is absent, and the response instead ends with a prose statement containing a boxed value. The intermediate notation 21:65 is not a valid clock time, but it is clearly presented as a temporary arithmetic result and is immediately normalized correctly.

Judge B: J-A (anthropic/claude-haiku-5.5) 9/10

Verdict is correct and the reference is 22:05. The response adds 2 h 20 min to 19:45 correctly, carrying 65 minutes into 1 hour 5 minutes, and gives the result via two consistent methods. Minor redundancy from the second method and a boxed final line that does not use the required ANSWER: format keep it from a 10.

The response reaches the correct end time of 22:05 by adding the hours and minutes separately and correctly carrying 65 minutes into one hour and five minutes. Its main strength is that the arithmetic is accurate and the carry step is explicitly justified. Its weakness is redundancy: it repeats the same calculation through a second, alternative method that adds nothing new. It also ends with a boxed LaTeX-style line rather than the requested ANSWER: line, though the value itself is in the required HH:MM format.

GLM-4.5-Air gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 10/10

The response correctly adds 2 hours to 19:45 to obtain 21:45, then adds 20 minutes to reach 22:05. It provides the required final ANSWER line in the requested 24-hour format.

The response clearly identifies the start time and duration, then performs the addition in two transparent steps. Its intermediate times, 21:45 and 22:05, are accurate, and the final line follows the required ANSWER format exactly. The explanation is slightly more structured than necessary for this simple calculation, but that does not affect correctness or clarity. No unsupported claims, arithmetic errors, or formatting problems are present.

Judge B: J-A (anthropic/claude-haiku-5.5) 8/10

Final answer 22:05 matches the reference and the programmatic verdict is correct. The argument adds 2 hours to 19:45 to get 21:45, then adds 20 minutes to reach 22:05. The steps are sound and clearly shown, with only mild padding from the restated start time and duration list and the header-style step labels.

The response reaches the correct end time, 22:05, by adding the two hours and then the twenty minutes in sequential steps. Each intermediate value, 21:45 and then 22:05, is arithmetically correct and follows from the previous step, so the derivation is easy to verify. Its main weakness is verbosity: it restates the start time and duration as a separate list and adds an introductory sentence before the calculation, none of which is needed for a two-step addition. A terser presentation would have met the task equally well. The final line follows the requested ANSWER format.

Lfm2.5 8B A1B gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 10/10

The response correctly adds 2 hours to 19:45 to get 21:45, then adds 20 minutes to reach 22:05. It provides the required final line in the requested ANSWER format.

The response demonstrates a clear and correct calculation: it separates the duration into two hours and twenty minutes, performs each addition accurately, and ends with the required 24-hour time. Its final line matches the reference answer and the requested syntax. One minor issue is that it exposes internal reasoning within think tags, which is unnecessary for this task and may violate expectations about presenting only the answer. This does not affect the correctness or completeness of the submitted result.

Judge B: J-A (anthropic/claude-haiku-5.5) 8/10

Final answer 22:05 matches the reference and the programmatic verdict of correct. The argument is shown in brief (19:45 + 2:00 = 21:45, then +20 min = 22:05), which justifies the answer. The response leaks its reasoning inside think tags, which the task did not ask for.

The response reaches the correct end time of 22:05 by adding the two hours and then the twenty minutes to 19:45, and it places the required answer on the final line in the requested format. Its strength is the correct, verifiable arithmetic performed in clear steps. Its weakness is that the reasoning is wrapped in think tags and includes a restatement of the format requirement, adding some unnecessary padding beyond the short derivation needed. The overall answer is accurate and the derivation supports it.

Lfm2.5 Thinking 1.2B gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 9/10

The response gives the correct end time, 22:05, and its final line follows the required `ANSWER: <value>` format. The extra repeated answer and explanatory sentence are unnecessary but do not affect correctness.

The response correctly calculates the ending time and presents `ANSWER: 22:05` on its final line, satisfying the central formatting requirement. Its main weakness is unnecessary repetition: it states the answer twice and adds an explanatory sentence despite the task requesting a concise final answer. The added material remains relevant and does not introduce any factual error, so it only slightly reduces precision and cleanliness.

Judge B: J-A (anthropic/claude-haiku-5.5) 7/10

The final answer 22:05 matches the programmatic verdict of correct. The response gives no worked argument, only the bare time, and then repeats the answer line with a stray fragment ('24-hour HH:MM format. The calculation confirms...') that asserts a calculation without showing it. The duplicate ANSWER line is a minor formatting blemish.

The response reaches the correct end time, 22:05, for a 19:45 start with a 2 hour 20 minute runtime, and states it in the requested 24-hour format. Its main weakness is justification: it asserts that a calculation confirms the result but never shows the addition of 2 hours and 20 minutes to 19:45. The answer line is also repeated twice, with a stray sentence between the copies, which adds clutter without adding substance. The correct value is present, but the supporting reasoning is absent.

Llama-2-7b-chat-hf gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 3/10

The response correctly converts 2 hours 20 minutes to 130 minutes, but it does not add that duration to 19:45. It incorrectly treats 130 minutes as 2.17 hours and gives 21:17 instead of the correct 22:05.

The response’s strength is that it identifies the movie duration as 130 minutes, which is a correct conversion. Its weakness is the final time calculation: it fails to add 2 hours and 20 minutes to the 19:45 start time and instead produces 21:17 from an inappropriate decimal-hour conversion. The required answer format is present, but the stated ending time is incorrect. The error occurs after the initial duration calculation, making this a near miss rather than a fundamentally unsupported attempt.

Judge B: J-A (anthropic/claude-haiku-5.5) 1/10

The response correctly converts 2h20m to 130 minutes but then divides by 60 incorrectly and rounds, never adding the duration to 19:45. The final answer 21:17 is wrong (correct is 22:05). Reasoning is flawed from the conversion step onward.

The response correctly recognises that 2 hours 20 minutes equals 130 minutes, which is the right starting point for the calculation. It then converts 130 minutes by dividing by 60 and rounding to 2.17 hours, which discards the 10 minutes that matter for the end time. It never adds the duration to the 19:45 start time, so the final answer of 21:17 is off by 48 minutes from the correct 22:05. The response also omits any step-by-step addition that would have exposed the error.

Llama3.2 1B Instruct gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

The response correctly adds two hours and twenty minutes to 19:45, reaching 22:05. However, it does not follow the required final-line format `ANSWER: <value>`; it gives the result in prose instead.

The response clearly shows the time calculation in two valid steps: first advancing from 19:45 to 21:45, then adding 20 minutes to reach 22:05. This makes the reasoning easy to verify and the final time correct. Its main weakness is formatting: the task explicitly required the final answer to appear on the last line in the form “ANSWER: <value>,” but the response ends with a prose sentence and never supplies that labeled line. The content is therefore correct despite the instruction-following omission.

Judge B: J-A (anthropic/claude-haiku-5.5) 9/10

Programmatic verdict is correct. The response adds 2 hours to 19:45 to get 21:45, then adds 20 minutes to reach 22:05, which matches the reference. The argument is clear and sound. It does not state the ANSWER: line explicitly in the required form, so it loses a point for format, though the final time is given in the right format.

The response correctly reaches the end time of 22:05 by adding the two hours and then the twenty minutes in two clear, verifiable steps. Each intermediate value is stated, so the arithmetic can be checked directly. The main weakness is that it does not close with the required 'ANSWER:' line; the final time appears only in a prose sentence. The reasoning itself is complete and contains no unnecessary padding or unstated leaps.

Marco O1 7B gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 9/10

The response correctly computes 19:45 + 2 hours 20 minutes as 22:05 and gives the required final answer in the specified format. The arithmetic is sound, though the repeated verification and unrelated theater-planning commentary are unnecessary.

The response’s main strength is that it presents multiple consistent checks of the time addition and ends with the exact required answer, “ANSWER: 22:05.” Its weakness is excessive repetition for a simple calculation: it independently recomputes the same result several times and adds irrelevant commentary about arriving at the theater. The reasoning remains correct and understandable, and the final answer is placed on the last line as requested. The exposed internal-style deliberation is unnecessary but does not affect the result.

Judge B: J-A (anthropic/claude-haiku-5.5) 7/10

Final answer 22:05 matches the verdict and reference. Argument is correct: adds 2h to reach 21:45, then 20 min, and cross-checks via minutes (1185+140=1325 → 22:05). Output exposes a visible thinking block with extended verification and filler, and the required final line is present. Padding and an unnecessary aside about buying tickets reduce the score.

The response reaches the correct end time, 22:05, and supports it with a clear step-by-step addition followed by an independent check converting both times to minutes since midnight. That cross-check is a genuine strength and makes the derivation easy to verify. The main weakness is verbosity: the response repeats the same calculation several times and adds an unnecessary note about arriving at the theater, which contributes nothing to the answer. It also places its reasoning inside a thinking-style block rather than presenting a concise justification. The required final ANSWER line is correctly included.

Medgemma 27B IT gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

The response correctly adds 2 hours 20 minutes to 19:45, carrying 65 minutes into 1 hour 5 minutes, and reaches 22:05. However, it does not follow the required final-line format `ANSWER: <value>`; its last line instead uses prose and boxed LaTeX.

The response clearly explains the time addition and correctly handles the minute overflow from 65 minutes to one additional hour and five minutes. Its repeated alternative calculation remains consistent with the first derivation. The main weakness is instruction compliance: although the correct value appears, the required last line beginning exactly with “ANSWER:” is absent, and the final line is phrased as a prose statement with LaTeX boxing. This is a minor presentation and format defect rather than a mathematical error.

Judge B: J-A (anthropic/claude-haiku-5.5) 9/10

Verdict is correct and the final answer 22:05 matches the reference. The argument adds 19+2 and 45+20, carries the 65 minutes into an extra hour, and reaches 22:05. The work is sound with some redundancy from giving the calculation twice. The final line uses boxed LaTeX rather than the requested ANSWER: line, a minor format deviation.

The response correctly determines that the movie ends at 22:05. Its main strength is a clear, step-by-step addition showing the 65-minute carry into the hour field, which makes the reasoning easy to verify. Its main weakness is redundancy: it repeats the same calculation in an alternative section without adding new information. It also closes with a boxed LaTeX expression rather than the requested 'ANSWER:' line, a small deviation from the specified output format.

Meta-Llama-3-8B-Instruct gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 3/10

The response correctly converts 2 hours 20 minutes to 140 minutes, but it adds that duration incorrectly: 19:45 + 140 minutes is 22:05, not 21:25. The required final answer is therefore wrong.

The response clearly identifies the duration and correctly converts it to 140 minutes, showing an appropriate method for solving the time addition. However, the final time calculation is incorrect: adding two hours to 19:45 gives 21:45, and adding another 20 minutes gives 22:05. Because the response concludes with 21:25, it fails the required exact answer despite making progress with the setup.

Judge B: J-A (anthropic/claude-haiku-5.5) 1/10

The response correctly converts 2h20m to 140 minutes but then adds incorrectly: 19:45 plus 140 minutes is 22:05, not 21:25. The final answer 21:25 is wrong, and the error is in the arithmetic at the final step, though the setup was sound.

The response handles the first step well by converting the 2 hour 20 minute duration into 140 minutes, which is accurate and clearly stated. Its key weakness is the final addition: adding 140 minutes to 19:45 should yield 22:05, but the response gives 21:25, apparently undercounting the hours. The answer is delivered in the required ANSWER line format, but the value is incorrect. Because the programmatic verdict marks the answer as wrong, the result cannot be counted as a correct solution despite the sound initial conversion.

Ministral 3 3B Instruct 2512 gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 8/10

The final answer is correct and matches the required HH:MM format. The response reaches 22:05, but its calculation incorrectly writes the starting time as 20:45 instead of 19:45; this appears to be a minor transcription error because the stated result is correct.

The response correctly gives 22:05 on the required final-answer line and uses the requested 24-hour format. Its main weakness is that the displayed calculation begins with 20:45, whereas the movie starts at 19:45. Although that inconsistency does not affect the reported answer, it makes the supporting arithmetic technically inaccurate as written. The response is otherwise concise, directly relevant, and easy to interpret, with the correct end time clearly identified.

Judge B: J-A (anthropic/claude-haiku-5.5) 6/10

The final answer 22:05 is correct, but the supporting line restates the start time as 20:45 instead of the prompt's 19:45. The arithmetic shown only reaches the right answer because of that misread input, so the argument does not justify the result.

The response reaches the correct end time, 22:05, and gives it in the requested 24-hour format on the final line. Its main weakness is the working line, which begins from 20:45 rather than the stated 19:45 start. Adding 2 hours 20 minutes to the correct start would give 22:05, so the answer holds, but the displayed derivation misquotes the problem and does not actually verify the result it reports.

Mistral Nemo 12B Instruct 2407 gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 10/10

The response correctly adds 2 hours 20 minutes to 19:45, showing 19:45 + 2 hours = 21:45 and then +20 minutes = 22:05. It gives the required final-line format and matches the reference answer.

The response correctly identifies the required time addition and presents the intermediate calculations clearly. Its final line uses the requested “ANSWER: 22:05” format, and the result matches the reference. The only minor weakness is unnecessary repetition: it converts 20 minutes to one-third of an hour and then converts it back without contributing to the calculation. This does not affect correctness, and the direct time arithmetic that follows is sound and fully supports the final answer.

Judge B: J-A (anthropic/claude-haiku-5.5) 8/10

Final answer 22:05 matches the reference and the programmatic verdict is correct. The arithmetic is shown step by step (19:45 + 2h = 21:45, + 20 min = 22:05). The opening detour converting 20 minutes to 1/3 hour and back is unnecessary padding, and the sentence claiming the total is 2 hours 20 minutes is redundant.

The response reaches the correct end time of 22:05 and presents the addition in a clear two-step sequence, adding the two hours first and then the twenty minutes, which is easy to verify. Its main weakness is an unnecessary detour at the start, where it converts 20 minutes into one-third of an hour and then converts it back to minutes. This adds length without contributing to the calculation. The final answer line is in the required format.

Nanbeige_Nanbeige4.3-33B gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 9/10

The response reaches the correct final answer, 22:05, and shows the appropriate hour and minute additions. The intermediate notation 21:65 is nonstandard, but it is explicitly normalized correctly to 22:05.

The response clearly decomposes the duration into two hours and twenty minutes, then correctly converts the resulting 65 minutes into one additional hour and five minutes. This provides a sound, traceable calculation and matches the required final format. Its only weakness is the temporary use of “21:65,” which is not a valid clock time, although the response immediately explains and corrects that notation. The final answer is unambiguous and appears on the last line as requested.

Judge B: J-A (anthropic/claude-haiku-5.5) 7/10

Final answer 22:05 matches the reference and the programmatic verdict is correct. The argument takes a shortcut: it writes 21:65 before carrying, which is not a valid clock notation, though the carry to 22:05 is explained correctly. Ends with the required ANSWER line.

The response reaches the correct end time of 22:05 and places it on the required final line. Its main strength is the clear step-by-step addition of two hours and then twenty minutes. Its weakness is the intermediate value 21:65, which is not a valid clock reading; the response does correct it by converting 65 minutes into one hour and five minutes, but the step is presented as if minutes can exceed 59 within a time. The argument is sound overall, with a small presentational gap in the carry step.

Openthinker 7B V2 gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 2/10

The response eventually identifies 22:05 through the minutes calculation and several explicit checks, but it repeatedly asserts the incorrect 22:50 result, remains unresolved, and ends without the required `ANSWER: <value>` line. The supplied verdict is incorrect.

The response’s strength is that it contains a valid conversion to minutes and repeatedly recognizes 22:05 as the mathematically correct end time. Its weakness is that it spends most of the output repeating an invalid minutes-carrying operation, alternates between contradictory conclusions, and never provides the requested final-answer line. The output also ends mid-thought, so the intended answer is not presented in the required format despite appearing in the reasoning.

Judge B: J-A (anthropic/claude-haiku-5.5) 0/10

The response is an extended, looping think-aloud that repeatedly contradicts itself (22:50, then 22:05, then back to 22:50) and is cut off mid-sentence without ever giving an ANSWER line. The programmatic verdict is incorrect and no usable final answer can be read.

The response attempts the correct method of converting times and adding the duration, and its minute-based conversion (1185 + 140 = 1325, giving 22:05) is actually the right arithmetic. However, it does not commit to that result and instead keeps re-checking and contradicting itself, arriving at 22:50 at several points. The output never reaches a final answer line and stops mid-sentence, so the task requirement of a clearly stated HH:MM answer is not met.

Phi3 3.8B Mini 128K Instruct gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 10/10

The response correctly adds 2 hours 20 minutes to 19:45, obtaining 22:05, and presents the required final-answer format on the last line.

The response directly addresses the time-calculation task and gives the correct end time. Its brief explanation is consistent with the arithmetic: 19:45 plus two hours reaches 21:45, and another 20 minutes reaches 22:05. It also follows the requested 24-hour HH:MM format and places the labeled answer on the final line. There are no substantive omissions, unsupported claims, or formatting problems in the observed output.

Judge B: J-A (anthropic/claude-haiku-5.5) 8/10

Verdict is correct: 19:45 plus 2h20m gives 22:05, matching the reference and the ANSWER line. The argument is a single unsupported sentence with no intermediate arithmetic shown, so the justification is minimal but the final answer is right and clearly stated.

The response gives the correct end time, 22:05, and places it on the required final ANSWER line in HH:MM format, which satisfies the task's output constraints. Its main strength is brevity and correct final formatting. Its weakness is that it offers no working: it asserts the result without showing how the 2 hours 20 minutes is added to 19:45, so a reader cannot check the reasoning from the response itself. The answer is right, but the supporting argument is thin.

YI Coder 1.5B Chat gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 3/10

The response reaches the incorrect final answer 21:05, despite correctly identifying the need to add 2 hours 20 minutes. The arithmetic should yield 22:05; the displayed Python code would also produce 22:05, so the claimed output reflects a final transcription or calculation error.

The response clearly identifies the relevant start time and duration and presents an appropriate datetime-based method. Its weakness is that the stated result, 21:05, is incorrect: adding 2 hours 20 minutes to 19:45 gives 22:05. Moreover, the code shown would calculate 22:05, contradicting the response’s claim about what the code outputs. The requested final-answer formatting is also not followed exactly because the answer appears within a sentence rather than as the final standalone line.

Judge B: J-A (anthropic/claude-haiku-5.5) 1/10

The response adds 2 hours 20 minutes to 19:45 and gets 21:05, which is wrong; the correct end time is 22:05. It also omits the 'datetime' import, so the code would fail, and it states the output without executing it. The answer line is incorrect, so the reasoning is wrong from the start.

The response correctly identifies that the task requires adding 2 hours and 20 minutes to the start time, and it uses a clear, tidy Python structure with a timedelta. Its main weakness is the arithmetic: 19:45 plus 2 hours 20 minutes is 22:05, not 21:05, so the final answer is wrong. The code also references datetime without importing it, so it would not run as written. The response asserts the printed output without evidence of execution, which adds no verification of the claimed result.

Apertus-8B-Instruct-2509 gap 1

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 10/10

The response correctly adds two hours and twenty minutes to 19:45, obtaining 22:05. It provides the required ANSWER line in the requested 24-hour HH:MM format.

The response clearly identifies the required time addition and performs it in two transparent steps: first adding two hours, then adding twenty minutes. This supports the correct ending time of 22:05 and satisfies the requested final-answer format. The response’s main strength is that its arithmetic is explicit and easy to verify. There is no substantive weakness; the brief explanatory detail is appropriate for the simple calculation, and the final line uses the exact requested ANSWER convention.

Judge B: J-A (anthropic/claude-haiku-5.5) 9/10

Verdict is correct and the final answer 22:05 matches the reference. The argument adds 2 hours to 19:45 to get 21:45, then 20 minutes to get 22:05, with each step shown. The ANSWER line is in the required form. Minor blemish: the working is slightly more verbose than necessary.

The response reaches the correct end time, 22:05, by adding the duration in two stages: first the two hours to reach 21:45, then the twenty minutes. Each intermediate step is stated explicitly, so the arithmetic can be checked directly. The final line follows the requested ANSWER format in 24-hour HH:MM. The main weakness is minor padding: the opening sentence and the restated start time add little beyond the calculation itself, though nothing in the reasoning is wrong or unsupported.

Aya Expanse 8B gap 1

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 3/10

The response correctly identifies that the duration must be added to 19:45, but it converts 20 minutes to an imprecise decimal and reports 22:08 instead of the correct 22:05. The final answer is therefore incorrect, though the overall approach is nearly right.

The response demonstrates the appropriate overall method by adding the two-hour, 20-minute duration to the stated start time. It also presents a clear final answer in the requested format. However, converting 20 minutes to 0.33 hours and then treating that approximation as exact produces 22:08, rather than the correct 22:05. The error is a small arithmetic and representation mistake at the final calculation, so the response comes close but does not satisfy the required answer.

Judge B: J-A (anthropic/claude-haiku-5.5) 2/10

The response converts 20 minutes to 2.33 hours, a decimal approximation, and adds it to 19:45 to get 22:08. The correct end time is 22:05, so the final answer is wrong. The flawed unit conversion is the error from the start, which places it in the wrong-answer band near the lower end.

The response attempts the task by adding the duration to the start time, which is the correct approach. However, it converts 20 minutes to a rounded decimal of 2.33 hours and then adds that to 19:45. This rounding introduces a drift of several minutes, producing 22:08 instead of the correct 22:05. The response does provide the required final answer line in the requested format, so the weakness is the arithmetic method rather than the structure of the output.

Codegeex4 9B All gap 1

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 3/10

The response correctly converts 2 hours 20 minutes to 140 minutes and recognizes that 45 + 140 minutes equals 3 hours 5 minutes. However, it then incorrectly adds 3:05 to 19:45, double-counting the starting 45 minutes, producing 22:50 instead of 22:05.

The response shows a sound initial approach by converting the duration into minutes and identifying the correct carry from 45 + 140 minutes. Its weakness is the final time addition: after deriving a three-hour-five-minute increment, it adds that increment to 19:45 without accounting for the already included 45 minutes, resulting in 22:50. The required final format is present, but the stated time is incorrect due to this arithmetic bookkeeping error.

Judge B: J-A (anthropic/claude-haiku-5.5) 2/10

The response correctly converts 2h20m to 140 minutes and correctly reaches 185 minutes past 19:45 as 3h05m. However, it then adds 3 hours 5 minutes to 19:45 incorrectly, giving 22:50 instead of 22:05. Final answer is wrong after a sound setup, which places it in the wrong band near the lower-middle range.

The response sets up the problem well: it converts the 2 hours 20 minutes duration into 140 minutes and correctly identifies that 185 minutes equals 3 hours 5 minutes. The main weakness is the final addition step, where 19:45 plus 3 hours 5 minutes is miscalculated as 22:50 rather than 22:05. The minutes component was mishandled when combining with the 45-minute start offset, so the stated end time is incorrect. The argument is clearly laid out, but the closing arithmetic error means the final answer does not match the correct end time.

Codegemma 2B Code V1.1 gap 1

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 1/10

The response gives 19:45, which is the movie's start time rather than the correct end time of 22:05. It provides no calculation or supporting reasoning.

The response does at least present an answer in the requested HH:MM style and includes an explicit ANSWER line. However, it repeats the starting time instead of adding the two-hour-20-minute runtime, so the final result is incorrect. The example text also adds unnecessary duplication without clarifying the calculation. Because the response reaches the wrong answer and offers no reasoning that could support a partial correction, its performance is substantially limited.

Judge B: J-A (anthropic/claude-haiku-5.5) 0/10

The response contains no reasoning and no attempt at the arithmetic. It only repeats an example line 'ANSWER: 19:45', which is the start time, not the end time. The verdict is incorrect and the answer is not derived from the task.

The response does not attempt the time calculation. It reproduces an example block and supplies 19:45 as the final answer, which is the movie's start time rather than the end time after a 2 hour 20 minute runtime. The output is incoherent relative to the task. The only element that matches the required format is the ANSWER line, though its value is wrong. No reasoning is shown that could be credited, and the response does not engage with the duration addition at all.

The other 198 models

Their judges were closer together. Their scores are listed here; open the model's page for the full answer.

ModelJudgesJudge AJudge BGap
Codegemma 7B Instruct V1.1 J-O + J-A 8 7 1
Codellama 7B Instruct J-O + J-A 6 7 1
Cogito 14B V1 Preview Qwen J-O + J-A 10 9 1
Cogito 3B V1 Preview Llama J-O + J-A 10 9 1
Deepcoder 1.5B Preview J-O + J-A 10 9 1
Deepscaler 1.5B Preview J-O + J-A 9 8 1
Deepseek Coder 6.7B Instruct J-O + J-A 2 1 1
Deepseek Llm 7B Chat J-O + J-A 2 1 1
Deepseek V2 16B Lite Chat J-O + J-A 2 1 1
DeepSeek-Coder-V2-Lite-Instruct J-O + J-A 3 2 1
DeepSeek-R1-0528-Qwen3-8B J-O + J-A 10 9 1
DeepSeek-R1-Distill-Qwen-14B J-O + J-A 10 9 1
DeepSeek-R1-Distill-Qwen-32B J-O + J-A 10 9 1
DeepSeek-R1-Distill-Qwen-7B J-O + J-A 10 9 1
Dolphin Phi 2.7B V2.6 J-O + J-A 2 1 1
Exaone Deep 7.8B J-O + J-A 9 8 1
Exaone3.5 2.4B Instruct J-O + J-A 10 9 1
Falcon3 1B Instruct J-O + J-A 2 1 1
Falcon3 3B Instruct J-O + J-A 10 9 1
Gemma 2B Instruct V1.1 J-O + J-A 1 0 1
Gemma 7B Instruct V1.1 J-O + J-A 10 9 1
gemma-2-9b-it J-O + J-A 2 1 1
gemma-4-26B-A4B-it J-O + J-A 10 9 1
Gemma2 27B Instruct J-O + J-A 3 2 1
Gemma2 2B Instruct J-O + J-A 3 2 1
Gemma3 1B IT J-O + J-A 3 2 1
Gemma4 E2B IT J-O + J-A 10 9 1
GLM-4.7-Flash J-O + J-A 10 9 1
Glm4 9B Chat J-O + J-A 10 9 1
gpt-oss-120b J-A + J-D 7 6 1
gpt-oss-20b J-A + J-D 7 6 1
Granite Code 8B Instruct J-O + J-A 3 2 1
Granite3 Dense 2B Instruct J-O + J-A 10 9 1
Granite3 Dense 8B Instruct J-O + J-A 10 9 1
Granite3 Moe 1B Instruct J-O + J-A 1 0 1
Granite3.1 Dense 2B Instruct J-O + J-A 3 2 1
Granite3.1 Dense 8B Instruct J-O + J-A 3 2 1
Granite3.1 Moe 3B Instruct J-O + J-A 3 2 1
Granite3.2 2B Instruct J-O + J-A 1 2 1
Granite4 3B J-O + J-A 10 9 1
Granite4 Micro H J-O + J-A 10 9 1
Granite4.2 8B J-O + J-A 2 1 1
Hermes-3-Llama-3.1-8B J-O + J-A 3 2 1
Internlm2 1.8B Chat V2.5 J-O + J-A 3 2 1
Laguna XS 2.1 J-O + J-A 10 9 1
Lfm2 24B J-O + J-A 10 9 1
Lfm3 1.7B J-O + J-A 9 8 1
Llama-3.1-8B-Instruct J-O + J-A 10 9 1
Llama-3.3-70B-Instruct J-O + J-A 10 9 1
Mathstral 7B V0.1 J-O + J-A 8 9 1
Ministral 3 14B Instruct (2512) J-O + J-A 10 9 1
Ministral 3 8B Instruct 2512 J-O + J-A 8 7 1
Mistral Small 3.2 24B Instruct (2506) J-O + J-A 10 9 1
Mistral Small3.1 24B Instruct 2503 J-O + J-A 10 9 1
Mixtral 8x7B Instruct V0.1 J-O + J-A 2 1 1
Nemotron 3.5 Lightning 30B A3B J-O + J-A 10 9 1
Nemotron Mini 4B Instruct J-O + J-A 1 0 1
Olmo 3.1 32B Instruct J-O + J-A 8 9 1
Opencoder 1.5B Instruct J-O + J-A 1 2 1
Ornith 9B J-O + J-A 10 9 1
Ornith-1.5-9B J-O + J-A 10 9 1
Phi-3-mini-4k-instruct J-O + J-A 10 9 1
phi-4 J-O + J-A 10 9 1
Phi-4-mini-instruct J-O + J-A 10 9 1
Phi3 14B Medium 128K Instruct J-O + J-A 10 9 1
Qwen 14B Chat V1.5 J-O + J-A 10 9 1
Qwen 7B Chat V1.5 J-O + J-A 2 1 1
Qwen2 0.5B Instruct J-O + J-A 2 1 1
Qwen2 1.5B Instruct J-O + J-A 3 2 1
Qwen2 Math 1.5B Instruct J-O + J-A 7 8 1
Qwen2 Math 7B Instruct J-O + J-A 7 8 1
Qwen2-VL-7B-Instruct J-O + J-A 10 9 1
Qwen2.5 0.5B Instruct J-O + J-A 3 2 1
Qwen2.5 Coder 1.5B Instruct J-O + J-A 8 9 1
Qwen2.5-3B-Instruct J-O + J-A 10 9 1
Qwen2.5-Coder-32B-Instruct J-O + J-A 10 9 1
Qwen2.5-Coder-3B-Instruct J-O + J-A 10 9 1
Qwen2.5-VL-32B-Instruct J-O + J-A 10 9 1
Qwen3 0.6B J-O + J-A 3 2 1
Qwen3 1.7B J-O + J-A 10 9 1
Qwen3-14B J-O + J-A 10 9 1
Qwen3-30B-A3B J-O + J-A 10 9 1
Qwen3-32B J-O + J-A 10 9 1
Qwen3-4B J-O + J-A 10 9 1
Qwen3-4B-Instruct-2507 J-O + J-A 10 9 1
Qwen3-4B-Thinking-2507 J-O + J-A 9 8 1
Qwen3-8B J-O + J-A 10 9 1
Qwen3-Coder-30B-A3B-Instruct J-O + J-A 10 9 1
Qwen3.5 2B J-O + J-A 10 9 1
Qwen3.5-4B J-O + J-A 10 9 1
Qwen3.5-9B J-O + J-A 10 9 1
Qwen3.6-27B J-O + J-A 10 9 1
Qwen3.6-35B-A3B J-O + J-A 10 9 1
Qwen3.8-27B J-O + J-A 10 9 1
Sailor2 1B Chat J-O + J-A 1 2 1
Sailor2 8B Chat J-O + J-A 3 2 1
Smallthinker 3B Preview J-O + J-A 3 2 1
Smollm2 1.7B Instruct J-O + J-A 1 0 1
Smollm2 135M Instruct J-O + J-A 1 0 1
Smollm2 360M Instruct J-O + J-A 2 1 1
Stable Code 3B Instruct J-O + J-A 3 2 1
Stablelm Zephyr 3B J-O + J-A 3 2 1
Stablelm2 12B Chat J-O + J-A 1 0 1
YI 34B Chat V1.5 J-O + J-A 3 2 1
YI 6B Chat V1.5 J-O + J-A 3 2 1
YI 9B Chat V1.5 J-O + J-A 8 9 1
YI Coder 9B Chat J-O + J-A 2 1 1
Aya 8B 23 J-O + J-A 6 6 0
Aya Expanse 32B J-O + J-A 2 2 0
Codeqwen 7B Chat V1.5 J-O + J-A 2 2 0
Codestral 22B V0.1 J-O + J-A 7 7 0
Cogito 8B V1 Preview Llama J-O + J-A 10 10 0
Command R7B (12-2024) J-O + J-A 6 6 0
Deepcoder 14B Preview J-O + J-A 0 0 0
Deepseek Coder 1.3B Instruct J-O + J-A 1 1 0
Deepseek Coder 33B Instruct J-O + J-A 6 6 0
Deepseek R1 1.5B Qwen Distill J-O + J-A 8 8 0
Deepseek R1 8B Llama Distill J-O + J-A 10 10 0
Devstral 24B Small 2505 J-O + J-A 10 10 0
Devstral Small 2 24B Instruct (2512) J-O + J-A 10 10 0
dolphin-2.9.1-yi-1.5-34b J-O + J-A 10 10 0
Exaone Deep 2.4B J-O + J-A 3 3 0
EXAONE-3.5-7.8B-Instruct J-O + J-A 10 10 0
Exaone3.5 32B Instruct J-O + J-A 10 10 0
Falcon2 11B J-O + J-A 2 2 0
Falcon3 10B Instruct J-O + J-A 10 10 0
Falcon3 7B Instruct J-O + J-A 9 9 0
Gemma 4 12B IT J-O + J-A 9 9 0
gemma-3-12b-it J-O + J-A 10 10 0
gemma-3-27b-it J-O + J-A 10 10 0
gemma-3-4b-it J-O + J-A 10 10 0
gemma-4-31B-it J-O + J-A 10 10 0
Gemma3n E2B IT J-O + J-A 2 2 0
Gemma4 E4B IT J-O + J-A 9 9 0
Granite Code 3B Instruct J-O + J-A 2 2 0
granite-4.1-3b J-O + J-A 10 10 0
Granite3.1 Moe 1B Instruct J-O + J-A 1 1 0
Granite3.2 8B Instruct J-O + J-A 2 2 0
Granite3.3 2B J-O + J-A 10 10 0
Granite3.3 8B J-O + J-A 2 2 0
Granite4 1B H J-O + J-A 10 10 0
Granite4 350M H J-O + J-A 1 1 0
Granite4 Small H J-O + J-A 10 10 0
Granite4 Tiny H J-O + J-A 10 10 0
Granite4.1 30B J-O + J-A 10 10 0
Granite4.1 8B J-O + J-A 10 10 0
Granite4.2 30B J-O + J-A 10 10 0
Granite4.2 3B J-O + J-A 10 10 0
Internlm2 7B Chat V2.5 J-O + J-A 2 2 0
Lfm3 VL 2B J-O + J-A 10 10 0
Llama 4 Scout 17B-16E Instruct J-O + J-A 10 10 0
Llama-3.2-3B-Instruct J-O + J-A 10 10 0
Llama2 13B Chat J-O + J-A 2 2 0
Magistral Small (2506) J-O + J-A 0 0 0
medgemma-4b-it J-O + J-A 10 10 0
Medgemma1.5 4B IT J-O + J-A 10 10 0
Mistral 7B Instruct V0.3 J-O + J-A 2 2 0
Mistral Small 22B Instruct 2409 J-O + J-A 10 10 0
Mistral Small 24B Instruct 2501 J-O + J-A 10 10 0
Mistral-7B-Instruct-v0.2 J-O + J-A 1 1 0
Muse Glimmer 30B J-O + J-A 10 10 0
Nemotron 3.5 Bolt E4B J-O + J-A 10 10 0
Nemotron Cascade 2 30B A3B J-O + J-A 6 6 0
Nemotron3 33B J-O + J-A 10 10 0
NVIDIA Nemotron 3 Nano 30B-A3B J-O + J-A 10 10 0
Olmo-3-7B-Instruct J-O + J-A 10 10 0
Olmo-3-7B-Think J-O + J-A 0 0 0
Olmo2 13B 1124 Instruct J-O + J-A 10 10 0
Olmo2 7B 1124 Instruct J-O + J-A 1 1 0
Phi4 Mini Reasoning 3.8B J-O + J-A 8 8 0
Qwen 0.5B Chat V1.5 J-O + J-A 1 1 0
Qwen 1.8B Chat V1.5 J-O + J-A 1 1 0
Qwen 4B Chat V1.5 J-O + J-A 1 1 0
Qwen2.5 1.5B Instruct J-O + J-A 2 2 0
Qwen2.5 Coder 0.5B Instruct J-O + J-A 1 1 0
Qwen2.5-14B-Instruct J-O + J-A 10 10 0
Qwen2.5-32B-Instruct J-O + J-A 10 10 0
Qwen2.5-7B-Instruct J-O + J-A 10 10 0
Qwen2.5-Coder-14B-Instruct J-O + J-A 10 10 0
Qwen2.5-Coder-7B-Instruct J-O + J-A 10 10 0
Qwen2.5-VL-3B-Instruct J-O + J-A 10 10 0
Qwen2.5-VL-7B-Instruct J-O + J-A 10 10 0
Qwen3 VL 2B Instruct J-O + J-A 10 10 0
Qwen3-30B-A3B-Instruct-2507 J-O + J-A 10 10 0
Qwen3-Coder-Next J-O + J-A 10 10 0
Qwen3-VL-30B-A3B-Instruct J-O + J-A 10 10 0
Qwen3-VL-32B-Instruct J-O + J-A 10 10 0
Qwen3-VL-4B-Instruct J-O + J-A 10 10 0
Qwen3-VL-8B-Instruct J-O + J-A 10 10 0
Qwen3.5-27B J-O + J-A 10 10 0
Qwen3.5-35B-A3B J-O + J-A 10 10 0
QwQ 32B J-O + J-A 6 6 0
Sage-A1.5-Q4_K_M J-O + J-A 9 9 0
SmolLM3-3B J-O + J-A 10 10 0
Solar 10.7B Instruct V1 J-O + J-A 3 3 0
Spark-X2.6-4B J-O + J-A 10 10 0
Stablelm2 1.6B Chat J-O + J-A 1 1 0
Translategemma 12B IT J-O + J-A 10 10 0

Withheld

These models are kept out of the ranking, so none of their scores or reasons are shown:

The judge room names the judges and shows where they disagree most.