← Judged questions

coding-010: both judges

A public question in Coding, battery v1. It starts: “Write a function that counts islands in a grid of `0`s (water) and `1`s (land), where an island is a maximal group of `1`s connected 4-direc…” Read the question and every model's answer.

228 models judged; 3 disputed (judges 3 or more points apart). The gap is the distance between the two judges' scores. Models are ordered by that gap, widest first.

Smallthinker 3B Preview gap 5, disputed

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

The provided function passes all 8 tests and correctly performs 4-directional DFS, handles empty grids, and avoids mutating the input. However, the response violates the required output shape by including extensive analysis and explanations instead of only one Python code block.

The response’s main strength is a correct, self-contained DFS implementation that uses a visited matrix, preserves the input grid, and handles empty input appropriately. Its main weakness is substantial noncompliance with the requested format: it includes lengthy reasoning, examples, headings, explanatory prose, and a code block rather than returning only the function in one code block. The implementation is reasonably clear, but the recursive approach could also encounter recursion-depth issues on very large islands, although the supplied tests do not expose that limitation.

Judge B: J-A (anthropic/claude-haiku-5.5) 2/10

All 8 tests passed per the programmatic verdict, so the response falls in the top band. However the output violates the explicit instruction to return only one code block with no explanation: it contains extensive visible reasoning, examples, and a second Final Solution section with a prose Explanation. Code itself is correct but the format violation and leaked reasoning lower the score.

The response produces a correct DFS-based island counter with a visited matrix and boundary checks, and the code passes all eight tests. Its main weakness is format: the task asked for only one Python code block with no explanation, yet the output contains lengthy reasoning, multiple test walkthroughs, a duplicated code block, and a prose explanation section. The reasoning also dwells on unnecessary alternatives such as sets and tuples, which adds length without improving the answer.

Openthinker 7B V2 gap 4, disputed

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 8/10

The BFS implementation is correct and passed all 8 tests, including empty grids and four-directional connectivity. However, the response violated the required output shape by including extensive reasoning and explanation around the code instead of returning only one Python code block.

The response provides a valid breadth-first traversal, marks discovered land cells to prevent duplicate visits, handles empty inputs, and uses the required four directions. Its main weakness is instruction compliance: it includes a long hidden-style reasoning section, explanatory prose, headings, and a code block rather than returning only the requested code block. The implementation also mutates the input grid, which was not prohibited but could be an implicit design concern if callers expect the input to remain unchanged.

Judge B: J-A (anthropic/claude-haiku-5.5) 4/10

All 8 tests passed, placing the response in the 7-10 band, but the output leaks a long visible reasoning block and includes extensive prose plus a second explanation despite the instruction to return only one code block with no explanation. The code itself is correct and idiomatic, but the format violation is significant, so the score is lowered to 4 to reflect the rubric's instruction-following concerns within the all-pass band.

The response implements a correct BFS-based island counter that marks visited land cells in place, handles empty grids, and uses 4-directional neighbours as specified, and all eight tests pass. Its main weakness is format: the task asked for only one Python code block with no explanation, yet the output contains a long thinking section, a multi-paragraph approach description, and an explanation after the code. It also includes a leaked thinking tag. The code block itself is clean, but the surrounding prose violates the explicit output-shape instruction.

Exaone Deep 2.4B gap 3, disputed

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

The BFS implementation is correct and passed all 8 tests, including four-directional connectivity, empty grids, and preserving the input via a copy. However, the response violated the required output shape by including extensive reasoning outside the single Python code block.

The response provides a functioning iterative BFS solution that marks visited land and correctly counts each four-directionally connected component. It also avoids mutating the caller’s grid, which is a reasonable implementation choice. The main weakness is instruction compliance: instead of returning only one code block, the response includes a long internal discussion followed by the code. The implementation itself is clean enough and handles the tested cases, but the excess text and exposed reasoning prevent full credit for output format.

Judge B: J-A (anthropic/claude-haiku-5.5) 4/10

All 8 tests passed per the programmatic verdict, placing the response in the 7-10 band. However, the output includes a visible reasoning block (thought tags) and a long preamble before the code, violating the instruction to return only one code block with no explanation. Code itself is clean BFS with a copy to avoid mutating input. Format violation and leaked reasoning reduce the score to the lower end of the band.

The response produces a correct BFS-based count_islands that copies the input grid so the caller's data is not mutated, and it handles empty grids and the 4-directional connectivity requirement. Its main weakness is format: the output exposes a long reasoning transcript inside thought tags before the code block, despite the instruction to return only one Python code block with no explanation. The code block itself is idiomatic and compact, but the surrounding extraneous text means the response does not follow the requested output shape.

Codegeex4 9B All gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

All 8 tests passed, and the DFS correctly marks visited land and uses only four directional neighbors. However, it directly accesses grid[0] during iteration, so an empty grid would raise IndexError, and recursive traversal can exceed Python's recursion limit on large islands.

The response provides a concise, idiomatic recursive flood-fill implementation that correctly counts four-directionally connected components and avoids revisiting cells by changing land to water. Its main weakness is an unhandled empty-grid case: both the loop bounds and DFS boundary check depend on grid[0] existing. Recursive DFS also imposes a scalability limitation for very large connected regions. These issues were not exposed by the supplied tests, which all passed, but they reduce robustness beyond the tested examples.

Judge B: J-A (anthropic/claude-haiku-5.5) 9/10

All 8 tests pass. The solution uses a recursive DFS that marks visited land cells in place, correctly applying 4-directional connectivity and bounds checks. It returns a single code block as requested. Mutating the input grid is a minor side effect, and recursion depth could be an issue on very large grids, so it is not a 10.

The response implements island counting with a depth-first search that flips visited land cells to water, which correctly enforces 4-directional connectivity and includes bounds checks. It passes the full programmatic test set and returns only a single code block, as instructed. Its main weakness is that it mutates the caller's input grid as a side effect, which may be unexpected for callers who reuse the grid. The recursive approach could also hit Python's recursion limit on very large contiguous land masses, a risk an iterative traversal would avoid.

Codegemma 7B Instruct V1.1 gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 6/10

The function correctly performs 4-directional DFS, marks visited land, and returns correct counts for standard rectangular nonempty grids. It passed 7 of 8 tests, placing it in the partial-credit band. The likely uncovered weakness is direct access to grid[0], which fails on an empty grid.

The response uses a straightforward recursive depth-first search and correctly restricts connectivity to the four orthogonal neighbors. It also avoids recounting land by marking visited cells as water, which supports the demonstrated examples. However, it assumes that the grid contains at least one row because it immediately evaluates len(grid[0]); an empty-grid test would therefore raise an exception. The function also mutates the caller’s input grid, which may be undesirable if preservation was expected, although that behavior is not necessarily prohibited by the task.

Judge B: J-A (anthropic/claude-haiku-5.5) 4/10

Programmatic verdict is 7/8 passed, placing the response in the partial band (3-6). The code is a correct-looking DFS flood fill that mutates the input grid and handles empty-row grids poorly: grid[0] raises IndexError on an empty list. The single failing test likely hit that edge case. Single code block returned as requested.

The response implements a standard recursive depth-first flood fill over 4-directional neighbours, which is the correct approach for counting islands and passes most of the test cases. It returns only one Python code block, matching the instruction. Its main weakness is that it indexes grid[0] without checking for an empty grid, so an input like an empty list fails. It also zeroes out the caller's grid in place, a side effect the task did not request, and recursion depth can overflow on large all-land grids.

Codellama 7B Instruct gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 3/10

The response defines the requested signature and correctly scans cells, but calls an undefined `dfs`, so land-containing inputs fail. It also assumes a nonempty grid. The verdict reports only 1/8 tests passed, placing it at the low end of the partial-credit band.

The response has the strength of using a visited set and iterating over every grid position, which reflects the intended island-counting structure. Its central weakness is that no `dfs` function is defined, causing a runtime error whenever an unvisited land cell is encountered. The implementation also accesses `grid[0]` without handling an empty grid. Additionally, it does not follow the requested output shape because the function is not enclosed in a single Python code block.

Judge B: J-A (anthropic/claude-haiku-5.5) 1/10

The verdict is 1/8 passed, so the band is 0-2 (runs but mostly fails). The code calls dfs, which is never defined, so every grid containing a 1 raises NameError. Only the all-water case returns correctly, consistent with the single passed test.

The response sets up a reasonable island-counting loop that scans every cell, skips visited land, and increments a count per new island. Its main weakness is that it calls a helper named dfs that is never defined anywhere in the output, so any grid containing land raises a NameError. The visited set is also never populated by the traversal because the helper is absent. As submitted, the function only works for grids with no land, which is why it passes only the trivial case.

Cogito 3B V1 Preview Llama gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 4/10

The function runs and handles empty or all-water grids, but its start condition requires a land cell to have no adjacent land. Consequently, multi-cell islands are never counted, and most nontrivial grids return incorrect results. The 2/8 test result places it in the partial-credit band.

The response correctly defines the requested function, handles empty grids, and uses four-directional DFS to mark visited land. However, the outer-loop condition is fundamentally incorrect: it starts DFS only for cells with no adjacent land, so connected islands containing multiple cells are omitted rather than counted once. The implementation also mutates the input grid, which may be undesirable if preservation was implicitly expected, though the primary failure is the island-detection condition. It additionally omits the requested Python code-block formatting.

Judge B: J-A (anthropic/claude-haiku-5.5) 2/10

Verdict is 2/8 passed, placing the response in the 0-2 band. The code only starts a DFS from land cells with no land neighbours, so any island with more than one cell is never traversed or counted as one unit. It runs without crashing, and the wrong counting logic explains the low pass rate. Single-cell islands are counted correctly.

The response produces one code block with the required signature and runs without error on the grid inputs. Its main strength is a clean, recursive depth-first traversal that marks visited cells by overwriting them with zero. The core weakness is the start condition: it only launches a search from a land cell with no land neighbours, so connected groups larger than one cell are never discovered as islands. As a result, it only counts isolated single cells and misses most multi-cell islands, which is why only a small fraction of the tests pass.

Cogito 8B V1 Preview Llama gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 10/10

The response provides exactly one Python code block with the requested function. It correctly tracks visited cells, explores only four directions, handles an empty grid, and increments the count once per connected land component. All 8/8 hidden tests passed.

The response’s main strength is a direct, idiomatic depth-first search that correctly marks each land cell and restricts connectivity to up, down, left, and right. It also follows the requested output shape by returning only one code block and includes a sensible empty-grid check. A minor latent limitation is that recursive traversal could hit Python’s recursion limit on an unusually large connected island, although this does not affect the supplied tests or the stated basic requirements.

Judge B: J-A (anthropic/claude-haiku-5.5) 8/10

All 8 tests pass. The code is a clean DFS flood-fill with correct 4-directional neighbors, bounds checks, and a visited matrix; it handles the empty grid. Recursion depth could overflow on very large connected grids, a latent edge case. Output is a single Python code block as requested.

The response implements island counting with a recursive depth-first flood fill over four-directional neighbors, correctly bounding indices and tracking visited cells so each island is counted once. It passes every supplied test and returns zero for an empty grid. A weakness is that the recursive traversal can exceed Python's default recursion limit on large contiguous land masses, since an iterative stack or BFS would avoid that risk. The output is limited to one code block with no extraneous explanation, matching the instructions.

Deepseek Llm 7B Chat gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

All 8 tests passed, and the recursive DFS correctly marks visited land and follows only four orthogonal directions. However, the implementation assumes a nonempty grid: both the DFS bounds check and iteration access grid[0], so count_islands([]) raises IndexError. It also mutates the caller's grid.

The response provides a concise, functioning depth-first search that correctly counts four-directionally connected components and satisfies the requested single-code-block format. Its main weakness is the unconditional use of grid[0], which makes an empty input fail rather than return zero. The function also changes land cells to zero in the supplied grid, an unstated side effect that may be undesirable for callers. These issues were not exposed by the eight passing tests but limit robustness beyond the tested cases.

Judge B: J-A (anthropic/claude-haiku-5.5) 9/10

All 8 tests pass. The DFS flood-fill correctly marks 4-directionally connected land cells and counts each component once. It mutates the input grid in place, a minor side effect, and returns 0 for empty-water grids. Returns only one code block as requested with no explanation.

The response implements island counting with a recursive depth-first flood fill that visits up, down, left and right neighbours, which matches the 4-directional connectivity requirement. Every provided test passes, and the code is compact and readable. The main weakness is that it mutates the caller's input grid by overwriting visited land cells with zero, a side effect the task did not ask for and which could surprise callers who reuse the grid. Recursion depth may also become a concern on very large contiguous islands, though this is not exercised here.

dolphin-2.9.1-yi-1.5-34b gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

All 8 tests pass, and the recursive DFS correctly marks visited land and uses only four-directional neighbors. However, the response omits the explicitly required Python code block and would fail on an empty grid because it accesses grid[0].

The response provides a concise, readable DFS implementation that correctly counts four-directionally connected components and avoids revisiting cells by marking them as water. Its main weaknesses are output compliance and robustness: it returns bare code rather than the requested single Python code block, and the nested loops and boundary checks assume the grid has at least one row and one column. It also mutates the input grid, which may be undesirable if preservation was expected, though that behavior was not explicitly prohibited.

Judge B: J-A (anthropic/claude-haiku-5.5) 9/10

All 8 tests passed, so the score is in the 7-10 band. The code is a clean recursive DFS that marks visited land as 0 and counts connected components with 4-directional moves. It returns only one code block with no extra prose, as instructed. Recursion depth could be an issue on very large grids, a minor latent limitation, and the function mutates the input grid.

The response implements island counting with a depth-first search over four-directional neighbours, flood-filling visited land cells to zero and incrementing a counter per component. It passes all eight tests and follows the instruction to return only a single code block with no explanation. Its main weakness is that it mutates the caller's input grid in place, which may surprise callers who reuse the grid. Recursive DFS also risks hitting Python's recursion limit on very large contiguous land masses, an edge case the tests did not exercise.

Falcon3 10B Instruct gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

All 8 tests passed, and the DFS correctly marks land and explores only the four orthogonal directions. However, it assumes the grid is nonempty because it repeatedly accesses len(grid[0]), so an empty grid would raise an error. It also mutates the input grid.

The response provides a compact recursive flood-fill implementation that correctly counts 4-directionally connected components and matches the requested function signature. Its main weakness is that it does not handle an empty grid: both the iteration logic and DFS bounds check depend on grid[0] existing. The function also changes every visited land cell to zero, which may be undesirable if callers expect the input to remain unchanged, although the task does not explicitly prohibit mutation. The single code block satisfies the requested output shape.

Judge B: J-A (anthropic/claude-haiku-5.5) 9/10

All 8 tests pass. The code uses a recursive DFS that marks visited land cells as 0 and counts each unvisited 1 as a new island, correctly applying 4-directional connectivity. It returns a single code block as instructed. Recursion depth could be an issue on very large grids, a minor latent concern not exercised by tests. It also mutates the input grid, which the task did not forbid but is worth noting.

The response implements island counting with a recursive depth-first search that flood-fills land cells and increments a counter for each new component. Its four-direction neighbour logic matches the stated connectivity rule, and it returns exactly one Python code block with no extra prose, as requested. A weakness is that it mutates the caller's input grid in place to mark visited cells, which may be unexpected for callers who reuse the grid. Recursive depth also grows with island size, so very large contiguous landmasses could exceed Python's recursion limit. Neither issue was exercised by the passing tests.

Falcon3 3B Instruct gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

The function passes all 8 tests and correctly uses 4-directional depth-first search, marking visited land and counting connected components. However, it accesses len(grid[0]) without handling an empty grid, causing an IndexError for a plausible edge case; recursion may also exceed limits on very large islands.

The response correctly implements island counting with a nested DFS, checks the four required neighboring directions, and marks visited cells to avoid recounting. It also follows the requested output shape by providing a single Python code block containing only the function. A weakness is that both the scan and DFS assume the grid has at least one row and that every row is nonempty, so an empty grid raises an IndexError. The recursive implementation could additionally hit Python’s recursion limit on a sufficiently large connected island.

Judge B: J-A (anthropic/claude-haiku-5.5) 9/10

All 8 tests pass. The response uses a recursive DFS that marks visited land cells as 0 and counts each connected component of 1s via 4-directional neighbours, with correct bounds checks. It returns only one Python code block with the requested signature, as asked. Minor deduction: the recursive DFS can hit recursion limits on very large grids, an edge case the tests did not force.

The response implements island counting with a depth-first search that flood-fills each land component in place and increments a counter per component. It correctly restricts movement to up, down, left and right, matching the task's 4-connectivity requirement, and includes bounds checks for every neighbour. It follows the output instruction by providing a single code block with no added explanation. A weakness is that the recursive traversal may exceed Python's default recursion limit on very large contiguous land regions, and it mutates the caller's input grid without mentioning this. Overall the solution is concise and idiomatic.

Falcon3 7B Instruct gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

All 8 tests passed, and the function correctly uses four-directional traversal while marking visited land. However, it assumes the grid is nonempty: both the DFS bounds check and iteration access len(grid[0]), so an empty grid would raise an IndexError.

The response provides a concise recursive flood-fill implementation that correctly counts four-directionally connected components and avoids revisiting cells by changing land to water. It also follows the requested output shape with exactly one Python code block and no explanation. Its main weakness is the unhandled empty-grid case: accessing `len(grid[0])` fails when the input has no rows. The recursive approach may also encounter recursion-depth limits on unusually large connected regions, though the supplied tests do not expose that issue.

Judge B: J-A (anthropic/claude-haiku-5.5) 9/10

All 8 tests pass. The response uses a recursive DFS that marks visited land cells as 0, correctly counting 4-directionally connected components, with bounds checks. Code is concise and idiomatic. Recursion depth could be an issue on very large grids, and the function mutates the input, which the task did not forbid but is a minor side effect. Output is a single code block as requested.

The response implements island counting with a nested depth-first search that flood-fills each discovered land cell by setting it to zero, then increments a counter per component. Bounds checks are present and connectivity is restricted to the four orthogonal directions as specified. The code is compact and clear, and it follows the single-code-block instruction. A weakness is that it mutates the caller's input grid in place without mention, and its recursive approach could hit Python's recursion limit on very large contiguous land regions.

Gemma 2B Instruct V1.1 gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 4/10

The function runs and iterates through the grid, but it counts each land cell's 4-directional neighbors rather than discovering connected components. Isolated islands contribute zero, adjacent cells are counted repeatedly, and the result is therefore incorrect for most nontrivial grids. The 3/8 test result places it in the partial-credit band.

The response has the strength of using the requested function signature and checking only the four permitted directions, while remaining syntactically valid and concise. Its central weakness is that it never marks cells as visited or traverses an island as a component. Instead, it sums neighboring-land counts, which double-counts connections and fails on isolated cells, among other cases. It also assumes a nonempty rectangular grid, although the main correctness issue is the island-counting logic.

Judge B: J-A (anthropic/claude-haiku-5.5) 2/10

The verdict is 3/8, which is partial and places the response in the 3-6 band, but the code is logically wrong: it sums each land cell's neighbour count instead of counting connected components, so the example count_islands([[1,1,0],[0,1,0],[0,0,1]]) would return 6, not 2. Since the verdict says some tests passed, the score should sit at the bottom of the partial band, with 2 being outside it, so the closest valid score is 3.

The response attempts a grid scan over land cells with a four-directional neighbour check, which reflects the correct connectivity rule. However, it never performs a traversal such as depth-first or breadth-first search to mark visited cells, so it adds neighbour counts rather than counting distinct islands. This produces inflated totals on any grid with adjacent land cells. The code is short and runs without error, but it fails most structural cases that require component identification.

Gemma 7B Instruct V1.1 gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

All 8 tests passed, and the implementation correctly uses four-directional DFS while marking visited land. However, it assumes the grid is nonempty by accessing len(grid[0]), so an empty grid would raise an error despite being a natural edge case.

The response provides a concise, working DFS implementation that counts each connected component once and correctly excludes diagonal connections. It also follows the requested single-code-block output format. Its main weakness is that it directly evaluates len(grid[0]) in both traversal and recursion bounds, which fails for an empty grid. The function mutates the supplied grid as part of visitation, a common approach, but that side effect is not documented or avoided. The helper function is valid Python and supports the requested public signature.

Judge B: J-A (anthropic/claude-haiku-5.5) 9/10

All 8 tests pass, so the score sits in the all-pass band. The solution is a clean DFS flood fill that marks land as visited by setting cells to 0 and correctly uses 4-directional neighbours. It mutates the input grid, a side effect the task did not mention, and it recurses without an explicit stack, which may hit limits on very large islands. Output is a single code block as requested.

The response implements island counting with a depth-first flood fill over four-directional neighbours, which matches the task's connectivity rule and passes every supplied test. The code is concise and readable, with a separate helper function that keeps the main loop simple. A weakness is that it mutates the caller's grid in place by overwriting land cells with zeros, a side effect the specification never mentions. The recursive helper also risks exceeding Python's recursion limit on very large contiguous islands, which the response does not guard against.

gemma-3-4b-it gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 9/10

The function passes all 8 tests, uses 4-directional DFS correctly, handles an empty grid, and returns the island count. It follows the requested single-code-block format. Its main latent limitation is recursive depth on very large connected grids, and it mutates the input grid while marking visited cells.

The response provides a concise, idiomatic depth-first search that correctly marks visited land and explores only the four permitted neighboring directions. It also handles an empty grid without indexing errors and satisfies the requested output shape. A weakness is that visitation is recorded by changing the caller’s grid in place, which may be unexpected if input preservation is required. The recursive implementation can also exceed Python’s recursion limit for a sufficiently large connected island, although neither issue affects the supplied passing tests.

Judge B: J-A (anthropic/claude-haiku-5.5) 7/10

All 8 tests pass. The response uses a correct 4-directional DFS flood fill and returns one code block as requested. It mutates the caller's grid in place by setting visited cells to 0, and its recursive DFS can exceed Python's recursion limit on large land masses. These latent issues keep it from the top band.

The response implements a standard flood-fill approach that counts each unvisited land cell as a new island and recursively clears its connected component. The four-direction neighbour logic correctly excludes diagonals, and the output is a single Python code block with no extra prose, as instructed. Its main weakness is that it destroys the input by overwriting land cells with zeros, so the caller's grid is altered after the call. The recursive depth also grows with island size, so very large contiguous islands could raise a recursion error. An iterative stack or a copied visited set would avoid both problems.

Glm4 9B Chat gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

All 8 tests pass, and the DFS correctly uses only four-directional neighbors while marking visited land. However, it unconditionally accesses grid[0], so an empty grid raises IndexError; it also mutates the caller’s grid.

The response provides a compact recursive DFS that correctly identifies four-directionally connected components and satisfies the requested single-code-block format. Its traversal and visited marking are straightforward, and the perfect test result confirms the tested behavior. The main weakness is that both the DFS bounds check and outer loop depend on len(grid[0]), causing an empty grid to fail instead of returning zero. The implementation also modifies the input grid in place, which may be undesirable if callers expect the original data to remain unchanged, although the task does not explicitly prohibit mutation.

Judge B: J-A (anthropic/claude-haiku-5.5) 9/10

All 8 tests pass. The response is a correct recursive DFS that marks visited land cells as 0 and counts each connected component, using only 4-directional moves as required. It returns a single code block with no extra prose. Edge cases such as empty grids or very large grids (recursion depth) are not handled, which keeps it from a perfect score.

The response implements island counting with a depth-first search that flood-fills land cells in place and increments a counter per component. It correctly restricts movement to up, down, left and right, matching the task's connectivity rule, and it follows the instruction to return only one code block. A weakness is that the recursive search may exceed Python's recursion limit on very large contiguous land regions, and the function mutates the caller's input grid as a side effect, which the task did not mention or forbid.

Granite3.1 Moe 3B Instruct gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

All 8 tests passed, and the recursive DFS correctly marks visited land and follows only four directions. However, the implementation accesses len(grid[0]) without handling an empty grid, creating a latent crash for a plausible edge case.

The response provides a concise, functioning recursive flood-fill implementation that counts four-directionally connected components and conforms to the requested single-code-block format. Its main weakness is that both the traversal loops and DFS bounds rely on grid[0], so an empty grid raises an IndexError rather than returning zero. It also mutates the supplied grid while marking cells, which may be undesirable if callers expect the input to remain unchanged, although that behavior was not prohibited by the task.

Judge B: J-A (anthropic/claude-haiku-5.5) 9/10

All 8 tests pass. The DFS flood-fill correctly marks 4-directionally connected land cells as visited and counts each component once. The code is compact and idiomatic. It mutates the input grid, which the task did not forbid but is a minor side effect. Signature matches the requested form. Only one code block is returned, as instructed.

The response implements island counting with a recursive depth-first flood fill over 4-neighbour moves, incrementing a counter per unvisited land cell. Its main strength is that it passes the full test set with concise, readable code and returns exactly one Python code block as requested. Its main weakness is that it overwrites the caller's input grid in place to mark visited cells, a side effect the task did not mention and that could surprise callers reusing the grid. It also uses recursion, which could exceed the recursion limit on very large contiguous islands.

Granite4 1B H gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

All 8 tests passed, and the function correctly performs 4-directional flood fill while counting each discovered island once. However, it does not follow the requested output shape: the function is not enclosed in a Python code block. It also assumes a nonempty, rectangular grid by indexing grid[0].

The response correctly implements recursive depth-first search, marks visited land as water, and explores only the four permitted directions. This is supported by the complete test pass result. Its main weakness is instruction compliance: the task explicitly requested one Python code block, but the output contains plain text code without fencing. The implementation also directly accesses the first row, so an empty grid would raise an exception rather than return zero, although that case was not represented among the supplied tests.

Judge B: J-A (anthropic/claude-haiku-5.5) 9/10

All 8 tests pass. The code uses a standard DFS flood-fill over 4-directional neighbours, marking visited land as water to avoid revisits. It handles empty rows and out-of-bounds indices via the boundary check. It mutates the input grid, a side effect the task did not forbid but that a careful solution might avoid. Clean, idiomatic, single code block.

The response implements island counting with a recursive depth-first flood fill that scans each cell and launches a search from every unvisited land cell. Neighbour checks are limited to the four orthogonal directions, as the task requires, and bounds are checked before access. Its main weakness is that it mutates the caller's grid in place to mark visited cells, which the task did not request and which would surprise a caller reusing the input. Recursion depth could also become an issue on very large connected islands. The output is a single code block with no extra prose.

Granite4 350M H gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

The implementation passes all 8 tests and correctly performs 4-directional flood fill by marking visited land as water. However, it does not return the requested single Python code block and assumes the grid is nonempty, so an empty grid would raise an error.

The response correctly counts connected components using depth-first search and mutates visited cells to prevent recounting, which is sufficient for the tested cases. Its main weakness is adherence to the requested output shape: it provides plain code rather than one Python code block. The implementation also accesses `grid[0]` without handling an empty grid, creating a latent edge-case failure even though the supplied tests all pass. The separate helper function is valid Python but was not included within the requested function-only presentation.

Judge B: J-A (anthropic/claude-haiku-5.5) 9/10

All 8 tests pass. The code runs a DFS from each unvisited land cell, flood-filling 4-directionally and zeroing visited cells, which correctly counts islands. The helper dfs is defined at module level and the solution is a single code block with no prose. Minor mutation of the input grid is a side effect not requested, and recursion depth could be a concern on very large grids.

The response implements the standard flood-fill approach: it scans the grid, increments a counter on each land cell, and recursively marks the connected land cells as water so they are not counted again. It correctly restricts connectivity to the four orthogonal directions, as the task requires. Its main weakness is that it mutates the caller's input grid in place without mentioning this, which may surprise callers who reuse the grid. Recursive DFS also risks exceeding the recursion limit on very large contiguous land regions, though this is not exercised by the tests. The output follows the single-code-block instruction.

Granite4.1 30B gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

All 8 tests pass, and the recursive DFS correctly marks visited land and restricts traversal to four directions. However, the implementation assumes the first row is nonempty, so `count_islands([[]])` raises an IndexError; recursion depth may also fail on very large grids.

The response provides a concise, idiomatic recursive flood-fill implementation that correctly counts four-directionally connected components and follows the requested single-code-block format. It also handles an empty grid and avoids recounting visited cells. Its main weakness is the unchecked assumption that `grid[0]` contains columns, which makes a valid empty-row input fail. The recursive approach additionally has a practical depth limitation for unusually large or elongated islands, although the supplied tests do not expose these issues.

Judge B: J-A (anthropic/claude-haiku-5.5) 9/10

All 8 tests pass. The response uses an iterative-looking but recursive DFS that marks visited cells in place with -1, counting each 4-connected component once. It returns 0 for empty input and handles the stated examples. Single code block, no extra prose. Minor deduction: recursive DFS can overflow on very large grids and mutates the input.

The response implements island counting with a recursive depth-first search over four-directional neighbours, scanning the grid and incrementing a counter at each unvisited land cell. It correctly handles the empty-grid case and passes every supplied test. It follows the instruction to return a single code block with no explanation. A weakness is that the recursion can exceed Python's default recursion limit on large contiguous land regions, and the function mutates the caller's grid by overwriting land cells with -1 rather than using a separate visited structure.

Hermes-3-Llama-3.1-8B gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

All 8 tests passed. The function correctly uses 4-directional DFS, counts each unvisited land component once, and handles the shown cases. However, the response is plain Python rather than the requested single fenced Python code block, and recursive DFS can hit recursion limits on large islands.

The response’s strength is its straightforward traversal: it marks visited land as water and explores exactly the four permitted neighboring cells, which is consistent with the task and confirmed by all tests passing. Its main weakness is output-shape compliance, since it supplies raw code without the requested Python code fence. The recursive implementation also has a practical depth limitation for unusually large connected regions, although that issue was not exposed by the test suite. The solution otherwise remains concise and directly implements the requested signature.

Judge B: J-A (anthropic/claude-haiku-5.5) 9/10

All 8 tests passed. The response is a clean recursive DFS that marks visited land cells as 0 and counts connected components with 4-directional moves, as the task requires. It returns a single code block with no extra prose. Minor concern: it mutates the input grid and uses recursion that could overflow on very large grids, which the tests did not exercise.

The response implements island counting with a depth-first search that flood-fills each discovered land component and increments a counter. It correctly restricts movement to up, down, left and right, matching the task's connectivity rule, and passes every provided test. Its main weakness is that it mutates the caller's input grid in place, setting visited land cells to zero, which may be unexpected for a pure counting function. Recursion depth also grows with island size, so very large contiguous islands could exceed Python's recursion limit. The output shape follows the instruction to return only code.

Lfm3 1.7B gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 9/10

The iterative DFS correctly counts 4-directionally connected components, handles an empty grid, and passes all 8 tests. It returns only one Python code block as requested. The implementation mutates the input and marks the initial cell only after traversal, causing avoidable duplicate processing, but this does not affect correctness for standard rectangular grids.

The response provides a concise iterative depth-first search that correctly explores the four permitted neighboring directions and counts each discovered island once. It also follows the requested output shape by supplying a single Python code block without explanation. A minor weakness is that the starting land cell is not marked before the search begins, so neighboring cells may push that cell onto the stack again, creating redundant work. The function also assumes a rectangular grid, which is conventional for this task but is not explicitly validated.

Judge B: J-A (anthropic/claude-haiku-5.5) 7/10

All 8 tests pass and the iterative DFS correctly counts 4-directional components. The code mutates the caller's grid in place, setting visited land cells to 0, which is an unrequested side effect. The redundant `return` and the `rows > 0` guard after an earlier `if not grid` check add mild clutter.

The response implements an iterative depth-first search over 4-connected land cells and returns a single code block with no extra prose, as requested. It handles the empty grid and all-water cases correctly. Its main weakness is that it destructively modifies the input grid, zeroing land cells as it visits them, so the caller's data is altered after the call. A non-mutating version using a separate visited set would avoid this side effect. Some minor redundancy, such as the unnecessary return statement in the helper, also reduces cleanliness.

Lfm3 VL 2B gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 4/10

The function runs and correctly performs four-directional DFS for grids containing string values, but the task specifies integer 0/1 cells. Its comparisons against `'1'` and assignment of `'0'` cause the provided integer examples and most relevant tests to fail; only 2 of 8 tests passed.

The response has a clear recursive traversal structure, checks bounds, explores exactly four directions, and returns zero for an empty grid. Its central weakness is a type mismatch with the specified input: it tests for string `'1'` rather than integer `1`, so valid integer grids are treated as entirely water. It also mutates the input grid and uses recursion without addressing potentially deep grids, though the type mismatch is the primary correctness failure reflected in the test result.

Judge B: J-A (anthropic/claude-haiku-5.5) 2/10

The code compares grid cells to the string '1' while the task's grid uses integer 1s, so no land is ever detected and the function returns 0 for every input. The programmatic verdict of 2/8 passed is consistent with only the zero-island cases succeeding. The response is a single code block as requested.

The response produces a clearly structured depth-first search with boundary checks and a visited-marking scheme, and it follows the single-code-block output instruction. Its central weakness is a type mismatch: the land test compares cells against the string '1' rather than the integer 1 used in the specified grid, so land is never recognised and the counter stays at zero. As a result only the all-water cases pass, and the solution fails the basic island-counting behaviour the task requires.

Llama 4 Scout 17B-16E Instruct gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

All 8 tests pass, and the function correctly performs 4-directional traversal while marking visited land. However, it assumes the first row exists and is nonempty; an input such as [[]] raises an IndexError, an implied empty-grid edge case not handled.

The response provides a concise, idiomatic DFS implementation that counts connected components using only the required four directions and avoids revisiting cells by marking them as water. It also follows the requested single-code-block output format and passed all supplied tests. A weakness is that it checks only whether the outer grid is empty before accessing grid[0], so a grid containing an empty first row causes an IndexError. The recursive traversal can also hit recursion limits on sufficiently large, highly connected grids.

Judge B: J-A (anthropic/claude-haiku-5.5) 9/10

All 8 tests pass per the programmatic verdict. The code uses a recursive DFS that marks visited land cells as water, counting each 4-connected component once. It handles the empty grid and all-water cases. It returns a single code block with no extra prose, as requested. Recursion depth could be an issue on very large grids, a minor latent risk.

The response implements island counting with a recursive depth-first search that flood-fills each land component by setting visited cells to zero, which correctly restricts connectivity to four directions. It passes all eight tests and handles the empty-grid case explicitly. It follows the instruction to return only one Python code block with no explanation. A weakness is the recursive approach, which can exceed Python's default recursion limit on very large grids, and an iterative traversal would be more robust. The code also mutates the input grid, a side effect the task did not mention.

Llama-3.2-3B-Instruct gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

All 8 tests passed, and the DFS correctly marks visited land and uses only four-directional traversal. However, the response does not follow the required output shape: it omits the single Python code fence and supplies a separate helper definition rather than one code block.

The response implements a functioning depth-first search that counts each connected land component once, and its boundary checks support the stated rectangular-grid examples. The main weakness is format compliance: the task explicitly required only one Python code block, but the model returned raw code without fencing. The implementation also mutates the input grid and relies on a separately defined helper, which may be acceptable functionally but is less self-contained than a nested helper or iterative approach. The hidden tests nevertheless all passed.

Judge B: J-A (anthropic/claude-haiku-5.5) 9/10

All 8 tests pass. The code uses a correct 4-directional DFS that marks visited land as 0, counting each connected component once. It handles empty grids and returns a single code block with no extra prose. It mutates the input grid, a minor side effect that the tests did not penalise.

The response implements island counting with a straightforward recursive depth-first search over four orthogonal neighbours, correctly excluding diagonal connections and passing all eight tests. It handles the empty-grid case explicitly and returns only one code block, as instructed. A weakness is that it mutates the caller's input grid by overwriting visited land cells with zeros, which may surprise callers who reuse the grid. Recursion depth could also become a concern on very large contiguous land regions, though this is not exercised by the tests.

Mathstral 7B V0.1 gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 5/10

The implementation passes 6 of 8 tests but does not perform a flood fill: it only clears the immediately adjacent cells, so larger or branching islands can be counted incorrectly. It also violates the requested output shape by including explanations and multiple code blocks.

The response provides a callable function with the requested name and correctly handles some simple grids, including the supplied examples. Its main weakness is that the algorithm clears only a land cell and its four immediate neighbors rather than traversing the entire connected component, causing incorrect counts for more complex islands. The accompanying prose also claims recursive traversal that the code does not implement. Finally, the response includes explanatory text and test code despite the requirement to return one Python code block only.

Judge B: J-A (anthropic/claude-haiku-5.5) 3/10

Programmatic verdict is 6/8, placing the response in the partial band (3-6). The code is a non-recursive scan that zeroes only the four direct neighbours of each found land cell, not the full connected component, so larger islands get counted multiple times. It also contains extra prose and two code blocks where one was requested.

The response attempts a grid scan with neighbour-clearing logic, which passes the simple cases the tests exercise, including both given examples. Its main weakness is that it only clears immediate neighbours rather than flood-filling the whole connected island, so any island wider than two cells is overcounted. The prose preamble and a second test-code block also violate the instruction to return only one Python code block with no explanation. The text describes a recursive approach that the code does not actually implement, which is an inconsistency worth noting.

Mixtral 8x7B Instruct V0.1 gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 4/10

The function implements a recursive four-direction traversal, but it checks for string values ('1') while the task specifies integer 1 values, so it fails the standard integer-grid cases. It also assumes a nonempty grid and mutates the input. The output omits the required single Python code block.

The response correctly attempts depth-first search with boundary checks and four-directional movement, which reflects the intended island definition. However, its comparisons use string markers rather than the integer values specified by the task, causing the implementation to return incorrect results for the provided input format. It also accesses grid[0] without handling an empty grid and modifies the input grid in place. Finally, it does not follow the requested output shape because the function is not enclosed in a Python code block.

Judge B: J-A (anthropic/claude-haiku-5.5) 2/10

The code compares grid cells to the string '1' while the task's examples use integer 1, so no island is ever detected and it returns 0 for any grid containing integer land cells. Only 2/8 tests pass, consistent with failing the basic case; it runs without error but is wrong on the primary input format.

The response implements a standard depth-first search flood fill over the grid with 4-directional neighbours and a correct island-counting loop structure, which is the right algorithmic approach. However, its land check compares cells against the string '1' rather than the integer 1 used in the task examples, so integer grids are never recognised as land. As a result the function returns 0 for inputs that contain islands, and it fails most of the test cases. The code also mutates the input grid in place, which the task did not authorise.

Olmo 3.1 32B Instruct gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

The implementation passes all 8 tests and correctly performs 4-directional DFS, including empty-grid handling. However, it returns raw Python text rather than the single Python code block explicitly requested, and it mutates the input grid while using recursive traversal, which can hit recursion limits on very large islands.

The response correctly identifies islands using depth-first search and marks visited land, including handling empty grids and the required four neighbors. Its main weakness is output-shape compliance: the function is not enclosed in the one Python code block requested by the task. The implementation also modifies the caller’s grid in place and relies on recursion, creating potential usability and scalability concerns beyond the provided tests. These issues do not affect the reported test results but prevent the response from fully satisfying every instruction.

Judge B: J-A (anthropic/claude-haiku-5.5) 9/10

All 8 tests pass per the programmatic verdict. The code is a clean, idiomatic DFS flood-fill with 4-directional neighbours, bounds checks, and an early return for empty input. It mutates the input grid in place, which is a minor side effect the task did not forbid. Only one code block is returned, with no extra explanation, as instructed.

The response implements island counting with a recursive depth-first flood fill that visits only up, down, left and right neighbours, matching the connectivity rule in the task. It handles empty grids and includes bounds checks before marking cells visited. Its main weakness is that it overwrites the caller's grid in place to mark visited land, a side effect that a caller might not expect. Recursion depth could also become a concern on very large contiguous islands, though this is not exercised by the tests. The output follows the single-code-block instruction without added prose.

Olmo-3-7B-Instruct gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

The function passes all 8 tests and correctly counts 4-directionally connected land using DFS and a visited matrix. However, it does not follow the required output shape: the function is not enclosed in a single Python code block.

The response correctly implements island counting with explicit bounds checks, visited tracking, and recursive traversal in all four permitted directions. It also handles empty grids and leaves the input unchanged. Its main weakness is format compliance: the task explicitly required one Python code block, but the output contains bare Python without code fences. The recursive approach also has a possible recursion-depth limitation on very large connected grids, although the supplied tests all pass.

Judge B: J-A (anthropic/claude-haiku-5.5) 9/10

All 8 tests pass. The function uses a recursive DFS with a visited matrix over 4-directional neighbours, correctly counting maximal land components and handling empty grids. It returns a single code block with no extra prose, as requested. Recursion depth could be an issue on very large grids, which the tests did not exercise, so it is not a 10.

The response implements island counting with a depth-first search that marks visited land cells and increments a counter for each unvisited land cell found during a row-major scan. It restricts movement to up, down, left and right, matching the specification, and returns zero for empty input. It follows the instruction to output only one code block. Its main weakness is the recursive DFS, which may exceed Python's default recursion limit on large contiguous land regions; an iterative stack or queue would be more robust. The code is otherwise concise and idiomatic.

The other 198 models

Their judges were closer together. Their scores are listed here; open the model's page for the full answer.

ModelJudgesJudge AJudge BGap
Phi3 14B Medium 128K Instruct J-O + J-A 7 9 2
Phi3 3.8B Mini 128K Instruct J-O + J-A 7 9 2
Qwen2 1.5B Instruct J-O + J-A 7 9 2
Qwen2.5-3B-Instruct J-O + J-A 9 7 2
Qwen2.5-Coder-7B-Instruct J-O + J-A 7 9 2
Qwen2.5-VL-3B-Instruct J-O + J-A 7 9 2
Qwen3 0.6B J-O + J-A 3 1 2
Qwen3-8B J-O + J-A 7 9 2
Qwen3-VL-8B-Instruct J-O + J-A 9 7 2
Sailor2 1B Chat J-O + J-A 3 1 2
Smollm2 135M Instruct J-O + J-A 3 1 2
Stablelm Zephyr 3B J-O + J-A 4 2 2
YI 34B Chat V1.5 J-O + J-A 7 9 2
YI 9B Chat V1.5 J-O + J-A 7 9 2
Aya 8B 23 J-O + J-A 8 9 1
Aya Expanse 8B J-O + J-A 7 8 1
Codellama 34B Instruct J-O + J-A 0 1 1
Codestral 22B V0.1 J-O + J-A 8 9 1
Command R 35B 08 2024 J-O + J-A 9 8 1
Command R7B (12-2024) J-O + J-A 7 8 1
Command R7B Arabic 7B 02 2025 J-O + J-A 7 8 1
Deepseek Coder 6.7B Instruct J-O + J-A 8 9 1
Deepseek R1 8B Llama Distill J-O + J-A 7 6 1
Deepseek V2 16B Lite Chat J-O + J-A 8 9 1
DeepSeek-Coder-V2-Lite-Instruct J-O + J-A 10 9 1
DeepSeek-R1-0528-Qwen3-8B J-O + J-A 7 6 1
DeepSeek-R1-Distill-Qwen-14B J-O + J-A 7 6 1
DeepSeek-R1-Distill-Qwen-7B J-O + J-A 7 6 1
Exaone Deep 7.8B J-O + J-A 7 6 1
EXAONE-3.5-7.8B-Instruct J-O + J-A 9 8 1
Falcon3 1B Instruct J-O + J-A 0 1 1
Flint 40B A2B J-O + J-A 8 9 1
Gemma 3n E4B IT J-O + J-A 8 9 1
Gemma 4 12B IT J-O + J-A 9 8 1
gemma-2-9b-it J-O + J-A 8 9 1
gemma-4-31B-it J-O + J-A 10 9 1
Gemma2 2B Instruct J-O + J-A 8 9 1
Gemma4 E2B IT J-O + J-A 9 8 1
Gemma4 E4B IT J-O + J-A 8 9 1
GLM-4.5-Air J-O + J-A 9 8 1
Granite Code 3B Instruct J-O + J-A 4 3 1
Granite Code 8B Instruct J-O + J-A 7 8 1
Granite3 Moe 1B Instruct J-O + J-A 3 2 1
Granite3 Moe 3B Instruct J-O + J-A 8 9 1
Granite3.1 Dense 8B Instruct J-O + J-A 9 8 1
Granite3.1 Moe 1B Instruct J-O + J-A 7 8 1
Granite3.2 8B Instruct J-O + J-A 9 8 1
Granite3.3 2B J-O + J-A 8 9 1
Granite3.3 8B J-O + J-A 8 9 1
Granite4 Small H J-O + J-A 8 9 1
Granite4 Tiny H J-O + J-A 8 9 1
Granite4.1 8B J-O + J-A 8 9 1
Granite4.2 30B J-O + J-A 10 9 1
Internlm2 1.8B Chat V2.5 J-O + J-A 8 9 1
Laguna XS 2.1 J-O + J-A 9 8 1
Lfm2.5 Thinking 1.2B J-O + J-A 8 9 1
Llama-2-7b-chat-hf J-O + J-A 3 2 1
Llama2 13B Chat J-O + J-A 0 1 1
Llama3.2 1B Instruct J-O + J-A 7 8 1
Marco O1 7B J-O + J-A 7 8 1
Medgemma1.5 4B IT J-O + J-A 10 9 1
Ministral 3 3B Instruct 2512 J-O + J-A 10 9 1
Ministral 3 8B Instruct 2512 J-O + J-A 10 9 1
Mistral Nemo 12B Instruct 2407 J-O + J-A 7 8 1
Mistral-7B-Instruct-v0.2 J-O + J-A 8 7 1
Nemotron 3.5 Bolt E4B J-O + J-A 8 9 1
Nemotron 3.5 Lightning 30B A3B J-O + J-A 8 9 1
Nemotron Cascade 2 30B A3B J-O + J-A 10 9 1
Nemotron Mini 4B Instruct J-O + J-A 8 9 1
Olmo2 13B 1124 Instruct J-O + J-A 3 2 1
Olmo2 7B 1124 Instruct J-O + J-A 3 2 1
Phi-3-mini-4k-instruct J-O + J-A 8 7 1
Phi-3.5-mini-instruct J-O + J-A 8 9 1
Phi4 Mini Reasoning 3.8B J-O + J-A 1 0 1
Qwen 1.8B Chat V1.5 J-O + J-A 4 3 1
Qwen2 Math 1.5B Instruct J-O + J-A 3 2 1
Qwen2 Math 7B Instruct J-O + J-A 8 9 1
Qwen2-VL-7B-Instruct J-O + J-A 8 9 1
Qwen2.5 1.5B Instruct J-O + J-A 8 9 1
Qwen2.5 Coder 0.5B Instruct J-O + J-A 8 9 1
Qwen2.5-14B-Instruct J-O + J-A 10 9 1
Qwen2.5-32B-Instruct J-O + J-A 7 8 1
Qwen2.5-Coder-32B-Instruct J-O + J-A 8 9 1
Qwen3 1.7B J-O + J-A 4 3 1
Qwen3 VL 2B Instruct J-O + J-A 10 9 1
Qwen3-4B-Instruct-2507 J-O + J-A 8 9 1
Qwen3-Coder-30B-A3B-Instruct J-O + J-A 10 9 1
Qwen3-Coder-Next J-O + J-A 10 9 1
Qwen3.5-35B-A3B J-O + J-A 8 7 1
Qwen3.5-9B J-O + J-A 10 9 1
Qwen3.6-27B J-O + J-A 9 8 1
Qwen3.6-35B-A3B J-O + J-A 10 9 1
Sage-A1.5-Q4_K_M J-O + J-A 8 9 1
Sailor2 8B Chat J-O + J-A 0 1 1
Smollm2 360M Instruct J-O + J-A 3 2 1
Solar 10.7B Instruct V1 J-O + J-A 3 2 1
Stablelm2 1.6B Chat J-O + J-A 3 2 1
Stablelm2 12B Chat J-O + J-A 4 3 1
Tulu3 8B J-O + J-A 6 7 1
Apertus-8B-Instruct-2509 J-O + J-A 6 6 0
Aya Expanse 32B J-O + J-A 9 9 0
Codegemma 2B Code V1.1 J-O + J-A 0 0 0
Codellama 13B Instruct J-O + J-A 0 0 0
Codeqwen 7B Chat V1.5 J-O + J-A 9 9 0
Cogito 14B V1 Preview Qwen J-O + J-A 9 9 0
Deepcoder 1.5B Preview J-O + J-A 0 0 0
Deepcoder 14B Preview J-O + J-A 7 7 0
Deepscaler 1.5B Preview J-O + J-A 0 0 0
Deepseek Coder 1.3B Instruct J-O + J-A 7 7 0
Deepseek Coder 33B Instruct J-O + J-A 7 7 0
Deepseek R1 1.5B Qwen Distill J-O + J-A 0 0 0
DeepSeek-R1-Distill-Qwen-32B J-O + J-A 0 0 0
Devstral 24B Small 2505 J-O + J-A 9 9 0
Devstral Small 2 24B Instruct (2512) J-O + J-A 9 9 0
Dolphin Phi 2.7B V2.6 J-O + J-A 0 0 0
Exaone3.5 2.4B Instruct J-O + J-A 9 9 0
Exaone3.5 32B Instruct J-O + J-A 9 9 0
Falcon2 11B J-O + J-A 0 0 0
gemma-3-12b-it J-O + J-A 9 9 0
gemma-3-27b-it J-O + J-A 9 9 0
gemma-4-26B-A4B-it J-O + J-A 9 9 0
Gemma2 27B Instruct J-O + J-A 9 9 0
Gemma3 1B IT J-O + J-A 9 9 0
Gemma3n E2B IT J-O + J-A 9 9 0
GLM-4.7-Flash J-O + J-A 9 9 0
gpt-oss-120b J-A + J-D 9 9 0
gpt-oss-20b J-A + J-D 9 9 0
granite-4.1-3b J-O + J-A 9 9 0
Granite3 Dense 2B Instruct J-O + J-A 9 9 0
Granite3 Dense 8B Instruct J-O + J-A 7 7 0
Granite3.1 Dense 2B Instruct J-O + J-A 9 9 0
Granite3.2 2B Instruct J-O + J-A 9 9 0
Granite4 3B J-O + J-A 9 9 0
Granite4 Micro H J-O + J-A 8 8 0
Granite4.2 3B J-O + J-A 9 9 0
Granite4.2 8B J-O + J-A 9 9 0
Hermes3 3B Llama3.2 J-O + J-A 0 0 0
Internlm2 7B Chat V2.5 J-O + J-A 9 9 0
Lfm2 24B J-O + J-A 9 9 0
Lfm2.5 8B A1B J-O + J-A 7 7 0
Llama-3.1-8B-Instruct J-O + J-A 9 9 0
Llama-3.3-70B-Instruct J-O + J-A 7 7 0
Magistral Small (2506) J-O + J-A 0 0 0
Medgemma 27B IT J-O + J-A 9 9 0
medgemma-4b-it J-O + J-A 9 9 0
Meta-Llama-3-8B-Instruct J-O + J-A 7 7 0
Ministral 3 14B Instruct (2512) J-O + J-A 9 9 0
Mistral 7B Instruct V0.3 J-O + J-A 7 7 0
Mistral Small 22B Instruct 2409 J-O + J-A 9 9 0
Mistral Small 24B Instruct 2501 J-O + J-A 9 9 0
Mistral Small 3.2 24B Instruct (2506) J-O + J-A 9 9 0
Mistral Small3.1 24B Instruct 2503 J-O + J-A 9 9 0
Muse Glimmer 30B J-O + J-A 9 9 0
Nanbeige_Nanbeige4.3-33B J-O + J-A 9 9 0
Nemotron3 33B J-O + J-A 9 9 0
NVIDIA Nemotron 3 Nano 30B-A3B J-O + J-A 9 9 0
Olmo-3-7B-Think J-O + J-A 9 9 0
Opencoder 1.5B Instruct J-O + J-A 7 7 0
Ornith 9B J-O + J-A 9 9 0
Ornith-1.5-9B J-O + J-A 9 9 0
Phi 2.7B Chat V2 J-O + J-A 7 7 0
phi-4 J-O + J-A 9 9 0
Phi-4-mini-instruct J-O + J-A 9 9 0
Qwen 0.5B Chat V1.5 J-O + J-A 3 3 0
Qwen 14B Chat V1.5 J-O + J-A 0 0 0
Qwen 4B Chat V1.5 J-O + J-A 0 0 0
Qwen 7B Chat V1.5 J-O + J-A 0 0 0
Qwen2 0.5B Instruct J-O + J-A 0 0 0
Qwen2.5 0.5B Instruct J-O + J-A 7 7 0
Qwen2.5 Coder 1.5B Instruct J-O + J-A 9 9 0
Qwen2.5-7B-Instruct J-O + J-A 9 9 0
Qwen2.5-Coder-14B-Instruct J-O + J-A 9 9 0
Qwen2.5-Coder-3B-Instruct J-O + J-A 8 8 0
Qwen2.5-VL-32B-Instruct J-O + J-A 9 9 0
Qwen2.5-VL-7B-Instruct J-O + J-A 8 8 0
Qwen3 VL 2B Thinking J-O + J-A 0 0 0
Qwen3-14B J-O + J-A 9 9 0
Qwen3-30B-A3B J-O + J-A 9 9 0
Qwen3-30B-A3B-Instruct-2507 J-O + J-A 9 9 0
Qwen3-32B J-O + J-A 9 9 0
Qwen3-4B J-O + J-A 9 9 0
Qwen3-4B-Thinking-2507 J-O + J-A 0 0 0
Qwen3-VL-30B-A3B-Instruct J-O + J-A 9 9 0
Qwen3-VL-32B-Instruct J-O + J-A 9 9 0
Qwen3-VL-4B-Instruct J-O + J-A 9 9 0
Qwen3.5 2B J-O + J-A 9 9 0
Qwen3.5-27B J-O + J-A 9 9 0
Qwen3.5-4B J-O + J-A 10 10 0
Qwen3.8-27B J-O + J-A 9 9 0
QwQ 32B J-O + J-A 9 9 0
Smollm2 1.7B Instruct J-O + J-A 4 4 0
SmolLM3-3B J-O + J-A 9 9 0
Spark-X2.6-4B J-O + J-A 9 9 0
Stable Code 3B Instruct J-O + J-A 0 0 0
Translategemma 12B IT J-O + J-A 9 9 0
YI 6B Chat V1.5 J-O + J-A 0 0 0
YI Coder 1.5B Chat J-O + J-A 7 7 0
YI Coder 9B Chat J-O + J-A 7 7 0

Withheld

These models are kept out of the ranking, so none of their scores or reasons are shown:

The judge room names the judges and shows where they disagree most.