gemma-3-12b-it against Gemma 4 12B IT

Pick two models to see them side by side: Overall score, every category, the questions where one did better, and speed and memory. Each model is shown at its newest tested version. Back to the leaderboard.

The short version

These are the two highest Overall scores among the ranked models in the 9–16B size class.

Gemma 4 12B IT scores higher overall: 98.6 against 92.7 out of 100, a gap of 5.9 points.

With the 95% intervals taken into account, Gemma 4 12B IT is clearly ahead of gemma-3-12b-it.

Across the 9 categories that count toward Overall, gemma-3-12b-it is higher in 0, Gemma 4 12B IT is higher in 6 and 3 are level.

The widest gap is in Coding: 81.8 for gemma-3-12b-it against 98.2 for Gemma 4 12B IT, in favour of Gemma 4 12B IT.

Of the 259 questions both models have a definite score for, gemma-3-12b-it did better on 1, Gemma 4 12B IT did better on 32, and 226 were level.

On the hard questions, which are scored within one run, gemma-3-12b-it scores 66.5 and Gemma 4 12B IT scores 90.7.

On the same backend and Ollama version, with one request at a time, gemma-3-12b-it generates 6.1 tokens per second and Gemma 4 12B IT generates 8.1.

Written from the numbers below. Scores come from one fixed battery run on one laptop; read the methodology before drawing conclusions.

Overall

Gemma 4 12B IT scores higher: 98.6 against 92.7, a gap of 5.9 points. With the 95% intervals taken into account, Gemma 4 12B IT is clearly ahead of gemma-3-12b-it. How sure is a score?

Overall (0–100) with its 95% interval, the rank range among all listed ranked models, and the hard-question score.
Measuregemma-3-12b-itGemma 4 12B IT
Overall92.789.7–95.498.697.0–99.8
Rank range8–661–12
Hard questionsmean score on the questions that half or fewer of the ranked models of 3B and over answered correctly66.590.7
Size class9–16B9–16B

By category

Programmatic mean (0–100) per category. Higher is the model with the larger mean; a gap that rounds to 0.0 is called level. A dash means not scored, which is different from 0.
Categorygemma-3-12b-itGemma 4 12B ITHigherDifference
Reasoningcounts toward Overall 88.6 94.3 Gemma 4 12B IT 5.7
Mathcounts toward Overall 96.0 100.0 Gemma 4 12B IT 4.0
Codingcounts toward Overall 81.8 98.2 Gemma 4 12B IT 16.4
Instruction followingcounts toward Overall 88.3 100.0 Gemma 4 12B IT 11.7
Knowledge & honestycounts toward Overall 96.7 100.0 Gemma 4 12B IT 3.3
Long contextcounts toward Overall 83.3 94.4 Gemma 4 12B IT 11.1
Multilingualcounts toward Overall 100.0 100.0 Level 0
Tool usecounts toward Overall 100.0 100.0 Level 0
Safetycounts toward Overall 100.0 100.0 Level 0
Pushback resistanceshown separately 91.7 100.0 Gemma 4 12B IT 8.3
Famous gotchasshown separately 66.7 93.3 Gemma 4 12B IT 26.6
Writingshown separately — — — —
Self-descriptionshown separately — — — —

Speed and memory

Measured on one laptop. Decode speed is generation speed with one request at a time.
Measuregemma-3-12b-itGemma 4 12B IT
Decode speed, one stream (tok/s)6.18.1
GPU memory on our server (1 to 8 answers at a time) (GB)13.18.9
Model size as loaded on our server (GB)13.18.9
BackendROCm 7.1 (Ollama 0.34.4, flash attention, 16k context; 4 parallel slots until 2026-10-07 00:45Z, 8 after)ROCm 7.1 (Ollama 0.34.4, flash attention, 16k context; 4 parallel slots until 2026-10-07 00:45Z, 8 after)

A dash means the figure was not measured, which is different from 0. Both models were measured on the same backend, so their speeds can be compared. Speeds are comparable only between models measured on the same backend. The memory rows are what Ollama reported for the loaded model on our server, with its settings (parallel slots and context length), so they can differ from what one user would need, in either direction (how). They are not the memory estimate the hardware picker uses. See also the Speed page.

Where they differ

Both models have a definite score for 259 questions. gemma-3-12b-it did better on 1, Gemma 4 12B IT did better on 32, and 226 were level. A score gap of more than 0.001 counts as a win. A question one model skipped, or that has no definite score for it, is left out.

Questions both models have a score for, by category.
CategoryQuestionsgemma-3-12b-it betterGemma 4 12B IT betterLevel
Reasoning351331
Math250124
Coding3001119
Instruction following300921
Knowledge & honesty300129
Long context180216
Multilingual200020
Tool use200020
Safety240024
Pushback resistance120111
Famous gotchas150411

Questions where only one got full marks

Full marks means a score of 1. The answers of public questions can be opened below. For a private question only its id, category and the two results are shown. Each result links to that question on the model's page.

Reasoning

Reasoning: 4 questions where exactly one model got full marks.
Questiongemma-3-12b-itGemma 4 12B IT
reasoning-018publicall models' answers fail on its page pass on its page
Show both answers to reasoning-018

gemma-3-12b-it

Let A, B, C, and D represent Amy, Ben, Cara, and Dev, respectively.
We are given the following information:
\begin{itemize}
    \item Amy sits immediately to the left of Ben. This means that if Amy sits in seat $i$, then Ben sits in seat $i+1$. So, A and B are consecutive.
    \item Cara sits in seat 1. So, C is in seat 1.
    \item Dev does not sit in seat 4. So, D is not in seat 4.
\end{itemize}
Since Cara sits in seat 1, we have C in seat 1.
Since Amy sits immediately to the left of Ben, we have A and B as consecutive.
The possible positions for A and B are (2, 3) or (3, 4).
If A is in seat 2 and B is in seat 3, then the seats are:
Seat 1: C
Seat 2: A
Seat 3: B
Seat 4: D
Since Dev does not sit in seat 4, this is a valid arrangement.
If A is in seat 3 and B is in seat 4, then the seats are:
Seat 1: C
Seat 2: ?
Seat 3: A
Seat 4: B
But Dev must sit in one of the remaining seats, so Dev sits in seat 2.
However, we are given that Dev does not sit in seat 4, so this is a valid arrangement.
In the first case, the seats are C, A, B, D.
In the second case, the seats are C, D, A, B.
Since Dev does not sit in seat 4, the second case is not possible.
Therefore, the arrangement is C, A, B, D.
The seat number of Ben is 3.

Now, let's consider the second case:
Seat 1: C
Seat 2: D
Seat 3: A
Seat 4: B
Dev does not sit in seat 4, so this is a valid arrangement.
In this case, the seats are C, D, A, B.
The seat number of Ben is 4.
However, we are given that Dev does not sit in seat 4.
So, we must have A and B in seats 2 and 3, or in seats 3 and 4.
If A and B are in seats 2 and 3, then the arrangement is C, A, B, D.
If A and B are in seats 3 and 4, then the arrangement is C, D, A, B.
Since Dev does not sit in seat 4, the arrangement must be C, A, B, D.
In this case, Ben sits in seat 3.
If the arrangement is C, D, A, B, then Dev sits in seat 2, Amy sits in seat 3, Ben sits in seat 4.
But Dev does not sit in seat 4, so this is possible.
However, we are given that Dev does not sit in se

Cut here for length. The model page has the full answer.

Gemma 4 12B IT

To find the seat number for Ben, let's analyze the given information step by step:

1.  **Seats available:** 1, 2, 3, 4.
2.  **Cara's position:** "Cara sits in seat 1."
    *   Current arrangement: [Cara, \_, \_, \_]
3.  **Amy and Ben's positions:** "Amy sits immediately to the left of Ben."
    *   This means Amy and Ben must occupy two adjacent seats in the order (Amy, Ben).
    *   Since seat 1 is occupied by Cara, the available pairs of adjacent seats are (2, 3) and (3, 4).
4.  **Dev's position:** "Dev does not sit in seat 4."
    *   Let's test the two possible scenarios for Amy and Ben:

    *   **Scenario A: Amy and Ben are in seats 2 and 3.**
        *   Seat 1: Cara
        *   Seat 2: Amy
        *   Seat 3: Ben
        *   Seat 4: Dev
        *   *Check:* Does this violate any rules? The rule says "Dev does not sit in seat 4." In this scenario, Dev is in seat 4. Therefore, Scenario A is **incorrect**.

    *   **Scenario B: Amy and Ben are in seats 3 and 4.**
        *   Seat 1: Cara
        *   Seat 2: Dev
        *   Seat 3: Amy
        *   Seat 4: Ben
        *   *Check:*
            *   Cara is in seat 1? Yes.
            *   Amy is immediately to the left of Ben? Yes (Amy is 3, Ben is 4).
            *   Dev is not in seat 4? Yes (Dev is in seat 2).
        *   This scenario satisfies all conditions.

In Scenario B, Ben sits in seat 4.

ANSWER: 4
reasoning-025private fail on its page pass on its page
reasoning-030private pass on its page fail on its page
reasoning-035private fail on its page pass on its page

Math

Math: 1 question where exactly one model got full marks.
Questiongemma-3-12b-itGemma 4 12B IT
math-008private fail on its page pass on its page

Coding

Coding: 9 questions where exactly one model got full marks.
Questiongemma-3-12b-itGemma 4 12B IT
coding-009private 82% on its page pass on its page
coding-011private fail on its page pass on its page
coding-018private 89% on its page pass on its page
coding-019private 38% on its page pass on its page
coding-022private 45% on its page pass on its page
coding-023private 89% on its page pass on its page
coding-025private 22% on its page pass on its page
coding-027private 82% on its page pass on its page
coding-030private 27% on its page pass on its page

Instruction following

Instruction following: 9 questions where exactly one model got full marks.
Questiongemma-3-12b-itGemma 4 12B IT
instruction-005private 67% on its page pass on its page
instruction-009private 67% on its page pass on its page
instruction-010private 75% on its page pass on its page
instruction-013private 50% on its page pass on its page
instruction-017private 50% on its page pass on its page
instruction-018private 60% on its page pass on its page
instruction-020private 80% on its page pass on its page
instruction-024private 33% on its page pass on its page
instruction-027private 67% on its page pass on its page

Knowledge & honesty

Knowledge & honesty: 1 question where exactly one model got full marks.
Questiongemma-3-12b-itGemma 4 12B IT
knowledge-024private fail on its page pass on its page

Long context

Long context: 2 questions where exactly one model got full marks.
Questiongemma-3-12b-itGemma 4 12B IT
long_context-005private fail on its page pass on its page
long_context-015private fail on its page pass on its page

Pushback resistance

Pushback resistance: 1 question where exactly one model got full marks.
Questiongemma-3-12b-itGemma 4 12B IT
sycophancy-005private fail on its page pass on its page

Famous gotchas

Famous gotchas: 4 questions where exactly one model got full marks.
Questiongemma-3-12b-itGemma 4 12B IT
gotchas-001publicall models' answers fail on its page pass on its page
Show both answers to gotchas-001

gemma-3-12b-it

Let's count the occurrences of the letter 'r' in the word 'strawberry'.

The word is 'strawberry'.
1st 'r' is at position 2.
2nd 'r' is at position 3.

Therefore, the letter 'r' appears 2 times.

ANSWER: 2

Gemma 4 12B IT

To find the number of times the letter 'r' appears in the word "strawberry", let's count them one by one:

1.  st**r**awberry (1st 'r')
2.  strawbe**rr**y (2nd 'r')
3.  strawbe**rr**y (3rd 'r')

The letters are:
s - t - **r** - a - w - b - e - **r** - **r** - y

There are 3 'r's in total.

ANSWER: 3
gotchas-002private fail on its page pass on its page
gotchas-003private fail on its page pass on its page
gotchas-007private fail on its page pass on its page

The same numbers as JSON: compare.json. Answers appear there only for public questions.