gemma-3-12b-it against Gemma 4 12B IT
Pick two models to see them side by side: Overall score, every category, the questions where one did better, and speed and memory. Each model is shown at its newest tested version. Back to the leaderboard.
The short version
These are the two highest Overall scores among the ranked models in the 9–16B size class.
Gemma 4 12B IT scores higher overall: 98.6 against 92.7 out of 100, a gap of 5.9 points.
With the 95% intervals taken into account, Gemma 4 12B IT is clearly ahead of gemma-3-12b-it.
Across the 9 categories that count toward Overall, gemma-3-12b-it is higher in 0, Gemma 4 12B IT is higher in 6 and 3 are level.
The widest gap is in Coding: 81.8 for gemma-3-12b-it against 98.2 for Gemma 4 12B IT, in favour of Gemma 4 12B IT.
Of the 259 questions both models have a definite score for, gemma-3-12b-it did better on 1, Gemma 4 12B IT did better on 32, and 226 were level.
On the hard questions, which are scored within one run, gemma-3-12b-it scores 66.5 and Gemma 4 12B IT scores 90.7.
On the same backend and Ollama version, with one request at a time, gemma-3-12b-it generates 6.1 tokens per second and Gemma 4 12B IT generates 8.1.
Written from the numbers below. Scores come from one fixed battery run on one laptop; read the methodology before drawing conclusions.
Overall
Gemma 4 12B IT scores higher: 98.6 against 92.7, a gap of 5.9 points. With the 95% intervals taken into account, Gemma 4 12B IT is clearly ahead of gemma-3-12b-it. How sure is a score?
| Measure | gemma-3-12b-it | Gemma 4 12B IT |
|---|---|---|
| Overall | 92.789.7–95.4 | 98.697.0–99.8 |
| Rank range | 8–66 | 1–12 |
| Hard questionsmean score on the questions that half or fewer of the ranked models of 3B and over answered correctly | 66.5 | 90.7 |
| Size class | 9–16B | 9–16B |
By category
| Category | gemma-3-12b-it | Gemma 4 12B IT | Higher | Difference |
|---|---|---|---|---|
| Reasoningcounts toward Overall | 88.6 | 94.3 | Gemma 4 12B IT | 5.7 |
| Mathcounts toward Overall | 96.0 | 100.0 | Gemma 4 12B IT | 4.0 |
| Codingcounts toward Overall | 81.8 | 98.2 | Gemma 4 12B IT | 16.4 |
| Instruction followingcounts toward Overall | 88.3 | 100.0 | Gemma 4 12B IT | 11.7 |
| Knowledge & honestycounts toward Overall | 96.7 | 100.0 | Gemma 4 12B IT | 3.3 |
| Long contextcounts toward Overall | 83.3 | 94.4 | Gemma 4 12B IT | 11.1 |
| Multilingualcounts toward Overall | 100.0 | 100.0 | Level | 0 |
| Tool usecounts toward Overall | 100.0 | 100.0 | Level | 0 |
| Safetycounts toward Overall | 100.0 | 100.0 | Level | 0 |
| Pushback resistanceshown separately | 91.7 | 100.0 | Gemma 4 12B IT | 8.3 |
| Famous gotchasshown separately | 66.7 | 93.3 | Gemma 4 12B IT | 26.6 |
| Writingshown separately | — | — | — | — |
| Self-descriptionshown separately | — | — | — | — |
Speed and memory
| Measure | gemma-3-12b-it | Gemma 4 12B IT |
|---|---|---|
| Decode speed, one stream (tok/s) | 6.1 | 8.1 |
| GPU memory on our server (1 to 8 answers at a time) (GB) | 13.1 | 8.9 |
| Model size as loaded on our server (GB) | 13.1 | 8.9 |
| Backend | ROCm 7.1 (Ollama 0.34.4, flash attention, 16k context; 4 parallel slots until 2026-10-07 00:45Z, 8 after) | ROCm 7.1 (Ollama 0.34.4, flash attention, 16k context; 4 parallel slots until 2026-10-07 00:45Z, 8 after) |
A dash means the figure was not measured, which is different from 0. Both models were measured on the same backend, so their speeds can be compared. Speeds are comparable only between models measured on the same backend. The memory rows are what Ollama reported for the loaded model on our server, with its settings (parallel slots and context length), so they can differ from what one user would need, in either direction (how). They are not the memory estimate the hardware picker uses. See also the Speed page.
Where they differ
Both models have a definite score for 259 questions. gemma-3-12b-it did better on 1, Gemma 4 12B IT did better on 32, and 226 were level. A score gap of more than 0.001 counts as a win. A question one model skipped, or that has no definite score for it, is left out.
| Category | Questions | gemma-3-12b-it better | Gemma 4 12B IT better | Level |
|---|---|---|---|---|
| Reasoning | 35 | 1 | 3 | 31 |
| Math | 25 | 0 | 1 | 24 |
| Coding | 30 | 0 | 11 | 19 |
| Instruction following | 30 | 0 | 9 | 21 |
| Knowledge & honesty | 30 | 0 | 1 | 29 |
| Long context | 18 | 0 | 2 | 16 |
| Multilingual | 20 | 0 | 0 | 20 |
| Tool use | 20 | 0 | 0 | 20 |
| Safety | 24 | 0 | 0 | 24 |
| Pushback resistance | 12 | 0 | 1 | 11 |
| Famous gotchas | 15 | 0 | 4 | 11 |
Questions where only one got full marks
Full marks means a score of 1. The answers of public questions can be opened below. For a private question only its id, category and the two results are shown. Each result links to that question on the model's page.
Reasoning
| Question | gemma-3-12b-it | Gemma 4 12B IT |
|---|---|---|
| reasoning-018publicall models' answers | fail on its page | pass on its page |
Show both answers to reasoning-018gemma-3-12b-it Let A, B, C, and D represent Amy, Ben, Cara, and Dev, respectively.
We are given the following information:
\begin{itemize}
\item Amy sits immediately to the left of Ben. This means that if Amy sits in seat $i$, then Ben sits in seat $i+1$. So, A and B are consecutive.
\item Cara sits in seat 1. So, C is in seat 1.
\item Dev does not sit in seat 4. So, D is not in seat 4.
\end{itemize}
Since Cara sits in seat 1, we have C in seat 1.
Since Amy sits immediately to the left of Ben, we have A and B as consecutive.
The possible positions for A and B are (2, 3) or (3, 4).
If A is in seat 2 and B is in seat 3, then the seats are:
Seat 1: C
Seat 2: A
Seat 3: B
Seat 4: D
Since Dev does not sit in seat 4, this is a valid arrangement.
If A is in seat 3 and B is in seat 4, then the seats are:
Seat 1: C
Seat 2: ?
Seat 3: A
Seat 4: B
But Dev must sit in one of the remaining seats, so Dev sits in seat 2.
However, we are given that Dev does not sit in seat 4, so this is a valid arrangement.
In the first case, the seats are C, A, B, D.
In the second case, the seats are C, D, A, B.
Since Dev does not sit in seat 4, the second case is not possible.
Therefore, the arrangement is C, A, B, D.
The seat number of Ben is 3.
Now, let's consider the second case:
Seat 1: C
Seat 2: D
Seat 3: A
Seat 4: B
Dev does not sit in seat 4, so this is a valid arrangement.
In this case, the seats are C, D, A, B.
The seat number of Ben is 4.
However, we are given that Dev does not sit in seat 4.
So, we must have A and B in seats 2 and 3, or in seats 3 and 4.
If A and B are in seats 2 and 3, then the arrangement is C, A, B, D.
If A and B are in seats 3 and 4, then the arrangement is C, D, A, B.
Since Dev does not sit in seat 4, the arrangement must be C, A, B, D.
In this case, Ben sits in seat 3.
If the arrangement is C, D, A, B, then Dev sits in seat 2, Amy sits in seat 3, Ben sits in seat 4.
But Dev does not sit in seat 4, so this is possible.
However, we are given that Dev does not sit in seCut here for length. The model page has the full answer. Gemma 4 12B IT To find the seat number for Ben, let's analyze the given information step by step:
1. **Seats available:** 1, 2, 3, 4.
2. **Cara's position:** "Cara sits in seat 1."
* Current arrangement: [Cara, \_, \_, \_]
3. **Amy and Ben's positions:** "Amy sits immediately to the left of Ben."
* This means Amy and Ben must occupy two adjacent seats in the order (Amy, Ben).
* Since seat 1 is occupied by Cara, the available pairs of adjacent seats are (2, 3) and (3, 4).
4. **Dev's position:** "Dev does not sit in seat 4."
* Let's test the two possible scenarios for Amy and Ben:
* **Scenario A: Amy and Ben are in seats 2 and 3.**
* Seat 1: Cara
* Seat 2: Amy
* Seat 3: Ben
* Seat 4: Dev
* *Check:* Does this violate any rules? The rule says "Dev does not sit in seat 4." In this scenario, Dev is in seat 4. Therefore, Scenario A is **incorrect**.
* **Scenario B: Amy and Ben are in seats 3 and 4.**
* Seat 1: Cara
* Seat 2: Dev
* Seat 3: Amy
* Seat 4: Ben
* *Check:*
* Cara is in seat 1? Yes.
* Amy is immediately to the left of Ben? Yes (Amy is 3, Ben is 4).
* Dev is not in seat 4? Yes (Dev is in seat 2).
* This scenario satisfies all conditions.
In Scenario B, Ben sits in seat 4.
ANSWER: 4
| ||
| reasoning-025private | fail on its page | pass on its page |
| reasoning-030private | pass on its page | fail on its page |
| reasoning-035private | fail on its page | pass on its page |
Math
| Question | gemma-3-12b-it | Gemma 4 12B IT |
|---|---|---|
| math-008private | fail on its page | pass on its page |
Coding
| Question | gemma-3-12b-it | Gemma 4 12B IT |
|---|---|---|
| coding-009private | 82% on its page | pass on its page |
| coding-011private | fail on its page | pass on its page |
| coding-018private | 89% on its page | pass on its page |
| coding-019private | 38% on its page | pass on its page |
| coding-022private | 45% on its page | pass on its page |
| coding-023private | 89% on its page | pass on its page |
| coding-025private | 22% on its page | pass on its page |
| coding-027private | 82% on its page | pass on its page |
| coding-030private | 27% on its page | pass on its page |
Instruction following
| Question | gemma-3-12b-it | Gemma 4 12B IT |
|---|---|---|
| instruction-005private | 67% on its page | pass on its page |
| instruction-009private | 67% on its page | pass on its page |
| instruction-010private | 75% on its page | pass on its page |
| instruction-013private | 50% on its page | pass on its page |
| instruction-017private | 50% on its page | pass on its page |
| instruction-018private | 60% on its page | pass on its page |
| instruction-020private | 80% on its page | pass on its page |
| instruction-024private | 33% on its page | pass on its page |
| instruction-027private | 67% on its page | pass on its page |
Knowledge & honesty
| Question | gemma-3-12b-it | Gemma 4 12B IT |
|---|---|---|
| knowledge-024private | fail on its page | pass on its page |
Long context
| Question | gemma-3-12b-it | Gemma 4 12B IT |
|---|---|---|
| long_context-005private | fail on its page | pass on its page |
| long_context-015private | fail on its page | pass on its page |
Pushback resistance
| Question | gemma-3-12b-it | Gemma 4 12B IT |
|---|---|---|
| sycophancy-005private | fail on its page | pass on its page |
Famous gotchas
| Question | gemma-3-12b-it | Gemma 4 12B IT |
|---|---|---|
| gotchas-001publicall models' answers | fail on its page | pass on its page |
Show both answers to gotchas-001gemma-3-12b-it Let's count the occurrences of the letter 'r' in the word 'strawberry'. The word is 'strawberry'. 1st 'r' is at position 2. 2nd 'r' is at position 3. Therefore, the letter 'r' appears 2 times. ANSWER: 2 Gemma 4 12B IT To find the number of times the letter 'r' appears in the word "strawberry", let's count them one by one: 1. st**r**awberry (1st 'r') 2. strawbe**rr**y (2nd 'r') 3. strawbe**rr**y (3rd 'r') The letters are: s - t - **r** - a - w - b - e - **r** - **r** - y There are 3 'r's in total. ANSWER: 3 | ||
| gotchas-002private | fail on its page | pass on its page |
| gotchas-003private | fail on its page | pass on its page |
| gotchas-007private | fail on its page | pass on its page |
The same numbers as JSON: compare.json. Answers appear there only for public questions.