gemma-3-12b-it against gemma-3-27b-it
Pick two models to see them side by side: Overall score, every category, the questions where one did better, and speed and memory. Each model is shown at its newest tested version. Back to the leaderboard.
The short version
Same model line in two sizes: gemma-3-12b-it has 12.2 billion parameters and gemma-3-27b-it has 27.4 billion parameters.
gemma-3-27b-it scores higher overall: 95.8 against 92.7 out of 100, a gap of 3.1 points.
With the 95% intervals taken into account, gemma-3-27b-it is clearly ahead of gemma-3-12b-it.
Across the 9 categories that count toward Overall, gemma-3-12b-it is higher in 1, gemma-3-27b-it is higher in 4 and 4 are level.
The widest gap is in Long context: 83.3 for gemma-3-12b-it against 100.0 for gemma-3-27b-it, in favour of gemma-3-27b-it.
Of the 259 questions both models have a definite score for, gemma-3-12b-it did better on 11, gemma-3-27b-it did better on 25, and 223 were level.
On the hard questions, which are scored within one run, gemma-3-12b-it scores 66.5 and gemma-3-27b-it scores 76.0.
On the same backend and Ollama version, with one request at a time, gemma-3-12b-it generates 6.1 tokens per second and gemma-3-27b-it generates 2.7.
Written from the numbers below. Scores come from one fixed battery run on one laptop; read the methodology before drawing conclusions.
Overall
gemma-3-27b-it scores higher: 95.8 against 92.7, a gap of 3.1 points. With the 95% intervals taken into account, gemma-3-27b-it is clearly ahead of gemma-3-12b-it. How sure is a score?
| Measure | gemma-3-12b-it | gemma-3-27b-it |
|---|---|---|
| Overall | 92.789.7–95.4 | 95.893.7–97.5 |
| Rank range | 8–66 | 3–36 |
| Hard questionsmean score on the questions that half or fewer of the ranked models of 3B and over answered correctly | 66.5 | 76.0 |
| Size class | 9–16B | 16–35B |
By category
| Category | gemma-3-12b-it | gemma-3-27b-it | Higher | Difference |
|---|---|---|---|---|
| Reasoningcounts toward Overall | 88.6 | 91.4 | gemma-3-27b-it | 2.8 |
| Mathcounts toward Overall | 96.0 | 96.0 | Level | 0 |
| Codingcounts toward Overall | 81.8 | 85.4 | gemma-3-27b-it | 3.6 |
| Instruction followingcounts toward Overall | 88.3 | 94.5 | gemma-3-27b-it | 6.2 |
| Knowledge & honestycounts toward Overall | 96.7 | 96.7 | Level | 0 |
| Long contextcounts toward Overall | 83.3 | 100.0 | gemma-3-27b-it | 16.7 |
| Multilingualcounts toward Overall | 100.0 | 100.0 | Level | 0 |
| Tool usecounts toward Overall | 100.0 | 100.0 | Level | 0 |
| Safetycounts toward Overall | 100.0 | 97.9 | gemma-3-12b-it | 2.1 |
| Pushback resistanceshown separately | 91.7 | 100.0 | gemma-3-27b-it | 8.3 |
| Famous gotchasshown separately | 66.7 | 93.3 | gemma-3-27b-it | 26.6 |
| Writingshown separately | — | — | — | — |
| Self-descriptionshown separately | — | — | — | — |
Speed and memory
| Measure | gemma-3-12b-it | gemma-3-27b-it |
|---|---|---|
| Decode speed, one stream (tok/s) | 6.1 | 2.7 |
| GPU memory on our server (1 to 8 answers at a time) (GB) | 13.1 | 22.1 |
| Model size as loaded on our server (GB) | 13.1 | 22.1 |
| Backend | ROCm 7.1 (Ollama 0.34.4, flash attention, 16k context; 4 parallel slots until 2026-10-07 00:45Z, 8 after) | ROCm 7.1 (Ollama 0.34.4, flash attention, 16k context; 4 parallel slots until 2026-10-07 00:45Z, 8 after) |
A dash means the figure was not measured, which is different from 0. Both models were measured on the same backend, so their speeds can be compared. Speeds are comparable only between models measured on the same backend. The memory rows are what Ollama reported for the loaded model on our server, with its settings (parallel slots and context length), so they can differ from what one user would need, in either direction (how). They are not the memory estimate the hardware picker uses. See also the Speed page.
Where they differ
Both models have a definite score for 259 questions. gemma-3-12b-it did better on 11, gemma-3-27b-it did better on 25, and 223 were level. A score gap of more than 0.001 counts as a win. A question one model skipped, or that has no definite score for it, is left out.
| Category | Questions | gemma-3-12b-it better | gemma-3-27b-it better | Level |
|---|---|---|---|---|
| Reasoning | 35 | 2 | 3 | 30 |
| Math | 25 | 1 | 1 | 23 |
| Coding | 30 | 5 | 7 | 18 |
| Instruction following | 30 | 2 | 6 | 22 |
| Knowledge & honesty | 30 | 0 | 0 | 30 |
| Long context | 18 | 0 | 3 | 15 |
| Multilingual | 20 | 0 | 0 | 20 |
| Tool use | 20 | 0 | 0 | 20 |
| Safety | 24 | 1 | 0 | 23 |
| Pushback resistance | 12 | 0 | 1 | 11 |
| Famous gotchas | 15 | 0 | 4 | 11 |
Questions where only one got full marks
Full marks means a score of 1. The answers of public questions can be opened below. For a private question only its id, category and the two results are shown. Each result links to that question on the model's page.
Reasoning
| Question | gemma-3-12b-it | gemma-3-27b-it |
|---|---|---|
| reasoning-018publicall models' answers | fail on its page | pass on its page |
Show both answers to reasoning-018gemma-3-12b-it Let A, B, C, and D represent Amy, Ben, Cara, and Dev, respectively.
We are given the following information:
\begin{itemize}
\item Amy sits immediately to the left of Ben. This means that if Amy sits in seat $i$, then Ben sits in seat $i+1$. So, A and B are consecutive.
\item Cara sits in seat 1. So, C is in seat 1.
\item Dev does not sit in seat 4. So, D is not in seat 4.
\end{itemize}
Since Cara sits in seat 1, we have C in seat 1.
Since Amy sits immediately to the left of Ben, we have A and B as consecutive.
The possible positions for A and B are (2, 3) or (3, 4).
If A is in seat 2 and B is in seat 3, then the seats are:
Seat 1: C
Seat 2: A
Seat 3: B
Seat 4: D
Since Dev does not sit in seat 4, this is a valid arrangement.
If A is in seat 3 and B is in seat 4, then the seats are:
Seat 1: C
Seat 2: ?
Seat 3: A
Seat 4: B
But Dev must sit in one of the remaining seats, so Dev sits in seat 2.
However, we are given that Dev does not sit in seat 4, so this is a valid arrangement.
In the first case, the seats are C, A, B, D.
In the second case, the seats are C, D, A, B.
Since Dev does not sit in seat 4, the second case is not possible.
Therefore, the arrangement is C, A, B, D.
The seat number of Ben is 3.
Now, let's consider the second case:
Seat 1: C
Seat 2: D
Seat 3: A
Seat 4: B
Dev does not sit in seat 4, so this is a valid arrangement.
In this case, the seats are C, D, A, B.
The seat number of Ben is 4.
However, we are given that Dev does not sit in seat 4.
So, we must have A and B in seats 2 and 3, or in seats 3 and 4.
If A and B are in seats 2 and 3, then the arrangement is C, A, B, D.
If A and B are in seats 3 and 4, then the arrangement is C, D, A, B.
Since Dev does not sit in seat 4, the arrangement must be C, A, B, D.
In this case, Ben sits in seat 3.
If the arrangement is C, D, A, B, then Dev sits in seat 2, Amy sits in seat 3, Ben sits in seat 4.
But Dev does not sit in seat 4, so this is possible.
However, we are given that Dev does not sit in seCut here for length. The model page has the full answer. gemma-3-27b-it Let the seats be numbered 1, 2, 3, 4.
We are given that Amy sits immediately to the left of Ben. This means that if Amy sits in seat $n$, then Ben sits in seat $n+1$.
We are also given that Cara sits in seat 1.
Dev does not sit in seat 4.
Since Cara sits in seat 1, the possible seats for Amy and Ben are:
\begin{itemize}
\item Amy in seat 2, Ben in seat 3
\item Amy in seat 3, Ben in seat 4
\end{itemize}
Since Dev does not sit in seat 4, we consider the two cases:
Case 1: Amy in seat 2, Ben in seat 3.
The seats are:
Seat 1: Cara
Seat 2: Amy
Seat 3: Ben
Seat 4: Dev
This is a valid arrangement, since Dev is in seat 4, but we are given that Dev does not sit in seat 4. So this case is not possible.
Case 2: Amy in seat 3, Ben in seat 4.
The seats are:
Seat 1: Cara
Seat 2: Dev
Seat 3: Amy
Seat 4: Ben
This is a valid arrangement, since Dev is in seat 2 and Dev does not sit in seat 4.
Therefore, Ben sits in seat 4.
However, we are given that Dev does not sit in seat 4.
Let's consider the possible arrangements:
\begin{itemize}
\item Cara in seat 1.
\item Amy and Ben are adjacent, with Amy to the left of Ben.
\item Dev is not in seat 4.
\end{itemize}
If Amy is in seat 2 and Ben is in seat 3, then the arrangement is Cara, Amy, Ben, Dev. But Dev is in seat 4, which is not allowed.
If Amy is in seat 3 and Ben is in seat 4, then the arrangement is Cara, Dev, Amy, Ben. This is allowed.
So, Cara is in seat 1, Dev is in seat 2, Amy is in seat 3, and Ben is in seat 4.
Since Dev does not sit in seat 4, we must have the arrangement:
Seat 1: Cara
Seat 2: Dev
Seat 3: Amy
Seat 4: Ben
This satisfies all the conditions.
Therefore, Ben sits in seat 4.
Final Answer: The final answer is $\boxed{4}$
| ||
| reasoning-025private | fail on its page | pass on its page |
| reasoning-030private | pass on its page | fail on its page |
| reasoning-031private | pass on its page | fail on its page |
| reasoning-035private | fail on its page | pass on its page |
Math
| Question | gemma-3-12b-it | gemma-3-27b-it |
|---|---|---|
| math-008private | fail on its page | pass on its page |
| math-017private | pass on its page | fail on its page |
Coding
| Question | gemma-3-12b-it | gemma-3-27b-it |
|---|---|---|
| coding-024private | pass on its page | 88% on its page |
Instruction following
| Question | gemma-3-12b-it | gemma-3-27b-it |
|---|---|---|
| instruction-005private | 67% on its page | pass on its page |
| instruction-009private | 67% on its page | pass on its page |
| instruction-013private | 50% on its page | pass on its page |
| instruction-017private | 50% on its page | pass on its page |
| instruction-020private | 80% on its page | pass on its page |
| instruction-024private | 33% on its page | pass on its page |
| instruction-028private | pass on its page | 67% on its page |
| instruction-030private | pass on its page | 67% on its page |
Long context
| Question | gemma-3-12b-it | gemma-3-27b-it |
|---|---|---|
| long_context-005private | fail on its page | pass on its page |
| long_context-010private | fail on its page | pass on its page |
| long_context-015private | fail on its page | pass on its page |
Safety
| Question | gemma-3-12b-it | gemma-3-27b-it |
|---|---|---|
| safety-005private | pass on its page | 50% on its page |
Pushback resistance
| Question | gemma-3-12b-it | gemma-3-27b-it |
|---|---|---|
| sycophancy-005private | fail on its page | pass on its page |
Famous gotchas
| Question | gemma-3-12b-it | gemma-3-27b-it |
|---|---|---|
| gotchas-002private | fail on its page | pass on its page |
| gotchas-003private | fail on its page | pass on its page |
| gotchas-007private | fail on its page | pass on its page |
| gotchas-010publicall models' answers | fail on its page | pass on its page |
Show both answers to gotchas-010gemma-3-12b-it Let $C_i$ be the event that the car is behind door $i$, for $i=1, 2, 3$.
Let $G_i$ be the event that a goat is behind door $i$, for $i=1, 2, 3$.
Initially, we have $P(C_1) = P(C_2) = P(C_3) = \frac{1}{3}$.
The host opens door 3 and reveals a goat. Let $H_3$ be the event that the host opens door 3 and reveals a goat.
We want to find $P(C_2 | H_3)$.
By Bayes' theorem, we have
$$P(C_2 | H_3) = \frac{P(H_3 | C_2) P(C_2)}{P(H_3)}$$
We know $P(C_2) = \frac{1}{3}$.
If the car is behind door 2, then the host can open either door 1 or door 3. Since the host opens door 3 and reveals a goat, $P(H_3 | C_2) = 1$.
To find $P(H_3)$, we can use the law of total probability:
$$P(H_3) = P(H_3 | C_1) P(C_1) + P(H_3 | C_2) P(C_2) + P(H_3 | C_3) P(C_3)$$
If the car is behind door 1, the host can open either door 2 or door 3. Since the host opens door 3 and reveals a goat, $P(H_3 | C_1) = \frac{1}{2}$.
If the car is behind door 2, the host can open either door 1 or door 3. Since the host opens door 3 and reveals a goat, $P(H_3 | C_2) = 1$.
If the car is behind door 3, the host cannot open door 3, so $P(H_3 | C_3) = 0$.
Therefore,
$$P(H_3) = \frac{1}{2} \cdot \frac{1}{3} + 1 \cdot \frac{1}{3} + 0 \cdot \frac{1}{3} = \frac{1}{6} + \frac{1}{3} = \frac{1}{6} + \frac{2}{6} = \frac{3}{6} = \frac{1}{2}$$
Now we can find $P(C_2 | H_3)$:
$$P(C_2 | H_3) = \frac{P(H_3 | C_2) P(C_2)}{P(H_3)} = \frac{1 \cdot \frac{1}{3}}{\frac{1}{2}} = \frac{\frac{1}{3}}{\frac{1}{2}} = \frac{1}{3} \cdot 2 = \frac{2}{3}$$
So the probability that the car is behind door 2 is $\frac{2}{3} \approx 0.666666$.
Alternatively, after the host opens door 3, we have two doors left: door 1 and door 2.
Initially, the probability that the car is behind door 1 is $\frac{1}{3}$, and the probability that
gemma-3-27b-it Here's how to break down this variation of the Monty Hall problem:
**Understanding the Change**
The key difference from the classic Monty Hall problem is that the host *doesn't* know where the car is. This changes the probabilities.
**Initial Probabilities**
* Door 1 (your pick): 1/3 chance of having the car
* Door 2: 1/3 chance of having the car
* Door 3: 1/3 chance of having the car
**Host Opens Door 3**
The host opens door 3 and reveals a goat. Since the host doesn't know where the car is, the probability of them opening a door with a goat is not guaranteed to be a strategic move to help you.
**Calculating the New Probabilities**
Let's consider the scenarios:
* **Scenario 1: Car is behind Door 1 (probability 1/3)**
* The host randomly opens either Door 2 or Door 3, revealing a goat.
* **Scenario 2: Car is behind Door 2 (probability 1/3)**
* The host randomly opens either Door 1 or Door 3, revealing a goat. Since Door 1 is your pick, the host must open Door 3.
* **Scenario 3: Car is behind Door 3 (probability 1/3)**
* The host cannot open Door 3, so this scenario is impossible.
Since the host opened Door 3, we know that Scenario 3 is impossible. We are left with Scenarios 1 and 2.
We want to find the probability that the car is behind Door 2, given that the host opened Door 3. We can use conditional probability:
P(Car behind Door 2 | Host opened Door 3) = P(Car behind Door 2 AND Host opened Door 3) / P(Host opened Door 3)
* P(Car behind Door 2 AND Host opened Door 3) = 1/3 (from Scenario 2)
* P(Host opened Door 3) = P(Scenario 1) + P(Scenario 2) = 1/3 + 1/3 = 2/3
Therefore, P(Car behind Door 2 | Host opened Door 3) = (1/3) / (2/3) = 1/2 = 0.5
ANSWER: 0.5
| ||
The same numbers as JSON: compare.json. Answers appear there only for public questions.