Gemma 4 12B IT against gemma-4-31B-it

Pick two models to see them side by side: Overall score, every category, the questions where one did better, and speed and memory. Each model is shown at its newest tested version. Back to the leaderboard.

The short version

Same model line in two sizes: Gemma 4 12B IT has 12 billion parameters and gemma-4-31B-it has 31.3 billion parameters.

gemma-4-31B-it scores higher overall: 99.3 against 98.6 out of 100, a gap of 0.7 points.

The battery is too small to tell these two apart with confidence: neither is clearly ahead once the intervals are taken into account.

Across the 9 categories that count toward Overall, Gemma 4 12B IT is higher in 1, gemma-4-31B-it is higher in 2 and 6 are level.

The widest gap is in Long context: 94.4 for Gemma 4 12B IT against 100.0 for gemma-4-31B-it, in favour of gemma-4-31B-it.

Of the 259 questions both models have a definite score for, Gemma 4 12B IT did better on 1, gemma-4-31B-it did better on 3, and 255 were level.

On the hard questions, which are scored within one run, Gemma 4 12B IT scores 90.7 and gemma-4-31B-it scores 94.7.

On the same backend and Ollama version, with one request at a time, Gemma 4 12B IT generates 8.1 tokens per second and gemma-4-31B-it generates 3.8.

Written from the numbers below. Scores come from one fixed battery run on one laptop; read the methodology before drawing conclusions.

Overall

gemma-4-31B-it scores higher: 99.3 against 98.6, a gap of 0.7 points. The battery is too small to tell these two apart with confidence: neither is clearly ahead once the intervals are taken into account. How sure is a score?

Overall (0–100) with its 95% interval, the rank range among all listed ranked models, and the hard-question score.
MeasureGemma 4 12B ITgemma-4-31B-it
Overall98.697.0–99.899.398.3–100.0
Rank range1–121–5
Hard questionsmean score on the questions that half or fewer of the ranked models of 3B and over answered correctly90.794.7
Size class9–16B16–35B

By category

Programmatic mean (0–100) per category. Higher is the model with the larger mean; a gap that rounds to 0.0 is called level. A dash means not scored, which is different from 0.
CategoryGemma 4 12B ITgemma-4-31B-itHigherDifference
Reasoningcounts toward Overall 94.3 97.1 gemma-4-31B-it 2.8
Mathcounts toward Overall 100.0 100.0 Level 0
Codingcounts toward Overall 98.2 96.6 Gemma 4 12B IT 1.6
Instruction followingcounts toward Overall 100.0 100.0 Level 0
Knowledge & honestycounts toward Overall 100.0 100.0 Level 0
Long contextcounts toward Overall 94.4 100.0 gemma-4-31B-it 5.6
Multilingualcounts toward Overall 100.0 100.0 Level 0
Tool usecounts toward Overall 100.0 100.0 Level 0
Safetycounts toward Overall 100.0 100.0 Level 0
Pushback resistanceshown separately 100.0 100.0 Level 0
Famous gotchasshown separately 93.3 93.3 Level 0
Writingshown separately — — — —
Self-descriptionshown separately — — — —

Speed and memory

Measured on one laptop. Decode speed is generation speed with one request at a time.
MeasureGemma 4 12B ITgemma-4-31B-it
Decode speed, one stream (tok/s)8.13.8
GPU memory on our server (1 to 8 answers at a time) (GB)8.911.2
Model size as loaded on our server (GB)8.911.2
BackendROCm 7.1 (Ollama 0.34.4, flash attention, 16k context; 4 parallel slots until 2026-10-07 00:45Z, 8 after)ROCm 7.1 (Ollama 0.34.4, flash attention, 16k context; 4 parallel slots until 2026-10-07 00:45Z, 8 after)

A dash means the figure was not measured, which is different from 0. Both models were measured on the same backend, so their speeds can be compared. Speeds are comparable only between models measured on the same backend. The memory rows are what Ollama reported for the loaded model on our server, with its settings (parallel slots and context length), so they can differ from what one user would need, in either direction (how). They are not the memory estimate the hardware picker uses. See also the Speed page.

Where they differ

Both models have a definite score for 259 questions. Gemma 4 12B IT did better on 1, gemma-4-31B-it did better on 3, and 255 were level. A score gap of more than 0.001 counts as a win. A question one model skipped, or that has no definite score for it, is left out.

Questions both models have a score for, by category.
CategoryQuestionsGemma 4 12B IT bettergemma-4-31B-it betterLevel
Reasoning350134
Math250025
Coding301128
Instruction following300030
Knowledge & honesty300030
Long context180117
Multilingual200020
Tool use200020
Safety240024
Pushback resistance120012
Famous gotchas150015

Questions where only one got full marks

Full marks means a score of 1. The answers of public questions can be opened below. For a private question only its id, category and the two results are shown. Each result links to that question on the model's page.

Reasoning

Reasoning: 1 question where exactly one model got full marks.
QuestionGemma 4 12B ITgemma-4-31B-it
reasoning-030private fail on its page pass on its page

Coding

Coding: 1 question where exactly one model got full marks.
QuestionGemma 4 12B ITgemma-4-31B-it
coding-016private 91% on its page pass on its page

Long context

Long context: 1 question where exactly one model got full marks.
QuestionGemma 4 12B ITgemma-4-31B-it
long_context-010private fail on its page pass on its page

The same numbers as JSON: compare.json. Answers appear there only for public questions.