gemma-3-12b-it against Translategemma 12B IT

Pick two models to see them side by side: Overall score, every category, the questions where one did better, and speed and memory. Each model is shown at its newest tested version. Back to the leaderboard.

The short version

Translategemma 12B IT is built on Gemma 3, its listed base, and gemma-3-12b-it is the Gemma 3 model closest to it in size on this site.

gemma-3-12b-it scores higher overall: 92.7 against 89.6 out of 100, a gap of 3.1 points.

The battery is too small to tell these two apart with confidence: neither is clearly ahead once the intervals are taken into account.

Across the 9 categories that count toward Overall, gemma-3-12b-it is higher in 5, Translategemma 12B IT is higher in 1 and 3 are level.

The widest gap is in Long context: 83.3 for gemma-3-12b-it against 66.7 for Translategemma 12B IT, in favour of gemma-3-12b-it.

Of the 259 questions both models have a definite score for, gemma-3-12b-it did better on 16, Translategemma 12B IT did better on 10, and 233 were level.

The two were tested in different runs, so their hard-question scores are not compared.

Their speeds were not measured on one shared backend and Ollama version, so they are not compared.

Written from the numbers below. Scores come from one fixed battery run on one laptop; read the methodology before drawing conclusions.

Overall

gemma-3-12b-it scores higher: 92.7 against 89.6, a gap of 3.1 points. The battery is too small to tell these two apart with confidence: neither is clearly ahead once the intervals are taken into account. How sure is a score?

Overall (0–100) with its 95% interval, the rank range among all listed ranked models, and the hard-question score.
Measuregemma-3-12b-itTranslategemma 12B IT
Overall92.789.7–95.489.686.4–92.9
Rank range8–6613–80
Hard questionsmean score on the questions that half or fewer of the ranked models of 3B and over in the model's own run answered correctly66.571.7
Size class9–16B9–16B
Fine-tune ofa community fine-tune is scored like any other modelan original modelGemma 3
Built fora specialised model counts toward Overall like any othergeneral usetranslation

By category

Programmatic mean (0–100) per category. Higher is the model with the larger mean; a gap that rounds to 0.0 is called level. A dash means not scored, which is different from 0.
Categorygemma-3-12b-itTranslategemma 12B ITHigherDifference
Reasoningcounts toward Overall 88.6 82.9 gemma-3-12b-it 5.7
Mathcounts toward Overall 96.0 92.0 gemma-3-12b-it 4.0
Codingcounts toward Overall 81.8 78.9 gemma-3-12b-it 2.9
Instruction followingcounts toward Overall 88.3 86.3 gemma-3-12b-it 2.0
Knowledge & honestycounts toward Overall 96.7 100.0 Translategemma 12B IT 3.3
Long contextcounts toward Overall 83.3 66.7 gemma-3-12b-it 16.6
Multilingualcounts toward Overall 100.0 100.0 Level 0
Tool usecounts toward Overall 100.0 100.0 Level 0
Safetycounts toward Overall 100.0 100.0 Level 0
Pushback resistanceshown separately 91.7 91.7 Level 0
Famous gotchasshown separately 66.7 86.7 Translategemma 12B IT 20.0
Writingshown separately — — — —
Self-descriptionshown separately — — — —

Speed and memory

Measured on one laptop. Decode speed is generation speed with one request at a time.
Measuregemma-3-12b-itTranslategemma 12B IT
Decode speed, one stream (tok/s)6.17.6
GPU memory on our server (1 to 8 answers at a time) (GB)13.113.1
Model size as loaded on our server (GB)13.113.1
BackendROCm 7.1 (Ollama 0.34.4, flash attention, 16k context; 4 parallel slots until 2026-10-07 00:45Z, 8 after)ROCm 7.1 (Ollama 0.40.4, flash attention, 16k context; 8 parallel slots)

A dash means the figure was not measured, which is different from 0. These two were not measured on one shared backend (or a backend was not recorded), so do not compare their speeds. Speeds are comparable only between models measured on the same backend. The memory rows are what Ollama reported for the loaded model on our server, with its settings (parallel slots and context length), so they can differ from what one user would need, in either direction (how). They are not the memory estimate the hardware picker uses. See also the Speed page.

Where they differ

Both models have a definite score for 259 questions. gemma-3-12b-it did better on 16, Translategemma 12B IT did better on 10, and 233 were level. A score gap of more than 0.001 counts as a win. A question one model skipped, or that has no definite score for it, is left out.

Questions both models have a score for, by category.
CategoryQuestionsgemma-3-12b-it betterTranslategemma 12B IT betterLevel
Reasoning352033
Math251024
Coding306222
Instruction following304422
Knowledge & honesty300129
Long context183015
Multilingual200020
Tool use200020
Safety240024
Pushback resistance120012
Famous gotchas150312

Questions where only one got full marks

Full marks means a score of 1. The answers of public questions can be opened below. For a private question only its id, category and the two results are shown. Each result links to that question on the model's page.

Reasoning

Reasoning: 2 questions where exactly one model got full marks.
Questiongemma-3-12b-itTranslategemma 12B IT
reasoning-027private pass on its page fail on its page
reasoning-032private pass on its page fail on its page

Math

Math: 1 question where exactly one model got full marks.
Questiongemma-3-12b-itTranslategemma 12B IT
math-017private pass on its page fail on its page

Coding

Coding: 3 questions where exactly one model got full marks.
Questiongemma-3-12b-itTranslategemma 12B IT
coding-017private pass on its page 60% on its page
coding-021private pass on its page 89% on its page
coding-024private pass on its page 38% on its page

Instruction following

Instruction following: 6 questions where exactly one model got full marks.
Questiongemma-3-12b-itTranslategemma 12B IT
instruction-005private 67% on its page pass on its page
instruction-008publicall models' answers pass on its page 33% on its page
Show both answers to instruction-008

gemma-3-12b-it

{"name": "Elara Meadowlight", "age": 27}

Translategemma 12B IT

```json
{
  "name": "Elara Meadowlight",
  "age": 27
}
```
instruction-010private 75% on its page pass on its page
instruction-019private pass on its page 75% on its page
instruction-020private 80% on its page pass on its page
instruction-028private pass on its page 67% on its page

Knowledge & honesty

Knowledge & honesty: 1 question where exactly one model got full marks.
Questiongemma-3-12b-itTranslategemma 12B IT
knowledge-024private fail on its page pass on its page

Long context

Long context: 3 questions where exactly one model got full marks.
Questiongemma-3-12b-itTranslategemma 12B IT
long_context-006private pass on its page fail on its page
long_context-014private pass on its page fail on its page
long_context-018private pass on its page fail on its page

Famous gotchas

Famous gotchas: 3 questions where exactly one model got full marks.
Questiongemma-3-12b-itTranslategemma 12B IT
gotchas-001publicall models' answers fail on its page pass on its page
Show both answers to gotchas-001

gemma-3-12b-it

Let's count the occurrences of the letter 'r' in the word 'strawberry'.

The word is 'strawberry'.
1st 'r' is at position 2.
2nd 'r' is at position 3.

Therefore, the letter 'r' appears 2 times.

ANSWER: 2

Translategemma 12B IT

The letter 'r' appears three times in the word 'strawberry'.

ANSWER: 3
gotchas-002private fail on its page pass on its page
gotchas-003private fail on its page pass on its page

The same numbers as JSON: compare.json. Answers appear there only for public questions.