gemma-4-31B-it against Muse Glimmer 30B

Pick two models to see them side by side: Overall score, every category, the questions where one did better, and speed and memory. Each model is shown at its newest tested version. Back to the leaderboard.

The short version

These are the two highest Overall scores among the ranked models in the 16–35B size class.

gemma-4-31B-it scores higher overall: 99.3 against 97.9 out of 100, a gap of 1.4 points.

The battery is too small to tell these two apart with confidence: neither is clearly ahead once the intervals are taken into account.

Across the 9 categories that count toward Overall, gemma-4-31B-it is higher in 3, Muse Glimmer 30B is higher in 0 and 6 are level.

The widest gap is in Long context: 100.0 for gemma-4-31B-it against 94.4 for Muse Glimmer 30B, in favour of gemma-4-31B-it.

Of the 259 questions both models have a definite score for, gemma-4-31B-it did better on 4, Muse Glimmer 30B did better on 2, and 253 were level.

The two were tested in different runs, so their hard-question scores are not compared.

Their speeds were not measured on one shared backend and Ollama version, so they are not compared.

Written from the numbers below. Scores come from one fixed battery run on one laptop; read the methodology before drawing conclusions.

Overall

gemma-4-31B-it scores higher: 99.3 against 97.9, a gap of 1.4 points. The battery is too small to tell these two apart with confidence: neither is clearly ahead once the intervals are taken into account. How sure is a score?

Overall (0–100) with its 95% interval, the rank range among all listed ranked models, and the hard-question score.
Measuregemma-4-31B-itMuse Glimmer 30B
Overall99.398.3–100.097.996.0–99.3
Rank range1–51–23
Hard questionsmean score on the questions that half or fewer of the ranked models of 3B and over in the model's own run answered correctly94.784.9
Size class16–35B16–35B

By category

Programmatic mean (0–100) per category. Higher is the model with the larger mean; a gap that rounds to 0.0 is called level. A dash means not scored, which is different from 0.
Categorygemma-4-31B-itMuse Glimmer 30BHigherDifference
Reasoningcounts toward Overall 97.1 94.3 gemma-4-31B-it 2.8
Mathcounts toward Overall 100.0 100.0 Level 0
Codingcounts toward Overall 96.6 92.4 gemma-4-31B-it 4.2
Instruction followingcounts toward Overall 100.0 100.0 Level 0
Knowledge & honestycounts toward Overall 100.0 100.0 Level 0
Long contextcounts toward Overall 100.0 94.4 gemma-4-31B-it 5.6
Multilingualcounts toward Overall 100.0 100.0 Level 0
Tool usecounts toward Overall 100.0 100.0 Level 0
Safetycounts toward Overall 100.0 100.0 Level 0
Pushback resistanceshown separately 100.0 100.0 Level 0
Famous gotchasshown separately 93.3 100.0 Muse Glimmer 30B 6.7
Writingshown separately — — — —
Self-descriptionshown separately — — — —

Speed and memory

Measured on one laptop. Decode speed is generation speed with one request at a time.
Measuregemma-4-31B-itMuse Glimmer 30B
Decode speed, one stream (tok/s)3.83.6
GPU memory on our server (1 to 8 answers at a time) (GB)11.218.3
Model size as loaded on our server (GB)11.218.3
BackendROCm 7.1 (Ollama 0.34.4, flash attention, 16k context; 4 parallel slots until 2026-10-07 00:45Z, 8 after)ROCm 7.1 (Ollama 0.40.13, flash attention, 16k context; 8 parallel slots)

A dash means the figure was not measured, which is different from 0. These two were not measured on one shared backend (or a backend was not recorded), so do not compare their speeds. Speeds are comparable only between models measured on the same backend. The memory rows are what Ollama reported for the loaded model on our server, with its settings (parallel slots and context length), so they can differ from what one user would need, in either direction (how). They are not the memory estimate the hardware picker uses. See also the Speed page.

Where they differ

Both models have a definite score for 259 questions. gemma-4-31B-it did better on 4, Muse Glimmer 30B did better on 2, and 253 were level. A score gap of more than 0.001 counts as a win. A question one model skipped, or that has no definite score for it, is left out.

Questions both models have a score for, by category.
CategoryQuestionsgemma-4-31B-it betterMuse Glimmer 30B betterLevel
Reasoning351034
Math250025
Coding302127
Instruction following300030
Knowledge & honesty300030
Long context181017
Multilingual200020
Tool use200020
Safety240024
Pushback resistance120012
Famous gotchas150114

Questions where only one got full marks

Full marks means a score of 1. The answers of public questions can be opened below. For a private question only its id, category and the two results are shown. Each result links to that question on the model's page.

Reasoning

Reasoning: 1 question where exactly one model got full marks.
Questiongemma-4-31B-itMuse Glimmer 30B
reasoning-030private pass on its page fail on its page

Coding

Coding: 2 questions where exactly one model got full marks.
Questiongemma-4-31B-itMuse Glimmer 30B
coding-017private pass on its page fail on its page
coding-028private pass on its page fail on its page

Long context

Long context: 1 question where exactly one model got full marks.
Questiongemma-4-31B-itMuse Glimmer 30B
long_context-010private pass on its page fail on its page

Famous gotchas

Famous gotchas: 1 question where exactly one model got full marks.
Questiongemma-4-31B-itMuse Glimmer 30B
gotchas-010publicall models' answers fail on its page pass on its page
Show both answers to gotchas-010

gemma-4-31B-it

To solve this problem, we must distinguish it from the standard Monty Hall problem. In the standard version, the host *knows* where the car is and *intentionally* avoids it. In this version, the host does *not* know where the car is and opens a door *at random*.

Let $C$ be the door hiding the car, and $H$ be the door the host opens.
The possible locations for the car are $C \in \{1, 2, 3\}$, each with probability $1/3$.
The host opens door 3 ($H=3$). We are looking for $P(C=2 | H=3)$.

Using Bayes' Theorem:
$P(C=2 | H=3) = \frac{P(H=3 | C=2) P(C=2)}{P(H=3)}$

1. **Prior Probabilities:**
   $P(C=1) = 1/3$
   $P(C=2) = 1/3$
   $P(C=3) = 1/3$

2. **Conditional Probabilities (The host opens door 3 at random from the remaining doors 2 and 3):**
   - If the car is behind door 1 ($C=1$), the host can open door 2 or 3. Since it's random: $P(H=3 | C=1) = 1/2$.
   - If the car is behind door 2 ($C=2$), the host can open door 2 or 3. Since it's random: $P(H=3 | C=2) = 1/2$.
   - If the car is behind door 3 ($C=3$), the host opens door 3: $P(H=3 | C=3) = 1/2$. (Wait, the prompt says the host opens door 3 and it *happens* to be a goat. This means we are conditioning on the event that the host opened door 3 AND it was a goat).

Let's refine the event. We are given that the host opened door 3 and it revealed a goat. This means $H=3$ and $C \neq 3$.
The probability of the host opening door 3 is $1/2$ regardless of where the car is (since he chooses between 2 and 3).
The probability that the car is behind door 2, given that the host opened door 3 and it was a goat:
$P(C=2 | H=3 \text{ and } C \neq 3) = \frac{P(H=3 \text{ and } C \neq 3 | C=2) P(C=2)}{P(H=3 \text{ and } C \neq 3)}$

- $P(H=3 \text{ and } C \neq 3 | C=2) = 1/2$ (The host picks door 3, and since the car is at 2, it is guaranteed to be a goat).
- $P(H=3 \text{ and } C \neq 3 | C=1) = 1/2$ (The host picks door 3, and since the car is at 1, it is guaranteed to be a goat).
- $P(H=3 \text{ and } C \neq 3 | C=3) = 0$ (If th

Cut here for length. The model page has the full answer.

Muse Glimmer 30B

The prior probabilities are

* $P(C_1)=P(C_2)=P(C_3)=1/3$.

Let $E$ be “door 3 is opened and a goat is seen”.  
If $C_1$ or $C_2$ the door 3 is a goat, so the host opens it with probability $1/2$ among the two unchosen doors. If $C_3$ the event is impossible

The same numbers as JSON: compare.json. Answers appear there only for public questions.