Qwen3-4B-Instruct-2507 against Qwen3-4B-Thinking-2507
Pick two models to see them side by side: Overall score, every category, the questions where one did better, and speed and memory. Each model is shown at its newest tested version. Back to the leaderboard.
The short version
Qwen3-4B-Thinking-2507 is named as a reasoning variant of Qwen3-4B-Instruct-2507: the same family and size.
Qwen3-4B-Instruct-2507 scores higher overall: 90.3 against 83.5 out of 100, a gap of 6.8 points.
With the 95% intervals taken into account, Qwen3-4B-Instruct-2507 is clearly ahead of Qwen3-4B-Thinking-2507.
Across the 9 categories that count toward Overall, Qwen3-4B-Instruct-2507 is higher in 5, Qwen3-4B-Thinking-2507 is higher in 2 and 2 are level.
The widest gap is in Coding: 69.7 for Qwen3-4B-Instruct-2507 against 26.7 for Qwen3-4B-Thinking-2507, in favour of Qwen3-4B-Instruct-2507.
Of the 259 questions both models have a definite score for, Qwen3-4B-Instruct-2507 did better on 37, Qwen3-4B-Thinking-2507 did better on 11, and 211 were level.
Their speeds were not measured on one shared backend and Ollama version, so they are not compared.
Written from the numbers below. Scores come from one fixed battery run on one laptop; read the methodology before drawing conclusions.
Overall
Qwen3-4B-Instruct-2507 scores higher: 90.3 against 83.5, a gap of 6.8 points. With the 95% intervals taken into account, Qwen3-4B-Instruct-2507 is clearly ahead of Qwen3-4B-Thinking-2507. How sure is a score?
| Measure | Qwen3-4B-Instruct-2507 | Qwen3-4B-Thinking-2507 |
|---|---|---|
| Overall | 90.387.2–93.3 | 83.580.2–87.0 |
| Rank range | 13–73 | 47–111 |
| Hard questionsmean score on the questions that half or fewer of the ranked models (chosen per run, see the methodology) answered correctly | 57.2 | — |
| Size class | 4–9B | 4–9B |
By category
| Category | Qwen3-4B-Instruct-2507 | Qwen3-4B-Thinking-2507 | Higher | Difference |
|---|---|---|---|---|
| Reasoningcounts toward Overall | 82.9 | 74.3 | Qwen3-4B-Instruct-2507 | 8.6 |
| Mathcounts toward Overall | 80.0 | 76.0 | Qwen3-4B-Instruct-2507 | 4.0 |
| Codingcounts toward Overall | 69.7 | 26.7 | Qwen3-4B-Instruct-2507 | 43.0 |
| Instruction followingcounts toward Overall | 94.2 | 88.6 | Qwen3-4B-Instruct-2507 | 5.6 |
| Knowledge & honestycounts toward Overall | 96.7 | 100.0 | Qwen3-4B-Thinking-2507 | 3.3 |
| Long contextcounts toward Overall | 88.9 | 94.4 | Qwen3-4B-Thinking-2507 | 5.5 |
| Multilingualcounts toward Overall | 100.0 | 91.7 | Qwen3-4B-Instruct-2507 | 8.3 |
| Tool usecounts toward Overall | 100.0 | 100.0 | Level | 0 |
| Safetycounts toward Overall | 100.0 | 100.0 | Level | 0 |
| Pushback resistanceshown separately | 100.0 | 100.0 | Level | 0 |
| Famous gotchasshown separately | 93.3 | 93.3 | Level | 0 |
| Writingshown separately | — | — | — | — |
| Self-descriptionshown separately | — | — | — | — |
Speed and memory
| Measure | Qwen3-4B-Instruct-2507 | Qwen3-4B-Thinking-2507 |
|---|---|---|
| Decode speed, one stream (tok/s) | 17.1 | 21.5 |
| GPU memory on our server (1 to 8 answers at a time) (GB) | 12.5 | 2.5 |
| Model size as loaded on our server (GB) | 12.5 | 2.5 |
| Backend | ROCm 7.1 (Ollama 0.34.4, flash attention, 16k context; 4 parallel slots until 2026-10-07 00:45Z, 8 after) | ROCm 7.1 (Ollama 0.40.13, flash attention, 16k context; 8 parallel slots) |
A dash means the figure was not measured, which is different from 0. These two were not measured on one shared backend (or a backend was not recorded), so do not compare their speeds. Speeds are comparable only between models measured on the same backend. The memory rows are what Ollama reported for the loaded model on our server, with its settings (parallel slots and context length), so they can differ from what one user would need, in either direction (how). They are not the memory estimate the hardware picker uses. See also the Speed page.
Where they differ
Both models have a definite score for 259 questions. Qwen3-4B-Instruct-2507 did better on 37, Qwen3-4B-Thinking-2507 did better on 11, and 211 were level. A score gap of more than 0.001 counts as a win. A question one model skipped, or that has no definite score for it, is left out.
| Category | Questions | Qwen3-4B-Instruct-2507 better | Qwen3-4B-Thinking-2507 better | Level |
|---|---|---|---|---|
| Reasoning | 35 | 5 | 2 | 28 |
| Math | 25 | 4 | 3 | 18 |
| Coding | 30 | 19 | 1 | 10 |
| Instruction following | 30 | 5 | 2 | 23 |
| Knowledge & honesty | 30 | 0 | 1 | 29 |
| Long context | 18 | 1 | 2 | 15 |
| Multilingual | 20 | 3 | 0 | 17 |
| Tool use | 20 | 0 | 0 | 20 |
| Safety | 24 | 0 | 0 | 24 |
| Pushback resistance | 12 | 0 | 0 | 12 |
| Famous gotchas | 15 | 0 | 0 | 15 |
Questions where only one got full marks
Full marks means a score of 1. The answers of public questions can be opened below. For a private question only its id, category and the two results are shown. Each result links to that question on the model's page.
Reasoning
| Question | Qwen3-4B-Instruct-2507 | Qwen3-4B-Thinking-2507 |
|---|---|---|
| reasoning-021private | pass on its page | fail on its page |
| reasoning-023private | pass on its page | fail on its page |
| reasoning-024private | pass on its page | fail on its page |
| reasoning-027private | fail on its page | pass on its page |
| reasoning-030private | pass on its page | fail on its page |
| reasoning-032private | fail on its page | pass on its page |
| reasoning-035private | pass on its page | fail on its page |
Math
| Question | Qwen3-4B-Instruct-2507 | Qwen3-4B-Thinking-2507 |
|---|---|---|
| math-008private | pass on its page | fail on its page |
| math-017private | fail on its page | pass on its page |
| math-019private | pass on its page | fail on its page |
| math-020private | fail on its page | pass on its page |
| math-022private | pass on its page | fail on its page |
| math-024private | pass on its page | fail on its page |
| math-025private | fail on its page | pass on its page |
Coding
| Question | Qwen3-4B-Instruct-2507 | Qwen3-4B-Thinking-2507 |
|---|---|---|
| coding-001private | pass on its page | fail on its page |
| coding-002private | pass on its page | fail on its page |
| coding-006private | pass on its page | fail on its page |
| coding-010publicall models' answers | pass on its page | fail on its page |
Show both answers to coding-010Qwen3-4B-Instruct-2507 ```python
def count_islands(grid: list) -> int:
if not grid or not grid[0]:
return 0
rows = len(grid)
cols = len(grid[0])
visited = [[False] * cols for _ in range(rows)]
count = 0
def dfs(r, c):
if (r < 0 or r >= rows or c < 0 or c >= cols or
visited[r][c] or grid[r][c] == 0):
return
visited[r][c] = True
# Explore all 4 directions
dfs(r + 1, c)
dfs(r - 1, c)
dfs(r, c + 1)
dfs(r, c - 1)
for i in range(rows):
for j in range(cols):
if grid[i][j] == 1 and not visited[i][j]:
dfs(i, j)
count += 1
return count
```
Qwen3-4B-Thinking-2507 (no text stored) | ||
| coding-011private | pass on its page | fail on its page |
| coding-014publicall models' answers | 43% on its page | pass on its page |
Show both answers to coding-014Qwen3-4B-Instruct-2507 ```python
def reverse_words(s: str) -> str:
return ' '.join(s.strip().split())
```
Qwen3-4B-Thinking-2507 def reverse_words(s: str) -> str:
return ' '.join(s.split()[::-1])
| ||
| coding-025private | pass on its page | fail on its page |
Instruction following
| Question | Qwen3-4B-Instruct-2507 | Qwen3-4B-Thinking-2507 |
|---|---|---|
| instruction-006private | pass on its page | 67% on its page |
| instruction-014publicall models' answers | pass on its page | 25% on its page |
Show both answers to instruction-014Qwen3-4B-Instruct-2507 "Thank you for your email and the opportunity to discuss this matter further. I appreciate your attention to the details and am confident we can reach a mutually beneficial resolution. regards" Qwen3-4B-Thinking-2507 (no text stored) | ||
| instruction-020private | 60% on its page | pass on its page |
| instruction-021private | 50% on its page | pass on its page |
| instruction-022private | pass on its page | 33% on its page |
| instruction-030private | pass on its page | 67% on its page |
Knowledge & honesty
| Question | Qwen3-4B-Instruct-2507 | Qwen3-4B-Thinking-2507 |
|---|---|---|
| knowledge-004private | fail on its page | pass on its page |
Long context
| Question | Qwen3-4B-Instruct-2507 | Qwen3-4B-Thinking-2507 |
|---|---|---|
| long_context-005private | pass on its page | fail on its page |
| long_context-014private | fail on its page | pass on its page |
| long_context-016private | fail on its page | pass on its page |
Multilingual
| Question | Qwen3-4B-Instruct-2507 | Qwen3-4B-Thinking-2507 |
|---|---|---|
| multilingual-017private | pass on its page | 33% on its page |
| multilingual-018private | pass on its page | 67% on its page |
| multilingual-019private | pass on its page | 33% on its page |
The same numbers as JSON: compare.json. Answers appear there only for public questions.