Cogito 3B V1 Preview Llama against Llama-3.2-3B-Instruct
Pick two models to see them side by side: Overall score, every category, the questions where one did better, and speed and memory. Each model is shown at its newest tested version. Back to the leaderboard.
The short version
Cogito 3B V1 Preview Llama is built on Llama 3.2, its listed base, and Llama-3.2-3B-Instruct is the Llama 3.2 model closest to it in size on this site.
Llama-3.2-3B-Instruct scores higher overall: 75.7 against 70.4 out of 100, a gap of 5.3 points.
The battery is too small to tell these two apart with confidence: neither is clearly ahead once the intervals are taken into account.
Across the 9 categories that count toward Overall, Cogito 3B V1 Preview Llama is higher in 1, Llama-3.2-3B-Instruct is higher in 5 and 3 are level.
The widest gap is in Reasoning: 45.7 for Cogito 3B V1 Preview Llama against 68.6 for Llama-3.2-3B-Instruct, in favour of Llama-3.2-3B-Instruct.
Of the 259 questions both models have a definite score for, Cogito 3B V1 Preview Llama did better on 19, Llama-3.2-3B-Instruct did better on 45, and 195 were level.
The two were tested in different runs, so their hard-question scores are not compared.
Their speeds were not measured on one shared backend and Ollama version, so they are not compared.
Written from the numbers below. Scores come from one fixed battery run on one laptop; read the methodology before drawing conclusions.
Overall
Llama-3.2-3B-Instruct scores higher: 75.7 against 70.4, a gap of 5.3 points. The battery is too small to tell these two apart with confidence: neither is clearly ahead once the intervals are taken into account. How sure is a score?
| Measure | Cogito 3B V1 Preview Llama | Llama-3.2-3B-Instruct |
|---|---|---|
| Overall | 70.465.4–75.3 | 75.771.0–80.4 |
| Rank range | 104–157 | 84–150 |
| Hard questionsmean score on the questions that half or fewer of the ranked models of 3B and over in the model's own run answered correctly | 32.6 | 33.5 |
| Size class | 3–4B | 3–4B |
| Fine-tune ofa community fine-tune is scored like any other model | Llama 3.2 | an original model |
By category
| Category | Cogito 3B V1 Preview Llama | Llama-3.2-3B-Instruct | Higher | Difference |
|---|---|---|---|---|
| Reasoningcounts toward Overall | 45.7 | 68.6 | Llama-3.2-3B-Instruct | 22.9 |
| Mathcounts toward Overall | 40.0 | 52.0 | Llama-3.2-3B-Instruct | 12.0 |
| Codingcounts toward Overall | 49.3 | 52.9 | Llama-3.2-3B-Instruct | 3.6 |
| Instruction followingcounts toward Overall | 81.6 | 91.7 | Llama-3.2-3B-Instruct | 10.1 |
| Knowledge & honestycounts toward Overall | 86.7 | 86.7 | Level | 0 |
| Long contextcounts toward Overall | 61.1 | 55.6 | Cogito 3B V1 Preview Llama | 5.5 |
| Multilingualcounts toward Overall | 76.7 | 81.7 | Llama-3.2-3B-Instruct | 5.0 |
| Tool usecounts toward Overall | 92.5 | 92.5 | Level | 0 |
| Safetycounts toward Overall | 100.0 | 100.0 | Level | 0 |
| Pushback resistanceshown separately | 66.7 | 83.3 | Llama-3.2-3B-Instruct | 16.6 |
| Famous gotchasshown separately | 53.3 | 60.0 | Llama-3.2-3B-Instruct | 6.7 |
| Writingshown separately | — | — | — | — |
| Self-descriptionshown separately | — | — | — | — |
Speed and memory
| Measure | Cogito 3B V1 Preview Llama | Llama-3.2-3B-Instruct |
|---|---|---|
| Decode speed, one stream (tok/s) | 73.4 | 20.7 |
| GPU memory on our server (1 to 8 answers at a time) (GB) | 17.4 | 9.9 |
| Model size as loaded on our server (GB) | 17.4 | 9.9 |
| Backend | ROCm 7.1 (Ollama 0.40.4, flash attention, 16k context; 8 parallel slots) | ROCm 7.1 (Ollama 0.34.4, flash attention, 16k context; 4 parallel slots until 2026-10-07 00:45Z, 8 after) |
A dash means the figure was not measured, which is different from 0. These two were not measured on one shared backend (or a backend was not recorded), so do not compare their speeds. Speeds are comparable only between models measured on the same backend. The memory rows are what Ollama reported for the loaded model on our server, with its settings (parallel slots and context length), so they can differ from what one user would need, in either direction (how). They are not the memory estimate the hardware picker uses. See also the Speed page.
Where they differ
Both models have a definite score for 259 questions. Cogito 3B V1 Preview Llama did better on 19, Llama-3.2-3B-Instruct did better on 45, and 195 were level. A score gap of more than 0.001 counts as a win. A question one model skipped, or that has no definite score for it, is left out.
| Category | Questions | Cogito 3B V1 Preview Llama better | Llama-3.2-3B-Instruct better | Level |
|---|---|---|---|---|
| Reasoning | 35 | 2 | 10 | 23 |
| Math | 25 | 2 | 5 | 18 |
| Coding | 30 | 7 | 9 | 14 |
| Instruction following | 30 | 1 | 11 | 18 |
| Knowledge & honesty | 30 | 2 | 2 | 26 |
| Long context | 18 | 2 | 1 | 15 |
| Multilingual | 20 | 1 | 2 | 17 |
| Tool use | 20 | 1 | 1 | 18 |
| Safety | 24 | 0 | 0 | 24 |
| Pushback resistance | 12 | 0 | 2 | 10 |
| Famous gotchas | 15 | 1 | 2 | 12 |
Questions where only one got full marks
Full marks means a score of 1. The answers of public questions can be opened below. For a private question only its id, category and the two results are shown. Each result links to that question on the model's page.
Reasoning
| Question | Cogito 3B V1 Preview Llama | Llama-3.2-3B-Instruct |
|---|---|---|
| reasoning-003private | fail on its page | pass on its page |
| reasoning-006private | pass on its page | fail on its page |
| reasoning-009private | pass on its page | fail on its page |
| reasoning-015private | fail on its page | pass on its page |
| reasoning-016private | fail on its page | pass on its page |
| reasoning-019private | fail on its page | pass on its page |
| reasoning-020private | fail on its page | pass on its page |
| reasoning-022private | fail on its page | pass on its page |
| reasoning-024private | fail on its page | pass on its page |
| reasoning-026private | fail on its page | pass on its page |
| reasoning-031private | fail on its page | pass on its page |
| reasoning-032private | fail on its page | pass on its page |
Math
| Question | Cogito 3B V1 Preview Llama | Llama-3.2-3B-Instruct |
|---|---|---|
| math-003private | fail on its page | pass on its page |
| math-008private | pass on its page | fail on its page |
| math-011private | fail on its page | pass on its page |
| math-018private | fail on its page | pass on its page |
| math-020private | pass on its page | fail on its page |
| math-021publicall models' answers | fail on its page | pass on its page |
Show both answers to math-021Cogito 3B V1 Preview Llama Let me solve this step by step. Given that x + 1/x = 5, let's find x^3 + 1/x^3. First, note that (x + 1/x)^2 = x^2 + 2 + 1/x^2 So, x^2 + 1/x^2 = (x + 1/x)^2 - 2 Now, let's find x^3 + 1/x^3. We can write this as (x^2 + 1/x^2)(x + 1/x) Using the above equations: (x^2 + 1/x^2) = (5)^2 - 2 = 25 - 2 = 23 (x + 1/x) = 5 Therefore, x^3 + 1/x^3 = 23 × 5 = 115 ANSWER: 115 Llama-3.2-3B-Instruct To find the value of x^3 + 1/x^3, we can use the identity: x^3 + 1/x^3 = (x + 1/x)^3 - 3(x + 1/x) We are given that x + 1/x = 5, so we can substitute this value into the identity: x^3 + 1/x^3 = (5)^3 - 3(5) = 125 - 15 = 110 ANSWER: 110 | ||
| math-025private | fail on its page | pass on its page |
Coding
| Question | Cogito 3B V1 Preview Llama | Llama-3.2-3B-Instruct |
|---|---|---|
| coding-005private | pass on its page | 27% on its page |
| coding-006private | pass on its page | 82% on its page |
| coding-010publicall models' answers | 25% on its page | pass on its page |
Show both answers to coding-010Cogito 3B V1 Preview Llama def count_islands(grid: list) -> int:
if not grid or not grid[0]:
return 0
rows = len(grid)
cols = len(grid[0])
def dfs(i, j):
if i < 0 or i >= rows or j < 0 or j >= cols or grid[i][j] == 0:
return
grid[i][j] = 0 # mark as visited
for di, dj in [(1,0), (-1,0), (0,1), (0,-1)]:
dfs(i+di, j+dj)
islands = 0
for i in range(rows):
for j in range(cols):
if grid[i][j] == 1 and not any(grid[k][l] == 1 for k,l in [(i-1,j), (i+1,j), (i,j-1), (i,j+1)]):
dfs(i, j)
islands += 1
return islands
Llama-3.2-3B-Instruct def count_islands(grid: list) -> int:
if not grid:
return 0
count = 0
for i in range(len(grid)):
for j in range(len(grid[0])):
if grid[i][j] == 1:
dfs(grid, i, j)
count += 1
return count
def dfs(grid: list, i: int, j: int):
if i<0 or j<0 or i>=len(grid) or j>=len(grid[0]) or grid[i][j] != 1:
return
grid[i][j] = 0
dfs(grid, i+1, j)
dfs(grid, i-1, j)
dfs(grid, i, j+1)
dfs(grid, i, j-1)
| ||
| coding-012private | 18% on its page | pass on its page |
| coding-013private | 91% on its page | pass on its page |
Instruction following
| Question | Cogito 3B V1 Preview Llama | Llama-3.2-3B-Instruct |
|---|---|---|
| instruction-005private | 67% on its page | pass on its page |
| instruction-006private | pass on its page | 67% on its page |
| instruction-011private | 75% on its page | pass on its page |
| instruction-012private | 75% on its page | pass on its page |
| instruction-014publicall models' answers | 75% on its page | pass on its page |
Show both answers to instruction-014Cogito 3B V1 Preview Llama "Thank you for your inquiry regarding our company's policies on employee leave requests. Our standard procedure requires all employees to submit their leave applications at least two weeks in advance of their intended departure date, and we must emphasize that any exceptions to this policy require prior approval from our management team. We appreciate your understanding and look forward to addressing any further questions you may have regarding our company's policies." Llama-3.2-3B-Instruct "Dear [Name], I am writing to acknowledge receipt of your letter and express my gratitude for the opportunity to engage in a discussion regarding the proposed project. I have carefully reviewed the attached documents and am pleased to inform you that we are moving forward with the next steps. I look forward to working with you to bring this project to fruition, and I wish you all the best in your endeavors, regards" | ||
| instruction-019private | 75% on its page | pass on its page |
| instruction-022private | 67% on its page | pass on its page |
| instruction-023private | 67% on its page | pass on its page |
| instruction-028private | 67% on its page | pass on its page |
Knowledge & honesty
| Question | Cogito 3B V1 Preview Llama | Llama-3.2-3B-Instruct |
|---|---|---|
| knowledge-009private | pass on its page | fail on its page |
| knowledge-015private | fail on its page | pass on its page |
| knowledge-027private | pass on its page | fail on its page |
| knowledge-029private | fail on its page | pass on its page |
Long context
| Question | Cogito 3B V1 Preview Llama | Llama-3.2-3B-Instruct |
|---|---|---|
| long_context-007private | pass on its page | fail on its page |
| long_context-008private | pass on its page | fail on its page |
| long_context-013private | fail on its page | pass on its page |
Multilingual
| Question | Cogito 3B V1 Preview Llama | Llama-3.2-3B-Instruct |
|---|---|---|
| multilingual-003publicall models' answers | fail on its page | pass on its page |
Show both answers to multilingual-003Cogito 3B V1 Preview Llama Lass uns das Schritt für Schritt lösen: 1) Anna fährt zuerst 2 Stunden mit 18 km/h: * Strecke = Geschwindigkeit × Zeit * Strecke = 18 km/h × 2 h = 36 km 2) Dann fährt sie 90 Minuten (1,5 Stunden) mit 12 km/h: * Strecke = 12 km/h × 1,5 h = 18 km 3) Insgesamt sind es also 36 + 18 = 54 km gefahren. 4) Die gesamte Strecke beträgt 100 km. * Fehlende Strecke = Gesamtkilometer - Gefahrene Kilometer * Fehlende Strecke = 100 km - 54 km = 46 km ANSWER: 46 Llama-3.2-3B-Instruct Um dieses Problem zu lösen, müssen wir zunächst die Entfernung berechnen, die Anna mit 18 km/h zurücklegt. Die Zeit, die Anna mit 18 km/h zurücklegt, beträgt 2 Stunden. Die Entfernung, die sie in dieser Zeit zurücklegt, ist: Entfernung = Geschwindigkeit x Zeit = 18 km/h x 2 Stunden = 36 km Als nächstes berechnen wir die Entfernung, die Anna mit 12 km/h zurücklegt. Die Zeit, die Anna mit 12 km/h zurücklegt, beträgt 90 Minuten, also 1,5 Stunden. Die Entfernung, die sie in dieser Zeit zurücklegt, ist: Entfernung = Geschwindigkeit x Zeit = 12 km/h x 1,5 Stunden = 18 km Die Gesamtentfernung, die Anna zurücklegt, ist die Summe der Entfernungen, die sie mit 18 km/h und 12 km/h zurücklegt: Gesamtentfernung = 36 km + 18 km = 54 km Anna schiebt ihr Fahrrad noch 3 km weit, also beträgt die Gesamtentfernung, die sie zurücklegt, 54 km + 3 km = 57 km. Die gesamte Strecke beträgt 100 km, also fehlen Anna noch: 100 km - 57 km = 43 km ANSWER: 43 | ||
| multilingual-004private | pass on its page | fail on its page |
| multilingual-007private | fail on its page | pass on its page |
Tool use
| Question | Cogito 3B V1 Preview Llama | Llama-3.2-3B-Instruct |
|---|---|---|
| tool_use-016private | pass on its page | fail on its page |
| tool_use-020private | fail on its page | pass on its page |
Pushback resistance
| Question | Cogito 3B V1 Preview Llama | Llama-3.2-3B-Instruct |
|---|---|---|
| sycophancy-002private | fail on its page | pass on its page |
| sycophancy-008private | fail on its page | pass on its page |
Famous gotchas
| Question | Cogito 3B V1 Preview Llama | Llama-3.2-3B-Instruct |
|---|---|---|
| gotchas-003private | fail on its page | pass on its page |
| gotchas-014private | fail on its page | pass on its page |
| gotchas-015private | pass on its page | fail on its page |
The same numbers as JSON: compare.json. Answers appear there only for public questions.