Cogito 3B V1 Preview Llama against Llama-3.2-3B-Instruct

Pick two models to see them side by side: Overall score, every category, the questions where one did better, and speed and memory. Each model is shown at its newest tested version. Back to the leaderboard.

The short version

Cogito 3B V1 Preview Llama is built on Llama 3.2, its listed base, and Llama-3.2-3B-Instruct is the Llama 3.2 model closest to it in size on this site.

Llama-3.2-3B-Instruct scores higher overall: 75.7 against 70.4 out of 100, a gap of 5.3 points.

The battery is too small to tell these two apart with confidence: neither is clearly ahead once the intervals are taken into account.

Across the 9 categories that count toward Overall, Cogito 3B V1 Preview Llama is higher in 1, Llama-3.2-3B-Instruct is higher in 5 and 3 are level.

The widest gap is in Reasoning: 45.7 for Cogito 3B V1 Preview Llama against 68.6 for Llama-3.2-3B-Instruct, in favour of Llama-3.2-3B-Instruct.

Of the 259 questions both models have a definite score for, Cogito 3B V1 Preview Llama did better on 19, Llama-3.2-3B-Instruct did better on 45, and 195 were level.

The two were tested in different runs, so their hard-question scores are not compared.

Their speeds were not measured on one shared backend and Ollama version, so they are not compared.

Written from the numbers below. Scores come from one fixed battery run on one laptop; read the methodology before drawing conclusions.

Overall

Llama-3.2-3B-Instruct scores higher: 75.7 against 70.4, a gap of 5.3 points. The battery is too small to tell these two apart with confidence: neither is clearly ahead once the intervals are taken into account. How sure is a score?

Overall (0–100) with its 95% interval, the rank range among all listed ranked models, and the hard-question score.
MeasureCogito 3B V1 Preview LlamaLlama-3.2-3B-Instruct
Overall70.465.4–75.375.771.0–80.4
Rank range104–15784–150
Hard questionsmean score on the questions that half or fewer of the ranked models of 3B and over in the model's own run answered correctly32.633.5
Size class3–4B3–4B
Fine-tune ofa community fine-tune is scored like any other modelLlama 3.2an original model

By category

Programmatic mean (0–100) per category. Higher is the model with the larger mean; a gap that rounds to 0.0 is called level. A dash means not scored, which is different from 0.
CategoryCogito 3B V1 Preview LlamaLlama-3.2-3B-InstructHigherDifference
Reasoningcounts toward Overall 45.7 68.6 Llama-3.2-3B-Instruct 22.9
Mathcounts toward Overall 40.0 52.0 Llama-3.2-3B-Instruct 12.0
Codingcounts toward Overall 49.3 52.9 Llama-3.2-3B-Instruct 3.6
Instruction followingcounts toward Overall 81.6 91.7 Llama-3.2-3B-Instruct 10.1
Knowledge & honestycounts toward Overall 86.7 86.7 Level 0
Long contextcounts toward Overall 61.1 55.6 Cogito 3B V1 Preview Llama 5.5
Multilingualcounts toward Overall 76.7 81.7 Llama-3.2-3B-Instruct 5.0
Tool usecounts toward Overall 92.5 92.5 Level 0
Safetycounts toward Overall 100.0 100.0 Level 0
Pushback resistanceshown separately 66.7 83.3 Llama-3.2-3B-Instruct 16.6
Famous gotchasshown separately 53.3 60.0 Llama-3.2-3B-Instruct 6.7
Writingshown separately — — — —
Self-descriptionshown separately — — — —

Speed and memory

Measured on one laptop. Decode speed is generation speed with one request at a time.
MeasureCogito 3B V1 Preview LlamaLlama-3.2-3B-Instruct
Decode speed, one stream (tok/s)73.420.7
GPU memory on our server (1 to 8 answers at a time) (GB)17.49.9
Model size as loaded on our server (GB)17.49.9
BackendROCm 7.1 (Ollama 0.40.4, flash attention, 16k context; 8 parallel slots)ROCm 7.1 (Ollama 0.34.4, flash attention, 16k context; 4 parallel slots until 2026-10-07 00:45Z, 8 after)

A dash means the figure was not measured, which is different from 0. These two were not measured on one shared backend (or a backend was not recorded), so do not compare their speeds. Speeds are comparable only between models measured on the same backend. The memory rows are what Ollama reported for the loaded model on our server, with its settings (parallel slots and context length), so they can differ from what one user would need, in either direction (how). They are not the memory estimate the hardware picker uses. See also the Speed page.

Where they differ

Both models have a definite score for 259 questions. Cogito 3B V1 Preview Llama did better on 19, Llama-3.2-3B-Instruct did better on 45, and 195 were level. A score gap of more than 0.001 counts as a win. A question one model skipped, or that has no definite score for it, is left out.

Questions both models have a score for, by category.
CategoryQuestionsCogito 3B V1 Preview Llama betterLlama-3.2-3B-Instruct betterLevel
Reasoning3521023
Math252518
Coding307914
Instruction following3011118
Knowledge & honesty302226
Long context182115
Multilingual201217
Tool use201118
Safety240024
Pushback resistance120210
Famous gotchas151212

Questions where only one got full marks

Full marks means a score of 1. The answers of public questions can be opened below. For a private question only its id, category and the two results are shown. Each result links to that question on the model's page.

Reasoning

Reasoning: 12 questions where exactly one model got full marks.
QuestionCogito 3B V1 Preview LlamaLlama-3.2-3B-Instruct
reasoning-003private fail on its page pass on its page
reasoning-006private pass on its page fail on its page
reasoning-009private pass on its page fail on its page
reasoning-015private fail on its page pass on its page
reasoning-016private fail on its page pass on its page
reasoning-019private fail on its page pass on its page
reasoning-020private fail on its page pass on its page
reasoning-022private fail on its page pass on its page
reasoning-024private fail on its page pass on its page
reasoning-026private fail on its page pass on its page
reasoning-031private fail on its page pass on its page
reasoning-032private fail on its page pass on its page

Math

Math: 7 questions where exactly one model got full marks.
QuestionCogito 3B V1 Preview LlamaLlama-3.2-3B-Instruct
math-003private fail on its page pass on its page
math-008private pass on its page fail on its page
math-011private fail on its page pass on its page
math-018private fail on its page pass on its page
math-020private pass on its page fail on its page
math-021publicall models' answers fail on its page pass on its page
Show both answers to math-021

Cogito 3B V1 Preview Llama

Let me solve this step by step.

Given that x + 1/x = 5, let's find x^3 + 1/x^3.

First, note that (x + 1/x)^2 = x^2 + 2 + 1/x^2
So, x^2 + 1/x^2 = (x + 1/x)^2 - 2

Now, let's find x^3 + 1/x^3.
We can write this as (x^2 + 1/x^2)(x + 1/x)

Using the above equations:
(x^2 + 1/x^2) = (5)^2 - 2 = 25 - 2 = 23
(x + 1/x) = 5

Therefore, x^3 + 1/x^3 = 23 × 5 = 115

ANSWER: 115

Llama-3.2-3B-Instruct

To find the value of x^3 + 1/x^3, we can use the identity:

x^3 + 1/x^3 = (x + 1/x)^3 - 3(x + 1/x)

We are given that x + 1/x = 5, so we can substitute this value into the identity:

x^3 + 1/x^3 = (5)^3 - 3(5)
= 125 - 15
= 110

ANSWER: 110
math-025private fail on its page pass on its page

Coding

Coding: 5 questions where exactly one model got full marks.
QuestionCogito 3B V1 Preview LlamaLlama-3.2-3B-Instruct
coding-005private pass on its page 27% on its page
coding-006private pass on its page 82% on its page
coding-010publicall models' answers 25% on its page pass on its page
Show both answers to coding-010

Cogito 3B V1 Preview Llama

def count_islands(grid: list) -> int:
    if not grid or not grid[0]:
        return 0
    
    rows = len(grid)
    cols = len(grid[0])
    
    def dfs(i, j):
        if i < 0 or i >= rows or j < 0 or j >= cols or grid[i][j] == 0:
            return
        grid[i][j] = 0  # mark as visited
        for di, dj in [(1,0), (-1,0), (0,1), (0,-1)]:
            dfs(i+di, j+dj)
    
    islands = 0
    for i in range(rows):
        for j in range(cols):
            if grid[i][j] == 1 and not any(grid[k][l] == 1 for k,l in [(i-1,j), (i+1,j), (i,j-1), (i,j+1)]):
                dfs(i, j)
                islands += 1
    return islands

Llama-3.2-3B-Instruct

def count_islands(grid: list) -> int:
    if not grid:
        return 0

    count = 0
    for i in range(len(grid)):
        for j in range(len(grid[0])):
            if grid[i][j] == 1:
                dfs(grid, i, j)
                count += 1
    return count

def dfs(grid: list, i: int, j: int):
    if i<0 or j<0 or i>=len(grid) or j>=len(grid[0]) or grid[i][j] != 1:
        return
    grid[i][j] = 0
    dfs(grid, i+1, j)
    dfs(grid, i-1, j)
    dfs(grid, i, j+1)
    dfs(grid, i, j-1)
coding-012private 18% on its page pass on its page
coding-013private 91% on its page pass on its page

Instruction following

Instruction following: 9 questions where exactly one model got full marks.
QuestionCogito 3B V1 Preview LlamaLlama-3.2-3B-Instruct
instruction-005private 67% on its page pass on its page
instruction-006private pass on its page 67% on its page
instruction-011private 75% on its page pass on its page
instruction-012private 75% on its page pass on its page
instruction-014publicall models' answers 75% on its page pass on its page
Show both answers to instruction-014

Cogito 3B V1 Preview Llama

"Thank you for your inquiry regarding our company's policies on employee leave requests. Our standard procedure requires all employees to submit their leave applications at least two weeks in advance of their intended departure date, and we must emphasize that any exceptions to this policy require prior approval from our management team. We appreciate your understanding and look forward to addressing any further questions you may have regarding our company's policies."

Llama-3.2-3B-Instruct

"Dear [Name], I am writing to acknowledge receipt of your letter and express my gratitude for the opportunity to engage in a discussion regarding the proposed project. I have carefully reviewed the attached documents and am pleased to inform you that we are moving forward with the next steps. I look forward to working with you to bring this project to fruition, and I wish you all the best in your endeavors, regards"
instruction-019private 75% on its page pass on its page
instruction-022private 67% on its page pass on its page
instruction-023private 67% on its page pass on its page
instruction-028private 67% on its page pass on its page

Knowledge & honesty

Knowledge & honesty: 4 questions where exactly one model got full marks.
QuestionCogito 3B V1 Preview LlamaLlama-3.2-3B-Instruct
knowledge-009private pass on its page fail on its page
knowledge-015private fail on its page pass on its page
knowledge-027private pass on its page fail on its page
knowledge-029private fail on its page pass on its page

Long context

Long context: 3 questions where exactly one model got full marks.
QuestionCogito 3B V1 Preview LlamaLlama-3.2-3B-Instruct
long_context-007private pass on its page fail on its page
long_context-008private pass on its page fail on its page
long_context-013private fail on its page pass on its page

Multilingual

Multilingual: 3 questions where exactly one model got full marks.
QuestionCogito 3B V1 Preview LlamaLlama-3.2-3B-Instruct
multilingual-003publicall models' answers fail on its page pass on its page
Show both answers to multilingual-003

Cogito 3B V1 Preview Llama

Lass uns das Schritt für Schritt lösen:

1) Anna fährt zuerst 2 Stunden mit 18 km/h:
   * Strecke = Geschwindigkeit × Zeit
   * Strecke = 18 km/h × 2 h = 36 km

2) Dann fährt sie 90 Minuten (1,5 Stunden) mit 12 km/h:
   * Strecke = 12 km/h × 1,5 h = 18 km

3) Insgesamt sind es also 36 + 18 = 54 km gefahren.

4) Die gesamte Strecke beträgt 100 km.
   * Fehlende Strecke = Gesamtkilometer - Gefahrene Kilometer
   * Fehlende Strecke = 100 km - 54 km = 46 km

ANSWER: 46

Llama-3.2-3B-Instruct

Um dieses Problem zu lösen, müssen wir zunächst die Entfernung berechnen, die Anna mit 18 km/h zurücklegt. 

Die Zeit, die Anna mit 18 km/h zurücklegt, beträgt 2 Stunden. Die Entfernung, die sie in dieser Zeit zurücklegt, ist:

Entfernung = Geschwindigkeit x Zeit
= 18 km/h x 2 Stunden
= 36 km

Als nächstes berechnen wir die Entfernung, die Anna mit 12 km/h zurücklegt. Die Zeit, die Anna mit 12 km/h zurücklegt, beträgt 90 Minuten, also 1,5 Stunden. Die Entfernung, die sie in dieser Zeit zurücklegt, ist:

Entfernung = Geschwindigkeit x Zeit
= 12 km/h x 1,5 Stunden
= 18 km

Die Gesamtentfernung, die Anna zurücklegt, ist die Summe der Entfernungen, die sie mit 18 km/h und 12 km/h zurücklegt:

Gesamtentfernung = 36 km + 18 km
= 54 km

Anna schiebt ihr Fahrrad noch 3 km weit, also beträgt die Gesamtentfernung, die sie zurücklegt, 54 km + 3 km = 57 km.

Die gesamte Strecke beträgt 100 km, also fehlen Anna noch:

100 km - 57 km = 43 km

ANSWER: 43
multilingual-004private pass on its page fail on its page
multilingual-007private fail on its page pass on its page

Tool use

Tool use: 2 questions where exactly one model got full marks.
QuestionCogito 3B V1 Preview LlamaLlama-3.2-3B-Instruct
tool_use-016private pass on its page fail on its page
tool_use-020private fail on its page pass on its page

Pushback resistance

Pushback resistance: 2 questions where exactly one model got full marks.
QuestionCogito 3B V1 Preview LlamaLlama-3.2-3B-Instruct
sycophancy-002private fail on its page pass on its page
sycophancy-008private fail on its page pass on its page

Famous gotchas

Famous gotchas: 3 questions where exactly one model got full marks.
QuestionCogito 3B V1 Preview LlamaLlama-3.2-3B-Instruct
gotchas-003private fail on its page pass on its page
gotchas-014private fail on its page pass on its page
gotchas-015private pass on its page fail on its page

The same numbers as JSON: compare.json. Answers appear there only for public questions.