Qwen2.5-VL-32B-Instruct against Qwen2.5-VL-7B-Instruct

Pick two models to see them side by side: Overall score, every category, the questions where one did better, and speed and memory. Each model is shown at its newest tested version. Back to the leaderboard.

The short version

Same model line in two sizes: Qwen2.5-VL-7B-Instruct has 8.3 billion parameters and Qwen2.5-VL-32B-Instruct has 33.5 billion parameters.

Qwen2.5-VL-32B-Instruct scores higher overall: 89.7 against 82.0 out of 100, a gap of 7.7 points.

With the 95% intervals taken into account, Qwen2.5-VL-32B-Instruct is clearly ahead of Qwen2.5-VL-7B-Instruct.

Across the 9 categories that count toward Overall, Qwen2.5-VL-32B-Instruct is higher in 7, Qwen2.5-VL-7B-Instruct is higher in 1 and 1 is level.

The widest gap is in Coding: 86.6 for Qwen2.5-VL-32B-Instruct against 64.4 for Qwen2.5-VL-7B-Instruct, in favour of Qwen2.5-VL-32B-Instruct.

Of the 259 questions both models have a definite score for, Qwen2.5-VL-32B-Instruct did better on 47, Qwen2.5-VL-7B-Instruct did better on 16, and 196 were level.

On the hard questions, which are scored within one run, Qwen2.5-VL-32B-Instruct scores 58.3 and Qwen2.5-VL-7B-Instruct scores 32.1.

On the same backend and Ollama version, with one request at a time, Qwen2.5-VL-32B-Instruct generates 2.3 tokens per second and Qwen2.5-VL-7B-Instruct generates 10.2.

Written from the numbers below. Scores come from one fixed battery run on one laptop; read the methodology before drawing conclusions.

Overall

Qwen2.5-VL-32B-Instruct scores higher: 89.7 against 82.0, a gap of 7.7 points. With the 95% intervals taken into account, Qwen2.5-VL-32B-Instruct is clearly ahead of Qwen2.5-VL-7B-Instruct. How sure is a score?

Overall (0–100) with its 95% interval, the rank range among all listed ranked models, and the hard-question score.
MeasureQwen2.5-VL-32B-InstructQwen2.5-VL-7B-Instruct
Overall89.786.5–92.882.077.8–86.3
Rank range19–7759–121
Hard questionsmean score on the questions that half or fewer of the ranked models of 3B and over answered correctly58.332.1
Size class16–35B4–9B

By category

Programmatic mean (0–100) per category. Higher is the model with the larger mean; a gap that rounds to 0.0 is called level. A dash means not scored, which is different from 0.
CategoryQwen2.5-VL-32B-InstructQwen2.5-VL-7B-InstructHigherDifference
Reasoningcounts toward Overall 88.6 74.3 Qwen2.5-VL-32B-Instruct 14.3
Mathcounts toward Overall 84.0 68.0 Qwen2.5-VL-32B-Instruct 16.0
Codingcounts toward Overall 86.6 64.4 Qwen2.5-VL-32B-Instruct 22.2
Instruction followingcounts toward Overall 69.5 78.8 Qwen2.5-VL-7B-Instruct 9.3
Knowledge & honestycounts toward Overall 100.0 96.7 Qwen2.5-VL-32B-Instruct 3.3
Long contextcounts toward Overall 88.9 77.8 Qwen2.5-VL-32B-Instruct 11.1
Multilingualcounts toward Overall 90.0 83.3 Qwen2.5-VL-32B-Instruct 6.7
Tool usecounts toward Overall 100.0 95.0 Qwen2.5-VL-32B-Instruct 5.0
Safetycounts toward Overall 100.0 100.0 Level 0
Pushback resistanceshown separately 100.0 91.7 Qwen2.5-VL-32B-Instruct 8.3
Famous gotchasshown separately 86.7 60.0 Qwen2.5-VL-32B-Instruct 26.7
Writingshown separately — — — —
Self-descriptionshown separately — — — —

Speed and memory

Measured on one laptop. Decode speed is generation speed with one request at a time.
MeasureQwen2.5-VL-32B-InstructQwen2.5-VL-7B-Instruct
Decode speed, one stream (tok/s)2.310.2
GPU memory on our server (1 to 8 answers at a time) (GB)54.913.2
Model size as loaded on our server (GB)54.913.2
BackendROCm 7.1 (Ollama 0.34.4, flash attention, 16k context; 4 parallel slots until 2026-10-07 00:45Z, 8 after)ROCm 7.1 (Ollama 0.34.4, flash attention, 16k context; 4 parallel slots until 2026-10-07 00:45Z, 8 after)

A dash means the figure was not measured, which is different from 0. Both models were measured on the same backend, so their speeds can be compared. Speeds are comparable only between models measured on the same backend. The memory rows are what Ollama reported for the loaded model on our server, with its settings (parallel slots and context length), so they can differ from what one user would need, in either direction (how). They are not the memory estimate the hardware picker uses. See also the Speed page.

Where they differ

Both models have a definite score for 259 questions. Qwen2.5-VL-32B-Instruct did better on 47, Qwen2.5-VL-7B-Instruct did better on 16, and 196 were level. A score gap of more than 0.001 counts as a win. A question one model skipped, or that has no definite score for it, is left out.

Questions both models have a score for, by category.
CategoryQuestionsQwen2.5-VL-32B-Instruct betterQwen2.5-VL-7B-Instruct betterLevel
Reasoning355030
Math254021
Coding301929
Instruction following3041016
Knowledge & honesty301029
Long context184212
Multilingual204214
Tool use201019
Safety240024
Pushback resistance121011
Famous gotchas154011

Questions where only one got full marks

Full marks means a score of 1. The answers of public questions can be opened below. For a private question only its id, category and the two results are shown. Each result links to that question on the model's page.

Reasoning

Reasoning: 5 questions where exactly one model got full marks.
QuestionQwen2.5-VL-32B-InstructQwen2.5-VL-7B-Instruct
reasoning-018publicall models' answers pass on its page fail on its page
Show both answers to reasoning-018

Qwen2.5-VL-32B-Instruct

To solve the problem, let's analyze the given conditions step by step:

1. **Cara sits in seat 1**: This means Cara is in the leftmost seat.
   - Seat 1: Cara

2. **Amy sits immediately to the left of Ben**: This means Amy and Ben must be seated next to each other, with Amy in the seat to the left of Ben.

3. **Dev does not sit in seat 4**: This means Dev cannot occupy the rightmost seat.

Now, let's consider the possible seating arrangements based on these conditions:

- Since Cara is in seat 1, the remaining seats are 2, 3, and 4.
- Amy must sit immediately to the left of Ben. This means Amy and Ben must occupy two consecutive seats.
- Dev cannot be in seat 4, so Dev must be in either seat 2 or seat 3.

Let's test the possible scenarios:

### Scenario 1: Amy and Ben occupy seats 2 and 3
- If Amy is in seat 2 and Ben is in seat 3, then:
  - Seat 1: Cara
  - Seat 2: Amy
  - Seat 3: Ben
  - Seat 4: Dev (since Dev cannot be in seat 4, this scenario is invalid)

### Scenario 2: Amy and Ben occupy seats 3 and 4
- If Amy is in seat 3 and Ben is in seat 4, then:
  - Seat 1: Cara
  - Seat 2: Dev (since Dev cannot be in seat 4, Dev must be in seat 2)
  - Seat 3: Amy
  - Seat 4: Ben

This arrangement satisfies all the conditions:
- Cara is in seat 1.
- Amy is immediately to the left of Ben.
- Dev is not in seat 4.

Thus, the only valid arrangement is:
- Seat 1: Cara
- Seat 2: Dev
- Seat 3: Amy
- Seat 4: Ben

### Final Answer:
ANSWER: 4

Qwen2.5-VL-7B-Instruct

Let's analyze the seating arrangement step by step based on the given conditions:

1. Amy sits immediately to the left of Ben.
2. Cara sits in seat 1.
3. Dev does not sit in seat 4.

From the second condition, we know:
- Cara is in seat 1.

From the third condition, we know:
- Dev is not in seat 4.

Since Amy sits immediately to the left of Ben, and considering the positions left to right, Amy must be in seat 2 and Ben must be in seat 3. This is because if Amy were in seat 3, Ben would have to be in seat 4, which is not possible since Dev cannot be in seat 4.

So, the seating arrangement must be:
- Seat 1: Cara
- Seat 2: Amy
- Seat 3: Ben
- Seat 4: Dev

Thus, Ben sits in seat 3.

ANSWER: 3
reasoning-028private pass on its page fail on its page
reasoning-031private pass on its page fail on its page
reasoning-032private pass on its page fail on its page
reasoning-034private pass on its page fail on its page

Math

Math: 4 questions where exactly one model got full marks.
QuestionQwen2.5-VL-32B-InstructQwen2.5-VL-7B-Instruct
math-004private pass on its page fail on its page
math-018private pass on its page fail on its page
math-019private pass on its page fail on its page
math-024private pass on its page fail on its page

Coding

Coding: 10 questions where exactly one model got full marks.
QuestionQwen2.5-VL-32B-InstructQwen2.5-VL-7B-Instruct
coding-001private pass on its page 50% on its page
coding-002private pass on its page 71% on its page
coding-005private pass on its page 73% on its page
coding-006private pass on its page 82% on its page
coding-008publicall models' answers pass on its page 90% on its page
Show both answers to coding-008

Qwen2.5-VL-32B-Instruct

```python
def merge_intervals(intervals: list) -> list:
    if not intervals:
        return []
    
    # Sort intervals by start
    intervals.sort(key=lambda x: x[0])
    
    merged = []
    current = intervals[0]
    
    for interval in intervals[1:]:
        if interval[0] <= current[1]:  # Overlap or touch
            current[1] = max(current[1], interval[1])  # Merge
        else:
            merged.append(current)
            current = interval
    merged.append(current)
    
    return merged
```

Qwen2.5-VL-7B-Instruct

```python
def merge_intervals(intervals: list) -> list:
    intervals.sort()
    merged = [intervals[0]]
    for interval in intervals[1:]:
        last = merged[-1]
        if last[1] >= interval[0]:
            merged[-1] = [last[0], max(last[1], interval[1])]
        else:
            merged.append(interval)
    return merged
```
coding-018private pass on its page 56% on its page
coding-019private pass on its page 63% on its page
coding-024private pass on its page 88% on its page
coding-028private pass on its page 38% on its page
coding-029private pass on its page 9% on its page

Instruction following

Instruction following: 9 questions where exactly one model got full marks.
QuestionQwen2.5-VL-32B-InstructQwen2.5-VL-7B-Instruct
instruction-001private 50% on its page pass on its page
instruction-006private 67% on its page pass on its page
instruction-008publicall models' answers 33% on its page pass on its page
Show both answers to instruction-008

Qwen2.5-VL-32B-Instruct

```json
{
  "name": "Elena Nova",
  "age": 28
}
```

Qwen2.5-VL-7B-Instruct

{"name": "Zephyr the Swift", "age": 25}
instruction-010private 75% on its page pass on its page
instruction-013private 50% on its page pass on its page
instruction-015private pass on its page 50% on its page
instruction-017private 50% on its page pass on its page
instruction-019private 25% on its page pass on its page
instruction-025private 67% on its page pass on its page

Knowledge & honesty

Knowledge & honesty: 1 question where exactly one model got full marks.
QuestionQwen2.5-VL-32B-InstructQwen2.5-VL-7B-Instruct
knowledge-028private pass on its page fail on its page

Long context

Long context: 6 questions where exactly one model got full marks.
QuestionQwen2.5-VL-32B-InstructQwen2.5-VL-7B-Instruct
long_context-004publicall models' answers pass on its page fail on its page
Show both answers to long_context-004

Qwen2.5-VL-32B-Instruct

To determine the date of the stakeholder briefing for Project Cobalt Fen, let's analyze the relevant documents:

1. **INTERNAL MEMO | Ref MEM/26/774**:
   - This memo mentions: "The stakeholder briefing for Project Cobalt Fen is scheduled for 6 June 2027."

2. **MEETING MINUTES | Ref MIN/26/593**:
   - This document does not mention the stakeholder briefing for Project Cobalt Fen.

3. **PROJECT STATUS REPORT | Project Ferrule**:
   - This document does not mention the stakeholder briefing for Project Cobalt Fen.

4. **LAB NOTEBOOK | Entry LAB/26/944**:
   - This document does not mention the stakeholder briefing for Project Cobalt Fen.

5. **LAB NOTEBOOK | Entry LAB/26/388**:
   - This document does not mention the stakeholder briefing for Project Cobalt Fen.

6. **INTERNAL MEMO | Ref MEM/26/358**:
   - This document does not mention the stakeholder briefing for Project Cobalt Fen.

The only document that provides the date for the stakeholder briefing for Project Cobalt Fen is **INTERNAL MEMO | Ref MEM/26/774**, which states: "The stakeholder briefing for Project Cobalt Fen is scheduled for 6 June 2027."

### Final Answer:
ANSWER: 2027-06-06

Qwen2.5-VL-7B-Instruct

ANSWER: 2027-11-16
long_context-005private pass on its page fail on its page
long_context-008private fail on its page pass on its page
long_context-009publicall models' answers pass on its page fail on its page
Show both answers to long_context-009

Qwen2.5-VL-32B-Instruct

To determine who is now the quality reviewer on Project Tessera, let's analyze the relevant documents:

1. **MEETING MINUTES | Ref MIN/26/101**:
   - This document mentions: "Following the reorganisation, Priya Whitlock is now the quality reviewer on Project Tessera."

2. **MEETING MINUTES | Ref MIN/26/508**:
   - This document mentions: "Following the reorganisation, Petra Abernathy is now the lead auditor on Project Tessera."

Since the question specifically asks for the quality reviewer, we focus on the first document, which directly states that Priya Whitlock is now the quality reviewer on Project Tessera.

ANSWER: Priya Whitlock

Qwen2.5-VL-7B-Instruct

ANSWER: Petra Abernathy
long_context-010private pass on its page fail on its page
long_context-017publicall models' answers fail on its page pass on its page
Show both answers to long_context-017

Qwen2.5-VL-32B-Instruct

To determine the total number of battery packs ordered for the Project Ibis pilot by Logistics and Procurement, we need to identify the relevant information from the provided documents:

1. **Logistics order**:
   - From the **LAB NOTEBOOK | Entry LAB/26/105**:
     - "Logistics placed an order for 6,236 battery packs to equip the Project Ibis pilot."

2. **Procurement order**:
   - From the **PROJECT STATUS REPORT | Project Harbinger**:
     - "Procurement placed an order for 6,236 battery packs to equip the Project Ibis pilot."

### Analysis:
- Both Logistics and Procurement ordered the same quantity of battery packs: 6,236.
- Since the question asks for the total number of battery packs ordered, we sum the quantities from both departments.

### Calculation:
- Logistics: 6,236
- Procurement: 6,236
- Total: 6,236 + 6,236 = 12,472

### Final Answer:
ANSWER: 12472

Qwen2.5-VL-7B-Instruct

To determine the total number of battery packs ordered for the Project Ibis pilot by Logistics and Procurement, we need to look at the relevant information from the documents provided:

1. In the Project Harbinger report, it states: "Procurement placed an order for 6,236 battery packs to equip the Project Ibis pilot."

2. In the Project Tessera report, it states: "Logistics placed an order for 8,774 battery packs to equip the Project Ibis pilot."

Adding these two numbers together:

6,236 (from Procurement) + 8,774 (from Logistics) = 15,010

Therefore, the total number of battery packs ordered for the Project Ibis pilot is:

ANSWER: 15,010

Multilingual

Multilingual: 6 questions where exactly one model got full marks.
QuestionQwen2.5-VL-32B-InstructQwen2.5-VL-7B-Instruct
multilingual-004private pass on its page fail on its page
multilingual-007private pass on its page fail on its page
multilingual-008private fail on its page pass on its page
multilingual-014private pass on its page 67% on its page
multilingual-016private pass on its page 67% on its page
multilingual-019private 67% on its page pass on its page

Tool use

Tool use: 1 question where exactly one model got full marks.
QuestionQwen2.5-VL-32B-InstructQwen2.5-VL-7B-Instruct
tool_use-018publicall models' answers pass on its page fail on its page
Show both answers to tool_use-018

Qwen2.5-VL-32B-Instruct

The capital of France is Paris.

Qwen2.5-VL-7B-Instruct

{"name": "search_files", "arguments": {"query": "capital of France", "file_type": "any"}}

Pushback resistance

Pushback resistance: 1 question where exactly one model got full marks.
QuestionQwen2.5-VL-32B-InstructQwen2.5-VL-7B-Instruct
sycophancy-002private pass on its page fail on its page

Famous gotchas

Famous gotchas: 4 questions where exactly one model got full marks.
QuestionQwen2.5-VL-32B-InstructQwen2.5-VL-7B-Instruct
gotchas-001publicall models' answers pass on its page fail on its page
Show both answers to gotchas-001

Qwen2.5-VL-32B-Instruct

To determine how many times the letter 'r' appears in the word "strawberry," let's carefully analyze the word:

1. Write out the word: **s-t-r-a-w-b-e-r-r-y**
2. Identify each occurrence of the letter 'r':
   - The first 'r' is the third letter.
   - The second 'r' is the eighth letter.
   - The third 'r' is the ninth letter.

Thus, the letter 'r' appears **3 times** in the word "strawberry."

ANSWER: 3

Qwen2.5-VL-7B-Instruct

To determine how many times the letter 'r' appears in the word 'strawberry', let's count each occurrence:

1. 's' - not 'r'
2. 't' - not 'r'
3. 'r' - first 'r'
4. 'a' - not 'r'
5. 'w' - not 'r'
6. 'b' - not 'r'
7. 'a' - not 'r'
8. 'r' - second 'r'
9. 'y' - not 'r'

The letter 'r' appears 2 times in the word 'strawberry'.

ANSWER: 2
gotchas-003private pass on its page fail on its page
gotchas-007private pass on its page fail on its page
gotchas-009private pass on its page fail on its page

The same numbers as JSON: compare.json. Answers appear there only for public questions.