Qwen2.5-14B-Instruct against Qwen2.5-7B-Instruct

Pick two models to see them side by side: Overall score, every category, the questions where one did better, and speed and memory. Each model is shown at its newest tested version. Back to the leaderboard.

The short version

Same model line in two sizes: Qwen2.5-7B-Instruct has 7.6 billion parameters and Qwen2.5-14B-Instruct has 14.8 billion parameters.

Qwen2.5-14B-Instruct scores higher overall: 91.3 against 85.6 out of 100, a gap of 5.7 points.

With the 95% intervals taken into account, Qwen2.5-14B-Instruct is clearly ahead of Qwen2.5-7B-Instruct.

Across the 9 categories that count toward Overall, Qwen2.5-14B-Instruct is higher in 6, Qwen2.5-7B-Instruct is higher in 1 and 2 are level.

The widest gap is in Long context: 94.4 for Qwen2.5-14B-Instruct against 77.8 for Qwen2.5-7B-Instruct, in favour of Qwen2.5-14B-Instruct.

Of the 259 questions both models have a definite score for, Qwen2.5-14B-Instruct did better on 37, Qwen2.5-7B-Instruct did better on 13, and 209 were level.

On the hard questions, which are scored within one run, Qwen2.5-14B-Instruct scores 63.6 and Qwen2.5-7B-Instruct scores 44.7.

On the same backend and Ollama version, with one request at a time, Qwen2.5-14B-Instruct generates 21.7 tokens per second and Qwen2.5-7B-Instruct generates 10.1.

Written from the numbers below. Scores come from one fixed battery run on one laptop; read the methodology before drawing conclusions.

Overall

Qwen2.5-14B-Instruct scores higher: 91.3 against 85.6, a gap of 5.7 points. With the 95% intervals taken into account, Qwen2.5-14B-Instruct is clearly ahead of Qwen2.5-7B-Instruct. How sure is a score?

Overall (0–100) with its 95% interval, the rank range among all listed ranked models, and the hard-question score.
MeasureQwen2.5-14B-InstructQwen2.5-7B-Instruct
Overall91.388.0–94.285.681.4–89.3
Rank range11–7245–100
Hard questionsmean score on the questions that half or fewer of the ranked models of 3B and over answered correctly63.644.7
Size class9–16B4–9B

By category

Programmatic mean (0–100) per category. Higher is the model with the larger mean; a gap that rounds to 0.0 is called level. A dash means not scored, which is different from 0.
CategoryQwen2.5-14B-InstructQwen2.5-7B-InstructHigherDifference
Reasoningcounts toward Overall 88.6 77.1 Qwen2.5-14B-Instruct 11.5
Mathcounts toward Overall 84.0 76.0 Qwen2.5-14B-Instruct 8.0
Codingcounts toward Overall 76.5 63.2 Qwen2.5-14B-Instruct 13.3
Instruction followingcounts toward Overall 91.3 88.4 Qwen2.5-14B-Instruct 2.9
Knowledge & honestycounts toward Overall 100.0 100.0 Level 0
Long contextcounts toward Overall 94.4 77.8 Qwen2.5-14B-Instruct 16.6
Multilingualcounts toward Overall 91.7 91.7 Level 0
Tool usecounts toward Overall 95.0 100.0 Qwen2.5-7B-Instruct 5.0
Safetycounts toward Overall 100.0 95.8 Qwen2.5-14B-Instruct 4.2
Pushback resistanceshown separately 91.7 91.7 Level 0
Famous gotchasshown separately 86.7 66.7 Qwen2.5-14B-Instruct 20.0
Writingshown separately — — — —
Self-descriptionshown separately — — — —

Speed and memory

Measured on one laptop. Decode speed is generation speed with one request at a time.
MeasureQwen2.5-14B-InstructQwen2.5-7B-Instruct
Decode speed, one stream (tok/s)21.710.1
GPU memory on our server (1 to 8 answers at a time) (GB)34.88.7
Model size as loaded on our server (GB)34.88.7
BackendROCm 7.1 (Ollama 0.34.4, flash attention, 16k context; 4 parallel slots until 2026-10-07 00:45Z, 8 after)ROCm 7.1 (Ollama 0.34.4, flash attention, 16k context; 4 parallel slots until 2026-10-07 00:45Z, 8 after)

A dash means the figure was not measured, which is different from 0. Both models were measured on the same backend, so their speeds can be compared. Speeds are comparable only between models measured on the same backend. The memory rows are what Ollama reported for the loaded model on our server, with its settings (parallel slots and context length), so they can differ from what one user would need, in either direction (how). They are not the memory estimate the hardware picker uses. See also the Speed page.

Where they differ

Both models have a definite score for 259 questions. Qwen2.5-14B-Instruct did better on 37, Qwen2.5-7B-Instruct did better on 13, and 209 were level. A score gap of more than 0.001 counts as a win. A question one model skipped, or that has no definite score for it, is left out.

Questions both models have a score for, by category.
CategoryQuestionsQwen2.5-14B-Instruct betterQwen2.5-7B-Instruct betterLevel
Reasoning354031
Math253121
Coding3012315
Instruction following305322
Knowledge & honesty300030
Long context184113
Multilingual203215
Tool use200119
Safety241023
Pushback resistance120012
Famous gotchas15528

Questions where only one got full marks

Full marks means a score of 1. The answers of public questions can be opened below. For a private question only its id, category and the two results are shown. Each result links to that question on the model's page.

Reasoning

Reasoning: 4 questions where exactly one model got full marks.
QuestionQwen2.5-14B-InstructQwen2.5-7B-Instruct
reasoning-022private pass on its page fail on its page
reasoning-030private pass on its page fail on its page
reasoning-032private pass on its page fail on its page
reasoning-034private pass on its page fail on its page

Math

Math: 4 questions where exactly one model got full marks.
QuestionQwen2.5-14B-InstructQwen2.5-7B-Instruct
math-010private pass on its page fail on its page
math-017private fail on its page pass on its page
math-019private pass on its page fail on its page
math-022private pass on its page fail on its page

Coding

Coding: 4 questions where exactly one model got full marks.
QuestionQwen2.5-14B-InstructQwen2.5-7B-Instruct
coding-005private pass on its page 91% on its page
coding-012private pass on its page 18% on its page
coding-024private pass on its page 63% on its page
coding-029private pass on its page 18% on its page

Instruction following

Instruction following: 5 questions where exactly one model got full marks.
QuestionQwen2.5-14B-InstructQwen2.5-7B-Instruct
instruction-006private 67% on its page pass on its page
instruction-009private pass on its page 67% on its page
instruction-016private pass on its page 67% on its page
instruction-018private pass on its page 60% on its page
instruction-021private pass on its page 75% on its page

Long context

Long context: 5 questions where exactly one model got full marks.
QuestionQwen2.5-14B-InstructQwen2.5-7B-Instruct
long_context-004publicall models' answers pass on its page fail on its page
Show both answers to long_context-004

Qwen2.5-14B-Instruct

ANSWER: 2027-06-06

Qwen2.5-7B-Instruct

ANSWER: 2027-06-05
long_context-009publicall models' answers pass on its page fail on its page
Show both answers to long_context-009

Qwen2.5-14B-Instruct

ANSWER: Priya Whitlock

Qwen2.5-7B-Instruct

ANSWER: Petra Abernathy
long_context-010private pass on its page fail on its page
long_context-016private pass on its page fail on its page
long_context-017publicall models' answers fail on its page pass on its page
Show both answers to long_context-017

Qwen2.5-14B-Instruct

To find the total number of battery packs ordered for the Project Ibis pilot, we need to sum up the orders from both Logistics and Procurement.

From the documents:
- Procurement placed an order for 6,236 battery packs to equip the Project Ibis pilot.
- There is no mention of Logistics placing a separate order for battery packs for Project Ibis.

Therefore, the total number of battery packs ordered for the Project Ibis pilot is 6,236.

ANSWER: 6236

Qwen2.5-7B-Instruct

To determine the total number of battery packs ordered for the Project Ibis pilot by Logistics and Procurement, we need to look at the relevant information provided in the documents.

1. From the PROJECT STATUS REPORT | Project Tessera document, we see that Corvid Freight placed an order for 8,774 battery packs to equip the Project Ibis pilot.
2. From the LAB NOTEBOOK | Entry LAB/26/882, we see that Procurement placed an order for 6,236 battery packs to equip the Project Ibis pilot.

Adding these two quantities together:

8,774 (from Corvid Freight) + 6,236 (from Procurement) = 15,010

Therefore, the total number of battery packs ordered for the Project Ibis pilot is 15,010.

ANSWER: 15010

Multilingual

Multilingual: 5 questions where exactly one model got full marks.
QuestionQwen2.5-14B-InstructQwen2.5-7B-Instruct
multilingual-007private pass on its page fail on its page
multilingual-008private fail on its page pass on its page
multilingual-016private pass on its page 67% on its page
multilingual-017private pass on its page 67% on its page
multilingual-018private 33% on its page pass on its page

Tool use

Tool use: 1 question where exactly one model got full marks.
QuestionQwen2.5-14B-InstructQwen2.5-7B-Instruct
tool_use-019private fail on its page pass on its page

Safety

Safety: 1 question where exactly one model got full marks.
QuestionQwen2.5-14B-InstructQwen2.5-7B-Instruct
safety-006private pass on its page fail on its page

Famous gotchas

Famous gotchas: 7 questions where exactly one model got full marks.
QuestionQwen2.5-14B-InstructQwen2.5-7B-Instruct
gotchas-001publicall models' answers fail on its page pass on its page
Show both answers to gotchas-001

Qwen2.5-14B-Instruct

To determine how many times the letter 'r' appears in the word 'strawberry', we need to count each occurrence of 'r' in the word.

The word 'strawberry' is spelled as follows: s-t-r-a-w-b-e-r-r-y.

Counting the 'r's, we find there are two 'r's in the word 'strawberry'.

ANSWER: 2

Qwen2.5-7B-Instruct

ANSWER: 3
gotchas-002private fail on its page pass on its page
gotchas-003private pass on its page fail on its page
gotchas-007private pass on its page fail on its page
gotchas-009private pass on its page fail on its page
gotchas-010publicall models' answers pass on its page fail on its page
Show both answers to gotchas-010

Qwen2.5-14B-Instruct

To solve this problem, we need to carefully consider the conditions given and the nature of probability under these conditions. The problem states that the host does not know where the car is and opens door 3 at random, revealing a goat. This is different from the classic Monty Hall problem where the host knows where the car is and always opens a door with a goat.

Given that the host opens door 3 randomly and it reveals a goat, we need to calculate the probability that the car is behind door 2.

Let's break down the possibilities:

1. **Initial Setup:**
   - There are 3 doors: Door 1, Door 2, and Door 3.
   - One door hides a car, and the other two hide goats.
   - You pick Door 1.

2. **Random Opening:**
   - The host opens Door 3 randomly and it reveals a goat.
   - This means Door 3 could have been chosen with equal probability (1/2) since it could have been either Door 2 or Door 3 if the car was behind Door 1 (but it wasn't, since Door 3 revealed a goat).

3. **Possible Scenarios:**
   - If the car is behind Door 1, the host could have opened either Door 2 or Door 3, but since Door 3 was opened and it revealed a goat, the car must be behind Door 1.
   - If the car is behind Door 2, the host would have opened Door 3, revealing a goat.
   - If the car is behind Door 3, the host would not have opened Door 3 (since it would reveal the car).

Since the host opened Door 3 and it revealed a goat, the car cannot be behind Door 3. This leaves us with two possibilities:
- The car is behind Door 1.
- The car is behind Door 2.

Since the host's action of opening Door 3 does not provide additional information about the location of the car (because the host's action was random and not based on knowledge of the car's location), each of the remaining doors (Door 1 and Door 2) is equally likely to have the car.

Therefore, the probability that the car is behind Door 2 is 1/2.

ANSWER: 0.5

Qwen2.5-7B-Instruct

To solve this problem, we can use basic probability principles. Let's break it down step by step.

1. **Initial Setup**: There are 3 doors, and the car is behind one of them. The probability that the car is behind any specific door is \( \frac{1}{3} \).

2. **Your Choice**: You pick door 1. The probability that the car is behind door 1 is \( \frac{1}{3} \), and the probability that the car is behind either door 2 or door 3 is \( \frac{2}{3} \).

3. **Host's Action**: The host, who does not know where the car is, opens door 3 and reveals a goat. This action does not change the initial probability that the car is behind door 2, but it does change the probability that the car is behind door 1. Since door 3 has a goat, the probability that the car is behind door 1 is now 0, and the \( \frac{2}{3} \) probability that the car is behind either door 2 or door 3 is now entirely on door 2.

4. **Conclusion**: The probability that the car is behind door 2 is \( \frac{2}{3} \).

Therefore, the probability that the car is behind door 2 is \( \frac{2}{3} \).

ANSWER: 0.6667
gotchas-013publicall models' answers pass on its page fail on its page
Show both answers to gotchas-013

Qwen2.5-14B-Instruct

To determine how many minutes after taking the first pill you take the third pill, let's break down the timing:

1. You take the first pill at time 0 minutes.
2. You take the second pill 30 minutes after the first pill.
3. You take the third pill 30 minutes after the second pill.

So, the time between the first and third pills is 60 minutes.

ANSWER: 60

Qwen2.5-7B-Instruct

ANSWER: 1 hour or 60 minutes

The same numbers as JSON: compare.json. Answers appear there only for public questions.