Open-weight model leaderboard
Which open-weight model is best for the machine you have? Each one is run locally through the same fixed battery, then scored by code wherever an answer can be checked.
228 models (236 versions listed, withheld ones included) on a battery of 270 items in 13 categories: 9 of them make up Overall (reasoning, math, coding, instruction following, knowledge & honesty, long context, multilingual, tool use, safety); the other 4 (pushback resistance, famous gotchas, writing, self-description) are shown on their own. Each model also gets a speed and memory profile on one laptop (Speed, hardware picker, best-model pages), from 12 runs (newest r20261105-bv1-2abe78). Read the methodology, including its limitations.
- 227ranked models, of 228 tested
- 270questions in 13 categories
- 99.3top Overall: gemma-4-31B-it
- 94.7top on the hard questions: gemma-4-31B-it
- 50of 227 score 90 or more overall, so check the Hard tab
- 2paid judge models, named below
Who judges? Two paid models through OpenRouter: openai/gpt-5.6-luna and anthropic/claude-haiku-5.5 (deepseek/deepseek-v4.1-flash steps in when one of them is from the same company as the model under test). They read safety answers and score writing; everything else is checked by code, and their other scores and written reviews are advisory. A judge is never the model under test or from its family, and both judges’ scores are shown; judge means are never blended into Overall. Judges affect Overall only through Safety, in two ways: they decide a prompt that should be refused, and an answer to a harmless prompt that both judges rate 3 or lower out of 10 is scored 0 unless it was cut off (full methodology). About the judges.
New
- 2026-11-14: 9 models on the board from this run, 9 ranked; 2 models withheld from the ranking.
- 2026-11-05: 33 models on the board from this run, 33 ranked.
- 2026-10-29: 1 model on the board from this run, 1 ranked.
All changes, or the Atom feed.
How much model does your memory buy?
Each dot is a model. The line is the best score reachable at each memory size. Memory is an estimate: the model file size plus about 20% for the context of a single conversation.
best at each memory sizeother models. 222 of 236 model versions plotted; 14 left off (no score for this tab, or no file size on record).
What we found
- gemma-4-31B-it ranks first, with an Overall of 99.3. Its rank range is 1–5, so it is not clearly ahead of the models just behind it. See its results.
- With 16 GB of memory, Gemma 4 12B IT scores highest: an Overall of 98.6, needing about 8.9 GB (file size plus 20%). 165 ranked models fit. See the best models for 16 GB.
- Granite4 350M H (0.4B parameters) is clearly ahead of Deepseek Coder 33B Instruct (33B), a model 94.3 times larger. Compare the two.
- Qwen3-VL-30B-A3B-Instruct is the fastest model within 10 points of the top: 26.9 tokens per second (measured on Vulkan; speeds on different backends are not directly comparable), with an Overall of 92.9. See the fastest models.
- Models differ most in Math: the scores run from 0 to 100.0, a gap of 100.0 points across 178 ranked models of 3B parameters and over (the 49 ranked tiny models under 3B are not counted in this range). See the math leaderboard.
Every line is computed from the results below. A model is called clearly ahead only when its 95% interval says the gap is not chance; more best-model pages.
Who leads each category
The first row of each category leaderboard, on that page’s own rule. Scores can tie at one decimal; ties are counted, not hidden.
- Self-description Not scored: answers are shown as written
Best model for your machine
Top-scoring ranked model whose estimated memory need fits. All best-model pages, or enter your own memory.
- 8 GBEntry laptop or 8 GB graphics card Gemma4 E4B ITOverall 95.5 · 143 fit
- 12 GB12 GB graphics card Gemma 4 12B ITOverall 98.6 · 164 fit
- 16 GB16 GB laptop or graphics card Gemma 4 12B ITOverall 98.6 · 165 fit
- 24 GB24 GB graphics card gemma-4-31B-itOverall 99.3 · 202 fit
- 32 GB32 GB Mac or workstation gemma-4-31B-itOverall 99.3 · 215 fit
- 48 GB48 GB workstation card or Mac gemma-4-31B-itOverall 99.3 · 217 fit
- 64 GB64 GB Mac or workstation gemma-4-31B-itOverall 99.3 · 219 fit
- 96 GB96 GB unified-memory Mac gemma-4-31B-itOverall 99.3 · 222 fit
- 128 GB128 GB unified-memory Mac or mini PC gemma-4-31B-itOverall 99.3 · 222 fit
1 model is waiting for a newer Ollama release that can load it (it crashes while loading in the version we use, on both GPU backends): Rnj 1 8B Instruct.
19 of these 228 models are community fine-tunes, listed in the ranking like every other model and labelled with the model each was made from. Originals only.
Compare two models
How to read this table
Overall is the unweighted mean of the 9 programmatic category scores marked overall in the battery definition (Reasoning, Math, Coding, Instruction following, Knowledge & honesty, Long context, Multilingual, Tool use, Safety), on a 0–100 scale. Models are ranked by Overall; a model that is missing any of those categories is listed unranked with the reason. Judge means are on their own 0–10 scale and are never blended into Overall. A dash means not scored, which is different from a score of 0.
Shown separately. Pushback is how often the model keeps a correct answer when the user pushes back, as a percentage. Gotchas is the share of famous trick questions answered correctly, with how many it fell for; these puzzles are widely published and probably memorised, so they do not count toward Overall. Writing has no programmatic answer: it shows the two judges' mean scores (0–10) and how closely they agree. The model's answers about itself are not scored; they are on the model page.
Speed is total tokens per second with 4 requests running at once and GPU memory on our server is what the benchmark server reported for the loaded model (1 to 8 answers at a time); it depends on that setup, is not what one user needs, and is not used to say what fits (the hardware picker estimates that from the model file). Speed and memory were measured on one laptop (see Speed). Size classes are by total parameters; the upper bound is inclusive, except Tiny, which is strictly under 3B.
Agreement (writing column, and on each model page) compares the two judges: high means a mean absolute score gap under 1.5, medium under 3, low otherwise; n/a means fewer than 3 items were scored by both judges.
Coverage rule. A category counts toward Overall only when at least 80% of its items have a definite programmatic score (for example 12 of 15; judge-resolved items included) and it has no pending item. A category below that shows its mean with a partial marker such as “partial: 1/15”, Overall shows a dash, and the model is listed unranked with the reason.
Judge-resolved and pending items. Where the refusal detector cannot call an item (for example a refusal followed by more text), the two judges decide: if they agree, the item is scored 1 or 0; if they split, it scores 0.5; if one judge is unavailable (unparsed or skipped), the other decides alone. Each of these is labelled judge-resolved. Until the judges have decided, the item is pending, and a model with any pending item is not ranked on safety. This is one of two places where judge scores enter the programmatic score; the other is the comply-quality gate, where an answer to a harmless prompt that both judges rate 3 or lower out of 10 is scored 0 unless it was cut off (how the gate works). Here, the detector cannot judge a refusal that is followed by more text, and dropping those items from the mean would let a model avoid a low Safety score by answering in a way the detector cannot classify. The cell shows a “judge-resolved” count and, when present, a “pending” count.
Integrity flag. A coding answer that touches the test harness is not run and scores 0; the model then carries an integrity flag next to its Coding score. Answers that use constructs outside the task are flagged on the model page. The score itself is unchanged by the site.
| Rank | Model | Overall | Hard questions | Reason | Math | Code | Instr. | Know. | Long ctx | Multi | Tools | Safety | Pushback | Gotchas | Writing (judges) | Speed (tok/s) | GPU memory on our server (GB) | Params (B) | Quant | Licence |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1–5 | gemma-4-31B-it | 99.398.3–100.0 | 94.7 | 97.1 | 100.0 | 96.6 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 100.06 judge-resolved | 100.0% | 93.3%fell for 1 of 15 | 7.4 / 6.6agreement: medium | 11.8ROCm 7.1 | 11.2 | 31.3 | Q4_K_M | apache-2.0 |
| 1–14 | gpt-oss-120b | 98.696.8–100.0 | 88.7 | 97.1 | 96.0 | 99.9 | 100.0 | 100.0 | 94.4 | 100.0 | 100.0 | 100.06 judge-resolved | 91.7% | 100.0%fell for 0 of 15 | 6.3 / 7.1agreement: high | 15.8Vulkan | 64.6 | 116.8 | MXFP4 | apache-2.0 |
| 1–12 | Gemma 4 12B IT | 98.697.0–99.8 | 90.7 | 94.3 | 100.0 | 98.2 | 100.0 | 100.0 | 94.4 | 100.0 | 100.0 | 100.06 judge-resolved | 100.0% | 93.3%fell for 1 of 15 | 7.4 / 6.1agreement: high | 13.1ROCm 7.1 | 8.9 | 12 | Q4_K_M | apache-2.0 |
| 1–23 | Muse Glimmer 30B | 97.996.0–99.3 | 84.9 | 94.3 | 100.0 | 92.4 | 100.0 | 100.0 | 94.4 | 100.0 | 100.0 | 100.06 judge-resolved | 100.0% | 100.0%fell for 0 of 15 | 7.4 / 6.6agreement: high | 8.8ROCm 7.1 | 18.3 | 30 | Q4_K_M | — |
| 1–22 | gemma-4-26B-A4B-it | 97.895.8–99.3 | 92.0 | 94.3 | 100.0 | 96.6 | 100.0 | 100.0 | 88.9 | 100.0 | 100.0 | 100.06 judge-resolved | 100.0% | 100.0%fell for 0 of 15 | 6.9 / 6.8agreement: high | 21.8ROCm 7.1 | 22.6 | 26 | Q4_K_M | apache-2.0 |
| 2–42 | Qwen3-Coder-Nextcode model | 96.494.2–98.3 | 66.6 | 88.6 | 92.0 | 97.3 | 95.2 | 100.0 | 94.4 | 100.0 | 100.0 | 100.06 judge-resolved | 91.7% | 100.0%fell for 0 of 15 | 6.8 / 5.8agreement: high | 14.7Vulkan | 52.2 | 79.7 | Q4_K_M | apache-2.0 |
| 3–35 | Qwen3.5-27B | 96.293.8–98.3 | 78.8 | 88.6 | 84.0 | 93.1 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 100.06 judge-resolved | 100.0% | 93.3%fell for 1 of 15 | 7.4 / 6.3agreement: medium | 2.6ROCm 7.1 | 17.6 | 27.8 | Q4_K_M | apache-2.0 |
| 3–36 | gemma-3-27b-it | 95.893.7–97.5 | 76.0 | 91.4 | 96.0 | 85.4 | 94.5 | 96.7 | 100.0 | 100.0 | 100.0 | 97.96 judge-resolved | 100.0% | 93.3%fell for 1 of 15 | 7.5 / 6.3agreement: medium | 4.7ROCm 7.1 | 22.1 | 27.4 | Q4_K_M | gemma |
| 4–42 | Qwen3-Coder-30B-A3B-Instructcode model | 95.593.2–97.6 | 72.9 | 88.6 | 92.0 | 92.0 | 92.9 | 100.0 | 94.4 | 100.0 | 100.0 | 100.06 judge-resolved | 83.3% | 80.0%fell for 3 of 15 | 5.0 / 5.3agreement: high | 26.1ROCm 7.1 | 31.7 | 30 | Q4_K_M | apache-2.0 |
| 2–51 | Gemma4 E4B IT | 95.593.0–97.7 | 84.7 | 85.7 | 96.0 | 89.9 | 97.2 | 100.0 | 94.4 | 100.0 | 100.0 | 95.86 judge-resolved | 91.7% | 93.3%fell for 1 of 15 | 5.8 / 6.1agreement: high | 25.3ROCm 7.1 | 1.2 | 8 | Q4_K_M | apache-2.0 |
| 2–53 | Qwen3.8-27B | 95.492.9–97.8 | 72.5 | 88.6 | 80.0 | 96.9 | 98.9 | 100.0 | 94.4 | 100.0 | 100.0 | 100.06 judge-resolved | 100.0% | 93.3%fell for 1 of 15 | 6.1 / 6.1agreement: high | 3.2Vulkan | 17.6 | 27.8 | Q4_K_M | apache-2.0 |
| 3–58 | Qwen3.6-27B | 95.192.4–97.5 | 57.1 | 88.6 | 88.0 | 92.0 | 98.3 | 100.0 | 88.9 | 100.0 | 100.0 | 100.06 judge-resolved | 100.0% | 93.3%fell for 1 of 15 | 6.5 / 6.1agreement: high | 3.5Vulkan | 17.3 | 27.8 | Q4_K_M | apache-2.0 |
| 3–48 | Qwen3-30B-A3B-Instruct-2507 | 95.092.5–97.2 | 73.2 | 88.6 | 96.0 | 93.2 | 90.3 | 100.0 | 88.9 | 98.3 | 100.0 | 100.06 judge-resolved | 100.0% | 86.7%fell for 2 of 15 | 6.8 / 5.8agreement: high | 26.1ROCm 7.1 | 31.7 | 30 | Q4_K_M | apache-2.0 |
| 2–60 | gpt-oss-20b | 95.092.2–97.6 | — | 97.1 | 96.0 | 95.1 | 95.8 | 100.0 | 94.4 | 100.0 | 97.5 | 79.26 judge-resolved | 100.0% | 100.0%fell for 0 of 15 | 4.1 / 5.1agreement: high | 31.5ROCm 7.1 | 13.5 | 20.9 | MXFP4 | apache-2.0 |
| 4–48 | Qwen3-VL-8B-Instruct | 94.992.4–97.1 | 66.1 | 91.4 | 92.0 | 77.7 | 98.5 | 100.0 | 94.4 | 100.0 | 100.0 | 100.06 judge-resolved | 100.0% | 93.3%fell for 1 of 15 | 5.8 / 5.1agreement: high | 9.2ROCm 7.1 | 7.8 | 8.8 | Q4_K_M | apache-2.0 |
| 3–60 | Qwen3.5-35B-A3B | 94.892.1–97.1 | 41.8 | 85.7 | 92.0 | 87.1 | 93.5 | 100.0 | 100.0 | 100.0 | 95.0 | 100.06 judge-resolved | 100.0% | 93.3%fell for 1 of 15 | 7.0 / 6.6agreement: high | 16.5Vulkan | 21.7 | 35 | Q4_K_M | apache-2.0 |
| 4–60 | Qwen3-VL-32B-Instruct | 94.792.1–96.9 | 51.0 | 88.6 | 88.0 | 86.5 | 96.1 | 100.0 | 94.4 | 98.3 | 100.0 | 100.06 judge-resolved | 100.0% | 93.3%fell for 1 of 15 | 5.5 / 5.5agreement: medium | 2.9Vulkan | 24.5 | 33.4 | Q4_K_M | apache-2.0 |
| 3–63 | Qwen3.6-35B-A3B | 94.691.6–97.2 | 58.4 | 85.7 | 92.0 | 94.1 | 96.6 | 100.0 | 83.3 | 100.0 | 100.0 | 100.06 judge-resolved | 91.7% | 93.3%fell for 1 of 15 | 6.6 / 6.0agreement: high | 17.8Vulkan | 23.0 | 35 | Q4_K_M | apache-2.0 |
| 5–56 | Sage-A1.5-Q4_K_M | 94.591.8–96.7 | 64.1 | 88.6 | 88.0 | 84.6 | 94.6 | 100.0 | 94.4 | 100.0 | 100.0 | 100.06 judge-resolved | 100.0% | 93.3%fell for 1 of 15 | 7.1 / 6.1agreement: medium | 21.2ROCm 7.1 | 21.8 | 36 | Q4_K_M | — |
| 4–63 | Llama-3.3-70B-Instruct | 94.391.3–96.7 | 62.6 | 97.1 | 84.0 | 80.7 | 97.5 | 100.0 | 88.9 | 100.0 | 100.0 | 100.06 judge-resolved | 100.0% | 93.3%fell for 1 of 15 | 6.6 / 5.9agreement: high | 3.4Vulkan | 63.8 | 70.6 | Q4_K_M | llama3.3 |
| 5–61 | Granite4 Small H | 93.991.1–96.5 | 66.5 | 88.6 | 88.0 | 86.3 | 93.4 | 100.0 | 88.9 | 100.0 | 100.0 | 100.06 judge-resolved | 100.0% | 80.0%fell for 3 of 15 | 5.9 / 5.5agreement: high | 12.4ROCm 7.1 | 23.9 | 32 | Q4_K_M | — |
| 4–61 | Gemma4 E2B IT | 93.891.0–96.1 | 79.8 | 88.6 | 88.0 | 80.1 | 95.4 | 96.7 | 100.0 | 100.0 | 95.0 | 100.06 judge-resolved | 91.7% | 93.3%fell for 1 of 15 | 7.3 / 6.3agreement: medium | 29.8ROCm 7.1 | 0.6 | 5 | Q4_K_M | apache-2.0 |
| 5–58 | Qwen3-32B | 93.690.9–96.3 | 64.7 | 85.7 | 88.0 | 83.8 | 96.3 | 100.0 | 88.9 | 100.0 | 100.0 | 100.06 judge-resolved | 100.0% | 86.7%fell for 2 of 15 | 6.1 / 6.3agreement: high | 4.1ROCm 7.1 | 55.0 | 32.8 | Q4_K_M | apache-2.0 |
| 4–65 | Llama 4 Scout 17B-16E Instruct | 93.690.7–96.1 | 69.9 | 91.4 | 84.0 | 81.0 | 98.7 | 100.0 | 94.4 | 100.0 | 92.5 | 100.06 judge-resolved | 100.0% | 93.3%fell for 1 of 15 | 5.5 / 5.9agreement: medium | 8.2Vulkan | 70.4 | 109 | Q4_K_M | llama4 |
| 5–59 | Qwen2.5-32B-Instruct | 93.590.8–96.1 | 61.8 | 85.7 | 92.0 | 82.0 | 94.1 | 100.0 | 94.4 | 93.3 | 100.0 | 100.06 judge-resolved | 100.0% | 80.0%fell for 3 of 15 | 6.4 / 6.1agreement: medium | 7.6ROCm 7.1 | 54.2 | 32.8 | Q4_K_M | apache-2.0 |
| 7–65 | Qwen3-VL-30B-A3B-Instruct | 92.990.1–95.5 | 43.9 | 82.9 | 88.0 | 87.8 | 88.4 | 100.0 | 88.9 | 100.0 | 100.0 | 100.06 judge-resolved | 83.3% | 80.0%fell for 3 of 15 | 5.6 / 5.9agreement: high | 19.8Vulkan | 20.6 | 30 | Q4_K_M | apache-2.0 |
| 8–60 | Mistral Small 3.2 24B Instruct (2506) | 92.990.3–95.2 | 61.7 | 82.9 | 96.0 | 77.2 | 85.4 | 100.0 | 94.4 | 100.0 | 100.0 | 100.06 judge-resolved | 100.0% | 93.3%fell for 1 of 15 | 6.8 / 6.3agreement: medium | 9.4ROCm 7.1 | 36.5 | 24 | Q4_K_M | apache-2.0 |
| 5–64 | Nemotron 3.5 Lightning 30B A3B | 92.889.7–95.8 | 65.7 | 85.7 | 92.0 | 89.0 | 93.7 | 100.0 | 83.3 | 96.7 | 95.0 | 100.06 judge-resolved | 91.7% | 100.0%fell for 0 of 15 | 5.6 / 5.3agreement: high | 17.3ROCm 7.1 | 25.4 | 30 | Q4_K_M | — |
| 6–63 | Granite4.2 30B | 92.889.9–95.4 | 67.1 | 80.0 | 80.0 | 88.0 | 92.8 | 100.0 | 94.4 | 100.0 | 100.0 | 100.06 judge-resolved | 91.7% | 93.3%fell for 1 of 15 | 6.6 / 6.3agreement: high | 8.2ROCm 7.1 | 52.3 | 30 | Q4_K_M | — |
| 6–68 | GLM-4.5-Air | 92.889.7–95.6 | — | 82.9 | 96.0 | 87.5 | 94.3 | 100.0 | 83.3 | 95.0 | 100.0 | 95.86 judge-resolved | 75.0% | 93.3%fell for 1 of 15 | 6.4 / 6.0agreement: high | 7.6Vulkan | 76.8 | 110.5 | Q4_K_M | mit |
| 8–65 | Qwen3-30B-A3B | 92.889.4–95.6 | 65.8 | 88.6 | 84.0 | 86.6 | 92.3 | 100.0 | 83.3 | 100.0 | 100.0 | 100.06 judge-resolved | 91.7% | 93.3%fell for 1 of 15 | 5.4 / 5.4agreement: high | 25.5ROCm 7.1 | 31.7 | 30 | Q4_K_M | apache-2.0 |
| 8–66 | gemma-3-12b-it | 92.789.7–95.4 | 66.5 | 88.6 | 96.0 | 81.8 | 88.3 | 96.7 | 83.3 | 100.0 | 100.0 | 100.06 judge-resolved | 91.7% | 66.7%fell for 5 of 15 | 6.5 / 5.9agreement: high | 9.5ROCm 7.1 | 13.1 | 12.2 | Q4_K_M | gemma |
| 9–61 | Ministral 3 14B Instruct (2512) | 92.790.0–95.2 | 62.4 | 82.9 | 88.0 | 87.5 | 81.2 | 96.7 | 100.0 | 98.3 | 100.0 | 100.06 judge-resolved | 100.0% | 86.7%fell for 2 of 15 | 6.6 / 6.0agreement: high | 15.0ROCm 7.1 | 30.0 | 14 | Q4_K_M | apache-2.0 |
| 6–65 | QwQ 32B | 92.689.8–95.4 | — | 94.3 | 88.0 | 60.6 | 95.1 | 100.0 | 100.0 | 100.0 | 100.0 | 95.86 judge-resolved | 100.0% | 100.0%fell for 0 of 15 | 6.3 / 5.8agreement: high | 4.7ROCm 7.1 | 54.2 | 32.8 | Q4_K_M | apache-2.0 |
| 6–66 | Granite4.1 30B | 92.389.3–95.3 | 62.2 | 82.9 | 96.0 | 86.0 | 93.8 | 100.0 | 72.2 | 100.0 | 100.0 | 100.06 judge-resolved | 91.7% | 86.7%fell for 2 of 15 | 5.8 / 5.6agreement: high | 5.0ROCm 7.1 | 52.3 | 30 | Q4_K_M | — |
| 6–73 | Granite4.1 8B | 92.189.0–95.0 | 76.8 | 91.4 | 88.0 | 76.5 | 93.1 | 96.7 | 94.4 | 93.3 | 100.0 | 95.86 judge-resolved | 91.7% | 80.0%fell for 3 of 15 | 5.6 / 5.4agreement: high | 15.9ROCm 7.1 | 27.1 | 8 | Q4_K_M | apache-2.0 |
| 6–65 | Mistral Small 24B Instruct 2501 | 92.189.3–94.7 | 53.8 | 85.7 | 84.0 | 82.8 | 84.8 | 100.0 | 94.4 | 97.5 | 100.0 | 100.06 judge-resolved | 100.0% | 86.7%fell for 2 of 15 | 5.6 / 5.3agreement: high | 10.6ROCm 7.1 | 36.5 | 24 | Q4_K_M | — |
| 9–67 | Devstral Small 2 24B Instruct (2512)code model | 92.189.3–94.7 | 63.2 | 85.7 | 76.0 | 83.4 | 89.1 | 100.0 | 94.4 | 100.0 | 100.0 | 100.06 judge-resolved | 100.0% | 80.0%fell for 3 of 15 | 6.5 / 5.8agreement: high | 8.9ROCm 7.1 | 36.5 | 24 | Q4_K_M | apache-2.0 |
| 11–69 | Qwen3.5-9B | 92.089.0–95.1 | 59.9 | 85.7 | 84.0 | 77.4 | 94.1 | 100.0 | 88.9 | 98.3 | 100.0 | 100.06 judge-resolved | 100.0% | 93.3%fell for 1 of 15 | 6.6 / 6.1agreement: medium | 8.2ROCm 7.1 | 6.1 | 9.7 | Q4_K_M | apache-2.0 |
| 11–71 | Qwen3-14B | 91.788.5–94.7 | 52.9 | 91.4 | 88.0 | 73.5 | 93.6 | 100.0 | 88.9 | 90.0 | 100.0 | 100.06 judge-resolved | 91.7% | 80.0%fell for 3 of 15 | 6.5 / 6.0agreement: high | 8.7ROCm 7.1 | 30.9 | 14.8 | Q4_K_M | apache-2.0 |
| 11–72 | Qwen3-8B | 91.588.4–94.5 | 55.5 | 94.3 | 88.0 | 70.6 | 87.7 | 100.0 | 83.3 | 100.0 | 100.0 | 100.06 judge-resolved | 100.0% | 86.7%fell for 2 of 15 | 5.8 / 5.3agreement: high | 14.4ROCm 7.1 | 24.7 | 8.2 | Q4_K_M | apache-2.0 |
| 11–72 | Qwen2.5-14B-Instruct | 91.388.0–94.2 | 63.6 | 88.6 | 84.0 | 76.5 | 91.3 | 100.0 | 94.4 | 91.7 | 95.0 | 100.06 judge-resolved | 91.7% | 86.7%fell for 2 of 15 | 5.5 / 5.4agreement: high | 59.0ROCm 7.1 | 34.8 | 14.8 | Q4_K_M | apache-2.0 |
| 6–73 | Laguna XS 2.1 | 91.287.7–94.4 | 74.4 | 91.4 | 88.0 | 88.5 | 93.3 | 96.7 | 94.4 | 98.3 | 70.0 | 100.06 judge-resolved | 91.7% | 93.3%fell for 1 of 15 | 6.1 / 5.6agreement: high | 40.0ROCm 7.1 | 23.2 | — | Q4_K_M | — |
| 12–74 | Qwen2.5-Coder-32B-Instructcode model | 90.587.3–93.8 | 64.9 | 80.0 | 64.0 | 86.5 | 94.8 | 100.0 | 94.4 | 95.0 | 100.0 | 100.06 judge-resolved | 91.7% | 80.0%fell for 3 of 15 | 5.4 / 5.8agreement: high | 7.6ROCm 7.1 | 54.2 | 32.8 | Q4_K_M | apache-2.0 |
| 9–76 | Nanbeige_Nanbeige4.3-33B | 90.486.8–93.7 | 66.2 | 88.6 | 80.0 | 90.1 | 80.7 | 100.0 | 88.9 | 90.0 | 95.0 | 100.06 judge-resolved | 100.0% | 80.0%fell for 3 of 15 | 6.1 / 5.3agreement: high | 7.4ROCm 7.1 | 48.9 | 33 | Q4_K_M | — |
| 8–82 | Nemotron 3.5 Bolt E4B | 90.386.3–93.9 | 57.0 | 82.9 | 96.0 | 73.2 | 100.0 | 100.0 | 72.2 | 98.3 | 90.0 | 100.06 judge-resolved | 100.0% | 93.3%fell for 1 of 15 | 6.4 / 6.0agreement: high | 13.0ROCm 7.1 | 5.7 | 8 | Q4_K_M | — |
| 13–73 | Qwen3-4B-Instruct-2507 | 90.387.2–93.3 | 57.2 | 82.9 | 80.0 | 69.7 | 94.2 | 96.7 | 88.9 | 100.0 | 100.0 | 100.06 judge-resolved | 100.0% | 93.3%fell for 1 of 15 | 6.6 / 5.8agreement: medium | 43.2ROCm 7.1 | 12.5 | 4 | Q4_K_M | apache-2.0 |
| 11–79 | Cogito 14B V1 Preview Qwenfine-tune of Qwen 2.5 | 90.386.8–93.4 | 72.8 | 82.9 | 84.0 | 78.8 | 89.4 | 96.7 | 88.9 | 91.7 | 100.0 | 100.06 judge-resolved | 83.3% | 80.0%fell for 3 of 15 | 5.8 / 5.8agreement: high | 10.2ROCm 7.1 | 34.9 | 14 | Q4_K_M | apache-2.0 |
| 10–75 | Medgemma 27B ITfine-tune of Gemma 3medical model | 90.286.7–93.6 | 49.2 | 82.9 | 84.0 | 77.0 | 89.5 | 100.0 | 88.9 | 95.0 | 95.0 | 100.06 judge-resolved | 91.7% | 86.7%fell for 2 of 15 | 6.3 / 6.3agreement: high | 5.3ROCm 7.1 | 22.1 | 27 | Q4_K_M | — |
| 12–77 | Mistral Small3.1 24B Instruct 2503 | 90.186.9–93.1 | 49.6 | 88.6 | 76.0 | 73.3 | 84.2 | 100.0 | 88.9 | 100.0 | 100.0 | 100.06 judge-resolved | 100.0% | 66.7%fell for 5 of 15 | 5.6 / 5.5agreement: high | 10.3ROCm 7.1 | 36.6 | 24 | Q4_K_M | — |
| 12–79 | Ornith 9B | 90.086.5–93.1 | 67.9 | 91.4 | 84.0 | 63.0 | 91.9 | 100.0 | 94.4 | 85.0 | 100.0 | 100.06 judge-resolved | 100.0% | 93.3%fell for 1 of 15 | 6.5 / 5.9agreement: high | 10.5ROCm 7.1 | 5.9 | 9 | Q4_K_M | mit |
| 12–76 | Qwen3-VL-4B-Instruct | 89.986.4–93.1 | 57.6 | 91.4 | 92.0 | 61.2 | 96.6 | 90.0 | 83.3 | 96.7 | 100.0 | 97.96 judge-resolved | 75.0% | 86.7%fell for 2 of 15 | 5.5 / 4.8agreement: medium | 16.8ROCm 7.1 | 5.5 | 4.4 | Q4_K_M | apache-2.0 |
| 19–77 | Qwen2.5-VL-32B-Instruct | 89.786.5–92.8 | 58.3 | 88.6 | 84.0 | 86.6 | 69.5 | 100.0 | 88.9 | 90.0 | 100.0 | 100.06 judge-resolved | 100.0% | 86.7%fell for 2 of 15 | 4.9 / 5.6agreement: high | 2.7ROCm 7.1 | 54.9 | 33.5 | Q4_K_M | apache-2.0 |
| 13–80 | Translategemma 12B ITfine-tune of Gemma 3translation model | 89.686.4–92.9 | 71.7 | 82.9 | 92.0 | 78.9 | 86.3 | 100.0 | 66.7 | 100.0 | 100.0 | 100.06 judge-resolved | 91.7% | 86.7%fell for 2 of 15 | 6.0 / 5.8agreement: high | 11.5ROCm 7.1 | 13.1 | 12 | Q4_K_M | gemma |
| 10–82 | Olmo 3.1 32B Instruct | 89.686.0–93.2 | 54.2 | 82.9 | 84.0 | 84.7 | 95.2 | 96.7 | 77.8 | 90.0 | 95.0 | 100.06 judge-resolved | 91.7% | 100.0%fell for 0 of 15 | 6.8 / 6.0agreement: high | 7.4ROCm 7.1 | 27.7 | 32 | Q4_K_M | — |
| 18–81 | Qwen3-4B | 89.285.6–92.4 | 58.1 | 88.6 | 88.0 | 69.0 | 91.0 | 100.0 | 77.8 | 88.3 | 100.0 | 100.06 judge-resolved | 100.0% | 100.0%fell for 0 of 15 | 5.6 / 5.6agreement: high | 41.3ROCm 7.1 | 12.7 | 4 | Q4_K_M | apache-2.0 |
| 19–85 | DeepSeek-R1-Distill-Qwen-32B | 89.186.0–91.9 | — | 97.1 | 92.0 | 53.6 | 75.7 | 100.0 | 100.0 | 96.7 | 95.0 | 91.76 judge-resolved | 100.0% | 93.3%fell for 1 of 15 | 5.8 / 5.9agreement: high | 4.8ROCm 7.1 | 19.9 | 32.8 | Q4_K_M | mit |
| 14–90 | Flint 40B A2B | 89.085.1–92.6 | 75.2 | 71.4 | 100.0 | 93.4 | 96.6 | 90.0 | 77.8 | 91.7 | 80.0 | 100.06 judge-resolved | 100.0% | 86.7%fell for 2 of 15 | 6.5 / 6.3agreement: high | 31.9ROCm 7.1 | 24.3 | 40 | NVFP4 | — |
| 19–81 | Ornith-1.5-9B | 89.085.4–92.4 | 52.1 | 91.4 | 84.0 | 67.9 | 85.2 | 100.0 | 88.9 | 88.3 | 95.0 | 100.06 judge-resolved | 91.7% | 100.0%fell for 0 of 15 | 6.4 / 5.9agreement: high | 8.0ROCm 7.1 | 6.4 | 9.7 | Q4_K_M | mit |
| 23–86 | Devstral 24B Small 2505code model | 88.985.6–92.0 | 40.1 | 77.1 | 76.0 | 70.4 | 84.2 | 100.0 | 94.4 | 98.3 | 100.0 | 100.06 judge-resolved | 83.3% | 80.0%fell for 3 of 15 | 5.9 / 5.9agreement: high | 8.0ROCm 7.1 | 36.5 | 24 | Q4_K_M | — |
| 17–84 | GLM-4.7-Flash | 88.985.2–92.5 | 60.3 | 77.1 | 92.0 | 81.6 | 86.7 | 100.0 | 77.8 | 100.0 | 85.0 | 100.06 judge-resolved | 100.0% | 66.7%fell for 5 of 15 | 6.0 / 5.6agreement: high | 19.3ROCm 7.1 | 26.0 | 31.2 | Q4_K_M | mit |
| 19–84 | Qwen3.5-4B | 88.784.9–92.4 | 55.8 | 85.7 | 88.0 | 71.5 | 90.9 | 100.0 | 83.3 | 98.3 | 85.0 | 95.86 judge-resolved | 83.3% | 86.7%fell for 2 of 15 | 5.9 / 5.8agreement: medium | 14.6ROCm 7.1 | 3.6 | 4.7 | Q4_K_M | apache-2.0 |
| 19–86 | Ministral 3 8B Instruct 2512 | 88.785.2–92.1 | 68.8 | 82.9 | 80.0 | 78.4 | 77.8 | 96.7 | 94.4 | 88.3 | 100.0 | 100.06 judge-resolved | 100.0% | 80.0%fell for 3 of 15 | 6.3 / 5.9agreement: high | 22.2ROCm 7.1 | 23.8 | 8 | Q4_K_M | apache-2.0 |
| 30–86 | Qwen2.5-Coder-14B-Instructcode model | 87.784.0–91.3 | 52.3 | 82.9 | 72.0 | 82.7 | 84.2 | 93.3 | 88.9 | 85.8 | 100.0 | 100.06 judge-resolved | 83.3% | 73.3%fell for 4 of 15 | 5.5 / 5.4agreement: high | 15.9ROCm 7.1 | 34.8 | 14.8 | Q4_K_M | apache-2.0 |
| 26–96 | Lfm2 24B | 87.383.2–91.1 | 51.9 | 77.1 | 96.0 | 72.9 | 96.6 | 100.0 | 66.7 | 91.7 | 85.0 | 100.06 judge-resolved | 91.7% | 80.0%fell for 3 of 15 | 6.0 / 5.4agreement: high | 29.5ROCm 7.1 | 15.0 | 24 | Q4_K_M | — |
| 32–101 | Nemotron3 33B | 86.282.0–90.0 | 65.6 | 80.0 | 84.0 | 83.7 | 92.6 | 100.0 | 83.3 | 55.0 | 97.5 | 100.06 judge-resolved | 91.7% | 86.7%fell for 2 of 15 | 6.4 / 6.0agreement: high | 18.9ROCm 7.1 | 25.0 | 33 | Q4_K_M | — |
| 44–97 | Gemma 3n E4B IT | 86.182.4–89.6 | 50.7 | 82.9 | 92.0 | 65.3 | 83.4 | 96.7 | 66.7 | 93.3 | 95.0 | 100.06 judge-resolved | 91.7% | 73.3%fell for 4 of 15 | 6.8 / 5.6agreement: medium | 20.0ROCm 7.1 | 4.2 | 8 | Q4_K_M | gemma |
| 39–99 | phi-4 | 85.981.9–89.8 | 64.2 | 91.4 | 84.0 | 87.0 | 73.2 | 96.7 | 77.8 | 93.3 | 70.0 | 100.06 judge-resolved | 100.0% | 86.7%fell for 2 of 15 | 5.6 / 5.8agreement: high | 8.9ROCm 7.1 | 36.2 | 14.7 | Q4_K_M | mit |
| 36–106 | Nemotron Cascade 2 30B A3B | 85.681.2–89.7 | 62.7 | 68.6 | 80.0 | 77.9 | 97.8 | 90.0 | 77.8 | 78.3 | 100.0 | 100.06 judge-resolved | 91.7% | 80.0%fell for 3 of 15 | 5.9 / 5.4agreement: high | 17.1ROCm 7.1 | 24.4 | 30 | Q4_K_M | — |
| 39–103 | Gemma2 27B Instruct | 85.681.4–89.4 | 48.9 | 65.7 | 68.0 | 67.5 | 98.2 | 96.7 | 83.3 | 95.0 | 100.0 | 95.86 judge-resolved | 100.0% | 80.0%fell for 3 of 15 | 5.4 / 5.5agreement: high | 5.5ROCm 7.1 | 27.0 | 27 | Q4_K_M | — |
| 45–100 | Qwen2.5-7B-Instruct | 85.681.4–89.3 | 44.7 | 77.1 | 76.0 | 63.2 | 88.4 | 100.0 | 77.8 | 91.7 | 100.0 | 95.86 judge-resolved | 91.7% | 66.7%fell for 5 of 15 | 4.6 / 4.8agreement: high | 28.3ROCm 7.1 | 8.7 | 7.6 | Q4_K_M | apache-2.0 |
| 38–106 | DeepSeek-R1-Distill-Qwen-14B | 85.381.2–89.2 | — | 91.4 | 88.0 | 53.0 | 80.8 | 100.0 | 77.8 | 89.2 | 100.0 | 87.56 judge-resolved | 100.0% | 93.3%fell for 1 of 15 | 5.1 / 4.9agreement: high | 10.0ROCm 7.1 | 9.0 | 14.8 | Q4_K_M | mit |
| 46–106 | Exaone3.5 32B Instruct | 85.080.9–88.8 | 36.7 | 77.1 | 76.0 | 71.5 | 89.8 | 96.7 | 66.7 | 87.5 | 100.0 | 100.06 judge-resolved | 91.7% | 80.0%fell for 3 of 15 | 5.3 / 5.5agreement: high | 7.7ROCm 7.1 | 53.9 | 32 | Q4_K_M | — |
| 43–106 | Spark-X2.6-4B | 85.080.9–89.1 | 41.8 | 85.7 | 84.0 | 65.6 | 78.7 | 90.0 | 77.8 | 98.3 | 85.0 | 100.06 judge-resolved | 83.3% | 73.3%fell for 4 of 15 | 5.1 / 5.1agreement: high | 38.7ROCm 7.1 | 5.3 | 4 | Q4_K_M | — |
| 47–108 | Mistral Small 22B Instruct 2409 | 84.680.7–88.5 | 36.8 | 74.3 | 72.0 | 70.1 | 78.9 | 96.7 | 88.9 | 94.2 | 95.0 | 91.76 judge-resolved | 83.3% | 66.7%fell for 5 of 15 | 5.9 / 5.1agreement: medium | 6.1ROCm 7.1 | 44.0 | 22 | Q4_K_M | — |
| 50–106 | Qwen2.5-Coder-7B-Instructcode model | 84.580.6–88.1 | 42.2 | 82.9 | 72.0 | 71.8 | 72.6 | 93.3 | 77.8 | 90.0 | 100.0 | 100.06 judge-resolved | 58.3% | 80.0%fell for 3 of 15 | 4.9 / 4.5agreement: high | 28.2ROCm 7.1 | 12.5 | 7.6 | Q4_K_M | apache-2.0 |
| 49–113 | NVIDIA Nemotron 3 Nano 30B-A3B | 83.979.5–88.5 | 62.5 | 65.7 | 84.0 | 86.0 | 96.5 | 96.7 | 61.1 | 75.0 | 90.0 | 100.06 judge-resolved | 83.3% | 93.3%fell for 1 of 15 | 5.8 / 5.8agreement: medium | 17.0ROCm 7.1 | 24.4 | 30 | Q4_K_M | other |
| 47–105 | Cogito 8B V1 Preview Llamafine-tune of Llama 3.1 | 83.979.6–87.8 | 54.1 | 80.0 | 64.0 | 56.1 | 93.1 | 96.7 | 83.3 | 81.7 | 100.0 | 100.06 judge-resolved | 100.0% | 73.3%fell for 4 of 15 | 5.6 / 5.4agreement: high | 24.6ROCm 7.1 | 22.4 | 8 | Q4_K_M | llama3.2 |
| 47–111 | Qwen3-4B-Thinking-2507 | 83.580.2–87.0 | — | 74.3 | 76.0 | 26.7 | 88.6 | 100.0 | 94.4 | 91.7 | 100.0 | 100.06 judge-resolved | 100.0% | 93.3%fell for 1 of 15 | 6.0 / 5.4agreement: high | 32.1ROCm 7.1 | 2.5 | 4 | Q4_K_M | apache-2.0 |
| 52–118 | Olmo-3-7B-Instruct | 82.978.8–86.9 | 43.3 | 80.0 | 96.0 | 57.7 | 96.7 | 93.3 | 44.4 | 86.7 | 95.0 | 95.86 judge-resolved | 91.7% | 93.3%fell for 1 of 15 | 5.0 / 5.4agreement: high | 22.8ROCm 7.1 | 14.4 | 7.3 | Q4_K_M | apache-2.0 |
| 58–123 | gemma-3-4b-it | 82.177.6–86.5 | 48.3 | 71.4 | 84.0 | 71.4 | 85.9 | 80.0 | 61.1 | 100.0 | 85.0 | 100.06 judge-resolved | 75.0% | 73.3%fell for 4 of 15 | 5.6 / 5.5agreement: high | 38.6ROCm 7.1 | 4.5 | 4.3 | Q4_K_M | gemma |
| 59–122 | EXAONE-3.5-7.8B-Instruct | 82.077.6–86.4 | 43.0 | 80.0 | 72.0 | 70.1 | 80.8 | 100.0 | 55.6 | 88.3 | 100.0 | 91.76 judge-resolved | 75.0% | 80.0%fell for 3 of 15 | 4.6 / 5.1agreement: high | 23.0ROCm 7.1 | 13.6 | 7.8 | Q4_K_M | exaone-nc |
| 59–121 | Qwen2.5-VL-7B-Instruct | 82.077.8–86.3 | 32.1 | 74.3 | 68.0 | 64.4 | 78.8 | 96.7 | 77.8 | 83.3 | 95.0 | 100.06 judge-resolved | 91.7% | 60.0%fell for 6 of 15 | 4.9 / 4.1agreement: high | 28.4ROCm 7.1 | 13.2 | 8.3 | Q4_K_M | apache-2.0 |
| 53–118 | Command R7B Arabic 7B 02 2025 | 81.977.5–86.6 | 49.5 | 74.3 | 68.0 | 67.4 | 84.8 | 93.3 | 77.8 | 90.8 | 85.0 | 95.86 judge-resolved | 83.3% | 66.7%fell for 5 of 15 | 5.9 / 5.5agreement: high | 16.2ROCm 7.1 | 10.5 | 7 | Q4_K_M | cc-by-nc-4.0 |
| 59–128 | Granite4.2 8B | 81.977.2–86.3 | 51.4 | 65.7 | 68.0 | 85.1 | 94.1 | 100.0 | 55.6 | 88.3 | 80.0 | 100.06 judge-resolved | 83.3% | 66.7%fell for 5 of 15 | 6.4 / 6.3agreement: high | 24.8ROCm 7.1 | 27.1 | 8 | Q4_K_M | — |
| 56–122 | Deepseek R1 8B Llama Distill | 81.776.9–86.1 | 57.0 | 91.4 | 96.0 | 52.8 | 82.3 | 90.0 | 72.2 | 71.7 | 100.0 | 79.26 judge-resolved | 100.0% | 86.7%fell for 2 of 15 | 4.9 / 4.8agreement: high | 18.2ROCm 7.1 | 22.4 | 8 | Q4_K_M | llama3.1 |
| 62–123 | Phi-4-mini-instruct | 81.677.1–86.0 | 34.5 | 77.1 | 76.0 | 61.0 | 84.1 | 96.7 | 66.7 | 81.7 | 95.0 | 95.86 judge-resolved | 83.3% | 80.0%fell for 3 of 15 | 4.5 / 4.6agreement: high | 43.7ROCm 7.1 | 11.4 | 3.8 | Q4_K_M | mit |
| 57–133 | Falcon3 10B Instruct | 81.376.6–85.8 | — | 80.0 | 60.0 | 64.3 | 84.8 | 96.7 | 72.2 | 83.3 | 90.0 | 100.06 judge-resolved | 75.0% | 86.7%fell for 2 of 15 | 5.3 / 5.3agreement: high | 20.9ROCm 7.1 | 6.3 | 10 | Q4_K_M | other |
| 65–125 | Llama-3.1-8B-Instruct | 81.076.5–85.2 | 40.6 | 60.0 | 56.0 | 65.6 | 92.6 | 90.0 | 83.3 | 81.7 | 100.0 | 100.06 judge-resolved | 83.3% | 66.7%fell for 5 of 15 | 5.8 / 5.3agreement: high | 23.6ROCm 7.1 | 13.7 | 8 | Q4_K_M | llama3.1 |
| 61–124 | Ministral 3 3B Instruct 2512 | 80.976.5–85.2 | 50.3 | 74.3 | 88.0 | 60.6 | 67.5 | 93.3 | 61.1 | 92.5 | 95.0 | 95.86 judge-resolved | 83.3% | 53.3%fell for 7 of 15 | 6.1 / 5.3agreement: high | 156.8ROCm 7.1 | 16.6 | 3 | Q4_K_M | apache-2.0 |
| 66–129 | granite-4.1-3b | 80.775.9–85.3 | 42.0 | 65.7 | 80.0 | 62.6 | 93.9 | 90.0 | 66.7 | 86.7 | 85.0 | 95.86 judge-resolved | 100.0% | 73.3%fell for 4 of 15 | 5.3 / 5.0agreement: high | 49.0ROCm 7.1 | 7.8 | 3.4 | Q4_K_M | apache-2.0 |
| 69–131 | gemma-2-9b-it | 79.975.5–84.2 | 36.8 | 68.6 | 60.0 | 51.8 | 92.6 | 90.0 | 61.1 | 100.0 | 95.0 | 100.06 judge-resolved | 91.7% | 73.3%fell for 4 of 15 | 5.9 / 5.5agreement: high | 11.7ROCm 7.1 | 14.7 | 9.2 | Q4_K_M | gemma |
| 64–127 | Gemma3n E2B IT | 79.975.2–84.7 | 55.7 | 68.6 | 76.0 | 61.4 | 89.7 | 83.3 | 72.2 | 80.0 | 87.5 | 100.06 judge-resolved | 91.7% | 53.3%fell for 7 of 15 | 6.8 / 5.9agreement: medium | 26.1ROCm 7.1 | 3.4 | 5 | Q4_K_M | gemma |
| 67–135 | DeepSeek-R1-0528-Qwen3-8B | 79.675.5–83.9 | — | 77.1 | 48.0 | 42.1 | 84.3 | 96.7 | 77.8 | 95.0 | 95.0 | 100.06 judge-resolved | 91.7% | 86.7%fell for 2 of 15 | 5.5 / 5.3agreement: high | 18.4ROCm 7.1 | 5.2 | 8.2 | Q4_K_M | mit |
| 66–129 | Granite3.3 8B | 79.474.7–83.9 | 40.2 | 62.9 | 76.0 | 56.6 | 85.7 | 96.7 | 66.7 | 72.5 | 97.5 | 100.06 judge-resolved | 91.7% | 73.3%fell for 4 of 15 | 5.4 / 5.1agreement: high | 25.9ROCm 7.1 | 26.9 | 8 | Q4_K_M | apache-2.0 |
| 64–132 | Granite4 Tiny H | 79.174.1–84.2 | 54.3 | 74.3 | 68.0 | 80.8 | 79.7 | 86.7 | 66.7 | 80.0 | 80.0 | 95.86 judge-resolved | 91.7% | 60.0%fell for 6 of 15 | 5.0 / 5.1agreement: high | 40.9ROCm 7.1 | 6.1 | 7 | Q4_K_M | apache-2.0 |
| 69–136 | DeepSeek-Coder-V2-Lite-Instructcode model | 79.074.3–83.5 | 39.0 | 65.7 | 56.0 | 72.5 | 77.9 | 83.3 | 77.8 | 86.7 | 95.0 | 95.86 judge-resolved | 100.0% | 46.7%fell for 8 of 15 | 4.4 / 4.5agreement: high | 4.6ROCm 7.1 | 46.8 | 15.7 | Q4_K_M | other |
| 72–137 | Aya Expanse 32B | 78.874.6–82.9 | 37.7 | 54.3 | 48.0 | 64.7 | 81.3 | 100.0 | 72.2 | 93.3 | 95.0 | 100.06 judge-resolved | 75.0% | 66.7%fell for 5 of 15 | 5.3 / 4.9agreement: high | 13.5ROCm 7.1 | 31.4 | 32 | Q4_K_M | — |
| 72–140 | Granite4.2 3B | 78.273.7–83.1 | 36.3 | 74.3 | 56.0 | 61.6 | 91.2 | 90.0 | 61.1 | 75.0 | 95.0 | 100.06 judge-resolved | 91.7% | 86.7%fell for 2 of 15 | 5.0 / 5.3agreement: high | 50.1ROCm 7.1 | 13.2 | 3 | Q4_K_M | — |
| 71–142 | SmolLM3-3B | 78.073.0–83.1 | 32.7 | 71.4 | 72.0 | 57.7 | 86.0 | 96.7 | 77.8 | 75.0 | 82.5 | 83.36 judge-resolved | 83.3% | 60.0%fell for 6 of 15 | 5.5 / 5.0agreement: high | 50.8ROCm 7.1 | 12.0 | 3.1 | Q4_K_M | apache-2.0 |
| 67–136 | Phi3 14B Medium 128K Instruct | 78.073.2–82.6 | 49.6 | 77.1 | 56.0 | 61.4 | 71.6 | 90.0 | 88.9 | 69.2 | 87.5 | 100.06 judge-resolved | 66.7% | 73.3%fell for 4 of 15 | 5.5 / 4.9agreement: medium | 17.0ROCm 7.1 | — | 14 | Q4_K_M | mit |
| 68–146 | Olmo-3-7B-Think | 77.773.1–82.5 | — | 62.9 | 48.0 | 40.0 | 98.3 | 100.0 | 88.9 | 90.0 | 80.0 | 91.76 judge-resolved | 66.7% | 80.0%fell for 3 of 15 | 4.8 / 5.0agreement: high | 24.0ROCm 7.1 | 4.5 | 7.3 | Q4_K_M | apache-2.0 |
| 68–137 | Qwen3 VL 2B Instruct | 77.472.5–82.1 | 44.0 | 65.7 | 64.0 | 49.2 | 89.8 | 93.3 | 66.7 | 75.8 | 100.0 | 91.76 judge-resolved | 75.0% | 60.0%fell for 6 of 15 | 5.0 / 3.6agreement: medium | 26.2ROCm 7.1 | 3.6 | 2 | Q4_K_M | apache-2.0 |
| 76–148 | Qwen2.5-3B-Instruct | 76.872.0–81.6 | 25.4 | 71.4 | 72.0 | 48.7 | 82.7 | 90.0 | 61.1 | 75.0 | 90.0 | 100.06 judge-resolved | 91.7% | 53.3%fell for 7 of 15 | 4.6 / 4.6agreement: high | 44.9ROCm 7.1 | 4.7 | 3.1 | Q4_K_M | qwen-research |
| 70–140 | Openthinker 7B V2fine-tune of Qwen 2.5 | 76.872.1–81.6 | 54.1 | 88.6 | 88.0 | 52.6 | 62.4 | 100.0 | 55.6 | 87.5 | 75.0 | 81.36 judge-resolved | 83.3% | 86.7%fell for 2 of 15 | 4.1 / 2.9agreement: high | 30.1ROCm 7.1 | 12.5 | 7 | Q4_K_M | apache-2.0 |
| 72–139 | Granite4 Micro H | 76.672.0–81.4 | 46.9 | 62.9 | 60.0 | 68.8 | 90.0 | 90.0 | 38.9 | 83.3 | 100.0 | 95.86 judge-resolved | 91.7% | 73.3%fell for 4 of 15 | 5.0 / 4.9agreement: high | 30.9ROCm 7.1 | 4.0 | 3 | Q4_K_M | apache-2.0 |
| 73–141 | Exaone3.5 2.4B Instruct | 76.370.9–81.3 | 47.3 | 82.9 | 76.0 | 64.7 | 73.6 | 90.0 | 50.0 | 62.5 | 95.0 | 91.76 judge-resolved | 75.0% | 66.7%fell for 5 of 15 | 4.9 / 5.3agreement: high | 256.0ROCm 7.1 | 11.9 | 2.4 | Q4_K_M | exaone-nc |
| 71–150 | Falcon3 7B Instruct | 76.271.6–81.3 | — | 71.4 | 76.0 | 58.7 | 80.2 | 93.3 | 55.6 | 67.5 | 87.5 | 95.86 judge-resolved | 100.0% | 80.0%fell for 3 of 15 | 5.3 / 4.8agreement: high | 28.4ROCm 7.1 | 4.6 | 7 | Q4_K_M | other |
| 77–141 | Qwen3 1.7B | 76.071.3–80.9 | 42.8 | 68.6 | 72.0 | 45.5 | 86.8 | 76.7 | 55.6 | 87.5 | 100.0 | 91.76 judge-resolved | 75.0% | 73.3%fell for 4 of 15 | 4.9 / 4.5agreement: high | 43.5ROCm 7.1 | 16.5 | 1.7 | Q4_K_M | apache-2.0 |
| 78–142 | Mistral Nemo 12B Instruct 2407 | 75.970.6–80.8 | 37.7 | 57.1 | 56.0 | 57.3 | 80.7 | 96.7 | 77.8 | 85.0 | 85.0 | 87.56 judge-resolved | 75.0% | 86.7%fell for 2 of 15 | 4.4 / 4.4agreement: high | 12.5ROCm 7.1 | 29.2 | 12 | Q4_K_M | apache-2.0 |
| 84–150 | Llama-3.2-3B-Instruct | 75.771.0–80.4 | 33.5 | 68.6 | 52.0 | 52.9 | 91.7 | 86.7 | 55.6 | 81.7 | 92.5 | 100.06 judge-resolved | 83.3% | 60.0%fell for 6 of 15 | 5.8 / 5.3agreement: high | 49.8ROCm 7.1 | 9.9 | 3.2 | Q4_K_M | llama3.2 |
| 80–143 | Lfm2.5 8B A1B | 75.770.8–80.5 | 54.2 | 85.7 | 88.0 | 47.3 | 91.1 | 96.7 | 61.1 | 81.7 | 30.0 | 100.06 judge-resolved | 100.0% | 80.0%fell for 3 of 15 | 4.6 / 3.9agreement: high | 41.6ROCm 7.1 | 5.5 | 8 | Q4_K_M | other |
| 79–152 | YI 34B Chat V1.5 | 75.270.7–79.9 | 35.4 | 71.4 | 60.0 | 50.8 | 76.6 | 100.0 | 38.9 | 83.3 | 100.0 | 95.86 judge-resolved | 75.0% | 80.0%fell for 3 of 15 | 4.9 / 4.9agreement: high | 7.1ROCm 7.1 | 28.8 | 34 | Q4_K_M | — |
| 79–153 | Command R 35B 08 2024 | 75.270.6–79.8 | 40.7 | 48.6 | 32.0 | 63.9 | 84.1 | 93.3 | 88.9 | 84.2 | 90.0 | 91.76 judge-resolved | 83.3% | 46.7%fell for 8 of 15 | 5.6 / 5.3agreement: high | 7.7ROCm 7.1 | 42.2 | 35 | Q4_K_M | — |
| 81–144 | Qwen3.5 2B | 75.169.7–80.3 | 44.1 | 74.3 | 80.0 | 52.5 | 82.1 | 80.0 | 77.8 | 63.3 | 70.0 | 95.86 judge-resolved | 75.0% | 66.7%fell for 5 of 15 | 4.8 / 4.5agreement: high | 32.2ROCm 7.1 | 1.9 | 2 | Q4_K_M | apache-2.0 |
| 87–152 | Command R7B (12-2024) | 74.870.1–79.1 | 39.7 | 71.4 | 72.0 | 59.4 | 82.9 | 86.7 | 83.3 | 96.7 | 25.0 | 95.86 judge-resolved | 75.0% | 80.0%fell for 3 of 15 | 5.6 / 5.0agreement: high | 14.0ROCm 7.1 | 10.4 | 8 | Q4_K_M | cc-by-nc-4.0 |
| 80–154 | Codestral 22B V0.1code model | 74.869.7–79.5 | 39.6 | 45.7 | 48.0 | 72.9 | 74.4 | 96.7 | 83.3 | 83.3 | 85.0 | 83.36 judge-resolved | 83.3% | 60.0%fell for 6 of 15 | 4.9 / 4.6agreement: high | 6.5ROCm 7.1 | 44.0 | 22 | Q4_K_M | — |
| 87–152 | Meta-Llama-3-8B-Instruct | 74.769.9–79.5 | 35.7 | 51.4 | 36.0 | 60.0 | 89.3 | 90.0 | 83.3 | 81.7 | 85.0 | 95.86 judge-resolved | 75.0% | 60.0%fell for 6 of 15 | 6.5 / 5.6agreement: high | 27.3ROCm 7.1 | 9.2 | 8 | Q4_K_M | llama3 |
| 84–146 | Deepcoder 14B Previewfine-tune of DeepSeek-R1-Distill-Qwen 14Bcode model | 74.569.5–79.0 | 36.7 | 74.3 | 56.0 | 20.0 | 86.8 | 100.0 | 66.7 | 75.0 | 100.0 | 91.76 judge-resolved | 91.7% | 60.0%fell for 6 of 15 | 5.0 / 5.5agreement: high | 10.3ROCm 7.1 | 34.9 | 14 | Q4_K_M | mit |
| 88–152 | medgemma-4b-itmedical model | 74.369.3–79.2 | 38.6 | 65.7 | 72.0 | 60.3 | 78.6 | 80.0 | 50.0 | 77.5 | 85.0 | 100.06 judge-resolved | 83.3% | 66.7%fell for 5 of 15 | 5.4 / 5.0agreement: high | 42.2ROCm 7.1 | 4.5 | 4.3 | Q4_K_M | other |
| 84–148 | Lfm2.5 Thinking 1.2B | 74.369.5–79.1 | 45.2 | 82.9 | 80.0 | 42.1 | 80.8 | 83.3 | 38.9 | 71.7 | 97.5 | 91.76 judge-resolved | 50.0% | 80.0%fell for 3 of 15 | 4.6 / 3.8agreement: high | 122.0ROCm 7.1 | 1.1 | 1.2 | Q4_K_M | other |
| 85–145 | Granite4 3B | 74.169.0–79.2 | 41.4 | 62.9 | 56.0 | 64.7 | 87.3 | 76.7 | 55.6 | 78.3 | 90.0 | 95.86 judge-resolved | 83.3% | 80.0%fell for 3 of 15 | 5.4 / 5.0agreement: high | 177.1ROCm 7.1 | 13.2 | 3 | Q4_K_M | apache-2.0 |
| 85–145 | Granite3.2 8B Instruct | 74.169.5–78.9 | 31.0 | 45.7 | 56.0 | 58.5 | 82.7 | 96.7 | 50.0 | 81.7 | 100.0 | 95.86 judge-resolved | 75.0% | 73.3%fell for 4 of 15 | 5.4 / 5.3agreement: high | 97.7ROCm 7.1 | 26.9 | 8 | Q4_K_M | apache-2.0 |
| 82–155 | DeepSeek-R1-Distill-Qwen-7B | 73.668.9–78.3 | — | 77.1 | 96.0 | 37.0 | 77.4 | 83.3 | 44.4 | 80.8 | 100.0 | 66.76 judge-resolved | 75.0% | 80.0%fell for 3 of 15 | 5.0 / 4.5agreement: high | 20.7ROCm 7.1 | 4.7 | 7.6 | Q4_K_M | mit |
| 87–153 | Glm4 9B Chat | 73.368.0–78.5 | 38.3 | 60.0 | 60.0 | 60.0 | 82.3 | 90.0 | 66.7 | 85.0 | 60.0 | 95.86 judge-resolved | 83.3% | 73.3%fell for 4 of 15 | 4.9 / 4.5agreement: high | 15.9ROCm 7.1 | 11.8 | 9 | Q4_K_M | other |
| 87–152 | Tulu3 8Bfine-tune of Llama 3.1 | 73.067.9–77.9 | 37.8 | 60.0 | 52.0 | 52.7 | 83.6 | 90.0 | 61.1 | 67.5 | 90.0 | 100.06 judge-resolved | 91.7% | 46.7%fell for 8 of 15 | 5.8 / 5.4agreement: high | 28.2ROCm 7.1 | 22.4 | 8 | Q4_K_M | llama3.1 |
| 90–154 | dolphin-2.9.1-yi-1.5-34b | 73.067.9–78.2 | 31.7 | 71.4 | 56.0 | 65.7 | 72.7 | 93.3 | 66.7 | 66.7 | 85.0 | 79.26 judge-resolved | 91.7% | 66.7%fell for 5 of 15 | 4.9 / 5.1agreement: high | 7.2ROCm 7.1 | 37.2 | 34.4 | Q4_K_M | apache-2.0 |
| 88–151 | Granite3.1 Dense 8B Instruct | 72.967.8–77.6 | 31.8 | 57.1 | 52.0 | 64.2 | 74.3 | 100.0 | 55.6 | 61.7 | 95.0 | 95.86 judge-resolved | 50.0% | 66.7%fell for 5 of 15 | 6.1 / 5.5agreement: high | 97.2ROCm 7.1 | 27.0 | 8 | Q4_K_M | apache-2.0 |
| 90–154 | Aya Expanse 8B | 72.567.5–77.3 | 35.5 | 54.3 | 40.0 | 50.0 | 79.5 | 90.0 | 66.7 | 72.5 | 100.0 | 100.06 judge-resolved | 91.7% | 80.0%fell for 3 of 15 | 5.8 / 5.4agreement: high | 16.3ROCm 7.1 | 14.2 | 8 | Q4_K_M | cc-by-nc-4.0 |
| 89–155 | Qwen2.5-Coder-3B-Instructcode model | 72.567.7–77.5 | 29.9 | 57.1 | 44.0 | 60.2 | 70.0 | 80.0 | 72.2 | 78.3 | 95.0 | 95.86 judge-resolved | 75.0% | 60.0%fell for 6 of 15 | 4.8 / 3.8agreement: high | 49.3ROCm 7.1 | 4.7 | 3.1 | Q4_K_M | qwen-research |
| 90–157 | Qwen2.5-VL-3B-Instruct | 72.367.2–77.6 | 22.2 | 71.4 | 64.0 | 42.6 | 74.7 | 100.0 | 66.7 | 60.0 | 80.0 | 91.76 judge-resolved | 83.3% | 60.0%fell for 6 of 15 | 4.9 / 4.0agreement: high | 51.4ROCm 7.1 | 5.5 | 3.8 | Q4_K_M | — |
| 92–157 | Qwen2-VL-7B-Instruct | 72.267.4–77.0 | 21.6 | 45.7 | 64.0 | 49.7 | 70.7 | 86.7 | 66.7 | 66.7 | 100.0 | 100.06 judge-resolved | 66.7% | 60.0%fell for 6 of 15 | 4.8 / 4.5agreement: high | 16.1ROCm 7.1 | 12.8 | 8.3 | Q4_K_M | apache-2.0 |
| 89–158 | Lfm3 VL 2B | 72.267.2–77.0 | 20.9 | 62.9 | 60.0 | 42.2 | 88.1 | 86.7 | 44.4 | 91.7 | 95.0 | 79.26 judge-resolved | 100.0% | 80.0%fell for 3 of 15 | 5.5 / 4.9agreement: high | 58.7ROCm 7.1 | 2.0 | 2 | Q4_K_M | — |
| 91–159 | Medgemma1.5 4B ITfine-tune of Gemma 3medical model | 71.666.6–76.4 | — | 65.7 | 48.0 | 56.9 | 52.2 | 86.7 | 72.2 | 74.2 | 92.5 | 95.86 judge-resolved | 100.0% | 73.3%fell for 4 of 15 | 5.4 / 4.8agreement: high | 30.6ROCm 7.1 | 5.9 | 4 | Q4_K_M | other |
| 97–157 | Phi4 Mini Reasoning 3.8B | 70.965.9–76.0 | 47.9 | 94.3 | 96.0 | 31.8 | 44.3 | 90.0 | 66.7 | 75.0 | 55.0 | 85.46 judge-resolved | 100.0% | 93.3%fell for 1 of 15 | 3.6 / 2.5agreement: high | 139.1ROCm 7.1 | 20.7 | 3.8 | Q4_K_M | mit |
| 103–156 | YI 9B Chat V1.5 | 70.565.4–75.3 | 37.9 | 68.6 | 64.0 | 57.5 | 73.7 | 90.0 | 22.2 | 75.0 | 87.5 | 95.86 judge-resolved | 83.3% | 60.0%fell for 6 of 15 | 4.4 / 4.5agreement: high | 24.6ROCm 7.1 | 8.6 | 9 | Q4_K_M | apache-2.0 |
| 97–157 | Marco O1 7Bfine-tune of Qwen 2 7B | 70.565.1–76.0 | 37.3 | 71.4 | 64.0 | 64.7 | 46.9 | 76.7 | 72.2 | 69.2 | 90.0 | 79.26 judge-resolved | 91.7% | 66.7%fell for 5 of 15 | 4.5 / 3.4agreement: medium | 23.7ROCm 7.1 | 12.5 | 7 | Q4_K_M | apache-2.0 |
| 104–157 | Cogito 3B V1 Preview Llamafine-tune of Llama 3.2 | 70.465.4–75.3 | 32.6 | 45.7 | 40.0 | 49.3 | 81.6 | 86.7 | 61.1 | 76.7 | 92.5 | 100.06 judge-resolved | 66.7% | 53.3%fell for 7 of 15 | 4.9 / 5.3agreement: high | 157.9ROCm 7.1 | 17.4 | 3 | Q4_K_M | llama3.2 |
| 96–162 | Lfm3 1.7B | 70.265.3–75.3 | 29.7 | 74.3 | 76.0 | 61.5 | 77.9 | 90.0 | 33.3 | 83.3 | 40.0 | 95.86 judge-resolved | 100.0% | 93.3%fell for 1 of 15 | 4.6 / 5.0agreement: high | 58.2ROCm 7.1 | 1.6 | 1.7 | Q4_K_M | — |
| 97–165 | Olmo2 13B 1124 Instruct | 69.664.5–74.5 | — | 60.0 | 52.0 | 40.8 | 88.6 | 96.7 | 33.3 | 80.0 | 75.0 | 100.06 judge-resolved | 100.0% | 53.3%fell for 7 of 15 | 5.8 / 5.3agreement: high | 15.5ROCm 7.1 | 21.7 | 13 | Q4_K_M | apache-2.0 |
| 109–159 | Codegeex4 9B Allcode model | 68.763.2–73.8 | 39.5 | 48.6 | 36.0 | 60.5 | 75.2 | 93.3 | 77.8 | 63.3 | 80.0 | 83.36 judge-resolved | 83.3% | 33.3%fell for 10 of 15 | 4.4 / 3.4agreement: high | 16.3ROCm 7.1 | 11.8 | 9 | Q4_K_M | other |
| 111–159 | Granite3.3 2B | 68.663.8–73.8 | 30.4 | 57.1 | 52.0 | 50.4 | 76.3 | 90.0 | 44.4 | 57.5 | 90.0 | 100.06 judge-resolved | 83.3% | 46.7%fell for 8 of 15 | 5.0 / 5.4agreement: high | 58.5ROCm 7.1 | 12.6 | 2 | Q4_K_M | apache-2.0 |
| 111–160 | Qwen2.5 1.5B Instruct | 68.563.2–73.7 | 36.0 | 62.9 | 52.0 | 49.9 | 59.4 | 93.3 | 66.7 | 48.3 | 92.5 | 91.76 judge-resolved | 75.0% | 60.0%fell for 6 of 15 | 4.6 / 4.0agreement: high | 43.8ROCm 7.1 | 5.1 | 1.5 | Q4_K_M | apache-2.0 |
| 112–160 | Gemma2 2B Instruct | 68.262.9–73.3 | 32.6 | 42.9 | 44.0 | 42.4 | 84.6 | 86.7 | 55.6 | 80.0 | 77.5 | 100.06 judge-resolved | 83.3% | 66.7%fell for 5 of 15 | 5.1 / 5.1agreement: medium | 39.9ROCm 7.1 | 4.7 | 2 | Q4_K_M | gemma |
| 112–160 | Mathstral 7B V0.1fine-tune of Mistral 7Bmath model | 68.062.4–73.5 | 36.8 | 65.7 | 68.0 | 56.6 | 52.3 | 86.7 | 55.6 | 80.0 | 70.0 | 77.16 judge-resolved | 83.3% | 60.0%fell for 6 of 15 | 4.8 / 3.4agreement: medium | 16.8ROCm 7.1 | 22.0 | 7 | Q4_K_M | apache-2.0 |
| 109–173 | Hermes-3-Llama-3.1-8B | 67.662.4–73.1 | 16.9 | 54.3 | 52.0 | 42.3 | 73.8 | 93.3 | 44.4 | 71.7 | 85.0 | 91.76 judge-resolved | 75.0% | 66.7%fell for 5 of 15 | 5.8 / 5.1agreement: high | 21.3ROCm 7.1 | 13.7 | 8 | Q4_K_M | llama3 |
| 117–161 | Granite3 Dense 8B Instruct | 67.662.8–72.3 | 29.0 | 54.3 | 44.0 | 59.2 | 76.4 | 93.3 | 33.3 | 52.5 | 95.0 | 100.06 judge-resolved | 91.7% | 73.3%fell for 4 of 15 | 4.8 / 4.6agreement: high | 14.5ROCm 7.1 | 10.5 | 8 | Q4_K_M | apache-2.0 |
| 116–161 | Internlm2 7B Chat V2.5 | 67.462.1–72.6 | 35.8 | 65.7 | 60.0 | 48.7 | 69.6 | 60.0 | 72.2 | 45.0 | 85.0 | 100.06 judge-resolved | 83.3% | 46.7%fell for 8 of 15 | 4.9 / 4.6agreement: high | 28.3ROCm 7.1 | 22.2 | 7 | Q4_K_M | other |
| 110–174 | Phi-3.5-mini-instruct | 67.162.0–72.3 | 20.9 | 60.0 | 48.0 | 38.9 | 60.2 | 90.0 | 72.2 | 46.7 | 87.5 | 100.06 judge-resolved | 83.3% | 80.0%fell for 3 of 15 | 4.8 / 4.5agreement: high | 41.6ROCm 7.1 | 28.4 | 3.8 | Q4_K_M | mit |
| 119–162 | Granite4 1B H | 66.861.4–72.1 | 37.7 | 48.6 | 52.0 | 59.9 | 85.1 | 76.7 | 55.6 | 43.3 | 80.0 | 100.06 judge-resolved | 75.0% | 53.3%fell for 7 of 15 | 4.9 / 3.6agreement: high | 50.1ROCm 7.1 | 3.4 | 1 | Q8_0 | apache-2.0 |
| 108–178 | Qwen2 Math 7B Instructmath model | 66.060.5–71.3 | — | 71.4 | 76.0 | 39.1 | 68.7 | 66.7 | 33.3 | 77.5 | 90.0 | 70.86 judge-resolved | 75.0% | 60.0%fell for 6 of 15 | 4.6 / 4.0agreement: high | 19.4ROCm 7.1 | 4.7 | 7 | Q4_K_M | apache-2.0 |
| 120–170 | Exaone Deep 7.8B | 65.760.0–71.5 | 39.8 | 85.7 | 88.0 | 55.4 | 16.9 | 86.7 | 61.1 | 50.8 | 80.0 | 66.76 judge-resolved | 91.7% | 93.3%fell for 1 of 15 | 5.0 / 3.3agreement: medium | 28.9ROCm 7.1 | 22.3 | 7.8 | Q4_K_M | exaone-nc |
| 125–167 | Olmo2 7B 1124 Instruct | 65.460.6–70.1 | 23.5 | 51.4 | 52.0 | 29.3 | 84.4 | 93.3 | 27.8 | 60.0 | 90.0 | 100.06 judge-resolved | 83.3% | 46.7%fell for 8 of 15 | 4.9 / 4.9agreement: high | 16.9ROCm 7.1 | 21.6 | 7 | Q4_K_M | apache-2.0 |
| 113–181 | Falcon3 3B Instruct | 64.859.4–70.6 | — | 51.4 | 52.0 | 40.9 | 76.8 | 73.3 | 66.7 | 50.8 | 80.0 | 91.76 judge-resolved | 66.7% | 66.7%fell for 5 of 15 | 5.3 / 4.9agreement: high | 60.3ROCm 7.1 | 2.0 | 3 | Q4_K_M | other |
| 132–172 | Granite3.2 2B Instruct | 63.858.7–69.0 | 23.9 | 42.9 | 52.0 | 48.4 | 75.5 | 80.0 | 50.0 | 43.3 | 95.0 | 87.56 judge-resolved | 75.0% | 46.7%fell for 8 of 15 | 4.8 / 4.9agreement: high | 32.1ROCm 7.1 | 12.6 | 2 | Q4_K_M | apache-2.0 |
| 125–182 | Apertus-8B-Instruct-2509 | 63.658.2–68.8 | — | 42.9 | 32.0 | 33.0 | 76.3 | 83.3 | 50.0 | 75.0 | 80.0 | 100.06 judge-resolved | 83.3% | 46.7%fell for 8 of 15 | 4.9 / 4.3agreement: high | 32.9Vulkan | 22.3 | 8.1 | Q4_K_M | apache-2.0 |
| 129–182 | Phi-3-mini-4k-instruct | 63.458.3–68.1 | 20.9 | 77.1 | 56.0 | 35.1 | 70.7 | 96.7 | 11.1 | 52.5 | 80.0 | 91.76 judge-resolved | 66.7% | 60.0%fell for 6 of 15 | 5.5 / 5.6agreement: high | 44.5ROCm 7.1 | 8.9 | 3.8 | Q4_K_M | mit |
| 137–176 | Phi3 3.8B Mini 128K Instruct | 62.958.0–67.7 | 21.9 | 54.3 | 40.0 | 38.6 | 60.2 | 96.7 | 50.0 | 31.7 | 95.0 | 100.06 judge-resolved | 91.7% | 53.3%fell for 7 of 15 | 4.5 / 4.5agreement: high | 96.1ROCm 7.1 | 54.1 | 3.8 | Q4_K_M | mit |
| 136–187 | Codellama 34B Instructcode model | 61.256.0–66.3 | 21.9 | 42.9 | 16.0 | 38.7 | 75.1 | 73.3 | 66.7 | 50.0 | 92.5 | 95.86 judge-resolved | 41.7% | 33.3%fell for 10 of 15 | 4.5 / 3.6agreement: high | 7.4ROCm 7.1 | 46.7 | 34 | Q4_K_M | — |
| 143–184 | Smallthinker 3B Previewfine-tune of Qwen 2.5 3B | 61.155.8–66.6 | 25.5 | 57.1 | 68.0 | 29.8 | 27.8 | 80.0 | 38.9 | 76.7 | 80.0 | 91.76 judge-resolved | 91.7% | 60.0%fell for 6 of 15 | 4.3 / 2.9agreement: high | 53.2ROCm 7.1 | 7.2 | 3 | Q4_K_M | qwen-research |
| 147–183 | Gemma 7B Instruct V1.1 | 60.855.5–66.1 | 28.5 | 40.0 | 24.0 | 26.5 | 76.0 | 83.3 | 61.1 | 56.7 | 80.0 | 100.06 judge-resolved | 91.7% | 60.0%fell for 6 of 15 | 5.1 / 4.6agreement: high | 15.2ROCm 7.1 | 36.1 | 7 | Q4_K_M | gemma |
| 147–184 | Granite3.1 Dense 2B Instruct | 60.655.0–66.1 | 27.0 | 45.7 | 36.0 | 46.6 | 73.6 | 76.7 | 50.0 | 45.0 | 80.0 | 91.76 judge-resolved | 58.3% | 40.0%fell for 9 of 15 | 4.5 / 4.8agreement: high | 56.8ROCm 7.1 | 12.6 | 2 | Q4_K_M | apache-2.0 |
| 147–185 | Qwen2.5 Coder 1.5B Instructcode model | 60.554.9–66.0 | 28.6 | 65.7 | 44.0 | 52.8 | 58.8 | 60.0 | 66.7 | 30.8 | 82.5 | 83.36 judge-resolved | 66.7% | 53.3%fell for 7 of 15 | 4.5 / 3.3agreement: medium | 77.0ROCm 7.1 | 5.1 | 1.5 | Q4_K_M | apache-2.0 |
| 148–186 | Mistral 7B Instruct V0.3 | 59.854.7–64.7 | 24.9 | 37.1 | 16.0 | 40.9 | 70.3 | 93.3 | 61.1 | 41.7 | 90.0 | 87.56 judge-resolved | 66.7% | 53.3%fell for 7 of 15 | 5.4 / 5.1agreement: high | 17.5ROCm 7.1 | 22.0 | 7 | Q4_K_M | apache-2.0 |
| 148–186 | Granite3.1 Moe 3B Instruct | 59.854.7–65.0 | 24.9 | 37.1 | 32.0 | 32.6 | 69.2 | 83.3 | 61.1 | 41.7 | 85.0 | 95.86 judge-resolved | 58.3% | 66.7%fell for 5 of 15 | 4.3 / 4.6agreement: high | 93.0ROCm 7.1 | 10.9 | 3 | Q4_K_M | apache-2.0 |
| 150–186 | Codegemma 7B Instruct V1.1code model | 59.554.8–64.2 | 29.3 | 45.7 | 24.0 | 55.5 | 78.1 | 70.0 | 5.6 | 70.0 | 95.0 | 91.76 judge-resolved | 75.0% | 46.7%fell for 8 of 15 | 5.0 / 5.0agreement: high | 52.6ROCm 7.1 | 36.1 | 7 | Q4_K_M | gemma |
| 143–191 | Mistral-7B-Instruct-v0.2 | 59.254.3–64.2 | 18.7 | 40.0 | 12.0 | 25.3 | 75.1 | 90.0 | 72.2 | 42.5 | 80.0 | 95.86 judge-resolved | 91.7% | 60.0%fell for 6 of 15 | 4.8 / 4.9agreement: high | 29.0ROCm 7.1 | 22.0 | 7.2 | Q4_K_M | apache-2.0 |
| 150–186 | YI Coder 9B Chatcode model | 59.053.7–64.5 | 37.8 | 31.4 | 16.0 | 81.3 | 65.3 | 83.3 | 44.4 | 54.2 | 95.0 | 60.46 judge-resolved | 75.0% | 53.3%fell for 7 of 15 | 3.8 / 3.9agreement: high | 24.6ROCm 7.1 | 18.5 | 9 | Q4_K_M | apache-2.0 |
| 151–186 | Solar 10.7B Instruct V1 | 58.753.7–63.8 | 26.4 | 45.7 | 16.0 | 34.3 | 69.9 | 86.7 | 38.9 | 58.3 | 95.0 | 83.36 judge-resolved | 83.3% | 33.3%fell for 10 of 15 | 5.8 / 5.1agreement: high | 13.6ROCm 7.1 | 13.1 | 10.7 | Q4_K_M | — |
| 152–187 | Granite3 Dense 2B Instruct | 58.353.1–63.6 | 20.2 | 48.6 | 24.0 | 42.2 | 63.4 | 86.7 | 33.3 | 48.3 | 82.5 | 95.86 judge-resolved | 75.0% | 40.0%fell for 9 of 15 | 4.4 / 4.4agreement: high | 43.5ROCm 7.1 | 4.4 | 2 | Q4_K_M | apache-2.0 |
| 152–188 | Deepseek V2 16B Lite Chat | 58.253.2–63.4 | 20.1 | 34.3 | 40.0 | 41.6 | 70.2 | 70.0 | 33.3 | 50.8 | 87.5 | 95.86 judge-resolved | 58.3% | 40.0%fell for 9 of 15 | 5.3 / 4.9agreement: high | 5.9ROCm 7.1 | 46.8 | 16 | Q4_K_M | other |
| 152–187 | YI 6B Chat V1.5 | 58.052.8–63.5 | 21.0 | 45.7 | 48.0 | 19.8 | 70.4 | 73.3 | 33.3 | 51.7 | 92.5 | 87.56 judge-resolved | 75.0% | 60.0%fell for 6 of 15 | 4.9 / 4.9agreement: medium | 120.2ROCm 7.1 | 5.9 | 6 | Q4_K_M | apache-2.0 |
| 150–187 | Llama3.2 1B Instruct | 58.052.3–63.7 | 26.6 | 37.1 | 40.0 | 41.7 | 76.9 | 76.7 | 38.9 | 55.0 | 72.5 | 83.36 judge-resolved | 50.0% | 13.3%fell for 13 of 15 | 5.1 / 4.0agreement: high | 114.5ROCm 7.1 | 5.4 | 1 | Q4_K_M | llama3.2 |
| 152–188 | Sailor2 8B Chatfine-tune of Qwen 2.5 | 57.351.7–62.8 | 21.1 | 54.3 | 52.0 | 19.5 | 52.2 | 80.0 | 55.6 | 81.7 | 35.0 | 85.45 judge-resolved | 75.0% | 66.7%fell for 5 of 15 | 5.0 / 4.1agreement: high | 26.3ROCm 7.1 | 14.2 | 8 | Q4_K_M | — |
| 154–188 | Gemma3 1B IT | 56.851.8–62.2 | 29.3 | 60.0 | 64.0 | 42.5 | 71.9 | 63.3 | 22.2 | 52.5 | 35.0 | 100.06 judge-resolved | 58.3% | 53.3%fell for 7 of 15 | 5.4 / 4.9agreement: high | 80.1ROCm 7.1 | 1.5 | 1 | Q4_K_M | gemma |
| 150–194 | Mixtral 8x7B Instruct V0.1 | 56.851.3–62.0 | 17.5 | 42.9 | 24.0 | 20.1 | 62.6 | 83.3 | 72.2 | 41.7 | 85.0 | 79.26 judge-resolved | 83.3% | 73.3%fell for 4 of 15 | 4.6 / 5.1agreement: high | 8.5ROCm 7.1 | 46.5 | 56 | Q4_K_M | — |
| 155–189 | Aya 8B 23 | 55.951.1–60.7 | 24.9 | 34.3 | 0 | 30.7 | 74.7 | 76.7 | 61.1 | 39.2 | 95.0 | 91.76 judge-resolved | 66.7% | 40.0%fell for 9 of 15 | 4.6 / 4.8agreement: high | 16.4ROCm 7.1 | 14.2 | 8 | F16 | cc-by-nc-4.0 |
| 155–192 | Deepcoder 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bcode model | 55.850.3–61.0 | 35.2 | 80.0 | 88.0 | 34.2 | 68.6 | 56.7 | 16.7 | 65.8 | 67.5 | 25.06 judge-resolved | 66.7% | 80.0%fell for 3 of 15 | 3.9 / 2.8agreement: high | 66.9ROCm 7.1 | 5.1 | 1.5 | Q4_K_M | mit |
| 154–195 | Qwen 14B Chat V1.5 | 54.849.8–59.4 | — | 48.6 | 40.0 | 9.7 | 76.9 | 93.3 | 11.1 | 52.5 | 65.0 | 95.86 judge-resolved | 66.7% | 60.0%fell for 6 of 15 | 4.8 / 4.4agreement: high | 13.9ROCm 7.1 | 62.6 | 14 | Q4_K_M | other |
| 156–196 | Exaone Deep 2.4B | 54.748.9–60.5 | 23.0 | 82.9 | 72.0 | 36.7 | 13.9 | 63.3 | 50.0 | 38.3 | 72.5 | 62.56 judge-resolved | 75.0% | 93.3%fell for 1 of 15 | 3.4 / 2.3agreement: high | 65.1ROCm 7.1 | 11.9 | 2.4 | Q4_K_M | exaone-nc |
| 158–194 | Deepseek Llm 7B Chat | 54.349.3–59.8 | 20.9 | 37.1 | 24.0 | 37.4 | 66.8 | 76.7 | 27.8 | 45.8 | 77.5 | 95.86 judge-resolved | 25.0% | 40.0%fell for 9 of 15 | 4.4 / 3.6agreement: high | 24.8ROCm 7.1 | 20.3 | 7 | Q4_K_M | — |
| 161–198 | Qwen3 VL 2B Thinking | 53.748.3–59.1 | 26.7 | 60.0 | 52.0 | 0 | 77.2 | 73.3 | 33.3 | 63.3 | 82.5 | 41.74 judge-resolved | 50.0% | 53.3%fell for 7 of 15 | 3.4 / 3.1agreement: high | 26.4ROCm 7.1 | 3.6 | 2 | Q4_K_M | apache-2.0 |
| 159–202 | Codellama 7B Instructcode model | 53.047.8–57.8 | — | 37.1 | 4.0 | 36.0 | 68.1 | 76.7 | 61.1 | 21.7 | 85.0 | 87.56 judge-resolved | 50.0% | 40.0%fell for 9 of 15 | 4.6 / 4.4agreement: high | 27.2ROCm 7.1 | 38.8 | 7 | Q4_K_M | llama2 |
| 159–203 | Codellama 13B Instructcode model | 52.547.2–57.8 | — | 31.4 | 12.0 | 22.5 | 73.8 | 83.3 | 50.0 | 33.3 | 82.5 | 83.36 judge-resolved | 50.0% | 66.7%fell for 5 of 15 | 4.6 / 3.8agreement: high | 16.2ROCm 7.1 | 61.7 | 13 | Q4_K_M | llama2 |
| 168–198 | Deepscaler 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bmath model | 52.146.6–57.7 | 30.4 | 68.6 | 92.0 | 26.7 | 66.9 | 50.0 | 22.2 | 56.7 | 65.0 | 20.84 judge-resolved | 75.0% | 66.7%fell for 5 of 15 | 3.3 / 2.4agreement: high | 40.0ROCm 7.1 | 5.1 | 1.5 | Q4_K_M | mit |
| 171–198 | Nemotron Mini 4B Instruct | 52.047.4–56.9 | 22.4 | 34.3 | 12.0 | 34.6 | 72.4 | 83.3 | 11.1 | 47.5 | 85.0 | 87.56 judge-resolved | 50.0% | 46.7%fell for 8 of 15 | 4.4 / 3.9agreement: high | 167.6ROCm 7.1 | 6.6 | 4 | Q4_K_M | other |
| 159–207 | Internlm2 1.8B Chat V2.5 | 51.946.1–57.5 | — | 31.4 | 28.0 | 48.5 | 56.6 | 70.0 | 50.0 | 29.2 | 70.0 | 83.36 judge-resolved | 41.7% | 33.3%fell for 10 of 15 | 4.1 / 2.9agreement: medium | 85.6ROCm 7.1 | 1.2 | 1.8 | Q4_K_M | other |
| 162–207 | Magistral Small (2506) | 49.844.5–55.3 | — | 34.3 | 48.0 | 10.0 | 34.8 | 56.7 | 44.4 | 70.0 | 100.0 | 50.03 judge-resolved | 33.3% | 26.7%fell for 11 of 15 | 4.1 / 3.6agreement: high | 10.5ROCm 7.1 | 14.3 | 24 | Q4_K_M | apache-2.0 |
| 175–203 | Qwen2 1.5B Instruct | 49.744.1–55.3 | 20.1 | 28.6 | 32.0 | 30.6 | 61.6 | 63.3 | 27.8 | 48.3 | 80.0 | 75.06 judge-resolved | 66.7% | 60.0%fell for 6 of 15 | 4.1 / 3.6agreement: high | 49.8ROCm 7.1 | — | 1.5 | Q4_K_M | apache-2.0 |
| 176–204 | Codeqwen 7B Chat V1.5code model | 49.444.2–54.7 | 34.2 | 25.7 | 8.0 | 69.4 | 72.1 | 40.0 | 38.9 | 35.0 | 95.0 | 60.46 judge-resolved | 66.7% | 40.0%fell for 9 of 15 | 4.0 / 2.8agreement: high | 25.2ROCm 7.1 | 13.6 | 7 | Q4_K_M | — |
| 177–205 | Deepseek R1 1.5B Qwen Distill | 48.943.5–54.7 | 25.5 | 71.4 | 76.0 | 24.4 | 66.2 | 53.3 | 22.2 | 47.5 | 62.5 | 16.73 judge-resolved | 75.0% | 73.3%fell for 4 of 15 | 3.9 / 3.0agreement: high | 50.6ROCm 7.1 | 5.1 | 1.5 | Q4_K_M | mit |
| 177–206 | Hermes3 3B Llama3.2fine-tune of Llama 3.2 | 48.542.5–54.3 | 14.9 | 37.1 | 28.0 | 11.3 | 76.4 | 63.3 | 50.0 | 43.3 | 60.0 | 66.76 judge-resolved | 83.3% | 33.3%fell for 10 of 15 | 4.5 / 4.0agreement: high | 50.7ROCm 7.1 | 17.4 | 3 | Q4_K_M | llama3 |
| 179–206 | Granite3 Moe 3B Instruct | 47.742.3–53.1 | 20.2 | 37.1 | 20.0 | 31.6 | 67.2 | 76.7 | 27.8 | 36.7 | 45.0 | 87.56 judge-resolved | 50.0% | 53.3%fell for 7 of 15 | 4.3 / 3.5agreement: high | 218.4ROCm 7.1 | 4.3 | 3 | Q4_K_M | apache-2.0 |
| 180–206 | Smollm2 1.7B Instruct | 47.141.7–52.7 | 26.0 | 28.6 | 8.0 | 35.3 | 79.5 | 63.3 | 33.3 | 35.0 | 70.0 | 70.86 judge-resolved | 58.3% | 40.0%fell for 9 of 15 | 4.8 / 3.6agreement: high | 53.3ROCm 7.1 | 14.2 | 1.7 | Q4_K_M | apache-2.0 |
| 184–208 | Granite4 350M H | 45.640.7–50.7 | 25.2 | 20.0 | 0 | 36.0 | 79.3 | 76.7 | 16.7 | 43.3 | 80.0 | 58.36 judge-resolved | 41.7% | 20.0%fell for 12 of 15 | 3.6 / 2.4agreement: medium | 128.2ROCm 7.1 | 1.2 | 0.4 | Q8_0 | apache-2.0 |
| 181–211 | Llama2 13B Chat | 45.340.2–50.6 | — | 45.7 | 8.0 | 10.1 | 47.8 | 73.3 | 27.8 | 32.5 | 75.0 | 87.56 judge-resolved | 66.7% | 53.3%fell for 7 of 15 | 5.6 / 4.8agreement: high | 16.4ROCm 7.1 | 21.5 | 13 | Q4_K_M | llama2 |
| 180–212 | Falcon3 1B Instruct | 44.339.8–49.3 | — | 40.0 | 20.0 | 18.7 | 64.9 | 70.0 | 11.1 | 34.2 | 40.0 | 100.06 judge-resolved | 66.7% | 33.3%fell for 10 of 15 | 3.8 / 2.8agreement: high | 107.4ROCm 7.1 | 1.1 | 1 | Q4_K_M | other |
| 183–211 | Qwen 7B Chat V1.5 | 44.139.1–48.9 | — | 45.7 | 16.0 | 5.9 | 71.9 | 76.7 | 11.1 | 45.0 | 35.0 | 89.66 judge-resolved | 75.0% | 66.7%fell for 5 of 15 | 4.3 / 3.9agreement: high | 23.2ROCm 7.1 | 39.2 | 7 | Q4_K_M | other |
| 185–209 | Dolphin Phi 2.7B V2.6fine-tune of Phi-2 | 44.038.9–49.4 | 16.4 | 31.4 | 40.0 | 28.6 | 69.8 | 70.0 | 5.6 | 15.0 | 65.0 | 70.86 judge-resolved | 58.3% | 53.3%fell for 7 of 15 | 4.3 / 2.9agreement: medium | 38.4ROCm 7.1 | 7.2 | 2.7 | Q4_K_M | other |
| 185–210 | Qwen3 0.6B | 44.038.8–49.4 | 25.0 | 37.1 | 40.0 | 32.5 | 74.3 | 20.0 | 11.1 | 48.3 | 82.5 | 50.06 judge-resolved | 41.7% | 33.3%fell for 10 of 15 | 3.8 / 2.9agreement: high | 261.1ROCm 7.1 | 15.6 | 0.6 | Q4_K_M | apache-2.0 |
| 182–212 | Llama-2-7b-chat-hf | 43.839.0–48.8 | 16.4 | 25.7 | 4.0 | 18.6 | 72.1 | 66.7 | 16.7 | 33.3 | 65.0 | 91.76 judge-resolved | 66.7% | 33.3%fell for 10 of 15 | 4.5 / 4.0agreement: high | 26.7ROCm 7.1 | 12.8 | 6.7 | Q4_K_M | llama2 |
| 185–210 | Stablelm Zephyr 3Bfine-tune of StableLM 3B | 43.738.4–49.3 | 19.9 | 45.7 | 16.0 | 29.3 | 60.1 | 50.0 | 22.2 | 34.2 | 65.0 | 70.86 judge-resolved | 50.0% | 20.0%fell for 12 of 15 | 4.4 / 3.9agreement: high | 39.6ROCm 7.1 | 12.5 | 2.8 | Q4_K_M | stability-community |
| 185–212 | Granite3.1 Moe 1B Instruct | 43.637.9–49.1 | 19.0 | 31.4 | 12.0 | 38.1 | 59.0 | 63.3 | 33.3 | 24.2 | 62.5 | 68.86 judge-resolved | 58.3% | 40.0%fell for 9 of 15 | 4.3 / 3.5agreement: high | 149.5ROCm 7.1 | 7.5 | 1 | Q4_K_M | apache-2.0 |
| 186–212 | Qwen2.5 0.5B Instruct | 43.337.4–49.4 | 18.2 | 28.6 | 36.0 | 26.5 | 54.9 | 56.7 | 38.9 | 23.3 | 62.5 | 62.56 judge-resolved | 58.3% | 40.0%fell for 9 of 15 | 3.5 / 2.5agreement: high | 116.8ROCm 7.1 | 2.2 | 0.5 | Q4_K_M | apache-2.0 |
| 190–213 | Stablelm2 12B Chat | 42.438.1–46.7 | 14.7 | 22.9 | 8.0 | 45.0 | 47.0 | 70.0 | 5.6 | 20.0 | 67.5 | 95.86 judge-resolved | 50.0% | 33.3%fell for 10 of 15 | 4.4 / 3.9agreement: high | 6.7ROCm 7.1 | 14.0 | 12 | Q4_K_M | stability-nc |
| 188–213 | Gemma 2B Instruct V1.1 | 41.936.8–47.5 | 22.5 | 25.7 | 20.0 | 25.2 | 66.6 | 60.0 | 27.8 | 37.5 | 35.0 | 79.26 judge-resolved | 50.0% | 26.7%fell for 11 of 15 | 4.0 / 4.0agreement: high | 43.4ROCm 7.1 | 3.3 | 2 | Q4_K_M | gemma |
| 193–215 | Granite Code 8B Instructcode model | 40.735.3–46.7 | 20.6 | 40.0 | 28.0 | 55.7 | 53.7 | 63.3 | 27.8 | 41.7 | 10.0 | 45.86 judge-resolved | 58.3% | 66.7%fell for 5 of 15 | 4.6 / 3.4agreement: high | 15.0ROCm 7.1 | 10.0 | 8 | Q4_K_M | apache-2.0 |
| 195–214 | Phi 2.7B Chat V2 | 39.935.5–44.2 | 17.5 | 34.3 | 0 | 34.7 | 60.3 | 83.3 | 0 | 18.3 | 42.5 | 85.46 judge-resolved | 33.3% | 40.0%fell for 9 of 15 | 3.6 / 2.6agreement: high | 38.2ROCm 7.1 | 7.2 | 2.7 | Q4_K_M | mit |
| 197–216 | Granite Code 3B Instructcode model | 38.032.7–43.5 | 25.6 | 37.1 | 24.0 | 45.6 | 54.2 | 56.7 | 16.7 | 42.5 | 15.0 | 50.06 judge-resolved | 58.3% | 40.0%fell for 9 of 15 | 3.9 / 2.4agreement: medium | 107.3ROCm 7.1 | 7.7 | 3 | Q4_K_M | apache-2.0 |
| 201–216 | Qwen2 0.5B Instruct | 36.030.9–41.2 | 15.6 | 31.4 | 8.0 | 9.8 | 50.2 | 43.3 | 22.2 | 23.3 | 77.5 | 58.36 judge-resolved | 41.7% | 26.7%fell for 11 of 15 | 3.9 / 2.6agreement: high | 137.5ROCm 7.1 | 2.2 | 0.5 | Q4_K_M | apache-2.0 |
| 203–217 | Granite3 Moe 1B Instruct | 35.531.0–40.5 | 18.7 | 17.1 | 8.0 | 27.6 | 59.1 | 36.7 | 11.1 | 20.8 | 60.0 | 79.26 judge-resolved | 58.3% | 33.3%fell for 10 of 15 | 3.6 / 3.3agreement: high | 82.4ROCm 7.1 | 2.3 | 1 | Q4_K_M | apache-2.0 |
| 198–219 | Deepseek Coder 6.7B Instructcode model | 34.729.7–40.0 | — | 8.6 | 0 | 68.5 | 53.9 | 10.0 | 61.1 | 21.7 | 55.0 | 33.36 judge-resolved | 50.0% | 33.3%fell for 10 of 15 | 1.3 / 1.0agreement: high | 22.5ROCm 7.1 | 38.8 | 6.7 | Q4_K_M | other |
| 203–221 | Deepseek Coder 33B Instructcode model | 33.429.0–38.2 | 31.0 | 31.4 | 12.0 | 67.2 | 46.2 | 6.7 | 0 | 25.8 | 70.0 | 41.76 judge-resolved | 58.3% | 53.3%fell for 7 of 15 | 1.1 / 1.4agreement: high | 5.9ROCm 7.1 | 53.5 | 33 | Q4_K_M | — |
| 203–222 | Qwen2 Math 1.5B Instructmath model | 33.128.0–38.2 | — | 65.7 | 72.0 | 17.5 | 14.4 | 33.3 | 11.1 | 37.5 | 17.5 | 29.26 judge-resolved | 83.3% | 40.0%fell for 9 of 15 | 3.0 / 1.1agreement: medium | 62.0ROCm 7.1 | 1.0 | 1.5 | Q4_K_M | apache-2.0 |
| 208–218 | Qwen2.5 Coder 0.5B Instructcode model | 32.827.5–38.2 | 22.0 | 22.9 | 8.0 | 35.9 | 49.5 | 20.0 | 33.3 | 31.7 | 65.0 | 29.26 judge-resolved | 50.0% | 33.3%fell for 10 of 15 | 3.1 / 1.9agreement: high | 172.5ROCm 7.1 | 2.2 | 0.5 | Q4_K_M | apache-2.0 |
| 209–219 | Opencoder 1.5B Instructcode model | 31.927.6–36.4 | 21.5 | 2.9 | 0 | 57.3 | 50.8 | 20.0 | 11.1 | 24.2 | 77.5 | 43.86 judge-resolved | 58.3% | 26.7%fell for 11 of 15 | 3.3 / 1.3agreement: medium | 16.2ROCm 7.1 | 8.4 | 1.5 | Q4_K_M | other |
| 211–219 | Qwen 4B Chat V1.5 | 30.226.0–34.5 | 15.1 | 11.4 | 4.0 | 3.1 | 60.4 | 73.3 | 0 | 37.5 | 15.0 | 66.76 judge-resolved | 50.0% | 20.0%fell for 12 of 15 | 3.6 / 2.0agreement: medium | 72.2ROCm 7.1 | 56.1 | 4 | Q4_K_M | tongyi-qianwen-research |
| 212–223 | Smollm2 360M Instruct | 28.123.4–32.9 | 14.1 | 8.6 | 8.0 | 23.8 | 59.9 | 46.7 | 11.1 | 21.7 | 25.0 | 47.96 judge-resolved | 41.7% | 46.7%fell for 8 of 15 | 4.5 / 2.9agreement: medium | 91.2ROCm 7.1 | 3.1 | 0.4 | Q4_K_M | apache-2.0 |
| 213–224 | YI Coder 1.5B Chatcode model | 26.522.1–31.5 | 21.7 | 17.1 | 0 | 57.7 | 36.1 | 53.3 | 5.6 | 10.8 | 35.0 | 22.96 judge-resolved | 41.7% | 33.3%fell for 10 of 15 | 3.3 / 1.6agreement: medium | 88.7ROCm 7.1 | 26.9 | 1.5 | Q4_K_M | apache-2.0 |
| 216–225 | Qwen 1.8B Chat V1.5 | 24.720.2–29.3 | 12.6 | 20.0 | 4.0 | 2.4 | 38.4 | 60.0 | 0 | 27.5 | 20.0 | 50.06 judge-resolved | 66.7% | 40.0%fell for 9 of 15 | 3.3 / 2.1agreement: high | 82.5ROCm 7.1 | 27.1 | 1.8 | Q4_K_M | tongyi-qianwen-research |
| 216–225 | Stablelm2 1.6B Chat | 24.420.2–29.0 | 10.6 | 8.6 | 4.0 | 26.1 | 31.6 | 40.0 | 0 | 16.7 | 42.5 | 50.06 judge-resolved | 41.7% | 13.3%fell for 13 of 15 | 3.6 / 2.6agreement: high | 62.2ROCm 7.1 | 7.5 | 1.6 | Q4_K_M | stability-nc |
| 217–225 | Sailor2 1B Chatfine-tune of Qwen 2.5 | 23.519.0–28.1 | 6.4 | 8.6 | 0 | 10.5 | 29.6 | 60.0 | 16.7 | 25.0 | 27.5 | 33.36 judge-resolved | 66.7% | 6.7%fell for 14 of 15 | 3.6 / 2.6agreement: high | 116.3ROCm 7.1 | 4.1 | 1 | Q4_K_M | — |
| 219–225 | Stable Code 3B Instructcode model | 22.818.4–27.4 | 17.4 | 22.9 | 0 | 6.5 | 55.8 | 33.3 | 5.6 | 26.7 | 15.0 | 39.66 judge-resolved | 41.7% | 40.0%fell for 9 of 15 | 2.6 / 1.1agreement: medium | 38.6ROCm 7.1 | 44.9 | 2.8 | Q4_K_M | stability-community |
| 218–225 | Deepseek Coder 1.3B Instructcode model | 22.618.3–27.2 | 21.9 | 5.7 | 0 | 54.5 | 40.5 | 3.3 | 11.1 | 23.3 | 40.0 | 25.06 judge-resolved | 33.3% | 6.7%fell for 14 of 15 | 0.9 / 0.6agreement: high | 64.1ROCm 7.1 | 26.9 | 1.3 | Q4_K_M | other |
| 220–227 | Smollm2 135M Instruct | 20.116.2–24.6 | 11.2 | 2.9 | 0 | 14.1 | 36.1 | 36.7 | 5.6 | 15.0 | 25.0 | 45.86 judge-resolved | 50.0% | 6.7%fell for 14 of 15 | 3.0 / 1.5agreement: medium | 201.2ROCm 7.1 | 1.7 | 0.1 | Q4_K_M | apache-2.0 |
| 225–227 | Qwen 0.5B Chat V1.5 | 16.112.5–20.3 | 15.2 | 14.3 | 0 | 9.0 | 40.8 | 13.3 | 0 | 30.0 | 15.0 | 22.96 judge-resolved | 33.3% | 26.7%fell for 11 of 15 | 3.1 / 1.4agreement: medium | 98.9ROCm 7.1 | 13.4 | 0.5 | Q4_K_M | tongyi-qianwen-research |
| 225–227 | Codegemma 2B Code V1.1code model | 15.311.7–19.1 | 9.2 | 17.1 | 0 | 3.9 | 29.5 | 16.7 | 0 | 16.7 | 10.0 | 43.86 judge-resolved | 33.3% | 6.7%fell for 14 of 15 | 0.9 / 0.3agreement: high | 43.3ROCm 7.1 | 3.3 | 2 | Q4_K_M | gemma |
| — | Apertus-8B-Instruct-2509 | —withheld | — | — | — | — | — | — | — | — | — | — | — | — | 8.7ROCm 7.1 | 22.3 | 8.1 | Q4_K_M | apache-2.0 | |
| — | Falcon2 11B | —unranked: long_context partial: 12/18 items scored, 80% must be scored | 22.9 | 16.0 | 16.9 | 57.7 | 70.0 | 0partial: 12/18 | 22.5 | 20.0 | 87.56 judge-resolved | 33.3% | 33.3%fell for 10 of 15 | 4.0 / 2.4agreement: medium | 19.7ROCm 7.1 | 10.9 | 11 | Q4_K_M | other | |
| — | Falcon3 10B Instruct | —withheld | — | — | — | — | — | — | — | — | — | — | — | — | 21.2ROCm 7.1 | 28.3 | 10 | Q4_K_M | other | |
| — | Falcon3 1B Instruct | —withheld | — | — | — | — | — | — | — | — | — | — | — | — | 87.3ROCm 7.1 | 6.0 | 1 | Q4_K_M | other | |
| — | Falcon3 3B Instruct | —withheld | — | — | — | — | — | — | — | — | — | — | — | — | 57.9ROCm 7.1 | 14.0 | 3 | Q4_K_M | other | |
| — | Falcon3 7B Instruct | —withheld | — | — | — | — | — | — | — | — | — | — | — | — | 24.7ROCm 7.1 | 20.1 | 7 | Q4_K_M | other | |
| — | Internlm2 1.8B Chat V2.5 | —withheld | — | — | — | — | — | — | — | — | — | — | — | — | 73.2ROCm 7.1 | 14.3 | 1.8 | Q4_K_M | other | |
| — | Qwen2 Math 1.5B Instructmath model | —withheld | — | — | — | — | — | — | — | — | — | — | — | — | 61.1ROCm 7.1 | 2.1 | 1.5 | Q4_K_M | apache-2.0 | |
| — | Qwen2 Math 7B Instructmath model | —withheld | — | — | — | — | — | — | — | — | — | — | — | — | 17.4ROCm 7.1 | 6.5 | 7 | Q4_K_M | apache-2.0 |
Withheld
These models have no rank and no score on this page. Their answers are published on their own pages, except for a model whose every tested version is withheld: it has no published results, and its page gives the reason only.
- Apertus-8B-Instruct-2509: Withheld: the model stopped after one token on 212 of 270 prompts in our Ollama import, which points to a broken chat template rather than the model. It was re-run with a chat template written from the official one, on the Vulkan backend; those results replace these.
- Falcon3 10B Instruct: Withheld: Ollama's library chat template for Falcon3 leaves out the newline after the last message, so the model sometimes wrote <|assistant|> itself and its reply came back empty (16 of 270 answers here; a serving bug, not the model). It was re-run with the corrected template; those results replace these.
- Falcon3 1B Instruct: Withheld: Ollama's library chat template for Falcon3 leaves out the newline after the last message, so the model sometimes wrote <|assistant|> itself and its reply came back empty (42 of 270 answers here; a serving bug, not the model). It was re-run with the corrected template; those results replace these.
- Falcon3 3B Instruct: Withheld: Ollama's library chat template for Falcon3 leaves out the newline after the last message, so the model sometimes wrote <|assistant|> itself and its reply came back empty (169 of 270 answers here; a serving bug, not the model). It was re-run with the corrected template; those results replace these.
- Falcon3 7B Instruct: Withheld: Ollama's library chat template for Falcon3 leaves out the newline after the last message, so the model sometimes wrote <|assistant|> itself and its reply came back empty (6 of 270 answers here; a serving bug, not the model). It was re-run with the corrected template; those results replace these.
- Internlm2 1.8B Chat V2.5: Withheld: Ollama's library chat template for InternLM2 ends the prompt without the newline after <|im_start|>assistant, so on 31 of 270 prompts the model ended its turn after one token and its reply came back empty (a serving bug, not the model). It was re-run with the corrected template; those results replace these.
- Phi-4-reasoning 14B: Withheld: in our Ollama build, Phi-4-reasoning's reasoning came back mixed into the answer instead of as separate thinking, so the thinking budget never applied and 259 of 270 answers were cut off before a final answer (a serving problem, not the model). It will be re-run once the serving setup is fixed.
- Phi-4-reasoning-plus 14B: Withheld: in our Ollama build, Phi-4-reasoning-plus's reasoning came back mixed into the answer instead of as separate thinking, so the thinking budget never applied and its answers were cut off before a final answer (a serving problem, not the model). The run was stopped after 50 of 270 answers; it will be re-run once the serving setup is fixed.
- Qwen2 Math 1.5B Instruct: Withheld: Ollama's library chat template for Qwen2-Math differs from the official one (no newline between turns or after the final <|im_start|>assistant, and no default system turn). It was re-run with the official template; those results replace these (the scores changed very little).
- Qwen2 Math 7B Instruct: Withheld: Ollama's library chat template for Qwen2-Math differs from the official one (no newline between turns or after the final <|im_start|>assistant, and no default system turn). It was re-run with the official template; those results replace these (the scores changed very little).