Open-weight model leaderboard

Which open-weight model is best for the machine you have? Each one is run locally through the same fixed battery, then scored by code wherever an answer can be checked.

228 models (236 versions listed, withheld ones included) on a battery of 270 items in 13 categories: 9 of them make up Overall (reasoning, math, coding, instruction following, knowledge & honesty, long context, multilingual, tool use, safety); the other 4 (pushback resistance, famous gotchas, writing, self-description) are shown on their own. Each model also gets a speed and memory profile on one laptop (Speed, hardware picker, best-model pages), from 12 runs (newest r20261105-bv1-2abe78). Read the methodology, including its limitations.

How much model does your memory buy?

Each dot is a model. The line is the best score reachable at each memory size. Memory is an estimate: the model file size plus about 20% for the context of a single conversation.

Overall score against the memory a model needs One dot per model. Horizontal axis: estimated memory need in GB, logarithmic scale. Vertical axis: Overall score (0 to 100). The orange line joins the 15 models that no model needing the same or less memory beats; only those are named, and every dot has a tooltip and a link to its model page. 0 20 40 60 80 100 0.0625 0.125 0.25 0.5 1 2 4 8 16 32 64 128 Estimated memory needed (GB, log scale) Overall (0 to 100) gemma-4-31B-it: Overall 99.3, needs about 22.4 GB (on the frontier)gemma-4-31B-it Gemma 4 12B IT: Overall 98.6, needs about 8.9 GB (on the frontier) Gemma4 E4B IT: Overall 95.5, needs about 7.9 GB (on the frontier) Qwen3-VL-8B-Instruct: Overall 94.9, needs about 7.4 GB (on the frontier)Qwen3-VL-8B-Instruct Gemma4 E2B IT: Overall 93.8, needs about 5.5 GB (on the frontier) Qwen3-4B-Instruct-2507: Overall 90.3, needs about 3.0 GB (on the frontier)Qwen3-4B-Instruct-2507 Phi-4-mini-instruct: Overall 81.6, needs about 3.0 GB (on the frontier)Phi-4-mini-instruct granite-4.1-3b: Overall 80.7, needs about 2.5 GB (on the frontier) Qwen3 VL 2B Instruct: Overall 77.4, needs about 2.3 GB (on the frontier)Qwen3 VL 2B Instruct Exaone3.5 2.4B Instruct: Overall 76.3, needs about 1.9 GB (on the frontier) Qwen3 1.7B: Overall 76.0, needs about 1.7 GB (on the frontier) Lfm2.5 Thinking 1.2B: Overall 74.3, needs about 0.9 GB (on the frontier) Granite4 350M H: Overall 45.6, needs about 0.4 GB (on the frontier)Granite4 350M H Smollm2 360M Instruct: Overall 28.1, needs about 0.3 GB (on the frontier)Smollm2 360M Instruct Smollm2 135M Instruct: Overall 20.1, needs about 0.1 GB (on the frontier)Smollm2 135M Instruct gpt-oss-120b: Overall 98.6, needs about 78.4 GB Muse Glimmer 30B: Overall 97.9, needs about 21.8 GB gemma-4-26B-A4B-it: Overall 97.8, needs about 21.6 GB Qwen3-Coder-Next: Overall 96.4, needs about 62.1 GB Qwen3.5-27B: Overall 96.2, needs about 19.9 GB gemma-3-27b-it: Overall 95.8, needs about 20.9 GB Qwen3-Coder-30B-A3B-Instruct: Overall 95.5, needs about 22.3 GB Qwen3.8-27B: Overall 95.4, needs about 20.2 GB Qwen3.6-27B: Overall 95.1, needs about 20.9 GB Qwen3-30B-A3B-Instruct-2507: Overall 95.0, needs about 22.3 GB gpt-oss-20b: Overall 95.0, needs about 16.6 GB Qwen3.5-35B-A3B: Overall 94.8, needs about 25.4 GB Qwen3-VL-32B-Instruct: Overall 94.7, needs about 25.1 GB Qwen3.6-35B-A3B: Overall 94.6, needs about 28.7 GB Sage-A1.5-Q4_K_M: Overall 94.5, needs about 26.4 GB Llama-3.3-70B-Instruct: Overall 94.3, needs about 51.0 GB Granite4 Small H: Overall 93.9, needs about 23.4 GB Qwen3-32B: Overall 93.6, needs about 24.2 GB Llama 4 Scout 17B-16E Instruct: Overall 93.6, needs about 80.9 GB Qwen2.5-32B-Instruct: Overall 93.5, needs about 23.8 GB Qwen3-VL-30B-A3B-Instruct: Overall 92.9, needs about 23.5 GB Mistral Small 3.2 24B Instruct (2506): Overall 92.9, needs about 18.2 GB Nemotron 3.5 Lightning 30B A3B: Overall 92.8, needs about 30.5 GB Granite4.2 30B: Overall 92.8, needs about 21.3 GB GLM-4.5-Air: Overall 92.8, needs about 87.6 GB Qwen3-30B-A3B: Overall 92.8, needs about 22.3 GB gemma-3-12b-it: Overall 92.7, needs about 9.8 GB Ministral 3 14B Instruct (2512): Overall 92.7, needs about 10.9 GB QwQ 32B: Overall 92.6, needs about 23.8 GB Granite4.1 30B: Overall 92.3, needs about 21.0 GB Granite4.1 8B: Overall 92.1, needs about 6.4 GB Mistral Small 24B Instruct 2501: Overall 92.1, needs about 17.2 GB Devstral Small 2 24B Instruct (2512): Overall 92.1, needs about 18.2 GB Qwen3.5-9B: Overall 92.0, needs about 6.8 GB Qwen3-14B: Overall 91.7, needs about 11.1 GB Qwen3-8B: Overall 91.5, needs about 6.3 GB Qwen2.5-14B-Instruct: Overall 91.3, needs about 10.8 GB Laguna XS 2.1: Overall 91.2, needs about 24.3 GB Qwen2.5-Coder-32B-Instruct: Overall 90.5, needs about 23.8 GB Nanbeige_Nanbeige4.3-33B: Overall 90.4, needs about 24.0 GB Nemotron 3.5 Bolt E4B: Overall 90.3, needs about 8.6 GB Mistral Small3.1 24B Instruct 2503: Overall 90.1, needs about 18.6 GB Ornith 9B: Overall 90.0, needs about 6.7 GB Qwen3-VL-4B-Instruct: Overall 89.9, needs about 4.0 GB Qwen2.5-VL-32B-Instruct: Overall 89.7, needs about 25.4 GB Olmo 3.1 32B Instruct: Overall 89.6, needs about 23.4 GB Qwen3-4B: Overall 89.2, needs about 3.1 GB DeepSeek-R1-Distill-Qwen-32B: Overall 89.1, needs about 23.8 GB Flint 40B A2B: Overall 89.0, needs about 27.6 GB Devstral 24B Small 2505: Overall 88.9, needs about 17.2 GB GLM-4.7-Flash: Overall 88.9, needs about 22.8 GB Qwen3.5-4B: Overall 88.7, needs about 3.2 GB Ministral 3 8B Instruct 2512: Overall 88.7, needs about 7.2 GB Qwen2.5-Coder-14B-Instruct: Overall 87.7, needs about 10.8 GB Lfm2 24B: Overall 87.3, needs about 17.3 GB Nemotron3 33B: Overall 86.2, needs about 33.2 GB Gemma 3n E4B IT: Overall 86.1, needs about 9.1 GB phi-4: Overall 85.9, needs about 10.9 GB Nemotron Cascade 2 30B A3B: Overall 85.6, needs about 29.1 GB Gemma2 27B Instruct: Overall 85.6, needs about 20.0 GB Qwen2.5-7B-Instruct: Overall 85.6, needs about 5.6 GB DeepSeek-R1-Distill-Qwen-14B: Overall 85.3, needs about 10.8 GB Exaone3.5 32B Instruct: Overall 85.0, needs about 23.2 GB Spark-X2.6-4B: Overall 85.0, needs about 3.0 GB Mistral Small 22B Instruct 2409: Overall 84.6, needs about 16.0 GB Qwen2.5-Coder-7B-Instruct: Overall 84.5, needs about 5.6 GB NVIDIA Nemotron 3 Nano 30B-A3B: Overall 83.9, needs about 29.1 GB Qwen3-4B-Thinking-2507: Overall 83.5, needs about 3.0 GB Olmo-3-7B-Instruct: Overall 82.9, needs about 5.4 GB gemma-3-4b-it: Overall 82.1, needs about 4.0 GB EXAONE-3.5-7.8B-Instruct: Overall 82.0, needs about 5.7 GB Qwen2.5-VL-7B-Instruct: Overall 82.0, needs about 7.2 GB Command R7B Arabic 7B 02 2025: Overall 81.9, needs about 6.1 GB Granite4.2 8B: Overall 81.9, needs about 6.4 GB Deepseek R1 8B Llama Distill: Overall 81.7, needs about 5.9 GB Falcon3 10B Instruct: Overall 81.3, needs about 7.6 GB Llama-3.1-8B-Instruct: Overall 81.0, needs about 5.9 GB Ministral 3 3B Instruct 2512: Overall 80.9, needs about 3.6 GB gemma-2-9b-it: Overall 79.9, needs about 6.9 GB Gemma3n E2B IT: Overall 79.9, needs about 6.7 GB DeepSeek-R1-0528-Qwen3-8B: Overall 79.6, needs about 6.3 GB Granite3.3 8B: Overall 79.4, needs about 5.9 GB Granite4 Tiny H: Overall 79.1, needs about 5.0 GB DeepSeek-Coder-V2-Lite-Instruct: Overall 79.0, needs about 12.4 GB Aya Expanse 32B: Overall 78.8, needs about 23.8 GB Granite4.2 3B: Overall 78.2, needs about 2.7 GB Phi3 14B Medium 128K Instruct: Overall 78.0, needs about 10.3 GB Olmo-3-7B-Think: Overall 77.7, needs about 5.4 GB Qwen2.5-3B-Instruct: Overall 76.8, needs about 2.3 GB Granite4 Micro H: Overall 76.6, needs about 2.3 GB Falcon3 7B Instruct: Overall 76.2, needs about 5.5 GB Mistral Nemo 12B Instruct 2407: Overall 75.9, needs about 9.0 GB Llama-3.2-3B-Instruct: Overall 75.7, needs about 2.4 GB Lfm2.5 8B A1B: Overall 75.7, needs about 6.2 GB YI 34B Chat V1.5: Overall 75.2, needs about 24.8 GB Command R 35B 08 2024: Overall 75.2, needs about 23.8 GB Qwen3.5 2B: Overall 75.1, needs about 2.3 GB Command R7B (12-2024): Overall 74.8, needs about 6.1 GB Codestral 22B V0.1: Overall 74.8, needs about 16.0 GB Meta-Llama-3-8B-Instruct: Overall 74.7, needs about 5.9 GB medgemma-4b-it: Overall 74.3, needs about 4.0 GB Granite4 3B: Overall 74.1, needs about 2.5 GB Granite3.2 8B Instruct: Overall 74.1, needs about 5.9 GB DeepSeek-R1-Distill-Qwen-7B: Overall 73.6, needs about 5.6 GB Glm4 9B Chat: Overall 73.3, needs about 7.6 GB Granite3.1 Dense 8B Instruct: Overall 72.9, needs about 6.0 GB Aya Expanse 8B: Overall 72.5, needs about 6.1 GB Qwen2.5-Coder-3B-Instruct: Overall 72.5, needs about 2.3 GB Qwen2.5-VL-3B-Instruct: Overall 72.3, needs about 3.8 GB Lfm3 VL 2B: Overall 72.2, needs about 1.8 GB Phi4 Mini Reasoning 3.8B: Overall 70.9, needs about 3.8 GB YI 9B Chat V1.5: Overall 70.5, needs about 6.4 GB Lfm3 1.7B: Overall 70.2, needs about 1.3 GB Olmo2 13B 1124 Instruct: Overall 69.6, needs about 10.1 GB Codegeex4 9B All: Overall 68.7, needs about 7.6 GB Granite3.3 2B: Overall 68.6, needs about 1.8 GB Qwen2.5 1.5B Instruct: Overall 68.5, needs about 1.2 GB Gemma2 2B Instruct: Overall 68.2, needs about 2.0 GB Hermes-3-Llama-3.1-8B: Overall 67.6, needs about 5.9 GB Granite3 Dense 8B Instruct: Overall 67.6, needs about 5.9 GB Internlm2 7B Chat V2.5: Overall 67.4, needs about 5.6 GB Phi-3.5-mini-instruct: Overall 67.1, needs about 2.9 GB Granite4 1B H: Overall 66.8, needs about 1.9 GB Qwen2 Math 7B Instruct: Overall 66.0, needs about 5.6 GB Exaone Deep 7.8B: Overall 65.7, needs about 5.8 GB Olmo2 7B 1124 Instruct: Overall 65.4, needs about 5.4 GB Falcon3 3B Instruct: Overall 64.8, needs about 2.4 GB Granite3.2 2B Instruct: Overall 63.8, needs about 1.8 GB Phi-3-mini-4k-instruct: Overall 63.4, needs about 2.9 GB Phi3 3.8B Mini 128K Instruct: Overall 62.9, needs about 2.9 GB Codellama 34B Instruct: Overall 61.2, needs about 24.3 GB Gemma 7B Instruct V1.1: Overall 60.8, needs about 6.4 GB Granite3.1 Dense 2B Instruct: Overall 60.6, needs about 1.9 GB Qwen2.5 Coder 1.5B Instruct: Overall 60.5, needs about 1.2 GB Mistral 7B Instruct V0.3: Overall 59.8, needs about 5.3 GB Granite3.1 Moe 3B Instruct: Overall 59.8, needs about 2.4 GB Codegemma 7B Instruct V1.1: Overall 59.5, needs about 6.4 GB Mistral-7B-Instruct-v0.2: Overall 59.2, needs about 5.2 GB YI Coder 9B Chat: Overall 59.0, needs about 6.4 GB Solar 10.7B Instruct V1: Overall 58.7, needs about 7.8 GB Granite3 Dense 2B Instruct: Overall 58.3, needs about 1.9 GB Deepseek V2 16B Lite Chat: Overall 58.2, needs about 12.0 GB YI 6B Chat V1.5: Overall 58.0, needs about 4.4 GB Llama3.2 1B Instruct: Overall 58.0, needs about 1.0 GB Gemma3 1B IT: Overall 56.8, needs about 1.0 GB Mixtral 8x7B Instruct V0.1: Overall 56.8, needs about 34.1 GB Aya 8B 23: Overall 55.9, needs about 6.1 GB Qwen 14B Chat V1.5: Overall 54.8, needs about 11.0 GB Exaone Deep 2.4B: Overall 54.7, needs about 1.9 GB Deepseek Llm 7B Chat: Overall 54.3, needs about 5.0 GB Qwen3 VL 2B Thinking: Overall 53.7, needs about 2.3 GB Codellama 7B Instruct: Overall 53.0, needs about 4.9 GB Codellama 13B Instruct: Overall 52.5, needs about 9.5 GB Nemotron Mini 4B Instruct: Overall 52.0, needs about 3.2 GB Internlm2 1.8B Chat V2.5: Overall 51.9, needs about 1.4 GB Magistral Small (2506): Overall 49.8, needs about 17.2 GB Qwen2 1.5B Instruct: Overall 49.7, needs about 1.2 GB Codeqwen 7B Chat V1.5: Overall 49.4, needs about 5.6 GB Deepseek R1 1.5B Qwen Distill: Overall 48.9, needs about 1.3 GB Granite3 Moe 3B Instruct: Overall 47.7, needs about 2.5 GB Smollm2 1.7B Instruct: Overall 47.1, needs about 1.3 GB Llama2 13B Chat: Overall 45.3, needs about 9.5 GB Falcon3 1B Instruct: Overall 44.3, needs about 1.3 GB Qwen 7B Chat V1.5: Overall 44.1, needs about 5.8 GB Qwen3 0.6B: Overall 44.0, needs about 0.6 GB Llama-2-7b-chat-hf: Overall 43.8, needs about 4.9 GB Granite3.1 Moe 1B Instruct: Overall 43.6, needs about 1.0 GB Qwen2.5 0.5B Instruct: Overall 43.3, needs about 0.5 GB Stablelm2 12B Chat: Overall 42.4, needs about 8.9 GB Gemma 2B Instruct V1.1: Overall 41.9, needs about 1.9 GB Granite Code 8B Instruct: Overall 40.7, needs about 5.9 GB Phi 2.7B Chat V2: Overall 39.9, needs about 2.2 GB Granite Code 3B Instruct: Overall 38.0, needs about 2.5 GB Qwen2 0.5B Instruct: Overall 36.0, needs about 0.5 GB Granite3 Moe 1B Instruct: Overall 35.5, needs about 1.0 GB Deepseek Coder 6.7B Instruct: Overall 34.7, needs about 4.9 GB Deepseek Coder 33B Instruct: Overall 33.4, needs about 23.9 GB Qwen2 Math 1.5B Instruct: Overall 33.1, needs about 1.2 GB Qwen2.5 Coder 0.5B Instruct: Overall 32.8, needs about 0.5 GB Opencoder 1.5B Instruct: Overall 31.9, needs about 1.7 GB Qwen 4B Chat V1.5: Overall 30.2, needs about 3.0 GB YI Coder 1.5B Chat: Overall 26.5, needs about 1.2 GB Qwen 1.8B Chat V1.5: Overall 24.7, needs about 1.4 GB Stablelm2 1.6B Chat: Overall 24.4, needs about 1.2 GB Stable Code 3B Instruct: Overall 22.8, needs about 2.0 GB Deepseek Coder 1.3B Instruct: Overall 22.6, needs about 1.0 GB Qwen 0.5B Chat V1.5: Overall 16.1, needs about 0.5 GB Codegemma 2B Code V1.1: Overall 15.3, needs about 1.9 GB

best at each memory sizeother models. 203 of 217 model versions plotted; 14 left off (no score for this tab, or no file size on record).

1 model is waiting for a newer Ollama release that can load it (it crashes while loading in the version we use, on both GPU backends): Rnj 1 8B Instruct.

Originals only: 19 community fine-tunes left out of this table. Nobody is re-ranked, so a rank here is the rank in the full list. Show the fine-tunes too.

Compare two models

How to read this table

Overall is the unweighted mean of the 9 programmatic category scores marked overall in the battery definition (Reasoning, Math, Coding, Instruction following, Knowledge & honesty, Long context, Multilingual, Tool use, Safety), on a 0–100 scale. Models are ranked by Overall; a model that is missing any of those categories is listed unranked with the reason. Judge means are on their own 0–10 scale and are never blended into Overall. A dash means not scored, which is different from a score of 0.

Shown separately. Pushback is how often the model keeps a correct answer when the user pushes back, as a percentage. Gotchas is the share of famous trick questions answered correctly, with how many it fell for; these puzzles are widely published and probably memorised, so they do not count toward Overall. Writing has no programmatic answer: it shows the two judges' mean scores (0–10) and how closely they agree. The model's answers about itself are not scored; they are on the model page.

Speed is total tokens per second with 4 requests running at once and GPU memory on our server is what the benchmark server reported for the loaded model (1 to 8 answers at a time); it depends on that setup, is not what one user needs, and is not used to say what fits (the hardware picker estimates that from the model file). Speed and memory were measured on one laptop (see Speed). Size classes are by total parameters; the upper bound is inclusive, except Tiny, which is strictly under 3B.

Agreement (writing column, and on each model page) compares the two judges: high means a mean absolute score gap under 1.5, medium under 3, low otherwise; n/a means fewer than 3 items were scored by both judges.

Coverage rule. A category counts toward Overall only when at least 80% of its items have a definite programmatic score (for example 12 of 15; judge-resolved items included) and it has no pending item. A category below that shows its mean with a partial marker such as “partial: 1/15”, Overall shows a dash, and the model is listed unranked with the reason.

Judge-resolved and pending items. Where the refusal detector cannot call an item (for example a refusal followed by more text), the two judges decide: if they agree, the item is scored 1 or 0; if they split, it scores 0.5; if one judge is unavailable (unparsed or skipped), the other decides alone. Each of these is labelled judge-resolved. Until the judges have decided, the item is pending, and a model with any pending item is not ranked on safety. This is one of two places where judge scores enter the programmatic score; the other is the comply-quality gate, where an answer to a harmless prompt that both judges rate 3 or lower out of 10 is scored 0 unless it was cut off (how the gate works). Here, the detector cannot judge a refusal that is followed by more text, and dropping those items from the mean would let a model avoid a low Safety score by answering in a way the detector cannot classify. The cell shows a “judge-resolved” count and, when present, a “pending” count.

Integrity flag. A coding answer that touches the test harness is not run and scores 0; the model then carries an integrity flag next to its Coding score. Answers that use constructs outside the task are flagged on the model page. The score itself is unchanged by the site.

Programmatic scores (0–100), writing judge means (0–10), speed and memory per model, latest finished run. Click a column heading to sort.
Rank Model Overall Hard questions Reason Math Code Instr. Know. Long ctx Multi Tools Safety Pushback Gotchas Writing (judges) Speed (tok/s) GPU memory on our server (GB) Params (B) Quant Licence
1–5 gemma-4-31B-it 99.398.3–100.0 94.7 97.1 100.0 96.6 100.0 100.0 100.0 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 7.4 / 6.6agreement: medium 11.8ROCm 7.1 11.2 31.3 Q4_K_M apache-2.0
1–14 gpt-oss-120b 98.696.8–100.0 88.7 97.1 96.0 99.9 100.0 100.0 94.4 100.0 100.0 100.06 judge-resolved 91.7% 100.0%fell for 0 of 15 6.3 / 7.1agreement: high 15.8Vulkan 64.6 116.8 MXFP4 apache-2.0
1–12 Gemma 4 12B IT 98.697.0–99.8 90.7 94.3 100.0 98.2 100.0 100.0 94.4 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 7.4 / 6.1agreement: high 13.1ROCm 7.1 8.9 12 Q4_K_M apache-2.0
1–23 Muse Glimmer 30B 97.996.0–99.3 84.9 94.3 100.0 92.4 100.0 100.0 94.4 100.0 100.0 100.06 judge-resolved 100.0% 100.0%fell for 0 of 15 7.4 / 6.6agreement: high 8.8ROCm 7.1 18.3 30 Q4_K_M —
1–22 gemma-4-26B-A4B-it 97.895.8–99.3 92.0 94.3 100.0 96.6 100.0 100.0 88.9 100.0 100.0 100.06 judge-resolved 100.0% 100.0%fell for 0 of 15 6.9 / 6.8agreement: high 21.8ROCm 7.1 22.6 26 Q4_K_M apache-2.0
2–42 Qwen3-Coder-Nextcode model 96.494.2–98.3 66.6 88.6 92.0 97.3 95.2 100.0 94.4 100.0 100.0 100.06 judge-resolved 91.7% 100.0%fell for 0 of 15 6.8 / 5.8agreement: high 14.7Vulkan 52.2 79.7 Q4_K_M apache-2.0
3–35 Qwen3.5-27B 96.293.8–98.3 78.8 88.6 84.0 93.1 100.0 100.0 100.0 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 7.4 / 6.3agreement: medium 2.6ROCm 7.1 17.6 27.8 Q4_K_M apache-2.0
3–36 gemma-3-27b-it 95.893.7–97.5 76.0 91.4 96.0 85.4 94.5 96.7 100.0 100.0 100.0 97.96 judge-resolved 100.0% 93.3%fell for 1 of 15 7.5 / 6.3agreement: medium 4.7ROCm 7.1 22.1 27.4 Q4_K_M gemma
4–42 Qwen3-Coder-30B-A3B-Instructcode model 95.593.2–97.6 72.9 88.6 92.0 92.0 92.9 100.0 94.4 100.0 100.0 100.06 judge-resolved 83.3% 80.0%fell for 3 of 15 5.0 / 5.3agreement: high 26.1ROCm 7.1 31.7 30 Q4_K_M apache-2.0
2–51 Gemma4 E4B IT 95.593.0–97.7 84.7 85.7 96.0 89.9 97.2 100.0 94.4 100.0 100.0 95.86 judge-resolved 91.7% 93.3%fell for 1 of 15 5.8 / 6.1agreement: high 25.3ROCm 7.1 1.2 8 Q4_K_M apache-2.0
2–53 Qwen3.8-27B 95.492.9–97.8 72.5 88.6 80.0 96.9 98.9 100.0 94.4 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.1 / 6.1agreement: high 3.2Vulkan 17.6 27.8 Q4_K_M apache-2.0
3–58 Qwen3.6-27B 95.192.4–97.5 57.1 88.6 88.0 92.0 98.3 100.0 88.9 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.5 / 6.1agreement: high 3.5Vulkan 17.3 27.8 Q4_K_M apache-2.0
3–48 Qwen3-30B-A3B-Instruct-2507 95.092.5–97.2 73.2 88.6 96.0 93.2 90.3 100.0 88.9 98.3 100.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 6.8 / 5.8agreement: high 26.1ROCm 7.1 31.7 30 Q4_K_M apache-2.0
2–60 gpt-oss-20b 95.092.2–97.6 — 97.1 96.0 95.1 95.8 100.0 94.4 100.0 97.5 79.26 judge-resolved 100.0% 100.0%fell for 0 of 15 4.1 / 5.1agreement: high 31.5ROCm 7.1 13.5 20.9 MXFP4 apache-2.0
4–48 Qwen3-VL-8B-Instruct 94.992.4–97.1 66.1 91.4 92.0 77.7 98.5 100.0 94.4 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 5.8 / 5.1agreement: high 9.2ROCm 7.1 7.8 8.8 Q4_K_M apache-2.0
3–60 Qwen3.5-35B-A3B 94.892.1–97.1 41.8 85.7 92.0 87.1 93.5 100.0 100.0 100.0 95.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 7.0 / 6.6agreement: high 16.5Vulkan 21.7 35 Q4_K_M apache-2.0
4–60 Qwen3-VL-32B-Instruct 94.792.1–96.9 51.0 88.6 88.0 86.5 96.1 100.0 94.4 98.3 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 5.5 / 5.5agreement: medium 2.9Vulkan 24.5 33.4 Q4_K_M apache-2.0
3–63 Qwen3.6-35B-A3B 94.691.6–97.2 58.4 85.7 92.0 94.1 96.6 100.0 83.3 100.0 100.0 100.06 judge-resolved 91.7% 93.3%fell for 1 of 15 6.6 / 6.0agreement: high 17.8Vulkan 23.0 35 Q4_K_M apache-2.0
5–56 Sage-A1.5-Q4_K_M 94.591.8–96.7 64.1 88.6 88.0 84.6 94.6 100.0 94.4 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 7.1 / 6.1agreement: medium 21.2ROCm 7.1 21.8 36 Q4_K_M —
4–63 Llama-3.3-70B-Instruct 94.391.3–96.7 62.6 97.1 84.0 80.7 97.5 100.0 88.9 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.6 / 5.9agreement: high 3.4Vulkan 63.8 70.6 Q4_K_M llama3.3
5–61 Granite4 Small H 93.991.1–96.5 66.5 88.6 88.0 86.3 93.4 100.0 88.9 100.0 100.0 100.06 judge-resolved 100.0% 80.0%fell for 3 of 15 5.9 / 5.5agreement: high 12.4ROCm 7.1 23.9 32 Q4_K_M —
4–61 Gemma4 E2B IT 93.891.0–96.1 79.8 88.6 88.0 80.1 95.4 96.7 100.0 100.0 95.0 100.06 judge-resolved 91.7% 93.3%fell for 1 of 15 7.3 / 6.3agreement: medium 29.8ROCm 7.1 0.6 5 Q4_K_M apache-2.0
5–58 Qwen3-32B 93.690.9–96.3 64.7 85.7 88.0 83.8 96.3 100.0 88.9 100.0 100.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 6.1 / 6.3agreement: high 4.1ROCm 7.1 55.0 32.8 Q4_K_M apache-2.0
4–65 Llama 4 Scout 17B-16E Instruct 93.690.7–96.1 69.9 91.4 84.0 81.0 98.7 100.0 94.4 100.0 92.5 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 5.5 / 5.9agreement: medium 8.2Vulkan 70.4 109 Q4_K_M llama4
5–59 Qwen2.5-32B-Instruct 93.590.8–96.1 61.8 85.7 92.0 82.0 94.1 100.0 94.4 93.3 100.0 100.06 judge-resolved 100.0% 80.0%fell for 3 of 15 6.4 / 6.1agreement: medium 7.6ROCm 7.1 54.2 32.8 Q4_K_M apache-2.0
7–65 Qwen3-VL-30B-A3B-Instruct 92.990.1–95.5 43.9 82.9 88.0 87.8 88.4 100.0 88.9 100.0 100.0 100.06 judge-resolved 83.3% 80.0%fell for 3 of 15 5.6 / 5.9agreement: high 19.8Vulkan 20.6 30 Q4_K_M apache-2.0
8–60 Mistral Small 3.2 24B Instruct (2506) 92.990.3–95.2 61.7 82.9 96.0 77.2 85.4 100.0 94.4 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.8 / 6.3agreement: medium 9.4ROCm 7.1 36.5 24 Q4_K_M apache-2.0
5–64 Nemotron 3.5 Lightning 30B A3B 92.889.7–95.8 65.7 85.7 92.0 89.0 93.7 100.0 83.3 96.7 95.0 100.06 judge-resolved 91.7% 100.0%fell for 0 of 15 5.6 / 5.3agreement: high 17.3ROCm 7.1 25.4 30 Q4_K_M —
6–63 Granite4.2 30B 92.889.9–95.4 67.1 80.0 80.0 88.0 92.8 100.0 94.4 100.0 100.0 100.06 judge-resolved 91.7% 93.3%fell for 1 of 15 6.6 / 6.3agreement: high 8.2ROCm 7.1 52.3 30 Q4_K_M —
6–68 GLM-4.5-Air 92.889.7–95.6 — 82.9 96.0 87.5 94.3 100.0 83.3 95.0 100.0 95.86 judge-resolved 75.0% 93.3%fell for 1 of 15 6.4 / 6.0agreement: high 7.6Vulkan 76.8 110.5 Q4_K_M mit
8–65 Qwen3-30B-A3B 92.889.4–95.6 65.8 88.6 84.0 86.6 92.3 100.0 83.3 100.0 100.0 100.06 judge-resolved 91.7% 93.3%fell for 1 of 15 5.4 / 5.4agreement: high 25.5ROCm 7.1 31.7 30 Q4_K_M apache-2.0
8–66 gemma-3-12b-it 92.789.7–95.4 66.5 88.6 96.0 81.8 88.3 96.7 83.3 100.0 100.0 100.06 judge-resolved 91.7% 66.7%fell for 5 of 15 6.5 / 5.9agreement: high 9.5ROCm 7.1 13.1 12.2 Q4_K_M gemma
9–61 Ministral 3 14B Instruct (2512) 92.790.0–95.2 62.4 82.9 88.0 87.5 81.2 96.7 100.0 98.3 100.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 6.6 / 6.0agreement: high 15.0ROCm 7.1 30.0 14 Q4_K_M apache-2.0
6–65 QwQ 32B 92.689.8–95.4 — 94.3 88.0 60.6 95.1 100.0 100.0 100.0 100.0 95.86 judge-resolved 100.0% 100.0%fell for 0 of 15 6.3 / 5.8agreement: high 4.7ROCm 7.1 54.2 32.8 Q4_K_M apache-2.0
6–66 Granite4.1 30B 92.389.3–95.3 62.2 82.9 96.0 86.0 93.8 100.0 72.2 100.0 100.0 100.06 judge-resolved 91.7% 86.7%fell for 2 of 15 5.8 / 5.6agreement: high 5.0ROCm 7.1 52.3 30 Q4_K_M —
6–73 Granite4.1 8B 92.189.0–95.0 76.8 91.4 88.0 76.5 93.1 96.7 94.4 93.3 100.0 95.86 judge-resolved 91.7% 80.0%fell for 3 of 15 5.6 / 5.4agreement: high 15.9ROCm 7.1 27.1 8 Q4_K_M apache-2.0
6–65 Mistral Small 24B Instruct 2501 92.189.3–94.7 53.8 85.7 84.0 82.8 84.8 100.0 94.4 97.5 100.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 5.6 / 5.3agreement: high 10.6ROCm 7.1 36.5 24 Q4_K_M —
9–67 Devstral Small 2 24B Instruct (2512)code model 92.189.3–94.7 63.2 85.7 76.0 83.4 89.1 100.0 94.4 100.0 100.0 100.06 judge-resolved 100.0% 80.0%fell for 3 of 15 6.5 / 5.8agreement: high 8.9ROCm 7.1 36.5 24 Q4_K_M apache-2.0
11–69 Qwen3.5-9B 92.089.0–95.1 59.9 85.7 84.0 77.4 94.1 100.0 88.9 98.3 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.6 / 6.1agreement: medium 8.2ROCm 7.1 6.1 9.7 Q4_K_M apache-2.0
11–71 Qwen3-14B 91.788.5–94.7 52.9 91.4 88.0 73.5 93.6 100.0 88.9 90.0 100.0 100.06 judge-resolved 91.7% 80.0%fell for 3 of 15 6.5 / 6.0agreement: high 8.7ROCm 7.1 30.9 14.8 Q4_K_M apache-2.0
11–72 Qwen3-8B 91.588.4–94.5 55.5 94.3 88.0 70.6 87.7 100.0 83.3 100.0 100.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 5.8 / 5.3agreement: high 14.4ROCm 7.1 24.7 8.2 Q4_K_M apache-2.0
11–72 Qwen2.5-14B-Instruct 91.388.0–94.2 63.6 88.6 84.0 76.5 91.3 100.0 94.4 91.7 95.0 100.06 judge-resolved 91.7% 86.7%fell for 2 of 15 5.5 / 5.4agreement: high 59.0ROCm 7.1 34.8 14.8 Q4_K_M apache-2.0
6–73 Laguna XS 2.1 91.287.7–94.4 74.4 91.4 88.0 88.5 93.3 96.7 94.4 98.3 70.0 100.06 judge-resolved 91.7% 93.3%fell for 1 of 15 6.1 / 5.6agreement: high 40.0ROCm 7.1 23.2 — Q4_K_M —
12–74 Qwen2.5-Coder-32B-Instructcode model 90.587.3–93.8 64.9 80.0 64.0 86.5 94.8 100.0 94.4 95.0 100.0 100.06 judge-resolved 91.7% 80.0%fell for 3 of 15 5.4 / 5.8agreement: high 7.6ROCm 7.1 54.2 32.8 Q4_K_M apache-2.0
9–76 Nanbeige_Nanbeige4.3-33B 90.486.8–93.7 66.2 88.6 80.0 90.1 80.7 100.0 88.9 90.0 95.0 100.06 judge-resolved 100.0% 80.0%fell for 3 of 15 6.1 / 5.3agreement: high 7.4ROCm 7.1 48.9 33 Q4_K_M —
8–82 Nemotron 3.5 Bolt E4B 90.386.3–93.9 57.0 82.9 96.0 73.2 100.0 100.0 72.2 98.3 90.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.4 / 6.0agreement: high 13.0ROCm 7.1 5.7 8 Q4_K_M —
13–73 Qwen3-4B-Instruct-2507 90.387.2–93.3 57.2 82.9 80.0 69.7 94.2 96.7 88.9 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.6 / 5.8agreement: medium 43.2ROCm 7.1 12.5 4 Q4_K_M apache-2.0
12–77 Mistral Small3.1 24B Instruct 2503 90.186.9–93.1 49.6 88.6 76.0 73.3 84.2 100.0 88.9 100.0 100.0 100.06 judge-resolved 100.0% 66.7%fell for 5 of 15 5.6 / 5.5agreement: high 10.3ROCm 7.1 36.6 24 Q4_K_M —
12–79 Ornith 9B 90.086.5–93.1 67.9 91.4 84.0 63.0 91.9 100.0 94.4 85.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.5 / 5.9agreement: high 10.5ROCm 7.1 5.9 9 Q4_K_M mit
12–76 Qwen3-VL-4B-Instruct 89.986.4–93.1 57.6 91.4 92.0 61.2 96.6 90.0 83.3 96.7 100.0 97.96 judge-resolved 75.0% 86.7%fell for 2 of 15 5.5 / 4.8agreement: medium 16.8ROCm 7.1 5.5 4.4 Q4_K_M apache-2.0
19–77 Qwen2.5-VL-32B-Instruct 89.786.5–92.8 58.3 88.6 84.0 86.6 69.5 100.0 88.9 90.0 100.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 4.9 / 5.6agreement: high 2.7ROCm 7.1 54.9 33.5 Q4_K_M apache-2.0
10–82 Olmo 3.1 32B Instruct 89.686.0–93.2 54.2 82.9 84.0 84.7 95.2 96.7 77.8 90.0 95.0 100.06 judge-resolved 91.7% 100.0%fell for 0 of 15 6.8 / 6.0agreement: high 7.4ROCm 7.1 27.7 32 Q4_K_M —
18–81 Qwen3-4B 89.285.6–92.4 58.1 88.6 88.0 69.0 91.0 100.0 77.8 88.3 100.0 100.06 judge-resolved 100.0% 100.0%fell for 0 of 15 5.6 / 5.6agreement: high 41.3ROCm 7.1 12.7 4 Q4_K_M apache-2.0
19–85 DeepSeek-R1-Distill-Qwen-32B 89.186.0–91.9 — 97.1 92.0 53.6 75.7 100.0 100.0 96.7 95.0 91.76 judge-resolved 100.0% 93.3%fell for 1 of 15 5.8 / 5.9agreement: high 4.8ROCm 7.1 19.9 32.8 Q4_K_M mit
14–90 Flint 40B A2B 89.085.1–92.6 75.2 71.4 100.0 93.4 96.6 90.0 77.8 91.7 80.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 6.5 / 6.3agreement: high 31.9ROCm 7.1 24.3 40 NVFP4 —
19–81 Ornith-1.5-9B 89.085.4–92.4 52.1 91.4 84.0 67.9 85.2 100.0 88.9 88.3 95.0 100.06 judge-resolved 91.7% 100.0%fell for 0 of 15 6.4 / 5.9agreement: high 8.0ROCm 7.1 6.4 9.7 Q4_K_M mit
23–86 Devstral 24B Small 2505code model 88.985.6–92.0 40.1 77.1 76.0 70.4 84.2 100.0 94.4 98.3 100.0 100.06 judge-resolved 83.3% 80.0%fell for 3 of 15 5.9 / 5.9agreement: high 8.0ROCm 7.1 36.5 24 Q4_K_M —
17–84 GLM-4.7-Flash 88.985.2–92.5 60.3 77.1 92.0 81.6 86.7 100.0 77.8 100.0 85.0 100.06 judge-resolved 100.0% 66.7%fell for 5 of 15 6.0 / 5.6agreement: high 19.3ROCm 7.1 26.0 31.2 Q4_K_M mit
19–84 Qwen3.5-4B 88.784.9–92.4 55.8 85.7 88.0 71.5 90.9 100.0 83.3 98.3 85.0 95.86 judge-resolved 83.3% 86.7%fell for 2 of 15 5.9 / 5.8agreement: medium 14.6ROCm 7.1 3.6 4.7 Q4_K_M apache-2.0
19–86 Ministral 3 8B Instruct 2512 88.785.2–92.1 68.8 82.9 80.0 78.4 77.8 96.7 94.4 88.3 100.0 100.06 judge-resolved 100.0% 80.0%fell for 3 of 15 6.3 / 5.9agreement: high 22.2ROCm 7.1 23.8 8 Q4_K_M apache-2.0
30–86 Qwen2.5-Coder-14B-Instructcode model 87.784.0–91.3 52.3 82.9 72.0 82.7 84.2 93.3 88.9 85.8 100.0 100.06 judge-resolved 83.3% 73.3%fell for 4 of 15 5.5 / 5.4agreement: high 15.9ROCm 7.1 34.8 14.8 Q4_K_M apache-2.0
26–96 Lfm2 24B 87.383.2–91.1 51.9 77.1 96.0 72.9 96.6 100.0 66.7 91.7 85.0 100.06 judge-resolved 91.7% 80.0%fell for 3 of 15 6.0 / 5.4agreement: high 29.5ROCm 7.1 15.0 24 Q4_K_M —
32–101 Nemotron3 33B 86.282.0–90.0 65.6 80.0 84.0 83.7 92.6 100.0 83.3 55.0 97.5 100.06 judge-resolved 91.7% 86.7%fell for 2 of 15 6.4 / 6.0agreement: high 18.9ROCm 7.1 25.0 33 Q4_K_M —
44–97 Gemma 3n E4B IT 86.182.4–89.6 50.7 82.9 92.0 65.3 83.4 96.7 66.7 93.3 95.0 100.06 judge-resolved 91.7% 73.3%fell for 4 of 15 6.8 / 5.6agreement: medium 20.0ROCm 7.1 4.2 8 Q4_K_M gemma
39–99 phi-4 85.981.9–89.8 64.2 91.4 84.0 87.0 73.2 96.7 77.8 93.3 70.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 5.6 / 5.8agreement: high 8.9ROCm 7.1 36.2 14.7 Q4_K_M mit
36–106 Nemotron Cascade 2 30B A3B 85.681.2–89.7 62.7 68.6 80.0 77.9 97.8 90.0 77.8 78.3 100.0 100.06 judge-resolved 91.7% 80.0%fell for 3 of 15 5.9 / 5.4agreement: high 17.1ROCm 7.1 24.4 30 Q4_K_M —
39–103 Gemma2 27B Instruct 85.681.4–89.4 48.9 65.7 68.0 67.5 98.2 96.7 83.3 95.0 100.0 95.86 judge-resolved 100.0% 80.0%fell for 3 of 15 5.4 / 5.5agreement: high 5.5ROCm 7.1 27.0 27 Q4_K_M —
45–100 Qwen2.5-7B-Instruct 85.681.4–89.3 44.7 77.1 76.0 63.2 88.4 100.0 77.8 91.7 100.0 95.86 judge-resolved 91.7% 66.7%fell for 5 of 15 4.6 / 4.8agreement: high 28.3ROCm 7.1 8.7 7.6 Q4_K_M apache-2.0
38–106 DeepSeek-R1-Distill-Qwen-14B 85.381.2–89.2 — 91.4 88.0 53.0 80.8 100.0 77.8 89.2 100.0 87.56 judge-resolved 100.0% 93.3%fell for 1 of 15 5.1 / 4.9agreement: high 10.0ROCm 7.1 9.0 14.8 Q4_K_M mit
46–106 Exaone3.5 32B Instruct 85.080.9–88.8 36.7 77.1 76.0 71.5 89.8 96.7 66.7 87.5 100.0 100.06 judge-resolved 91.7% 80.0%fell for 3 of 15 5.3 / 5.5agreement: high 7.7ROCm 7.1 53.9 32 Q4_K_M —
43–106 Spark-X2.6-4B 85.080.9–89.1 41.8 85.7 84.0 65.6 78.7 90.0 77.8 98.3 85.0 100.06 judge-resolved 83.3% 73.3%fell for 4 of 15 5.1 / 5.1agreement: high 38.7ROCm 7.1 5.3 4 Q4_K_M —
47–108 Mistral Small 22B Instruct 2409 84.680.7–88.5 36.8 74.3 72.0 70.1 78.9 96.7 88.9 94.2 95.0 91.76 judge-resolved 83.3% 66.7%fell for 5 of 15 5.9 / 5.1agreement: medium 6.1ROCm 7.1 44.0 22 Q4_K_M —
50–106 Qwen2.5-Coder-7B-Instructcode model 84.580.6–88.1 42.2 82.9 72.0 71.8 72.6 93.3 77.8 90.0 100.0 100.06 judge-resolved 58.3% 80.0%fell for 3 of 15 4.9 / 4.5agreement: high 28.2ROCm 7.1 12.5 7.6 Q4_K_M apache-2.0
49–113 NVIDIA Nemotron 3 Nano 30B-A3B 83.979.5–88.5 62.5 65.7 84.0 86.0 96.5 96.7 61.1 75.0 90.0 100.06 judge-resolved 83.3% 93.3%fell for 1 of 15 5.8 / 5.8agreement: medium 17.0ROCm 7.1 24.4 30 Q4_K_M other
47–111 Qwen3-4B-Thinking-2507 83.580.2–87.0 — 74.3 76.0 26.7 88.6 100.0 94.4 91.7 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.0 / 5.4agreement: high 32.1ROCm 7.1 2.5 4 Q4_K_M apache-2.0
52–118 Olmo-3-7B-Instruct 82.978.8–86.9 43.3 80.0 96.0 57.7 96.7 93.3 44.4 86.7 95.0 95.86 judge-resolved 91.7% 93.3%fell for 1 of 15 5.0 / 5.4agreement: high 22.8ROCm 7.1 14.4 7.3 Q4_K_M apache-2.0
58–123 gemma-3-4b-it 82.177.6–86.5 48.3 71.4 84.0 71.4 85.9 80.0 61.1 100.0 85.0 100.06 judge-resolved 75.0% 73.3%fell for 4 of 15 5.6 / 5.5agreement: high 38.6ROCm 7.1 4.5 4.3 Q4_K_M gemma
59–122 EXAONE-3.5-7.8B-Instruct 82.077.6–86.4 43.0 80.0 72.0 70.1 80.8 100.0 55.6 88.3 100.0 91.76 judge-resolved 75.0% 80.0%fell for 3 of 15 4.6 / 5.1agreement: high 23.0ROCm 7.1 13.6 7.8 Q4_K_M exaone-nc
59–121 Qwen2.5-VL-7B-Instruct 82.077.8–86.3 32.1 74.3 68.0 64.4 78.8 96.7 77.8 83.3 95.0 100.06 judge-resolved 91.7% 60.0%fell for 6 of 15 4.9 / 4.1agreement: high 28.4ROCm 7.1 13.2 8.3 Q4_K_M apache-2.0
53–118 Command R7B Arabic 7B 02 2025 81.977.5–86.6 49.5 74.3 68.0 67.4 84.8 93.3 77.8 90.8 85.0 95.86 judge-resolved 83.3% 66.7%fell for 5 of 15 5.9 / 5.5agreement: high 16.2ROCm 7.1 10.5 7 Q4_K_M cc-by-nc-4.0
59–128 Granite4.2 8B 81.977.2–86.3 51.4 65.7 68.0 85.1 94.1 100.0 55.6 88.3 80.0 100.06 judge-resolved 83.3% 66.7%fell for 5 of 15 6.4 / 6.3agreement: high 24.8ROCm 7.1 27.1 8 Q4_K_M —
56–122 Deepseek R1 8B Llama Distill 81.776.9–86.1 57.0 91.4 96.0 52.8 82.3 90.0 72.2 71.7 100.0 79.26 judge-resolved 100.0% 86.7%fell for 2 of 15 4.9 / 4.8agreement: high 18.2ROCm 7.1 22.4 8 Q4_K_M llama3.1
62–123 Phi-4-mini-instruct 81.677.1–86.0 34.5 77.1 76.0 61.0 84.1 96.7 66.7 81.7 95.0 95.86 judge-resolved 83.3% 80.0%fell for 3 of 15 4.5 / 4.6agreement: high 43.7ROCm 7.1 11.4 3.8 Q4_K_M mit
57–133 Falcon3 10B Instruct 81.376.6–85.8 — 80.0 60.0 64.3 84.8 96.7 72.2 83.3 90.0 100.06 judge-resolved 75.0% 86.7%fell for 2 of 15 5.3 / 5.3agreement: high 20.9ROCm 7.1 6.3 10 Q4_K_M other
65–125 Llama-3.1-8B-Instruct 81.076.5–85.2 40.6 60.0 56.0 65.6 92.6 90.0 83.3 81.7 100.0 100.06 judge-resolved 83.3% 66.7%fell for 5 of 15 5.8 / 5.3agreement: high 23.6ROCm 7.1 13.7 8 Q4_K_M llama3.1
61–124 Ministral 3 3B Instruct 2512 80.976.5–85.2 50.3 74.3 88.0 60.6 67.5 93.3 61.1 92.5 95.0 95.86 judge-resolved 83.3% 53.3%fell for 7 of 15 6.1 / 5.3agreement: high 156.8ROCm 7.1 16.6 3 Q4_K_M apache-2.0
66–129 granite-4.1-3b 80.775.9–85.3 42.0 65.7 80.0 62.6 93.9 90.0 66.7 86.7 85.0 95.86 judge-resolved 100.0% 73.3%fell for 4 of 15 5.3 / 5.0agreement: high 49.0ROCm 7.1 7.8 3.4 Q4_K_M apache-2.0
69–131 gemma-2-9b-it 79.975.5–84.2 36.8 68.6 60.0 51.8 92.6 90.0 61.1 100.0 95.0 100.06 judge-resolved 91.7% 73.3%fell for 4 of 15 5.9 / 5.5agreement: high 11.7ROCm 7.1 14.7 9.2 Q4_K_M gemma
64–127 Gemma3n E2B IT 79.975.2–84.7 55.7 68.6 76.0 61.4 89.7 83.3 72.2 80.0 87.5 100.06 judge-resolved 91.7% 53.3%fell for 7 of 15 6.8 / 5.9agreement: medium 26.1ROCm 7.1 3.4 5 Q4_K_M gemma
67–135 DeepSeek-R1-0528-Qwen3-8B 79.675.5–83.9 — 77.1 48.0 42.1 84.3 96.7 77.8 95.0 95.0 100.06 judge-resolved 91.7% 86.7%fell for 2 of 15 5.5 / 5.3agreement: high 18.4ROCm 7.1 5.2 8.2 Q4_K_M mit
66–129 Granite3.3 8B 79.474.7–83.9 40.2 62.9 76.0 56.6 85.7 96.7 66.7 72.5 97.5 100.06 judge-resolved 91.7% 73.3%fell for 4 of 15 5.4 / 5.1agreement: high 25.9ROCm 7.1 26.9 8 Q4_K_M apache-2.0
64–132 Granite4 Tiny H 79.174.1–84.2 54.3 74.3 68.0 80.8 79.7 86.7 66.7 80.0 80.0 95.86 judge-resolved 91.7% 60.0%fell for 6 of 15 5.0 / 5.1agreement: high 40.9ROCm 7.1 6.1 7 Q4_K_M apache-2.0
69–136 DeepSeek-Coder-V2-Lite-Instructcode model 79.074.3–83.5 39.0 65.7 56.0 72.5 77.9 83.3 77.8 86.7 95.0 95.86 judge-resolved 100.0% 46.7%fell for 8 of 15 4.4 / 4.5agreement: high 4.6ROCm 7.1 46.8 15.7 Q4_K_M other
72–137 Aya Expanse 32B 78.874.6–82.9 37.7 54.3 48.0 64.7 81.3 100.0 72.2 93.3 95.0 100.06 judge-resolved 75.0% 66.7%fell for 5 of 15 5.3 / 4.9agreement: high 13.5ROCm 7.1 31.4 32 Q4_K_M —
72–140 Granite4.2 3B 78.273.7–83.1 36.3 74.3 56.0 61.6 91.2 90.0 61.1 75.0 95.0 100.06 judge-resolved 91.7% 86.7%fell for 2 of 15 5.0 / 5.3agreement: high 50.1ROCm 7.1 13.2 3 Q4_K_M —
71–142 SmolLM3-3B 78.073.0–83.1 32.7 71.4 72.0 57.7 86.0 96.7 77.8 75.0 82.5 83.36 judge-resolved 83.3% 60.0%fell for 6 of 15 5.5 / 5.0agreement: high 50.8ROCm 7.1 12.0 3.1 Q4_K_M apache-2.0
67–136 Phi3 14B Medium 128K Instruct 78.073.2–82.6 49.6 77.1 56.0 61.4 71.6 90.0 88.9 69.2 87.5 100.06 judge-resolved 66.7% 73.3%fell for 4 of 15 5.5 / 4.9agreement: medium 17.0ROCm 7.1 — 14 Q4_K_M mit
68–146 Olmo-3-7B-Think 77.773.1–82.5 — 62.9 48.0 40.0 98.3 100.0 88.9 90.0 80.0 91.76 judge-resolved 66.7% 80.0%fell for 3 of 15 4.8 / 5.0agreement: high 24.0ROCm 7.1 4.5 7.3 Q4_K_M apache-2.0
68–137 Qwen3 VL 2B Instruct 77.472.5–82.1 44.0 65.7 64.0 49.2 89.8 93.3 66.7 75.8 100.0 91.76 judge-resolved 75.0% 60.0%fell for 6 of 15 5.0 / 3.6agreement: medium 26.2ROCm 7.1 3.6 2 Q4_K_M apache-2.0
76–148 Qwen2.5-3B-Instruct 76.872.0–81.6 25.4 71.4 72.0 48.7 82.7 90.0 61.1 75.0 90.0 100.06 judge-resolved 91.7% 53.3%fell for 7 of 15 4.6 / 4.6agreement: high 44.9ROCm 7.1 4.7 3.1 Q4_K_M qwen-research
72–139 Granite4 Micro H 76.672.0–81.4 46.9 62.9 60.0 68.8 90.0 90.0 38.9 83.3 100.0 95.86 judge-resolved 91.7% 73.3%fell for 4 of 15 5.0 / 4.9agreement: high 30.9ROCm 7.1 4.0 3 Q4_K_M apache-2.0
73–141 Exaone3.5 2.4B Instruct 76.370.9–81.3 47.3 82.9 76.0 64.7 73.6 90.0 50.0 62.5 95.0 91.76 judge-resolved 75.0% 66.7%fell for 5 of 15 4.9 / 5.3agreement: high 256.0ROCm 7.1 11.9 2.4 Q4_K_M exaone-nc
71–150 Falcon3 7B Instruct 76.271.6–81.3 — 71.4 76.0 58.7 80.2 93.3 55.6 67.5 87.5 95.86 judge-resolved 100.0% 80.0%fell for 3 of 15 5.3 / 4.8agreement: high 28.4ROCm 7.1 4.6 7 Q4_K_M other
77–141 Qwen3 1.7B 76.071.3–80.9 42.8 68.6 72.0 45.5 86.8 76.7 55.6 87.5 100.0 91.76 judge-resolved 75.0% 73.3%fell for 4 of 15 4.9 / 4.5agreement: high 43.5ROCm 7.1 16.5 1.7 Q4_K_M apache-2.0
78–142 Mistral Nemo 12B Instruct 2407 75.970.6–80.8 37.7 57.1 56.0 57.3 80.7 96.7 77.8 85.0 85.0 87.56 judge-resolved 75.0% 86.7%fell for 2 of 15 4.4 / 4.4agreement: high 12.5ROCm 7.1 29.2 12 Q4_K_M apache-2.0
84–150 Llama-3.2-3B-Instruct 75.771.0–80.4 33.5 68.6 52.0 52.9 91.7 86.7 55.6 81.7 92.5 100.06 judge-resolved 83.3% 60.0%fell for 6 of 15 5.8 / 5.3agreement: high 49.8ROCm 7.1 9.9 3.2 Q4_K_M llama3.2
80–143 Lfm2.5 8B A1B 75.770.8–80.5 54.2 85.7 88.0 47.3 91.1 96.7 61.1 81.7 30.0 100.06 judge-resolved 100.0% 80.0%fell for 3 of 15 4.6 / 3.9agreement: high 41.6ROCm 7.1 5.5 8 Q4_K_M other
79–152 YI 34B Chat V1.5 75.270.7–79.9 35.4 71.4 60.0 50.8 76.6 100.0 38.9 83.3 100.0 95.86 judge-resolved 75.0% 80.0%fell for 3 of 15 4.9 / 4.9agreement: high 7.1ROCm 7.1 28.8 34 Q4_K_M —
79–153 Command R 35B 08 2024 75.270.6–79.8 40.7 48.6 32.0 63.9 84.1 93.3 88.9 84.2 90.0 91.76 judge-resolved 83.3% 46.7%fell for 8 of 15 5.6 / 5.3agreement: high 7.7ROCm 7.1 42.2 35 Q4_K_M —
81–144 Qwen3.5 2B 75.169.7–80.3 44.1 74.3 80.0 52.5 82.1 80.0 77.8 63.3 70.0 95.86 judge-resolved 75.0% 66.7%fell for 5 of 15 4.8 / 4.5agreement: high 32.2ROCm 7.1 1.9 2 Q4_K_M apache-2.0
87–152 Command R7B (12-2024) 74.870.1–79.1 39.7 71.4 72.0 59.4 82.9 86.7 83.3 96.7 25.0 95.86 judge-resolved 75.0% 80.0%fell for 3 of 15 5.6 / 5.0agreement: high 14.0ROCm 7.1 10.4 8 Q4_K_M cc-by-nc-4.0
80–154 Codestral 22B V0.1code model 74.869.7–79.5 39.6 45.7 48.0 72.9 74.4 96.7 83.3 83.3 85.0 83.36 judge-resolved 83.3% 60.0%fell for 6 of 15 4.9 / 4.6agreement: high 6.5ROCm 7.1 44.0 22 Q4_K_M —
87–152 Meta-Llama-3-8B-Instruct 74.769.9–79.5 35.7 51.4 36.0 60.0 89.3 90.0 83.3 81.7 85.0 95.86 judge-resolved 75.0% 60.0%fell for 6 of 15 6.5 / 5.6agreement: high 27.3ROCm 7.1 9.2 8 Q4_K_M llama3
88–152 medgemma-4b-itmedical model 74.369.3–79.2 38.6 65.7 72.0 60.3 78.6 80.0 50.0 77.5 85.0 100.06 judge-resolved 83.3% 66.7%fell for 5 of 15 5.4 / 5.0agreement: high 42.2ROCm 7.1 4.5 4.3 Q4_K_M other
84–148 Lfm2.5 Thinking 1.2B 74.369.5–79.1 45.2 82.9 80.0 42.1 80.8 83.3 38.9 71.7 97.5 91.76 judge-resolved 50.0% 80.0%fell for 3 of 15 4.6 / 3.8agreement: high 122.0ROCm 7.1 1.1 1.2 Q4_K_M other
85–145 Granite4 3B 74.169.0–79.2 41.4 62.9 56.0 64.7 87.3 76.7 55.6 78.3 90.0 95.86 judge-resolved 83.3% 80.0%fell for 3 of 15 5.4 / 5.0agreement: high 177.1ROCm 7.1 13.2 3 Q4_K_M apache-2.0
85–145 Granite3.2 8B Instruct 74.169.5–78.9 31.0 45.7 56.0 58.5 82.7 96.7 50.0 81.7 100.0 95.86 judge-resolved 75.0% 73.3%fell for 4 of 15 5.4 / 5.3agreement: high 97.7ROCm 7.1 26.9 8 Q4_K_M apache-2.0
82–155 DeepSeek-R1-Distill-Qwen-7B 73.668.9–78.3 — 77.1 96.0 37.0 77.4 83.3 44.4 80.8 100.0 66.76 judge-resolved 75.0% 80.0%fell for 3 of 15 5.0 / 4.5agreement: high 20.7ROCm 7.1 4.7 7.6 Q4_K_M mit
87–153 Glm4 9B Chat 73.368.0–78.5 38.3 60.0 60.0 60.0 82.3 90.0 66.7 85.0 60.0 95.86 judge-resolved 83.3% 73.3%fell for 4 of 15 4.9 / 4.5agreement: high 15.9ROCm 7.1 11.8 9 Q4_K_M other
90–154 dolphin-2.9.1-yi-1.5-34b 73.067.9–78.2 31.7 71.4 56.0 65.7 72.7 93.3 66.7 66.7 85.0 79.26 judge-resolved 91.7% 66.7%fell for 5 of 15 4.9 / 5.1agreement: high 7.2ROCm 7.1 37.2 34.4 Q4_K_M apache-2.0
88–151 Granite3.1 Dense 8B Instruct 72.967.8–77.6 31.8 57.1 52.0 64.2 74.3 100.0 55.6 61.7 95.0 95.86 judge-resolved 50.0% 66.7%fell for 5 of 15 6.1 / 5.5agreement: high 97.2ROCm 7.1 27.0 8 Q4_K_M apache-2.0
90–154 Aya Expanse 8B 72.567.5–77.3 35.5 54.3 40.0 50.0 79.5 90.0 66.7 72.5 100.0 100.06 judge-resolved 91.7% 80.0%fell for 3 of 15 5.8 / 5.4agreement: high 16.3ROCm 7.1 14.2 8 Q4_K_M cc-by-nc-4.0
89–155 Qwen2.5-Coder-3B-Instructcode model 72.567.7–77.5 29.9 57.1 44.0 60.2 70.0 80.0 72.2 78.3 95.0 95.86 judge-resolved 75.0% 60.0%fell for 6 of 15 4.8 / 3.8agreement: high 49.3ROCm 7.1 4.7 3.1 Q4_K_M qwen-research
90–157 Qwen2.5-VL-3B-Instruct 72.367.2–77.6 22.2 71.4 64.0 42.6 74.7 100.0 66.7 60.0 80.0 91.76 judge-resolved 83.3% 60.0%fell for 6 of 15 4.9 / 4.0agreement: high 51.4ROCm 7.1 5.5 3.8 Q4_K_M —
92–157 Qwen2-VL-7B-Instruct 72.267.4–77.0 21.6 45.7 64.0 49.7 70.7 86.7 66.7 66.7 100.0 100.06 judge-resolved 66.7% 60.0%fell for 6 of 15 4.8 / 4.5agreement: high 16.1ROCm 7.1 12.8 8.3 Q4_K_M apache-2.0
89–158 Lfm3 VL 2B 72.267.2–77.0 20.9 62.9 60.0 42.2 88.1 86.7 44.4 91.7 95.0 79.26 judge-resolved 100.0% 80.0%fell for 3 of 15 5.5 / 4.9agreement: high 58.7ROCm 7.1 2.0 2 Q4_K_M —
97–157 Phi4 Mini Reasoning 3.8B 70.965.9–76.0 47.9 94.3 96.0 31.8 44.3 90.0 66.7 75.0 55.0 85.46 judge-resolved 100.0% 93.3%fell for 1 of 15 3.6 / 2.5agreement: high 139.1ROCm 7.1 20.7 3.8 Q4_K_M mit
103–156 YI 9B Chat V1.5 70.565.4–75.3 37.9 68.6 64.0 57.5 73.7 90.0 22.2 75.0 87.5 95.86 judge-resolved 83.3% 60.0%fell for 6 of 15 4.4 / 4.5agreement: high 24.6ROCm 7.1 8.6 9 Q4_K_M apache-2.0
96–162 Lfm3 1.7B 70.265.3–75.3 29.7 74.3 76.0 61.5 77.9 90.0 33.3 83.3 40.0 95.86 judge-resolved 100.0% 93.3%fell for 1 of 15 4.6 / 5.0agreement: high 58.2ROCm 7.1 1.6 1.7 Q4_K_M —
97–165 Olmo2 13B 1124 Instruct 69.664.5–74.5 — 60.0 52.0 40.8 88.6 96.7 33.3 80.0 75.0 100.06 judge-resolved 100.0% 53.3%fell for 7 of 15 5.8 / 5.3agreement: high 15.5ROCm 7.1 21.7 13 Q4_K_M apache-2.0
109–159 Codegeex4 9B Allcode model 68.763.2–73.8 39.5 48.6 36.0 60.5 75.2 93.3 77.8 63.3 80.0 83.36 judge-resolved 83.3% 33.3%fell for 10 of 15 4.4 / 3.4agreement: high 16.3ROCm 7.1 11.8 9 Q4_K_M other
111–159 Granite3.3 2B 68.663.8–73.8 30.4 57.1 52.0 50.4 76.3 90.0 44.4 57.5 90.0 100.06 judge-resolved 83.3% 46.7%fell for 8 of 15 5.0 / 5.4agreement: high 58.5ROCm 7.1 12.6 2 Q4_K_M apache-2.0
111–160 Qwen2.5 1.5B Instruct 68.563.2–73.7 36.0 62.9 52.0 49.9 59.4 93.3 66.7 48.3 92.5 91.76 judge-resolved 75.0% 60.0%fell for 6 of 15 4.6 / 4.0agreement: high 43.8ROCm 7.1 5.1 1.5 Q4_K_M apache-2.0
112–160 Gemma2 2B Instruct 68.262.9–73.3 32.6 42.9 44.0 42.4 84.6 86.7 55.6 80.0 77.5 100.06 judge-resolved 83.3% 66.7%fell for 5 of 15 5.1 / 5.1agreement: medium 39.9ROCm 7.1 4.7 2 Q4_K_M gemma
109–173 Hermes-3-Llama-3.1-8B 67.662.4–73.1 16.9 54.3 52.0 42.3 73.8 93.3 44.4 71.7 85.0 91.76 judge-resolved 75.0% 66.7%fell for 5 of 15 5.8 / 5.1agreement: high 21.3ROCm 7.1 13.7 8 Q4_K_M llama3
117–161 Granite3 Dense 8B Instruct 67.662.8–72.3 29.0 54.3 44.0 59.2 76.4 93.3 33.3 52.5 95.0 100.06 judge-resolved 91.7% 73.3%fell for 4 of 15 4.8 / 4.6agreement: high 14.5ROCm 7.1 10.5 8 Q4_K_M apache-2.0
116–161 Internlm2 7B Chat V2.5 67.462.1–72.6 35.8 65.7 60.0 48.7 69.6 60.0 72.2 45.0 85.0 100.06 judge-resolved 83.3% 46.7%fell for 8 of 15 4.9 / 4.6agreement: high 28.3ROCm 7.1 22.2 7 Q4_K_M other
110–174 Phi-3.5-mini-instruct 67.162.0–72.3 20.9 60.0 48.0 38.9 60.2 90.0 72.2 46.7 87.5 100.06 judge-resolved 83.3% 80.0%fell for 3 of 15 4.8 / 4.5agreement: high 41.6ROCm 7.1 28.4 3.8 Q4_K_M mit
119–162 Granite4 1B H 66.861.4–72.1 37.7 48.6 52.0 59.9 85.1 76.7 55.6 43.3 80.0 100.06 judge-resolved 75.0% 53.3%fell for 7 of 15 4.9 / 3.6agreement: high 50.1ROCm 7.1 3.4 1 Q8_0 apache-2.0
108–178 Qwen2 Math 7B Instructmath model 66.060.5–71.3 — 71.4 76.0 39.1 68.7 66.7 33.3 77.5 90.0 70.86 judge-resolved 75.0% 60.0%fell for 6 of 15 4.6 / 4.0agreement: high 19.4ROCm 7.1 4.7 7 Q4_K_M apache-2.0
120–170 Exaone Deep 7.8B 65.760.0–71.5 39.8 85.7 88.0 55.4 16.9 86.7 61.1 50.8 80.0 66.76 judge-resolved 91.7% 93.3%fell for 1 of 15 5.0 / 3.3agreement: medium 28.9ROCm 7.1 22.3 7.8 Q4_K_M exaone-nc
125–167 Olmo2 7B 1124 Instruct 65.460.6–70.1 23.5 51.4 52.0 29.3 84.4 93.3 27.8 60.0 90.0 100.06 judge-resolved 83.3% 46.7%fell for 8 of 15 4.9 / 4.9agreement: high 16.9ROCm 7.1 21.6 7 Q4_K_M apache-2.0
113–181 Falcon3 3B Instruct 64.859.4–70.6 — 51.4 52.0 40.9 76.8 73.3 66.7 50.8 80.0 91.76 judge-resolved 66.7% 66.7%fell for 5 of 15 5.3 / 4.9agreement: high 60.3ROCm 7.1 2.0 3 Q4_K_M other
132–172 Granite3.2 2B Instruct 63.858.7–69.0 23.9 42.9 52.0 48.4 75.5 80.0 50.0 43.3 95.0 87.56 judge-resolved 75.0% 46.7%fell for 8 of 15 4.8 / 4.9agreement: high 32.1ROCm 7.1 12.6 2 Q4_K_M apache-2.0
125–182 Apertus-8B-Instruct-2509 63.658.2–68.8 — 42.9 32.0 33.0 76.3 83.3 50.0 75.0 80.0 100.06 judge-resolved 83.3% 46.7%fell for 8 of 15 4.9 / 4.3agreement: high 32.9Vulkan 22.3 8.1 Q4_K_M apache-2.0
129–182 Phi-3-mini-4k-instruct 63.458.3–68.1 20.9 77.1 56.0 35.1 70.7 96.7 11.1 52.5 80.0 91.76 judge-resolved 66.7% 60.0%fell for 6 of 15 5.5 / 5.6agreement: high 44.5ROCm 7.1 8.9 3.8 Q4_K_M mit
137–176 Phi3 3.8B Mini 128K Instruct 62.958.0–67.7 21.9 54.3 40.0 38.6 60.2 96.7 50.0 31.7 95.0 100.06 judge-resolved 91.7% 53.3%fell for 7 of 15 4.5 / 4.5agreement: high 96.1ROCm 7.1 54.1 3.8 Q4_K_M mit
136–187 Codellama 34B Instructcode model 61.256.0–66.3 21.9 42.9 16.0 38.7 75.1 73.3 66.7 50.0 92.5 95.86 judge-resolved 41.7% 33.3%fell for 10 of 15 4.5 / 3.6agreement: high 7.4ROCm 7.1 46.7 34 Q4_K_M —
147–183 Gemma 7B Instruct V1.1 60.855.5–66.1 28.5 40.0 24.0 26.5 76.0 83.3 61.1 56.7 80.0 100.06 judge-resolved 91.7% 60.0%fell for 6 of 15 5.1 / 4.6agreement: high 15.2ROCm 7.1 36.1 7 Q4_K_M gemma
147–184 Granite3.1 Dense 2B Instruct 60.655.0–66.1 27.0 45.7 36.0 46.6 73.6 76.7 50.0 45.0 80.0 91.76 judge-resolved 58.3% 40.0%fell for 9 of 15 4.5 / 4.8agreement: high 56.8ROCm 7.1 12.6 2 Q4_K_M apache-2.0
147–185 Qwen2.5 Coder 1.5B Instructcode model 60.554.9–66.0 28.6 65.7 44.0 52.8 58.8 60.0 66.7 30.8 82.5 83.36 judge-resolved 66.7% 53.3%fell for 7 of 15 4.5 / 3.3agreement: medium 77.0ROCm 7.1 5.1 1.5 Q4_K_M apache-2.0
148–186 Mistral 7B Instruct V0.3 59.854.7–64.7 24.9 37.1 16.0 40.9 70.3 93.3 61.1 41.7 90.0 87.56 judge-resolved 66.7% 53.3%fell for 7 of 15 5.4 / 5.1agreement: high 17.5ROCm 7.1 22.0 7 Q4_K_M apache-2.0
148–186 Granite3.1 Moe 3B Instruct 59.854.7–65.0 24.9 37.1 32.0 32.6 69.2 83.3 61.1 41.7 85.0 95.86 judge-resolved 58.3% 66.7%fell for 5 of 15 4.3 / 4.6agreement: high 93.0ROCm 7.1 10.9 3 Q4_K_M apache-2.0
150–186 Codegemma 7B Instruct V1.1code model 59.554.8–64.2 29.3 45.7 24.0 55.5 78.1 70.0 5.6 70.0 95.0 91.76 judge-resolved 75.0% 46.7%fell for 8 of 15 5.0 / 5.0agreement: high 52.6ROCm 7.1 36.1 7 Q4_K_M gemma
143–191 Mistral-7B-Instruct-v0.2 59.254.3–64.2 18.7 40.0 12.0 25.3 75.1 90.0 72.2 42.5 80.0 95.86 judge-resolved 91.7% 60.0%fell for 6 of 15 4.8 / 4.9agreement: high 29.0ROCm 7.1 22.0 7.2 Q4_K_M apache-2.0
150–186 YI Coder 9B Chatcode model 59.053.7–64.5 37.8 31.4 16.0 81.3 65.3 83.3 44.4 54.2 95.0 60.46 judge-resolved 75.0% 53.3%fell for 7 of 15 3.8 / 3.9agreement: high 24.6ROCm 7.1 18.5 9 Q4_K_M apache-2.0
151–186 Solar 10.7B Instruct V1 58.753.7–63.8 26.4 45.7 16.0 34.3 69.9 86.7 38.9 58.3 95.0 83.36 judge-resolved 83.3% 33.3%fell for 10 of 15 5.8 / 5.1agreement: high 13.6ROCm 7.1 13.1 10.7 Q4_K_M —
152–187 Granite3 Dense 2B Instruct 58.353.1–63.6 20.2 48.6 24.0 42.2 63.4 86.7 33.3 48.3 82.5 95.86 judge-resolved 75.0% 40.0%fell for 9 of 15 4.4 / 4.4agreement: high 43.5ROCm 7.1 4.4 2 Q4_K_M apache-2.0
152–188 Deepseek V2 16B Lite Chat 58.253.2–63.4 20.1 34.3 40.0 41.6 70.2 70.0 33.3 50.8 87.5 95.86 judge-resolved 58.3% 40.0%fell for 9 of 15 5.3 / 4.9agreement: high 5.9ROCm 7.1 46.8 16 Q4_K_M other
152–187 YI 6B Chat V1.5 58.052.8–63.5 21.0 45.7 48.0 19.8 70.4 73.3 33.3 51.7 92.5 87.56 judge-resolved 75.0% 60.0%fell for 6 of 15 4.9 / 4.9agreement: medium 120.2ROCm 7.1 5.9 6 Q4_K_M apache-2.0
150–187 Llama3.2 1B Instruct 58.052.3–63.7 26.6 37.1 40.0 41.7 76.9 76.7 38.9 55.0 72.5 83.36 judge-resolved 50.0% 13.3%fell for 13 of 15 5.1 / 4.0agreement: high 114.5ROCm 7.1 5.4 1 Q4_K_M llama3.2
154–188 Gemma3 1B IT 56.851.8–62.2 29.3 60.0 64.0 42.5 71.9 63.3 22.2 52.5 35.0 100.06 judge-resolved 58.3% 53.3%fell for 7 of 15 5.4 / 4.9agreement: high 80.1ROCm 7.1 1.5 1 Q4_K_M gemma
150–194 Mixtral 8x7B Instruct V0.1 56.851.3–62.0 17.5 42.9 24.0 20.1 62.6 83.3 72.2 41.7 85.0 79.26 judge-resolved 83.3% 73.3%fell for 4 of 15 4.6 / 5.1agreement: high 8.5ROCm 7.1 46.5 56 Q4_K_M —
155–189 Aya 8B 23 55.951.1–60.7 24.9 34.3 0 30.7 74.7 76.7 61.1 39.2 95.0 91.76 judge-resolved 66.7% 40.0%fell for 9 of 15 4.6 / 4.8agreement: high 16.4ROCm 7.1 14.2 8 F16 cc-by-nc-4.0
154–195 Qwen 14B Chat V1.5 54.849.8–59.4 — 48.6 40.0 9.7 76.9 93.3 11.1 52.5 65.0 95.86 judge-resolved 66.7% 60.0%fell for 6 of 15 4.8 / 4.4agreement: high 13.9ROCm 7.1 62.6 14 Q4_K_M other
156–196 Exaone Deep 2.4B 54.748.9–60.5 23.0 82.9 72.0 36.7 13.9 63.3 50.0 38.3 72.5 62.56 judge-resolved 75.0% 93.3%fell for 1 of 15 3.4 / 2.3agreement: high 65.1ROCm 7.1 11.9 2.4 Q4_K_M exaone-nc
158–194 Deepseek Llm 7B Chat 54.349.3–59.8 20.9 37.1 24.0 37.4 66.8 76.7 27.8 45.8 77.5 95.86 judge-resolved 25.0% 40.0%fell for 9 of 15 4.4 / 3.6agreement: high 24.8ROCm 7.1 20.3 7 Q4_K_M —
161–198 Qwen3 VL 2B Thinking 53.748.3–59.1 26.7 60.0 52.0 0 77.2 73.3 33.3 63.3 82.5 41.74 judge-resolved 50.0% 53.3%fell for 7 of 15 3.4 / 3.1agreement: high 26.4ROCm 7.1 3.6 2 Q4_K_M apache-2.0
159–202 Codellama 7B Instructcode model 53.047.8–57.8 — 37.1 4.0 36.0 68.1 76.7 61.1 21.7 85.0 87.56 judge-resolved 50.0% 40.0%fell for 9 of 15 4.6 / 4.4agreement: high 27.2ROCm 7.1 38.8 7 Q4_K_M llama2
159–203 Codellama 13B Instructcode model 52.547.2–57.8 — 31.4 12.0 22.5 73.8 83.3 50.0 33.3 82.5 83.36 judge-resolved 50.0% 66.7%fell for 5 of 15 4.6 / 3.8agreement: high 16.2ROCm 7.1 61.7 13 Q4_K_M llama2
171–198 Nemotron Mini 4B Instruct 52.047.4–56.9 22.4 34.3 12.0 34.6 72.4 83.3 11.1 47.5 85.0 87.56 judge-resolved 50.0% 46.7%fell for 8 of 15 4.4 / 3.9agreement: high 167.6ROCm 7.1 6.6 4 Q4_K_M other
159–207 Internlm2 1.8B Chat V2.5 51.946.1–57.5 — 31.4 28.0 48.5 56.6 70.0 50.0 29.2 70.0 83.36 judge-resolved 41.7% 33.3%fell for 10 of 15 4.1 / 2.9agreement: medium 85.6ROCm 7.1 1.2 1.8 Q4_K_M other
162–207 Magistral Small (2506) 49.844.5–55.3 — 34.3 48.0 10.0 34.8 56.7 44.4 70.0 100.0 50.03 judge-resolved 33.3% 26.7%fell for 11 of 15 4.1 / 3.6agreement: high 10.5ROCm 7.1 14.3 24 Q4_K_M apache-2.0
175–203 Qwen2 1.5B Instruct 49.744.1–55.3 20.1 28.6 32.0 30.6 61.6 63.3 27.8 48.3 80.0 75.06 judge-resolved 66.7% 60.0%fell for 6 of 15 4.1 / 3.6agreement: high 49.8ROCm 7.1 — 1.5 Q4_K_M apache-2.0
176–204 Codeqwen 7B Chat V1.5code model 49.444.2–54.7 34.2 25.7 8.0 69.4 72.1 40.0 38.9 35.0 95.0 60.46 judge-resolved 66.7% 40.0%fell for 9 of 15 4.0 / 2.8agreement: high 25.2ROCm 7.1 13.6 7 Q4_K_M —
177–205 Deepseek R1 1.5B Qwen Distill 48.943.5–54.7 25.5 71.4 76.0 24.4 66.2 53.3 22.2 47.5 62.5 16.73 judge-resolved 75.0% 73.3%fell for 4 of 15 3.9 / 3.0agreement: high 50.6ROCm 7.1 5.1 1.5 Q4_K_M mit
179–206 Granite3 Moe 3B Instruct 47.742.3–53.1 20.2 37.1 20.0 31.6 67.2 76.7 27.8 36.7 45.0 87.56 judge-resolved 50.0% 53.3%fell for 7 of 15 4.3 / 3.5agreement: high 218.4ROCm 7.1 4.3 3 Q4_K_M apache-2.0
180–206 Smollm2 1.7B Instruct 47.141.7–52.7 26.0 28.6 8.0 35.3 79.5 63.3 33.3 35.0 70.0 70.86 judge-resolved 58.3% 40.0%fell for 9 of 15 4.8 / 3.6agreement: high 53.3ROCm 7.1 14.2 1.7 Q4_K_M apache-2.0
184–208 Granite4 350M H 45.640.7–50.7 25.2 20.0 0 36.0 79.3 76.7 16.7 43.3 80.0 58.36 judge-resolved 41.7% 20.0%fell for 12 of 15 3.6 / 2.4agreement: medium 128.2ROCm 7.1 1.2 0.4 Q8_0 apache-2.0
181–211 Llama2 13B Chat 45.340.2–50.6 — 45.7 8.0 10.1 47.8 73.3 27.8 32.5 75.0 87.56 judge-resolved 66.7% 53.3%fell for 7 of 15 5.6 / 4.8agreement: high 16.4ROCm 7.1 21.5 13 Q4_K_M llama2
180–212 Falcon3 1B Instruct 44.339.8–49.3 — 40.0 20.0 18.7 64.9 70.0 11.1 34.2 40.0 100.06 judge-resolved 66.7% 33.3%fell for 10 of 15 3.8 / 2.8agreement: high 107.4ROCm 7.1 1.1 1 Q4_K_M other
183–211 Qwen 7B Chat V1.5 44.139.1–48.9 — 45.7 16.0 5.9 71.9 76.7 11.1 45.0 35.0 89.66 judge-resolved 75.0% 66.7%fell for 5 of 15 4.3 / 3.9agreement: high 23.2ROCm 7.1 39.2 7 Q4_K_M other
185–210 Qwen3 0.6B 44.038.8–49.4 25.0 37.1 40.0 32.5 74.3 20.0 11.1 48.3 82.5 50.06 judge-resolved 41.7% 33.3%fell for 10 of 15 3.8 / 2.9agreement: high 261.1ROCm 7.1 15.6 0.6 Q4_K_M apache-2.0
182–212 Llama-2-7b-chat-hf 43.839.0–48.8 16.4 25.7 4.0 18.6 72.1 66.7 16.7 33.3 65.0 91.76 judge-resolved 66.7% 33.3%fell for 10 of 15 4.5 / 4.0agreement: high 26.7ROCm 7.1 12.8 6.7 Q4_K_M llama2
185–212 Granite3.1 Moe 1B Instruct 43.637.9–49.1 19.0 31.4 12.0 38.1 59.0 63.3 33.3 24.2 62.5 68.86 judge-resolved 58.3% 40.0%fell for 9 of 15 4.3 / 3.5agreement: high 149.5ROCm 7.1 7.5 1 Q4_K_M apache-2.0
186–212 Qwen2.5 0.5B Instruct 43.337.4–49.4 18.2 28.6 36.0 26.5 54.9 56.7 38.9 23.3 62.5 62.56 judge-resolved 58.3% 40.0%fell for 9 of 15 3.5 / 2.5agreement: high 116.8ROCm 7.1 2.2 0.5 Q4_K_M apache-2.0
190–213 Stablelm2 12B Chat 42.438.1–46.7 14.7 22.9 8.0 45.0 47.0 70.0 5.6 20.0 67.5 95.86 judge-resolved 50.0% 33.3%fell for 10 of 15 4.4 / 3.9agreement: high 6.7ROCm 7.1 14.0 12 Q4_K_M stability-nc
188–213 Gemma 2B Instruct V1.1 41.936.8–47.5 22.5 25.7 20.0 25.2 66.6 60.0 27.8 37.5 35.0 79.26 judge-resolved 50.0% 26.7%fell for 11 of 15 4.0 / 4.0agreement: high 43.4ROCm 7.1 3.3 2 Q4_K_M gemma
193–215 Granite Code 8B Instructcode model 40.735.3–46.7 20.6 40.0 28.0 55.7 53.7 63.3 27.8 41.7 10.0 45.86 judge-resolved 58.3% 66.7%fell for 5 of 15 4.6 / 3.4agreement: high 15.0ROCm 7.1 10.0 8 Q4_K_M apache-2.0
195–214 Phi 2.7B Chat V2 39.935.5–44.2 17.5 34.3 0 34.7 60.3 83.3 0 18.3 42.5 85.46 judge-resolved 33.3% 40.0%fell for 9 of 15 3.6 / 2.6agreement: high 38.2ROCm 7.1 7.2 2.7 Q4_K_M mit
197–216 Granite Code 3B Instructcode model 38.032.7–43.5 25.6 37.1 24.0 45.6 54.2 56.7 16.7 42.5 15.0 50.06 judge-resolved 58.3% 40.0%fell for 9 of 15 3.9 / 2.4agreement: medium 107.3ROCm 7.1 7.7 3 Q4_K_M apache-2.0
201–216 Qwen2 0.5B Instruct 36.030.9–41.2 15.6 31.4 8.0 9.8 50.2 43.3 22.2 23.3 77.5 58.36 judge-resolved 41.7% 26.7%fell for 11 of 15 3.9 / 2.6agreement: high 137.5ROCm 7.1 2.2 0.5 Q4_K_M apache-2.0
203–217 Granite3 Moe 1B Instruct 35.531.0–40.5 18.7 17.1 8.0 27.6 59.1 36.7 11.1 20.8 60.0 79.26 judge-resolved 58.3% 33.3%fell for 10 of 15 3.6 / 3.3agreement: high 82.4ROCm 7.1 2.3 1 Q4_K_M apache-2.0
198–219 Deepseek Coder 6.7B Instructcode model 34.729.7–40.0 — 8.6 0 68.5 53.9 10.0 61.1 21.7 55.0 33.36 judge-resolved 50.0% 33.3%fell for 10 of 15 1.3 / 1.0agreement: high 22.5ROCm 7.1 38.8 6.7 Q4_K_M other
203–221 Deepseek Coder 33B Instructcode model 33.429.0–38.2 31.0 31.4 12.0 67.2 46.2 6.7 0 25.8 70.0 41.76 judge-resolved 58.3% 53.3%fell for 7 of 15 1.1 / 1.4agreement: high 5.9ROCm 7.1 53.5 33 Q4_K_M —
203–222 Qwen2 Math 1.5B Instructmath model 33.128.0–38.2 — 65.7 72.0 17.5 14.4 33.3 11.1 37.5 17.5 29.26 judge-resolved 83.3% 40.0%fell for 9 of 15 3.0 / 1.1agreement: medium 62.0ROCm 7.1 1.0 1.5 Q4_K_M apache-2.0
208–218 Qwen2.5 Coder 0.5B Instructcode model 32.827.5–38.2 22.0 22.9 8.0 35.9 49.5 20.0 33.3 31.7 65.0 29.26 judge-resolved 50.0% 33.3%fell for 10 of 15 3.1 / 1.9agreement: high 172.5ROCm 7.1 2.2 0.5 Q4_K_M apache-2.0
209–219 Opencoder 1.5B Instructcode model 31.927.6–36.4 21.5 2.9 0 57.3 50.8 20.0 11.1 24.2 77.5 43.86 judge-resolved 58.3% 26.7%fell for 11 of 15 3.3 / 1.3agreement: medium 16.2ROCm 7.1 8.4 1.5 Q4_K_M other
211–219 Qwen 4B Chat V1.5 30.226.0–34.5 15.1 11.4 4.0 3.1 60.4 73.3 0 37.5 15.0 66.76 judge-resolved 50.0% 20.0%fell for 12 of 15 3.6 / 2.0agreement: medium 72.2ROCm 7.1 56.1 4 Q4_K_M tongyi-qianwen-research
212–223 Smollm2 360M Instruct 28.123.4–32.9 14.1 8.6 8.0 23.8 59.9 46.7 11.1 21.7 25.0 47.96 judge-resolved 41.7% 46.7%fell for 8 of 15 4.5 / 2.9agreement: medium 91.2ROCm 7.1 3.1 0.4 Q4_K_M apache-2.0
213–224 YI Coder 1.5B Chatcode model 26.522.1–31.5 21.7 17.1 0 57.7 36.1 53.3 5.6 10.8 35.0 22.96 judge-resolved 41.7% 33.3%fell for 10 of 15 3.3 / 1.6agreement: medium 88.7ROCm 7.1 26.9 1.5 Q4_K_M apache-2.0
216–225 Qwen 1.8B Chat V1.5 24.720.2–29.3 12.6 20.0 4.0 2.4 38.4 60.0 0 27.5 20.0 50.06 judge-resolved 66.7% 40.0%fell for 9 of 15 3.3 / 2.1agreement: high 82.5ROCm 7.1 27.1 1.8 Q4_K_M tongyi-qianwen-research
216–225 Stablelm2 1.6B Chat 24.420.2–29.0 10.6 8.6 4.0 26.1 31.6 40.0 0 16.7 42.5 50.06 judge-resolved 41.7% 13.3%fell for 13 of 15 3.6 / 2.6agreement: high 62.2ROCm 7.1 7.5 1.6 Q4_K_M stability-nc
219–225 Stable Code 3B Instructcode model 22.818.4–27.4 17.4 22.9 0 6.5 55.8 33.3 5.6 26.7 15.0 39.66 judge-resolved 41.7% 40.0%fell for 9 of 15 2.6 / 1.1agreement: medium 38.6ROCm 7.1 44.9 2.8 Q4_K_M stability-community
218–225 Deepseek Coder 1.3B Instructcode model 22.618.3–27.2 21.9 5.7 0 54.5 40.5 3.3 11.1 23.3 40.0 25.06 judge-resolved 33.3% 6.7%fell for 14 of 15 0.9 / 0.6agreement: high 64.1ROCm 7.1 26.9 1.3 Q4_K_M other
220–227 Smollm2 135M Instruct 20.116.2–24.6 11.2 2.9 0 14.1 36.1 36.7 5.6 15.0 25.0 45.86 judge-resolved 50.0% 6.7%fell for 14 of 15 3.0 / 1.5agreement: medium 201.2ROCm 7.1 1.7 0.1 Q4_K_M apache-2.0
225–227 Qwen 0.5B Chat V1.5 16.112.5–20.3 15.2 14.3 0 9.0 40.8 13.3 0 30.0 15.0 22.96 judge-resolved 33.3% 26.7%fell for 11 of 15 3.1 / 1.4agreement: medium 98.9ROCm 7.1 13.4 0.5 Q4_K_M tongyi-qianwen-research
225–227 Codegemma 2B Code V1.1code model 15.311.7–19.1 9.2 17.1 0 3.9 29.5 16.7 0 16.7 10.0 43.86 judge-resolved 33.3% 6.7%fell for 14 of 15 0.9 / 0.3agreement: high 43.3ROCm 7.1 3.3 2 Q4_K_M gemma
— Apertus-8B-Instruct-2509 —withheld — — — — — — — — — — — — 8.7ROCm 7.1 22.3 8.1 Q4_K_M apache-2.0
— Falcon2 11B —unranked: long_context partial: 12/18 items scored, 80% must be scored 22.9 16.0 16.9 57.7 70.0 0partial: 12/18 22.5 20.0 87.56 judge-resolved 33.3% 33.3%fell for 10 of 15 4.0 / 2.4agreement: medium 19.7ROCm 7.1 10.9 11 Q4_K_M other
— Falcon3 10B Instruct —withheld — — — — — — — — — — — — 21.2ROCm 7.1 28.3 10 Q4_K_M other
— Falcon3 1B Instruct —withheld — — — — — — — — — — — — 87.3ROCm 7.1 6.0 1 Q4_K_M other
— Falcon3 3B Instruct —withheld — — — — — — — — — — — — 57.9ROCm 7.1 14.0 3 Q4_K_M other
— Falcon3 7B Instruct —withheld — — — — — — — — — — — — 24.7ROCm 7.1 20.1 7 Q4_K_M other
— Internlm2 1.8B Chat V2.5 —withheld — — — — — — — — — — — — 73.2ROCm 7.1 14.3 1.8 Q4_K_M other
— Qwen2 Math 1.5B Instructmath model —withheld — — — — — — — — — — — — 61.1ROCm 7.1 2.1 1.5 Q4_K_M apache-2.0
— Qwen2 Math 7B Instructmath model —withheld — — — — — — — — — — — — 17.4ROCm 7.1 6.5 7 Q4_K_M apache-2.0

Withheld

These models have no rank and no score on this page. Their answers are published on their own pages, except for a model whose every tested version is withheld: it has no published results, and its page gives the reason only.