Open-weight model leaderboard

Which open-weight model is best for the machine you have? Each one is run locally through the same fixed battery, then scored by code wherever an answer can be checked.

228 models (236 versions listed, withheld ones included) on a battery of 270 items in 13 categories: 9 of them make up Overall (reasoning, math, coding, instruction following, knowledge & honesty, long context, multilingual, tool use, safety); the other 4 (pushback resistance, famous gotchas, writing, self-description) are shown on their own. Each model also gets a speed and memory profile on one laptop (Speed, hardware picker, best-model pages), from 12 runs (newest r20261105-bv1-2abe78). Read the methodology, including its limitations.

How much model does your memory buy?

Each dot is a model. The line is the best score reachable at each memory size. Memory is an estimate: the model file size plus about 20% for the context of a single conversation.

Coding score against the memory a model needs One dot per model. Horizontal axis: estimated memory need in GB, logarithmic scale. Vertical axis: Coding score (0 to 100). The orange line joins the 18 models that no model needing the same or less memory beats; only those are named, and every dot has a tooltip and a link to its model page. 0 20 40 60 80 100 0.0625 0.125 0.25 0.5 1 2 4 8 16 32 64 128 Estimated memory needed (GB, log scale) Coding (0 to 100) gpt-oss-120b: Coding 99.9, needs about 78.4 GB (on the frontier)gpt-oss-120b Gemma 4 12B IT: Coding 98.2, needs about 8.9 GB (on the frontier)Gemma 4 12B IT Gemma4 E4B IT: Coding 89.9, needs about 7.9 GB (on the frontier)Gemma4 E4B IT Granite4.2 8B: Coding 85.1, needs about 6.4 GB (on the frontier)Granite4.2 8B YI Coder 9B Chat: Coding 81.3, needs about 6.4 GB (on the frontier) Granite4 Tiny H: Coding 80.8, needs about 5.0 GB (on the frontier)Granite4 Tiny H Qwen3.5-4B: Coding 71.5, needs about 3.2 GB (on the frontier)Qwen3.5-4B Qwen3-4B-Instruct-2507: Coding 69.7, needs about 3.0 GB (on the frontier) Granite4 Micro H: Coding 68.8, needs about 2.3 GB (on the frontier) Exaone3.5 2.4B Instruct: Coding 64.7, needs about 1.9 GB (on the frontier)Exaone3.5 2.4B Instruct Lfm3 1.7B: Coding 61.5, needs about 1.3 GB (on the frontier) YI Coder 1.5B Chat: Coding 57.7, needs about 1.2 GB (on the frontier)YI Coder 1.5B Chat Deepseek Coder 1.3B Instruct: Coding 54.5, needs about 1.0 GB (on the frontier) Gemma3 1B IT: Coding 42.5, needs about 1.0 GB (on the frontier)Gemma3 1B IT Lfm2.5 Thinking 1.2B: Coding 42.1, needs about 0.9 GB (on the frontier) Granite4 350M H: Coding 36.0, needs about 0.4 GB (on the frontier)Granite4 350M H Smollm2 360M Instruct: Coding 23.8, needs about 0.3 GB (on the frontier)Smollm2 360M Instruct Smollm2 135M Instruct: Coding 14.1, needs about 0.1 GB (on the frontier)Smollm2 135M Instruct Qwen3-Coder-Next: Coding 97.3, needs about 62.1 GB Qwen3.8-27B: Coding 96.9, needs about 20.2 GB gemma-4-31B-it: Coding 96.6, needs about 22.4 GB gemma-4-26B-A4B-it: Coding 96.6, needs about 21.6 GB gpt-oss-20b: Coding 95.1, needs about 16.6 GB Qwen3.6-35B-A3B: Coding 94.1, needs about 28.7 GB Flint 40B A2B: Coding 93.4, needs about 27.6 GB Qwen3-30B-A3B-Instruct-2507: Coding 93.2, needs about 22.3 GB Qwen3.5-27B: Coding 93.1, needs about 19.9 GB Muse Glimmer 30B: Coding 92.4, needs about 21.8 GB Qwen3.6-27B: Coding 92.0, needs about 20.9 GB Qwen3-Coder-30B-A3B-Instruct: Coding 92.0, needs about 22.3 GB Nanbeige_Nanbeige4.3-33B: Coding 90.1, needs about 24.0 GB Nemotron 3.5 Lightning 30B A3B: Coding 89.0, needs about 30.5 GB Laguna XS 2.1: Coding 88.5, needs about 24.3 GB Granite4.2 30B: Coding 88.0, needs about 21.3 GB Qwen3-VL-30B-A3B-Instruct: Coding 87.8, needs about 23.5 GB Ministral 3 14B Instruct (2512): Coding 87.5, needs about 10.9 GB GLM-4.5-Air: Coding 87.5, needs about 87.6 GB Qwen3.5-35B-A3B: Coding 87.1, needs about 25.4 GB phi-4: Coding 87.0, needs about 10.9 GB Qwen2.5-VL-32B-Instruct: Coding 86.6, needs about 25.4 GB Qwen3-30B-A3B: Coding 86.6, needs about 22.3 GB Qwen2.5-Coder-32B-Instruct: Coding 86.5, needs about 23.8 GB Qwen3-VL-32B-Instruct: Coding 86.5, needs about 25.1 GB Granite4 Small H: Coding 86.3, needs about 23.4 GB NVIDIA Nemotron 3 Nano 30B-A3B: Coding 86.0, needs about 29.1 GB Granite4.1 30B: Coding 86.0, needs about 21.0 GB gemma-3-27b-it: Coding 85.4, needs about 20.9 GB Olmo 3.1 32B Instruct: Coding 84.7, needs about 23.4 GB Sage-A1.5-Q4_K_M: Coding 84.6, needs about 26.4 GB Qwen3-32B: Coding 83.8, needs about 24.2 GB Nemotron3 33B: Coding 83.7, needs about 33.2 GB Devstral Small 2 24B Instruct (2512): Coding 83.4, needs about 18.2 GB Mistral Small 24B Instruct 2501: Coding 82.8, needs about 17.2 GB Qwen2.5-Coder-14B-Instruct: Coding 82.7, needs about 10.8 GB Qwen2.5-32B-Instruct: Coding 82.0, needs about 23.8 GB gemma-3-12b-it: Coding 81.8, needs about 9.8 GB GLM-4.7-Flash: Coding 81.6, needs about 22.8 GB Llama 4 Scout 17B-16E Instruct: Coding 81.0, needs about 80.9 GB Llama-3.3-70B-Instruct: Coding 80.7, needs about 51.0 GB Gemma4 E2B IT: Coding 80.1, needs about 5.5 GB Ministral 3 8B Instruct 2512: Coding 78.4, needs about 7.2 GB Nemotron Cascade 2 30B A3B: Coding 77.9, needs about 29.1 GB Qwen3-VL-8B-Instruct: Coding 77.7, needs about 7.4 GB Qwen3.5-9B: Coding 77.4, needs about 6.8 GB Mistral Small 3.2 24B Instruct (2506): Coding 77.2, needs about 18.2 GB Qwen2.5-14B-Instruct: Coding 76.5, needs about 10.8 GB Granite4.1 8B: Coding 76.5, needs about 6.4 GB Qwen3-14B: Coding 73.5, needs about 11.1 GB Mistral Small3.1 24B Instruct 2503: Coding 73.3, needs about 18.6 GB Nemotron 3.5 Bolt E4B: Coding 73.2, needs about 8.6 GB Codestral 22B V0.1: Coding 72.9, needs about 16.0 GB Lfm2 24B: Coding 72.9, needs about 17.3 GB DeepSeek-Coder-V2-Lite-Instruct: Coding 72.5, needs about 12.4 GB Qwen2.5-Coder-7B-Instruct: Coding 71.8, needs about 5.6 GB Exaone3.5 32B Instruct: Coding 71.5, needs about 23.2 GB gemma-3-4b-it: Coding 71.4, needs about 4.0 GB Qwen3-8B: Coding 70.6, needs about 6.3 GB Devstral 24B Small 2505: Coding 70.4, needs about 17.2 GB Mistral Small 22B Instruct 2409: Coding 70.1, needs about 16.0 GB EXAONE-3.5-7.8B-Instruct: Coding 70.1, needs about 5.7 GB Codeqwen 7B Chat V1.5: Coding 69.4, needs about 5.6 GB Qwen3-4B: Coding 69.0, needs about 3.1 GB Deepseek Coder 6.7B Instruct: Coding 68.5, needs about 4.9 GB Gemma2 27B Instruct: Coding 67.5, needs about 20.0 GB Command R7B Arabic 7B 02 2025: Coding 67.4, needs about 6.1 GB Deepseek Coder 33B Instruct: Coding 67.2, needs about 23.9 GB Llama-3.1-8B-Instruct: Coding 65.6, needs about 5.9 GB Spark-X2.6-4B: Coding 65.6, needs about 3.0 GB Gemma 3n E4B IT: Coding 65.3, needs about 9.1 GB Aya Expanse 32B: Coding 64.7, needs about 23.8 GB Granite4 3B: Coding 64.7, needs about 2.5 GB Qwen2.5-VL-7B-Instruct: Coding 64.4, needs about 7.2 GB Falcon3 10B Instruct: Coding 64.3, needs about 7.6 GB Granite3.1 Dense 8B Instruct: Coding 64.2, needs about 6.0 GB Command R 35B 08 2024: Coding 63.9, needs about 23.8 GB Qwen2.5-7B-Instruct: Coding 63.2, needs about 5.6 GB Ornith 9B: Coding 63.0, needs about 6.7 GB granite-4.1-3b: Coding 62.6, needs about 2.5 GB Granite4.2 3B: Coding 61.6, needs about 2.7 GB Phi3 14B Medium 128K Instruct: Coding 61.4, needs about 10.3 GB Gemma3n E2B IT: Coding 61.4, needs about 6.7 GB Qwen3-VL-4B-Instruct: Coding 61.2, needs about 4.0 GB Phi-4-mini-instruct: Coding 61.0, needs about 3.0 GB Ministral 3 3B Instruct 2512: Coding 60.6, needs about 3.6 GB QwQ 32B: Coding 60.6, needs about 23.8 GB Codegeex4 9B All: Coding 60.5, needs about 7.6 GB medgemma-4b-it: Coding 60.3, needs about 4.0 GB Qwen2.5-Coder-3B-Instruct: Coding 60.2, needs about 2.3 GB Meta-Llama-3-8B-Instruct: Coding 60.0, needs about 5.9 GB Glm4 9B Chat: Coding 60.0, needs about 7.6 GB Granite4 1B H: Coding 59.9, needs about 1.9 GB Command R7B (12-2024): Coding 59.4, needs about 6.1 GB Granite3 Dense 8B Instruct: Coding 59.2, needs about 5.9 GB Falcon3 7B Instruct: Coding 58.7, needs about 5.5 GB Granite3.2 8B Instruct: Coding 58.5, needs about 5.9 GB Olmo-3-7B-Instruct: Coding 57.7, needs about 5.4 GB YI 9B Chat V1.5: Coding 57.5, needs about 6.4 GB Opencoder 1.5B Instruct: Coding 57.3, needs about 1.7 GB Mistral Nemo 12B Instruct 2407: Coding 57.3, needs about 9.0 GB Granite3.3 8B: Coding 56.6, needs about 5.9 GB Granite Code 8B Instruct: Coding 55.7, needs about 5.9 GB Codegemma 7B Instruct V1.1: Coding 55.5, needs about 6.4 GB Exaone Deep 7.8B: Coding 55.4, needs about 5.8 GB DeepSeek-R1-Distill-Qwen-32B: Coding 53.6, needs about 23.8 GB DeepSeek-R1-Distill-Qwen-14B: Coding 53.0, needs about 10.8 GB Llama-3.2-3B-Instruct: Coding 52.9, needs about 2.4 GB Deepseek R1 8B Llama Distill: Coding 52.8, needs about 5.9 GB Qwen2.5 Coder 1.5B Instruct: Coding 52.8, needs about 1.2 GB Qwen3.5 2B: Coding 52.5, needs about 2.3 GB gemma-2-9b-it: Coding 51.8, needs about 6.9 GB YI 34B Chat V1.5: Coding 50.8, needs about 24.8 GB Granite3.3 2B: Coding 50.4, needs about 1.8 GB Aya Expanse 8B: Coding 50.0, needs about 6.1 GB Qwen2.5 1.5B Instruct: Coding 49.9, needs about 1.2 GB Qwen3 VL 2B Instruct: Coding 49.2, needs about 2.3 GB Qwen2.5-3B-Instruct: Coding 48.7, needs about 2.3 GB Internlm2 7B Chat V2.5: Coding 48.7, needs about 5.6 GB Internlm2 1.8B Chat V2.5: Coding 48.5, needs about 1.4 GB Granite3.2 2B Instruct: Coding 48.4, needs about 1.8 GB Lfm2.5 8B A1B: Coding 47.2, needs about 6.2 GB Granite3.1 Dense 2B Instruct: Coding 46.6, needs about 1.9 GB Granite Code 3B Instruct: Coding 45.6, needs about 2.5 GB Qwen3 1.7B: Coding 45.5, needs about 1.7 GB Stablelm2 12B Chat: Coding 45.0, needs about 8.9 GB Qwen2.5-VL-3B-Instruct: Coding 42.6, needs about 3.8 GB Gemma2 2B Instruct: Coding 42.4, needs about 2.0 GB Hermes-3-Llama-3.1-8B: Coding 42.3, needs about 5.9 GB Lfm3 VL 2B: Coding 42.2, needs about 1.8 GB Granite3 Dense 2B Instruct: Coding 42.2, needs about 1.9 GB DeepSeek-R1-0528-Qwen3-8B: Coding 42.1, needs about 6.3 GB Llama3.2 1B Instruct: Coding 41.7, needs about 1.0 GB Deepseek V2 16B Lite Chat: Coding 41.6, needs about 12.0 GB Mistral 7B Instruct V0.3: Coding 40.9, needs about 5.3 GB Falcon3 3B Instruct: Coding 40.9, needs about 2.4 GB Olmo2 13B 1124 Instruct: Coding 40.8, needs about 10.1 GB Olmo-3-7B-Think: Coding 40.0, needs about 5.4 GB Qwen2 Math 7B Instruct: Coding 39.1, needs about 5.6 GB Phi-3.5-mini-instruct: Coding 38.9, needs about 2.9 GB Codellama 34B Instruct: Coding 38.7, needs about 24.3 GB Phi3 3.8B Mini 128K Instruct: Coding 38.6, needs about 2.9 GB Granite3.1 Moe 1B Instruct: Coding 38.1, needs about 1.0 GB Deepseek Llm 7B Chat: Coding 37.4, needs about 5.0 GB DeepSeek-R1-Distill-Qwen-7B: Coding 37.0, needs about 5.6 GB Exaone Deep 2.4B: Coding 36.7, needs about 1.9 GB Codellama 7B Instruct: Coding 36.0, needs about 4.9 GB Qwen2.5 Coder 0.5B Instruct: Coding 35.9, needs about 0.5 GB Smollm2 1.7B Instruct: Coding 35.3, needs about 1.3 GB Phi-3-mini-4k-instruct: Coding 35.1, needs about 2.9 GB Phi 2.7B Chat V2: Coding 34.7, needs about 2.2 GB Nemotron Mini 4B Instruct: Coding 34.6, needs about 3.2 GB Solar 10.7B Instruct V1: Coding 34.3, needs about 7.8 GB Granite3.1 Moe 3B Instruct: Coding 32.6, needs about 2.4 GB Qwen3 0.6B: Coding 32.5, needs about 0.6 GB Phi4 Mini Reasoning 3.8B: Coding 31.8, needs about 3.8 GB Granite3 Moe 3B Instruct: Coding 31.6, needs about 2.5 GB Aya 8B 23: Coding 30.7, needs about 6.1 GB Qwen2 1.5B Instruct: Coding 30.6, needs about 1.2 GB Olmo2 7B 1124 Instruct: Coding 29.3, needs about 5.4 GB Granite3 Moe 1B Instruct: Coding 27.6, needs about 1.0 GB Qwen3-4B-Thinking-2507: Coding 26.7, needs about 3.0 GB Qwen2.5 0.5B Instruct: Coding 26.5, needs about 0.5 GB Gemma 7B Instruct V1.1: Coding 26.5, needs about 6.4 GB Stablelm2 1.6B Chat: Coding 26.1, needs about 1.2 GB Mistral-7B-Instruct-v0.2: Coding 25.3, needs about 5.2 GB Gemma 2B Instruct V1.1: Coding 25.2, needs about 1.9 GB Deepseek R1 1.5B Qwen Distill: Coding 24.4, needs about 1.3 GB Codellama 13B Instruct: Coding 22.5, needs about 9.5 GB Mixtral 8x7B Instruct V0.1: Coding 20.1, needs about 34.1 GB YI 6B Chat V1.5: Coding 19.8, needs about 4.4 GB Falcon3 1B Instruct: Coding 18.7, needs about 1.3 GB Llama-2-7b-chat-hf: Coding 18.6, needs about 4.9 GB Qwen2 Math 1.5B Instruct: Coding 17.5, needs about 1.2 GB Falcon2 11B: Coding 16.9, needs about 8.2 GB Llama2 13B Chat: Coding 10.1, needs about 9.5 GB Magistral Small (2506): Coding 10.0, needs about 17.2 GB Qwen2 0.5B Instruct: Coding 9.8, needs about 0.5 GB Qwen 14B Chat V1.5: Coding 9.7, needs about 11.0 GB Qwen 0.5B Chat V1.5: Coding 9.0, needs about 0.5 GB Stable Code 3B Instruct: Coding 6.5, needs about 2.0 GB Qwen 7B Chat V1.5: Coding 5.9, needs about 5.8 GB Codegemma 2B Code V1.1: Coding 3.9, needs about 1.9 GB Qwen 4B Chat V1.5: Coding 3.1, needs about 3.0 GB Qwen 1.8B Chat V1.5: Coding 2.4, needs about 1.4 GB Qwen3 VL 2B Thinking: Coding 0.0, needs about 2.3 GB

best at each memory sizeother models. 204 of 217 model versions plotted; 13 left off (no score for this tab, or no file size on record).

1 model is waiting for a newer Ollama release that can load it (it crashes while loading in the version we use, on both GPU backends): Rnj 1 8B Instruct.

Originals only: 19 community fine-tunes left out of this table. Nobody is re-ranked, so a rank here is the rank in the full list. Show the fine-tunes too.

Compare two models

How to read this table

Overall is the unweighted mean of the 9 programmatic category scores marked overall in the battery definition (Reasoning, Math, Coding, Instruction following, Knowledge & honesty, Long context, Multilingual, Tool use, Safety), on a 0–100 scale. Models are ranked by Overall; a model that is missing any of those categories is listed unranked with the reason. Judge means are on their own 0–10 scale and are never blended into Overall. A dash means not scored, which is different from a score of 0.

Shown separately. Pushback is how often the model keeps a correct answer when the user pushes back, as a percentage. Gotchas is the share of famous trick questions answered correctly, with how many it fell for; these puzzles are widely published and probably memorised, so they do not count toward Overall. Writing has no programmatic answer: it shows the two judges' mean scores (0–10) and how closely they agree. The model's answers about itself are not scored; they are on the model page.

Speed is total tokens per second with 4 requests running at once and GPU memory on our server is what the benchmark server reported for the loaded model (1 to 8 answers at a time); it depends on that setup, is not what one user needs, and is not used to say what fits (the hardware picker estimates that from the model file). Speed and memory were measured on one laptop (see Speed). Size classes are by total parameters; the upper bound is inclusive, except Tiny, which is strictly under 3B.

Agreement (writing column, and on each model page) compares the two judges: high means a mean absolute score gap under 1.5, medium under 3, low otherwise; n/a means fewer than 3 items were scored by both judges.

Coverage rule. A category counts toward Overall only when at least 80% of its items have a definite programmatic score (for example 12 of 15; judge-resolved items included) and it has no pending item. A category below that shows its mean with a partial marker such as “partial: 1/15”, Overall shows a dash, and the model is listed unranked with the reason.

Judge-resolved and pending items. Where the refusal detector cannot call an item (for example a refusal followed by more text), the two judges decide: if they agree, the item is scored 1 or 0; if they split, it scores 0.5; if one judge is unavailable (unparsed or skipped), the other decides alone. Each of these is labelled judge-resolved. Until the judges have decided, the item is pending, and a model with any pending item is not ranked on safety. This is one of two places where judge scores enter the programmatic score; the other is the comply-quality gate, where an answer to a harmless prompt that both judges rate 3 or lower out of 10 is scored 0 unless it was cut off (how the gate works). Here, the detector cannot judge a refusal that is followed by more text, and dropping those items from the mean would let a model avoid a low Safety score by answering in a way the detector cannot classify. The cell shows a “judge-resolved” count and, when present, a “pending” count.

Integrity flag. A coding answer that touches the test harness is not run and scores 0; the model then carries an integrity flag next to its Coding score. Answers that use constructs outside the task are flagged on the model page. The score itself is unchanged by the site.

Programmatic scores (0–100), writing judge means (0–10), speed and memory per model, latest finished run. Click a column heading to sort.
Rank Model Overall Hard questions Reason Math Code Instr. Know. Long ctx Multi Tools Safety Pushback Gotchas Writing (judges) Speed (tok/s) GPU memory on our server (GB) Params (B) Quant Licence
1–5 gemma-4-31B-it 99.398.3–100.0 94.7 97.1 100.0 96.6 100.0 100.0 100.0 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 7.4 / 6.6agreement: medium 11.8ROCm 7.1 11.2 31.3 Q4_K_M apache-2.0
1–14 gpt-oss-120b 98.696.8–100.0 88.7 97.1 96.0 99.9 100.0 100.0 94.4 100.0 100.0 100.06 judge-resolved 91.7% 100.0%fell for 0 of 15 6.3 / 7.1agreement: high 15.8Vulkan 64.6 116.8 MXFP4 apache-2.0
1–12 Gemma 4 12B IT 98.697.0–99.8 90.7 94.3 100.0 98.2 100.0 100.0 94.4 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 7.4 / 6.1agreement: high 13.1ROCm 7.1 8.9 12 Q4_K_M apache-2.0
1–23 Muse Glimmer 30B 97.996.0–99.3 84.9 94.3 100.0 92.4 100.0 100.0 94.4 100.0 100.0 100.06 judge-resolved 100.0% 100.0%fell for 0 of 15 7.4 / 6.6agreement: high 8.8ROCm 7.1 18.3 30 Q4_K_M —
1–22 gemma-4-26B-A4B-it 97.895.8–99.3 92.0 94.3 100.0 96.6 100.0 100.0 88.9 100.0 100.0 100.06 judge-resolved 100.0% 100.0%fell for 0 of 15 6.9 / 6.8agreement: high 21.8ROCm 7.1 22.6 26 Q4_K_M apache-2.0
2–42 Qwen3-Coder-Nextcode model 96.494.2–98.3 66.6 88.6 92.0 97.3 95.2 100.0 94.4 100.0 100.0 100.06 judge-resolved 91.7% 100.0%fell for 0 of 15 6.8 / 5.8agreement: high 14.7Vulkan 52.2 79.7 Q4_K_M apache-2.0
3–35 Qwen3.5-27B 96.293.8–98.3 78.8 88.6 84.0 93.1 100.0 100.0 100.0 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 7.4 / 6.3agreement: medium 2.6ROCm 7.1 17.6 27.8 Q4_K_M apache-2.0
3–36 gemma-3-27b-it 95.893.7–97.5 76.0 91.4 96.0 85.4 94.5 96.7 100.0 100.0 100.0 97.96 judge-resolved 100.0% 93.3%fell for 1 of 15 7.5 / 6.3agreement: medium 4.7ROCm 7.1 22.1 27.4 Q4_K_M gemma
4–42 Qwen3-Coder-30B-A3B-Instructcode model 95.593.2–97.6 72.9 88.6 92.0 92.0 92.9 100.0 94.4 100.0 100.0 100.06 judge-resolved 83.3% 80.0%fell for 3 of 15 5.0 / 5.3agreement: high 26.1ROCm 7.1 31.7 30 Q4_K_M apache-2.0
2–51 Gemma4 E4B IT 95.593.0–97.7 84.7 85.7 96.0 89.9 97.2 100.0 94.4 100.0 100.0 95.86 judge-resolved 91.7% 93.3%fell for 1 of 15 5.8 / 6.1agreement: high 25.3ROCm 7.1 1.2 8 Q4_K_M apache-2.0
2–53 Qwen3.8-27B 95.492.9–97.8 72.5 88.6 80.0 96.9 98.9 100.0 94.4 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.1 / 6.1agreement: high 3.2Vulkan 17.6 27.8 Q4_K_M apache-2.0
3–58 Qwen3.6-27B 95.192.4–97.5 57.1 88.6 88.0 92.0 98.3 100.0 88.9 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.5 / 6.1agreement: high 3.5Vulkan 17.3 27.8 Q4_K_M apache-2.0
3–48 Qwen3-30B-A3B-Instruct-2507 95.092.5–97.2 73.2 88.6 96.0 93.2 90.3 100.0 88.9 98.3 100.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 6.8 / 5.8agreement: high 26.1ROCm 7.1 31.7 30 Q4_K_M apache-2.0
2–60 gpt-oss-20b 95.092.2–97.6 — 97.1 96.0 95.1 95.8 100.0 94.4 100.0 97.5 79.26 judge-resolved 100.0% 100.0%fell for 0 of 15 4.1 / 5.1agreement: high 31.5ROCm 7.1 13.5 20.9 MXFP4 apache-2.0
4–48 Qwen3-VL-8B-Instruct 94.992.4–97.1 66.1 91.4 92.0 77.7 98.5 100.0 94.4 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 5.8 / 5.1agreement: high 9.2ROCm 7.1 7.8 8.8 Q4_K_M apache-2.0
3–60 Qwen3.5-35B-A3B 94.892.1–97.1 41.8 85.7 92.0 87.1 93.5 100.0 100.0 100.0 95.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 7.0 / 6.6agreement: high 16.5Vulkan 21.7 35 Q4_K_M apache-2.0
4–60 Qwen3-VL-32B-Instruct 94.792.1–96.9 51.0 88.6 88.0 86.5 96.1 100.0 94.4 98.3 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 5.5 / 5.5agreement: medium 2.9Vulkan 24.5 33.4 Q4_K_M apache-2.0
3–63 Qwen3.6-35B-A3B 94.691.6–97.2 58.4 85.7 92.0 94.1 96.6 100.0 83.3 100.0 100.0 100.06 judge-resolved 91.7% 93.3%fell for 1 of 15 6.6 / 6.0agreement: high 17.8Vulkan 23.0 35 Q4_K_M apache-2.0
5–56 Sage-A1.5-Q4_K_M 94.591.8–96.7 64.1 88.6 88.0 84.6 94.6 100.0 94.4 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 7.1 / 6.1agreement: medium 21.2ROCm 7.1 21.8 36 Q4_K_M —
4–63 Llama-3.3-70B-Instruct 94.391.3–96.7 62.6 97.1 84.0 80.7 97.5 100.0 88.9 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.6 / 5.9agreement: high 3.4Vulkan 63.8 70.6 Q4_K_M llama3.3
5–61 Granite4 Small H 93.991.1–96.5 66.5 88.6 88.0 86.3 93.4 100.0 88.9 100.0 100.0 100.06 judge-resolved 100.0% 80.0%fell for 3 of 15 5.9 / 5.5agreement: high 12.4ROCm 7.1 23.9 32 Q4_K_M —
4–61 Gemma4 E2B IT 93.891.0–96.1 79.8 88.6 88.0 80.1 95.4 96.7 100.0 100.0 95.0 100.06 judge-resolved 91.7% 93.3%fell for 1 of 15 7.3 / 6.3agreement: medium 29.8ROCm 7.1 0.6 5 Q4_K_M apache-2.0
5–58 Qwen3-32B 93.690.9–96.3 64.7 85.7 88.0 83.8 96.3 100.0 88.9 100.0 100.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 6.1 / 6.3agreement: high 4.1ROCm 7.1 55.0 32.8 Q4_K_M apache-2.0
4–65 Llama 4 Scout 17B-16E Instruct 93.690.7–96.1 69.9 91.4 84.0 81.0 98.7 100.0 94.4 100.0 92.5 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 5.5 / 5.9agreement: medium 8.2Vulkan 70.4 109 Q4_K_M llama4
5–59 Qwen2.5-32B-Instruct 93.590.8–96.1 61.8 85.7 92.0 82.0 94.1 100.0 94.4 93.3 100.0 100.06 judge-resolved 100.0% 80.0%fell for 3 of 15 6.4 / 6.1agreement: medium 7.6ROCm 7.1 54.2 32.8 Q4_K_M apache-2.0
7–65 Qwen3-VL-30B-A3B-Instruct 92.990.1–95.5 43.9 82.9 88.0 87.8 88.4 100.0 88.9 100.0 100.0 100.06 judge-resolved 83.3% 80.0%fell for 3 of 15 5.6 / 5.9agreement: high 19.8Vulkan 20.6 30 Q4_K_M apache-2.0
8–60 Mistral Small 3.2 24B Instruct (2506) 92.990.3–95.2 61.7 82.9 96.0 77.2 85.4 100.0 94.4 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.8 / 6.3agreement: medium 9.4ROCm 7.1 36.5 24 Q4_K_M apache-2.0
5–64 Nemotron 3.5 Lightning 30B A3B 92.889.7–95.8 65.7 85.7 92.0 89.0 93.7 100.0 83.3 96.7 95.0 100.06 judge-resolved 91.7% 100.0%fell for 0 of 15 5.6 / 5.3agreement: high 17.3ROCm 7.1 25.4 30 Q4_K_M —
6–63 Granite4.2 30B 92.889.9–95.4 67.1 80.0 80.0 88.0 92.8 100.0 94.4 100.0 100.0 100.06 judge-resolved 91.7% 93.3%fell for 1 of 15 6.6 / 6.3agreement: high 8.2ROCm 7.1 52.3 30 Q4_K_M —
6–68 GLM-4.5-Air 92.889.7–95.6 — 82.9 96.0 87.5 94.3 100.0 83.3 95.0 100.0 95.86 judge-resolved 75.0% 93.3%fell for 1 of 15 6.4 / 6.0agreement: high 7.6Vulkan 76.8 110.5 Q4_K_M mit
8–65 Qwen3-30B-A3B 92.889.4–95.6 65.8 88.6 84.0 86.6 92.3 100.0 83.3 100.0 100.0 100.06 judge-resolved 91.7% 93.3%fell for 1 of 15 5.4 / 5.4agreement: high 25.5ROCm 7.1 31.7 30 Q4_K_M apache-2.0
8–66 gemma-3-12b-it 92.789.7–95.4 66.5 88.6 96.0 81.8 88.3 96.7 83.3 100.0 100.0 100.06 judge-resolved 91.7% 66.7%fell for 5 of 15 6.5 / 5.9agreement: high 9.5ROCm 7.1 13.1 12.2 Q4_K_M gemma
9–61 Ministral 3 14B Instruct (2512) 92.790.0–95.2 62.4 82.9 88.0 87.5 81.2 96.7 100.0 98.3 100.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 6.6 / 6.0agreement: high 15.0ROCm 7.1 30.0 14 Q4_K_M apache-2.0
6–65 QwQ 32B 92.689.8–95.4 — 94.3 88.0 60.6 95.1 100.0 100.0 100.0 100.0 95.86 judge-resolved 100.0% 100.0%fell for 0 of 15 6.3 / 5.8agreement: high 4.7ROCm 7.1 54.2 32.8 Q4_K_M apache-2.0
6–66 Granite4.1 30B 92.389.3–95.3 62.2 82.9 96.0 86.0 93.8 100.0 72.2 100.0 100.0 100.06 judge-resolved 91.7% 86.7%fell for 2 of 15 5.8 / 5.6agreement: high 5.0ROCm 7.1 52.3 30 Q4_K_M —
6–73 Granite4.1 8B 92.189.0–95.0 76.8 91.4 88.0 76.5 93.1 96.7 94.4 93.3 100.0 95.86 judge-resolved 91.7% 80.0%fell for 3 of 15 5.6 / 5.4agreement: high 15.9ROCm 7.1 27.1 8 Q4_K_M apache-2.0
6–65 Mistral Small 24B Instruct 2501 92.189.3–94.7 53.8 85.7 84.0 82.8 84.8 100.0 94.4 97.5 100.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 5.6 / 5.3agreement: high 10.6ROCm 7.1 36.5 24 Q4_K_M —
9–67 Devstral Small 2 24B Instruct (2512)code model 92.189.3–94.7 63.2 85.7 76.0 83.4 89.1 100.0 94.4 100.0 100.0 100.06 judge-resolved 100.0% 80.0%fell for 3 of 15 6.5 / 5.8agreement: high 8.9ROCm 7.1 36.5 24 Q4_K_M apache-2.0
11–69 Qwen3.5-9B 92.089.0–95.1 59.9 85.7 84.0 77.4 94.1 100.0 88.9 98.3 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.6 / 6.1agreement: medium 8.2ROCm 7.1 6.1 9.7 Q4_K_M apache-2.0
11–71 Qwen3-14B 91.788.5–94.7 52.9 91.4 88.0 73.5 93.6 100.0 88.9 90.0 100.0 100.06 judge-resolved 91.7% 80.0%fell for 3 of 15 6.5 / 6.0agreement: high 8.7ROCm 7.1 30.9 14.8 Q4_K_M apache-2.0
11–72 Qwen3-8B 91.588.4–94.5 55.5 94.3 88.0 70.6 87.7 100.0 83.3 100.0 100.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 5.8 / 5.3agreement: high 14.4ROCm 7.1 24.7 8.2 Q4_K_M apache-2.0
11–72 Qwen2.5-14B-Instruct 91.388.0–94.2 63.6 88.6 84.0 76.5 91.3 100.0 94.4 91.7 95.0 100.06 judge-resolved 91.7% 86.7%fell for 2 of 15 5.5 / 5.4agreement: high 59.0ROCm 7.1 34.8 14.8 Q4_K_M apache-2.0
6–73 Laguna XS 2.1 91.287.7–94.4 74.4 91.4 88.0 88.5 93.3 96.7 94.4 98.3 70.0 100.06 judge-resolved 91.7% 93.3%fell for 1 of 15 6.1 / 5.6agreement: high 40.0ROCm 7.1 23.2 — Q4_K_M —
12–74 Qwen2.5-Coder-32B-Instructcode model 90.587.3–93.8 64.9 80.0 64.0 86.5 94.8 100.0 94.4 95.0 100.0 100.06 judge-resolved 91.7% 80.0%fell for 3 of 15 5.4 / 5.8agreement: high 7.6ROCm 7.1 54.2 32.8 Q4_K_M apache-2.0
9–76 Nanbeige_Nanbeige4.3-33B 90.486.8–93.7 66.2 88.6 80.0 90.1 80.7 100.0 88.9 90.0 95.0 100.06 judge-resolved 100.0% 80.0%fell for 3 of 15 6.1 / 5.3agreement: high 7.4ROCm 7.1 48.9 33 Q4_K_M —
8–82 Nemotron 3.5 Bolt E4B 90.386.3–93.9 57.0 82.9 96.0 73.2 100.0 100.0 72.2 98.3 90.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.4 / 6.0agreement: high 13.0ROCm 7.1 5.7 8 Q4_K_M —
13–73 Qwen3-4B-Instruct-2507 90.387.2–93.3 57.2 82.9 80.0 69.7 94.2 96.7 88.9 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.6 / 5.8agreement: medium 43.2ROCm 7.1 12.5 4 Q4_K_M apache-2.0
12–77 Mistral Small3.1 24B Instruct 2503 90.186.9–93.1 49.6 88.6 76.0 73.3 84.2 100.0 88.9 100.0 100.0 100.06 judge-resolved 100.0% 66.7%fell for 5 of 15 5.6 / 5.5agreement: high 10.3ROCm 7.1 36.6 24 Q4_K_M —
12–79 Ornith 9B 90.086.5–93.1 67.9 91.4 84.0 63.0 91.9 100.0 94.4 85.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.5 / 5.9agreement: high 10.5ROCm 7.1 5.9 9 Q4_K_M mit
12–76 Qwen3-VL-4B-Instruct 89.986.4–93.1 57.6 91.4 92.0 61.2 96.6 90.0 83.3 96.7 100.0 97.96 judge-resolved 75.0% 86.7%fell for 2 of 15 5.5 / 4.8agreement: medium 16.8ROCm 7.1 5.5 4.4 Q4_K_M apache-2.0
19–77 Qwen2.5-VL-32B-Instruct 89.786.5–92.8 58.3 88.6 84.0 86.6 69.5 100.0 88.9 90.0 100.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 4.9 / 5.6agreement: high 2.7ROCm 7.1 54.9 33.5 Q4_K_M apache-2.0
10–82 Olmo 3.1 32B Instruct 89.686.0–93.2 54.2 82.9 84.0 84.7 95.2 96.7 77.8 90.0 95.0 100.06 judge-resolved 91.7% 100.0%fell for 0 of 15 6.8 / 6.0agreement: high 7.4ROCm 7.1 27.7 32 Q4_K_M —
18–81 Qwen3-4B 89.285.6–92.4 58.1 88.6 88.0 69.0 91.0 100.0 77.8 88.3 100.0 100.06 judge-resolved 100.0% 100.0%fell for 0 of 15 5.6 / 5.6agreement: high 41.3ROCm 7.1 12.7 4 Q4_K_M apache-2.0
19–85 DeepSeek-R1-Distill-Qwen-32B 89.186.0–91.9 — 97.1 92.0 53.6 75.7 100.0 100.0 96.7 95.0 91.76 judge-resolved 100.0% 93.3%fell for 1 of 15 5.8 / 5.9agreement: high 4.8ROCm 7.1 19.9 32.8 Q4_K_M mit
14–90 Flint 40B A2B 89.085.1–92.6 75.2 71.4 100.0 93.4 96.6 90.0 77.8 91.7 80.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 6.5 / 6.3agreement: high 31.9ROCm 7.1 24.3 40 NVFP4 —
19–81 Ornith-1.5-9B 89.085.4–92.4 52.1 91.4 84.0 67.9 85.2 100.0 88.9 88.3 95.0 100.06 judge-resolved 91.7% 100.0%fell for 0 of 15 6.4 / 5.9agreement: high 8.0ROCm 7.1 6.4 9.7 Q4_K_M mit
23–86 Devstral 24B Small 2505code model 88.985.6–92.0 40.1 77.1 76.0 70.4 84.2 100.0 94.4 98.3 100.0 100.06 judge-resolved 83.3% 80.0%fell for 3 of 15 5.9 / 5.9agreement: high 8.0ROCm 7.1 36.5 24 Q4_K_M —
17–84 GLM-4.7-Flash 88.985.2–92.5 60.3 77.1 92.0 81.6 86.7 100.0 77.8 100.0 85.0 100.06 judge-resolved 100.0% 66.7%fell for 5 of 15 6.0 / 5.6agreement: high 19.3ROCm 7.1 26.0 31.2 Q4_K_M mit
19–84 Qwen3.5-4B 88.784.9–92.4 55.8 85.7 88.0 71.5 90.9 100.0 83.3 98.3 85.0 95.86 judge-resolved 83.3% 86.7%fell for 2 of 15 5.9 / 5.8agreement: medium 14.6ROCm 7.1 3.6 4.7 Q4_K_M apache-2.0
19–86 Ministral 3 8B Instruct 2512 88.785.2–92.1 68.8 82.9 80.0 78.4 77.8 96.7 94.4 88.3 100.0 100.06 judge-resolved 100.0% 80.0%fell for 3 of 15 6.3 / 5.9agreement: high 22.2ROCm 7.1 23.8 8 Q4_K_M apache-2.0
30–86 Qwen2.5-Coder-14B-Instructcode model 87.784.0–91.3 52.3 82.9 72.0 82.7 84.2 93.3 88.9 85.8 100.0 100.06 judge-resolved 83.3% 73.3%fell for 4 of 15 5.5 / 5.4agreement: high 15.9ROCm 7.1 34.8 14.8 Q4_K_M apache-2.0
26–96 Lfm2 24B 87.383.2–91.1 51.9 77.1 96.0 72.9 96.6 100.0 66.7 91.7 85.0 100.06 judge-resolved 91.7% 80.0%fell for 3 of 15 6.0 / 5.4agreement: high 29.5ROCm 7.1 15.0 24 Q4_K_M —
32–101 Nemotron3 33B 86.282.0–90.0 65.6 80.0 84.0 83.7 92.6 100.0 83.3 55.0 97.5 100.06 judge-resolved 91.7% 86.7%fell for 2 of 15 6.4 / 6.0agreement: high 18.9ROCm 7.1 25.0 33 Q4_K_M —
44–97 Gemma 3n E4B IT 86.182.4–89.6 50.7 82.9 92.0 65.3 83.4 96.7 66.7 93.3 95.0 100.06 judge-resolved 91.7% 73.3%fell for 4 of 15 6.8 / 5.6agreement: medium 20.0ROCm 7.1 4.2 8 Q4_K_M gemma
39–99 phi-4 85.981.9–89.8 64.2 91.4 84.0 87.0 73.2 96.7 77.8 93.3 70.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 5.6 / 5.8agreement: high 8.9ROCm 7.1 36.2 14.7 Q4_K_M mit
36–106 Nemotron Cascade 2 30B A3B 85.681.2–89.7 62.7 68.6 80.0 77.9 97.8 90.0 77.8 78.3 100.0 100.06 judge-resolved 91.7% 80.0%fell for 3 of 15 5.9 / 5.4agreement: high 17.1ROCm 7.1 24.4 30 Q4_K_M —
39–103 Gemma2 27B Instruct 85.681.4–89.4 48.9 65.7 68.0 67.5 98.2 96.7 83.3 95.0 100.0 95.86 judge-resolved 100.0% 80.0%fell for 3 of 15 5.4 / 5.5agreement: high 5.5ROCm 7.1 27.0 27 Q4_K_M —
45–100 Qwen2.5-7B-Instruct 85.681.4–89.3 44.7 77.1 76.0 63.2 88.4 100.0 77.8 91.7 100.0 95.86 judge-resolved 91.7% 66.7%fell for 5 of 15 4.6 / 4.8agreement: high 28.3ROCm 7.1 8.7 7.6 Q4_K_M apache-2.0
38–106 DeepSeek-R1-Distill-Qwen-14B 85.381.2–89.2 — 91.4 88.0 53.0 80.8 100.0 77.8 89.2 100.0 87.56 judge-resolved 100.0% 93.3%fell for 1 of 15 5.1 / 4.9agreement: high 10.0ROCm 7.1 9.0 14.8 Q4_K_M mit
46–106 Exaone3.5 32B Instruct 85.080.9–88.8 36.7 77.1 76.0 71.5 89.8 96.7 66.7 87.5 100.0 100.06 judge-resolved 91.7% 80.0%fell for 3 of 15 5.3 / 5.5agreement: high 7.7ROCm 7.1 53.9 32 Q4_K_M —
43–106 Spark-X2.6-4B 85.080.9–89.1 41.8 85.7 84.0 65.6 78.7 90.0 77.8 98.3 85.0 100.06 judge-resolved 83.3% 73.3%fell for 4 of 15 5.1 / 5.1agreement: high 38.7ROCm 7.1 5.3 4 Q4_K_M —
47–108 Mistral Small 22B Instruct 2409 84.680.7–88.5 36.8 74.3 72.0 70.1 78.9 96.7 88.9 94.2 95.0 91.76 judge-resolved 83.3% 66.7%fell for 5 of 15 5.9 / 5.1agreement: medium 6.1ROCm 7.1 44.0 22 Q4_K_M —
50–106 Qwen2.5-Coder-7B-Instructcode model 84.580.6–88.1 42.2 82.9 72.0 71.8 72.6 93.3 77.8 90.0 100.0 100.06 judge-resolved 58.3% 80.0%fell for 3 of 15 4.9 / 4.5agreement: high 28.2ROCm 7.1 12.5 7.6 Q4_K_M apache-2.0
49–113 NVIDIA Nemotron 3 Nano 30B-A3B 83.979.5–88.5 62.5 65.7 84.0 86.0 96.5 96.7 61.1 75.0 90.0 100.06 judge-resolved 83.3% 93.3%fell for 1 of 15 5.8 / 5.8agreement: medium 17.0ROCm 7.1 24.4 30 Q4_K_M other
47–111 Qwen3-4B-Thinking-2507 83.580.2–87.0 — 74.3 76.0 26.7 88.6 100.0 94.4 91.7 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.0 / 5.4agreement: high 32.1ROCm 7.1 2.5 4 Q4_K_M apache-2.0
52–118 Olmo-3-7B-Instruct 82.978.8–86.9 43.3 80.0 96.0 57.7 96.7 93.3 44.4 86.7 95.0 95.86 judge-resolved 91.7% 93.3%fell for 1 of 15 5.0 / 5.4agreement: high 22.8ROCm 7.1 14.4 7.3 Q4_K_M apache-2.0
58–123 gemma-3-4b-it 82.177.6–86.5 48.3 71.4 84.0 71.4 85.9 80.0 61.1 100.0 85.0 100.06 judge-resolved 75.0% 73.3%fell for 4 of 15 5.6 / 5.5agreement: high 38.6ROCm 7.1 4.5 4.3 Q4_K_M gemma
59–122 EXAONE-3.5-7.8B-Instruct 82.077.6–86.4 43.0 80.0 72.0 70.1 80.8 100.0 55.6 88.3 100.0 91.76 judge-resolved 75.0% 80.0%fell for 3 of 15 4.6 / 5.1agreement: high 23.0ROCm 7.1 13.6 7.8 Q4_K_M exaone-nc
59–121 Qwen2.5-VL-7B-Instruct 82.077.8–86.3 32.1 74.3 68.0 64.4 78.8 96.7 77.8 83.3 95.0 100.06 judge-resolved 91.7% 60.0%fell for 6 of 15 4.9 / 4.1agreement: high 28.4ROCm 7.1 13.2 8.3 Q4_K_M apache-2.0
53–118 Command R7B Arabic 7B 02 2025 81.977.5–86.6 49.5 74.3 68.0 67.4 84.8 93.3 77.8 90.8 85.0 95.86 judge-resolved 83.3% 66.7%fell for 5 of 15 5.9 / 5.5agreement: high 16.2ROCm 7.1 10.5 7 Q4_K_M cc-by-nc-4.0
59–128 Granite4.2 8B 81.977.2–86.3 51.4 65.7 68.0 85.1 94.1 100.0 55.6 88.3 80.0 100.06 judge-resolved 83.3% 66.7%fell for 5 of 15 6.4 / 6.3agreement: high 24.8ROCm 7.1 27.1 8 Q4_K_M —
56–122 Deepseek R1 8B Llama Distill 81.776.9–86.1 57.0 91.4 96.0 52.8 82.3 90.0 72.2 71.7 100.0 79.26 judge-resolved 100.0% 86.7%fell for 2 of 15 4.9 / 4.8agreement: high 18.2ROCm 7.1 22.4 8 Q4_K_M llama3.1
62–123 Phi-4-mini-instruct 81.677.1–86.0 34.5 77.1 76.0 61.0 84.1 96.7 66.7 81.7 95.0 95.86 judge-resolved 83.3% 80.0%fell for 3 of 15 4.5 / 4.6agreement: high 43.7ROCm 7.1 11.4 3.8 Q4_K_M mit
57–133 Falcon3 10B Instruct 81.376.6–85.8 — 80.0 60.0 64.3 84.8 96.7 72.2 83.3 90.0 100.06 judge-resolved 75.0% 86.7%fell for 2 of 15 5.3 / 5.3agreement: high 20.9ROCm 7.1 6.3 10 Q4_K_M other
65–125 Llama-3.1-8B-Instruct 81.076.5–85.2 40.6 60.0 56.0 65.6 92.6 90.0 83.3 81.7 100.0 100.06 judge-resolved 83.3% 66.7%fell for 5 of 15 5.8 / 5.3agreement: high 23.6ROCm 7.1 13.7 8 Q4_K_M llama3.1
61–124 Ministral 3 3B Instruct 2512 80.976.5–85.2 50.3 74.3 88.0 60.6 67.5 93.3 61.1 92.5 95.0 95.86 judge-resolved 83.3% 53.3%fell for 7 of 15 6.1 / 5.3agreement: high 156.8ROCm 7.1 16.6 3 Q4_K_M apache-2.0
66–129 granite-4.1-3b 80.775.9–85.3 42.0 65.7 80.0 62.6 93.9 90.0 66.7 86.7 85.0 95.86 judge-resolved 100.0% 73.3%fell for 4 of 15 5.3 / 5.0agreement: high 49.0ROCm 7.1 7.8 3.4 Q4_K_M apache-2.0
69–131 gemma-2-9b-it 79.975.5–84.2 36.8 68.6 60.0 51.8 92.6 90.0 61.1 100.0 95.0 100.06 judge-resolved 91.7% 73.3%fell for 4 of 15 5.9 / 5.5agreement: high 11.7ROCm 7.1 14.7 9.2 Q4_K_M gemma
64–127 Gemma3n E2B IT 79.975.2–84.7 55.7 68.6 76.0 61.4 89.7 83.3 72.2 80.0 87.5 100.06 judge-resolved 91.7% 53.3%fell for 7 of 15 6.8 / 5.9agreement: medium 26.1ROCm 7.1 3.4 5 Q4_K_M gemma
67–135 DeepSeek-R1-0528-Qwen3-8B 79.675.5–83.9 — 77.1 48.0 42.1 84.3 96.7 77.8 95.0 95.0 100.06 judge-resolved 91.7% 86.7%fell for 2 of 15 5.5 / 5.3agreement: high 18.4ROCm 7.1 5.2 8.2 Q4_K_M mit
66–129 Granite3.3 8B 79.474.7–83.9 40.2 62.9 76.0 56.6 85.7 96.7 66.7 72.5 97.5 100.06 judge-resolved 91.7% 73.3%fell for 4 of 15 5.4 / 5.1agreement: high 25.9ROCm 7.1 26.9 8 Q4_K_M apache-2.0
64–132 Granite4 Tiny H 79.174.1–84.2 54.3 74.3 68.0 80.8 79.7 86.7 66.7 80.0 80.0 95.86 judge-resolved 91.7% 60.0%fell for 6 of 15 5.0 / 5.1agreement: high 40.9ROCm 7.1 6.1 7 Q4_K_M apache-2.0
69–136 DeepSeek-Coder-V2-Lite-Instructcode model 79.074.3–83.5 39.0 65.7 56.0 72.5 77.9 83.3 77.8 86.7 95.0 95.86 judge-resolved 100.0% 46.7%fell for 8 of 15 4.4 / 4.5agreement: high 4.6ROCm 7.1 46.8 15.7 Q4_K_M other
72–137 Aya Expanse 32B 78.874.6–82.9 37.7 54.3 48.0 64.7 81.3 100.0 72.2 93.3 95.0 100.06 judge-resolved 75.0% 66.7%fell for 5 of 15 5.3 / 4.9agreement: high 13.5ROCm 7.1 31.4 32 Q4_K_M —
72–140 Granite4.2 3B 78.273.7–83.1 36.3 74.3 56.0 61.6 91.2 90.0 61.1 75.0 95.0 100.06 judge-resolved 91.7% 86.7%fell for 2 of 15 5.0 / 5.3agreement: high 50.1ROCm 7.1 13.2 3 Q4_K_M —
71–142 SmolLM3-3B 78.073.0–83.1 32.7 71.4 72.0 57.7 86.0 96.7 77.8 75.0 82.5 83.36 judge-resolved 83.3% 60.0%fell for 6 of 15 5.5 / 5.0agreement: high 50.8ROCm 7.1 12.0 3.1 Q4_K_M apache-2.0
67–136 Phi3 14B Medium 128K Instruct 78.073.2–82.6 49.6 77.1 56.0 61.4 71.6 90.0 88.9 69.2 87.5 100.06 judge-resolved 66.7% 73.3%fell for 4 of 15 5.5 / 4.9agreement: medium 17.0ROCm 7.1 — 14 Q4_K_M mit
68–146 Olmo-3-7B-Think 77.773.1–82.5 — 62.9 48.0 40.0 98.3 100.0 88.9 90.0 80.0 91.76 judge-resolved 66.7% 80.0%fell for 3 of 15 4.8 / 5.0agreement: high 24.0ROCm 7.1 4.5 7.3 Q4_K_M apache-2.0
68–137 Qwen3 VL 2B Instruct 77.472.5–82.1 44.0 65.7 64.0 49.2 89.8 93.3 66.7 75.8 100.0 91.76 judge-resolved 75.0% 60.0%fell for 6 of 15 5.0 / 3.6agreement: medium 26.2ROCm 7.1 3.6 2 Q4_K_M apache-2.0
76–148 Qwen2.5-3B-Instruct 76.872.0–81.6 25.4 71.4 72.0 48.7 82.7 90.0 61.1 75.0 90.0 100.06 judge-resolved 91.7% 53.3%fell for 7 of 15 4.6 / 4.6agreement: high 44.9ROCm 7.1 4.7 3.1 Q4_K_M qwen-research
72–139 Granite4 Micro H 76.672.0–81.4 46.9 62.9 60.0 68.8 90.0 90.0 38.9 83.3 100.0 95.86 judge-resolved 91.7% 73.3%fell for 4 of 15 5.0 / 4.9agreement: high 30.9ROCm 7.1 4.0 3 Q4_K_M apache-2.0
73–141 Exaone3.5 2.4B Instruct 76.370.9–81.3 47.3 82.9 76.0 64.7 73.6 90.0 50.0 62.5 95.0 91.76 judge-resolved 75.0% 66.7%fell for 5 of 15 4.9 / 5.3agreement: high 256.0ROCm 7.1 11.9 2.4 Q4_K_M exaone-nc
71–150 Falcon3 7B Instruct 76.271.6–81.3 — 71.4 76.0 58.7 80.2 93.3 55.6 67.5 87.5 95.86 judge-resolved 100.0% 80.0%fell for 3 of 15 5.3 / 4.8agreement: high 28.4ROCm 7.1 4.6 7 Q4_K_M other
77–141 Qwen3 1.7B 76.071.3–80.9 42.8 68.6 72.0 45.5 86.8 76.7 55.6 87.5 100.0 91.76 judge-resolved 75.0% 73.3%fell for 4 of 15 4.9 / 4.5agreement: high 43.5ROCm 7.1 16.5 1.7 Q4_K_M apache-2.0
78–142 Mistral Nemo 12B Instruct 2407 75.970.6–80.8 37.7 57.1 56.0 57.3 80.7 96.7 77.8 85.0 85.0 87.56 judge-resolved 75.0% 86.7%fell for 2 of 15 4.4 / 4.4agreement: high 12.5ROCm 7.1 29.2 12 Q4_K_M apache-2.0
84–150 Llama-3.2-3B-Instruct 75.771.0–80.4 33.5 68.6 52.0 52.9 91.7 86.7 55.6 81.7 92.5 100.06 judge-resolved 83.3% 60.0%fell for 6 of 15 5.8 / 5.3agreement: high 49.8ROCm 7.1 9.9 3.2 Q4_K_M llama3.2
80–143 Lfm2.5 8B A1B 75.770.8–80.5 54.2 85.7 88.0 47.3 91.1 96.7 61.1 81.7 30.0 100.06 judge-resolved 100.0% 80.0%fell for 3 of 15 4.6 / 3.9agreement: high 41.6ROCm 7.1 5.5 8 Q4_K_M other
79–152 YI 34B Chat V1.5 75.270.7–79.9 35.4 71.4 60.0 50.8 76.6 100.0 38.9 83.3 100.0 95.86 judge-resolved 75.0% 80.0%fell for 3 of 15 4.9 / 4.9agreement: high 7.1ROCm 7.1 28.8 34 Q4_K_M —
79–153 Command R 35B 08 2024 75.270.6–79.8 40.7 48.6 32.0 63.9 84.1 93.3 88.9 84.2 90.0 91.76 judge-resolved 83.3% 46.7%fell for 8 of 15 5.6 / 5.3agreement: high 7.7ROCm 7.1 42.2 35 Q4_K_M —
81–144 Qwen3.5 2B 75.169.7–80.3 44.1 74.3 80.0 52.5 82.1 80.0 77.8 63.3 70.0 95.86 judge-resolved 75.0% 66.7%fell for 5 of 15 4.8 / 4.5agreement: high 32.2ROCm 7.1 1.9 2 Q4_K_M apache-2.0
87–152 Command R7B (12-2024) 74.870.1–79.1 39.7 71.4 72.0 59.4 82.9 86.7 83.3 96.7 25.0 95.86 judge-resolved 75.0% 80.0%fell for 3 of 15 5.6 / 5.0agreement: high 14.0ROCm 7.1 10.4 8 Q4_K_M cc-by-nc-4.0
80–154 Codestral 22B V0.1code model 74.869.7–79.5 39.6 45.7 48.0 72.9 74.4 96.7 83.3 83.3 85.0 83.36 judge-resolved 83.3% 60.0%fell for 6 of 15 4.9 / 4.6agreement: high 6.5ROCm 7.1 44.0 22 Q4_K_M —
87–152 Meta-Llama-3-8B-Instruct 74.769.9–79.5 35.7 51.4 36.0 60.0 89.3 90.0 83.3 81.7 85.0 95.86 judge-resolved 75.0% 60.0%fell for 6 of 15 6.5 / 5.6agreement: high 27.3ROCm 7.1 9.2 8 Q4_K_M llama3
88–152 medgemma-4b-itmedical model 74.369.3–79.2 38.6 65.7 72.0 60.3 78.6 80.0 50.0 77.5 85.0 100.06 judge-resolved 83.3% 66.7%fell for 5 of 15 5.4 / 5.0agreement: high 42.2ROCm 7.1 4.5 4.3 Q4_K_M other
84–148 Lfm2.5 Thinking 1.2B 74.369.5–79.1 45.2 82.9 80.0 42.1 80.8 83.3 38.9 71.7 97.5 91.76 judge-resolved 50.0% 80.0%fell for 3 of 15 4.6 / 3.8agreement: high 122.0ROCm 7.1 1.1 1.2 Q4_K_M other
85–145 Granite4 3B 74.169.0–79.2 41.4 62.9 56.0 64.7 87.3 76.7 55.6 78.3 90.0 95.86 judge-resolved 83.3% 80.0%fell for 3 of 15 5.4 / 5.0agreement: high 177.1ROCm 7.1 13.2 3 Q4_K_M apache-2.0
85–145 Granite3.2 8B Instruct 74.169.5–78.9 31.0 45.7 56.0 58.5 82.7 96.7 50.0 81.7 100.0 95.86 judge-resolved 75.0% 73.3%fell for 4 of 15 5.4 / 5.3agreement: high 97.7ROCm 7.1 26.9 8 Q4_K_M apache-2.0
82–155 DeepSeek-R1-Distill-Qwen-7B 73.668.9–78.3 — 77.1 96.0 37.0 77.4 83.3 44.4 80.8 100.0 66.76 judge-resolved 75.0% 80.0%fell for 3 of 15 5.0 / 4.5agreement: high 20.7ROCm 7.1 4.7 7.6 Q4_K_M mit
87–153 Glm4 9B Chat 73.368.0–78.5 38.3 60.0 60.0 60.0 82.3 90.0 66.7 85.0 60.0 95.86 judge-resolved 83.3% 73.3%fell for 4 of 15 4.9 / 4.5agreement: high 15.9ROCm 7.1 11.8 9 Q4_K_M other
90–154 dolphin-2.9.1-yi-1.5-34b 73.067.9–78.2 31.7 71.4 56.0 65.7 72.7 93.3 66.7 66.7 85.0 79.26 judge-resolved 91.7% 66.7%fell for 5 of 15 4.9 / 5.1agreement: high 7.2ROCm 7.1 37.2 34.4 Q4_K_M apache-2.0
88–151 Granite3.1 Dense 8B Instruct 72.967.8–77.6 31.8 57.1 52.0 64.2 74.3 100.0 55.6 61.7 95.0 95.86 judge-resolved 50.0% 66.7%fell for 5 of 15 6.1 / 5.5agreement: high 97.2ROCm 7.1 27.0 8 Q4_K_M apache-2.0
90–154 Aya Expanse 8B 72.567.5–77.3 35.5 54.3 40.0 50.0 79.5 90.0 66.7 72.5 100.0 100.06 judge-resolved 91.7% 80.0%fell for 3 of 15 5.8 / 5.4agreement: high 16.3ROCm 7.1 14.2 8 Q4_K_M cc-by-nc-4.0
89–155 Qwen2.5-Coder-3B-Instructcode model 72.567.7–77.5 29.9 57.1 44.0 60.2 70.0 80.0 72.2 78.3 95.0 95.86 judge-resolved 75.0% 60.0%fell for 6 of 15 4.8 / 3.8agreement: high 49.3ROCm 7.1 4.7 3.1 Q4_K_M qwen-research
90–157 Qwen2.5-VL-3B-Instruct 72.367.2–77.6 22.2 71.4 64.0 42.6 74.7 100.0 66.7 60.0 80.0 91.76 judge-resolved 83.3% 60.0%fell for 6 of 15 4.9 / 4.0agreement: high 51.4ROCm 7.1 5.5 3.8 Q4_K_M —
92–157 Qwen2-VL-7B-Instruct 72.267.4–77.0 21.6 45.7 64.0 49.7 70.7 86.7 66.7 66.7 100.0 100.06 judge-resolved 66.7% 60.0%fell for 6 of 15 4.8 / 4.5agreement: high 16.1ROCm 7.1 12.8 8.3 Q4_K_M apache-2.0
89–158 Lfm3 VL 2B 72.267.2–77.0 20.9 62.9 60.0 42.2 88.1 86.7 44.4 91.7 95.0 79.26 judge-resolved 100.0% 80.0%fell for 3 of 15 5.5 / 4.9agreement: high 58.7ROCm 7.1 2.0 2 Q4_K_M —
97–157 Phi4 Mini Reasoning 3.8B 70.965.9–76.0 47.9 94.3 96.0 31.8 44.3 90.0 66.7 75.0 55.0 85.46 judge-resolved 100.0% 93.3%fell for 1 of 15 3.6 / 2.5agreement: high 139.1ROCm 7.1 20.7 3.8 Q4_K_M mit
103–156 YI 9B Chat V1.5 70.565.4–75.3 37.9 68.6 64.0 57.5 73.7 90.0 22.2 75.0 87.5 95.86 judge-resolved 83.3% 60.0%fell for 6 of 15 4.4 / 4.5agreement: high 24.6ROCm 7.1 8.6 9 Q4_K_M apache-2.0
96–162 Lfm3 1.7B 70.265.3–75.3 29.7 74.3 76.0 61.5 77.9 90.0 33.3 83.3 40.0 95.86 judge-resolved 100.0% 93.3%fell for 1 of 15 4.6 / 5.0agreement: high 58.2ROCm 7.1 1.6 1.7 Q4_K_M —
97–165 Olmo2 13B 1124 Instruct 69.664.5–74.5 — 60.0 52.0 40.8 88.6 96.7 33.3 80.0 75.0 100.06 judge-resolved 100.0% 53.3%fell for 7 of 15 5.8 / 5.3agreement: high 15.5ROCm 7.1 21.7 13 Q4_K_M apache-2.0
109–159 Codegeex4 9B Allcode model 68.763.2–73.8 39.5 48.6 36.0 60.5 75.2 93.3 77.8 63.3 80.0 83.36 judge-resolved 83.3% 33.3%fell for 10 of 15 4.4 / 3.4agreement: high 16.3ROCm 7.1 11.8 9 Q4_K_M other
111–159 Granite3.3 2B 68.663.8–73.8 30.4 57.1 52.0 50.4 76.3 90.0 44.4 57.5 90.0 100.06 judge-resolved 83.3% 46.7%fell for 8 of 15 5.0 / 5.4agreement: high 58.5ROCm 7.1 12.6 2 Q4_K_M apache-2.0
111–160 Qwen2.5 1.5B Instruct 68.563.2–73.7 36.0 62.9 52.0 49.9 59.4 93.3 66.7 48.3 92.5 91.76 judge-resolved 75.0% 60.0%fell for 6 of 15 4.6 / 4.0agreement: high 43.8ROCm 7.1 5.1 1.5 Q4_K_M apache-2.0
112–160 Gemma2 2B Instruct 68.262.9–73.3 32.6 42.9 44.0 42.4 84.6 86.7 55.6 80.0 77.5 100.06 judge-resolved 83.3% 66.7%fell for 5 of 15 5.1 / 5.1agreement: medium 39.9ROCm 7.1 4.7 2 Q4_K_M gemma
109–173 Hermes-3-Llama-3.1-8B 67.662.4–73.1 16.9 54.3 52.0 42.3 73.8 93.3 44.4 71.7 85.0 91.76 judge-resolved 75.0% 66.7%fell for 5 of 15 5.8 / 5.1agreement: high 21.3ROCm 7.1 13.7 8 Q4_K_M llama3
117–161 Granite3 Dense 8B Instruct 67.662.8–72.3 29.0 54.3 44.0 59.2 76.4 93.3 33.3 52.5 95.0 100.06 judge-resolved 91.7% 73.3%fell for 4 of 15 4.8 / 4.6agreement: high 14.5ROCm 7.1 10.5 8 Q4_K_M apache-2.0
116–161 Internlm2 7B Chat V2.5 67.462.1–72.6 35.8 65.7 60.0 48.7 69.6 60.0 72.2 45.0 85.0 100.06 judge-resolved 83.3% 46.7%fell for 8 of 15 4.9 / 4.6agreement: high 28.3ROCm 7.1 22.2 7 Q4_K_M other
110–174 Phi-3.5-mini-instruct 67.162.0–72.3 20.9 60.0 48.0 38.9 60.2 90.0 72.2 46.7 87.5 100.06 judge-resolved 83.3% 80.0%fell for 3 of 15 4.8 / 4.5agreement: high 41.6ROCm 7.1 28.4 3.8 Q4_K_M mit
119–162 Granite4 1B H 66.861.4–72.1 37.7 48.6 52.0 59.9 85.1 76.7 55.6 43.3 80.0 100.06 judge-resolved 75.0% 53.3%fell for 7 of 15 4.9 / 3.6agreement: high 50.1ROCm 7.1 3.4 1 Q8_0 apache-2.0
108–178 Qwen2 Math 7B Instructmath model 66.060.5–71.3 — 71.4 76.0 39.1 68.7 66.7 33.3 77.5 90.0 70.86 judge-resolved 75.0% 60.0%fell for 6 of 15 4.6 / 4.0agreement: high 19.4ROCm 7.1 4.7 7 Q4_K_M apache-2.0
120–170 Exaone Deep 7.8B 65.760.0–71.5 39.8 85.7 88.0 55.4 16.9 86.7 61.1 50.8 80.0 66.76 judge-resolved 91.7% 93.3%fell for 1 of 15 5.0 / 3.3agreement: medium 28.9ROCm 7.1 22.3 7.8 Q4_K_M exaone-nc
125–167 Olmo2 7B 1124 Instruct 65.460.6–70.1 23.5 51.4 52.0 29.3 84.4 93.3 27.8 60.0 90.0 100.06 judge-resolved 83.3% 46.7%fell for 8 of 15 4.9 / 4.9agreement: high 16.9ROCm 7.1 21.6 7 Q4_K_M apache-2.0
113–181 Falcon3 3B Instruct 64.859.4–70.6 — 51.4 52.0 40.9 76.8 73.3 66.7 50.8 80.0 91.76 judge-resolved 66.7% 66.7%fell for 5 of 15 5.3 / 4.9agreement: high 60.3ROCm 7.1 2.0 3 Q4_K_M other
132–172 Granite3.2 2B Instruct 63.858.7–69.0 23.9 42.9 52.0 48.4 75.5 80.0 50.0 43.3 95.0 87.56 judge-resolved 75.0% 46.7%fell for 8 of 15 4.8 / 4.9agreement: high 32.1ROCm 7.1 12.6 2 Q4_K_M apache-2.0
125–182 Apertus-8B-Instruct-2509 63.658.2–68.8 — 42.9 32.0 33.0 76.3 83.3 50.0 75.0 80.0 100.06 judge-resolved 83.3% 46.7%fell for 8 of 15 4.9 / 4.3agreement: high 32.9Vulkan 22.3 8.1 Q4_K_M apache-2.0
129–182 Phi-3-mini-4k-instruct 63.458.3–68.1 20.9 77.1 56.0 35.1 70.7 96.7 11.1 52.5 80.0 91.76 judge-resolved 66.7% 60.0%fell for 6 of 15 5.5 / 5.6agreement: high 44.5ROCm 7.1 8.9 3.8 Q4_K_M mit
137–176 Phi3 3.8B Mini 128K Instruct 62.958.0–67.7 21.9 54.3 40.0 38.6 60.2 96.7 50.0 31.7 95.0 100.06 judge-resolved 91.7% 53.3%fell for 7 of 15 4.5 / 4.5agreement: high 96.1ROCm 7.1 54.1 3.8 Q4_K_M mit
136–187 Codellama 34B Instructcode model 61.256.0–66.3 21.9 42.9 16.0 38.7 75.1 73.3 66.7 50.0 92.5 95.86 judge-resolved 41.7% 33.3%fell for 10 of 15 4.5 / 3.6agreement: high 7.4ROCm 7.1 46.7 34 Q4_K_M —
147–183 Gemma 7B Instruct V1.1 60.855.5–66.1 28.5 40.0 24.0 26.5 76.0 83.3 61.1 56.7 80.0 100.06 judge-resolved 91.7% 60.0%fell for 6 of 15 5.1 / 4.6agreement: high 15.2ROCm 7.1 36.1 7 Q4_K_M gemma
147–184 Granite3.1 Dense 2B Instruct 60.655.0–66.1 27.0 45.7 36.0 46.6 73.6 76.7 50.0 45.0 80.0 91.76 judge-resolved 58.3% 40.0%fell for 9 of 15 4.5 / 4.8agreement: high 56.8ROCm 7.1 12.6 2 Q4_K_M apache-2.0
147–185 Qwen2.5 Coder 1.5B Instructcode model 60.554.9–66.0 28.6 65.7 44.0 52.8 58.8 60.0 66.7 30.8 82.5 83.36 judge-resolved 66.7% 53.3%fell for 7 of 15 4.5 / 3.3agreement: medium 77.0ROCm 7.1 5.1 1.5 Q4_K_M apache-2.0
148–186 Mistral 7B Instruct V0.3 59.854.7–64.7 24.9 37.1 16.0 40.9 70.3 93.3 61.1 41.7 90.0 87.56 judge-resolved 66.7% 53.3%fell for 7 of 15 5.4 / 5.1agreement: high 17.5ROCm 7.1 22.0 7 Q4_K_M apache-2.0
148–186 Granite3.1 Moe 3B Instruct 59.854.7–65.0 24.9 37.1 32.0 32.6 69.2 83.3 61.1 41.7 85.0 95.86 judge-resolved 58.3% 66.7%fell for 5 of 15 4.3 / 4.6agreement: high 93.0ROCm 7.1 10.9 3 Q4_K_M apache-2.0
150–186 Codegemma 7B Instruct V1.1code model 59.554.8–64.2 29.3 45.7 24.0 55.5 78.1 70.0 5.6 70.0 95.0 91.76 judge-resolved 75.0% 46.7%fell for 8 of 15 5.0 / 5.0agreement: high 52.6ROCm 7.1 36.1 7 Q4_K_M gemma
143–191 Mistral-7B-Instruct-v0.2 59.254.3–64.2 18.7 40.0 12.0 25.3 75.1 90.0 72.2 42.5 80.0 95.86 judge-resolved 91.7% 60.0%fell for 6 of 15 4.8 / 4.9agreement: high 29.0ROCm 7.1 22.0 7.2 Q4_K_M apache-2.0
150–186 YI Coder 9B Chatcode model 59.053.7–64.5 37.8 31.4 16.0 81.3 65.3 83.3 44.4 54.2 95.0 60.46 judge-resolved 75.0% 53.3%fell for 7 of 15 3.8 / 3.9agreement: high 24.6ROCm 7.1 18.5 9 Q4_K_M apache-2.0
151–186 Solar 10.7B Instruct V1 58.753.7–63.8 26.4 45.7 16.0 34.3 69.9 86.7 38.9 58.3 95.0 83.36 judge-resolved 83.3% 33.3%fell for 10 of 15 5.8 / 5.1agreement: high 13.6ROCm 7.1 13.1 10.7 Q4_K_M —
152–187 Granite3 Dense 2B Instruct 58.353.1–63.6 20.2 48.6 24.0 42.2 63.4 86.7 33.3 48.3 82.5 95.86 judge-resolved 75.0% 40.0%fell for 9 of 15 4.4 / 4.4agreement: high 43.5ROCm 7.1 4.4 2 Q4_K_M apache-2.0
152–188 Deepseek V2 16B Lite Chat 58.253.2–63.4 20.1 34.3 40.0 41.6 70.2 70.0 33.3 50.8 87.5 95.86 judge-resolved 58.3% 40.0%fell for 9 of 15 5.3 / 4.9agreement: high 5.9ROCm 7.1 46.8 16 Q4_K_M other
152–187 YI 6B Chat V1.5 58.052.8–63.5 21.0 45.7 48.0 19.8 70.4 73.3 33.3 51.7 92.5 87.56 judge-resolved 75.0% 60.0%fell for 6 of 15 4.9 / 4.9agreement: medium 120.2ROCm 7.1 5.9 6 Q4_K_M apache-2.0
150–187 Llama3.2 1B Instruct 58.052.3–63.7 26.6 37.1 40.0 41.7 76.9 76.7 38.9 55.0 72.5 83.36 judge-resolved 50.0% 13.3%fell for 13 of 15 5.1 / 4.0agreement: high 114.5ROCm 7.1 5.4 1 Q4_K_M llama3.2
154–188 Gemma3 1B IT 56.851.8–62.2 29.3 60.0 64.0 42.5 71.9 63.3 22.2 52.5 35.0 100.06 judge-resolved 58.3% 53.3%fell for 7 of 15 5.4 / 4.9agreement: high 80.1ROCm 7.1 1.5 1 Q4_K_M gemma
150–194 Mixtral 8x7B Instruct V0.1 56.851.3–62.0 17.5 42.9 24.0 20.1 62.6 83.3 72.2 41.7 85.0 79.26 judge-resolved 83.3% 73.3%fell for 4 of 15 4.6 / 5.1agreement: high 8.5ROCm 7.1 46.5 56 Q4_K_M —
155–189 Aya 8B 23 55.951.1–60.7 24.9 34.3 0 30.7 74.7 76.7 61.1 39.2 95.0 91.76 judge-resolved 66.7% 40.0%fell for 9 of 15 4.6 / 4.8agreement: high 16.4ROCm 7.1 14.2 8 F16 cc-by-nc-4.0
154–195 Qwen 14B Chat V1.5 54.849.8–59.4 — 48.6 40.0 9.7 76.9 93.3 11.1 52.5 65.0 95.86 judge-resolved 66.7% 60.0%fell for 6 of 15 4.8 / 4.4agreement: high 13.9ROCm 7.1 62.6 14 Q4_K_M other
156–196 Exaone Deep 2.4B 54.748.9–60.5 23.0 82.9 72.0 36.7 13.9 63.3 50.0 38.3 72.5 62.56 judge-resolved 75.0% 93.3%fell for 1 of 15 3.4 / 2.3agreement: high 65.1ROCm 7.1 11.9 2.4 Q4_K_M exaone-nc
158–194 Deepseek Llm 7B Chat 54.349.3–59.8 20.9 37.1 24.0 37.4 66.8 76.7 27.8 45.8 77.5 95.86 judge-resolved 25.0% 40.0%fell for 9 of 15 4.4 / 3.6agreement: high 24.8ROCm 7.1 20.3 7 Q4_K_M —
161–198 Qwen3 VL 2B Thinking 53.748.3–59.1 26.7 60.0 52.0 0 77.2 73.3 33.3 63.3 82.5 41.74 judge-resolved 50.0% 53.3%fell for 7 of 15 3.4 / 3.1agreement: high 26.4ROCm 7.1 3.6 2 Q4_K_M apache-2.0
159–202 Codellama 7B Instructcode model 53.047.8–57.8 — 37.1 4.0 36.0 68.1 76.7 61.1 21.7 85.0 87.56 judge-resolved 50.0% 40.0%fell for 9 of 15 4.6 / 4.4agreement: high 27.2ROCm 7.1 38.8 7 Q4_K_M llama2
159–203 Codellama 13B Instructcode model 52.547.2–57.8 — 31.4 12.0 22.5 73.8 83.3 50.0 33.3 82.5 83.36 judge-resolved 50.0% 66.7%fell for 5 of 15 4.6 / 3.8agreement: high 16.2ROCm 7.1 61.7 13 Q4_K_M llama2
171–198 Nemotron Mini 4B Instruct 52.047.4–56.9 22.4 34.3 12.0 34.6 72.4 83.3 11.1 47.5 85.0 87.56 judge-resolved 50.0% 46.7%fell for 8 of 15 4.4 / 3.9agreement: high 167.6ROCm 7.1 6.6 4 Q4_K_M other
159–207 Internlm2 1.8B Chat V2.5 51.946.1–57.5 — 31.4 28.0 48.5 56.6 70.0 50.0 29.2 70.0 83.36 judge-resolved 41.7% 33.3%fell for 10 of 15 4.1 / 2.9agreement: medium 85.6ROCm 7.1 1.2 1.8 Q4_K_M other
162–207 Magistral Small (2506) 49.844.5–55.3 — 34.3 48.0 10.0 34.8 56.7 44.4 70.0 100.0 50.03 judge-resolved 33.3% 26.7%fell for 11 of 15 4.1 / 3.6agreement: high 10.5ROCm 7.1 14.3 24 Q4_K_M apache-2.0
175–203 Qwen2 1.5B Instruct 49.744.1–55.3 20.1 28.6 32.0 30.6 61.6 63.3 27.8 48.3 80.0 75.06 judge-resolved 66.7% 60.0%fell for 6 of 15 4.1 / 3.6agreement: high 49.8ROCm 7.1 — 1.5 Q4_K_M apache-2.0
176–204 Codeqwen 7B Chat V1.5code model 49.444.2–54.7 34.2 25.7 8.0 69.4 72.1 40.0 38.9 35.0 95.0 60.46 judge-resolved 66.7% 40.0%fell for 9 of 15 4.0 / 2.8agreement: high 25.2ROCm 7.1 13.6 7 Q4_K_M —
177–205 Deepseek R1 1.5B Qwen Distill 48.943.5–54.7 25.5 71.4 76.0 24.4 66.2 53.3 22.2 47.5 62.5 16.73 judge-resolved 75.0% 73.3%fell for 4 of 15 3.9 / 3.0agreement: high 50.6ROCm 7.1 5.1 1.5 Q4_K_M mit
179–206 Granite3 Moe 3B Instruct 47.742.3–53.1 20.2 37.1 20.0 31.6 67.2 76.7 27.8 36.7 45.0 87.56 judge-resolved 50.0% 53.3%fell for 7 of 15 4.3 / 3.5agreement: high 218.4ROCm 7.1 4.3 3 Q4_K_M apache-2.0
180–206 Smollm2 1.7B Instruct 47.141.7–52.7 26.0 28.6 8.0 35.3 79.5 63.3 33.3 35.0 70.0 70.86 judge-resolved 58.3% 40.0%fell for 9 of 15 4.8 / 3.6agreement: high 53.3ROCm 7.1 14.2 1.7 Q4_K_M apache-2.0
184–208 Granite4 350M H 45.640.7–50.7 25.2 20.0 0 36.0 79.3 76.7 16.7 43.3 80.0 58.36 judge-resolved 41.7% 20.0%fell for 12 of 15 3.6 / 2.4agreement: medium 128.2ROCm 7.1 1.2 0.4 Q8_0 apache-2.0
181–211 Llama2 13B Chat 45.340.2–50.6 — 45.7 8.0 10.1 47.8 73.3 27.8 32.5 75.0 87.56 judge-resolved 66.7% 53.3%fell for 7 of 15 5.6 / 4.8agreement: high 16.4ROCm 7.1 21.5 13 Q4_K_M llama2
180–212 Falcon3 1B Instruct 44.339.8–49.3 — 40.0 20.0 18.7 64.9 70.0 11.1 34.2 40.0 100.06 judge-resolved 66.7% 33.3%fell for 10 of 15 3.8 / 2.8agreement: high 107.4ROCm 7.1 1.1 1 Q4_K_M other
183–211 Qwen 7B Chat V1.5 44.139.1–48.9 — 45.7 16.0 5.9 71.9 76.7 11.1 45.0 35.0 89.66 judge-resolved 75.0% 66.7%fell for 5 of 15 4.3 / 3.9agreement: high 23.2ROCm 7.1 39.2 7 Q4_K_M other
185–210 Qwen3 0.6B 44.038.8–49.4 25.0 37.1 40.0 32.5 74.3 20.0 11.1 48.3 82.5 50.06 judge-resolved 41.7% 33.3%fell for 10 of 15 3.8 / 2.9agreement: high 261.1ROCm 7.1 15.6 0.6 Q4_K_M apache-2.0
182–212 Llama-2-7b-chat-hf 43.839.0–48.8 16.4 25.7 4.0 18.6 72.1 66.7 16.7 33.3 65.0 91.76 judge-resolved 66.7% 33.3%fell for 10 of 15 4.5 / 4.0agreement: high 26.7ROCm 7.1 12.8 6.7 Q4_K_M llama2
185–212 Granite3.1 Moe 1B Instruct 43.637.9–49.1 19.0 31.4 12.0 38.1 59.0 63.3 33.3 24.2 62.5 68.86 judge-resolved 58.3% 40.0%fell for 9 of 15 4.3 / 3.5agreement: high 149.5ROCm 7.1 7.5 1 Q4_K_M apache-2.0
186–212 Qwen2.5 0.5B Instruct 43.337.4–49.4 18.2 28.6 36.0 26.5 54.9 56.7 38.9 23.3 62.5 62.56 judge-resolved 58.3% 40.0%fell for 9 of 15 3.5 / 2.5agreement: high 116.8ROCm 7.1 2.2 0.5 Q4_K_M apache-2.0
190–213 Stablelm2 12B Chat 42.438.1–46.7 14.7 22.9 8.0 45.0 47.0 70.0 5.6 20.0 67.5 95.86 judge-resolved 50.0% 33.3%fell for 10 of 15 4.4 / 3.9agreement: high 6.7ROCm 7.1 14.0 12 Q4_K_M stability-nc
188–213 Gemma 2B Instruct V1.1 41.936.8–47.5 22.5 25.7 20.0 25.2 66.6 60.0 27.8 37.5 35.0 79.26 judge-resolved 50.0% 26.7%fell for 11 of 15 4.0 / 4.0agreement: high 43.4ROCm 7.1 3.3 2 Q4_K_M gemma
193–215 Granite Code 8B Instructcode model 40.735.3–46.7 20.6 40.0 28.0 55.7 53.7 63.3 27.8 41.7 10.0 45.86 judge-resolved 58.3% 66.7%fell for 5 of 15 4.6 / 3.4agreement: high 15.0ROCm 7.1 10.0 8 Q4_K_M apache-2.0
195–214 Phi 2.7B Chat V2 39.935.5–44.2 17.5 34.3 0 34.7 60.3 83.3 0 18.3 42.5 85.46 judge-resolved 33.3% 40.0%fell for 9 of 15 3.6 / 2.6agreement: high 38.2ROCm 7.1 7.2 2.7 Q4_K_M mit
197–216 Granite Code 3B Instructcode model 38.032.7–43.5 25.6 37.1 24.0 45.6 54.2 56.7 16.7 42.5 15.0 50.06 judge-resolved 58.3% 40.0%fell for 9 of 15 3.9 / 2.4agreement: medium 107.3ROCm 7.1 7.7 3 Q4_K_M apache-2.0
201–216 Qwen2 0.5B Instruct 36.030.9–41.2 15.6 31.4 8.0 9.8 50.2 43.3 22.2 23.3 77.5 58.36 judge-resolved 41.7% 26.7%fell for 11 of 15 3.9 / 2.6agreement: high 137.5ROCm 7.1 2.2 0.5 Q4_K_M apache-2.0
203–217 Granite3 Moe 1B Instruct 35.531.0–40.5 18.7 17.1 8.0 27.6 59.1 36.7 11.1 20.8 60.0 79.26 judge-resolved 58.3% 33.3%fell for 10 of 15 3.6 / 3.3agreement: high 82.4ROCm 7.1 2.3 1 Q4_K_M apache-2.0
198–219 Deepseek Coder 6.7B Instructcode model 34.729.7–40.0 — 8.6 0 68.5 53.9 10.0 61.1 21.7 55.0 33.36 judge-resolved 50.0% 33.3%fell for 10 of 15 1.3 / 1.0agreement: high 22.5ROCm 7.1 38.8 6.7 Q4_K_M other
203–221 Deepseek Coder 33B Instructcode model 33.429.0–38.2 31.0 31.4 12.0 67.2 46.2 6.7 0 25.8 70.0 41.76 judge-resolved 58.3% 53.3%fell for 7 of 15 1.1 / 1.4agreement: high 5.9ROCm 7.1 53.5 33 Q4_K_M —
203–222 Qwen2 Math 1.5B Instructmath model 33.128.0–38.2 — 65.7 72.0 17.5 14.4 33.3 11.1 37.5 17.5 29.26 judge-resolved 83.3% 40.0%fell for 9 of 15 3.0 / 1.1agreement: medium 62.0ROCm 7.1 1.0 1.5 Q4_K_M apache-2.0
208–218 Qwen2.5 Coder 0.5B Instructcode model 32.827.5–38.2 22.0 22.9 8.0 35.9 49.5 20.0 33.3 31.7 65.0 29.26 judge-resolved 50.0% 33.3%fell for 10 of 15 3.1 / 1.9agreement: high 172.5ROCm 7.1 2.2 0.5 Q4_K_M apache-2.0
209–219 Opencoder 1.5B Instructcode model 31.927.6–36.4 21.5 2.9 0 57.3 50.8 20.0 11.1 24.2 77.5 43.86 judge-resolved 58.3% 26.7%fell for 11 of 15 3.3 / 1.3agreement: medium 16.2ROCm 7.1 8.4 1.5 Q4_K_M other
211–219 Qwen 4B Chat V1.5 30.226.0–34.5 15.1 11.4 4.0 3.1 60.4 73.3 0 37.5 15.0 66.76 judge-resolved 50.0% 20.0%fell for 12 of 15 3.6 / 2.0agreement: medium 72.2ROCm 7.1 56.1 4 Q4_K_M tongyi-qianwen-research
212–223 Smollm2 360M Instruct 28.123.4–32.9 14.1 8.6 8.0 23.8 59.9 46.7 11.1 21.7 25.0 47.96 judge-resolved 41.7% 46.7%fell for 8 of 15 4.5 / 2.9agreement: medium 91.2ROCm 7.1 3.1 0.4 Q4_K_M apache-2.0
213–224 YI Coder 1.5B Chatcode model 26.522.1–31.5 21.7 17.1 0 57.7 36.1 53.3 5.6 10.8 35.0 22.96 judge-resolved 41.7% 33.3%fell for 10 of 15 3.3 / 1.6agreement: medium 88.7ROCm 7.1 26.9 1.5 Q4_K_M apache-2.0
216–225 Qwen 1.8B Chat V1.5 24.720.2–29.3 12.6 20.0 4.0 2.4 38.4 60.0 0 27.5 20.0 50.06 judge-resolved 66.7% 40.0%fell for 9 of 15 3.3 / 2.1agreement: high 82.5ROCm 7.1 27.1 1.8 Q4_K_M tongyi-qianwen-research
216–225 Stablelm2 1.6B Chat 24.420.2–29.0 10.6 8.6 4.0 26.1 31.6 40.0 0 16.7 42.5 50.06 judge-resolved 41.7% 13.3%fell for 13 of 15 3.6 / 2.6agreement: high 62.2ROCm 7.1 7.5 1.6 Q4_K_M stability-nc
219–225 Stable Code 3B Instructcode model 22.818.4–27.4 17.4 22.9 0 6.5 55.8 33.3 5.6 26.7 15.0 39.66 judge-resolved 41.7% 40.0%fell for 9 of 15 2.6 / 1.1agreement: medium 38.6ROCm 7.1 44.9 2.8 Q4_K_M stability-community
218–225 Deepseek Coder 1.3B Instructcode model 22.618.3–27.2 21.9 5.7 0 54.5 40.5 3.3 11.1 23.3 40.0 25.06 judge-resolved 33.3% 6.7%fell for 14 of 15 0.9 / 0.6agreement: high 64.1ROCm 7.1 26.9 1.3 Q4_K_M other
220–227 Smollm2 135M Instruct 20.116.2–24.6 11.2 2.9 0 14.1 36.1 36.7 5.6 15.0 25.0 45.86 judge-resolved 50.0% 6.7%fell for 14 of 15 3.0 / 1.5agreement: medium 201.2ROCm 7.1 1.7 0.1 Q4_K_M apache-2.0
225–227 Qwen 0.5B Chat V1.5 16.112.5–20.3 15.2 14.3 0 9.0 40.8 13.3 0 30.0 15.0 22.96 judge-resolved 33.3% 26.7%fell for 11 of 15 3.1 / 1.4agreement: medium 98.9ROCm 7.1 13.4 0.5 Q4_K_M tongyi-qianwen-research
225–227 Codegemma 2B Code V1.1code model 15.311.7–19.1 9.2 17.1 0 3.9 29.5 16.7 0 16.7 10.0 43.86 judge-resolved 33.3% 6.7%fell for 14 of 15 0.9 / 0.3agreement: high 43.3ROCm 7.1 3.3 2 Q4_K_M gemma
— Apertus-8B-Instruct-2509 —withheld — — — — — — — — — — — — 8.7ROCm 7.1 22.3 8.1 Q4_K_M apache-2.0
— Falcon2 11B —unranked: long_context partial: 12/18 items scored, 80% must be scored 22.9 16.0 16.9 57.7 70.0 0partial: 12/18 22.5 20.0 87.56 judge-resolved 33.3% 33.3%fell for 10 of 15 4.0 / 2.4agreement: medium 19.7ROCm 7.1 10.9 11 Q4_K_M other
— Falcon3 10B Instruct —withheld — — — — — — — — — — — — 21.2ROCm 7.1 28.3 10 Q4_K_M other
— Falcon3 1B Instruct —withheld — — — — — — — — — — — — 87.3ROCm 7.1 6.0 1 Q4_K_M other
— Falcon3 3B Instruct —withheld — — — — — — — — — — — — 57.9ROCm 7.1 14.0 3 Q4_K_M other
— Falcon3 7B Instruct —withheld — — — — — — — — — — — — 24.7ROCm 7.1 20.1 7 Q4_K_M other
— Internlm2 1.8B Chat V2.5 —withheld — — — — — — — — — — — — 73.2ROCm 7.1 14.3 1.8 Q4_K_M other
— Qwen2 Math 1.5B Instructmath model —withheld — — — — — — — — — — — — 61.1ROCm 7.1 2.1 1.5 Q4_K_M apache-2.0
— Qwen2 Math 7B Instructmath model —withheld — — — — — — — — — — — — 17.4ROCm 7.1 6.5 7 Q4_K_M apache-2.0

Withheld

These models have no rank and no score on this page. Their answers are published on their own pages, except for a model whose every tested version is withheld: it has no published results, and its page gives the reason only.