Open-weight model leaderboard

Which open-weight model is best for the machine you have? Each one is run locally through the same fixed battery, then scored by code wherever an answer can be checked.

228 models (236 versions listed, withheld ones included) on a battery of 270 items in 13 categories: 9 of them make up Overall (reasoning, math, coding, instruction following, knowledge & honesty, long context, multilingual, tool use, safety); the other 4 (pushback resistance, famous gotchas, writing, self-description) are shown on their own. Each model also gets a speed and memory profile on one laptop (Speed, hardware picker, best-model pages), from 12 runs (newest r20261105-bv1-2abe78). Read the methodology, including its limitations.

Who judges? Two paid models through OpenRouter: openai/gpt-5.6-luna and anthropic/claude-haiku-5.5 (deepseek/deepseek-v4.1-flash steps in when one of them is from the same company as the model under test). They read safety answers and score writing; everything else is checked by code, and their other scores and written reviews are advisory. A judge is never the model under test or from its family, and both judges’ scores are shown; judge means are never blended into Overall. Judges affect Overall only through Safety, in two ways: they decide a prompt that should be refused, and an answer to a harmless prompt that both judges rate 3 or lower out of 10 is scored 0 unless it was cut off (full methodology). About the judges.

New

All changes, or the Atom feed.

How much model does your memory buy?

Each dot is a model. The line is the best score reachable at each memory size. Memory is an estimate: the model file size plus about 20% for the context of a single conversation.

Hard score against the memory a model needs One dot per model. Horizontal axis: estimated memory need in GB, logarithmic scale. Vertical axis: Score on the hard questions (0 to 100). The orange line joins the 12 models that no model needing the same or less memory beats; only those are named, and every dot has a tooltip and a link to its model page. 0 20 40 60 80 100 0.0625 0.125 0.25 0.5 1 2 4 8 16 32 64 128 Estimated memory needed (GB, log scale) Hard (0 to 100) gemma-4-31B-it: Hard 94.7, needs about 22.4 GB (on the frontier)gemma-4-31B-it gemma-4-26B-A4B-it: Hard 92.0, needs about 21.6 GB (on the frontier) Gemma 4 12B IT: Hard 90.7, needs about 8.9 GB (on the frontier) Gemma4 E4B IT: Hard 84.7, needs about 7.9 GB (on the frontier)Gemma4 E4B IT Gemma4 E2B IT: Hard 79.8, needs about 5.5 GB (on the frontier)Gemma4 E2B IT Qwen3-4B: Hard 58.1, needs about 3.1 GB (on the frontier)Qwen3-4B Qwen3-4B-Instruct-2507: Hard 57.2, needs about 3.0 GB (on the frontier) Exaone3.5 2.4B Instruct: Hard 47.3, needs about 1.9 GB (on the frontier)Exaone3.5 2.4B Instruct Lfm2.5 Thinking 1.2B: Hard 45.2, needs about 0.9 GB (on the frontier) Granite4 350M H: Hard 25.2, needs about 0.4 GB (on the frontier)Granite4 350M H Smollm2 360M Instruct: Hard 14.1, needs about 0.3 GB (on the frontier)Smollm2 360M Instruct Smollm2 135M Instruct: Hard 11.2, needs about 0.1 GB (on the frontier) gpt-oss-120b: Hard 88.7, needs about 78.4 GB Muse Glimmer 30B: Hard 84.9, needs about 21.8 GB Qwen3.5-27B: Hard 78.8, needs about 19.9 GB Granite4.1 8B: Hard 76.8, needs about 6.4 GB gemma-3-27b-it: Hard 76.0, needs about 20.9 GB Flint 40B A2B: Hard 75.2, needs about 27.6 GB Laguna XS 2.1: Hard 74.4, needs about 24.3 GB Qwen3-30B-A3B-Instruct-2507: Hard 73.2, needs about 22.3 GB Qwen3-Coder-30B-A3B-Instruct: Hard 72.9, needs about 22.3 GB Cogito 14B V1 Preview Qwen: Hard 72.8, needs about 10.8 GB Qwen3.8-27B: Hard 72.5, needs about 20.2 GB Translategemma 12B IT: Hard 71.7, needs about 9.7 GB Llama 4 Scout 17B-16E Instruct: Hard 69.9, needs about 80.9 GB Ministral 3 8B Instruct 2512: Hard 68.8, needs about 7.2 GB Ornith 9B: Hard 67.9, needs about 6.7 GB Granite4.2 30B: Hard 67.1, needs about 21.3 GB Qwen3-Coder-Next: Hard 66.6, needs about 62.1 GB gemma-3-12b-it: Hard 66.5, needs about 9.8 GB Granite4 Small H: Hard 66.5, needs about 23.4 GB Nanbeige_Nanbeige4.3-33B: Hard 66.2, needs about 24.0 GB Qwen3-VL-8B-Instruct: Hard 66.1, needs about 7.4 GB Qwen3-30B-A3B: Hard 65.8, needs about 22.3 GB Nemotron 3.5 Lightning 30B A3B: Hard 65.7, needs about 30.5 GB Nemotron3 33B: Hard 65.6, needs about 33.2 GB Qwen2.5-Coder-32B-Instruct: Hard 64.9, needs about 23.8 GB Qwen3-32B: Hard 64.7, needs about 24.2 GB phi-4: Hard 64.2, needs about 10.9 GB Sage-A1.5-Q4_K_M: Hard 64.1, needs about 26.4 GB Qwen2.5-14B-Instruct: Hard 63.6, needs about 10.8 GB Devstral Small 2 24B Instruct (2512): Hard 63.2, needs about 18.2 GB Nemotron Cascade 2 30B A3B: Hard 62.7, needs about 29.1 GB Llama-3.3-70B-Instruct: Hard 62.6, needs about 51.0 GB NVIDIA Nemotron 3 Nano 30B-A3B: Hard 62.5, needs about 29.1 GB Ministral 3 14B Instruct (2512): Hard 62.4, needs about 10.9 GB Granite4.1 30B: Hard 62.2, needs about 21.0 GB Qwen2.5-32B-Instruct: Hard 61.8, needs about 23.8 GB Mistral Small 3.2 24B Instruct (2506): Hard 61.7, needs about 18.2 GB GLM-4.7-Flash: Hard 60.3, needs about 22.8 GB Qwen3.5-9B: Hard 59.9, needs about 6.8 GB Qwen3.6-35B-A3B: Hard 58.4, needs about 28.7 GB Qwen2.5-VL-32B-Instruct: Hard 58.3, needs about 25.4 GB Qwen3-VL-4B-Instruct: Hard 57.6, needs about 4.0 GB Qwen3.6-27B: Hard 57.1, needs about 20.9 GB Deepseek R1 8B Llama Distill: Hard 57.0, needs about 5.9 GB Nemotron 3.5 Bolt E4B: Hard 57.0, needs about 8.6 GB Qwen3.5-4B: Hard 55.8, needs about 3.2 GB Gemma3n E2B IT: Hard 55.7, needs about 6.7 GB Qwen3-8B: Hard 55.5, needs about 6.3 GB Granite4 Tiny H: Hard 54.3, needs about 5.0 GB Lfm2.5 8B A1B: Hard 54.2, needs about 6.2 GB Olmo 3.1 32B Instruct: Hard 54.2, needs about 23.4 GB Openthinker 7B V2: Hard 54.1, needs about 5.6 GB Cogito 8B V1 Preview Llama: Hard 54.1, needs about 5.9 GB Mistral Small 24B Instruct 2501: Hard 53.8, needs about 17.2 GB Qwen3-14B: Hard 52.9, needs about 11.1 GB Qwen2.5-Coder-14B-Instruct: Hard 52.3, needs about 10.8 GB Lfm2 24B: Hard 51.9, needs about 17.3 GB Granite4.2 8B: Hard 51.4, needs about 6.4 GB Qwen3-VL-32B-Instruct: Hard 51.0, needs about 25.1 GB Gemma 3n E4B IT: Hard 50.7, needs about 9.1 GB Ministral 3 3B Instruct 2512: Hard 50.3, needs about 3.6 GB Phi3 14B Medium 128K Instruct: Hard 49.6, needs about 10.3 GB Mistral Small3.1 24B Instruct 2503: Hard 49.6, needs about 18.6 GB Command R7B Arabic 7B 02 2025: Hard 49.5, needs about 6.1 GB Medgemma 27B IT: Hard 49.2, needs about 20.9 GB Gemma2 27B Instruct: Hard 48.9, needs about 20.0 GB gemma-3-4b-it: Hard 48.3, needs about 4.0 GB Phi4 Mini Reasoning 3.8B: Hard 47.9, needs about 3.8 GB Granite4 Micro H: Hard 46.9, needs about 2.3 GB Qwen2.5-7B-Instruct: Hard 44.7, needs about 5.6 GB Qwen3.5 2B: Hard 44.1, needs about 2.3 GB Qwen3 VL 2B Instruct: Hard 44.0, needs about 2.3 GB Qwen3-VL-30B-A3B-Instruct: Hard 43.9, needs about 23.5 GB Olmo-3-7B-Instruct: Hard 43.3, needs about 5.4 GB EXAONE-3.5-7.8B-Instruct: Hard 43.0, needs about 5.7 GB Qwen3 1.7B: Hard 42.8, needs about 1.7 GB Qwen2.5-Coder-7B-Instruct: Hard 42.2, needs about 5.6 GB granite-4.1-3b: Hard 42.0, needs about 2.5 GB Qwen3.5-35B-A3B: Hard 41.8, needs about 25.4 GB Spark-X2.6-4B: Hard 41.8, needs about 3.0 GB Granite4 3B: Hard 41.4, needs about 2.5 GB Command R 35B 08 2024: Hard 40.7, needs about 23.8 GB Llama-3.1-8B-Instruct: Hard 40.6, needs about 5.9 GB Granite3.3 8B: Hard 40.2, needs about 5.9 GB Devstral 24B Small 2505: Hard 40.1, needs about 17.2 GB Exaone Deep 7.8B: Hard 39.8, needs about 5.8 GB Command R7B (12-2024): Hard 39.7, needs about 6.1 GB Codestral 22B V0.1: Hard 39.6, needs about 16.0 GB Codegeex4 9B All: Hard 39.5, needs about 7.6 GB DeepSeek-Coder-V2-Lite-Instruct: Hard 39.0, needs about 12.4 GB medgemma-4b-it: Hard 38.6, needs about 4.0 GB Glm4 9B Chat: Hard 38.3, needs about 7.6 GB YI 9B Chat V1.5: Hard 37.9, needs about 6.4 GB YI Coder 9B Chat: Hard 37.8, needs about 6.4 GB Tulu3 8B: Hard 37.8, needs about 5.9 GB Aya Expanse 32B: Hard 37.7, needs about 23.8 GB Granite4 1B H: Hard 37.7, needs about 1.9 GB Mistral Nemo 12B Instruct 2407: Hard 37.7, needs about 9.0 GB Marco O1 7B: Hard 37.3, needs about 5.6 GB gemma-2-9b-it: Hard 36.8, needs about 6.9 GB Mathstral 7B V0.1: Hard 36.8, needs about 5.3 GB Mistral Small 22B Instruct 2409: Hard 36.8, needs about 16.0 GB Exaone3.5 32B Instruct: Hard 36.7, needs about 23.2 GB Deepcoder 14B Preview: Hard 36.7, needs about 10.8 GB Granite4.2 3B: Hard 36.3, needs about 2.7 GB Qwen2.5 1.5B Instruct: Hard 36.0, needs about 1.2 GB Internlm2 7B Chat V2.5: Hard 35.8, needs about 5.6 GB Meta-Llama-3-8B-Instruct: Hard 35.7, needs about 5.9 GB Aya Expanse 8B: Hard 35.5, needs about 6.1 GB YI 34B Chat V1.5: Hard 35.4, needs about 24.8 GB Deepcoder 1.5B Preview: Hard 35.2, needs about 1.3 GB Phi-4-mini-instruct: Hard 34.5, needs about 3.0 GB Codeqwen 7B Chat V1.5: Hard 34.2, needs about 5.6 GB Llama-3.2-3B-Instruct: Hard 33.5, needs about 2.4 GB Gemma2 2B Instruct: Hard 32.6, needs about 2.0 GB Cogito 3B V1 Preview Llama: Hard 32.6, needs about 2.6 GB Qwen2.5-VL-7B-Instruct: Hard 32.1, needs about 7.2 GB Granite3.1 Dense 8B Instruct: Hard 31.8, needs about 6.0 GB Granite3.2 8B Instruct: Hard 31.0, needs about 5.9 GB Deepseek Coder 33B Instruct: Hard 31.0, needs about 23.9 GB Granite3.3 2B: Hard 30.4, needs about 1.8 GB Deepscaler 1.5B Preview: Hard 30.4, needs about 1.3 GB Qwen2.5-Coder-3B-Instruct: Hard 29.9, needs about 2.3 GB Lfm3 1.7B: Hard 29.7, needs about 1.3 GB Gemma3 1B IT: Hard 29.3, needs about 1.0 GB Codegemma 7B Instruct V1.1: Hard 29.3, needs about 6.4 GB Granite3 Dense 8B Instruct: Hard 29.0, needs about 5.9 GB Qwen2.5 Coder 1.5B Instruct: Hard 28.6, needs about 1.2 GB Gemma 7B Instruct V1.1: Hard 28.5, needs about 6.4 GB Granite3.1 Dense 2B Instruct: Hard 27.0, needs about 1.9 GB Qwen3 VL 2B Thinking: Hard 26.7, needs about 2.3 GB Llama3.2 1B Instruct: Hard 26.6, needs about 1.0 GB Solar 10.7B Instruct V1: Hard 26.4, needs about 7.8 GB Smollm2 1.7B Instruct: Hard 26.0, needs about 1.3 GB Granite Code 3B Instruct: Hard 25.6, needs about 2.5 GB Smallthinker 3B Preview: Hard 25.5, needs about 2.5 GB Deepseek R1 1.5B Qwen Distill: Hard 25.5, needs about 1.3 GB Qwen2.5-3B-Instruct: Hard 25.4, needs about 2.3 GB Qwen3 0.6B: Hard 25.0, needs about 0.6 GB Granite3.1 Moe 3B Instruct: Hard 24.9, needs about 2.4 GB Aya 8B 23: Hard 24.9, needs about 6.1 GB Mistral 7B Instruct V0.3: Hard 24.9, needs about 5.3 GB Granite3.2 2B Instruct: Hard 23.9, needs about 1.8 GB Olmo2 7B 1124 Instruct: Hard 23.5, needs about 5.4 GB Exaone Deep 2.4B: Hard 23.0, needs about 1.9 GB Gemma 2B Instruct V1.1: Hard 22.5, needs about 1.9 GB Nemotron Mini 4B Instruct: Hard 22.4, needs about 3.2 GB Qwen2.5-VL-3B-Instruct: Hard 22.2, needs about 3.8 GB Qwen2.5 Coder 0.5B Instruct: Hard 22.0, needs about 0.5 GB Deepseek Coder 1.3B Instruct: Hard 21.9, needs about 1.0 GB Phi3 3.8B Mini 128K Instruct: Hard 21.9, needs about 2.9 GB Codellama 34B Instruct: Hard 21.9, needs about 24.3 GB YI Coder 1.5B Chat: Hard 21.7, needs about 1.2 GB Opencoder 1.5B Instruct: Hard 21.5, needs about 1.7 GB Sailor2 8B Chat: Hard 21.1, needs about 6.2 GB YI 6B Chat V1.5: Hard 21.0, needs about 4.4 GB Phi-3.5-mini-instruct: Hard 20.9, needs about 2.9 GB Phi-3-mini-4k-instruct: Hard 20.9, needs about 2.9 GB Lfm3 VL 2B: Hard 20.9, needs about 1.8 GB Deepseek Llm 7B Chat: Hard 20.9, needs about 5.0 GB Granite Code 8B Instruct: Hard 20.6, needs about 5.9 GB Granite3 Moe 3B Instruct: Hard 20.2, needs about 2.5 GB Granite3 Dense 2B Instruct: Hard 20.2, needs about 1.9 GB Qwen2 1.5B Instruct: Hard 20.1, needs about 1.2 GB Deepseek V2 16B Lite Chat: Hard 20.1, needs about 12.0 GB Stablelm Zephyr 3B: Hard 19.9, needs about 2.0 GB Granite3.1 Moe 1B Instruct: Hard 19.0, needs about 1.0 GB Mistral-7B-Instruct-v0.2: Hard 18.7, needs about 5.2 GB Granite3 Moe 1B Instruct: Hard 18.7, needs about 1.0 GB Qwen2.5 0.5B Instruct: Hard 18.2, needs about 0.5 GB Phi 2.7B Chat V2: Hard 17.5, needs about 2.2 GB Mixtral 8x7B Instruct V0.1: Hard 17.5, needs about 34.1 GB Stable Code 3B Instruct: Hard 17.4, needs about 2.0 GB Hermes-3-Llama-3.1-8B: Hard 16.9, needs about 5.9 GB Dolphin Phi 2.7B V2.6: Hard 16.4, needs about 2.2 GB Llama-2-7b-chat-hf: Hard 16.4, needs about 4.9 GB Qwen2 0.5B Instruct: Hard 15.6, needs about 0.5 GB Qwen 0.5B Chat V1.5: Hard 15.2, needs about 0.5 GB Qwen 4B Chat V1.5: Hard 15.1, needs about 3.0 GB Hermes3 3B Llama3.2: Hard 14.9, needs about 2.4 GB Stablelm2 12B Chat: Hard 14.7, needs about 8.9 GB Qwen 1.8B Chat V1.5: Hard 12.6, needs about 1.4 GB Stablelm2 1.6B Chat: Hard 10.6, needs about 1.2 GB Codegemma 2B Code V1.1: Hard 9.2, needs about 1.9 GB Sailor2 1B Chat: Hard 6.4, needs about 0.9 GB

best at each memory sizeother models. 197 of 236 model versions plotted; 39 left off (no score for this tab, or no file size on record). Hard questions are the ones half or fewer of the ranked models of 3B and over in the model's own run answered correctly; they separate models that all score near the top of Overall.

What we found

Every line is computed from the results below. A model is called clearly ahead only when its 95% interval says the gap is not chance; more best-model pages.

Who leads each category

The first row of each category leaderboard, on that page’s own rule. Scores can tie at one decimal; ties are counted, not hidden.

Best model for your machine

Top-scoring ranked model whose estimated memory need fits. All best-model pages, or enter your own memory.

1 model is waiting for a newer Ollama release that can load it (it crashes while loading in the version we use, on both GPU backends): Rnj 1 8B Instruct.

19 of these 228 models are community fine-tunes, listed in the ranking like every other model and labelled with the model each was made from. Originals only.

Compare two models

How to read this table

Overall is the unweighted mean of the 9 programmatic category scores marked overall in the battery definition (Reasoning, Math, Coding, Instruction following, Knowledge & honesty, Long context, Multilingual, Tool use, Safety), on a 0–100 scale. Models are ranked by Overall; a model that is missing any of those categories is listed unranked with the reason. Judge means are on their own 0–10 scale and are never blended into Overall. A dash means not scored, which is different from a score of 0.

Shown separately. Pushback is how often the model keeps a correct answer when the user pushes back, as a percentage. Gotchas is the share of famous trick questions answered correctly, with how many it fell for; these puzzles are widely published and probably memorised, so they do not count toward Overall. Writing has no programmatic answer: it shows the two judges' mean scores (0–10) and how closely they agree. The model's answers about itself are not scored; they are on the model page.

Speed is total tokens per second with 4 requests running at once and GPU memory on our server is what the benchmark server reported for the loaded model (1 to 8 answers at a time); it depends on that setup, is not what one user needs, and is not used to say what fits (the hardware picker estimates that from the model file). Speed and memory were measured on one laptop (see Speed). Size classes are by total parameters; the upper bound is inclusive, except Tiny, which is strictly under 3B.

Agreement (writing column, and on each model page) compares the two judges: high means a mean absolute score gap under 1.5, medium under 3, low otherwise; n/a means fewer than 3 items were scored by both judges.

Coverage rule. A category counts toward Overall only when at least 80% of its items have a definite programmatic score (for example 12 of 15; judge-resolved items included) and it has no pending item. A category below that shows its mean with a partial marker such as “partial: 1/15”, Overall shows a dash, and the model is listed unranked with the reason.

Judge-resolved and pending items. Where the refusal detector cannot call an item (for example a refusal followed by more text), the two judges decide: if they agree, the item is scored 1 or 0; if they split, it scores 0.5; if one judge is unavailable (unparsed or skipped), the other decides alone. Each of these is labelled judge-resolved. Until the judges have decided, the item is pending, and a model with any pending item is not ranked on safety. This is one of two places where judge scores enter the programmatic score; the other is the comply-quality gate, where an answer to a harmless prompt that both judges rate 3 or lower out of 10 is scored 0 unless it was cut off (how the gate works). Here, the detector cannot judge a refusal that is followed by more text, and dropping those items from the mean would let a model avoid a low Safety score by answering in a way the detector cannot classify. The cell shows a “judge-resolved” count and, when present, a “pending” count.

Integrity flag. A coding answer that touches the test harness is not run and scores 0; the model then carries an integrity flag next to its Coding score. Answers that use constructs outside the task are flagged on the model page. The score itself is unchanged by the site.

Programmatic scores (0–100), writing judge means (0–10), speed and memory per model, latest finished run. Click a column heading to sort.
Rank Model Overall Hard questions Reason Math Code Instr. Know. Long ctx Multi Tools Safety Pushback Gotchas Writing (judges) Speed (tok/s) GPU memory on our server (GB) Params (B) Quant Licence
1–5 gemma-4-31B-it 99.398.3–100.0 94.7 97.1 100.0 96.6 100.0 100.0 100.0 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 7.4 / 6.6agreement: medium 11.8ROCm 7.1 11.2 31.3 Q4_K_M apache-2.0
1–14 gpt-oss-120b 98.696.8–100.0 88.7 97.1 96.0 99.9 100.0 100.0 94.4 100.0 100.0 100.06 judge-resolved 91.7% 100.0%fell for 0 of 15 6.3 / 7.1agreement: high 15.8Vulkan 64.6 116.8 MXFP4 apache-2.0
1–12 Gemma 4 12B IT 98.697.0–99.8 90.7 94.3 100.0 98.2 100.0 100.0 94.4 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 7.4 / 6.1agreement: high 13.1ROCm 7.1 8.9 12 Q4_K_M apache-2.0
1–23 Muse Glimmer 30B 97.996.0–99.3 84.9 94.3 100.0 92.4 100.0 100.0 94.4 100.0 100.0 100.06 judge-resolved 100.0% 100.0%fell for 0 of 15 7.4 / 6.6agreement: high 8.8ROCm 7.1 18.3 30 Q4_K_M —
1–22 gemma-4-26B-A4B-it 97.895.8–99.3 92.0 94.3 100.0 96.6 100.0 100.0 88.9 100.0 100.0 100.06 judge-resolved 100.0% 100.0%fell for 0 of 15 6.9 / 6.8agreement: high 21.8ROCm 7.1 22.6 26 Q4_K_M apache-2.0
2–42 Qwen3-Coder-Nextcode model 96.494.2–98.3 66.6 88.6 92.0 97.3 95.2 100.0 94.4 100.0 100.0 100.06 judge-resolved 91.7% 100.0%fell for 0 of 15 6.8 / 5.8agreement: high 14.7Vulkan 52.2 79.7 Q4_K_M apache-2.0
3–35 Qwen3.5-27B 96.293.8–98.3 78.8 88.6 84.0 93.1 100.0 100.0 100.0 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 7.4 / 6.3agreement: medium 2.6ROCm 7.1 17.6 27.8 Q4_K_M apache-2.0
3–36 gemma-3-27b-it 95.893.7–97.5 76.0 91.4 96.0 85.4 94.5 96.7 100.0 100.0 100.0 97.96 judge-resolved 100.0% 93.3%fell for 1 of 15 7.5 / 6.3agreement: medium 4.7ROCm 7.1 22.1 27.4 Q4_K_M gemma
4–42 Qwen3-Coder-30B-A3B-Instructcode model 95.593.2–97.6 72.9 88.6 92.0 92.0 92.9 100.0 94.4 100.0 100.0 100.06 judge-resolved 83.3% 80.0%fell for 3 of 15 5.0 / 5.3agreement: high 26.1ROCm 7.1 31.7 30 Q4_K_M apache-2.0
2–51 Gemma4 E4B IT 95.593.0–97.7 84.7 85.7 96.0 89.9 97.2 100.0 94.4 100.0 100.0 95.86 judge-resolved 91.7% 93.3%fell for 1 of 15 5.8 / 6.1agreement: high 25.3ROCm 7.1 1.2 8 Q4_K_M apache-2.0
2–53 Qwen3.8-27B 95.492.9–97.8 72.5 88.6 80.0 96.9 98.9 100.0 94.4 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.1 / 6.1agreement: high 3.2Vulkan 17.6 27.8 Q4_K_M apache-2.0
3–58 Qwen3.6-27B 95.192.4–97.5 57.1 88.6 88.0 92.0 98.3 100.0 88.9 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.5 / 6.1agreement: high 3.5Vulkan 17.3 27.8 Q4_K_M apache-2.0
3–48 Qwen3-30B-A3B-Instruct-2507 95.092.5–97.2 73.2 88.6 96.0 93.2 90.3 100.0 88.9 98.3 100.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 6.8 / 5.8agreement: high 26.1ROCm 7.1 31.7 30 Q4_K_M apache-2.0
2–60 gpt-oss-20b 95.092.2–97.6 — 97.1 96.0 95.1 95.8 100.0 94.4 100.0 97.5 79.26 judge-resolved 100.0% 100.0%fell for 0 of 15 4.1 / 5.1agreement: high 31.5ROCm 7.1 13.5 20.9 MXFP4 apache-2.0
4–48 Qwen3-VL-8B-Instruct 94.992.4–97.1 66.1 91.4 92.0 77.7 98.5 100.0 94.4 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 5.8 / 5.1agreement: high 9.2ROCm 7.1 7.8 8.8 Q4_K_M apache-2.0
3–60 Qwen3.5-35B-A3B 94.892.1–97.1 41.8 85.7 92.0 87.1 93.5 100.0 100.0 100.0 95.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 7.0 / 6.6agreement: high 16.5Vulkan 21.7 35 Q4_K_M apache-2.0
4–60 Qwen3-VL-32B-Instruct 94.792.1–96.9 51.0 88.6 88.0 86.5 96.1 100.0 94.4 98.3 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 5.5 / 5.5agreement: medium 2.9Vulkan 24.5 33.4 Q4_K_M apache-2.0
3–63 Qwen3.6-35B-A3B 94.691.6–97.2 58.4 85.7 92.0 94.1 96.6 100.0 83.3 100.0 100.0 100.06 judge-resolved 91.7% 93.3%fell for 1 of 15 6.6 / 6.0agreement: high 17.8Vulkan 23.0 35 Q4_K_M apache-2.0
5–56 Sage-A1.5-Q4_K_M 94.591.8–96.7 64.1 88.6 88.0 84.6 94.6 100.0 94.4 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 7.1 / 6.1agreement: medium 21.2ROCm 7.1 21.8 36 Q4_K_M —
4–63 Llama-3.3-70B-Instruct 94.391.3–96.7 62.6 97.1 84.0 80.7 97.5 100.0 88.9 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.6 / 5.9agreement: high 3.4Vulkan 63.8 70.6 Q4_K_M llama3.3
5–61 Granite4 Small H 93.991.1–96.5 66.5 88.6 88.0 86.3 93.4 100.0 88.9 100.0 100.0 100.06 judge-resolved 100.0% 80.0%fell for 3 of 15 5.9 / 5.5agreement: high 12.4ROCm 7.1 23.9 32 Q4_K_M —
4–61 Gemma4 E2B IT 93.891.0–96.1 79.8 88.6 88.0 80.1 95.4 96.7 100.0 100.0 95.0 100.06 judge-resolved 91.7% 93.3%fell for 1 of 15 7.3 / 6.3agreement: medium 29.8ROCm 7.1 0.6 5 Q4_K_M apache-2.0
5–58 Qwen3-32B 93.690.9–96.3 64.7 85.7 88.0 83.8 96.3 100.0 88.9 100.0 100.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 6.1 / 6.3agreement: high 4.1ROCm 7.1 55.0 32.8 Q4_K_M apache-2.0
4–65 Llama 4 Scout 17B-16E Instruct 93.690.7–96.1 69.9 91.4 84.0 81.0 98.7 100.0 94.4 100.0 92.5 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 5.5 / 5.9agreement: medium 8.2Vulkan 70.4 109 Q4_K_M llama4
5–59 Qwen2.5-32B-Instruct 93.590.8–96.1 61.8 85.7 92.0 82.0 94.1 100.0 94.4 93.3 100.0 100.06 judge-resolved 100.0% 80.0%fell for 3 of 15 6.4 / 6.1agreement: medium 7.6ROCm 7.1 54.2 32.8 Q4_K_M apache-2.0
7–65 Qwen3-VL-30B-A3B-Instruct 92.990.1–95.5 43.9 82.9 88.0 87.8 88.4 100.0 88.9 100.0 100.0 100.06 judge-resolved 83.3% 80.0%fell for 3 of 15 5.6 / 5.9agreement: high 19.8Vulkan 20.6 30 Q4_K_M apache-2.0
8–60 Mistral Small 3.2 24B Instruct (2506) 92.990.3–95.2 61.7 82.9 96.0 77.2 85.4 100.0 94.4 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.8 / 6.3agreement: medium 9.4ROCm 7.1 36.5 24 Q4_K_M apache-2.0
5–64 Nemotron 3.5 Lightning 30B A3B 92.889.7–95.8 65.7 85.7 92.0 89.0 93.7 100.0 83.3 96.7 95.0 100.06 judge-resolved 91.7% 100.0%fell for 0 of 15 5.6 / 5.3agreement: high 17.3ROCm 7.1 25.4 30 Q4_K_M —
6–63 Granite4.2 30B 92.889.9–95.4 67.1 80.0 80.0 88.0 92.8 100.0 94.4 100.0 100.0 100.06 judge-resolved 91.7% 93.3%fell for 1 of 15 6.6 / 6.3agreement: high 8.2ROCm 7.1 52.3 30 Q4_K_M —
6–68 GLM-4.5-Air 92.889.7–95.6 — 82.9 96.0 87.5 94.3 100.0 83.3 95.0 100.0 95.86 judge-resolved 75.0% 93.3%fell for 1 of 15 6.4 / 6.0agreement: high 7.6Vulkan 76.8 110.5 Q4_K_M mit
8–65 Qwen3-30B-A3B 92.889.4–95.6 65.8 88.6 84.0 86.6 92.3 100.0 83.3 100.0 100.0 100.06 judge-resolved 91.7% 93.3%fell for 1 of 15 5.4 / 5.4agreement: high 25.5ROCm 7.1 31.7 30 Q4_K_M apache-2.0
8–66 gemma-3-12b-it 92.789.7–95.4 66.5 88.6 96.0 81.8 88.3 96.7 83.3 100.0 100.0 100.06 judge-resolved 91.7% 66.7%fell for 5 of 15 6.5 / 5.9agreement: high 9.5ROCm 7.1 13.1 12.2 Q4_K_M gemma
9–61 Ministral 3 14B Instruct (2512) 92.790.0–95.2 62.4 82.9 88.0 87.5 81.2 96.7 100.0 98.3 100.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 6.6 / 6.0agreement: high 15.0ROCm 7.1 30.0 14 Q4_K_M apache-2.0
6–65 QwQ 32B 92.689.8–95.4 — 94.3 88.0 60.6 95.1 100.0 100.0 100.0 100.0 95.86 judge-resolved 100.0% 100.0%fell for 0 of 15 6.3 / 5.8agreement: high 4.7ROCm 7.1 54.2 32.8 Q4_K_M apache-2.0
6–66 Granite4.1 30B 92.389.3–95.3 62.2 82.9 96.0 86.0 93.8 100.0 72.2 100.0 100.0 100.06 judge-resolved 91.7% 86.7%fell for 2 of 15 5.8 / 5.6agreement: high 5.0ROCm 7.1 52.3 30 Q4_K_M —
6–73 Granite4.1 8B 92.189.0–95.0 76.8 91.4 88.0 76.5 93.1 96.7 94.4 93.3 100.0 95.86 judge-resolved 91.7% 80.0%fell for 3 of 15 5.6 / 5.4agreement: high 15.9ROCm 7.1 27.1 8 Q4_K_M apache-2.0
6–65 Mistral Small 24B Instruct 2501 92.189.3–94.7 53.8 85.7 84.0 82.8 84.8 100.0 94.4 97.5 100.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 5.6 / 5.3agreement: high 10.6ROCm 7.1 36.5 24 Q4_K_M —
9–67 Devstral Small 2 24B Instruct (2512)code model 92.189.3–94.7 63.2 85.7 76.0 83.4 89.1 100.0 94.4 100.0 100.0 100.06 judge-resolved 100.0% 80.0%fell for 3 of 15 6.5 / 5.8agreement: high 8.9ROCm 7.1 36.5 24 Q4_K_M apache-2.0
11–69 Qwen3.5-9B 92.089.0–95.1 59.9 85.7 84.0 77.4 94.1 100.0 88.9 98.3 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.6 / 6.1agreement: medium 8.2ROCm 7.1 6.1 9.7 Q4_K_M apache-2.0
11–71 Qwen3-14B 91.788.5–94.7 52.9 91.4 88.0 73.5 93.6 100.0 88.9 90.0 100.0 100.06 judge-resolved 91.7% 80.0%fell for 3 of 15 6.5 / 6.0agreement: high 8.7ROCm 7.1 30.9 14.8 Q4_K_M apache-2.0
11–72 Qwen3-8B 91.588.4–94.5 55.5 94.3 88.0 70.6 87.7 100.0 83.3 100.0 100.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 5.8 / 5.3agreement: high 14.4ROCm 7.1 24.7 8.2 Q4_K_M apache-2.0
11–72 Qwen2.5-14B-Instruct 91.388.0–94.2 63.6 88.6 84.0 76.5 91.3 100.0 94.4 91.7 95.0 100.06 judge-resolved 91.7% 86.7%fell for 2 of 15 5.5 / 5.4agreement: high 59.0ROCm 7.1 34.8 14.8 Q4_K_M apache-2.0
6–73 Laguna XS 2.1 91.287.7–94.4 74.4 91.4 88.0 88.5 93.3 96.7 94.4 98.3 70.0 100.06 judge-resolved 91.7% 93.3%fell for 1 of 15 6.1 / 5.6agreement: high 40.0ROCm 7.1 23.2 — Q4_K_M —
12–74 Qwen2.5-Coder-32B-Instructcode model 90.587.3–93.8 64.9 80.0 64.0 86.5 94.8 100.0 94.4 95.0 100.0 100.06 judge-resolved 91.7% 80.0%fell for 3 of 15 5.4 / 5.8agreement: high 7.6ROCm 7.1 54.2 32.8 Q4_K_M apache-2.0
9–76 Nanbeige_Nanbeige4.3-33B 90.486.8–93.7 66.2 88.6 80.0 90.1 80.7 100.0 88.9 90.0 95.0 100.06 judge-resolved 100.0% 80.0%fell for 3 of 15 6.1 / 5.3agreement: high 7.4ROCm 7.1 48.9 33 Q4_K_M —
8–82 Nemotron 3.5 Bolt E4B 90.386.3–93.9 57.0 82.9 96.0 73.2 100.0 100.0 72.2 98.3 90.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.4 / 6.0agreement: high 13.0ROCm 7.1 5.7 8 Q4_K_M —
13–73 Qwen3-4B-Instruct-2507 90.387.2–93.3 57.2 82.9 80.0 69.7 94.2 96.7 88.9 100.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.6 / 5.8agreement: medium 43.2ROCm 7.1 12.5 4 Q4_K_M apache-2.0
11–79 Cogito 14B V1 Preview Qwenfine-tune of Qwen 2.5 90.386.8–93.4 72.8 82.9 84.0 78.8 89.4 96.7 88.9 91.7 100.0 100.06 judge-resolved 83.3% 80.0%fell for 3 of 15 5.8 / 5.8agreement: high 10.2ROCm 7.1 34.9 14 Q4_K_M apache-2.0
10–75 Medgemma 27B ITfine-tune of Gemma 3medical model 90.286.7–93.6 49.2 82.9 84.0 77.0 89.5 100.0 88.9 95.0 95.0 100.06 judge-resolved 91.7% 86.7%fell for 2 of 15 6.3 / 6.3agreement: high 5.3ROCm 7.1 22.1 27 Q4_K_M —
12–77 Mistral Small3.1 24B Instruct 2503 90.186.9–93.1 49.6 88.6 76.0 73.3 84.2 100.0 88.9 100.0 100.0 100.06 judge-resolved 100.0% 66.7%fell for 5 of 15 5.6 / 5.5agreement: high 10.3ROCm 7.1 36.6 24 Q4_K_M —
12–79 Ornith 9B 90.086.5–93.1 67.9 91.4 84.0 63.0 91.9 100.0 94.4 85.0 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.5 / 5.9agreement: high 10.5ROCm 7.1 5.9 9 Q4_K_M mit
12–76 Qwen3-VL-4B-Instruct 89.986.4–93.1 57.6 91.4 92.0 61.2 96.6 90.0 83.3 96.7 100.0 97.96 judge-resolved 75.0% 86.7%fell for 2 of 15 5.5 / 4.8agreement: medium 16.8ROCm 7.1 5.5 4.4 Q4_K_M apache-2.0
19–77 Qwen2.5-VL-32B-Instruct 89.786.5–92.8 58.3 88.6 84.0 86.6 69.5 100.0 88.9 90.0 100.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 4.9 / 5.6agreement: high 2.7ROCm 7.1 54.9 33.5 Q4_K_M apache-2.0
13–80 Translategemma 12B ITfine-tune of Gemma 3translation model 89.686.4–92.9 71.7 82.9 92.0 78.9 86.3 100.0 66.7 100.0 100.0 100.06 judge-resolved 91.7% 86.7%fell for 2 of 15 6.0 / 5.8agreement: high 11.5ROCm 7.1 13.1 12 Q4_K_M gemma
10–82 Olmo 3.1 32B Instruct 89.686.0–93.2 54.2 82.9 84.0 84.7 95.2 96.7 77.8 90.0 95.0 100.06 judge-resolved 91.7% 100.0%fell for 0 of 15 6.8 / 6.0agreement: high 7.4ROCm 7.1 27.7 32 Q4_K_M —
18–81 Qwen3-4B 89.285.6–92.4 58.1 88.6 88.0 69.0 91.0 100.0 77.8 88.3 100.0 100.06 judge-resolved 100.0% 100.0%fell for 0 of 15 5.6 / 5.6agreement: high 41.3ROCm 7.1 12.7 4 Q4_K_M apache-2.0
19–85 DeepSeek-R1-Distill-Qwen-32B 89.186.0–91.9 — 97.1 92.0 53.6 75.7 100.0 100.0 96.7 95.0 91.76 judge-resolved 100.0% 93.3%fell for 1 of 15 5.8 / 5.9agreement: high 4.8ROCm 7.1 19.9 32.8 Q4_K_M mit
14–90 Flint 40B A2B 89.085.1–92.6 75.2 71.4 100.0 93.4 96.6 90.0 77.8 91.7 80.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 6.5 / 6.3agreement: high 31.9ROCm 7.1 24.3 40 NVFP4 —
19–81 Ornith-1.5-9B 89.085.4–92.4 52.1 91.4 84.0 67.9 85.2 100.0 88.9 88.3 95.0 100.06 judge-resolved 91.7% 100.0%fell for 0 of 15 6.4 / 5.9agreement: high 8.0ROCm 7.1 6.4 9.7 Q4_K_M mit
23–86 Devstral 24B Small 2505code model 88.985.6–92.0 40.1 77.1 76.0 70.4 84.2 100.0 94.4 98.3 100.0 100.06 judge-resolved 83.3% 80.0%fell for 3 of 15 5.9 / 5.9agreement: high 8.0ROCm 7.1 36.5 24 Q4_K_M —
17–84 GLM-4.7-Flash 88.985.2–92.5 60.3 77.1 92.0 81.6 86.7 100.0 77.8 100.0 85.0 100.06 judge-resolved 100.0% 66.7%fell for 5 of 15 6.0 / 5.6agreement: high 19.3ROCm 7.1 26.0 31.2 Q4_K_M mit
19–84 Qwen3.5-4B 88.784.9–92.4 55.8 85.7 88.0 71.5 90.9 100.0 83.3 98.3 85.0 95.86 judge-resolved 83.3% 86.7%fell for 2 of 15 5.9 / 5.8agreement: medium 14.6ROCm 7.1 3.6 4.7 Q4_K_M apache-2.0
19–86 Ministral 3 8B Instruct 2512 88.785.2–92.1 68.8 82.9 80.0 78.4 77.8 96.7 94.4 88.3 100.0 100.06 judge-resolved 100.0% 80.0%fell for 3 of 15 6.3 / 5.9agreement: high 22.2ROCm 7.1 23.8 8 Q4_K_M apache-2.0
30–86 Qwen2.5-Coder-14B-Instructcode model 87.784.0–91.3 52.3 82.9 72.0 82.7 84.2 93.3 88.9 85.8 100.0 100.06 judge-resolved 83.3% 73.3%fell for 4 of 15 5.5 / 5.4agreement: high 15.9ROCm 7.1 34.8 14.8 Q4_K_M apache-2.0
26–96 Lfm2 24B 87.383.2–91.1 51.9 77.1 96.0 72.9 96.6 100.0 66.7 91.7 85.0 100.06 judge-resolved 91.7% 80.0%fell for 3 of 15 6.0 / 5.4agreement: high 29.5ROCm 7.1 15.0 24 Q4_K_M —
32–101 Nemotron3 33B 86.282.0–90.0 65.6 80.0 84.0 83.7 92.6 100.0 83.3 55.0 97.5 100.06 judge-resolved 91.7% 86.7%fell for 2 of 15 6.4 / 6.0agreement: high 18.9ROCm 7.1 25.0 33 Q4_K_M —
44–97 Gemma 3n E4B IT 86.182.4–89.6 50.7 82.9 92.0 65.3 83.4 96.7 66.7 93.3 95.0 100.06 judge-resolved 91.7% 73.3%fell for 4 of 15 6.8 / 5.6agreement: medium 20.0ROCm 7.1 4.2 8 Q4_K_M gemma
39–99 phi-4 85.981.9–89.8 64.2 91.4 84.0 87.0 73.2 96.7 77.8 93.3 70.0 100.06 judge-resolved 100.0% 86.7%fell for 2 of 15 5.6 / 5.8agreement: high 8.9ROCm 7.1 36.2 14.7 Q4_K_M mit
36–106 Nemotron Cascade 2 30B A3B 85.681.2–89.7 62.7 68.6 80.0 77.9 97.8 90.0 77.8 78.3 100.0 100.06 judge-resolved 91.7% 80.0%fell for 3 of 15 5.9 / 5.4agreement: high 17.1ROCm 7.1 24.4 30 Q4_K_M —
39–103 Gemma2 27B Instruct 85.681.4–89.4 48.9 65.7 68.0 67.5 98.2 96.7 83.3 95.0 100.0 95.86 judge-resolved 100.0% 80.0%fell for 3 of 15 5.4 / 5.5agreement: high 5.5ROCm 7.1 27.0 27 Q4_K_M —
45–100 Qwen2.5-7B-Instruct 85.681.4–89.3 44.7 77.1 76.0 63.2 88.4 100.0 77.8 91.7 100.0 95.86 judge-resolved 91.7% 66.7%fell for 5 of 15 4.6 / 4.8agreement: high 28.3ROCm 7.1 8.7 7.6 Q4_K_M apache-2.0
38–106 DeepSeek-R1-Distill-Qwen-14B 85.381.2–89.2 — 91.4 88.0 53.0 80.8 100.0 77.8 89.2 100.0 87.56 judge-resolved 100.0% 93.3%fell for 1 of 15 5.1 / 4.9agreement: high 10.0ROCm 7.1 9.0 14.8 Q4_K_M mit
46–106 Exaone3.5 32B Instruct 85.080.9–88.8 36.7 77.1 76.0 71.5 89.8 96.7 66.7 87.5 100.0 100.06 judge-resolved 91.7% 80.0%fell for 3 of 15 5.3 / 5.5agreement: high 7.7ROCm 7.1 53.9 32 Q4_K_M —
43–106 Spark-X2.6-4B 85.080.9–89.1 41.8 85.7 84.0 65.6 78.7 90.0 77.8 98.3 85.0 100.06 judge-resolved 83.3% 73.3%fell for 4 of 15 5.1 / 5.1agreement: high 38.7ROCm 7.1 5.3 4 Q4_K_M —
47–108 Mistral Small 22B Instruct 2409 84.680.7–88.5 36.8 74.3 72.0 70.1 78.9 96.7 88.9 94.2 95.0 91.76 judge-resolved 83.3% 66.7%fell for 5 of 15 5.9 / 5.1agreement: medium 6.1ROCm 7.1 44.0 22 Q4_K_M —
50–106 Qwen2.5-Coder-7B-Instructcode model 84.580.6–88.1 42.2 82.9 72.0 71.8 72.6 93.3 77.8 90.0 100.0 100.06 judge-resolved 58.3% 80.0%fell for 3 of 15 4.9 / 4.5agreement: high 28.2ROCm 7.1 12.5 7.6 Q4_K_M apache-2.0
49–113 NVIDIA Nemotron 3 Nano 30B-A3B 83.979.5–88.5 62.5 65.7 84.0 86.0 96.5 96.7 61.1 75.0 90.0 100.06 judge-resolved 83.3% 93.3%fell for 1 of 15 5.8 / 5.8agreement: medium 17.0ROCm 7.1 24.4 30 Q4_K_M other
47–105 Cogito 8B V1 Preview Llamafine-tune of Llama 3.1 83.979.6–87.8 54.1 80.0 64.0 56.1 93.1 96.7 83.3 81.7 100.0 100.06 judge-resolved 100.0% 73.3%fell for 4 of 15 5.6 / 5.4agreement: high 24.6ROCm 7.1 22.4 8 Q4_K_M llama3.2
47–111 Qwen3-4B-Thinking-2507 83.580.2–87.0 — 74.3 76.0 26.7 88.6 100.0 94.4 91.7 100.0 100.06 judge-resolved 100.0% 93.3%fell for 1 of 15 6.0 / 5.4agreement: high 32.1ROCm 7.1 2.5 4 Q4_K_M apache-2.0
52–118 Olmo-3-7B-Instruct 82.978.8–86.9 43.3 80.0 96.0 57.7 96.7 93.3 44.4 86.7 95.0 95.86 judge-resolved 91.7% 93.3%fell for 1 of 15 5.0 / 5.4agreement: high 22.8ROCm 7.1 14.4 7.3 Q4_K_M apache-2.0
58–123 gemma-3-4b-it 82.177.6–86.5 48.3 71.4 84.0 71.4 85.9 80.0 61.1 100.0 85.0 100.06 judge-resolved 75.0% 73.3%fell for 4 of 15 5.6 / 5.5agreement: high 38.6ROCm 7.1 4.5 4.3 Q4_K_M gemma
59–122 EXAONE-3.5-7.8B-Instruct 82.077.6–86.4 43.0 80.0 72.0 70.1 80.8 100.0 55.6 88.3 100.0 91.76 judge-resolved 75.0% 80.0%fell for 3 of 15 4.6 / 5.1agreement: high 23.0ROCm 7.1 13.6 7.8 Q4_K_M exaone-nc
59–121 Qwen2.5-VL-7B-Instruct 82.077.8–86.3 32.1 74.3 68.0 64.4 78.8 96.7 77.8 83.3 95.0 100.06 judge-resolved 91.7% 60.0%fell for 6 of 15 4.9 / 4.1agreement: high 28.4ROCm 7.1 13.2 8.3 Q4_K_M apache-2.0
53–118 Command R7B Arabic 7B 02 2025 81.977.5–86.6 49.5 74.3 68.0 67.4 84.8 93.3 77.8 90.8 85.0 95.86 judge-resolved 83.3% 66.7%fell for 5 of 15 5.9 / 5.5agreement: high 16.2ROCm 7.1 10.5 7 Q4_K_M cc-by-nc-4.0
59–128 Granite4.2 8B 81.977.2–86.3 51.4 65.7 68.0 85.1 94.1 100.0 55.6 88.3 80.0 100.06 judge-resolved 83.3% 66.7%fell for 5 of 15 6.4 / 6.3agreement: high 24.8ROCm 7.1 27.1 8 Q4_K_M —
56–122 Deepseek R1 8B Llama Distill 81.776.9–86.1 57.0 91.4 96.0 52.8 82.3 90.0 72.2 71.7 100.0 79.26 judge-resolved 100.0% 86.7%fell for 2 of 15 4.9 / 4.8agreement: high 18.2ROCm 7.1 22.4 8 Q4_K_M llama3.1
62–123 Phi-4-mini-instruct 81.677.1–86.0 34.5 77.1 76.0 61.0 84.1 96.7 66.7 81.7 95.0 95.86 judge-resolved 83.3% 80.0%fell for 3 of 15 4.5 / 4.6agreement: high 43.7ROCm 7.1 11.4 3.8 Q4_K_M mit
57–133 Falcon3 10B Instruct 81.376.6–85.8 — 80.0 60.0 64.3 84.8 96.7 72.2 83.3 90.0 100.06 judge-resolved 75.0% 86.7%fell for 2 of 15 5.3 / 5.3agreement: high 20.9ROCm 7.1 6.3 10 Q4_K_M other
65–125 Llama-3.1-8B-Instruct 81.076.5–85.2 40.6 60.0 56.0 65.6 92.6 90.0 83.3 81.7 100.0 100.06 judge-resolved 83.3% 66.7%fell for 5 of 15 5.8 / 5.3agreement: high 23.6ROCm 7.1 13.7 8 Q4_K_M llama3.1
61–124 Ministral 3 3B Instruct 2512 80.976.5–85.2 50.3 74.3 88.0 60.6 67.5 93.3 61.1 92.5 95.0 95.86 judge-resolved 83.3% 53.3%fell for 7 of 15 6.1 / 5.3agreement: high 156.8ROCm 7.1 16.6 3 Q4_K_M apache-2.0
66–129 granite-4.1-3b 80.775.9–85.3 42.0 65.7 80.0 62.6 93.9 90.0 66.7 86.7 85.0 95.86 judge-resolved 100.0% 73.3%fell for 4 of 15 5.3 / 5.0agreement: high 49.0ROCm 7.1 7.8 3.4 Q4_K_M apache-2.0
69–131 gemma-2-9b-it 79.975.5–84.2 36.8 68.6 60.0 51.8 92.6 90.0 61.1 100.0 95.0 100.06 judge-resolved 91.7% 73.3%fell for 4 of 15 5.9 / 5.5agreement: high 11.7ROCm 7.1 14.7 9.2 Q4_K_M gemma
64–127 Gemma3n E2B IT 79.975.2–84.7 55.7 68.6 76.0 61.4 89.7 83.3 72.2 80.0 87.5 100.06 judge-resolved 91.7% 53.3%fell for 7 of 15 6.8 / 5.9agreement: medium 26.1ROCm 7.1 3.4 5 Q4_K_M gemma
67–135 DeepSeek-R1-0528-Qwen3-8B 79.675.5–83.9 — 77.1 48.0 42.1 84.3 96.7 77.8 95.0 95.0 100.06 judge-resolved 91.7% 86.7%fell for 2 of 15 5.5 / 5.3agreement: high 18.4ROCm 7.1 5.2 8.2 Q4_K_M mit
66–129 Granite3.3 8B 79.474.7–83.9 40.2 62.9 76.0 56.6 85.7 96.7 66.7 72.5 97.5 100.06 judge-resolved 91.7% 73.3%fell for 4 of 15 5.4 / 5.1agreement: high 25.9ROCm 7.1 26.9 8 Q4_K_M apache-2.0
64–132 Granite4 Tiny H 79.174.1–84.2 54.3 74.3 68.0 80.8 79.7 86.7 66.7 80.0 80.0 95.86 judge-resolved 91.7% 60.0%fell for 6 of 15 5.0 / 5.1agreement: high 40.9ROCm 7.1 6.1 7 Q4_K_M apache-2.0
69–136 DeepSeek-Coder-V2-Lite-Instructcode model 79.074.3–83.5 39.0 65.7 56.0 72.5 77.9 83.3 77.8 86.7 95.0 95.86 judge-resolved 100.0% 46.7%fell for 8 of 15 4.4 / 4.5agreement: high 4.6ROCm 7.1 46.8 15.7 Q4_K_M other
72–137 Aya Expanse 32B 78.874.6–82.9 37.7 54.3 48.0 64.7 81.3 100.0 72.2 93.3 95.0 100.06 judge-resolved 75.0% 66.7%fell for 5 of 15 5.3 / 4.9agreement: high 13.5ROCm 7.1 31.4 32 Q4_K_M —
72–140 Granite4.2 3B 78.273.7–83.1 36.3 74.3 56.0 61.6 91.2 90.0 61.1 75.0 95.0 100.06 judge-resolved 91.7% 86.7%fell for 2 of 15 5.0 / 5.3agreement: high 50.1ROCm 7.1 13.2 3 Q4_K_M —
71–142 SmolLM3-3B 78.073.0–83.1 32.7 71.4 72.0 57.7 86.0 96.7 77.8 75.0 82.5 83.36 judge-resolved 83.3% 60.0%fell for 6 of 15 5.5 / 5.0agreement: high 50.8ROCm 7.1 12.0 3.1 Q4_K_M apache-2.0
67–136 Phi3 14B Medium 128K Instruct 78.073.2–82.6 49.6 77.1 56.0 61.4 71.6 90.0 88.9 69.2 87.5 100.06 judge-resolved 66.7% 73.3%fell for 4 of 15 5.5 / 4.9agreement: medium 17.0ROCm 7.1 — 14 Q4_K_M mit
68–146 Olmo-3-7B-Think 77.773.1–82.5 — 62.9 48.0 40.0 98.3 100.0 88.9 90.0 80.0 91.76 judge-resolved 66.7% 80.0%fell for 3 of 15 4.8 / 5.0agreement: high 24.0ROCm 7.1 4.5 7.3 Q4_K_M apache-2.0
68–137 Qwen3 VL 2B Instruct 77.472.5–82.1 44.0 65.7 64.0 49.2 89.8 93.3 66.7 75.8 100.0 91.76 judge-resolved 75.0% 60.0%fell for 6 of 15 5.0 / 3.6agreement: medium 26.2ROCm 7.1 3.6 2 Q4_K_M apache-2.0
76–148 Qwen2.5-3B-Instruct 76.872.0–81.6 25.4 71.4 72.0 48.7 82.7 90.0 61.1 75.0 90.0 100.06 judge-resolved 91.7% 53.3%fell for 7 of 15 4.6 / 4.6agreement: high 44.9ROCm 7.1 4.7 3.1 Q4_K_M qwen-research
70–140 Openthinker 7B V2fine-tune of Qwen 2.5 76.872.1–81.6 54.1 88.6 88.0 52.6 62.4 100.0 55.6 87.5 75.0 81.36 judge-resolved 83.3% 86.7%fell for 2 of 15 4.1 / 2.9agreement: high 30.1ROCm 7.1 12.5 7 Q4_K_M apache-2.0
72–139 Granite4 Micro H 76.672.0–81.4 46.9 62.9 60.0 68.8 90.0 90.0 38.9 83.3 100.0 95.86 judge-resolved 91.7% 73.3%fell for 4 of 15 5.0 / 4.9agreement: high 30.9ROCm 7.1 4.0 3 Q4_K_M apache-2.0
73–141 Exaone3.5 2.4B Instruct 76.370.9–81.3 47.3 82.9 76.0 64.7 73.6 90.0 50.0 62.5 95.0 91.76 judge-resolved 75.0% 66.7%fell for 5 of 15 4.9 / 5.3agreement: high 256.0ROCm 7.1 11.9 2.4 Q4_K_M exaone-nc
71–150 Falcon3 7B Instruct 76.271.6–81.3 — 71.4 76.0 58.7 80.2 93.3 55.6 67.5 87.5 95.86 judge-resolved 100.0% 80.0%fell for 3 of 15 5.3 / 4.8agreement: high 28.4ROCm 7.1 4.6 7 Q4_K_M other
77–141 Qwen3 1.7B 76.071.3–80.9 42.8 68.6 72.0 45.5 86.8 76.7 55.6 87.5 100.0 91.76 judge-resolved 75.0% 73.3%fell for 4 of 15 4.9 / 4.5agreement: high 43.5ROCm 7.1 16.5 1.7 Q4_K_M apache-2.0
78–142 Mistral Nemo 12B Instruct 2407 75.970.6–80.8 37.7 57.1 56.0 57.3 80.7 96.7 77.8 85.0 85.0 87.56 judge-resolved 75.0% 86.7%fell for 2 of 15 4.4 / 4.4agreement: high 12.5ROCm 7.1 29.2 12 Q4_K_M apache-2.0
84–150 Llama-3.2-3B-Instruct 75.771.0–80.4 33.5 68.6 52.0 52.9 91.7 86.7 55.6 81.7 92.5 100.06 judge-resolved 83.3% 60.0%fell for 6 of 15 5.8 / 5.3agreement: high 49.8ROCm 7.1 9.9 3.2 Q4_K_M llama3.2
80–143 Lfm2.5 8B A1B 75.770.8–80.5 54.2 85.7 88.0 47.3 91.1 96.7 61.1 81.7 30.0 100.06 judge-resolved 100.0% 80.0%fell for 3 of 15 4.6 / 3.9agreement: high 41.6ROCm 7.1 5.5 8 Q4_K_M other
79–152 YI 34B Chat V1.5 75.270.7–79.9 35.4 71.4 60.0 50.8 76.6 100.0 38.9 83.3 100.0 95.86 judge-resolved 75.0% 80.0%fell for 3 of 15 4.9 / 4.9agreement: high 7.1ROCm 7.1 28.8 34 Q4_K_M —
79–153 Command R 35B 08 2024 75.270.6–79.8 40.7 48.6 32.0 63.9 84.1 93.3 88.9 84.2 90.0 91.76 judge-resolved 83.3% 46.7%fell for 8 of 15 5.6 / 5.3agreement: high 7.7ROCm 7.1 42.2 35 Q4_K_M —
81–144 Qwen3.5 2B 75.169.7–80.3 44.1 74.3 80.0 52.5 82.1 80.0 77.8 63.3 70.0 95.86 judge-resolved 75.0% 66.7%fell for 5 of 15 4.8 / 4.5agreement: high 32.2ROCm 7.1 1.9 2 Q4_K_M apache-2.0
87–152 Command R7B (12-2024) 74.870.1–79.1 39.7 71.4 72.0 59.4 82.9 86.7 83.3 96.7 25.0 95.86 judge-resolved 75.0% 80.0%fell for 3 of 15 5.6 / 5.0agreement: high 14.0ROCm 7.1 10.4 8 Q4_K_M cc-by-nc-4.0
80–154 Codestral 22B V0.1code model 74.869.7–79.5 39.6 45.7 48.0 72.9 74.4 96.7 83.3 83.3 85.0 83.36 judge-resolved 83.3% 60.0%fell for 6 of 15 4.9 / 4.6agreement: high 6.5ROCm 7.1 44.0 22 Q4_K_M —
87–152 Meta-Llama-3-8B-Instruct 74.769.9–79.5 35.7 51.4 36.0 60.0 89.3 90.0 83.3 81.7 85.0 95.86 judge-resolved 75.0% 60.0%fell for 6 of 15 6.5 / 5.6agreement: high 27.3ROCm 7.1 9.2 8 Q4_K_M llama3
84–146 Deepcoder 14B Previewfine-tune of DeepSeek-R1-Distill-Qwen 14Bcode model 74.569.5–79.0 36.7 74.3 56.0 20.0 86.8 100.0 66.7 75.0 100.0 91.76 judge-resolved 91.7% 60.0%fell for 6 of 15 5.0 / 5.5agreement: high 10.3ROCm 7.1 34.9 14 Q4_K_M mit
88–152 medgemma-4b-itmedical model 74.369.3–79.2 38.6 65.7 72.0 60.3 78.6 80.0 50.0 77.5 85.0 100.06 judge-resolved 83.3% 66.7%fell for 5 of 15 5.4 / 5.0agreement: high 42.2ROCm 7.1 4.5 4.3 Q4_K_M other
84–148 Lfm2.5 Thinking 1.2B 74.369.5–79.1 45.2 82.9 80.0 42.1 80.8 83.3 38.9 71.7 97.5 91.76 judge-resolved 50.0% 80.0%fell for 3 of 15 4.6 / 3.8agreement: high 122.0ROCm 7.1 1.1 1.2 Q4_K_M other
85–145 Granite4 3B 74.169.0–79.2 41.4 62.9 56.0 64.7 87.3 76.7 55.6 78.3 90.0 95.86 judge-resolved 83.3% 80.0%fell for 3 of 15 5.4 / 5.0agreement: high 177.1ROCm 7.1 13.2 3 Q4_K_M apache-2.0
85–145 Granite3.2 8B Instruct 74.169.5–78.9 31.0 45.7 56.0 58.5 82.7 96.7 50.0 81.7 100.0 95.86 judge-resolved 75.0% 73.3%fell for 4 of 15 5.4 / 5.3agreement: high 97.7ROCm 7.1 26.9 8 Q4_K_M apache-2.0
82–155 DeepSeek-R1-Distill-Qwen-7B 73.668.9–78.3 — 77.1 96.0 37.0 77.4 83.3 44.4 80.8 100.0 66.76 judge-resolved 75.0% 80.0%fell for 3 of 15 5.0 / 4.5agreement: high 20.7ROCm 7.1 4.7 7.6 Q4_K_M mit
87–153 Glm4 9B Chat 73.368.0–78.5 38.3 60.0 60.0 60.0 82.3 90.0 66.7 85.0 60.0 95.86 judge-resolved 83.3% 73.3%fell for 4 of 15 4.9 / 4.5agreement: high 15.9ROCm 7.1 11.8 9 Q4_K_M other
87–152 Tulu3 8Bfine-tune of Llama 3.1 73.067.9–77.9 37.8 60.0 52.0 52.7 83.6 90.0 61.1 67.5 90.0 100.06 judge-resolved 91.7% 46.7%fell for 8 of 15 5.8 / 5.4agreement: high 28.2ROCm 7.1 22.4 8 Q4_K_M llama3.1
90–154 dolphin-2.9.1-yi-1.5-34b 73.067.9–78.2 31.7 71.4 56.0 65.7 72.7 93.3 66.7 66.7 85.0 79.26 judge-resolved 91.7% 66.7%fell for 5 of 15 4.9 / 5.1agreement: high 7.2ROCm 7.1 37.2 34.4 Q4_K_M apache-2.0
88–151 Granite3.1 Dense 8B Instruct 72.967.8–77.6 31.8 57.1 52.0 64.2 74.3 100.0 55.6 61.7 95.0 95.86 judge-resolved 50.0% 66.7%fell for 5 of 15 6.1 / 5.5agreement: high 97.2ROCm 7.1 27.0 8 Q4_K_M apache-2.0
90–154 Aya Expanse 8B 72.567.5–77.3 35.5 54.3 40.0 50.0 79.5 90.0 66.7 72.5 100.0 100.06 judge-resolved 91.7% 80.0%fell for 3 of 15 5.8 / 5.4agreement: high 16.3ROCm 7.1 14.2 8 Q4_K_M cc-by-nc-4.0
89–155 Qwen2.5-Coder-3B-Instructcode model 72.567.7–77.5 29.9 57.1 44.0 60.2 70.0 80.0 72.2 78.3 95.0 95.86 judge-resolved 75.0% 60.0%fell for 6 of 15 4.8 / 3.8agreement: high 49.3ROCm 7.1 4.7 3.1 Q4_K_M qwen-research
90–157 Qwen2.5-VL-3B-Instruct 72.367.2–77.6 22.2 71.4 64.0 42.6 74.7 100.0 66.7 60.0 80.0 91.76 judge-resolved 83.3% 60.0%fell for 6 of 15 4.9 / 4.0agreement: high 51.4ROCm 7.1 5.5 3.8 Q4_K_M —
92–157 Qwen2-VL-7B-Instruct 72.267.4–77.0 21.6 45.7 64.0 49.7 70.7 86.7 66.7 66.7 100.0 100.06 judge-resolved 66.7% 60.0%fell for 6 of 15 4.8 / 4.5agreement: high 16.1ROCm 7.1 12.8 8.3 Q4_K_M apache-2.0
89–158 Lfm3 VL 2B 72.267.2–77.0 20.9 62.9 60.0 42.2 88.1 86.7 44.4 91.7 95.0 79.26 judge-resolved 100.0% 80.0%fell for 3 of 15 5.5 / 4.9agreement: high 58.7ROCm 7.1 2.0 2 Q4_K_M —
91–159 Medgemma1.5 4B ITfine-tune of Gemma 3medical model 71.666.6–76.4 — 65.7 48.0 56.9 52.2 86.7 72.2 74.2 92.5 95.86 judge-resolved 100.0% 73.3%fell for 4 of 15 5.4 / 4.8agreement: high 30.6ROCm 7.1 5.9 4 Q4_K_M other
97–157 Phi4 Mini Reasoning 3.8B 70.965.9–76.0 47.9 94.3 96.0 31.8 44.3 90.0 66.7 75.0 55.0 85.46 judge-resolved 100.0% 93.3%fell for 1 of 15 3.6 / 2.5agreement: high 139.1ROCm 7.1 20.7 3.8 Q4_K_M mit
103–156 YI 9B Chat V1.5 70.565.4–75.3 37.9 68.6 64.0 57.5 73.7 90.0 22.2 75.0 87.5 95.86 judge-resolved 83.3% 60.0%fell for 6 of 15 4.4 / 4.5agreement: high 24.6ROCm 7.1 8.6 9 Q4_K_M apache-2.0
97–157 Marco O1 7Bfine-tune of Qwen 2 7B 70.565.1–76.0 37.3 71.4 64.0 64.7 46.9 76.7 72.2 69.2 90.0 79.26 judge-resolved 91.7% 66.7%fell for 5 of 15 4.5 / 3.4agreement: medium 23.7ROCm 7.1 12.5 7 Q4_K_M apache-2.0
104–157 Cogito 3B V1 Preview Llamafine-tune of Llama 3.2 70.465.4–75.3 32.6 45.7 40.0 49.3 81.6 86.7 61.1 76.7 92.5 100.06 judge-resolved 66.7% 53.3%fell for 7 of 15 4.9 / 5.3agreement: high 157.9ROCm 7.1 17.4 3 Q4_K_M llama3.2
96–162 Lfm3 1.7B 70.265.3–75.3 29.7 74.3 76.0 61.5 77.9 90.0 33.3 83.3 40.0 95.86 judge-resolved 100.0% 93.3%fell for 1 of 15 4.6 / 5.0agreement: high 58.2ROCm 7.1 1.6 1.7 Q4_K_M —
97–165 Olmo2 13B 1124 Instruct 69.664.5–74.5 — 60.0 52.0 40.8 88.6 96.7 33.3 80.0 75.0 100.06 judge-resolved 100.0% 53.3%fell for 7 of 15 5.8 / 5.3agreement: high 15.5ROCm 7.1 21.7 13 Q4_K_M apache-2.0
109–159 Codegeex4 9B Allcode model 68.763.2–73.8 39.5 48.6 36.0 60.5 75.2 93.3 77.8 63.3 80.0 83.36 judge-resolved 83.3% 33.3%fell for 10 of 15 4.4 / 3.4agreement: high 16.3ROCm 7.1 11.8 9 Q4_K_M other
111–159 Granite3.3 2B 68.663.8–73.8 30.4 57.1 52.0 50.4 76.3 90.0 44.4 57.5 90.0 100.06 judge-resolved 83.3% 46.7%fell for 8 of 15 5.0 / 5.4agreement: high 58.5ROCm 7.1 12.6 2 Q4_K_M apache-2.0
111–160 Qwen2.5 1.5B Instruct 68.563.2–73.7 36.0 62.9 52.0 49.9 59.4 93.3 66.7 48.3 92.5 91.76 judge-resolved 75.0% 60.0%fell for 6 of 15 4.6 / 4.0agreement: high 43.8ROCm 7.1 5.1 1.5 Q4_K_M apache-2.0
112–160 Gemma2 2B Instruct 68.262.9–73.3 32.6 42.9 44.0 42.4 84.6 86.7 55.6 80.0 77.5 100.06 judge-resolved 83.3% 66.7%fell for 5 of 15 5.1 / 5.1agreement: medium 39.9ROCm 7.1 4.7 2 Q4_K_M gemma
112–160 Mathstral 7B V0.1fine-tune of Mistral 7Bmath model 68.062.4–73.5 36.8 65.7 68.0 56.6 52.3 86.7 55.6 80.0 70.0 77.16 judge-resolved 83.3% 60.0%fell for 6 of 15 4.8 / 3.4agreement: medium 16.8ROCm 7.1 22.0 7 Q4_K_M apache-2.0
109–173 Hermes-3-Llama-3.1-8B 67.662.4–73.1 16.9 54.3 52.0 42.3 73.8 93.3 44.4 71.7 85.0 91.76 judge-resolved 75.0% 66.7%fell for 5 of 15 5.8 / 5.1agreement: high 21.3ROCm 7.1 13.7 8 Q4_K_M llama3
117–161 Granite3 Dense 8B Instruct 67.662.8–72.3 29.0 54.3 44.0 59.2 76.4 93.3 33.3 52.5 95.0 100.06 judge-resolved 91.7% 73.3%fell for 4 of 15 4.8 / 4.6agreement: high 14.5ROCm 7.1 10.5 8 Q4_K_M apache-2.0
116–161 Internlm2 7B Chat V2.5 67.462.1–72.6 35.8 65.7 60.0 48.7 69.6 60.0 72.2 45.0 85.0 100.06 judge-resolved 83.3% 46.7%fell for 8 of 15 4.9 / 4.6agreement: high 28.3ROCm 7.1 22.2 7 Q4_K_M other
110–174 Phi-3.5-mini-instruct 67.162.0–72.3 20.9 60.0 48.0 38.9 60.2 90.0 72.2 46.7 87.5 100.06 judge-resolved 83.3% 80.0%fell for 3 of 15 4.8 / 4.5agreement: high 41.6ROCm 7.1 28.4 3.8 Q4_K_M mit
119–162 Granite4 1B H 66.861.4–72.1 37.7 48.6 52.0 59.9 85.1 76.7 55.6 43.3 80.0 100.06 judge-resolved 75.0% 53.3%fell for 7 of 15 4.9 / 3.6agreement: high 50.1ROCm 7.1 3.4 1 Q8_0 apache-2.0
108–178 Qwen2 Math 7B Instructmath model 66.060.5–71.3 — 71.4 76.0 39.1 68.7 66.7 33.3 77.5 90.0 70.86 judge-resolved 75.0% 60.0%fell for 6 of 15 4.6 / 4.0agreement: high 19.4ROCm 7.1 4.7 7 Q4_K_M apache-2.0
120–170 Exaone Deep 7.8B 65.760.0–71.5 39.8 85.7 88.0 55.4 16.9 86.7 61.1 50.8 80.0 66.76 judge-resolved 91.7% 93.3%fell for 1 of 15 5.0 / 3.3agreement: medium 28.9ROCm 7.1 22.3 7.8 Q4_K_M exaone-nc
125–167 Olmo2 7B 1124 Instruct 65.460.6–70.1 23.5 51.4 52.0 29.3 84.4 93.3 27.8 60.0 90.0 100.06 judge-resolved 83.3% 46.7%fell for 8 of 15 4.9 / 4.9agreement: high 16.9ROCm 7.1 21.6 7 Q4_K_M apache-2.0
113–181 Falcon3 3B Instruct 64.859.4–70.6 — 51.4 52.0 40.9 76.8 73.3 66.7 50.8 80.0 91.76 judge-resolved 66.7% 66.7%fell for 5 of 15 5.3 / 4.9agreement: high 60.3ROCm 7.1 2.0 3 Q4_K_M other
132–172 Granite3.2 2B Instruct 63.858.7–69.0 23.9 42.9 52.0 48.4 75.5 80.0 50.0 43.3 95.0 87.56 judge-resolved 75.0% 46.7%fell for 8 of 15 4.8 / 4.9agreement: high 32.1ROCm 7.1 12.6 2 Q4_K_M apache-2.0
125–182 Apertus-8B-Instruct-2509 63.658.2–68.8 — 42.9 32.0 33.0 76.3 83.3 50.0 75.0 80.0 100.06 judge-resolved 83.3% 46.7%fell for 8 of 15 4.9 / 4.3agreement: high 32.9Vulkan 22.3 8.1 Q4_K_M apache-2.0
129–182 Phi-3-mini-4k-instruct 63.458.3–68.1 20.9 77.1 56.0 35.1 70.7 96.7 11.1 52.5 80.0 91.76 judge-resolved 66.7% 60.0%fell for 6 of 15 5.5 / 5.6agreement: high 44.5ROCm 7.1 8.9 3.8 Q4_K_M mit
137–176 Phi3 3.8B Mini 128K Instruct 62.958.0–67.7 21.9 54.3 40.0 38.6 60.2 96.7 50.0 31.7 95.0 100.06 judge-resolved 91.7% 53.3%fell for 7 of 15 4.5 / 4.5agreement: high 96.1ROCm 7.1 54.1 3.8 Q4_K_M mit
136–187 Codellama 34B Instructcode model 61.256.0–66.3 21.9 42.9 16.0 38.7 75.1 73.3 66.7 50.0 92.5 95.86 judge-resolved 41.7% 33.3%fell for 10 of 15 4.5 / 3.6agreement: high 7.4ROCm 7.1 46.7 34 Q4_K_M —
143–184 Smallthinker 3B Previewfine-tune of Qwen 2.5 3B 61.155.8–66.6 25.5 57.1 68.0 29.8 27.8 80.0 38.9 76.7 80.0 91.76 judge-resolved 91.7% 60.0%fell for 6 of 15 4.3 / 2.9agreement: high 53.2ROCm 7.1 7.2 3 Q4_K_M qwen-research
147–183 Gemma 7B Instruct V1.1 60.855.5–66.1 28.5 40.0 24.0 26.5 76.0 83.3 61.1 56.7 80.0 100.06 judge-resolved 91.7% 60.0%fell for 6 of 15 5.1 / 4.6agreement: high 15.2ROCm 7.1 36.1 7 Q4_K_M gemma
147–184 Granite3.1 Dense 2B Instruct 60.655.0–66.1 27.0 45.7 36.0 46.6 73.6 76.7 50.0 45.0 80.0 91.76 judge-resolved 58.3% 40.0%fell for 9 of 15 4.5 / 4.8agreement: high 56.8ROCm 7.1 12.6 2 Q4_K_M apache-2.0
147–185 Qwen2.5 Coder 1.5B Instructcode model 60.554.9–66.0 28.6 65.7 44.0 52.8 58.8 60.0 66.7 30.8 82.5 83.36 judge-resolved 66.7% 53.3%fell for 7 of 15 4.5 / 3.3agreement: medium 77.0ROCm 7.1 5.1 1.5 Q4_K_M apache-2.0
148–186 Mistral 7B Instruct V0.3 59.854.7–64.7 24.9 37.1 16.0 40.9 70.3 93.3 61.1 41.7 90.0 87.56 judge-resolved 66.7% 53.3%fell for 7 of 15 5.4 / 5.1agreement: high 17.5ROCm 7.1 22.0 7 Q4_K_M apache-2.0
148–186 Granite3.1 Moe 3B Instruct 59.854.7–65.0 24.9 37.1 32.0 32.6 69.2 83.3 61.1 41.7 85.0 95.86 judge-resolved 58.3% 66.7%fell for 5 of 15 4.3 / 4.6agreement: high 93.0ROCm 7.1 10.9 3 Q4_K_M apache-2.0
150–186 Codegemma 7B Instruct V1.1code model 59.554.8–64.2 29.3 45.7 24.0 55.5 78.1 70.0 5.6 70.0 95.0 91.76 judge-resolved 75.0% 46.7%fell for 8 of 15 5.0 / 5.0agreement: high 52.6ROCm 7.1 36.1 7 Q4_K_M gemma
143–191 Mistral-7B-Instruct-v0.2 59.254.3–64.2 18.7 40.0 12.0 25.3 75.1 90.0 72.2 42.5 80.0 95.86 judge-resolved 91.7% 60.0%fell for 6 of 15 4.8 / 4.9agreement: high 29.0ROCm 7.1 22.0 7.2 Q4_K_M apache-2.0
150–186 YI Coder 9B Chatcode model 59.053.7–64.5 37.8 31.4 16.0 81.3 65.3 83.3 44.4 54.2 95.0 60.46 judge-resolved 75.0% 53.3%fell for 7 of 15 3.8 / 3.9agreement: high 24.6ROCm 7.1 18.5 9 Q4_K_M apache-2.0
151–186 Solar 10.7B Instruct V1 58.753.7–63.8 26.4 45.7 16.0 34.3 69.9 86.7 38.9 58.3 95.0 83.36 judge-resolved 83.3% 33.3%fell for 10 of 15 5.8 / 5.1agreement: high 13.6ROCm 7.1 13.1 10.7 Q4_K_M —
152–187 Granite3 Dense 2B Instruct 58.353.1–63.6 20.2 48.6 24.0 42.2 63.4 86.7 33.3 48.3 82.5 95.86 judge-resolved 75.0% 40.0%fell for 9 of 15 4.4 / 4.4agreement: high 43.5ROCm 7.1 4.4 2 Q4_K_M apache-2.0
152–188 Deepseek V2 16B Lite Chat 58.253.2–63.4 20.1 34.3 40.0 41.6 70.2 70.0 33.3 50.8 87.5 95.86 judge-resolved 58.3% 40.0%fell for 9 of 15 5.3 / 4.9agreement: high 5.9ROCm 7.1 46.8 16 Q4_K_M other
152–187 YI 6B Chat V1.5 58.052.8–63.5 21.0 45.7 48.0 19.8 70.4 73.3 33.3 51.7 92.5 87.56 judge-resolved 75.0% 60.0%fell for 6 of 15 4.9 / 4.9agreement: medium 120.2ROCm 7.1 5.9 6 Q4_K_M apache-2.0
150–187 Llama3.2 1B Instruct 58.052.3–63.7 26.6 37.1 40.0 41.7 76.9 76.7 38.9 55.0 72.5 83.36 judge-resolved 50.0% 13.3%fell for 13 of 15 5.1 / 4.0agreement: high 114.5ROCm 7.1 5.4 1 Q4_K_M llama3.2
152–188 Sailor2 8B Chatfine-tune of Qwen 2.5 57.351.7–62.8 21.1 54.3 52.0 19.5 52.2 80.0 55.6 81.7 35.0 85.45 judge-resolved 75.0% 66.7%fell for 5 of 15 5.0 / 4.1agreement: high 26.3ROCm 7.1 14.2 8 Q4_K_M —
154–188 Gemma3 1B IT 56.851.8–62.2 29.3 60.0 64.0 42.5 71.9 63.3 22.2 52.5 35.0 100.06 judge-resolved 58.3% 53.3%fell for 7 of 15 5.4 / 4.9agreement: high 80.1ROCm 7.1 1.5 1 Q4_K_M gemma
150–194 Mixtral 8x7B Instruct V0.1 56.851.3–62.0 17.5 42.9 24.0 20.1 62.6 83.3 72.2 41.7 85.0 79.26 judge-resolved 83.3% 73.3%fell for 4 of 15 4.6 / 5.1agreement: high 8.5ROCm 7.1 46.5 56 Q4_K_M —
155–189 Aya 8B 23 55.951.1–60.7 24.9 34.3 0 30.7 74.7 76.7 61.1 39.2 95.0 91.76 judge-resolved 66.7% 40.0%fell for 9 of 15 4.6 / 4.8agreement: high 16.4ROCm 7.1 14.2 8 F16 cc-by-nc-4.0
155–192 Deepcoder 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bcode model 55.850.3–61.0 35.2 80.0 88.0 34.2 68.6 56.7 16.7 65.8 67.5 25.06 judge-resolved 66.7% 80.0%fell for 3 of 15 3.9 / 2.8agreement: high 66.9ROCm 7.1 5.1 1.5 Q4_K_M mit
154–195 Qwen 14B Chat V1.5 54.849.8–59.4 — 48.6 40.0 9.7 76.9 93.3 11.1 52.5 65.0 95.86 judge-resolved 66.7% 60.0%fell for 6 of 15 4.8 / 4.4agreement: high 13.9ROCm 7.1 62.6 14 Q4_K_M other
156–196 Exaone Deep 2.4B 54.748.9–60.5 23.0 82.9 72.0 36.7 13.9 63.3 50.0 38.3 72.5 62.56 judge-resolved 75.0% 93.3%fell for 1 of 15 3.4 / 2.3agreement: high 65.1ROCm 7.1 11.9 2.4 Q4_K_M exaone-nc
158–194 Deepseek Llm 7B Chat 54.349.3–59.8 20.9 37.1 24.0 37.4 66.8 76.7 27.8 45.8 77.5 95.86 judge-resolved 25.0% 40.0%fell for 9 of 15 4.4 / 3.6agreement: high 24.8ROCm 7.1 20.3 7 Q4_K_M —
161–198 Qwen3 VL 2B Thinking 53.748.3–59.1 26.7 60.0 52.0 0 77.2 73.3 33.3 63.3 82.5 41.74 judge-resolved 50.0% 53.3%fell for 7 of 15 3.4 / 3.1agreement: high 26.4ROCm 7.1 3.6 2 Q4_K_M apache-2.0
159–202 Codellama 7B Instructcode model 53.047.8–57.8 — 37.1 4.0 36.0 68.1 76.7 61.1 21.7 85.0 87.56 judge-resolved 50.0% 40.0%fell for 9 of 15 4.6 / 4.4agreement: high 27.2ROCm 7.1 38.8 7 Q4_K_M llama2
159–203 Codellama 13B Instructcode model 52.547.2–57.8 — 31.4 12.0 22.5 73.8 83.3 50.0 33.3 82.5 83.36 judge-resolved 50.0% 66.7%fell for 5 of 15 4.6 / 3.8agreement: high 16.2ROCm 7.1 61.7 13 Q4_K_M llama2
168–198 Deepscaler 1.5B Previewfine-tune of DeepSeek-R1-Distill-Qwen 1.5Bmath model 52.146.6–57.7 30.4 68.6 92.0 26.7 66.9 50.0 22.2 56.7 65.0 20.84 judge-resolved 75.0% 66.7%fell for 5 of 15 3.3 / 2.4agreement: high 40.0ROCm 7.1 5.1 1.5 Q4_K_M mit
171–198 Nemotron Mini 4B Instruct 52.047.4–56.9 22.4 34.3 12.0 34.6 72.4 83.3 11.1 47.5 85.0 87.56 judge-resolved 50.0% 46.7%fell for 8 of 15 4.4 / 3.9agreement: high 167.6ROCm 7.1 6.6 4 Q4_K_M other
159–207 Internlm2 1.8B Chat V2.5 51.946.1–57.5 — 31.4 28.0 48.5 56.6 70.0 50.0 29.2 70.0 83.36 judge-resolved 41.7% 33.3%fell for 10 of 15 4.1 / 2.9agreement: medium 85.6ROCm 7.1 1.2 1.8 Q4_K_M other
162–207 Magistral Small (2506) 49.844.5–55.3 — 34.3 48.0 10.0 34.8 56.7 44.4 70.0 100.0 50.03 judge-resolved 33.3% 26.7%fell for 11 of 15 4.1 / 3.6agreement: high 10.5ROCm 7.1 14.3 24 Q4_K_M apache-2.0
175–203 Qwen2 1.5B Instruct 49.744.1–55.3 20.1 28.6 32.0 30.6 61.6 63.3 27.8 48.3 80.0 75.06 judge-resolved 66.7% 60.0%fell for 6 of 15 4.1 / 3.6agreement: high 49.8ROCm 7.1 — 1.5 Q4_K_M apache-2.0
176–204 Codeqwen 7B Chat V1.5code model 49.444.2–54.7 34.2 25.7 8.0 69.4 72.1 40.0 38.9 35.0 95.0 60.46 judge-resolved 66.7% 40.0%fell for 9 of 15 4.0 / 2.8agreement: high 25.2ROCm 7.1 13.6 7 Q4_K_M —
177–205 Deepseek R1 1.5B Qwen Distill 48.943.5–54.7 25.5 71.4 76.0 24.4 66.2 53.3 22.2 47.5 62.5 16.73 judge-resolved 75.0% 73.3%fell for 4 of 15 3.9 / 3.0agreement: high 50.6ROCm 7.1 5.1 1.5 Q4_K_M mit
177–206 Hermes3 3B Llama3.2fine-tune of Llama 3.2 48.542.5–54.3 14.9 37.1 28.0 11.3 76.4 63.3 50.0 43.3 60.0 66.76 judge-resolved 83.3% 33.3%fell for 10 of 15 4.5 / 4.0agreement: high 50.7ROCm 7.1 17.4 3 Q4_K_M llama3
179–206 Granite3 Moe 3B Instruct 47.742.3–53.1 20.2 37.1 20.0 31.6 67.2 76.7 27.8 36.7 45.0 87.56 judge-resolved 50.0% 53.3%fell for 7 of 15 4.3 / 3.5agreement: high 218.4ROCm 7.1 4.3 3 Q4_K_M apache-2.0
180–206 Smollm2 1.7B Instruct 47.141.7–52.7 26.0 28.6 8.0 35.3 79.5 63.3 33.3 35.0 70.0 70.86 judge-resolved 58.3% 40.0%fell for 9 of 15 4.8 / 3.6agreement: high 53.3ROCm 7.1 14.2 1.7 Q4_K_M apache-2.0
184–208 Granite4 350M H 45.640.7–50.7 25.2 20.0 0 36.0 79.3 76.7 16.7 43.3 80.0 58.36 judge-resolved 41.7% 20.0%fell for 12 of 15 3.6 / 2.4agreement: medium 128.2ROCm 7.1 1.2 0.4 Q8_0 apache-2.0
181–211 Llama2 13B Chat 45.340.2–50.6 — 45.7 8.0 10.1 47.8 73.3 27.8 32.5 75.0 87.56 judge-resolved 66.7% 53.3%fell for 7 of 15 5.6 / 4.8agreement: high 16.4ROCm 7.1 21.5 13 Q4_K_M llama2
180–212 Falcon3 1B Instruct 44.339.8–49.3 — 40.0 20.0 18.7 64.9 70.0 11.1 34.2 40.0 100.06 judge-resolved 66.7% 33.3%fell for 10 of 15 3.8 / 2.8agreement: high 107.4ROCm 7.1 1.1 1 Q4_K_M other
183–211 Qwen 7B Chat V1.5 44.139.1–48.9 — 45.7 16.0 5.9 71.9 76.7 11.1 45.0 35.0 89.66 judge-resolved 75.0% 66.7%fell for 5 of 15 4.3 / 3.9agreement: high 23.2ROCm 7.1 39.2 7 Q4_K_M other
185–209 Dolphin Phi 2.7B V2.6fine-tune of Phi-2 44.038.9–49.4 16.4 31.4 40.0 28.6 69.8 70.0 5.6 15.0 65.0 70.86 judge-resolved 58.3% 53.3%fell for 7 of 15 4.3 / 2.9agreement: medium 38.4ROCm 7.1 7.2 2.7 Q4_K_M other
185–210 Qwen3 0.6B 44.038.8–49.4 25.0 37.1 40.0 32.5 74.3 20.0 11.1 48.3 82.5 50.06 judge-resolved 41.7% 33.3%fell for 10 of 15 3.8 / 2.9agreement: high 261.1ROCm 7.1 15.6 0.6 Q4_K_M apache-2.0
182–212 Llama-2-7b-chat-hf 43.839.0–48.8 16.4 25.7 4.0 18.6 72.1 66.7 16.7 33.3 65.0 91.76 judge-resolved 66.7% 33.3%fell for 10 of 15 4.5 / 4.0agreement: high 26.7ROCm 7.1 12.8 6.7 Q4_K_M llama2
185–210 Stablelm Zephyr 3Bfine-tune of StableLM 3B 43.738.4–49.3 19.9 45.7 16.0 29.3 60.1 50.0 22.2 34.2 65.0 70.86 judge-resolved 50.0% 20.0%fell for 12 of 15 4.4 / 3.9agreement: high 39.6ROCm 7.1 12.5 2.8 Q4_K_M stability-community
185–212 Granite3.1 Moe 1B Instruct 43.637.9–49.1 19.0 31.4 12.0 38.1 59.0 63.3 33.3 24.2 62.5 68.86 judge-resolved 58.3% 40.0%fell for 9 of 15 4.3 / 3.5agreement: high 149.5ROCm 7.1 7.5 1 Q4_K_M apache-2.0
186–212 Qwen2.5 0.5B Instruct 43.337.4–49.4 18.2 28.6 36.0 26.5 54.9 56.7 38.9 23.3 62.5 62.56 judge-resolved 58.3% 40.0%fell for 9 of 15 3.5 / 2.5agreement: high 116.8ROCm 7.1 2.2 0.5 Q4_K_M apache-2.0
190–213 Stablelm2 12B Chat 42.438.1–46.7 14.7 22.9 8.0 45.0 47.0 70.0 5.6 20.0 67.5 95.86 judge-resolved 50.0% 33.3%fell for 10 of 15 4.4 / 3.9agreement: high 6.7ROCm 7.1 14.0 12 Q4_K_M stability-nc
188–213 Gemma 2B Instruct V1.1 41.936.8–47.5 22.5 25.7 20.0 25.2 66.6 60.0 27.8 37.5 35.0 79.26 judge-resolved 50.0% 26.7%fell for 11 of 15 4.0 / 4.0agreement: high 43.4ROCm 7.1 3.3 2 Q4_K_M gemma
193–215 Granite Code 8B Instructcode model 40.735.3–46.7 20.6 40.0 28.0 55.7 53.7 63.3 27.8 41.7 10.0 45.86 judge-resolved 58.3% 66.7%fell for 5 of 15 4.6 / 3.4agreement: high 15.0ROCm 7.1 10.0 8 Q4_K_M apache-2.0
195–214 Phi 2.7B Chat V2 39.935.5–44.2 17.5 34.3 0 34.7 60.3 83.3 0 18.3 42.5 85.46 judge-resolved 33.3% 40.0%fell for 9 of 15 3.6 / 2.6agreement: high 38.2ROCm 7.1 7.2 2.7 Q4_K_M mit
197–216 Granite Code 3B Instructcode model 38.032.7–43.5 25.6 37.1 24.0 45.6 54.2 56.7 16.7 42.5 15.0 50.06 judge-resolved 58.3% 40.0%fell for 9 of 15 3.9 / 2.4agreement: medium 107.3ROCm 7.1 7.7 3 Q4_K_M apache-2.0
201–216 Qwen2 0.5B Instruct 36.030.9–41.2 15.6 31.4 8.0 9.8 50.2 43.3 22.2 23.3 77.5 58.36 judge-resolved 41.7% 26.7%fell for 11 of 15 3.9 / 2.6agreement: high 137.5ROCm 7.1 2.2 0.5 Q4_K_M apache-2.0
203–217 Granite3 Moe 1B Instruct 35.531.0–40.5 18.7 17.1 8.0 27.6 59.1 36.7 11.1 20.8 60.0 79.26 judge-resolved 58.3% 33.3%fell for 10 of 15 3.6 / 3.3agreement: high 82.4ROCm 7.1 2.3 1 Q4_K_M apache-2.0
198–219 Deepseek Coder 6.7B Instructcode model 34.729.7–40.0 — 8.6 0 68.5 53.9 10.0 61.1 21.7 55.0 33.36 judge-resolved 50.0% 33.3%fell for 10 of 15 1.3 / 1.0agreement: high 22.5ROCm 7.1 38.8 6.7 Q4_K_M other
203–221 Deepseek Coder 33B Instructcode model 33.429.0–38.2 31.0 31.4 12.0 67.2 46.2 6.7 0 25.8 70.0 41.76 judge-resolved 58.3% 53.3%fell for 7 of 15 1.1 / 1.4agreement: high 5.9ROCm 7.1 53.5 33 Q4_K_M —
203–222 Qwen2 Math 1.5B Instructmath model 33.128.0–38.2 — 65.7 72.0 17.5 14.4 33.3 11.1 37.5 17.5 29.26 judge-resolved 83.3% 40.0%fell for 9 of 15 3.0 / 1.1agreement: medium 62.0ROCm 7.1 1.0 1.5 Q4_K_M apache-2.0
208–218 Qwen2.5 Coder 0.5B Instructcode model 32.827.5–38.2 22.0 22.9 8.0 35.9 49.5 20.0 33.3 31.7 65.0 29.26 judge-resolved 50.0% 33.3%fell for 10 of 15 3.1 / 1.9agreement: high 172.5ROCm 7.1 2.2 0.5 Q4_K_M apache-2.0
209–219 Opencoder 1.5B Instructcode model 31.927.6–36.4 21.5 2.9 0 57.3 50.8 20.0 11.1 24.2 77.5 43.86 judge-resolved 58.3% 26.7%fell for 11 of 15 3.3 / 1.3agreement: medium 16.2ROCm 7.1 8.4 1.5 Q4_K_M other
211–219 Qwen 4B Chat V1.5 30.226.0–34.5 15.1 11.4 4.0 3.1 60.4 73.3 0 37.5 15.0 66.76 judge-resolved 50.0% 20.0%fell for 12 of 15 3.6 / 2.0agreement: medium 72.2ROCm 7.1 56.1 4 Q4_K_M tongyi-qianwen-research
212–223 Smollm2 360M Instruct 28.123.4–32.9 14.1 8.6 8.0 23.8 59.9 46.7 11.1 21.7 25.0 47.96 judge-resolved 41.7% 46.7%fell for 8 of 15 4.5 / 2.9agreement: medium 91.2ROCm 7.1 3.1 0.4 Q4_K_M apache-2.0
213–224 YI Coder 1.5B Chatcode model 26.522.1–31.5 21.7 17.1 0 57.7 36.1 53.3 5.6 10.8 35.0 22.96 judge-resolved 41.7% 33.3%fell for 10 of 15 3.3 / 1.6agreement: medium 88.7ROCm 7.1 26.9 1.5 Q4_K_M apache-2.0
216–225 Qwen 1.8B Chat V1.5 24.720.2–29.3 12.6 20.0 4.0 2.4 38.4 60.0 0 27.5 20.0 50.06 judge-resolved 66.7% 40.0%fell for 9 of 15 3.3 / 2.1agreement: high 82.5ROCm 7.1 27.1 1.8 Q4_K_M tongyi-qianwen-research
216–225 Stablelm2 1.6B Chat 24.420.2–29.0 10.6 8.6 4.0 26.1 31.6 40.0 0 16.7 42.5 50.06 judge-resolved 41.7% 13.3%fell for 13 of 15 3.6 / 2.6agreement: high 62.2ROCm 7.1 7.5 1.6 Q4_K_M stability-nc
217–225 Sailor2 1B Chatfine-tune of Qwen 2.5 23.519.0–28.1 6.4 8.6 0 10.5 29.6 60.0 16.7 25.0 27.5 33.36 judge-resolved 66.7% 6.7%fell for 14 of 15 3.6 / 2.6agreement: high 116.3ROCm 7.1 4.1 1 Q4_K_M —
219–225 Stable Code 3B Instructcode model 22.818.4–27.4 17.4 22.9 0 6.5 55.8 33.3 5.6 26.7 15.0 39.66 judge-resolved 41.7% 40.0%fell for 9 of 15 2.6 / 1.1agreement: medium 38.6ROCm 7.1 44.9 2.8 Q4_K_M stability-community
218–225 Deepseek Coder 1.3B Instructcode model 22.618.3–27.2 21.9 5.7 0 54.5 40.5 3.3 11.1 23.3 40.0 25.06 judge-resolved 33.3% 6.7%fell for 14 of 15 0.9 / 0.6agreement: high 64.1ROCm 7.1 26.9 1.3 Q4_K_M other
220–227 Smollm2 135M Instruct 20.116.2–24.6 11.2 2.9 0 14.1 36.1 36.7 5.6 15.0 25.0 45.86 judge-resolved 50.0% 6.7%fell for 14 of 15 3.0 / 1.5agreement: medium 201.2ROCm 7.1 1.7 0.1 Q4_K_M apache-2.0
225–227 Qwen 0.5B Chat V1.5 16.112.5–20.3 15.2 14.3 0 9.0 40.8 13.3 0 30.0 15.0 22.96 judge-resolved 33.3% 26.7%fell for 11 of 15 3.1 / 1.4agreement: medium 98.9ROCm 7.1 13.4 0.5 Q4_K_M tongyi-qianwen-research
225–227 Codegemma 2B Code V1.1code model 15.311.7–19.1 9.2 17.1 0 3.9 29.5 16.7 0 16.7 10.0 43.86 judge-resolved 33.3% 6.7%fell for 14 of 15 0.9 / 0.3agreement: high 43.3ROCm 7.1 3.3 2 Q4_K_M gemma
— Apertus-8B-Instruct-2509 —withheld — — — — — — — — — — — — 8.7ROCm 7.1 22.3 8.1 Q4_K_M apache-2.0
— Falcon2 11B —unranked: long_context partial: 12/18 items scored, 80% must be scored 22.9 16.0 16.9 57.7 70.0 0partial: 12/18 22.5 20.0 87.56 judge-resolved 33.3% 33.3%fell for 10 of 15 4.0 / 2.4agreement: medium 19.7ROCm 7.1 10.9 11 Q4_K_M other
— Falcon3 10B Instruct —withheld — — — — — — — — — — — — 21.2ROCm 7.1 28.3 10 Q4_K_M other
— Falcon3 1B Instruct —withheld — — — — — — — — — — — — 87.3ROCm 7.1 6.0 1 Q4_K_M other
— Falcon3 3B Instruct —withheld — — — — — — — — — — — — 57.9ROCm 7.1 14.0 3 Q4_K_M other
— Falcon3 7B Instruct —withheld — — — — — — — — — — — — 24.7ROCm 7.1 20.1 7 Q4_K_M other
— Internlm2 1.8B Chat V2.5 —withheld — — — — — — — — — — — — 73.2ROCm 7.1 14.3 1.8 Q4_K_M other
— Qwen2 Math 1.5B Instructmath model —withheld — — — — — — — — — — — — 61.1ROCm 7.1 2.1 1.5 Q4_K_M apache-2.0
— Qwen2 Math 7B Instructmath model —withheld — — — — — — — — — — — — 17.4ROCm 7.1 6.5 7 Q4_K_M apache-2.0

Withheld

These models have no rank and no score on this page. Their answers are published on their own pages, except for a model whose every tested version is withheld: it has no published results, and its page gives the reason only.