JSON API

Read-only, no authentication, no API key. Responses are cacheable for five minutes in browsers and an hour at the edge. Please fetch politely and cache what you fetch.

The results and scores on this site are licensed under CC BY 4.0: you may copy, share and reuse them, including commercially, as long as you credit OpenWeights Bench (openweightsbench.com). This covers our results and scores only. It does not cover the judges’ reviews, the models’ answers, the question texts or the models themselves; each model has its own licence terms.

GET /api/v1/leaderboard.json

One object per model version, using its latest finished run (count is the number of these objects, withheld versions included, so a model with a withheld older version is counted twice; count distinct slug values for the number of models): overall on a 0–100 scale and categories, an object keyed by category id with the programmatic mean of every category in the battery on the same scale (null means not scored, which is different from 0; writing and identity have no programmatic score, so theirs is always null). The top level lists the battery's categories (id, label, overall, judged, read from the battery definition) and overall_categories, the ids whose means make up overall (an unweighted mean; a model missing any of them is unranked). Each row also has size_class (tiny for under 3B, then le4 for 3–4B, 4-9, 9-16, 16-35 or gt35, by total parameters), derived_from (the base model label of a community fine-tune, null for an original), specialised (medical, translation, sql, code, math or tool-use for a purpose-built model by name pattern, else null; informational, such a model counts toward Overall like any other), backend_short (the backend name without its settings) and class_rank (rank among ranked models of that class), backend and gpu of its run, perf (speed and memory metrics, null when not measured) with perf_units, fell_for_gotchas, judge_a_mean and judge_b_mean on a 0–10 scale, agreement, disputed, tokens_per_s, tested_at and size_bytes (the model file size, null when not on record: the hardware page estimates the memory a model needs as this size plus about 20%, never from vram_gb). A category counts toward overall only when at least 80% of its items have a definite programmatic score (judge-resolved items included) and it has no pending item: category_detail carries n_items, n_scored (judge-resolved items included), n_judge_resolved, n_judge_split (judges disagreed, scored 0.5), n_pending_resolution (abstained, judging unfinished; any pending item keeps the category out of Overall) and coverage_ok per category, and each model has ranked, an unranked_reason when false, and n_tampering and n_flagged (coding answers that touched the test harness, and coding answers that used constructs outside the task; the score is unchanged by the site). A model withheld from the ranking (see the methodology) has ranked false, a null overall, null category means in this list, and withheld with its public reason and since; the top-level withheld list repeats them, and the model's own JSON keeps its raw category means. A model with only withheld versions has withheld_only true: every measured figure in its row (perf values, judge means, agreement, disputed, tokens_per_s, the counts of cut-off, tampering and flagged answers, and the per-category counts and means in category_detail) is null, it is not counted as a model on the site, and its own JSON keeps no category means either. withheld_only is false on every other row. With no published runs, models is an empty list and note says so.

Explorer fields (every row, always present): hf_downloads and hf_rank (Hugging Face download count and rank, integers, null when not on record), active_params_b (active parameters in billions of a mixture-of-experts model, null when not on record) and vision (true when the model takes images).

Statistics fields (read from stored statistics, see the methodology; every key is always present and is null, 0 or [] when there are none): has_stats (true when the model has statistics that agree with its overall to within 0.05), stats_stale (true when stored statistics exist but disagree, in which case every value below stays empty), ci_low and ci_high (95% interval of overall from resampling the questions), beats (model version ids this model is clearly ahead of), rank_lo and rank_hi (the rank range among all listed ranked models), class_rank_lo and class_rank_hi (the same inside the size class), hard_score and hard_n (mean score on the hard questions, and how many there are), ext_overall (overall with cut-off answers replaced by their second attempt at double the length), n_truncated and n_extended (cut-off answers in the Overall categories, and how many of them have a scored second attempt) and n_gated (safety answers counted as failed by the comply-quality gate). Rows stay sorted by overall.

GET /api/v1/models/{slug}.json

Model and version metadata (the model also carries derived_from and specialised, as on the leaderboard, and import_note, a short public note on how a non-standard import was made, null for most models), the run, the category summary (every category keyed by id, with perf, size_class and backend as on the leaderboard), the perf list of speed and memory metrics, the pushback counts (held or accepted versus caved, control items separately when the database marks them), the gotchas counts, the safety block, n_input_truncated and loaded_context_length (answers whose prompt the host cut, and the window it loaded), n_recovered (how many tool-call answers came from a second ask with the tools declared natively; each such generation has a non-null recovered), and every generation of the model's latest finished run. A model with only withheld versions has none of these: its JSON has withheld true with withheld_reason and withheld_since, the model, version and run metadata, a summary with ranked false, a null overall and the reason, an empty perf and an empty generations list. The battery is private except for a fixed public sample (see the methodology), so a generation entry always carries only numbers and statuses:

Only when public is true does the entry also carry prompt (a long-context prompt is cut to its start and end, with prompt_truncated true), messages (the earlier turns of a multi-turn item as {role, content}, or null), system_prompt, the verbatim output_text, thinking_text, the scorer details (in place of summary), the second attempt's output_text inside extension, and each judge's rationale and review. For private items these fields are absent. Coding items, public or not, only ever carry the summary (test counts, extraction method, failure reason), never test output. Reference answers and hidden tests are never published. Add ?v=<version id> to select an older model version.

GET /api/v1/speed.json

Speed and memory per model version, fastest 4-stream speed first: perf holds decode_tps (one stream), decode_tps_c4 (total with 4 parallel streams), ttft_s, prefill_tps_4k, prefill_tps_12k, load_seconds, size_gb, vram_gb, gpu_fraction and context_length (null when not measured), plus backend, gpu, size_class, overall and ranked, and concurrency_min, concurrency_max and concurrency_text: how many answers were generated at a time for the model (null when not recorded; min and max differ when the setting changed during its run). The memory figures are what Ollama reported with this server's settings (parallel slots and context length), so they can differ from what one user would need, in either direction. All numbers come from one laptop; speeds are comparable only between rows that share a backend. With nothing measured, models is an empty list and note says so.

GET /api/v1/compare.json?a={slug}&b={slug}

Two models side by side, each at its newest tested version (the page is /compare). a and b hold the two models: overall, ci_low and ci_high, rank_lo and rank_hi, hard_score, backend, and perf with the single-stream decode_tps and the memory figures vram_gb and size_gb (null when not measured). overall says which model is higher, by how much (difference) and whether the stored statistics put one clearly_ahead; categories does the same for every category. items counts, over the questions both models have a definite score for, how many each model did better on (a_wins, b_wins) and how many were level (ties); a score gap of more than tie_margin counts as a win. items.differing lists the questions where exactly one model got full marks (a score of at least full_marks_at), with each model's score and verdict. Only a public question carries output_text (cut at 2,000 characters, with output_truncated true when it was); a private question has numbers and verdicts only. items is null when the two models ran different battery versions. The speed and memory caveats are the same as for speed.json. An unknown slug returns 404, and a missing slug or the same slug twice returns 400, both as {"error": "..."}.

GET /api/v1/category/{id}.json

The ranking for one battery category (the page is /category). category has the id, label, overall, judged and a plain-language description of the test. models are ranked by value: the programmatic mean on a 0–100 scale for a category that code scores, only for models with at least 80% of its items scored and none pending (ranked_by says so), or the average of the two judges' means on a 0–10 scale for a category only the judges score (writing). Each entry has rank (ties share a rank), n_scored, n_items, and for a judged category judge_a_mean, judge_b_mean and agreement. not_ranked lists the other models with the reason. item_stats has how many questions every ranked model answered correctly (n_all_right out of n_items) and, for a category that counts toward Overall, how many are hard (n_hard). hard_set says whether a hard-question set exists yet: it is computed only once enough models are ranked, and until then n_hard is null. item_stats is null when no statistics are stored. An unknown id returns 404.

GET /api/v1/questions/{item_id}.json

One public question and every published model's answer to it (the page is /questions). question has item_id, category, battery_version, system_prompt, prompt (a long-context prompt is cut to its start and end, with prompt_truncated true) and messages (the earlier turns of a multi-turn question, or null). ranked_models says how many ranked models answered it correctly (answered_correctly out of n_models, and their share), or is null when no statistics are stored. answers are sorted by score, best first, then by model name; each has the model's slug, ranked, withheld (true for a withheld model: its answer is listed as recorded, its score, verdict and full_marks are null, it is last in the list, and it is left out of count, which has n_withheld beside it), status, score (null means no definite score, which is different from 0), verdict, full_marks, recovered (null, or the kind of the second ask when the answer was recovered that way), input_truncated (true when the host cut the question for this model; the score is as answered), the judges' scores on a 0–10 scale, and output_text (cut at 2,000 characters, with output_truncated true when it was). Only public questions exist here. A private question, a question that expects a refusal and an unknown id all return 404 with {"error": "not found"}, and never any text.

GET /api/v1/runs/{run_id}.json

Run metadata, a hardware summary (Ollama version, GPU or backend, OS family when known), per-model generation counts (n_generations, n_ok, n_truncated, n_failed, n_pending; all null for a model with only withheld versions, which has withheld_only true) and the latest events for the run. Events carry only their timestamp, level and kind; event text is not published.

Notes