Trust / Reproduce
Reproduce an answer
Almost every answer on this site came from one request to Ollama; two kinds did not. This page takes one stored public answer and shows the values behind it, copied from the data, so you can send the same request yourself and compare. It also says what you will not be able to repeat.
A worked example
Question gotchas-001, asked of Gemma4 E4B IT in run r20261016-bv1-473145. Every value below is read from the stored answer.
1. The same weights
ollama pull gemma4:e4b-it-q4_K_M
The Ollama tag is gemma4:e4b-it-q4_K_M (Q4_K_M). The run recorded this model digest:
dc35e8d9c6061baa6f0fa870975ab6932e2542b579b13ea0f199fa4bb7300c9c
Ollama's own ollama list shows the first 12 characters, dc35e8d9c606, as the ID; the /api/tags endpoint returns all of it. If yours differs, you are not running the same file as we did.
2. The same prompt
Prompt
How many times does the letter 'r' appear in the word 'strawberry'? Give your final answer on the last line in the form: ANSWER: <value>
Its stored hash is:
7d60d2a6f1017df8647fbe7bae3938faee9db72e52c7ef950bdda5c1bdaf34c8
This hash is the SHA-256 of the system prompt and the prompt, written as JSON with sorted keys and no ASCII escaping. We recomputed it from the text above and it matches. To check your copy, set system to the system prompt text, or to the JSON null value (not an empty string) when there is none, and prompt to the prompt exactly as above:
import hashlib, json
payload = {"system": system, "prompt": prompt}
text = json.dumps(payload, sort_keys=True, ensure_ascii=False)
print(hashlib.sha256(text.encode("utf-8")).hexdigest())
3. The same request
The runner sends one chat request per question, bar the two exceptions below; this answer is not one of them, because it ended normally and is not a tool-call question. This is the body it sent for this answer: the stored options and think set to false (this model can think; thinking was switched off).
{
"model": "gemma4:e4b-it-q4_K_M",
"messages": [
{
"role": "user",
"content": "How many times does the letter 'r' appear in the word 'strawberry'?\n\nGive your final answer on the last line in the form: ANSWER: <value>"
}
],
"stream": true,
"keep_alive": "30m",
"options": {
"num_ctx": 16384,
"num_predict": 768,
"seed": 42,
"temperature": 0
},
"think": false
}
Save this body to a file and send it to a local Ollama with curl http://localhost:11434/api/chat -d @your-file. The reply is streamed; set "stream": false to get a single JSON object. Temperature 0 and seed 42 are the same for every model on this site; the context size and token limit are the stored values above.
4. Compare
Our stored answer
Let's examine the word "strawberry" and count the occurrences of the letter 'r': s - t - **r** - a - w - b - e - **rr** - y 1. The first 'r' is in the third position. 2. The second 'r' is in the eighth position. 3. The third 'r' is in the ninth position. The letter 'r' appears 3 times in the word "strawberry". ANSWER: 3
It ended normally (105 tokens), and the programmatic check numeric gave it full marks. The run used ROCm 7.1 (Ollama 0.40.4, flash attention, 16k context; 8 parallel slots) on AMD Radeon 8060S (Ryzen AI Max+ 395, 128 GB unified, ~78 GB GPU), with 8 requests in flight at a time. Your answer may not match ours word for word, for the reasons below.
What you cannot reproduce
- Answers that took a second request. A tool-call question whose first reply came back empty because Ollama's handler swallowed the model's own tool-call syntax is asked once more with the tools declared the official way, and the call that comes back is the stored answer. An answer cut off at the length limit is asked again at double the limit; that longer answer is shown beside the cut-off one and does not change the leaderboard score. In the runs behind the leaderboard, 73 of 61,554 stored answers are recovered tool calls and 3,232 cut-off answers have a second, longer answer. To repeat either, send the second request too. See the second chance and the truncation page.
- Private items. 44 of the 270 items in battery v1 are public. The other 226 stay private to limit contamination, so their prompts and answers are not published and nobody outside can ask them again. For a private item the site shows only the numbers, the statuses and the prompt hash. See the public sample.
- Refusal prompts. Prompts that ask a model to refuse something harmful are never published, whatever else is public.
- Byte-for-byte answers. Temperature 0 does not make a repeat identical. When we repeated the battery on 6 models, 14.0% of answers were scored differently between runs. Requests served in parallel can change an answer. See reproducibility.
- The same backend and hardware. The runs used one laptop. A different Ollama version, GPU backend or number of parallel requests can change an answer, and a backend can break a model outright. See backends and the errata. Speed and memory figures only transfer to the same setup.
- The judges. The judge models are paid models reached through a hosted API. We do not control their weights or how they decode, so asking them again can give a different score. Each judgement stores the hash of the rubric it used, but not the judge's state. The programmatic checks are the part you can repeat. See the judges.
- A moved tag. An Ollama tag can be re-pushed. The digest above is what ties an answer to a file; the tag alone does not.
Found a different answer from the same digest, options and prompt? That is useful to us: write to [email protected].