Trust / Reproduce

Reproduce an answer

Almost every answer on this site came from one request to Ollama; two kinds did not. This page takes one stored public answer and shows the values behind it, copied from the data, so you can send the same request yourself and compare. It also says what you will not be able to repeat.

A worked example

Question gotchas-001, asked of Gemma4 E4B IT in run r20261016-bv1-473145. Every value below is read from the stored answer.

1. The same weights

ollama pull gemma4:e4b-it-q4_K_M

The Ollama tag is gemma4:e4b-it-q4_K_M (Q4_K_M). The run recorded this model digest:

dc35e8d9c6061baa6f0fa870975ab6932e2542b579b13ea0f199fa4bb7300c9c

Ollama's own ollama list shows the first 12 characters, dc35e8d9c606, as the ID; the /api/tags endpoint returns all of it. If yours differs, you are not running the same file as we did.

2. The same prompt

Prompt

How many times does the letter 'r' appear in the word 'strawberry'?

Give your final answer on the last line in the form: ANSWER: <value>

Its stored hash is:

7d60d2a6f1017df8647fbe7bae3938faee9db72e52c7ef950bdda5c1bdaf34c8

This hash is the SHA-256 of the system prompt and the prompt, written as JSON with sorted keys and no ASCII escaping. We recomputed it from the text above and it matches. To check your copy, set system to the system prompt text, or to the JSON null value (not an empty string) when there is none, and prompt to the prompt exactly as above:

import hashlib, json
payload = {"system": system, "prompt": prompt}
text = json.dumps(payload, sort_keys=True, ensure_ascii=False)
print(hashlib.sha256(text.encode("utf-8")).hexdigest())

3. The same request

The runner sends one chat request per question, bar the two exceptions below; this answer is not one of them, because it ended normally and is not a tool-call question. This is the body it sent for this answer: the stored options and think set to false (this model can think; thinking was switched off).

{
  "model": "gemma4:e4b-it-q4_K_M",
  "messages": [
    {
      "role": "user",
      "content": "How many times does the letter 'r' appear in the word 'strawberry'?\n\nGive your final answer on the last line in the form: ANSWER: <value>"
    }
  ],
  "stream": true,
  "keep_alive": "30m",
  "options": {
    "num_ctx": 16384,
    "num_predict": 768,
    "seed": 42,
    "temperature": 0
  },
  "think": false
}

Save this body to a file and send it to a local Ollama with curl http://localhost:11434/api/chat -d @your-file. The reply is streamed; set "stream": false to get a single JSON object. Temperature 0 and seed 42 are the same for every model on this site; the context size and token limit are the stored values above.

4. Compare

Our stored answer

Let's examine the word "strawberry" and count the occurrences of the letter 'r':

s - t - **r** - a - w - b - e - **rr** - y

1. The first 'r' is in the third position.
2. The second 'r' is in the eighth position.
3. The third 'r' is in the ninth position.

The letter 'r' appears 3 times in the word "strawberry".

ANSWER: 3

It ended normally (105 tokens), and the programmatic check numeric gave it full marks. The run used ROCm 7.1 (Ollama 0.40.4, flash attention, 16k context; 8 parallel slots) on AMD Radeon 8060S (Ryzen AI Max+ 395, 128 GB unified, ~78 GB GPU), with 8 requests in flight at a time. Your answer may not match ours word for word, for the reasons below.

What you cannot reproduce

Found a different answer from the same digest, options and prompt? That is useful to us: write to [email protected].