{
  "generated_at": "2026-11-19T15:02:07Z",
  "a": {
    "slug": "google-gemma-4-31b-it",
    "display_name": "gemma-4-31B-it",
    "model_version_id": 12,
    "run_id": "r20261006-bv1-53e146",
    "overall": 99.3091,
    "ranked": true,
    "ci_low": 98.33,
    "ci_high": 99.99,
    "rank_lo": 1,
    "rank_hi": 5,
    "hard_score": 94.7151,
    "hard_n": 38,
    "size_class": "16-35",
    "params_b": 31.27,
    "derived_from": null,
    "specialised": null,
    "backend": "ROCm 7.1 (Ollama 0.34.4, flash attention, 16k context; 4 parallel slots until 2026-10-07 00:45Z, 8 after)",
    "gpu": "AMD Radeon 8060S (Ryzen AI Max+ 395, 128 GB unified, ~78 GB GPU)",
    "withheld": null,
    "perf": {
      "decode_tps": 3.8389,
      "vram_gb": 11.1594,
      "size_gb": 11.1594
    },
    "perf_units": {
      "decode_tps": "tok/s",
      "vram_gb": "GB",
      "size_gb": "GB"
    }
  },
  "b": {
    "slug": "muse-glimmer-30b",
    "display_name": "Muse Glimmer 30B",
    "model_version_id": 222,
    "run_id": "r20261102-bv1-5246ab",
    "overall": 97.8982,
    "ranked": true,
    "ci_low": 96.03,
    "ci_high": 99.32,
    "rank_lo": 1,
    "rank_hi": 23,
    "hard_score": 84.8744,
    "hard_n": 35,
    "size_class": "16-35",
    "params_b": 30.0,
    "derived_from": null,
    "specialised": null,
    "backend": "ROCm 7.1 (Ollama 0.40.13, flash attention, 16k context; 8 parallel slots)",
    "gpu": "AMD Radeon 8060S (Ryzen AI Max+ 395, 128 GB unified, ~78 GB GPU)",
    "withheld": null,
    "perf": {
      "decode_tps": 3.6388,
      "vram_gb": 18.329,
      "size_gb": 18.329
    },
    "perf_units": {
      "decode_tps": "tok/s",
      "vram_gb": "GB",
      "size_gb": "GB"
    }
  },
  "overall": {
    "higher": "google-gemma-4-31b-it",
    "difference": 1.4,
    "clearly_ahead": "tie"
  },
  "categories": [
    {
      "id": "reasoning",
      "label": "Reasoning",
      "overall": true,
      "a": 97.1429,
      "b": 94.2857,
      "higher": "google-gemma-4-31b-it",
      "difference": 2.8
    },
    {
      "id": "math",
      "label": "Math",
      "overall": true,
      "a": 100.0,
      "b": 100.0,
      "higher": "tie",
      "difference": 0.0
    },
    {
      "id": "coding",
      "label": "Coding",
      "overall": true,
      "a": 96.6392,
      "b": 92.3534,
      "higher": "google-gemma-4-31b-it",
      "difference": 4.2
    },
    {
      "id": "instruction",
      "label": "Instruction following",
      "overall": true,
      "a": 100.0,
      "b": 100.0,
      "higher": "tie",
      "difference": 0.0
    },
    {
      "id": "knowledge",
      "label": "Knowledge & honesty",
      "overall": true,
      "a": 100.0,
      "b": 100.0,
      "higher": "tie",
      "difference": 0.0
    },
    {
      "id": "long_context",
      "label": "Long context",
      "overall": true,
      "a": 100.0,
      "b": 94.4444,
      "higher": "google-gemma-4-31b-it",
      "difference": 5.6
    },
    {
      "id": "multilingual",
      "label": "Multilingual",
      "overall": true,
      "a": 100.0,
      "b": 100.0,
      "higher": "tie",
      "difference": 0.0
    },
    {
      "id": "tool_use",
      "label": "Tool use",
      "overall": true,
      "a": 100.0,
      "b": 100.0,
      "higher": "tie",
      "difference": 0.0
    },
    {
      "id": "safety",
      "label": "Safety",
      "overall": true,
      "a": 100.0,
      "b": 100.0,
      "higher": "tie",
      "difference": 0.0
    },
    {
      "id": "sycophancy",
      "label": "Pushback resistance",
      "overall": false,
      "a": 100.0,
      "b": 100.0,
      "higher": "tie",
      "difference": 0.0
    },
    {
      "id": "gotchas",
      "label": "Famous gotchas",
      "overall": false,
      "a": 93.3333,
      "b": 100.0,
      "higher": "muse-glimmer-30b",
      "difference": 6.7
    },
    {
      "id": "writing",
      "label": "Writing",
      "overall": false,
      "a": null,
      "b": null,
      "higher": null,
      "difference": null
    },
    {
      "id": "identity",
      "label": "Self-description",
      "overall": false,
      "a": null,
      "b": null,
      "higher": null,
      "difference": null
    }
  ],
  "same_backend": false,
  "same_battery": true,
  "items": {
    "compared": 259,
    "a_wins": 4,
    "b_wins": 2,
    "ties": 253,
    "tie_margin": 0.001,
    "full_marks_at": 0.999,
    "by_category": [
      {
        "id": "reasoning",
        "label": "Reasoning",
        "compared": 35,
        "a_wins": 1,
        "b_wins": 0,
        "ties": 34
      },
      {
        "id": "math",
        "label": "Math",
        "compared": 25,
        "a_wins": 0,
        "b_wins": 0,
        "ties": 25
      },
      {
        "id": "coding",
        "label": "Coding",
        "compared": 30,
        "a_wins": 2,
        "b_wins": 1,
        "ties": 27
      },
      {
        "id": "instruction",
        "label": "Instruction following",
        "compared": 30,
        "a_wins": 0,
        "b_wins": 0,
        "ties": 30
      },
      {
        "id": "knowledge",
        "label": "Knowledge & honesty",
        "compared": 30,
        "a_wins": 0,
        "b_wins": 0,
        "ties": 30
      },
      {
        "id": "long_context",
        "label": "Long context",
        "compared": 18,
        "a_wins": 1,
        "b_wins": 0,
        "ties": 17
      },
      {
        "id": "multilingual",
        "label": "Multilingual",
        "compared": 20,
        "a_wins": 0,
        "b_wins": 0,
        "ties": 20
      },
      {
        "id": "tool_use",
        "label": "Tool use",
        "compared": 20,
        "a_wins": 0,
        "b_wins": 0,
        "ties": 20
      },
      {
        "id": "safety",
        "label": "Safety",
        "compared": 24,
        "a_wins": 0,
        "b_wins": 0,
        "ties": 24
      },
      {
        "id": "sycophancy",
        "label": "Pushback resistance",
        "compared": 12,
        "a_wins": 0,
        "b_wins": 0,
        "ties": 12
      },
      {
        "id": "gotchas",
        "label": "Famous gotchas",
        "compared": 15,
        "a_wins": 0,
        "b_wins": 1,
        "ties": 14
      }
    ],
    "differing": [
      {
        "item_id": "reasoning-030",
        "category": "reasoning",
        "public": false,
        "a": {
          "score": 1.0,
          "verdict": "pass",
          "full_marks": true
        },
        "b": {
          "score": 0.0,
          "verdict": "fail",
          "full_marks": false
        }
      },
      {
        "item_id": "coding-017",
        "category": "coding",
        "public": false,
        "a": {
          "score": 1.0,
          "verdict": "pass",
          "full_marks": true
        },
        "b": {
          "score": 0.0,
          "verdict": "fail",
          "full_marks": false
        }
      },
      {
        "item_id": "coding-028",
        "category": "coding",
        "public": false,
        "a": {
          "score": 1.0,
          "verdict": "pass",
          "full_marks": true
        },
        "b": {
          "score": 0.0,
          "verdict": "fail",
          "full_marks": false
        }
      },
      {
        "item_id": "long_context-010",
        "category": "long_context",
        "public": false,
        "a": {
          "score": 1.0,
          "verdict": "pass",
          "full_marks": true
        },
        "b": {
          "score": 0.0,
          "verdict": "fail",
          "full_marks": false
        }
      },
      {
        "item_id": "gotchas-010",
        "category": "gotchas",
        "public": true,
        "a": {
          "score": 0.0,
          "verdict": "fail",
          "full_marks": false,
          "output_text": "To solve this problem, we must distinguish it from the standard Monty Hall problem. In the standard version, the host *knows* where the car is and *intentionally* avoids it. In this version, the host does *not* know where the car is and opens a door *at random*.\n\nLet $C$ be the door hiding the car, and $H$ be the door the host opens.\nThe possible locations for the car are $C \\in \\{1, 2, 3\\}$, each with probability $1/3$.\nThe host opens door 3 ($H=3$). We are looking for $P(C=2 | H=3)$.\n\nUsing Bayes' Theorem:\n$P(C=2 | H=3) = \\frac{P(H=3 | C=2) P(C=2)}{P(H=3)}$\n\n1. **Prior Probabilities:**\n   $P(C=1) = 1/3$\n   $P(C=2) = 1/3$\n   $P(C=3) = 1/3$\n\n2. **Conditional Probabilities (The host opens door 3 at random from the remaining doors 2 and 3):**\n   - If the car is behind door 1 ($C=1$), the host can open door 2 or 3. Since it's random: $P(H=3 | C=1) = 1/2$.\n   - If the car is behind door 2 ($C=2$), the host can open door 2 or 3. Since it's random: $P(H=3 | C=2) = 1/2$.\n   - If the car is behind door 3 ($C=3$), the host opens door 3: $P(H=3 | C=3) = 1/2$. (Wait, the prompt says the host opens door 3 and it *happens* to be a goat. This means we are conditioning on the event that the host opened door 3 AND it was a goat).\n\nLet's refine the event. We are given that the host opened door 3 and it revealed a goat. This means $H=3$ and $C \\neq 3$.\nThe probability of the host opening door 3 is $1/2$ regardless of where the car is (since he chooses between 2 and 3).\nThe probability that the car is behind door 2, given that the host opened door 3 and it was a goat:\n$P(C=2 | H=3 \\text{ and } C \\neq 3) = \\frac{P(H=3 \\text{ and } C \\neq 3 | C=2) P(C=2)}{P(H=3 \\text{ and } C \\neq 3)}$\n\n- $P(H=3 \\text{ and } C \\neq 3 | C=2) = 1/2$ (The host picks door 3, and since the car is at 2, it is guaranteed to be a goat).\n- $P(H=3 \\text{ and } C \\neq 3 | C=1) = 1/2$ (The host picks door 3, and since the car is at 1, it is guaranteed to be a goat).\n- $P(H=3 \\text{ and } C \\neq 3 | C=3) = 0$ (If th",
          "output_truncated": true
        },
        "b": {
          "score": 1.0,
          "verdict": "pass",
          "full_marks": true,
          "output_text": "The prior probabilities are\n\n* $P(C_1)=P(C_2)=P(C_3)=1/3$.\n\nLet $E$ be “door 3 is opened and a goat is seen”.  \nIf $C_1$ or $C_2$ the door 3 is a goat, so the host opens it with probability $1/2$ among the two unchosen doors. If $C_3$ the event is impossible",
          "output_truncated": false
        }
      }
    ]
  },
  "note": "Speed and memory were measured on one laptop with this server's settings; speeds compare only between models on the same backend, and memory can differ from what one user would need, in either direction. Answer text appears only for public items."
}
