← Judged questions

writing-003: both judges

A public question in Writing, battery v1. It starts: “Write a polite but firm complaint email (120 to 180 words) to an internet provider. Facts: the customer, Dana Whitfield, has had no service…” Read the question and every model's answer.

228 models judged; 7 disputed (judges 3 or more points apart). The gap is the distance between the two judges' scores. Models are ordered by that gap, widest first.

Exaone Deep 2.4B gap 3, disputed

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

The response delivers a polite, firm complaint email with a subject line and includes all required facts and requests. Its stated 178-word count is within the 120–180 range. However, it unnecessarily says the calls occurred “earlier today” and includes redundant placeholder signature fields. The exposed planning text is also inappropriate as part of the final response.

The response clearly states the six-day outage, references two promised but missed technician visits, requests service within 48 hours, and asks for a credit. Its tone is appropriately formal and firm, and the email appears to remain within the requested length. A weakness is the unsupported specificity that both calls occurred “earlier today,” which was not supplied in the task. The signature also contains unnecessary placeholders such as a second full-name field and optional company details. Planning commentary is exposed before the email rather than being omitted from the final answer.

Judge B: J-A (anthropic/claude-haiku-5.5) 4/10

The email delivers a complaint with a subject line and covers the six-day outage, two promised visits, the 48-hour visit request and credit. However, it leaks a visible thinking block, and the body plus signature runs past 180 words, violating the length constraint. Its self-reported word count of 178 is unreliable.

The response delivers the requested complaint email with a clear subject line and covers the core facts: six days without service, two support calls with unfulfilled promises, a 48-hour visit request, and a credit for lost days. Its tone stays polite yet firm. Weaknesses are that the output opens with a long visible reasoning block that should not appear in the final text, and the email body with signature appears to exceed the 180-word maximum. Phrasing such as 'critical state of service interruption' is also somewhat stock.

Exaone Deep 7.8B gap 3, disputed

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

The response delivers a polite, firm complaint email with a subject line and includes all required facts and requests: six days without service, two missed technician commitments, a 48-hour visit, and a credit. The email is within the stated length tolerance. However, it is surrounded by leaked reasoning and extra meta-commentary, and includes unnecessary fabricated-looking account details and placeholders.

The response’s main strength is that the email itself clearly states the outage duration, the two unfulfilled technician promises, and both requested remedies in an appropriately professional tone. Its main weakness is presentation: extensive internal reasoning appears before the email, followed by word-count and “key elements” commentary that does not belong in a finished complaint. The use of placeholder contact information and a specific-looking account number also makes the draft less ready to send, although these do not obscure the requested content.

Judge B: J-A (anthropic/claude-haiku-5.5) 4/10

The email delivers a subject line and covers all facts, but the body is roughly 170 words, within range. However, the response includes visible thinking and meta notes (word count, tone summary) outside the email, and the letter has placeholder brackets. The credit is phrased loosely as per policy. Two hard-constraint issues: an unrequested notes section and a leaked thought block violating the text-only brief.

The response delivers a complaint email with a clear subject line that covers the six-day outage, the two broken technician promises, the 48-hour visit request and the credit for lost days. The tone is polite yet firm and the structure is easy to follow. However, the output opens with a long visible reasoning block and closes with a word-count and tone summary, neither of which the brief asked for. The letter also leaves several placeholders unfilled, so it is not ready to send.

Granite4 350M H gap 3, disputed

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 3/10

The response supplies only a subject line and omits the requested complaint email. It does not mention Dana Whitfield, the six-day outage, missed technician visits, the 48-hour deadline, or the requested credit, so the central task is substantially incomplete.

The response makes a minimal attempt to engage with the requested format by providing a subject line related to a service dispute. However, it does not deliver the email itself, including the customer’s name, outage duration, prior unanswered technician promises, requested resolution timeframe, or compensation request. Because the body is entirely absent, the response cannot function as a polite but firm complaint or communicate any of the specified facts to the provider.

Judge B: J-A (anthropic/claude-haiku-5.5) 0/10

The response is only a subject line, "Subject: Request for Service Dispute", with no email body. The requested 120 to 180 word complaint is absent, so there is no usable text to grade under the writing rubric.

The response consists solely of a subject line reading Request for Service Dispute. It does not deliver the requested complaint email: there is no greeting, no account of the six days without service, no mention of the two earlier support calls, and no request for a visit within 48 hours or a credit. The one element it does provide is a relevant subject line, which is the only part of the brief it addresses. The body of the email is missing entirely, so the piece cannot be assessed against the length or content constraints. The response is therefore effectively empty and fails to produce the requested text.

Marco O1 7B gap 3, disputed

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

The email satisfies the requested length and includes a subject line, Dana Whitfield’s name, six days without service, two unsuccessful support contacts, a 48-hour technician request, and a credit request. It is polite, firm, clear, and professionally formatted, though its wording is conventional rather than especially distinctive.

The response delivers a complete complaint email with an appropriate subject line, clear description of the six-day outage, and specific requests for a technician visit within 48 hours and a service credit. Its tone remains professional while stating that the lack of follow-through is unacceptable. However, the phrasing is largely standard and offers little distinctive detail beyond the required facts. The embedded planning text before the email is also inappropriate as part of a final response because it exposes internal drafting rather than presenting only the requested message.

Judge B: J-A (anthropic/claude-haiku-5.5) 4/10

Email covers all facts and requests with a subject line and polite tone, but the body including greeting and signature is well under 120 words, roughly 110 to 120, and the leaked Thought block is visible. Word count is at or just below the 120-word minimum, a hard constraint violation, and the response also exposes leaked thinking.

The response delivers the requested complaint email on the correct subject, with a subject line and all key facts: six days without service, two unfulfilled support calls, a 48-hour visit request and a credit. The tone stays polite yet firm in the body. However, the response opens with a visible reasoning block that the brief did not ask for, and the email body falls short of the required 120-word minimum once counted. The closing paragraph is also somewhat generic, adding little beyond restating the request.

Nanbeige_Nanbeige4.3-33B gap 3, disputed

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

The response delivers a clear, polite but firm complaint email with a subject line and all required facts: six days without service, two missed technician commitments, a visit within 48 hours, and a credit request. The email is within the requested length and uses an appropriate professional tone. The added self-evaluation is unnecessary but does not undermine the email itself.

The response’s main strength is its complete and well-organized email: the outage duration, prior support calls, requested appointment window, and compensation request are all stated plainly. Its tone balances professionalism with firmness, especially when describing missed commitments and requesting confirmation. The main weakness is the appended commentary about word count, tone, and structure, followed by an offer to make adjustments. That material is not part of the requested complaint email and makes the overall response less clean, even though the email itself satisfies the substantive brief.

Judge B: J-A (anthropic/claude-haiku-5.5) 4/10

The email delivers the requested complaint with subject line and covers all facts, but the response appends a word-count claim, tone note, structure breakdown and a closing offer, which the brief did not ask for. The email body itself is roughly 130 words, and the self-reported 158-word count is unverified. The brief says to reply with the text alone, so the extra commentary is a violation.

The response delivers a complaint email with a clear subject line and accurately includes the six-day outage, the two unfulfilled technician promises, the 48-hour visit request and the credit demand. Its polite-but-firm register is appropriate and the requests are stated plainly. The main weakness is that it appends an unrequested analysis section with word count, tone and structure notes plus a closing offer, although the brief asked for the email alone. The body's word count also appears to fall short of the 120-word minimum, so the self-reported figure is not reliable.

Opencoder 1.5B Instruct gap 3, disputed

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 6/10

The response is a complete 120–180-word email with a subject line and includes all required facts, the 48-hour visit request, and the credit request. However, its complaint is weakened by indirect phrasing such as expressing “interest” and offering to help the provider, rather than making a firm escalation. The language is also somewhat awkward.

The response’s strength is that it includes the customer’s name, six-day outage, two missed technician commitments, requested 48-hour visit, and compensation request within a properly formatted email. Its main weakness is tone and phrasing: calling the message an expression of “interest” makes it sound like a service inquiry rather than a complaint, while the offer to help arrange a technician shifts responsibility away from the provider. The request is therefore clear but insufficiently firm and somewhat unnatural in places.

Judge B: J-A (anthropic/claude-haiku-5.5) 3/10

Delivers a subject line and an email to a provider, but it is written in the third person about Dana rather than as her firm complaint. It never states the 6-day outage, the broken promises, or a credit demand directly, and the body is vague and muddled (it offers to 'assist in finding' a professional). Word count is roughly 110, below the 120-word minimum.

The response includes a subject line and a signed email addressed to a provider, so it recognisably attempts the requested piece. Its main weakness is that it does not deliver a firm complaint from Dana: the facts are reported in the third person as background rather than asserted, the outage duration and promised visits are mentioned only vaguely, and the requested credit is referred to without being demanded. The closing offer to help find a technician undercuts the firm tone. The body also falls short of the 120-word minimum, which is a hard constraint violation. The prose is fluent but generic.

Phi-3.5-mini-instruct gap 3, disputed

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 8/10

The email includes a subject line, identifies Dana Whitfield, states the six-day outage and two missed technician promises, requests a visit within 48 hours, and seeks a six-day credit. Its tone is polite but firm and the text appears within the requested 120–180-word range, despite the inaccurate stated count of 116.

The response clearly delivers the requested complaint email and covers every required fact and remedy. Its strongest feature is the explicit connection between the six-day outage and the requested service credit, alongside a concrete 48-hour deadline. The main weakness is that several sentences use generic customer-service language, such as praising the provider’s values, which makes the complaint less direct and slightly dilutes its firmness. The appended word-count claim is also inaccurate, although the email itself appears to satisfy the required length.

Judge B: J-A (anthropic/claude-haiku-5.5) 5/10

The email delivers the requested complaint with a subject line and covers the outage, the two broken promises, the 48-hour visit request and the credit. However, the body runs roughly 200 words, above the 120 to 180 range, and the self-reported '(Word Count: 116)' is inaccurate.

The response delivers the requested complaint with a subject line. It covers the six-day outage, the two unkept support-call promises, the 48-hour visit request and the credit for lost days, in a firm but courteous register. Its main weakness is length: the body is roughly 200 words, exceeding the 120 to 180 range, and the appended word count of 116 misstates this. Phrasing such as 'esteemed company' is also generic.

Cogito 3B V1 Preview Llama gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 5/10

The email is polite, clear, and within the 120–180-word range, with the subject line and core outage history included. However, it requests service restoration rather than explicitly requesting a technician visit within 48 hours, and asks for generic compensation rather than a credit for the lost days. It also adds an unsupported claim that visits were promised within 24 hours.

The response presents a usable complaint email with an appropriate subject line, professional tone, and clear description of six days without service and two missed technician appointments. Its requests for prompt restoration and compensation address the general problem. However, it does not precisely state that a technician visit must occur within 48 hours, nor does it specifically request a service credit, which are central details in the brief. The added claim that each visit was promised within 24 hours is also not supplied by the task and weakens factual precision.

Judge B: J-A (anthropic/claude-haiku-5.5) 7/10

Subject line present and body is about 135 words, within range. Covers six days, two broken promises, 48-hour restoration and compensation, in a polite but firm tone. It invents a '24 hours' promise window not in the brief, and the compensation is not named as a credit.

The response delivers the requested complaint email with a subject line and a body within the 120 to 180 word range. Its firm but courteous register suits the brief, and it covers the six-day outage, the two missed technician visits, the 48-hour demand and compensation. Its main weakness is the added detail that technicians were promised within 24 hours, which the brief never states. It also refers to compensation rather than the specific credit requested, and its closing request for an explanation goes beyond the brief. The prose is clear and competent but largely conventional.

Command R7B Arabic 7B 02 2025 gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 8/10

The response delivers a 120–180-word complaint email with a subject line, accurately states the six-day outage and two missed technician commitments, and requests both a visit within 48 hours and a service credit. It is polite, firm, clear, and professionally formatted, though some wording is repetitive and generic.

The response’s main strength is complete task coverage: it identifies Dana Whitfield, explains the six-day outage and two failed commitments, and makes both requested remedies explicit. Its tone remains appropriately courteous while communicating dissatisfaction. A minor weakness is that the closing repeats the request for prompt action without adding useful escalation details, and phrases such as “significant inconvenience and disruption” are conventional rather than especially precise. Overall, the email is polished and functional, with only modest stylistic room for improvement.

Judge B: J-A (anthropic/claude-haiku-5.5) 6/10

Delivers a complete complaint email with a subject line and the required facts: six days offline, two broken promises, a 48-hour visit request, and a credit. Body length is about 165 words, within range. The prose is flat and generic ('deep dissatisfaction', 'growing frustration') and repeats 'promptly' and 'resolution'.

The response delivers the requested complaint in full, including a subject line, the six-day outage, the two earlier support promises, and the explicit requests for a 48-hour visit and a credit. Its body length falls within the stated range. Its main weakness is generic phrasing and repetition: 'deep dissatisfaction', 'growing frustration', and 'promptly' recur, so the email reads as a template rather than a specific account. It also cites unspecified 'terms and conditions' that the brief never mentioned.

Deepseek V2 16B Lite Chat gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 5/10

The email is clear, polite, firm, and includes a subject line, the six-day outage, earlier promised technician visits, a 48-hour request, and a credit request. However, it substantially exceeds the required 120–180-word range, with extensive generic wording and placeholders such as dates and provider name.

The response delivers the requested complaint format and covers the core facts, including the six-day outage, missed technician commitment, requested 48-hour visit, and compensation. Its tone is appropriately professional and assertive. The principal weakness is length: the email is well over the 180-word maximum, largely because of repeated contextual and courtesy language. Placeholder fields for the provider and dates also make it less ready to send, although they do not undermine the basic complaint.

Judge B: J-A (anthropic/claude-haiku-5.5) 7/10

Delivers a complete complaint email with a subject line, covers six days without service, two broken promises of a technician visit, a 48-hour visit request and a credit for lost days. Word count is roughly 170, within 120-180. Tone is polite but firm and clear, though phrasing like 'I hope this message finds you well' is stock.

The response delivers the requested complaint email in full, with a subject line, and covers every fact in the brief: six days without service, two earlier promised visits that did not occur, a request for a visit within 48 hours, and a credit for days lost. Its word count falls within the requested 120 to 180 range and the tone stays polite yet firm. Its weakness is that much of the phrasing is conventional, such as the opening pleasantry and 'valued customer', and it leaves several bracketed placeholders for dates and provider name, which makes the letter less specific than it could be.

Falcon3 1B Instruct gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

The response delivers a polite, firm complaint email with a subject line and includes the six-day outage, two missed technician promises, a 48-hour visit request, and a credit request. It is approximately within the requested length tolerance. However, it awkwardly uses customer-name placeholders and adds an unsupported claim of substantial data loss.

The response clearly presents the main service complaint and makes both requested remedies explicit: a technician visit within 48 hours and a credit for lost service. Its professional tone and conventional email structure suit the task. However, the wording is less precise than it could be because it describes the connection as unreliable rather than directly stating that service has been unavailable, and it introduces substantial data loss without that fact being provided. The repeated customer-name placeholders also make the email feel insufficiently personalized.

Judge B: J-A (anthropic/claude-haiku-5.5) 5/10

Delivers a subject line and a complaint email covering the 6-day outage, two broken technician promises, a 48-hour visit request and a credit. Word count is roughly 170, within range. However it says the customer's name is 'Customer's Name' placeholder and claims data loss and an attached log that were never stated, and the tone is more demanding than polite-firm. One hard constraint arguably fails: the invented P.S. about an attachment and data loss adds unrequested facts.

The response delivers the requested complaint email with a subject line, covering the six-day outage, the two earlier support calls, the 48-hour technician request and the credit for lost days. Its main strength is that it hits every required element in a clear, orderly structure. Its main weakness is that it adds facts not given in the brief, such as substantial data loss and an attached log in a postscript, and it leaves the customer name as a bracketed placeholder alongside the real name. The closing sentences are generic and the prose leans on stock phrasing like expedite and rectify, which weakens the firm but courteous voice the brief asked for.

Falcon3 7B Instruct gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 8/10

The email is within the requested 120–180-word range and includes a subject line, all specified facts, a 48-hour technician request, and a credit request. It is clear, polite, and appropriately firm, though its wording is conventional rather than especially distinctive.

The response effectively delivers a complete complaint email with a clear subject line, professional salutation, and appropriate closing. It accurately states the six-day outage, the two unfulfilled technician promises, the requested 48-hour visit, and the desired service credit. The tone remains courteous while communicating urgency. Its main weakness is that the language is fairly generic, and the placeholder for the provider’s name makes the message slightly less ready to send. The complaint could also be firmer by requesting confirmation of the appointment and specifying when the credit will be applied.

Judge B: J-A (anthropic/claude-haiku-5.5) 6/10

Includes a subject line, covers the six-day outage, the two broken promises, the 48-hour visit and the credit. Body is about 105 words, just under the 120 minimum, but within the 10% tolerance. Tone is polite but soft rather than firm, and phrasing is generic ('significantly impacted my daily activities').

The response delivers the requested complaint email with a subject line and covers every stated fact: the six-day outage, the two unkept technician promises, the 48-hour visit request and the credit for lost days. Its main weakness is that the wording is generic and somewhat deferential, with stock phrases such as 'I look forward to a prompt resolution' that do not convey the firmness the brief asks for. The word count sits at the lower edge of the requested 120 to 180 range, close to the tolerance limit.

Gemma 3n E4B IT gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

The response delivers a polite but firm complaint email with a subject line and includes all required facts: six days without service, two unfulfilled technician promises, a visit requested within 48 hours, and a six-day service credit. It is clear and appropriately professional, though largely conventional and marred by stray underscore characters before “Furthermore.”

The response effectively organizes the complaint and states the requested remedy clearly, including both a technician visit within 48 hours and a credit for the six days without service. Its tone is appropriately firm without becoming hostile, and the email falls within the requested length. The main weakness is that it relies on numerous unresolved placeholders, such as account details and call dates, while the stray characters before “Furthermore” create a minor presentation defect. The prose is functional but mostly standard rather than especially distinctive.

Judge B: J-A (anthropic/claude-haiku-5.5) 5/10

Delivers a complaint email with a subject line, covering the six-day outage, two unfulfilled promised visits, a 48-hour visit request and a credit. Word count is roughly 150, within range. Polite but firm tone holds. Weaknesses: stray token artifact '▁▁' in the body, and many unfilled bracketed placeholders reduce polish.

The response delivers the requested complaint email with a subject line and covers every required fact: six days without service, two earlier calls with unkept technician promises, a 48-hour visit request, and a credit for lost days. Its length falls within the 120 to 180 word range and the tone stays polite yet firm. A real weakness is a stray special-character artifact in the middle of the request paragraph, which a reader would notice. The heavy use of bracketed placeholders for dates, account details and impact also makes the draft feel less finished than it could be.

gemma-2-9b-it gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 8/10

The response delivers a 120–180-word complaint email with a subject line, clearly identifies Dana Whitfield, states the six-day outage, mentions both prior unsuccessful support calls, and requests a technician within 48 hours plus a service credit. Its tone is appropriately polite but firm and the language is clear, though somewhat conventional.

The response handles the requested email format effectively, including a clear subject line, salutation, specific outage duration, prior support attempts, a 48-hour deadline, and the requested account credit. Its firm wording communicates the seriousness of the disruption without becoming abusive or unprofessional. The main weakness is that it relies on generic placeholders for the provider, account number, and call dates, and uses fairly standard complaint phrasing rather than a particularly distinctive voice. These placeholders are reasonable in a template, but they make the email feel less immediately ready to send.

Judge B: J-A (anthropic/claude-haiku-5.5) 6/10

Subject line present and body is about 135-147 words, within the 120-180 range. It states the six-day outage, the two prior calls with promised visits, the 48-hour visit request and the credit. Phrasing is generic ("look forward to your prompt response"), and the placeholders are left unfilled.

The response meets the brief's hard constraints: it includes a subject line, stays within the requested length, and covers the outage duration, the two unkept technician promises, the 48-hour visit request, and the credit for lost days. Its tone is firm, though "extreme dissatisfaction" leans slightly harsh for a polite letter. The closing sentences are stock formulas that could appear in any complaint, and the bracketed placeholders are left unfilled, which is reasonable for a template but limits its immediate usefulness to Dana Whitfield.

gemma-4-31B-it gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 8/10

The response delivers a complete 120–180-word complaint email with a subject line. It accurately states six days without service, two unsuccessful support calls, the requested 48-hour technician visit, and a credit for lost service. The tone is polite but firm, and the prose is clear and professional, though conventional rather than especially distinctive.

The response handles the requested email format effectively, including a clear subject line, appropriate greeting and closing, and all required factual details. Its strongest feature is the specific, actionable request for both a technician visit within 48 hours and a pro-rated credit. The language remains firm without becoming hostile. Its main limitation is that the phrasing is fairly standard and includes account and phone-number placeholders, which slightly reduces the sense of a fully finalized message. Overall, it is a polished and competent complaint that fulfills the brief.

Judge B: J-A (anthropic/claude-haiku-5.5) 6/10

The email has a subject line and about 145 body words, within the 120 to 180 range. It states the six-day outage, the two unkept technician promises, the 48-hour visit request and the credit. Phrasing is clear but largely expected, with stock lines like 'lack of reliability' and 'look forward to your prompt confirmation'.

The response delivers the requested complaint email with a subject line and stays within the word range. It covers every stated fact: the six-day outage, the two earlier calls with unkept promises, the 48-hour visit request and the credit for lost days. Its main weakness is that the language is generic and formulaic, with stock phrases such as 'failure to follow through' and 'unacceptable' that could appear in any complaint. The subject line's 'Total Loss of Service' is also somewhat melodramatic for a polite but firm tone.

Gemma3 1B IT gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 8/10

The response delivers a polite but firm complaint email with a subject line and appears to remain within the 120–180-word range. It accurately includes Dana Whitfield’s six-day outage, two missed promised technician visits, the requested 48-hour visit, and a request for credit. The phrasing is clear and appropriately professional, though somewhat conventional.

The response’s main strength is its complete coverage of the requested facts: it identifies the six-day interruption, documents two earlier unsuccessful support contacts, and makes both requested remedies explicit. Its tone remains courteous while communicating urgency, and the email structure is usable as written. The main weakness is that it relies on generic placeholders for the provider and call dates, although those details were not supplied in the task. The language is also fairly standard rather than especially distinctive, but it remains clear and appropriately firm.

Judge B: J-A (anthropic/claude-haiku-5.5) 6/10

Subject line present; body is about 150 words, within 120 to 180. It covers six days without service, two prior calls with unkept technician promises, a 48-hour visit request, and a credit. Tone is polite but firm. Phrasing is generic ('significant frustration', 'considerable disruption') and relies on bracketed placeholders.

The response meets the brief's hard constraints: it includes a subject line, stays within the requested word range, and covers every stated fact, including the 48-hour visit deadline and the request for a credit. Its tone is polite yet firm, and the structure is clear. Its main weakness is that the wording is generic and leans on stock complaint phrases, so it reads as a template rather than a letter tailored to this customer. The bracketed placeholders for dates and provider name are also left unfilled, which a reader would notice.

Gemma4 E2B IT gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 8/10

The response provides a complete 120–180-word complaint email with a subject line, accurately states six days without service and two missed technician promises, and requests both a visit within 48 hours and a credit. Its tone is appropriately firm and generally polite, though the wording is conventional rather than especially distinctive.

The response clearly presents the outage history and converts the requested remedies into explicit demands: a technician visit within 48 hours and a credit for six lost days. It also uses a recognizable email format with a subject, greeting, closing, and sender name. Its main weakness is that the language is fairly formulaic, and phrases such as “completely unacceptable” and “immediately” make the tone somewhat more confrontational than necessary for a polite complaint. The account and contact placeholders are practical but reduce the sense of a fully personalized message.

Judge B: J-A (anthropic/claude-haiku-5.5) 6/10

Subject line present, about 135 words including placeholders, and it covers all facts: six days without service, two calls with a broken promise, a visit within 48 hours, and a credit. Phrases like "extreme dissatisfaction" and "completely unacceptable" lean hostile rather than polite, and the wording is generic.

The response meets every hard constraint: it includes a subject line, stays within the requested word range, and makes all four requests clearly. Its strength is a clear structure that a provider could act on immediately. Its main weakness is tone. Phrases such as "extreme dissatisfaction" and "completely unacceptable" read as angry rather than polite but firm, and sentences like "This lack of follow-through is very frustrating" repeat the same complaint without adding specifics such as call dates or reference numbers.

Glm4 9B Chat gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 8/10

The response delivers a complete 120–180-word complaint email with a subject line. It accurately states six days without service, two missed promised technician visits, and requests both a visit within 48 hours and a credit. The tone is polite but appropriately firm, though some wording is conventional and the provider placeholder is slightly generic.

The response clearly fulfills the requested email format and includes all specified facts and remedies. Its tone remains courteous while communicating urgency through the outage duration, missed appointments, and 48-hour deadline. The wording is clean and professional, and the length appears within the required range. Its main weakness is that several phrases, such as the appeal to being a loyal customer and the request for a “swift resolution,” are standard rather than especially specific or forceful. The “[Provider’s Name]” placeholder is also generic, though reasonable in a template complaint.

Judge B: J-A (anthropic/claude-haiku-5.5) 6/10

The email includes a subject line and runs about 140 words in the body, within the 120 to 180 range. It requests a visit within 48 hours and a credit for lost days, and stays polite but firm. Phrasing is generic, with stock lines like 'deep concern' and repeated 'inconvenience'.

The response delivers the requested complaint email with a subject line, a length inside the required range, and all the stated facts: six days without service, two unkept support calls, a 48-hour visit request, and a credit. Its tone is courteous and direct, which suits the brief. Its main weakness is generic language. Phrases such as 'deep concern', 'swift resolution', and 'urgent matter' are stock wording, and 'inconvenience' appears twice, so the letter reads as competent but unremarkable rather than distinctive.

Granite3.1 Moe 3B Instruct gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 5/10

The response supplies a polite, firm complaint email with a subject line and includes the six-day outage, two prior calls, a 48-hour technician request, and a credit request. However, it substantially exceeds the 120–180-word limit at roughly 200 words, and it introduces placeholder dates and times not provided in the facts.

The response clearly presents the customer’s concern and includes all major requested remedies, including a technician visit within 48 hours and compensation for lost service. Its professional tone and readable structure suit an internet-provider complaint. The main weakness is that the email is materially over the specified word range, making it a hard-constraint violation. It also describes the calls as having scheduled dates and times, although the task only states that visits were promised; the placeholders weaken the factual precision of the complaint.

Judge B: J-A (anthropic/claude-haiku-5.5) 7/10

Delivers a complete complaint email with a subject line, covering six days without service, two broken promises of a technician visit, a 48-hour visit request, and a credit. Word count is roughly 165, within 120-180. Tone is polite but firm and language is clean, though phrasing like 'I hope this message finds you well' is expected.

The response delivers the requested complaint email in full, with a subject line, and meets the 120 to 180 word range. It covers every required fact: six days without service, two earlier calls with unkept promises, a 48-hour visit request, and a credit for the lost days. The tone stays polite yet firm, and the language is clear and correct. Its main weakness is that the opening and closing lines are stock phrases a reader will have seen many times, and the placeholders for dates and times are left unfilled, which makes the draft feel less specific to the customer's situation.

Granite4 Tiny H gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 8/10

The response delivers a complete complaint email within the requested 120–180-word range and includes a subject line. It accurately states six days without service, two missed technician promises, a requested visit within 48 hours, and a request for a service credit. The tone is polite but appropriately firm, though some phrasing is conventional.

The response clearly addresses the provider and presents all required facts in a professional email structure. Its strongest feature is the specific, actionable request for a technician within 48 hours alongside a credit for the outage period. The tone remains courteous while communicating dissatisfaction and urgency. Its main weakness is that several expressions, such as describing the situation as causing “frustration and inconvenience,” are fairly formulaic, and the opening could be more direct. Overall, it is polished and complete, but not especially distinctive in voice or phrasing.

Judge B: J-A (anthropic/claude-haiku-5.5) 6/10

Body runs about 150 words, within the 120-180 range, with a subject line and all facts present: six days without service, two promised visits missed, a 48-hour visit request and a credit. Phrasing is generic ('deep dissatisfaction', 'state of frustration'), and the firmness is mostly carried by the stated requests.

The response meets every hard constraint: a subject line, a body of roughly 150 words, the six-day outage, the two unkept technician promises, the 48-hour visit request and the requested credit. Its strength is that the requests are stated plainly and in the right order. Its weakness is that the prose leans on stock complaint language such as 'unfulfilled promises' and 'state of frustration', so the letter reads as expected rather than distinctive. The closing line is also a generic courtesy rather than a firm deadline or consequence.

Granite4.1 8B gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 8/10

The response delivers a complete complaint email with a subject line and all required facts: six days without service, two missed technician promises, a 48-hour visit request, and a request for a service credit. It is within the requested length and maintains a polite but firm tone. The phrase “lack of internet service ... uninterrupted” is slightly awkward.

The response clearly presents Dana Whitfield’s situation and includes each requested remedy, including a technician visit within 48 hours and a credit for the outage period. Its professional structure, specific subject line, and courteous wording make it suitable for contacting an internet provider. One weakness is the awkward phrasing describing the outage as a “lack of internet service” that was “uninterrupted,” which could momentarily confuse the reader because uninterrupted service normally implies that connectivity continued. The contact-information placeholder is also generic, though it does not undermine the requested complaint.

Judge B: J-A (anthropic/claude-haiku-5.5) 6/10

Meets the hard constraints: a subject line is present, the body is roughly 130 words, and all facts appear (six days, two calls, 48-hour visit, credit). Prose is clear but generic, with stock phrases like 'I hope this message finds you well' and 'swift response'. The 'uninterrupted' wording is slightly awkward.

The response delivers the requested complaint email with a clear subject line and a body length within the stated range. It covers every fact in the brief: six days without service, two earlier calls with unkept promises, a technician visit within 48 hours, and a credit for the lost days. Its tone is polite and firm as asked. The main weakness is that the phrasing is largely conventional, relying on stock openers and closers such as 'I hope this message finds you well' and 'swift response', and it describes the outage as 'uninterrupted', which is an awkward word choice for a service failure.

Internlm2 7B Chat V2.5 gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

The response provides a complete complaint email with a subject line, identifies six days without service and two missed technician promises, and requests both a visit within 48 hours and a service credit. It is clear and appropriately polite, though somewhat generic, slightly overlong before tolerance, and includes an unexplained token at the end.

The response clearly covers the central facts and requested remedies, including the six-day outage, two unfulfilled technician commitments, a 48-hour deadline, and compensation. Its courteous business tone is suitable for an internet-provider complaint. However, the wording is fairly formulaic and less firm than it could be, with phrases such as “I kindly request” and “I trust that you will prioritize.” It also adds an unsupported reference to business operations and ends with an apparent unused token, which reduces polish and professionalism.

Judge B: J-A (anthropic/claude-haiku-5.5) 5/10

Includes a subject line and covers all facts: six days offline, two broken promises, a 48-hour visit request and a credit. Body length is about 175 words, within tolerance. Prose is generic and cliché-driven ('I hope this message finds you well', 'valued customer'). A stray '[UNUSED_TOKEN_145]' artifact closes the email.

The response delivers the requested complaint email with a clear subject line and includes every stated fact: six days without service, two earlier support calls with unkept promises, a 48-hour technician visit, and a credit. The body stays polite yet firm and lands near the requested length. Its main weakness is generic phrasing, such as the stock opening pleasantry and the phrase describing the customer as valued, which makes the letter feel templated. A stray token artifact also appears after the signature block, which a reader would notice and which should have been removed before sending.

Laguna XS 2.1 gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 8/10

The response delivers a complete 120–180-word complaint email with a subject line, identifies Dana Whitfield, states the six-day outage and two missed technician commitments, and requests a visit within 48 hours plus a six-day credit. Its tone is firm yet professional, with only slight awkwardness in “technicians were promised.”

The response clearly presents the service outage, the two unsuccessful prior calls, and the requested remedies. Its subject line and closing make the message immediately usable, while the request for confirmation adds practical specificity. The tone remains appropriately firm without becoming hostile. A minor weakness is the phrase describing technicians as having been “promised,” which is slightly imprecise because the visits, rather than the technicians themselves, were promised. The added impact on work and personal matters is plausible but not required by the stated facts.

Judge B: J-A (anthropic/claude-haiku-5.5) 6/10

The email meets the 120-180 word range (about 145 words, counting the body) and includes a subject line. It covers six days without service, the two broken promises, a 48-hour visit and a credit. The phrasing is generic ('I trust you will prioritize this issue'), and 'I demand' sits uneasily with the brief's polite tone.

The response delivers the requested complaint email within the length range, with a subject line and all the stated facts: six days offline, two unkept support-call promises, a 48-hour visit request and a credit. Its main strength is clear structure and a firm, readable ask. Its main weakness is generic wording, such as 'I trust you will prioritize this issue', and 'I demand' pushes against the polite register the brief asks for. The trailing word-count note is unrequested but harmless.

Lfm2.5 8B A1B gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

The final email is within the requested length, includes a subject line, identifies the six-day outage and two missed technician promises, and requests service within 48 hours plus a credit. It is polite and appropriately firm. The salutation incorrectly addresses Dana Whitfield rather than the provider, and the visible planning text is extraneous.

The response clearly delivers a complete complaint email with all requested facts and remedies. Its wording is professional, and the request for a technician within 48 hours and a credit for six lost days is explicit. However, the salutation reads “Dear Dana Whitfield,” even though Dana is the customer and the email should address the provider. The inserted drafting and word-count discussion also appears before the final email, making the overall response less clean, although the email itself remains usable after that material is removed.

Judge B: J-A (anthropic/claude-haiku-5.5) 5/10

The delivered email meets the subject line, subject facts, and is about 120-135 words, so length is within range. However the body is addressed 'Dear Dana Whitfield' (the sender, not the provider), a clear error. The text also contains a leaked thinking block with word counting before the final answer. Writing is clear and competent but generic.

The response delivers a complete complaint email with a subject line that covers the six-day outage, the two broken promises of a technician visit, the 48-hour request, and the service credit. Its main strength is clear structure and a polite but firm register that fits the brief. Its main weakness is a salutation error: it addresses 'Dear Dana Whitfield' when the recipient should be the provider's support team, and the reply also exposes a long reasoning block before the email. The prose is clear but largely conventional.

Lfm2.5 Thinking 1.2B gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

The response provides a polite, firm email with a subject line and includes all required facts: six days without service, two missed technician visits, a 48-hour request, and a credit request. The email is within the requested length, though the sentence identifying the account is awkward and the appended explanation is unnecessary.

The response’s strength is its clear inclusion of the requested remedy and supporting history, especially the two earlier calls and the requested 48-hour technician visit. Its tone remains appropriately courteous while communicating urgency. A weakness is the awkward phrasing in “issue affecting my account, Dana Whitfield,” which incorrectly makes the customer name appear to identify the account. The added word-count claim and explanatory paragraph also make the output less clean as a ready-to-send email, even though the email itself substantially fulfills the brief.

Judge B: J-A (anthropic/claude-haiku-5.5) 5/10

The email has a subject line, delivers the complaint, and covers the six-day outage, failed technician promises, 48-hour request and credit. It is clear and polite. However, the body is roughly 110 words, below the 120-180 range, and the closing note outside the email adds a self-description that was not requested. Word count claim of 150 is inaccurate.

The response delivers a recognisable complaint email with a subject line that covers every stated fact: six days without service, two unfulfilled support calls, the 48-hour visit request and a credit. Its tone is polite yet firm, and the prose is clear. Its main weakness is length: the email body falls below the required 120 to 180 words, and the self-claimed word count of 150 is inaccurate. The trailing commentary also explains the email rather than being requested, and the wording leans on stock phrases such as 'maintain our partnership's integrity'.

Lfm3 VL 2B gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 7/10

The response delivers a 120–180-word complaint email with a subject line, identifies six days without service, notes two missed technician promises, and requests both a visit within 48 hours and a credit. Its tone is polite but firm, though the third-person phrasing and placeholders make it less natural and specific.

The response’s main strength is that it includes all required facts and requests in a concise, professional email structure. The subject line is clear, and the wording communicates urgency without becoming hostile. Its weakness is that it refers to Dana in the third person, even though the message otherwise appears intended to come from her, and it uses generic placeholders for the provider and sender. The closing request is also somewhat softened by broad language such as “at your earliest convenience,” which is less firm than the requested 48-hour deadline.

Judge B: J-A (anthropic/claude-haiku-5.5) 5/10

The email body counts about 98 words, below the required 120 to 180 and outside the 10% tolerance, so one hard constraint is violated. A subject line is present and the facts are covered. The prose is clear but generic, and it refers to Dana in third person despite being her complaint.

The response includes a subject line and covers the core facts: six days without service, two unkept promises of a visit, and the request for a visit within 48 hours plus a credit. Its main weakness is length: the body is roughly 98 words, short of the 120 to 180 range. It also writes about Dana in the third person while presenting itself as the complaint, which reads awkwardly. The phrasing is clear but largely stock, with lines such as 'committed to resolving it promptly' that add little specificity.

Llama 4 Scout 17B-16E Instruct gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 8/10

The response delivers a 120–180-word complaint email with a subject line, identifies Dana Whitfield, states the six-day outage and two missed promised visits, requests a technician within 48 hours, and asks for a six-day service credit. Its tone is polite but firm, with clear and professional language.

The response effectively presents the required complaint in a conventional email format. Its strongest feature is the clear inclusion of every requested fact and remedy, especially the 48-hour technician deadline and credit request. The tone remains appropriately firm without becoming hostile, and the wording is concise and readable. The main weakness is that the phrasing is somewhat conventional, relying on familiar expressions such as disappointment, frustration, and prompt resolution. It also adds an unrequested claim about being unable to work and access essential services, though this does not undermine fulfillment of the brief.

Judge B: J-A (anthropic/claude-haiku-5.5) 6/10

The response includes a subject line and a body of roughly 130 words, inside the 120 to 180 range. It asks for a visit within 48 hours and a credit for six days, and it cites the two missed promises. The prose is clear but generic, with stock phrases such as 'disappointment and frustration' and 'up and running again'.

The response meets the brief's hard constraints: it has a subject line, falls within the word range, and covers the outage, the two broken support promises, the 48-hour visit request, and the credit for lost days. Its strength is clarity, since the reader immediately understands the grievance and the demands. Its weakness is that the writing is largely expected, relying on familiar complaint phrasing and offering no specific detail such as an account reference or a concrete date of the earlier calls. The tone is firm enough without becoming hostile.

Llama-3.1-8B-Instruct gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 8/10

The response delivers a complete 120–180-word complaint email with a subject line. It accurately includes Dana Whitfield’s six-day outage, two missed promised technician visits, a request for service within 48 hours, and a credit. The tone is polite but firm, and the wording is clear, though conventional.

The response effectively organizes the complaint as a usable email, with a clear subject line, professional greeting, concise account of the failed support commitments, and explicit requests for both a technician visit and a billing credit. Its tone balances courtesy with firmness, particularly in noting that the lack of follow-through is unacceptable. The main weakness is that the phrasing is fairly formulaic and does not provide practical account details, such as a service address or reference number, although those details were not required by the task. The requested facts and length constraints are otherwise satisfied.

Judge B: J-A (anthropic/claude-haiku-5.5) 6/10

The body runs about 143 words, inside the 120 to 180 range, and a subject line is included. It asks for a visit within 48 hours and a credit for days lost, and the tone is polite but firm. The prose is generic, with stock phrasing such as 'I look forward to hearing from you soon' and 'unacceptable'.

The response meets every hard constraint: it has a subject line, falls within the word range, and covers the outage, the two broken support promises, the 48-hour visit request, and the credit. Its strongest element is the firm line that the lack of follow-through on promised visits is unacceptable, which names the core grievance clearly. Its weakness is that the writing is largely expected. Phrases such as 'express my frustration and disappointment' and 'look forward to hearing from you soon' are stock complaint-letter language, and the email offers no specific detail about the call dates or the promised technician that would make it more persuasive.

Llama-3.2-3B-Instruct gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 8/10

The response delivers a complete complaint email with a subject line and stays within the 120–180-word range. It includes all required facts and requests a technician within 48 hours plus a credit. The tone is polite but firm, though some phrasing is generic and it adds unsupported references to numerous attempts and a stable connection.

The response clearly presents Dana Whitfield’s service outage, the two missed technician commitments, the 48-hour request, and the request for a billing credit. Its subject line and closing make it immediately usable as an email, and the tone remains appropriately firm without becoming hostile. A minor weakness is that it introduces details not stated in the prompt, such as “numerous attempts” and a “stable internet connection,” while relying on fairly standard complaint language. These additions do not obscure the central request or undermine the email’s effectiveness.

Judge B: J-A (anthropic/claude-haiku-5.5) 6/10

Delivers a subject line and a complete email within the 120 to 180 word range. It includes the six days without service, the two broken technician promises, the 48-hour visit request and the credit request. Phrasing is generic ('disappointment and frustration', 'take immediate action') and it adds an unstated 'numerous attempts' detail.

The response meets every hard constraint: subject line, length, the 48-hour visit request, the credit request and both earlier broken promises. Its tone stays polite and firm throughout. The weakness is that the prose relies on stock complaint phrasing and gives the reader little that is specific to this customer. It also asserts 'numerous attempts' to contact support, which the brief does not state.

Llama2 13B Chat gap 2

Judged by J-O + J-A.

Judge A: J-O (openai/gpt-5.6-luna) 8/10

The response delivers a 120–180-word complaint email with a subject line, identifies Dana Whitfield, states the six-day outage and two missed technician promises, and requests a visit within 48 hours plus a six-day credit. It is polite, firm, clear, and well structured, though the wording is somewhat conventional and adds an unsupported claim of financial loss.

The response clearly fulfills the requested format and includes every specified fact and requested remedy. Its subject line is informative, and the progression from outage, to failed support commitments, to the 48-hour deadline and credit request is easy to follow. The tone remains appropriately courteous while expressing urgency. However, the prose relies on familiar complaint-email phrasing and adds a claim of “financial loss” that was not supplied in the prompt. That addition is minor, but a more precise version would avoid introducing consequences not established by the stated facts.

Judge B: J-A (anthropic/claude-haiku-5.5) 6/10

Meets the constraints: subject line present, body about 165 words, and it covers the six-day outage, the two broken technician promises, the 48-hour visit request and the credit. Phrasing is generic ('I have been patient and understanding', 'unacceptable length of time'), and it adds an unstated 'financial loss for my family' detail.

The response delivers the requested complaint email with a subject line and a body within the 120 to 180 word range. It covers every fact in the brief: the six days without service, the two earlier calls that promised a visit, the 48-hour deadline and the credit request. Its main weakness is generic, cliché-driven wording, such as 'unacceptable length of time' and 'take immediate action'. It also invents a detail the brief does not supply, a financial loss for the customer's family, which a careful reader would notice as an embellishment.

The other 198 models

Their judges were closer together. Their scores are listed here; open the model's page for the full answer.

ModelJudgesJudge AJudge BGap
Meta-Llama-3-8B-Instruct J-O + J-A 8 6 2
Mistral Small3.1 24B Instruct 2503 J-O + J-A 8 6 2
NVIDIA Nemotron 3 Nano 30B-A3B J-O + J-A 8 6 2
Olmo2 7B 1124 Instruct J-O + J-A 6 4 2
Phi3 14B Medium 128K Instruct J-O + J-A 7 5 2
Phi4 Mini Reasoning 3.8B J-O + J-A 5 3 2
Qwen 14B Chat V1.5 J-O + J-A 8 6 2
Qwen2.5 Coder 0.5B Instruct J-O + J-A 3 1 2
Qwen2.5-32B-Instruct J-O + J-A 7 5 2
Qwen2.5-Coder-32B-Instruct J-O + J-A 8 6 2
Qwen3-30B-A3B J-O + J-A 8 6 2
Qwen3-4B-Instruct-2507 J-O + J-A 8 6 2
Qwen3-8B J-O + J-A 8 6 2
Qwen3-VL-32B-Instruct J-O + J-A 8 6 2
Qwen3.5 2B J-O + J-A 4 6 2
Qwen3.5-4B J-O + J-A 4 6 2
YI 6B Chat V1.5 J-O + J-A 5 7 2
Apertus-8B-Instruct-2509 J-O + J-A 7 6 1
Aya 8B 23 J-O + J-A 7 6 1
Aya Expanse 32B J-O + J-A 7 6 1
Codegemma 7B Instruct V1.1 J-O + J-A 7 6 1
Codellama 34B Instruct J-O + J-A 5 6 1
Codellama 7B Instruct J-O + J-A 4 5 1
Cogito 14B V1 Preview Qwen J-O + J-A 7 6 1
Cogito 8B V1 Preview Llama J-O + J-A 7 6 1
Command R 35B 08 2024 J-O + J-A 7 6 1
Command R7B (12-2024) J-O + J-A 7 6 1
Deepcoder 1.5B Preview J-O + J-A 7 6 1
Deepcoder 14B Preview J-O + J-A 5 6 1
Deepseek Llm 7B Chat J-O + J-A 5 6 1
Deepseek R1 1.5B Qwen Distill J-O + J-A 5 4 1
Deepseek R1 8B Llama Distill J-O + J-A 5 6 1
DeepSeek-Coder-V2-Lite-Instruct J-O + J-A 5 6 1
DeepSeek-R1-0528-Qwen3-8B J-O + J-A 7 6 1
DeepSeek-R1-Distill-Qwen-14B J-O + J-A 7 6 1
DeepSeek-R1-Distill-Qwen-32B J-O + J-A 7 6 1
DeepSeek-R1-Distill-Qwen-7B J-O + J-A 7 6 1
Devstral 24B Small 2505 J-O + J-A 7 6 1
Devstral Small 2 24B Instruct (2512) J-O + J-A 7 6 1
EXAONE-3.5-7.8B-Instruct J-O + J-A 5 6 1
Exaone3.5 2.4B Instruct J-O + J-A 7 6 1
Exaone3.5 32B Instruct J-O + J-A 7 6 1
Falcon2 11B J-O + J-A 7 6 1
Falcon3 10B Instruct J-O + J-A 7 6 1
Falcon3 3B Instruct J-O + J-A 7 6 1
Flint 40B A2B J-O + J-A 5 6 1
Gemma 2B Instruct V1.1 J-O + J-A 4 5 1
Gemma 4 12B IT J-O + J-A 7 6 1
Gemma 7B Instruct V1.1 J-O + J-A 7 6 1
gemma-3-12b-it J-O + J-A 7 6 1
gemma-3-27b-it J-O + J-A 7 6 1
gemma-3-4b-it J-O + J-A 7 6 1
Gemma2 27B Instruct J-O + J-A 7 6 1
Gemma2 2B Instruct J-O + J-A 7 6 1
Gemma3n E2B IT J-O + J-A 7 6 1
Gemma4 E4B IT J-O + J-A 7 6 1
GLM-4.5-Air J-O + J-A 7 6 1
GLM-4.7-Flash J-O + J-A 7 6 1
gpt-oss-120b J-A + J-D 6 7 1
gpt-oss-20b J-A + J-D 6 7 1
granite-4.1-3b J-O + J-A 7 6 1
Granite3 Dense 2B Instruct J-O + J-A 7 6 1
Granite3 Dense 8B Instruct J-O + J-A 7 6 1
Granite3 Moe 3B Instruct J-O + J-A 7 6 1
Granite3.1 Dense 2B Instruct J-O + J-A 5 6 1
Granite3.1 Dense 8B Instruct J-O + J-A 7 6 1
Granite3.2 8B Instruct J-O + J-A 7 6 1
Granite3.3 8B J-O + J-A 7 6 1
Granite4 1B H J-O + J-A 7 6 1
Granite4 3B J-O + J-A 5 6 1
Granite4 Micro H J-O + J-A 5 6 1
Granite4 Small H J-O + J-A 7 6 1
Granite4.1 30B J-O + J-A 5 6 1
Granite4.2 30B J-O + J-A 7 6 1
Granite4.2 3B J-O + J-A 5 6 1
Hermes-3-Llama-3.1-8B J-O + J-A 7 6 1
Hermes3 3B Llama3.2 J-O + J-A 5 4 1
Internlm2 1.8B Chat V2.5 J-O + J-A 5 6 1
Lfm3 1.7B J-O + J-A 7 6 1
Llama-2-7b-chat-hf J-O + J-A 5 6 1
Llama-3.3-70B-Instruct J-O + J-A 7 6 1
Llama3.2 1B Instruct J-O + J-A 7 6 1
Magistral Small (2506) J-O + J-A 7 6 1
Mathstral 7B V0.1 J-O + J-A 7 6 1
medgemma-4b-it J-O + J-A 7 6 1
Medgemma1.5 4B IT J-O + J-A 7 6 1
Ministral 3 14B Instruct (2512) J-O + J-A 7 6 1
Ministral 3 3B Instruct 2512 J-O + J-A 7 6 1
Ministral 3 8B Instruct 2512 J-O + J-A 7 6 1
Mistral 7B Instruct V0.3 J-O + J-A 7 6 1
Mistral Small 22B Instruct 2409 J-O + J-A 7 6 1
Mistral Small 24B Instruct 2501 J-O + J-A 7 6 1
Mistral Small 3.2 24B Instruct (2506) J-O + J-A 7 6 1
Mistral-7B-Instruct-v0.2 J-O + J-A 5 6 1
Mixtral 8x7B Instruct V0.1 J-O + J-A 5 6 1
Muse Glimmer 30B J-O + J-A 7 6 1
Nemotron 3.5 Bolt E4B J-O + J-A 7 6 1
Nemotron 3.5 Lightning 30B A3B J-O + J-A 7 6 1
Nemotron Cascade 2 30B A3B J-O + J-A 7 6 1
Nemotron Mini 4B Instruct J-O + J-A 7 6 1
Olmo 3.1 32B Instruct J-O + J-A 7 6 1
Olmo-3-7B-Instruct J-O + J-A 7 6 1
Olmo2 13B 1124 Instruct J-O + J-A 7 6 1
Openthinker 7B V2 J-O + J-A 5 4 1
Ornith 9B J-O + J-A 7 6 1
Phi-3-mini-4k-instruct J-O + J-A 7 6 1
phi-4 J-O + J-A 7 6 1
Phi3 3.8B Mini 128K Instruct J-O + J-A 7 6 1
Qwen 0.5B Chat V1.5 J-O + J-A 4 5 1
Qwen 4B Chat V1.5 J-O + J-A 4 3 1
Qwen 7B Chat V1.5 J-O + J-A 7 6 1
Qwen2 1.5B Instruct J-O + J-A 4 5 1
Qwen2 Math 1.5B Instruct J-O + J-A 3 2 1
Qwen2-VL-7B-Instruct J-O + J-A 7 6 1
Qwen2.5 0.5B Instruct J-O + J-A 4 5 1
Qwen2.5 1.5B Instruct J-O + J-A 7 6 1
Qwen2.5 Coder 1.5B Instruct J-O + J-A 7 6 1
Qwen2.5-14B-Instruct J-O + J-A 7 6 1
Qwen2.5-3B-Instruct J-O + J-A 5 6 1
Qwen2.5-Coder-14B-Instruct J-O + J-A 7 6 1
Qwen2.5-Coder-3B-Instruct J-O + J-A 7 6 1
Qwen2.5-Coder-7B-Instruct J-O + J-A 5 6 1
Qwen2.5-VL-32B-Instruct J-O + J-A 5 6 1
Qwen2.5-VL-3B-Instruct J-O + J-A 7 6 1
Qwen2.5-VL-7B-Instruct J-O + J-A 5 6 1
Qwen3 0.6B J-O + J-A 5 4 1
Qwen3 1.7B J-O + J-A 5 4 1
Qwen3 VL 2B Instruct J-O + J-A 7 6 1
Qwen3 VL 2B Thinking J-O + J-A 7 6 1
Qwen3-14B J-O + J-A 7 6 1
Qwen3-30B-A3B-Instruct-2507 J-O + J-A 7 6 1
Qwen3-4B J-O + J-A 7 6 1
Qwen3-4B-Thinking-2507 J-O + J-A 7 6 1
Qwen3-VL-30B-A3B-Instruct J-O + J-A 7 6 1
Qwen3.5-35B-A3B J-O + J-A 7 6 1
Qwen3.5-9B J-O + J-A 7 6 1
Qwen3.6-35B-A3B J-O + J-A 7 6 1
QwQ 32B J-O + J-A 7 6 1
Sage-A1.5-Q4_K_M J-O + J-A 7 8 1
Smollm2 1.7B Instruct J-O + J-A 7 6 1
Smollm2 135M Instruct J-O + J-A 4 3 1
SmolLM3-3B J-O + J-A 7 6 1
Stablelm2 12B Chat J-O + J-A 5 4 1
Translategemma 12B IT J-O + J-A 7 6 1
Tulu3 8B J-O + J-A 7 6 1
YI 34B Chat V1.5 J-O + J-A 7 6 1
YI Coder 1.5B Chat J-O + J-A 4 3 1
YI Coder 9B Chat J-O + J-A 4 5 1
Aya Expanse 8B J-O + J-A 7 7 0
Codegeex4 9B All J-O + J-A 5 5 0
Codegemma 2B Code V1.1 J-O + J-A 1 1 0
Codellama 13B Instruct J-O + J-A 5 5 0
Codeqwen 7B Chat V1.5 J-O + J-A 4 4 0
Codestral 22B V0.1 J-O + J-A 5 5 0
Deepscaler 1.5B Preview J-O + J-A 5 5 0
Deepseek Coder 1.3B Instruct J-O + J-A 0 0 0
Deepseek Coder 33B Instruct J-O + J-A 0 0 0
Deepseek Coder 6.7B Instruct J-O + J-A 0 0 0
Dolphin Phi 2.7B V2.6 J-O + J-A 5 5 0
dolphin-2.9.1-yi-1.5-34b J-O + J-A 5 5 0
gemma-4-26B-A4B-it J-O + J-A 7 7 0
Granite Code 3B Instruct J-O + J-A 4 4 0
Granite Code 8B Instruct J-O + J-A 5 5 0
Granite3 Moe 1B Instruct J-O + J-A 4 4 0
Granite3.1 Moe 1B Instruct J-O + J-A 4 4 0
Granite3.2 2B Instruct J-O + J-A 4 4 0
Granite3.3 2B J-O + J-A 7 7 0
Granite4.2 8B J-O + J-A 7 7 0
Lfm2 24B J-O + J-A 7 7 0
Medgemma 27B IT J-O + J-A 7 7 0
Mistral Nemo 12B Instruct 2407 J-O + J-A 5 5 0
Nemotron3 33B J-O + J-A 7 7 0
Olmo-3-7B-Think J-O + J-A 7 7 0
Ornith-1.5-9B J-O + J-A 7 7 0
Phi 2.7B Chat V2 J-O + J-A 4 4 0
Phi-4-mini-instruct J-O + J-A 5 5 0
Qwen 1.8B Chat V1.5 J-O + J-A 4 4 0
Qwen2 0.5B Instruct J-O + J-A 6 6 0
Qwen2 Math 7B Instruct J-O + J-A 5 5 0
Qwen2.5-7B-Instruct J-O + J-A 5 5 0
Qwen3-32B J-O + J-A 8 8 0
Qwen3-Coder-30B-A3B-Instruct J-O + J-A 7 7 0
Qwen3-Coder-Next J-O + J-A 7 7 0
Qwen3-VL-4B-Instruct J-O + J-A 7 7 0
Qwen3-VL-8B-Instruct J-O + J-A 7 7 0
Qwen3.5-27B J-O + J-A 7 7 0
Qwen3.6-27B J-O + J-A 5 5 0
Qwen3.8-27B J-O + J-A 7 7 0
Sailor2 1B Chat J-O + J-A 4 4 0
Sailor2 8B Chat J-O + J-A 5 5 0
Smallthinker 3B Preview J-O + J-A 6 6 0
Smollm2 360M Instruct J-O + J-A 4 4 0
Solar 10.7B Instruct V1 J-O + J-A 5 5 0
Spark-X2.6-4B J-O + J-A 7 7 0
Stable Code 3B Instruct J-O + J-A 2 2 0
Stablelm Zephyr 3B J-O + J-A 4 4 0
Stablelm2 1.6B Chat J-O + J-A 4 4 0
YI 9B Chat V1.5 J-O + J-A 5 5 0

Withheld

These models are kept out of the ranking, so none of their scores or reasons are shown:

The judge room names the judges and shows where they disagree most.