How to evaluate AI writing without flattening its voice

Writing quality is several decisions

One overall score hides important failures. A polished paragraph can invent a source. A factual paragraph can waste the reader’s time. A draft can satisfy both checks while replacing a distinct voice with stock phrases. Editors need to see which part failed and whether revision can repair it.

OpenAI’s evaluation best-practices guide recommends task-specific datasets with typical, edge, and adversarial cases. Apply that advice to the editorial job rather than to an abstract measure of prose quality.

A six-step writing eval

1. Define the publication decision. Write down what the eval will permit. A drafting assistant may only need to produce useful raw material, while an automated product description needs strict factual, policy, and formatting gates.
2. Collect real briefs and outcomes. Start with 20 to 50 briefs that represent the actual mix. Keep accepted copy, rejected copy, editor notes, source packets, audience, format, and publication risk. Add hard cases rather than filling the set with easy examples.
3. Separate the dimensions. Write one criterion for each independent concern. Give every score an observable anchor, such as “all checkable claims have supporting evidence” or “the first paragraph answers the stated question.” Avoid labels such as good, engaging, or professional unless examples define them.
4. Run repeated blinded comparisons. Generate several drafts with the same brief, sources, prompt, length, and tool access. Randomize order and hide model identity. Pairwise choices are often easier for reviewers than assigning a precise score to each draft.
5. Calibrate automated graders. Test deterministic checks first, then rubric-based model graders. Compare each grader with editor labels on a held-out set. Measure agreement for every dimension and risk slice, not only one average.
6. Set release gates and review disagreements. State which failures block publication, which require revision, and which only affect ranking. Sample accepted and rejected work after release. Add failures to the test set without rewriting the held-out cases used for the current decision.

Worked example: an evidence-based software guide

Suppose a team uses an assistant to draft a guide that compares two developer tools. Build 30 cases: 10 straightforward comparisons, six cases with missing evidence, four with stale product behavior, four where tools suit different users, three that contain a tempting vendor claim, and three that require the assistant to say that evidence is insufficient.

For factual support, require each current product claim to point to official documentation and verify that the linked passage supports its scope. For usefulness, ask an engineer whether the decision table leads to the right choice for the stated scenario. For voice, give reviewers three accepted Learnetto paragraphs and three rejected paragraphs with reasons. For structure, check whether the direct answer appears before background material. Mark an invented limitation, false price, or unsupported security claim as a blocking failure.

Generate three drafts per brief. Two editors independently choose the better draft and label each dimension. Discuss disagreements before changing the rubric. If one editor rewards brevity while another expects implementation detail, the disagreement reveals an editorial rule that the team has not yet written down. Test the revised rule on new cases rather than changing scores after seeing the model names.

Test voice with positive and negative anchors

A style guide lists rules, while an eval needs decisions. Choose short passages that editors would accept and passages they would reject. Annotate the exact reason: vague opening, inflated claim, repeated conclusion, misplaced joke, indirect instruction, or wording that the publication avoids. These anchors make “sounds like us” less dependent on memory.

Towards AI’s ToneBench methodology offers one current example. It evaluates scripts for one creator with 10 briefs, five generations per model, nine weighted criteria, several model judges, and human pairwise comparisons. The design is useful because it defines a narrow task and repeats generation. Its leaderboard should not be treated as a general ranking of writing models because the target voice, format, prompts, and judging panel belong to that benchmark.

Preserve room for legitimate variation. Require the traits that matter and avoid rewarding surface imitation. If every accepted paragraph must use the same sentence length, heading pattern, and transition phrase, the eval will flatten the work it is meant to protect.

Choose the cheapest reliable grader for each check

• Code and validators suit links, required fields, forbidden phrases, length bounds, citation format, and schema. They give repeatable results but cannot decide whether an explanation helps its reader.
• Source-grounded checks suit factual claims. Split the draft into claims, retrieve the cited passage, and ask whether that passage supports the exact wording and scope. A valid URL alone does not pass.
• Model graders suit rubric-based comparisons at scale. Give them the brief, sources, criterion, anchored examples, and a structured output. Use separate calls for independent dimensions when one long rubric causes criteria to bleed together.
• Human editors suit ambiguous voice, audience fit, material risk, and final publication. They also provide the reference decisions used to test automated graders.

Anthropic’s guidance on defining success criteria tells teams to define success before building evaluations and combine methods where appropriate. Writing teams can follow the same rule: automate objective checks, calibrate judgment calls, and preserve review where consequences justify it.

Model judges have their own preferences

A model grader may reward verbosity, familiar phrasing, confident tone, or output from a related model family. Research on self-preference in rubric-based evaluation reports that judges can favor their own family and that an ensemble does not automatically remove this bias. Treat automated judgment as a measurement system that needs testing.

Use a confusion matrix for blocking labels, rank correlation for ordered scores, and raw disagreement examples for editorial review. Break results out by topic, format, language, risk, and source availability. If agreement is poor on a high-risk slice, route that slice to people even when the overall number looks acceptable.

Trade-offs and common mistakes

• One draft per brief: generation varies, so one result can exaggerate either success or failure. Repeat enough cases to see the distribution.
• One overall score: averages let a fluent draft hide a factual failure. Keep blocking dimensions separate.
• Style rules without examples: reviewers interpret abstract words differently. Add accepted and rejected anchors with reasons.
• Model identity shown to editors: expectations can influence judgment. Blind the comparison where practical.
• Judge agreement treated as truth: two models can share the same preference. Calibrate against people and deterministic evidence.
• Optimizing the test set: repeated prompt changes can overfit known cases. Preserve a held-out set and add fresh production samples.
• Replacing editorial responsibility: a threshold can route work, but a score cannot own the consequences of publication.