What this video shows
Bouchard defines weak generated writing through six recurring problems: stock wording, vague claims, low substance, repetition, unsupported claims, and a voice that could belong to anyone. His team tests a narrower question: whether a model can produce a YouTube script that one creator can read naturally.
The published setup uses 10 briefs, five drafts per model and four judges. The judges score nine criteria, while human reviewers compare 1,000 pairs. Use the method as an example of task-specific evaluation rather than a universal model ranking. Your rubric should reflect your own readers, format, evidence rules, and approval decision.
Read the ToneBench methodology for the current tasks, weights, judges, and limitations. OpenAI recommends task-specific evals and human calibration when automated scores influence a release decision.
What you will learn
- Split factual support, usefulness, voice, structure, and stock phrasing into separate checks.
- Generate several drafts because one run can flatter or punish a variable model.
- Hide model identity during review and randomize pair order.
- Keep a human editor responsible for the final publication decision.
How to apply this safely
- Collect 20 real briefs with accepted copy, rejected copy, and written reasons.
- Write anchored examples for each rubric score before testing a model.
- Run each candidate several times with the same brief, sources, prompt, and limits.
- Compare automated labels with blinded editor labels and inspect disagreements before scaling.
Important limitations
- ToneBench tests long-form scripts in one creator’s voice, so its model order does not establish general writing quality.
- Towards AI designed the benchmark and publishes the associated leaderboard, so its methodology and conclusions need independent scrutiny.
- Model judges can prefer output from their own family and can reward longer prose.
- Human preferences vary across audience, language, format, and publication risk.
Sources to check
- ToneBench methodology Towards AI documents the benchmark scope, criteria, weights, repeated drafts, and judging panel.
- Self-preference bias in rubric-based evaluation Research on model-family preference in LLM judges and the limits of judge ensembles.
- Why AI Writing Still Reads Like Slop AI Engineer video published October 22, 2026.
Continue learning on Learnetto
Evaluate AI writing
Build a rubric, test set, and calibrated review process.
AI evals guide
Choose graders and turn failures into regression cases.
Verify AI citations
Check factual support separately from prose quality.