Drafting with a reviewer

Every week a product team turns a handful of bullet points about what shipped into an update email a customer would actually want to read. Someone has to make sure the email covers everything that shipped, and separately, that it reads like the rest of the company’s writing: no unverified superlatives, no promises the bullets didn’t make.

Drafting with a reviewer splits those two checks apart, because they’re different kinds of question. Whether every bullet made it into the draft is something plain code can check for itself. Whether the draft’s tone matches the style guide is not: that needs a second prompt, told the rule, checking the first prompt’s work.

Example run

"- Shipped dark mode
- Fixed the sync bug reported by several users
- Raised the free-tier project limit from 3 to 5"

Walkthrough

The pipeline composes the runnable code already on the prompt chaining and evaluator-optimizer pages, in the order described above, with this job’s own outline-and-rubric content in place of their own citation-checking example. Neither example writes email in the repository; the diagram is their two loops chained and pointed at this job.

The run above starts with three bullets and an outline that covers all of them, so the chain’s own gate passes without a second model call. Decide in advance what a dropped bullet does: the gate can ask for the outline once more, or stop and hand the bullets back, but it must not pass a draft that was never going to mention the thing that shipped. The draft that follows reads fine on its surface but uses “amazing” (a word the style rubric flags as an unverified superlative), and the checker’s one-word verdict is specific enough that the revision fixes exactly that, nothing else, on the first try. PASS_TOKEN is the entire contract the checker and the code share: the code never has to judge what “good” means, only whether that exact token came back.

What to measure

Evaluator-optimizer and prompt chaining are each scored on their own by the site’s shared 60-question set (see docs/EVALS.md); this job needs its own set instead, since it drafts an email rather than answering a question about documents. Build one from real or synthetic bullet lists with a known-good outline and a small style rubric written down in advance. Three numbers: outline coverage (does every bullet’s subject appear, the same test the chain’s own gate runs); the average revisions per draft and the share that hit the cap still failing the rubric, the two numbers evaluator-optimizer’s own page tracks; and, run twice on the same draft, whether the checker’s verdict agrees with itself: a checker that doesn’t is adding cost without adding reliability, which its own page warns against directly. No result file exists for either technique on this task yet, so this recipe cannot claim a score for any of it.

Variations

• Add a second, unrelated checker for a second rubric (length, a required legal footer) rather than widening one prompt’s criteria; two narrow checks stay more testable than one broad one.
• Move to review and debate once the same kind of error keeps passing the checker: evidence the writer and the checker share a blind spot rather than that the rubric needs one more rule.
• Route by content type first with routing if the team drafts more than one kind of message (a release note, a status update) needing a different outline shape and a different rubric.
• Add human approval as a last step before sending, for the first several weeks, until the rubric has actually been checked against enough real drafts to trust unattended.

Design choices