Reviewing work you did not do

Reviewing is checking work you did not do yourself before it goes anywhere. It is a different skill from doing the work, and it gets harder rather than easier as a system does more on its own, because more of the work happened somewhere you were not watching.

Two things decide whether a piece of work can be reviewed at all. There has to be something to check it against (a source, a record, a number you can recompute) and enough of the work has to be visible to check, not just its conclusion. When neither is true, “review” means reading something plausible and agreeing with it.

How much review a result needs is not a fixed amount. It scales with what a wrong answer would cost and how long the mistake would sit there before anyone noticed. That is the same question delegating asks before handing a task over at all, asked again about the thing that came back; calibrating trust is what the answers to it, kept over time, add up to. All four skills on the operator craft topic start from what the request said, which is briefing.

This page is sourced, not measured: the checks below are drawn from primary sources, and no result file exists for any of them, so no number here is one this site took.

def figures_in(text: str) -> list[str]:
    """Every figure the text states, in one canonical spelling each, sorted and deduplicated."""
    figures = []
    for word in _PERCENT_RE.sub(r"\1%", text).split():
        word = word.strip(_TRIM)
        parts = [word] if _ISO_DATE_RE.match(word) else _RANGE_RE.split(word)
        for part in parts:
            figure = _figure(part)
            if figure is not None:
                figures.append(figure)
    return sorted(set(figures))

def run(answer: Answer, sections: dict[str, Section], tracer: Tracer) -> ReviewReport:
    figures = figures_in(answer.text)
    tracer.record(
        kind="code",
        decided_by="code",
        title="Read the figures the answer states",
        detail=", ".join(figures) or "none",
    )

    flags: list[Flag] = []
    found_somewhere: set[str] = set()
    for citation in answer.citations:
        section = sections.get(citation)
        if section is None:
            tracer.record(
                kind="code", decided_by="code", title=f"Open {citation}", detail="not in the corpus"
            )
            flags.append(Flag(citation, "cited section does not exist"))
            continue
        here = sorted(set(figures) & set(figures_in(section.text)))
        found_somewhere.update(here)
        tracer.record(
            kind="code",
            decided_by="code",
            title=f"Open {citation}",
            detail=f"{section.title}: {', '.join(here) or 'no claimed figure'}",
        )
        if figures and not here:
            flags.append(Flag(citation, "section carries none of the answer's figures"))

    for figure in figures:
        if figure not in found_somewhere:
            flags.append(Flag(figure, "figure appears in no cited section"))

    tracer.record(
        kind="code",
        decided_by="code",
        title="Report",
        detail=f"{len(flags)} thing(s) to look at across {len(answer.citations)} citation(s)",
    )
    return ReviewReport(figures_claimed=figures, checked=list(answer.citations), flags=flags)

At each level

• Conventional software: there is no output to review, only code to test.
• Direct prompting: one reply, reviewed against what you already know or can look up in the time you were willing to spend.
• Added context: the sources are on screen, so the cheap checks become possible, and a citation that is present is not yet a citation that supports the sentence.
• Workflows: the pipeline is fixed, so you can decide once, in advance, which step is worth a person’s eyes rather than deciding per result.
• Tool use: a proposed action can be reviewed before it runs, which is a cheaper and more useful check than reading about it afterwards.
• Agent loops: nobody reads the whole run, so review becomes a sample plus the final result, and the sample rate is now a number someone has to choose.
• Teams of Agents: several outputs agree with each other, which is not evidence: they can be wrong together, and often from the same starting assumption.
• Always-on agents: review is entirely after the fact, so the question becomes what would have to change to catch this before the next run, not this one.

Practices

• Check the specific claims first (figures, dates, names, citations) then what is missing, then the judgment. The order matters: the first two are fast and the third is where your expertise actually earns its keep.
• Open the source, not a summary of the source written by the system whose work you are checking.
• Set the sample rate before the week starts, not while reading the output.
• Read the record of what was done before any summary of what was done, and the irreversible actions before the rest.
• Count the flags and the skips. A run with none of either is a reason to look harder.

Run it

Try it

1. Use it

   Take the last piece of model-generated work you accepted without much scrutiny. Pull out three specific claims (a figure, a date, a citation) and check each against its source. Note how long it took; that number is what a review costs you.

2. Build it

   From the repo root run python -m examples.reviewing --scenario clean, then --scenario mismatch, then --scenario missing, and read the three reports. Then add a scenario of your own in examples/reviewing/__main__.py whose answer spells a corpus figure differently ("the drain pump costs 52 dollars" against a list price written $52.00) and confirm it still reports clean. Then break it on purpose: change 52 to 5.2 and read the two flags that come back.

3. Either lane

   Write down the sample rate you actually use on work you are supposed to be reviewing: one in three, one in ten, whatever it honestly is. Then write down the rate you would defend to someone whose money or safety depends on it. If the two differ, one of them is wrong.