Turn an incident write-up into a runbook

Somebody who was on call last night has a write-up: a timestamped account of what they checked, what they tried, and what happened, written while the incident was still open or right after it closed. They want something different from it than the postmortem will eventually hold: a short list of steps a person can follow the next time the same kind of thing happens, each one naming who does it and how they will know it worked. What they have is one incident’s account, in whatever mix of full sentences and shorthand a tired person types at three in the morning. What they want back is not a summary of the night and not a finding about the root cause; both already have a home, one in the postmortem and one in whatever ticket tracks the bug. This recipe turns the notes into steps, and nothing else. It does not decide what caused the incident, and it does not decide that any step belongs in the runbook forever. A runbook drawn from a single write-up is a first draft, built from whatever happened to be true on one particular night, and the person who was there is the one who says which parts of it still hold on an ordinary day.

Example run

"02:14 - on-call page fires for jrivera..."
27 lines, one incident, Thistledown Labs order-sync backlog

Walkthrough

No recorded run exists for this recipe yet, so what follows is a stepped walkthrough of the code against SAMPLE_INPUT, run with a scripted stand-in for the model’s replies, the same one the test file checks against.

def run(writeup: str, model: Model, tracer: Tracer) -> PendingApproval:
    tracer.record(kind="code", decided_by="code", title="Read the write-up", detail=f"{len(writeup.splitlines())} line(s)")
    timeline = _extract_timeline(writeup, model, tracer)
    drafts = _draft_steps(timeline, model, tracer)
    steps, dropped = _verify_steps(drafts, timeline, tracer)
    tracer.record(
        kind="code", decided_by="code", title="Hold the draft for the incident owner's approval",
        detail=f"{len(steps)} step(s) pending, {sum(1 for s in steps if s.incomplete)} incomplete, {len(dropped)} dropped",
    )
    return PendingApproval(writeup=writeup, timeline=tuple(timeline), steps=tuple(steps), dropped=tuple(dropped))

The write-up is 27 lines. The first model call returns seven events, timestamped from the page firing at 02:14 to the incident owner closing it at 03:25, and code assigns each one an id, e1 through e7, in the order they came back. The second call sees that numbered timeline and drafts four steps: check the queue-depth figure (e2), restart the worker pool (e4), roll back only alongside a restart rather than in place of one (e5, the one event that is a lesson more than an action), and check the lag metric came back down (e6). Two events do not become steps: deciding to wake a specific person (e3) and drafting an email to specific customers (e7), both true of this incident and not obviously true of the next one.

def _verify_steps(drafts: list[dict], timeline: list[TimelineEvent], tracer: Tracer) -> tuple[list[RunbookStep], list[str]]:
    known = {e.id for e in timeline}
    kept: list[RunbookStep] = []
    dropped: list[str] = []
    for draft in drafts:
        from_event = draft.get("from_event", "")
        if from_event not in known:
            dropped.append(f"{draft.get('action', '(no action)')!r} cites event {from_event!r}, which is not in the timeline")
            continue
        role = (draft.get("role") or "").strip()
        check = (draft.get("check") or "").strip()
        problems = [p for p, missing in (("no role", not role), ("no check", not check)) if missing]
        kept.append(RunbookStep(
            n=len(kept) + 1, action=draft.get("action", ""), role=role, check=check,
            from_event=from_event, incomplete=bool(problems), problems=tuple(problems),
        ))
    tracer.record(
        kind="code", decided_by="code",
        title="Verify every step traces to a real event and names a role and a check",
        detail=f"{len(kept)} kept, {len(dropped)} dropped, {sum(1 for s in kept if s.incomplete)} incomplete",
    )
    return kept, dropped

Verification is where this recipe keeps its promise, and the one step that never touches the model. A step whose from_event names an id nothing in the timeline has is dropped, with the made up citation recorded rather than silently discarded: test_a_step_citing_an_event_not_in_the_timeline_is_dropped_and_reported plants exactly that. A step with no check, or no role, is kept and flagged incomplete instead of dropped: a step nobody can tell has worked is still worth a person’s attention. What verification cannot do is tell a real repeatable action from a one-off the model generalized by mistake. “Call dcho and wake them” and “check the queue-depth figure” can both carry a perfectly formed role and check; nothing about their shape marks one as belonging to this incident alone. test_a_one_off_action_with_a_plausible_role_and_check_passes_verification_unflagged proves the negative directly: that step is not dropped and not flagged. Nothing in code catches it, which is the argument for what comes next.

def resume(pending: PendingApproval, decision: Decision, tracer: Tracer, *, note: str = "") -> Runbook:
    tracer.record(
        kind="code", decided_by="code", title="Resume from checkpoint with the incident owner's decision",
        detail=f"decision={decision}" + (f" note={note!r}" if note else ""),
    )
    if decision == "approve":
        return Runbook(text=pending.text, steps=pending.steps, approved=True)
    if decision == "edit":
        return Runbook(text=note, steps=pending.steps, approved=True)
    return Runbook(text="The incident owner rejected this draft; no runbook exists.", steps=(), approved=False)

The draft, however it came out, is a PendingApproval and never anything more until the incident owner reads it and calls resume. Approving ships the draft text as written; editing replaces it with whatever the reviewer typed, while still keeping the same steps for citation; rejecting ships nothing, because a runbook one person read and refused is not a runbook anybody should follow.

What it costs

Illustrative, not measured: no result file exists for this technique yet, so every number below is a worked example. See docs/EVALS.md for how a real run is recorded and scored.

The unit here is the incident, not the year and not the team. A company has a handful of incidents worth writing a runbook from, not thousands of units a day, so there is no volume to divide this cost by and no honest way to say what it costs “at scale.” Two calls and about 1,200 tokens turn one write-up into a draft; the figure that matters is how much of the hour or two a person would spend writing it by hand this saves, against how much of that hour they still spend reading the draft closely enough to catch what it got wrong.

How it fails

What to measure

A right answer here is not a label a script can compare against a key. It is a step a person who ran the incident reads and either approves as written, edits, or rejects, and the record worth building is that decision itself: keep every draft alongside what happened to it, incident by incident. There is no set to collect before this is usable, because there is nothing to hold out; a company sees a handful of incidents worth a runbook in a year, and the first real draft is already the first data point.

The confusion that matters is not accuracy across steps evenly. It is the difference between a step verification drops or flags and one it lets through clean. A step dropped for citing nothing, or flagged for missing a check, stays visible in the draft as a problem someone can see and fix in a minute: the cheap direction. A one-off promoted into a step that looks exactly as complete as the real ones is expensive, because nothing marks it and it is only caught if the reviewer reads closely enough to ask “would I really do this again.” Watch that direction, not the counts.

No result file exists for this recipe (see docs/EVALS.md), so nothing here is a score: only what to start writing down, chiefly how often a rejected or edited step turns out to have been a one-off nothing in code caught.

Variations

• Feed the same chain a different after-the-fact narrative: a deployment rollback log, a support escalation thread, a lab notebook entry from a failed experiment. The shape carries over unchanged; only the categories of “would be done again” and “specific to this one” move.
• Once a few months of approved and rejected drafts exist, retune what verification flags as incomplete from what reviewers actually caught, rather than from a guess.
• A runbook is not permanent because it was once approved. Nothing here re-checks a runbook against a system that has since changed, and nothing should be trusted to have done that silently; treat an old approval as a reason to re-run this on a fresh write-up, not as proof the steps still hold.
• Turning requirements into a test plan is the same shape, extract-draft-verify-approve, over a stronger source: a requirement is written down on purpose, and a write-up is only an account of what happened to occur.

Design choices