Turn a measurement session into a report somebody can review

This is an engineering-test page. Five revision C prototypes of the Orbeck SRB-5030 spent three days on the bench, swept over line, load and temperature, and the sweep is done: the margins are computed, the uncertainty budget behind them is computed, the guardbanded verdicts are computed. What is left is the afternoon’s other job, turning a lab notebook (evals/bench/corpus/characterization-notebook.md) and a table of numbers into a short characterization report somebody else can read without re-deriving any of it. Code has already decided every number in that report. What it has not decided is how to say it in sentences that connect one finding to the next and do not quietly drop a caveat the notebook is honest about.

The run, stepped

compute_results reads evals/bench/data/characterization-2026-09.csv and returns the corner margins for all five boards, the uncertainty budget for the thinnest one, and board SRB5030-2609-0005’s line regulation figures and verdicts at all three ambients. The corner budget is the same one that page computes, for the same five readings: board SRB5030-2609-0003 at 9.0 V in, 3.000 A out, 70 degC ambient, on the 10 V range, meter at 23 degC, one year calibration row.

That number belongs to those five readings and to nothing else, which matters because two nearby figures are in the documents this report is written from. The meter’s manual works a budget for a different reading and gets 349.5 uV, and the notebook works one for a generic reading of this session and gets 351.6 uV. A report that quotes either of those for this corner is quoting the wrong measurement, and the check below rejects it for that reason and not by accident.

Against that budget the corner margin is 20.4 mV, about fifty-eight times the expanded uncertainty, so the measurement is not in doubt even though the margin is thin. Board SRB5030-2609-0005’s line regulation at 25 degC is a different case: 0.299 percent against a 0.300 percent limit, with 0.0075 percentage points of expanded uncertainty on the difference. guarded_verdict returns "cannot say" there, "pass" at 0 degC and "fail" at 70 degC, and all three verdicts are figures the model is handed, not conclusions it draws. The k of 2 is stated with every one of those numbers because it is part of them: it covers roughly 95 percent of where the true value could be, on the usual assumption that the contributions are independent and roughly normal once they are combined, and it is not a guarantee that the value is inside.

Every figure is formatted once, as a Figure(label, text) pair, and that formatted string is the only form the model is allowed to reproduce:

def compute_results(csv_path: Path = CHARACTERIZATION_CSV) -> Results:
    """Read the characterization sweep and compute the corner margins, one uncertainty budget
    and the line-regulation verdicts this report needs. Nothing here is a model's arithmetic, and
    nothing the model is handed lets it redo this arithmetic differently.
    """
    rows = _read_rows(csv_path)
    serials = sorted({r["serial"] for r in rows})
    days = sorted({r["timestamp"][:10] for r in rows})

    figures: list[Figure] = [
        Figure("first day of the sweep", _mdy(days[0])),
        Figure("last day of the sweep", _mdy(days[-1])),
        Figure("output voltage minimum (V)", f"{VOUT_MIN_V:.3f}"),
        Figure("line regulation limit (%)", f"{LINE_REG_MAX_PCT:.3f}"),
        Figure("coverage factor (k)", f"{COVERAGE_FACTOR:.0f}"),
        Figure("corner ambient (degC)", CORNER_AMBIENT_TEXT),
        Figure("corner input voltage (V)", CORNER_VIN_V),
        Figure("corner load current (A)", CORNER_IOUT_A),
        Figure("uncertain block input voltage (V)", UNCERTAIN_BLOCK_VIN_TEXT),
        Figure("lead and connection uncertainty assumption (uV)", f"{LEAD_HALF_WIDTH_V * 1e6:.0f}"),
    ]

    corner_margins_mv: dict[str, float] = {}
    for serial in serials:
        mean = statistics.fmean(_readings(rows, serial, CORNER_TAMB_C, CORNER_VIN_V, CORNER_IOUT_A))
        margin_v = margin_to_limit(mean, VOUT_MIN_V, side="lower")
        corner_margins_mv[serial] = 1000.0 * margin_v
        figures.append(Figure(f"{serial} corner output voltage (V)", f"{mean:.5f}"))
        figures.append(Figure(f"{serial} corner margin (mV)", f"{1000.0 * margin_v:.1f}"))

    thin_serial = min(corner_margins_mv, key=corner_margins_mv.get)
    corner_readings = _readings(rows, thin_serial, CORNER_TAMB_C, CORNER_VIN_V, CORNER_IOUT_A)
    corner_expanded_v = expanded_uncertainty(
        combined_uncertainty(dc_voltage_budget(corner_readings, range_v=DC_RANGE_V))
    )
    corner_expanded_uv = 1e6 * corner_expanded_v
    corner_verdict = guarded_verdict(
        statistics.fmean(corner_readings), corner_expanded_v, lower=VOUT_MIN_V, upper=VOUT_MAX_V
    )
    figures.append(Figure("thin-margin board corner expanded uncertainty (uV)", f"{corner_expanded_uv:.1f}"))

    line_reg_pct = {tamb: line_regulation_pct(rows, MARGINAL_LINE_SERIAL, tamb) for tamb in ("0.0", "25.0", "70.0")}
    reg_readings = _readings(rows, MARGINAL_LINE_SERIAL, "25.0", "32.0", "1.000")
    per_reading = combined_uncertainty(
        dc_voltage_budget(reg_readings, range_v=DC_RANGE_V, lead_half_width_v=None)
    )
    reg_expanded_v = expanded_uncertainty(per_reading * math.sqrt(2.0))
    line_reg_uncertainty_pct = 100.0 * reg_expanded_v / VOUT_NOM_V
    figures.append(
        Figure("marginal-line board regulation uncertainty (percentage points)", f"{line_reg_uncertainty_pct:.4f}")
    )
    line_reg_verdicts: dict[str, str] = {}
    for tamb, pct in line_reg_pct.items():
        figures.append(Figure(f"marginal-line board regulation at {tamb} degC (%)", f"{pct:.3f}"))
        line_reg_verdicts[tamb] = guarded_verdict(pct, line_reg_uncertainty_pct, upper=LINE_REG_MAX_PCT)

    return Results(
        figures=tuple(figures),
        corner_margins_mv=corner_margins_mv,
        thin_serial=thin_serial,
        corner_expanded_uncertainty_uv=corner_expanded_uv,
        corner_verdict=corner_verdict,
        line_reg_pct=line_reg_pct,
        line_reg_uncertainty_pct=line_reg_uncertainty_pct,
        line_reg_verdicts=line_reg_verdicts,
    )

The notebook’s own prose (evals/bench/corpus/characterization-notebook.md, loaded whole) and the figures block go into one prompt. The system message is explicit that every number in the reply has to be copied from the figures, character for character, and that anything else (a count, a comparison between ambients) gets written in words, not digits. One call, and the model’s only job is the sentences:

def run(
    notes: str,
    model: Model,
    tracer: Tracer,
    *,
    csv_path: Path = CHARACTERIZATION_CSV,
) -> Report:
    """One model call. `notes` is free text a requester may add (which finding to lead with, a
    house style note); it is appended to the prompt and never reaches the check, because it never
    contributes a number of its own.
    """
    results = compute_results(csv_path)
    tracer.record(
        kind="code",
        decided_by="code",
        title="Compute the figures from the characterization data",
        detail=(
            f"{len(results.figures)} figures; thin-margin board {results.thin_serial} "
            f"({results.corner_margins_mv[results.thin_serial]:.1f} mV, {results.corner_verdict})"
        ),
    )

    notebook = load_bench_documents()["characterization-notebook"]
    tracer.record(kind="code", decided_by="code", title="Load the session notebook", detail=f"{len(notebook)} characters")

    user_parts = [
        f"Figures (use only these numbers, exactly as written):\n{_figures_block(results.figures)}",
        f"Notebook:\n{notebook}",
    ]
    if notes and notes.strip():
        user_parts.append(f"Additional guidance from the requester: {notes.strip()}")
    messages = [
        Message(role="system", content=SYSTEM_PROMPT),
        Message(role="user", content="\n\n".join(user_parts)),
    ]
    tracer.record(
        kind="code",
        decided_by="code",
        title="Build the prompt with the figures and the notebook",
        detail=f"{len(results.figures)} figures, {len(notebook)}-character notebook",
    )

    completion = model.complete(messages, max_tokens=700)
    tracer.record(
        kind="model",
        decided_by="code",
        title="Ask the model to draft the report",
        detail=completion.text[:200],
        tokens_in=completion.tokens_in,
        tokens_out=completion.tokens_out,
        ms=completion.ms,
    )

    unsupported = unsupported_numbers(completion.text, results.figures)
    tracer.record(
        kind="code",
        decided_by="code",
        title="Check the draft's numbers against the figures",
        detail="every number is supported" if not unsupported else f"unsupported: {', '.join(unsupported)}",
    )
    return Report(text=completion.text, figures=results.figures, unsupported=unsupported)

What it costs

Illustrative, not measured: no result file exists for this technique yet, so every number below is a worked example. See docs/EVALS.md for how a real run is recorded and scored.

This is an engineering-test cost, counted per run and per session, not per unit: the notebook this recipe reads took three days on the bench, and the report about it is written once. A tenth of a cent a run is not a number worth optimizing; the number worth comparing it against is the afternoon a person would otherwise spend writing the same paragraphs by hand, and the half hour lost if a report ships with a number nobody checked, which is what the check above exists to catch before it costs that much.

How it fails on a real bench, specifically

How to evaluate it

There is no 60-question set to grade this against: scripts/eval_run.py measures document question answering over evals/corpus/, and this recipe reads a different document set and a different data set entirely. What it measures instead is the share of drafts in which every figure in the prose is one that code produced, which unsupported_numbers turns into a plain pass or a fail rather than a judgment call: run a session’s worth of drafts, count how many come back clean, and a number under 100 percent means the prompt is losing the “copy the figure, do not compute one” instruction, not that the arithmetic underneath is wrong.

That check is necessary and not sufficient, for the reasons the list above it gives. Before trusting this recipe on a real session, keep a small set of past reports (five or six is enough at this volume, since there is no golden run to score against and no second unit to catch a mistake the way production volume would) and read each one against its own notebook, checking the claim each number sits next to and the open items the report was supposed to carry forward.

How to adapt it

Every figure above, and the notebook it is drawn from, is this bench’s own. A real characterization session has its own datasheet limits, its own uncertainty budget and its own notebook format; the part that ports is the shape, not the numbers: compute every figure first, format each one exactly once, hand the model the figures and the source text and nothing else it could compute a number from, and check the draft’s numbers against the figures before anyone reads it.

That shape is not specific to a bench. It is turning one piece of text into another with the facts pinned down in advance: a test report written from a results table a suite already produced, release notes drafted from a list of commits, a status update written from what a tracker already says about a sprint. In every case the numbers or the facts exist before the model is asked to write anything, so the same rule applies: a model writes the sentences, code supplies and checks the facts, and nothing downstream trusts the prose for a number it could have gotten from the table instead.

Where this stays level 1 and does not climb to the kind of pipeline accuracy-specs-from-the-manual needs, a schema, a bounded retry and a person’s recorded sign-off: one draft, read by one person, with nothing else consuming it. It would move to write and check the day a report started generating unattended on a schedule with nobody reading each one before it went out, or the day something downstream, a dashboard, another program, a second model, started parsing this report’s prose instead of a person reading it. Neither is true here, and a page that added a second model pass anyway would be spending a call to guard against a risk this recipe does not have.

Design choices

Build it

def unsupported_numbers(draft: str, figures: Sequence[Figure]) -> tuple[str, ...]:
    """Every numeric token in `draft` that is not, character for character, one of `figures`'
    own text. In reading order, and it may repeat a token, so a report that leans on one bad
    number three times shows all three.

    Dates are matched first and checked whole, so 09/14/2026 is one token and not the three
    numbers 09, 14 and 2026. Identifiers are then blanked, so a serial number is not read as a
    quoted measurement. What is left is scanned for numbers.

    This is the whole safety argument for making one model call write a report nobody re-derives
    by hand: it costs two regular-expression scans and a set lookup, and it is a pass or a fail,
    never a judgment call. What it cannot do is check that a real figure is sitting next to the
    claim it belongs to, notice a caveat the draft dropped, or see a digit buried inside a word;
    this recipe's page names each of those and says what catches it instead.
    """
    allowed = {figure.text for figure in figures}
    scanned = _blank_identifiers(DATE_RE.sub(lambda m: " " * len(m.group()), draft))
    hits = [(m.start(), m.group()) for m in DATE_RE.finditer(draft)]
    hits += [(m.start(), m.group()) for m in NUMBER_RE.finditer(scanned)]
    return tuple(token for _, token in sorted(hits) if token not in allowed)