Grade against a rubric, with a second reader

A teacher has a written rubric for a short assignment: four criteria, each with its own point levels (a stated position, a specific piece of evidence, an accurately described counterargument, a paragraph structure that holds together), and a stack of short submissions to get through. Scoring one is close reading: how many points a paragraph earns on each line, and the quote that earned it. That does not get easier by the fiftieth submission, and the lines most likely to be misread quickly are not the mechanical ones; they are the ones asking whether a sentence actually does what the rubric describes.

What the teacher gets back for each submission is one of two things: a proposed grade, criterion by criterion, with the quote behind every score, ready to enter as is; or a checkpoint naming which criterion is in dispute and why. Nothing here posts a grade to a gradebook. A proposed grade still waits on the teacher entering it, and a checkpoint hands them the disagreement rather than resolving it. Out of scope: writing feedback for the student, a separate job once a score exists; designing the rubric itself; and averaging two numbers into one, which this recipe deliberately never does, for reasons the failure modes below spell out.

Example run

A short essay on extending the lunch period, four rubric criteria: thesis,
evidence, counterargument, organization

Walkthrough

The submission below is the one the diagram plays: a hedged position, a general observation instead of a concrete detail, a paragraph claiming there is no opposing view “because everyone agrees,” and a thin close. The grader reads it against all four criteria in one call and returns a point value and a quote for each:

def run(
    submission: str,
    model: Model,
    tracer: Tracer,
    *,
    max_rounds: int = MAX_ROUNDS,
) -> ProposedGrade | Checkpoint:
    raw = _grade(submission, model, tracer)
    scored, unevidenced = _check_evidence(raw, submission, tracer)
    scores_text = _scores_summary(scored, unevidenced)
    verdict, forced = _review(submission, scores_text, model, tracer, max_rounds)

    if forced:
        reason: Literal["rejected", "unevidenced", "round_cap"] = "round_cap"
    elif verdict.upper().startswith("REJECT"):
        reason = "rejected"
    elif unevidenced:
        reason = "unevidenced"
    else:
        reason = None  # type: ignore[assignment]

    if reason is not None:
        tracer.record(kind="code", decided_by="code", title="Send to the teacher as a checkpoint", detail=f"reason={reason}: {verdict}")
        return Checkpoint(submission=submission, reason=reason, detail=verdict, scores=tuple(scored), unevidenced=tuple(unevidenced))

    total = sum(s.points for s in scored)
    max_total = sum(c["max_points"] for c in CRITERIA)
    tracer.record(kind="code", decided_by="code", title="Assemble the proposed grade", detail=f"{total}/{max_total}")
    return ProposedGrade(submission=submission, scores=tuple(scored), total_points=total, max_points=max_total)

The scores come back low across the board except counterargument, which the grader gives 3 of 4 points, quoting “I don’t really have a counterargument because everyone agrees lunch should be longer anyway.” Code checks all four quotes against the submission before the reviewer ever sees them:

def _check_evidence(raw_scores: list[dict], submission: str, tracer: Tracer) -> tuple[list[CriterionScore], list[str]]:
    """Code's own decision: a score stands only if its quote is actually in the submission,
    character for character. A quote that does not appear is not corrected or reworded -- the
    criterion it was scoring is dropped from the count and marked unevidenced instead."""
    scored: list[CriterionScore] = []
    unevidenced: list[str] = []
    for entry in raw_scores:
        cid = entry.get("criterion")
        quote = entry.get("quote", "")
        if cid in CRITERIA_BY_ID and quote and quote in submission:
            scored.append(CriterionScore(criterion=cid, points=entry["points"], quote=quote))
        elif cid in CRITERIA_BY_ID:
            unevidenced.append(cid)
    for cid in CRITERIA_BY_ID:
        if cid not in {s.criterion for s in scored} and cid not in unevidenced:
            unevidenced.append(cid)  # the grader never returned this criterion at all
    tracer.record(
        kind="code",
        decided_by="code",
        title="Check every quote against the submission",
        detail=f"evidenced: {', '.join(s.criterion for s in scored) or 'none'}; unevidenced: {', '.join(unevidenced) or 'none'}",
    )
    return scored, unevidenced

Every quote is real, so nothing gets marked unevidenced. That is the trap: a verbatim check proves the words are in the submission, not that they mean what the score claims. The reviewer gets the submission, the rubric, and these four scores and quotes, nothing else; the grader’s own sentence of reasoning for each score never reaches it. On its first turn it asks to check counterargument specifically; code hands back that criterion’s full rubric text (describes a real opposing position accurately and responds to it, rather than dismissing it or asserting that no one holds it). On its second turn the reviewer rejects, naming the criterion: a sentence that says an opposing view does not exist is not a sentence that describes one, whatever points the grader gave it.

def _review(submission: str, scores_text: str, model: Model, tracer: Tracer, max_rounds: int) -> tuple[str, bool]:
    """Runs the reviewer's turns until it gives a verdict or the round cap forces one. Returns
    the verdict text and whether the cap forced it."""
    checked: list[tuple[str, str]] = []
    rounds = 0
    while True:
        if rounds >= max_rounds:
            tracer.record(kind="code", decided_by="code", title="Round cap reached", detail=f"{rounds} checks >= {max_rounds}; forcing a verdict")
            completion = model.complete(
                [Message(role="system", content=REVIEWER_SYSTEM), Message(role="user", content=FORCE_VERDICT)],
                max_tokens=60,
            )
            verdict = completion.text.strip()
            tracer.record(
                kind="model",
                decided_by="code",
                title="Reviewer forced to a verdict",
                detail=verdict,
                tokens_in=completion.tokens_in,
                tokens_out=completion.tokens_out,
                ms=completion.ms,
            )
            return verdict, True
        turn = _reviewer_turn(submission, scores_text, checked, model, tracer)
        if turn.upper().startswith("CHECK:"):
            cid = turn.split(":", 1)[1].strip()
            text = _criterion_block(cid)
            tracer.record(kind="code", decided_by="code", title="Return that criterion's rubric text and the submission again", detail=text[:200])
            checked.append((cid, text))
            rounds += 1
        else:
            return turn, False

Both reviewer turns are decided_by: "model"; everything else in this run, including the lookup that answers the first turn, is decided_by: "code". A submission whose quotes and scores hold up on the reviewer’s first look needs only that one turn and comes back a proposed grade instead of a checkpoint. A submission the reviewer keeps circling on, past MAX_ROUNDS turns, never gets a voluntary verdict: code forces one, recorded as code’s decision, the same way examples/debate_review/run.py’s own round cap does, which this reviewer’s shape follows on purpose.

What it costs

Illustrative, not measured: no result file exists for this technique yet, so every number below is a worked example. See docs/EVALS.md for how a real run is recorded and scored.

The unit here is per submission, since a teacher reads a checkpoint as it appears rather than waiting for the whole stack to finish. Two of the three calls above are unconditional: the grader’s own pass always happens, and the reviewer has to see the proposed scores at least once before it can accept, reject or check something. What is not fixed is how far past that first look it goes: scores that hold up need nothing more, a real disagreement costs one more call for the check itself, and a submission the reviewer keeps circling on costs a fourth once the round cap steps in. Across a class of thirty short essays that is on the order of a hundred calls and tens of thousands of tokens a stack, and that cost lands on whoever is running the model, not on the teacher directly. It buys one thing: a second reader that checks what the rubric’s own words say on the criterion most likely to be misread, instead of every criterion getting the same shallow look twice.

How it fails

What to measure

Build the labeled set from submissions a teacher has already graded by hand: twenty to thirty short essays across a couple of assignments give well over a hundred individual criterion judgments to compare against, since four criteria score independently on every submission. Track two rates separately: how often a proposed grade (every criterion evidenced, the reviewer accepting) matches what the teacher gave by hand, and how often a checkpoint’s stated disagreement turns out to be one the teacher agrees was worth raising, not a false alarm.

The confusion that matters is not a point value off by one. It is a checkpoint never being raised on a criterion a teacher, reading alone, would have caught as wrong: a wrong score reaching a grade nobody double-checked, worth driving toward zero even at the cost of more checkpoints on submissions that turn out fine. A checkpoint raised too often costs a few minutes of a teacher’s attention; one that should have been raised and was not costs a grade nobody can defend later.

No result file exists for this example. Running the grader and reviewer against a set of already graded submissions is what would produce one, and until then this recipe claims no score.

Variations

• Swap in a different assignment’s rubric and submissions. CRITERIA and the schema’s criterion list are the only things that change; the grade-then-check pattern, the verbatim evidence check and the CHECK/ACCEPT/REJECT loop carry over unchanged.
• Loosen the gate once real hand-graded data shows most unevidenced criteria turn out fine on a second look, the same way human approval’s own advice is to set a threshold from answers that turned out wrong before, not from a guess.
• Drop to write and check for a rubric whose criteria really are mechanical, a word count or a citation format a single fixed test can check the same way twice; the day every line is that mechanical is the day this level stops earning its cost.
• The same shape checks a manuscript against a submission guideline or a grant application against a funder’s rules: written criteria, findings that quote the work, and a second independent reader for the criteria too expensive to get wrong on one pass.

Design choices