Check an agreement against your own checklist

A vendor sends over a draft agreement, and somebody on the team already keeps a short list of things it has to say: how long you have to pay an invoice, whether the vendor’s liability is capped, how much notice either side owes before walking away, whether the vendor can hand the contract to someone else, whose state’s law governs if there is ever a dispute, what happens to your data once the relationship ends. The checklist does not change from one vendor to the next; it is the same handful of rules a team decided mattered, usually after one agreement that went badly. What the person reviewing this agreement wants back is not an opinion on whether to sign it. It is a list: for each rule on the checklist, the clause of this agreement that addresses it, a verbatim quote from that clause, and whether the clause meets the rule, breaks it, or is simply absent. This recipe produces that list, from a checklist and an agreement, and nothing more. It does not draft a counter-clause, does not negotiate, does not read anything the checklist did not ask about, and it is not legal advice: the output is a list of things for a person to look at, never a recommendation to sign, walk away, or accept a term as written.

Example run

15-clause invented services agreement
6 rules: payment_terms, liability_cap, notice_period,
assignment, governing_law, data_deletion

Walkthrough

Every call starts the same way, and this is the whole of what one call ever sees:

def _prompt_for_rule(rule: Rule, agreement_text: str) -> str:
    """The whole of what one call sees: this rule, alone, and the agreement. No other rule's id
    or text appears here, which is what a test can check directly against this function's output
    without running a model at all."""
    return f"Checklist rule {rule.id}: {rule.text}\n\nAgreement:\n{agreement_text}"

Six of those go out at once against the invented fifteen-clause agreement this example ships with, a fulfillment services agreement between two invented companies, checked against a checklist a team might actually keep: a payment-term limit, a liability cap, a notice period, an assignment restriction, a governing-law requirement, a data-deletion requirement.

What comes back is uneven, which is the point of checking a real agreement instead of a synthetic yes-or-no test. payment_terms comes back breach: the agreement’s clause 5 gives forty-five days to pay an invoice against the checklist’s thirty-day limit, and the quoted evidence, “within forty-five (45) days of the invoice date,” is a verbatim piece of clause 5. liability_cap and notice_period both come back meets, against clauses 9 and 7. assignment comes back breach: clause 12 lets either party hand the agreement to a third party without the other’s consent, which is exactly what the rule forbids. governing_law comes back unclear: clause 13 sets governing law to “the state in which Vendor maintains its principal place of business” instead of naming one, so the finding says the text does not settle it rather than guessing which state that is. data_deletion comes back missing: nothing in the fifteen clauses says what happens to the client’s data once the agreement ends, which is a different finding from a rule the agreement quietly meets.

def _merge_finding(rule: Rule, completion: Completion, clauses: dict[int, str]) -> Finding:
    """The check this recipe teaches. A finding ships as drafted only if its status is one of the
    four allowed, and, for anything but `missing`, only if the clause it names exists in this
    agreement and the quote is an exact substring of that clause's own text -- not of the
    agreement as a whole, so a real sentence lifted from a different clause than the one cited
    still fails this check."""
    try:
        raw = json.loads(completion.text)
    except (json.JSONDecodeError, TypeError):
        return Finding(rule=rule.id, status="unclear", clause=None, quote="", checked_by="code",
                        note="the model's reply was not valid JSON")

    status = raw.get("status")
    clause_no = raw.get("clause")
    quote = raw.get("quote") or ""

    if status not in _STATUSES:
        return Finding(rule=rule.id, status="unclear", clause=clause_no if isinstance(clause_no, int) else None,
                        quote=quote, checked_by="code",
                        note=f"the model returned a status outside breach/meets/missing/unclear: {status!r}")
    if status == "missing":
        return Finding(rule=rule.id, status="missing", clause=None, quote="", checked_by="model")
    if not isinstance(clause_no, int) or clause_no not in clauses:
        return Finding(rule=rule.id, status="unclear", clause=clause_no if isinstance(clause_no, int) else None,
                        quote=quote, checked_by="code",
                        note=f"cites clause {clause_no!r}, which this agreement does not have")
    if not quote or quote not in clauses[clause_no]:
        return Finding(rule=rule.id, status="unclear", clause=clause_no, quote=quote, checked_by="code",
                        note=f"the quoted text does not appear in clause {clause_no}")
    return Finding(rule=rule.id, status=status, clause=clause_no, quote=quote, checked_by="model")

Every one of those six replies passes through this check before it is a finding rather than just a completion. A quote that is not an exact substring of the clause it names, a clause number this agreement does not have, or a status outside breach, meets, missing and unclear is downgraded to unclear with the reason recorded, never shipped as drafted. tests/test_example_contract_review.py attacks this directly: one test hands the check a reply that quotes clause 7’s own sentence while citing clause 12, a real sentence attached to the wrong clause, and the substring check catches it because it checks the quote against the clause actually cited, not against the agreement as a whole.

def run(
    agreement_text: str,
    model: Model,
    tracer: Tracer,
    *,
    checklist: tuple[Rule, ...] = CHECKLIST,
) -> ReviewCheckpoint:
    text = agreement_text.strip() if isinstance(agreement_text, str) and agreement_text.strip() else AGREEMENT_TEXT
    clauses = _parse_clauses(text)
    tracer.record(kind="code", decided_by="code", title="Read the agreement",
                  detail=f"{len(clauses)} numbered clauses, {len(checklist)} checklist rules")

    # One call per rule, all sent at once; each sees only its own rule and the whole agreement.
    with ThreadPoolExecutor(max_workers=len(checklist)) as pool:
        completions = list(pool.map(lambda r: _check_rule(r, text, model), checklist))

    for rule, completion in zip(checklist, completions):
        tracer.record(kind="model", decided_by="code", title=f"Check {rule.id} alone",
                       detail=completion.text[:200], tokens_in=completion.tokens_in,
                       tokens_out=completion.tokens_out, ms=completion.ms)

    findings = tuple(_merge_finding(rule, completion, clauses) for rule, completion in zip(checklist, completions))
    tracer.record(kind="code", decided_by="code", title="Check every quote against its cited clause",
                   detail=", ".join(f"{f.rule}:{f.status}" for f in findings))

    needs_review = tuple(f for f in findings if f.status != "meets")
    cleared = tuple(f for f in findings if f.status == "meets")
    tracer.record(kind="code", decided_by="code",
                   title="Gate: every breach, missing and unclear finding goes to a person",
                   detail=f"{len(needs_review)} to review, {len(cleared)} cleared")
    return ReviewCheckpoint(agreement=text, findings=findings, needs_review=needs_review, cleared=cleared)

The six findings become a checkpoint, never a final answer. Four of them, payment_terms, assignment, governing_law and data_deletion, are not meets, so all four wait for a person; liability_cap and notice_period are listed too, but need no action from anyone. resume is where a reviewer’s decision is actually recorded, whatever it turns out to be, the same shape as examples/human_in_the_loop/run.py’s own pause and resume.

What it costs

Illustrative, not measured: no result file exists for this technique yet, so every number below is a worked example. See docs/EVALS.md for how a real run is recorded and scored.

The unit worth counting in is per agreement: a checklist like this gets run once per vendor contract, not on a schedule and not per page. At six calls and under five thousand tokens, checking one agreement costs a few cents at any current model’s list price, which is not the number that matters here. What matters is the four findings a person actually has to read before this contract could be signed, against the two that need nothing more than a glance.

How it fails

What to measure

A right answer here is a finding whose status, clause number and quote match what a person reading the agreement by hand would write for the same rule; there is no other test, since the checklist itself defines what counts as right. Build a labeled set from agreements a person has already reviewed: the checklist’s six rules against a handful of past contracts with known outcomes, including at least one agreement where a rule is genuinely missing rather than met or broken, the way data_deletion is missing here. A review runs once per agreement, so a few dozen rule-and- finding pairs, not thousands, is both realistic and about all most teams will ever have.

The confusion that costs the most is a false meets: a clause that actually breaches or omits the rule, reported as satisfied, is the one nobody’s eyes ever land on again, since a meets finding needs no action. Watch that direction separately from a false breach or a false missing, which only cost a person a few minutes confirming a clause that was fine all along. No result file exists for this recipe (see docs/EVALS.md), so it claims no score; what is worth tracking by hand is how often the reviewer’s own decision agrees with each finding’s status, rule by rule, as agreements accumulate.

Variations

• Add a second, independent check on any rule where a false meets would be expensive: the same evaluator-optimizer shape design review against a checklist uses for its own two rules that need reading, run only on the rules a labeled set shows are actually getting missed.
• Swap the checklist for a different fixed set of written rules and nothing about the shape changes: a style and security guide checked against a pull request, a requirements document checked against a test plan, a coverage checklist checked against an insurance policy.
• Feed the agreement in from turning photos and PDFs into records once agreements arrive as scans rather than text with numbered clauses; the checklist and the merge check do not change, only where the text comes from.
• Move to a single agent only once which rules apply actually varies from one agreement to the next, a lease has no assignment-to-a-third-party question the same way a services agreement does; a checklist that is the same every time is exactly what keeps this recipe at level 3.

Design choices