Match invoices to purchase orders

An accounts payable clerk has three documents open at once: the invoice a supplier sent, the purchase order that authorized the buy, and the receiving log saying what actually came off the truck. The question in front of them is narrow and expensive to get wrong: does this invoice match well enough to pay, or does something about it need a person’s eyes first. Three things go wrong often enough to be worth checking every time: a supplier bills for more than was delivered, a unit price drifts from what was agreed, or an invoice’s own arithmetic does not add up to its own total. Any of those, paid without a look, is money out the door that nobody notices until the books do not close.

Nothing here reads a scan or a PDF directly; that step, turning a photographed or emailed document into text, is its own recipe and this one starts after it. Nothing here cuts a check either: posting means the match cleared for payment, not that money moved. And nothing here reconciles a vendor’s running account balance or handles a credit memo; it is one invoice against the one purchase order it names. The household version of this same seam is a household’s own paperwork, where nobody is billed by a stranger for a delivery a stranger also controls the record of; a business needs the extra check because the two sides of the transaction don’t trust each other by default.

Example run

"Corrigan Fasteners, Invoice #INV-77012, PO Number: PO-4410,
FST-2201 qty 40 @ $12.50, FST-2209 qty 40 @ $6.40,
Total due: $756.00"

Walkthrough

The read comes first, and it is the only step that touches a model:

def _extract_invoice(text: str, model: Model, tracer: Tracer) -> tuple[ExtractedInvoice | None, list[str]]:
    """Ask the model for the fixed fields, validate the reply, and retry once with the
    validation error appended if it fails. The only model call in this recipe."""
    messages = [Message(role="system", content=SYSTEM_PROMPT), Message(role="user", content=text)]
    problems: list[str] = []
    for attempt in range(MAX_RETRIES + 1):
        completion = model.complete(messages, schema=SCHEMA, max_tokens=300)
        tracer.record(
            kind="model", decided_by="code",
            title="Read the invoice into fixed fields" if attempt == 0 else "Ask again with the validation error",
            detail=completion.text[:200],
            tokens_in=completion.tokens_in, tokens_out=completion.tokens_out, ms=completion.ms,
        )
        try:
            record = json.loads(completion.text)
            problems = _validate(record)
        except json.JSONDecodeError as exc:
            record, problems = {}, [f"invalid JSON: {exc}"]
        tracer.record(kind="code", decided_by="code", title="Validate against the schema", detail="; ".join(problems) or "valid")
        if not problems:
            return _record_to_invoice(record), []
        if attempt < MAX_RETRIES:
            messages.append(Message(
                role="user",
                content=f"That did not validate: {'; '.join(problems)}. Reply again with corrected JSON only.",
            ))
    return None, problems

Against SAMPLE_INPUT, a clean invoice from Corrigan Fasteners naming PO-4410, the model replies with a supplier, a PO number, an invoice number, a currency, two line items and a stated total, all in the fixed shape the schema names. It validates on the first try, so there is no retry step in this run; a reply that didn’t parse as JSON, or that was missing a field, would get one more chance with the validation error appended to the prompt before the run gives up and pauses rather than posting on a guess.

def run(
    invoice_text: str,
    model: Model,
    tracer: Tracer,
    *,
    purchase_orders: dict[str, PurchaseOrder] = PURCHASE_ORDERS,
    goods_received: dict[str, tuple[ReceivedLine, ...]] = GOODS_RECEIVED,
) -> PostedInvoice | PendingMatch:
    invoice, problems = _extract_invoice(invoice_text, model, tracer)
    if invoice is None:
        return _pause(tracer, None, "", tuple(problems), "extraction_failed")

    po = purchase_orders.get(invoice.po_number)
    tracer.record(kind="code", decided_by="code", title="Look up the purchase order",
                  detail=f"{invoice.po_number}: found" if po else f"{invoice.po_number}: not on file")
    if po is None:
        return _pause(tracer, invoice, invoice.po_number, (f"no purchase order {invoice.po_number!r} on file",), "unknown_po")

    received = {line.code: line.quantity_received for line in goods_received.get(po.po_number, ())}
    discrepancies = _three_way_match(invoice, po, received)
    tracer.record(kind="code", decided_by="code", title="Compare invoiced, ordered and received",
                  detail="; ".join(d.detail for d in discrepancies) or "no discrepancies")
    if discrepancies:
        return _pause(tracer, invoice, po.po_number, tuple(d.detail for d in discrepancies), "mismatch")

    tracer.record(kind="code", decided_by="code", title="Post to accounts payable",
                  detail=f"{po.po_number} {invoice.invoice_number}: {invoice.stated_total_cents} cents")
    return PostedInvoice(
        po_number=po.po_number, invoice_number=invoice.invoice_number,
        supplier=invoice.supplier, posted_cents=invoice.stated_total_cents,
    )

From there run looks up PO-4410 (found, two lines, both fully received per GOODS_RECEIVED), runs the three subtractions above, finds nothing outside tolerance, and posts: $756.00 to Corrigan Fasteners against PO-4410, invoice INV-77012. A different invoice, one line billed a cent over what the order was placed at, takes the same path up through the comparison and then pauses instead, with the exact cent figure named in the reason a reviewer reads. resume is the second half, called separately once a person has actually looked: approve posts the invoice as the supplier stated it, on the reviewer’s own authority, and reject sends it back unposted with whatever note explains why. Nothing about resuming is a model decision either; it is the same decided_by="code" as everything upstream of it, recording a choice a person already made.

What it costs

Illustrative, not measured: no result file exists for this technique yet, so every number below is a worked example. See docs/EVALS.md for how a real run is recorded and scored.

The unit here is per invoice, and the monthly figure is worked from that at a stated volume, 500 invoices, rather than assumed: 236 tokens in and 68 out, measured from the scripted run above, times 500 is about 152,000 tokens a month, and a retry on every single one would roughly double it. Whatever a real accounts payable desk’s own volume is, the number that matters is not the token cost, which is small at any volume this job runs at: it is that the match itself costs nothing extra to run correctly every time, because it was never a model call to begin with.

How it fails

What to measure

A right answer here is not a label a person assigns; it is whatever the accounts payable clerk would have decided reading the same three documents, which makes building a labeled set mostly free: pull invoices already paid last month, and record whether each one should have posted cleanly or should have paused, against the purchase order and receiving record it actually matched. Twenty or thirty is enough to start, since the check itself is arithmetic and what is being scored is really the extraction step, not the comparison.

The confusion that matters is not symmetric. A false post, an invoice the match should have caught but didn’t, pays money out that has to be clawed back or written off. A false pause, an invoice that was actually fine, costs a clerk a few minutes of review and nothing else. Score the first direction, and only the first direction, as the number to drive toward zero; the zero-cent tolerance above is already a deliberate choice in that direction, and the right response to seeing too many false pauses is to look at why the extraction disagrees with the purchase order, not to loosen the tolerance until the gate stops catching real ones. No result file exists for this recipe, so it claims no score, only this method for building one.

Variations

• Widen the tolerance from zero to a small guardband once a real month of false pauses shows the drift is rounding rather than typos, and say so on the ledger: a tolerance is a business decision made once, in the open, not a default this recipe should quietly assume for you.
• Add the duplicate-invoice-number check this recipe deliberately leaves out, against a table of what has already posted, before the gate runs at all.
• Feed this recipe from document extraction once invoices arrive as scans or photographs instead of text a PDF reader already pulled out.
• Move the purchase order lookup from an in-memory table to a live query against a real ERP system once one exists; that is still level 0 code calling an API, not a reason to reach for function calling, unless the model itself starts choosing when to look something up.

Design choices

def _three_way_match(invoice: ExtractedInvoice, po: PurchaseOrder, received: dict[str, int]) -> list[Discrepancy]:
    """The join and its three subtractions: quantity against goods received, unit price against
    the order, and the invoice's own stated total against its own lines, recomputed. A line the
    purchase order does not carry is flagged on its own, before any of the three subtractions run
    against it."""
    discrepancies: list[Discrepancy] = []
    if invoice.currency != po.currency:
        discrepancies.append(Discrepancy(
            kind="currency", code=None,
            detail=f"invoice is in {invoice.currency}, {po.po_number} was placed in {po.currency}",
        ))
    po_lines = {line.code: line for line in po.lines}
    for line in invoice.lines:
        po_line = po_lines.get(line.code)
        if po_line is None:
            discrepancies.append(Discrepancy(
                kind="unknown_line", code=line.code,
                detail=f"{line.code} is not a line on {po.po_number}",
            ))
            continue
        received_qty = received.get(line.code, 0)
        if line.quantity != received_qty:
            discrepancies.append(Discrepancy(
                kind="quantity", code=line.code,
                detail=f"{line.code}: invoiced {line.quantity}, received {received_qty} ({line.quantity - received_qty:+d})",
            ))
        price_diff = line.unit_price_cents - po_line.unit_price_cents
        if abs(price_diff) > TOLERANCE_CENTS:
            discrepancies.append(Discrepancy(
                kind="unit_price", code=line.code,
                detail=f"{line.code}: invoiced at {line.unit_price_cents} cents, ordered at "
                       f"{po_line.unit_price_cents} cents ({price_diff:+d} cents)",
            ))
    recomputed = sum(line.quantity * line.unit_price_cents for line in invoice.lines)
    total_diff = invoice.stated_total_cents - recomputed
    if abs(total_diff) > TOLERANCE_CENTS:
        discrepancies.append(Discrepancy(
            kind="total", code=None,
            detail=f"invoice states {invoice.stated_total_cents} cents but its own "
                   f"{len(invoice.lines)} line(s) sum to {recomputed} cents ({total_diff:+d} cents)",
        ))
    return discrepancies