Turn photos and PDFs into records

A clinic’s front desk photographs each new patient’s paper intake form instead of re-typing it (name, date of birth, reason for visit, insurance ID) into the patient system by hand. The form is handwritten, sometimes hard to read, and the record it becomes matters enough that nobody wants a guessed date of birth saved silently.

Document extraction is that job: read the image, fill a fixed set of fields, and have a person confirm the record (correcting whatever the read got wrong) before it’s saved. Nothing here decides whether to save; that always happens once a person has looked.

Example run

one image + "Extract name, date of birth, reason for
visit and insurance ID as JSON. Use \"unknown\" for
anything you cannot read."

Walkthrough

The three steps compose the runnable code already on the multimodal, structured output and human approval pages, each written against the site’s own shared examples (a rating plate, a warranty record in text) rather than a literal intake form. Composing them here means keeping the same three functions and swapping in this job’s own schema (name, date of birth, reason for visit, insurance ID, each allowed to come back "unknown") and its own gate reason (low_confidence whenever a field reads "unknown"). Nothing in this repository ships that form; the diagram is those three examples with this job’s schema in them.

The run above shows a form with one field genuinely too faint to read. The first pass returns three clean fields and "unknown" for the date of birth; the code crops that corner of the image and asks again, the same one-retry contract structured output’s own example uses, and the model still can’t read it: an honest "unknown", not a guessed date. Cropping works because the clinic uses one fixed form, so the code knows where each field sits; a desk handling several layouts has to re-ask on the whole image instead. The record validates either way, since "unknown" is a well-typed string, so the gate is what catches it: low_confidence trips, the run pauses, and a person reads the paper form, fills in the real date and approves. The corrected record is what gets saved.

Say plainly where the data goes. A photographed intake form is patient data and it leaves the machine the moment the request is sent, so where the model runs is settled before how accurate it is. Log the extracted fields, the gate’s reason and the correction a person made, plus the attachment’s size and type rather than the image itself.

What to measure

None of the three techniques here answers a question the site’s shared 60-question set can grade; each has its own measurement instead, described on its own page. Build a labeled set for this job: a batch of photographed forms with the record a person would actually write down beside each one. Two numbers matter most. Field accuracy against those labels, character for character for a date or an ID (the same measure multimodal’s own page describes for a serial number read off a plate), tells you whether a field that validated is also correct. And the gate’s own numbers: the pause rate, and separately, a check of the answers that did not pause, to see whether an “unknown” ever slipped through as a confident-looking guess instead. No result file exists for multimodal, structured output or human approval yet (see docs/EVALS.md), so this recipe claims no score.

Variations

• Route by form type first (intake, insurance update, referral) with routing, if the clinic uses more than one form, each needing a different schema; the extraction and the gate stay the same underneath.
• Loosen the gate on fields that are cheap to get wrong (a misspelled name a later step can fix) while keeping it strict on a date of birth or an insurance ID, rather than one threshold for every field.
• Move to function calling only if saving stops being one fixed action: for instance, if the model must first decide whether this patient already has a record to update instead of a new one to create.
• Preprocess before reading: cropping, deskewing or upscaling the photographed corner that failed, the way the retry in the run above does, is order zero applied to one field rather than the whole form.

Design choices