Learning, from results over time, how much to rely on a model without checking.
Sourced
Concept at a glance
Ending or continuingRevisit reliance when the task, model, or evidence changes.
Trust should be specific to a task and revised when evidence changes.
Same concept, different task and consequences. Switching starts a fresh walkthrough; prior answers and approvals do not carry over.
Calibrating reliance on a model from observed performance in a particular task and context.
Follow a sequence of outcomes into a decision about how much oversight to use next. Inspect whether evidence of reliability transfers to the task now being attempted.
Decide how closely to review an assistant's event plans.
A small labeled performance history, per-task review policy, a novel-case failure, and a justified change in oversight.
The setting makes the example concrete. Carry the underlying pattern into your own work; adapt the sources, tools, and level of oversight to your task.
Decide how closely to review an assistant's event plans.
Authored case. Select any record below; nothing is sent to a model.What changed: Establish the facts supplied for this version of the task.
Success on familiar easy cases may not transfer to new contexts. Confidence should concern a specific capability under specific conditions.
Describe your task to your own model and use Calibrating trust as a reference. Ask whether it fits, which alternatives meet the same automation needs, and how you would implement and check the result.
Calibrating trust means keeping how much you rely on a model without checking in line with how often it has actually been right on tasks like the one in front of you. It is a record, not an impression, and it is built one checked result at a time. That is why reviewing comes first: the reviews are the entries. It is the last of the four skills on the operator craft topic, and what the record is for is the next round of delegating.
Both directions cost something. Over-trust looks exactly like things going well, right up until the wrong answer that mattered goes through unread. Under-trust looks like diligence: re-doing work a model has done reliably a hundred times, or avoiding a task it would genuinely help with because a different task went badly once. Neither shows up as an error anywhere.
The unit of trust is a task type, not a model. Reliably right at summarizing a document you can check says nothing about arithmetic, or about a question outside anything it was trained on. A single global “I trust this one” hides which specific things it has earned.
This page is sourced, not measured: what the makers claim below is quoted from their own pages, and no claim here has been checked against a run of this site’s own.
A record for one person is five columns, takes about twenty seconds a row, and lives wherever you already keep notes:
| Date | Task type | What you checked | Right? | What you changed |
|---|---|---|---|---|
| 3/12/2027 | renewal dates from a contract | all 4 dates against the clauses | yes | nothing |
| 3/12/2027 | plain-language summary of a policy | the 2 exclusions it listed | no | added the third exclusion |
Task type keeps the record usable later: separate lines even when the same product did both. What you changed is the check itself: nothing, a word, or the whole thing. After a month, read down that column for one task type. Mostly “nothing” means you can safely sample instead of reading every one; anything else means you aren’t ready to stop checking, whatever the product’s reputation is. Write the number down; it’s next month’s sampling rate, not a feeling you’ll remember correctly later.
Two rules keep it honest. Log the checks that came back fine, not only the corrections, or the record reads like a catalog of disasters. And start a fresh page whenever what you’re trusting changes: a new model version, an edited prompt, a different tool. Anthropic says exactly this about its own published techniques: where one names a specific model, “treat it as measured on that model and re-check it against your own evals before applying it to another.”[1] A record built on last quarter’s version doesn’t transfer just because the product name did.
Skip it for a one-off task you’ll never ask again, or something so low-stakes a wrong answer costs nothing to fix. Keep it for whatever you catch yourself about to trust from memory instead of a count.
Be wary of research that sounds like it settles this. A 2019 complacency scale was built on Mechanical Turk respondents whose “experience with automation was predominantly with relatively low-stakes and common forms of automation, such as in-car navigation systems,” which its own authors name as a limitation[2]; those participants weren’t supervising a model at work, so treat the finding as a reason to keep your own count, not a number about your job. A 2025 survey defines over-reliance as “relying on LLMs beyond their capabilities”[3] and argues for measurement over impression, which is exactly what the table above is.
A team cannot keep one person’s notebook. What replaces it is a fixed set of checkable questions, scored the same way every time and kept as a file: the evals topic pointed at a running system rather than at a prompt being drafted. “We have been using it a while and it seems fine” is the thing the file exists to replace.
Three properties make a team record worth keeping:
Sample by hand on top of that, at a rate you write down. The machine check and the human sample answer different questions, and an automated number rising while nobody has read an output in six weeks is exactly the state that looks safest and is not.
One design decision belongs here rather than in the record: what the system does when its own confidence is low. A system that can say “I did not find this” gives a reviewer a signal worth logging; one that always produces an answer makes every result look identical from outside, and a record over identical-looking results is much more expensive to keep.
Per task type, the edit rate (how often a result is accepted unchanged) next to the sampling rate actually being used. Those two numbers moving apart is the whole subject of this page, in either direction.
Keeping the record costs roughly the same per checked result at any volume, so the sampling rate, not the traffic, sets the bill. What volume changes is the cost of being miscalibrated: the same error rate is a nuisance at ten results a day and a recall at ten thousand.
The record stops being written before it stops being cited. Months later a decision is justified with 'it has been reliable', and the last entry anyone made was before two model upgrades and a prompt rewrite.
Every checked result against what was actually correct, tagged with the task type and with the model and prompt version that produced it. Untagged accuracy cannot be compared across a change, which is the only comparison that matters.
Pick a task you now let a model do without checking. Write down when you last verified one of its answers and what you found. If you cannot remember, that gap is the distance between your trust and your record, and the next five results are the cheapest rows you will ever add.
Take one output type from a system you use or built. Sketch the automatic check for it (what would a machine compare against what) and say what that check would NOT catch. The second half is what the human sample is for.
Name one task you trust a model on and one you deliberately do not. For each, write the evidence the position rests on. If either answer is a feeling rather than a count, that is the one to start a record for.
Track how a model performs on a specific recurring task before deciding which parts can receive lighter review.
An illustrative task example. No verified product or tool is currently listed for this concept.
No product, tool or model is registered against this page yet. The names index lists every one the site does name, and which technique each belongs to.
Names listed 09/19/2026. 223 of 223 registry entries have been checked against the maker's own page; the registry marks the rest as unchecked.
Last reviewed 09/19/2026. Pages unreviewed for 90 days are flagged for another pass. Markdown version of this page