Evals

Evals are how a claim that a change made things better gets checked instead of assumed. A golden set is a fixed list of questions with a known right answer, or a rubric for judging one, run under defined conditions. It can support a comparison before and after a change, but evaluations can also grade a single run or estimate performance across a dataset. Grading can use deterministic code, people, models, or a combination. A rubric grader is itself a model call and can be wrong, so some share of its verdicts needs checking by a person. None of this belongs to one level: a single prompt, a fixed workflow and an agent that runs for hours all make claims only an eval can check.

This is the site’s own second principle, on the Method page: a page here may say a level helped only when a result file backs it. This is a topic, not a level.

This page is sourced, not measured: how an eval works is checked against primary sources, but no eval on this site has a scored result file yet (see docs/EVALS.md), so what follows describes the measuring rather than reporting any of it.

class DryRunModel:
    """Stands in for a real model during `--dry`. Makes no network call, and projects an upper
    bound rather than a likely run.

    Input tokens are counted for real, with `count_tokens`, over the prompt the example actually
    builds. Output tokens are reported as the `max_tokens` the example asked for, on every call:
    that is the most the provider can bill for output, since the example caps every call. Whenever
    tools are offered it calls the first one, every time, so a tool-using example runs to its own
    step cap or token cap. That is a ceiling except where control flow branches on the model's
    own words: `routing` parses one word to pick a handler, so it projects its cheapest branch.
    """

    def __init__(self, requested_id: str) -> None:
        self.model_id = requested_id
        self.calls = 0

    def complete(
        self,
        messages: list[Message],
        *,
        tools: list[dict] | None = None,
        schema: dict | None = None,
        max_tokens: int = 1024,
    ) -> Completion:
        self.calls += 1
        tokens_in = sum(count_tokens(content_text(m.content)) for m in messages)
        tool_calls: list[ToolCall] = []
        if tools:
            first = tools[0]
            arg_name = next(iter(first["parameters"]["properties"]), "query")
            tool_calls = [ToolCall(name=first["name"], arguments={arg_name: "projected"})]
        text = "" if tool_calls else "[dry run projection, no model called]"
        return Completion(
            text=text, tool_calls=tool_calls, tokens_in=tokens_in, tokens_out=max_tokens, ms=0.0, model_id=self.model_id
        )

def stub_refusal(example: str, *, is_stub: bool, allow_stub: bool) -> str | None:
    """The line to print when a stub run is refused a result file, or None when it may write one.

    A stub answers nothing real, so a result file from one measures nothing. It is refused unless
    the caller asks for it outright, and even then the summary carries `"stub": true` so the site
    can refuse to chart it.
    """
    # Named, rather than three lines inside `main`, so a page can pin it by name: a pinned line
    # range over this file has now slid three times, once per wave that grew the runner.
    if is_stub and not allow_stub:
        return f"[{example}] stub model: result not written (pass --allow-stub to write one anyway, marked stub=true)."
    return None

Grading engineering work

The 60-question set grades text answers, which is one shape of grade among several. For a reader who writes code for electronics test, measurement or design, the shape follows the job, and two of the three below are not a score at all.

A drafted script is a pass or a fail. When a model drafts commands for an instrument from that instrument’s programming manual, there is nothing to rate. Each command is in the documented command set or it is not, and the script either runs on the simulated instrument with an empty error queue or it does not. Grading is running it, so the grade is repeatable and no rubric model takes part in it. Drafting an instrument control script works that case through.

A label is a confusion matrix, not an accuracy number. Sorting failing units and operator notes into causes is classification into a fixed set, and the two directions of a mistake cost different amounts: calling a real defect a fixture problem can let a bad board ship, while calling a fixture problem a defect costs an engineer an afternoon. A single accuracy figure averages the two together and hides the one that matters. Sorting failing units into causes names which direction to drive toward zero and which cheaper one to accept more of.

A reported measurement is not graded by a model at all. A margin, an uncertainty, a Cpk or a verdict is arithmetic, and the check on it is that code computed it and a person can reproduce it. Where a model writes the prose around numbers code produced, the grade is mechanical: every figure in the draft appears in the computed results, character for character, or the draft does not ship. Turning a measurement session into a report runs exactly that check in code.

How many graded examples exist to work with depends on the setting rather than on the technique. A production line yields thousands of labeled units a month, so a rate means something. Design verification has five prototype boards and one sweep, so there is no rate to compute and the honest eval is a person reading every case. A precise measurement may happen once, and what gets checked there is the uncertainty budget behind the number, not a score over a set. Prescribing a set size the reader cannot reach is the fastest way to lose them.

Failure modes

At each level

• Conventional software: there is no model call to grade, so this is ordinary software testing: fixed inputs, known outputs, run exhaustively rather than sampled, the way level 0’s own search can be tested exhaustively rather than sampled.
• Direct prompting: grading one prompt’s answer is the simplest case; exact match works when a task has one right phrasing, the way structured output can be scored on whether the reply is valid at all, separately from whether its values are right, and a rubric grader is needed as soon as a task has no one right phrasing.
• Added context: what got retrieved changes the answer, so grading has to check citations as well as the final text: whether RAG’s answer used the passage that actually holds the fact, not merely a plausible one.
• Workflows: a fixed chain of steps can be graded step by step, not only on the final output: prompt chaining’s own citation check is exactly this, so a wrong route or a dropped step shows up even when a later step happens to recover.
• Tool use: grading has to check whether the right tool was called with the right arguments, since function calling’s own failure modes show a wrong tool call can still produce an answer that reads fine.
• Agent loops: the model decides when to stop, so an eval has to weigh how many steps and tool calls a single agent took, and whether it stopped too early or kept going too long, alongside whether the final answer is right.
• Teams of Agents: one agent’s output being graded correct is not enough for a review or debate setup: the agreement between the agents needs measuring too, since two agents can share a blind spot and agree while both are wrong.
• Always-on agents: there is no single run to grade before it ships, the reason long-running tasks’ own eval section gives for why the site’s question set does not apply to it. Evaluation shifts from a one-time offline pass on a fixed set toward continuous scoring of what the agent actually did once it was already running[4].

Practices

• Ask how a number was graded before trusting it: exact match, or a model reading a rubric. The two carry different failure modes.
• Match the grade to the job before picking a tool. A pass or fail from running the thing, a confusion matrix with one named costly direction, and a rubric read by a model are three different measurements, and only the third needs a grader model at all.
• You do not have to build the machinery. Evaluation frameworks sets six tools that run a set, grade it and compare runs side by side, in their own documentation’s words.
• If a grader model is used, check some of its verdicts by hand. This site’s own runner writes 10% of them to a file for exactly that.
• Never treat an ungraded question as a wrong answer, and never let a report quietly do the same; a grader that returns nothing readable is a gap, not a zero.
• Re-run the same question set before and after a change, not a new set each time, or the comparison is not measuring the change.
• Report cost and latency next to the score, not separately. A prompt that scores two points higher at ten times the tokens is a different trade than the score alone shows; the operations topic is where those numbers are managed.

Run it

Try it

1. Use it

   Find a benchmark score a product or model page states, and check whether the page says how the answer was graded and whether a person checked any of the verdicts by hand. If it doesn't say, that's the gap this page is about.

2. Build it

   Run python scripts/eval_run.py --example rag --model stub --dry from the repo root and read the projected tokens it prints, then change TOP_K in examples/rag/run.py from 4 to 2 and run it again. Did the projection change, and does that match what you'd expect from fewer retrieved chunks?

3. Either lane

   Pick one of the five question kinds in evals/questions.json (lookup, multi-hop, numeric, unanswerable, conflicting sources) and write, on paper, one new question of that kind against evals/corpus/: the question, the accepted answer pattern, and which section it must cite.