Evaluation frameworks

Evaluation frameworks is a page under evals: tools that hold a dataset of examples, run a program or prompt against every one of them, grade each result, and let two runs be compared, instead of a team building that machinery from scratch. Some also record a trace of what happened inside each run, not only the final answer.

Five of them are set out side by side below, described only in their own documentation’s words, read September 19, 2026. There is no ranking and no “best for” here: what fits a team depends on things this page cannot know: what is already being traced, who has to read a result, and whether the data being graded may leave the machine at all. A sixth tool, OpenAI Evals, is mid-shutdown and is reported separately rather than compared. Prompt optimization is the technique that consumes whatever one of these produces.

This page is sourced, not measured: what each framework does comes from its own documentation, and no result from any of them exists for this site, so no number below is one it scored.

def sample_for_review(items: list[dict], frac: float = REVIEW_FRACTION, *, seed: int = 0) -> list[dict]:
    """A `frac` sample of the grader's verdicts to hand-check. Deterministic given `seed`: the
    same run re-sampled with the same seed picks the same questions, and a different seed picks
    a different set, so a reviewer can take a second sample without re-running anything."""
    if not items:
        return []
    count = max(1, round(len(items) * frac))
    indexes = sorted(random.Random(seed).sample(range(len(items)), count))
    return [items[i] for i in indexes]

def parse_verdict(text: str) -> bool | None:
    """Read a grader's reply. `True` for PASS, `False` for FAIL, `None` for anything else.

    The grader is told to answer with one word, so the first line, stripped of surrounding
    punctuation, must be exactly that word. Malformed output is ungraded, never correct: a
    grader that has drifted or been talked into prose must show up as an ungraded count on the
    result file, not as a silent run of zeros.
    """
    first_line = text.strip().splitlines()[0] if text.strip() else ""
    word = first_line.strip().strip(".,:;!*_`\"'()[]").upper()
    return VERDICTS.get(word)

Scoring the path, not only the answer

Everything above scores an answer against a golden set. From level 4 up there is a second thing to score: the path the run took to get there. A LangSmith tutorial builds a customer support agent and runs three kinds of evaluation on it. One of the three is the trajectory, defined there as “Evaluate whether the agent took the expected path (e.g., of tool calls) to arrive at the final answer”[11]. Its concepts page makes the same split when it asks what “good” looks like for an agent, listing “correct tool selection and proper argument formatting or trajectory that the agent took”[6].

This sits in the evals topic rather than on a rung because it changes nothing about who decides the next step. It only widens what a grader reads: from the last message to every step before it. The reason to bother starts where the steps are the model’s own. The wrong tool, the right tool with the wrong arguments, six calls where one would do, and a loop that never stops are all failures a final-answer grader can score as a pass.

You do not need it below level 4. A single call has no path, and in a fixed chain the path is the one your code wrote, so scoring it measures your code rather than the model.

The failure mode is scoring the path as an exact sequence. A run that reaches the right answer a different way then reads as a failure, and the score punishes the agent for not being your reference implementation. LangSmith’s own tutorial gives the shape that avoids it: a trajectory evaluation “Compares the actual sequence of steps the agent took against an expected sequence” and “Calculates a score based on how many of the expected steps were completed correctly”[11], which is partial credit rather than a match. Report it beside the answer score, never instead of it: a perfect path to a wrong answer is still a wrong answer.

Failure modes

How to Evaluate It

There is no example on this page, because adopting a framework is a decision rather than a technique the 60-question set can run. What can be checked is the framework itself, and the check is the same whichever one is installed: a claim that a system improved needs the same dataset and the same grading configuration run before and after, a hand-checked sample of any rubric grader’s verdicts, and an honest count of what went ungraded instead of being scored as a failure. If a tool cannot tell you which of those three it did, that is the finding.

For this site’s own runner, python scripts/eval_run.py --example rag --model stub --dry projects what a real run would cost before anything is spent; it is described in full on the evals page. Whether each tool above offers the same projection is a question for its own documentation.

Run it

Try it

1. Use it

   Find a dashboard or report at work, or in a project you use, that shows an eval score. Can you tell from it how items were graded, and how many were not graded at all? Ask whoever set it up; the answer is usually available and rarely on screen.

2. Build it

   Run python scripts/eval_run.py --example rag --model stub --dry from the repo root and read the projected token count. Then open the documentation of one tool above and look for its own dry run or cost projection. Note whether you found one, and how long it took to find.

3. Either lane

   Pick one of the five tools above and read its documentation for how it treats an item that errored or whose grader reply could not be parsed. Excluded from the score, counted as a failure, or not stated anywhere? All three answers happen.