The agent harness

Start with an ordinary request

The user supplies two starting documents. CLAUDE.md holds reusable framework instructions; DUT_BRIEF.md describes this particular DUT. Establish the framework instructions once and maintain them as conventions evolve. Write a new brief for each DUT.

The agent also needs access to framework documentation, framework source, previous projects, and the standard template. These remain the reference material; the two starting files do not replace them. Point to their actual locations in CLAUDE.md, and ask the agent to read DUT_BRIEF.md when starting the task. The brief, plan, and status filenames are conventions for this example, not special files automatically understood by every agent.

Configure permissions separately. Instructions in Markdown describe the boundaries; they do not themselves make the framework read-only or prevent instrument access.

With those files in place, the user can give this request:

“Read CLAUDE.md and DUT_BRIEF.md, then create a project for this new DUT using our shared Python test framework. Use the closest past project where one is suitable; otherwise use the standard template. Here is my description of how this DUT differs. Read the framework documentation and reuse its existing tools and instrument interfaces. Ask me about missing requirements before proceeding. Show me your proposed files and changes in PROJECT_PLAN.md, and wait for my approval before generating the project. Produce Python code, YAML/JSON configuration, and Markdown documentation, including PROJECT_STATUS.md, for my review. Obtain separate approval before creating any new project-local tool. Do not modify the shared framework or connect to instruments.”

What belongs where

The agent can generate code that imports existing framework functions. Each function does not need its own model-tool definition. The project’s use of an instrument API does not authorize the agent to execute it against connected equipment.

The feedback loop in practice

1. Understand: read the framework documentation, candidate reference projects, and the written description of this DUT’s differences. Ask about missing requirements before filling them in.
2. Plan and wait: identify the closest suitable project or the standard template. Propose the files, intended changes, framework tools to reuse, and non-hardware checks. Obtain the user’s approval before generating project files and code.
3. Generate: create the approved Python files, YAML/JSON configuration, and Markdown documents in the new project’s folder. Reuse the framework’s existing tools wherever applicable.
4. Check without hardware: run approved syntax and configuration checks that cannot connect to instruments. Do not import or execute project setup code unless its lack of hardware access is established. Report the checks performed and their results; do not label the project as hardware-tested.
5. Hand over: the user reviews the project, runs it on real instruments, and supplies logs, results, and observations. The documentation distinguishes generated work from verified behavior.
6. Refine: use that feedback to revise the project within the approved scope. Ask about newly missing requirements, and seek approval for changes that cross the boundaries below.

The agent automates project creation and revision. Hardware testing remains a human-controlled step. A passing syntax or configuration check does not establish that a measurement is correct.

Approval boundaries and hard controls

The shared framework must not be modified without explicit authorization. If a new framework feature is absolutely required, the agent must explain the requirement, why existing capabilities cannot satisfy it, and the proposed change, then wait for the user’s decision. Plan approval for a DUT project is not blanket permission to modify the framework.

A new project-local tool also requires approval. Before creating one, explain the gap, which framework tools were considered, and why a new tool is necessary. Placing a helper in the project folder does not bypass this rule. Missing requirements must be asked about first, rather than silently guessed or left as unapproved TODOs.

These are required boundaries, but a written instruction alone is not a hard enforcement mechanism. Read-only access to the shared framework is a proposed control whose availability still needs to be confirmed. The intended setup gives the agent write access only to the approved DUT project, withholds live instrument access, and keeps permission controls outside files it can rewrite. Any authorized framework change would need a separately scoped exception. This page does not claim those controls are already implemented.

Markdown documentation to hand over

• Capabilities and created files: supported tests, what was generated, and where each part lives.
• Differences from the reference: what changed for this DUT and why.
• Configuration and operation: parameters, limits, expected instruments, connections, setup, cleanup, and instructions for the user to run the project through the framework.
• Requirements checklist: each requested test mapped to its code, configuration, and validation status.
• Validation record: checks the agent actually ran, followed by hardware results the user supplies.
• Open questions, limitations, and approvals: unresolved issues and any requested tool or framework changes.

How the concepts fit together

The harness coordinates these concepts during one task: turn a DUT brief into a reviewable project using an existing framework. Some parts describe what the model sees, some decide what may happen, and others establish what actually happened. The controls below describe the intended setup; they are not a claim that custom checks or restrictions already exist.

One proposed helper, several different controls

Suppose the agent believes it needs a new unit-conversion helper for the DUT:

1. Context and tools: it reads the framework’s existing conversion API and relevant past code. If those already meet the requirement, it uses them in the generated project.
2. Guardrail and approval: if it proposes a new helper, a configured policy check should pause that creation until a specific approval exists. The agent explains the gap and asks the user. Without an implemented check, this remains an instruction the agent is expected to follow.
3. Permissions: approval for a helper in the project does not unlock the shared framework or instrument access. A framework change would require its own authorization and scoped access.
4. Execution and observation: after approval, it writes the helper within scope, runs only approved non-hardware checks, and records the change, approval, and actual results.
5. Evaluation and feedback: the reviewer checks the conversion against the agreed requirement. The user performs any required hardware validation and returns findings. The agent revises within scope or stops and reports the next decision it needs.

The guardrail checks the proposal; the person authorizes an exception; permissions constrain execution; logs record the outcome; evaluation judges whether the result meets the requirement. The harness brings these together around the model’s repeated calls.

Relating this to the code and run below

The runnable demonstration below uses a document lookup task, not this DUT framework. Its ContextPolicy corresponds to selecting the documentation and decisions the model sees; ToolRegistry corresponds to the actions the coding agent can call; a Hook illustrates checking a proposed action before execution; and the caps bound the loop. A veto hook alone is not a human approval workflow: that also needs a pause, a recorded decision, and a way to resume within scope. The trace illustrates observability. The DUT diagram describes how these responsibilities would apply to your project workflow, without claiming the demonstration implements its controls.

You do not need every technique on the site to start this workflow. Reading repository files does not by itself establish a RAG system, a Markdown instruction file is not automatically a packaged skill, and using Python APIs does not require MCP. Those are separate choices if retrieval, reusable procedures, or external tool connections become necessary.

"What does the DW-300's drain pump cost, and
how long is it under warranty?"

def trim_to_budget(budget_tokens: int) -> ContextPolicy:
    """The tight policy: keeps every non-tool-result message, and as many of the most recent
    tool results as fit under `budget_tokens`, dropping older ones first. Real context policies
    trim the same way -- see this page's Use it lane for how Anthropic describes tool result
    clearing and compaction -- this one trims by a plain token count to keep the point readable
    in a few lines.

    A tool result is recognized by its role, `tool`, never by how its text opens: the question is
    a user message, so a policy that matched on text could drop a question that happened to begin
    "Result of ...", leaving the model answering something it can no longer see. A trimmed result
    keeps its call id, because every tool call in the history still needs an answer."""

    def policy(messages: list[Message]) -> list[Message]:
        result_idx = [i for i, m in enumerate(messages) if _is_tool_result(m)]
        kept: set[int] = set()
        used = 0
        for i in reversed(result_idx):
            cost = count_tokens(content_text(messages[i].content))
            if used + cost > budget_tokens:
                break
            used += cost
            kept.add(i)
        out = []
        for i, m in enumerate(messages):
            if i in result_idx and i not in kept:
                out.append(
                    Message(
                        role="tool",
                        content="[earlier tool result trimmed by the context policy]",
                        tool_call_id=m.tool_call_id,
                        tool_name=m.tool_name,
                    )
                )
            else:
                out.append(m)
        return out

    return policy

def deny_after(allowed_calls: int) -> Hook:
    """A hook for demonstration and testing: allows the first `allowed_calls` tool calls the
    model attempts, vetoes every one after. A real hook would read the call's own name and
    arguments; this one only counts, to keep the point -- a hook can block an action the model
    already decided to take -- in three lines."""
    seen = {"n": 0}

    def hook(call: ToolCall) -> tuple[bool, str]:
        seen["n"] += 1
        if seen["n"] > allowed_calls:
            return False, f"tool budget of {allowed_calls} call(s) already spent"
        return True, ""

    return hook

def run(
    question: str,
    model: Model,
    embedder: Embedder | None,
    tracer: Tracer,
    *,
    corpus_dir=DEFAULT_CORPUS_DIR,
    registry: ToolRegistry = DEFAULT_REGISTRY,
    context_policy: ContextPolicy = keep_everything,
    hook: Hook = allow_everything,
    max_steps: int = MAX_STEPS,
    max_tokens: int = MAX_TOKENS,
) -> Answer:
    del embedder  # this harness retrieves through its tools, not a vector index
    sections = load_sections(corpus_dir)
    messages = [Message(role="system", content=SYSTEM), Message(role="user", content=question)]
    citations: list[str] = []
    tokens_used = 0

    for _ in range(max_steps):
        completion = model.complete(context_policy(messages), tools=registry.definitions, max_tokens=400)
        tokens_used += completion.tokens_in + completion.tokens_out

        if not completion.tool_calls:
            record_completion(tracer, decided_by="model", title="Model stops and answers", completion=completion)
            return Answer.from_text(completion.text, retrieved_sources=citations)

        calls_desc = ", ".join(f"{c.name}({c.arguments})" for c in completion.tool_calls)
        record_completion(tracer, decided_by="model", title="Model picks an action", completion=completion, detail=calls_desc)
        turn, calls = assistant_turn(completion, len(messages))
        messages.append(turn)

        for call in calls:
            allowed, reason = hook(call)
            if not allowed:
                tracer.record(kind="code", decided_by="code", title="Hook vetoes the call", detail=reason)
                messages.append(tool_result(call, f"Denied: {reason}"))
                continue
            if call.name not in registry.allowed:
                result_text, cites = toolkit.unknown_tool(call.name)
            else:
                result_text, cites = registry.call(call, sections)
            citations.extend(cites)
            tracer.record(kind="code", decided_by="code", title=f"Run tool: {call.name}", detail=result_text[:200])
            messages.append(tool_result(call, result_text))

        if tokens_used >= max_tokens:
            reason = f"token budget reached: {tokens_used} >= {max_tokens}"
            final = force_final(context_policy(messages), model, tracer, reason=reason, max_tokens=400)
            return Answer.from_text(final.text, retrieved_sources=citations)

    final = force_final(context_policy(messages), model, tracer, reason=f"step cap reached: {max_steps} steps", max_tokens=400)
    return Answer.from_text(final.text, retrieved_sources=citations)

python -m examples.agent_harness --model stub:scripted

Failure modes

Cost and latency

Illustrative, not measured: no result file exists for this technique yet, so every number below is a worked example. See docs/EVALS.md for how a real run is recorded and scored.

How to Evaluate It

agent_harness answers a question about the documents and cites what it used, the same task single_agent is scored on, so scripts/eval_run.py scores it against the site’s own 60-question set: run python scripts/eval_run.py --example agent_harness --model <spec> --dry to project the cost of a full run before spending anything. What is specific to this technique is running the same 60 questions through two harness configurations and comparing the two result files directly: a gap in citation hit rate or score between a generous context policy and a tight one is not model variance, it is what the harness cost the run.

Run it

Try it

1. Use it

   Pick two agent products (a coding agent and a research or deep-research tool, say) and try to name, for each, its step or turn cap, whether it asks before an irreversible action, and what it says when it hits a limit. That is the harness, not the model, and most products document at least part of it.

2. Build it

   Run python -m examples.agent_harness --model stub:scripted from the repo root. The same model searches, looks up the part, and stops with a priced, warranty-scoped answer, all of it the harness's doing rather than the model's. Then open tests/test_example_agent_harness.py and change trim_to_budget(15) to a much larger number in the harness-changes-the-outcome test. Does the answer come back the same as the generous policy's?

3. Either lane

   Pick one of the failure modes above and try to cause it on purpose, using the pattern in tests/test_example_agent_harness.py.

4. Build it

   Read READ_ONLY_HEADERS and is_read_only in examples/common/bench.py, then SafetyEnvelope and Approval right after them. That is the same allowlist-plus-hook shape this page's own ToolRegistry and Hook build, in a setting where the one class of command that actually energizes a board needs a person's approval naming the set point before it runs at all.