Code execution

Code execution lets the model write a small program instead of choosing among named tools, and a sandbox (not the model) runs it. The shape is the same as function calling: the model’s output picks what happens next, and your code always carries it out. Anthropic describes its own version this way: the tool “allows Claude to run Bash commands and manipulate files, including writing code, in a secure, sandboxed environment”[1]. What the model writes is data your code hands to an interpreter, never text your code trusts and runs directly.

Code execution sits at level 4, tools. The model’s one real choice is what code to write; the decided_by: "model" step below is picking that content, not deciding whether to run it: your code always runs whatever it wrote, inside a fixed sandbox, and always asks once more for an answer once the result is back. That fixed shape, one write-and-run cycle bounded from outside, is the line to level 5: a coding agent keeps writing and running code in a loop it exits on its own, reading each result before deciding what to write next.

This page provides primary references and illustrative examples. The examples are scripted, not measured model runs; source references do not establish the correctness of every implementation or outcome.

"What is the total price to replace the heating elements
on both a DW-300 and a DW-480?"

SYSTEM_PROMPT = (
    "You answer numeric questions about Halvorsen appliances using only the numbered sources "
    "below. Reply with exactly one line: a Python arithmetic expression using only numbers, "
    "+ - * / and parentheses, that computes the answer -- no words, no units, no code fences. "
    f"If the sources do not contain the numbers you would need, reply with the single word "
    f"{DECLINE} instead."
)

def _eval_node(node: ast.AST, depth: int = 0) -> float:
    if depth > MAX_DEPTH:
        raise UnsafeExpression(f"expression nests deeper than {MAX_DEPTH} levels")
    if isinstance(node, ast.Constant) and isinstance(node.value, (int, float)) and not isinstance(node.value, bool):
        return node.value
    if isinstance(node, ast.BinOp) and type(node.op) in _BINOPS:
        return _BINOPS[type(node.op)](_eval_node(node.left, depth + 1), _eval_node(node.right, depth + 1))
    if isinstance(node, ast.UnaryOp) and type(node.op) in _UNARYOPS:
        return _UNARYOPS[type(node.op)](_eval_node(node.operand, depth + 1))
    raise UnsafeExpression(f"{type(node).__name__} is not on the arithmetic whitelist")

    if len(expr) > MAX_EXPRESSION_CHARS:
        raise UnsafeExpression(f"expression is {len(expr)} characters; the limit is {MAX_EXPRESSION_CHARS}")
    try:
        tree = ast.parse(expr, mode="eval")
    except (SyntaxError, ValueError) as exc:
        raise UnsafeExpression(f"not a valid expression: {exc}") from exc
    value = _eval_node(tree.body)
    if isinstance(value, float) and not math.isfinite(value):
        raise UnsafeExpression(f"result is not a finite number: {value}")
    return value

    first = model.complete(messages, max_tokens=60)
    expr = first.text.strip()

    if not expr or expr.upper() == DECLINE:
        tracer.record(
            kind="model",
            decided_by="model",
            title="Model declines: not enough numbers in the passages",
            detail=expr or "(empty)",
            tokens_in=first.tokens_in,
            tokens_out=first.tokens_out,
            ms=first.ms,
        )
        return Answer(text="The documents don't give enough numbers to compute that.", citations=[])

    tracer.record(
        kind="model",
        decided_by="model",
        title="Model writes an expression",
        detail=expr,
        tokens_in=first.tokens_in,
        tokens_out=first.tokens_out,
        ms=first.ms,
    )

    try:
        value = safe_eval(expr)
    except (UnsafeExpression, ArithmeticError) as exc:
        tracer.record(kind="code", decided_by="code", title="Sandbox refused the expression", detail=str(exc))
        return Answer(text=f"Could not safely evaluate that expression: {exc}", citations=[])

    tracer.record(kind="code", decided_by="code", title="Sandbox evaluates the expression", detail=f"{expr} = {value}")

python -m examples.code_execution --model stub:scripted

Failure modes

Cost and latency

Illustrative, not measured: no result file exists for this technique yet, so every number below is a worked example. See docs/EVALS.md for how a real run is recorded and scored.

How to Evaluate It

Code execution is registered with the site’s runner as not scored against the shared 60-question set, with the reason (see docs/EVALS.md). Asking scripts/eval_run.py for it by name prints that reason and stops, rather than producing a number about a task the technique was never built to do. What it would be scored on instead: correctness on the 12 numeric questions in the shared set, the only kind this technique answers by design, plus two numbers the shared harness does not otherwise track: how often a deliberately malicious or malformed expression is refused rather than evaluated, and how cleanly the other four question kinds (lookup, multi-hop, unanswerable, conflicting sources) are declined instead of forced through an arithmetic answer that was never going to fit them.

Run it

Try it

1. Use it

   Ask a chat app's data-analysis feature a question that needs a calculation across numbers you give it, then ask it to show you the code it ran. Does the code actually compute what the final answer claims?

2. Build it

   Run python -m examples.code_execution --model stub:scripted from the repo root. Retrieval finds the two prices, the model writes 38.50 + 41.00, the sandbox evaluates it, and the answer is $79.50 with its citation. Run it again with --model stub: the echoed question goes to the sandbox instead of an expression and is refused as invalid syntax, which is the point, since nothing the model returns is trusted to be arithmetic. Then open a Python shell in the repo root and call examples.code_execution.run.safe_eval on a few strings of your own: "38.50 + 41.00", "2 ** 10", "__import__('os').listdir('.')", and "1" plus "+1" forty times. Read which bound each one hits.

3. Either lane

   Pick one of the failure modes above and try to cause it on purpose, using only the synthetic documents in evals/corpus/.

4. Build it

   Read examples/bench_test_data_by_conversation/README.md: on this site's electronics-test bench, a model writes a short snippet against a retest export whose volts column is actually millivolts under a header that says volts, and a sandbox runs it. Before that snippet or the model ever sees the table, code range-checks the column against the widest node the board has anywhere and corrects the mislabeled unit. Why does that check have to happen in code, on every table, whether or not the model's tool gets called at all, rather than being one more thing the written snippet is trusted to do?