Agentic RAG and deep research

Agentic RAG puts the search loop itself under the model’s control. RAG always searches once and asks the model once. Agentic RAG instead offers the model a search tool it can call as many times as it decides it needs, lets it read what comes back, and lets it decide whether to search again, read more closely, or stop and answer: the single-agent loop aimed at retrieval.

Google describes its own version this way: at each step, “the model has to ground itself on all information gathered so far, then identify missing information and discrepancies it wants to explore”, continuing until “the model determines enough information has been gathered”[1]. OpenAI’s deep research models work the same way: “agentic” systems that “conduct multi-step research” and return a listing of every search made along the way[3]. In both cases the model chooses the next query and when to stop; your code still runs every search and can cut the loop off with a hard cap regardless of what the model would have done next.

This page is measured: the cost and the score under How to Evaluate It come from a recorded run of this example on a real model, beside the RAG page’s run on the same model and questions, and hold for that model’s class. The step-through just below is still a scripted illustration, and source references do not establish the correctness of every implementation or outcome.

"How long is the warranty on the DW-480,
and what voids it?"

def run(
    question: str,
    model: Model,
    embedder: Embedder | None,
    tracer: Tracer,
    *,
    corpus_dir: Path = DEFAULT_CORPUS_DIR,
    max_steps: int = MAX_STEPS,
    max_tokens: int = MAX_TOKENS,
) -> Answer:
    del embedder  # level 5 retrieves through its tools, not a vector index
    sections = load_sections(corpus_dir)
    messages = [Message(role="system", content=SYSTEM_PROMPT), Message(role="user", content=question)]
    tracer.record(kind="code", decided_by="code", title="Build prompt with tool definitions", detail="search, read")

    citations: list[str] = []
    tokens_used = 0
    for _ in range(max_steps):
        completion = model.complete(messages, tools=TOOLS, max_tokens=400)
        tokens_used += completion.tokens_in + completion.tokens_out

        if not completion.tool_calls:
            tracer.record(
                kind="model",
                decided_by="model",
                title="Model stops and answers",
                detail=completion.text[:200],
                tokens_in=completion.tokens_in,
                tokens_out=completion.tokens_out,
                ms=completion.ms,
            )
            return Answer.from_text(completion.text, retrieved_sources=citations)

        calls_desc = ", ".join(f"{c.name}({json.dumps(c.arguments, sort_keys=True)})" for c in completion.tool_calls)
        tracer.record(
            kind="model",
            decided_by="model",
            title="Model calls tool(s)",
            detail=calls_desc,
            tokens_in=completion.tokens_in,
            tokens_out=completion.tokens_out,
            ms=completion.ms,
        )
        turn, calls = assistant_turn(completion, len(messages))
        messages.append(turn)
        for call in calls:
            result_text, cites = _run_tool(call, sections)
            citations.extend(cites)
            tracer.record(kind="code", decided_by="code", title=f"Run tool: {call.name}", detail=result_text[:200])
            messages.append(tool_result(call, result_text))

        if tokens_used >= max_tokens:
            final = force_final(messages, model, tracer, reason=f"token budget reached: {tokens_used} >= {max_tokens}", max_tokens=400)
            return Answer.from_text(final.text, retrieved_sources=citations)

    final = force_final(messages, model, tracer, reason=f"step cap reached: {max_steps} steps", max_tokens=400)
    return Answer.from_text(final.text, retrieved_sources=citations)

python -m examples.agentic_rag --model stub:scripted

Failure modes

Cost and latency

Measured: averages over the 60-question run on Muse Glimmer 30B, a model in the Large local (about 30B) class, on one local GPU. Tokens out include the model's hidden reasoning, which it spends before answering. Holds for this model class only.

Most of the extra input is the loop itself: every step sends the whole conversation again, searches and read sections included, so the prompt grows with each round. Anthropic separately reports that in its data agents typically use about 4x more tokens than chat interactions, and multi-agent systems about 15x more than chats; those are Anthropic’s figures, not ones measured here.

How to Evaluate It

agentic_rag is registered and scored on the same 60-question set as every other technique here: exact or rubric match, citation hit rate, and the count of model-decided steps the trace carries. Multi-hop and conflicting-source questions are where the extra cost is supposed to earn its keep: a multi-hop question needs two sections found and used together, which single-pass RAG often cannot do in one search, and a conflicting-source question needs the loop to notice two retrieved sections disagree rather than stopping after the first one that looks like an answer. A lookup question that RAG already answers in one call is the wrong place to look for agentic RAG’s advantage; if the extra cost does not show up as a better score on multi-hop and conflicting questions specifically, it is not paying for itself.

On this model the extra cost paid for itself where this page said it should: most of the multi-hop questions single-pass RAG missed, the loop answered, because after reading one section it searched again for the next. The other kinds moved less, since RAG already did well on them.

Most loops did not stop on their own. The token budget in run ended most questions and forced an answer from whatever had been read so far, and three of the answers this run got wrong say so outright: the section they needed had turned up in a search but had not been read yet. Whether a larger budget would answer more is not tested here. The budget is the setting that trades cost for completeness, and a real deployment has to choose it.

To run it yourself, python scripts/eval_run.py --example agentic_rag --model <spec> --dry projects the cost first; docs/FIRST-LIVE-RUN.md is the full sequence and the checks to read before the score.

Run it

Try it

1. Use it

   Give a deep-research mode a question with two parts that live in different sources, and check its shown searches or sources: did it actually run more than one search, and does each part of the answer trace to one it ran?

2. Build it

   Run python -m examples.agentic_rag --model stub:scripted from the repo root. The model searches, reads the one section its own search turned up, and stops: three turns, one citation. Run it again with --model stub, where the echo is never a tool call, and the loop ends on the first turn having retrieved nothing.

3. Either lane

   Compare this page's run to RAG's: RAG's five steps are all solid (code-decided); count how many of this run's eight are dashed instead. What does the difference buy, and what does it cost?