Parallel calls

Parallel calls run more than one model call at the same time instead of one after another, and combine the results in code. Anthropic describes two variations[1]. Sectioning breaks one task into independent subtasks that each run in parallel: one call screens a request for policy violations while another handles it, or several calls each evaluate a different aspect of the same output. Voting runs the same task several times and combines the outputs: several prompts review the same code for vulnerabilities, or several prompts judge the same content against different thresholds and the results are combined into one verdict.

Both stay at level 3 as long as your code decides how many calls to make, what each one gets, and how to combine what comes back, before any call goes out. The model fills in each call’s answer; it does not decide how many branches exist or how they are merged. The line runs through those two decisions: a system where the model reads the task and works out what the subtasks are, or reads the branches and writes the merged answer itself, is lead agent and workers at level 6, not this.

This page provides primary references and illustrative examples. The examples are scripted, not measured model runs; source references do not establish the correctness of every implementation or outcome.

"What is the DW-480's Normal cycle water use, and how
often should its filter be cleaned?"

LEVEL = 3
CANDIDATES_K = 3
NO_ANSWER = "NOT IN THIS SECTION"
PER_SECTION_SYSTEM = (
    "You are given exactly one source passage about Halvorsen appliances, and a question that "
    "may have more than one part. If this passage answers all or part of the question, answer "
    "briefly using only this passage. If it answers none of the question, reply with exactly "
    f"'{NO_ANSWER}' and nothing else."
)


def _answer_from_one_section(question: str, section: Section, model: Model) -> Completion:
    prompt = f"Passage [{section.cite}] {section.title}:\n{section.text}\n\nQuestion: {question}"
    return model.complete([Message(role="system", content=PER_SECTION_SYSTEM), Message(role="user", content=prompt)], max_tokens=200)


def run(
    question: str,
    model: Model,
    embedder: Embedder | None,
    tracer: Tracer,
    *,
    corpus_dir: Path = DEFAULT_CORPUS_DIR,
    k: int = CANDIDATES_K,
) -> Answer:
    del embedder  # candidates come from keyword search, not a vector index
    sections = load_sections(corpus_dir)
    candidates = [s for s, score in bm25_search(sections, question, k=k) if score > 0]
    tracer.record(
        kind="code", decided_by="code", title="Pick sections to answer in parallel",
        detail=", ".join(s.cite for s in candidates) or "none",
    )

    # .map submits every call to the pool at once and yields results back in candidate order,
    # so the calls run concurrently but the code below never has to sort them: determinism comes
    # from retrieval order, not from whichever call happens to finish first.
    with ThreadPoolExecutor(max_workers=max(1, len(candidates))) as pool:
        completions = list(pool.map(lambda s: _answer_from_one_section(question, s, model), candidates))

    for section, completion in zip(candidates, completions):
        tracer.record(
            kind="model", decided_by="code", title=f"Answer from {section.cite} alone", detail=completion.text[:200],
            tokens_in=completion.tokens_in, tokens_out=completion.tokens_out, ms=completion.ms,
        )

    used = [(s, c) for s, c in zip(candidates, completions) if NO_ANSWER not in c.text.upper()]
    tracer.record(
        kind="code", decided_by="code", title="Combine the sections that answered",
        detail=f"{len(used)} of {len(candidates)} sections answered part of the question",
    )
    if not used:
        return Answer(text="None of the retrieved sections answered the question.", citations=[], retrieved_sources=[s.cite for s in candidates])
    combined = " ".join(c.text.strip() for _, c in used)
    return Answer(text=combined, citations=sorted({s.cite for s, _ in used}), retrieved_sources=[s.cite for s in candidates])

python -m examples.parallelization --model stub:scripted

Failure modes

Cost and latency

Illustrative, not measured: no result file exists for this technique yet, so every number below is a worked example. See docs/EVALS.md for how a real run is recorded and scored.

How to Evaluate It

The same 60 questions, the same corpus, plus one number of its own: what share of a question’s must_cite sections were actually covered by some section’s answer. Multi-hop questions test that directly, since they need more than one section’s fact combined into a single answer.

Conflicting-source questions are the interesting case to watch: sectioning retrieves both sides of a deliberate contradiction as readily as RAG does, but its fixed combine step has no way to compare them, only to concatenate whatever each section said. Whether that scores better or worse than RAG’s single stuffed-context prompt is exactly the kind of question this site can only answer once a result file exists (see docs/EVALS.md); none does yet. Run python scripts/eval_run.py --example parallelization --model <spec> --dry to project the cost of a real run first.

Run it

Try it

1. Use it

   Ask a coding agent that advertises parallel subagents to work on three independent parts. Does it run them at once, and does the result repeat itself?

2. Build it

   Run python -m examples.parallelization --model stub:scripted from the repo root. Three branches answer from one section each, one says NOT IN THIS SECTION, and the combine keeps the two that did. Now change CANDIDATES_K from 3 to 5 in examples/parallelization/run.py: the run stops with ScriptExhausted, naming the passage no reply matches.

3. Either lane

   Cause the overlap failure on purpose: find a question evals/corpus/ answers in two places, run it, and count how often the combined answer repeats itself.