Chat

This page’s level 1 example is a plain model call: one request and one response, with no search or tool execution around it. You supply a message and inspect the reply. A chat interface does not guarantee that architecture: a modern chat product may search, run tools, use memory, or coordinate several model calls behind one visible reply. The level describes how the task runs, not the appearance of its message box.

Because there is only one step, the whole outcome depends on what happens inside that one call: how the model was trained, and what you put in the request. Later levels add retrieval, tools and loops around the model to make up for what one call alone gets wrong. Before any of that, it helps to be precise about what “the model” even refers to, because a chat app, the company behind it, and the model actually answering you are three different things often sharing one name.

This page provides primary references and illustrative examples. The examples are scripted, not measured model runs; source references do not establish the correctness of every implementation or outcome.

"What voltage does a DR-210 need?"

def run(question: str, model: Model, embedder: Embedder | None, tracer: Tracer) -> Answer:
    del embedder  # level 1 has no retrieval step
    messages = [
        Message(role="system", content=SYSTEM_PROMPT),
        Message(role="user", content=question),
    ]
    tracer.record(kind="code", decided_by="code", title="Build prompt", detail=question)
    completion = model.complete(messages, max_tokens=400)
    tracer.record(
        kind="model",
        decided_by="code",
        title="Ask the model",
        detail=completion.text[:200],
        tokens_in=completion.tokens_in,
        tokens_out=completion.tokens_out,
        ms=completion.ms,
    )
    return Answer.from_text(completion.text)

python -m examples.one_call --model stub:scripted

Completion in the editor

The editor that finishes your line as you type is this level too, and for people who write code it is usually the model they touch most hours of the week. GitHub’s documentation says “GitHub Copilot offers coding suggestions as you type”[5]. Cursor describes its own the same way: “Tab is Cursor’s AI-powered autocomplete. It suggests code as you type, based on your recent edits, surrounding code, and linter errors”[6].

Nobody decides the next step, which is what keeps it on this rung. The editor’s code decides when to ask and what context to send, the model fills in the rest of the line, and you accept it with a keystroke or keep typing. That is one request and one answer. It is not level 5: a coding agent picks each step and decides when it is finished, and a completion picks nothing.

You do not need it when you already know exactly what the line says, since reading a suggestion costs more attention than typing eight characters, and at a bench a register write that looks right is worse than a blank line.

The failure mode follows from that asymmetry: accepting takes one key and checking takes a minute, so a plausible wrong line lands in the file unread. There is a second thing an unread line can carry: GitHub’s documentation says “GitHub Copilot checks each suggestion for matches with publicly available code”[7], and that a match is either discarded or offered with a code reference, depending on a policy setting your account or organization controls[7].

Failure modes

Cost and latency

Illustrative, not measured: no result file exists for this technique yet, so every number below is a worked example. See docs/EVALS.md for how a real run is recorded and scored.

How to Evaluate It

Chat is scored on the same 60-question set as every other level (docs/EVALS.md). With no documents attached, its lookup and numeric scores should sit close to level 0’s floor: whatever it gets right, it gets right from training data alone, which for a fictional appliance maker like Halvorsen should be close to nothing. The one place a single call can beat a keyword score is unanswerable questions, if the system prompt’s instruction to decline rather than guess actually holds: it can say “I don’t know” in its own words instead of returning an irrelevant passage.

No result file exists yet for any level (see docs/EVALS.md). Run python scripts/eval_run.py --example one_call --model stub --dry to project the token cost of a run before spending anything on a real one.

Run it

Try it

1. Use it

   Open a chat app and check which model answered your last message, usually in a menu or settings panel. Ask it its own knowledge cutoff date and compare that against the product's documentation.

2. Build it

   Run python -m examples.one_call --model stub:scripted from the repo root. With no documents and no tools, the reply declines to give the DR-210's supply voltage and says where the number is: the best answer this level has. Run it again with --model stub: the echo prints the prompt back, the same shape, nothing in it.

3. Either lane

   Pick a name from the "Out there" list at the bottom of this page: which of the four kinds is it, developer, model, product or tool?

4. Build it

   At a bench, paste a paragraph from an instrument programming manual into a chat app and ask for a summary: a safe use, since the text is right there to check. Then ask it, with nothing attached, for a specific accuracy figure from memory. A fluent answer is not a reported measurement, and never becomes one.