Prompt engineering

Prompt engineering is writing the request itself well: instructions, examples, an assigned role, a required format, and sometimes an explicit ask to reason before answering. It does not change what level a technique sits at (this is still level 1, one call, decided entirely by code). It changes what happens inside that one call, by giving the model more to work with than the bare question alone.

OpenAI, Anthropic and Google each publish their own guidance for this, and the moves they have in common (structure, examples, a stated format) carry across models even though the details of how much each one helps do not[1][2][3]. None of it is worth doing once and trusting forever: write down what a correct answer looks like before changing a prompt, then check the new version against the same cases the old one had to pass. This page’s example makes that concrete. It uses the same question and the same source text, asked two ways, checked by a regular expression rather than by reading the reply and deciding it looks better.

This page is sourced, not measured: every instruction below is checked against a maker’s own prompting guide, and no wording here has been scored against another. It is illustrated.

"What is the DW-480's drain pump part number and price?"
passage: "...HLV-2205, available through authorized
service...priced at $52.00..."

def run(question: str, model: Model, tracer: Tracer, *, structured: bool = True) -> Answer:
    if structured:
        messages = [
            Message(role="system", content=STRUCTURED_SYSTEM),
            Message(role="user", content=f"{PASSAGE}\n\nQuestion: {question}"),
        ]
        tracer.record(
            kind="code",
            decided_by="code",
            title="Build the structured prompt",
            detail="role + output format + one worked example",
        )
    else:
        messages = [Message(role="user", content=f"{PASSAGE}\n\nQuestion: {question}")]
        tracer.record(kind="code", decided_by="code", title="Build the bare prompt", detail=question)
    completion = model.complete(messages, max_tokens=200)
    tracer.record(
        kind="model",
        decided_by="code",
        title="Ask the model",
        detail=completion.text[:200],
        tokens_in=completion.tokens_in,
        tokens_out=completion.tokens_out,
        ms=completion.ms,
    )
    match = FIELD_RE.search(completion.text)
    citations = ["dw480-manual#8", "parts-list#2"] if match else []
    tracer.record(
        kind="code",
        decided_by="code",
        title="Check the reply against the expected format",
        detail="matched PART/PRICE" if match else "did not match the expected format",
    )
    return Answer(text=completion.text, citations=citations)

python -m examples.prompt_engineering --model stub:scripted --structured
python -m examples.prompt_engineering --model stub:scripted --no-structured

Failure modes

Cost and latency

Illustrative, not measured: no result file exists for this technique yet, so every number below is a worked example. See docs/EVALS.md for how a real run is recorded and scored.

How to Evaluate It

Prompt engineering is not its own row in the site’s eval; it is a way of improving the score at whichever level you are already using, by holding the retrieval, the tools and the level fixed and comparing prompt versions against the same 60-question set (docs/EVALS.md). The right test is A/B, not before/after: run the old prompt and the new prompt over the same questions and compare scores, since a single “it reads better now” impression on a handful of examples is exactly the failure the iterating-against-test-cases move above exists to catch.

scripts/eval_run.py will not score this example against that set: it sends one fixed passage to the model two ways and never reads the documents the questions are about, so asking the runner for a score prints that reason and stops (docs/EVALS.md). What to measure for a prompt change is the pair, old prompt against new, on the same inputs: format adherence, the share of replies that come back in the shape you asked for, and accuracy of what is in them.

Run it

Try it

1. Use it

   Take a chat app request you make often and add one instruction, one example of the output you want, and a specific format. Compare the reply against what the plain version gave you.

2. Build it

   Run both commands from examples/prompt_engineering/README.md with --model stub, then edit STRUCTURED_SYSTEM in run.py to remove the worked example and see whether the test file still passes.

3. Either lane

   Before you try it, write down exactly what a correct answer to your own question would contain. That written-down version is the test case the iterating-against-test-cases move above depends on.