Observability

Observability is recording what each run did, in enough detail that a bad result can be traced back to the step that caused it: which passages a retrieval step picked, which tool a model called and with what arguments, which branch a workflow took, and how many tokens and how much time each step spent. OpenTelemetry, an open standard for this, describes a trace this way: “The path of a request through your application.” It defines a span, one of “the building blocks” of a trace, as a unit of work carrying attributes, its own “key-value pairs” of metadata about the operation it tracked[1].

This topic is not a level on the ladder; it applies at every level, the way safety and ops do. What is worth recording changes by level, which is why this page has its own “At each level” section rather than pointing only at ops’s.

This page is sourced, not measured: what each tool records comes from its own documentation, and no production trace exists for this site, so what follows shows the mechanism and the site’s own small version of it.

class Step:
    i: int
    kind: StepKind
    decided_by: DecidedBy
    title: str
    detail: str
    tokens_in: int
    tokens_out: int
    ms: float
    edge: str  # "solid" | "dashed"

def to_otel_spans(trace: dict[str, Any], *, capture_content: bool = False) -> list[dict[str, Any]]:
    """One span-shaped dict per step in `trace`, in the shape `Tracer.write` produces.

    `capture_content=False` (the default) never lets a step's `detail` leave this function.
    The span's `name` is the step's `title`, copied through either way; see the module docstring.
    """
    spans: list[dict[str, Any]] = []
    for step in trace["steps"]:
        attributes: dict[str, Any] = {SITE_DECIDED_BY: step["decided_by"]}
        if step["kind"] == "model":
            attributes[GEN_AI_OPERATION_NAME] = "chat"
            attributes[GEN_AI_INPUT_TOKENS] = step["tokens_in"]
            attributes[GEN_AI_OUTPUT_TOKENS] = step["tokens_out"]
        if capture_content:
            attributes[SITE_DETAIL] = step["detail"]
        spans.append({"name": step["title"], "duration_ms": step["ms"], "attributes": attributes})
    return spans

python -m examples.observability --demo

Failure modes

At each level

• Conventional software: there is nothing to trace that a normal application log does not already cover: this topic’s own concerns start at level 1.
• Direct prompting: one span, the way chat’s own trace is a single model step; the whole question is whether that one call’s tokens, time and outcome are recorded at all.
• Added context: what got retrieved is now part of the trace, not just the model call: RAG’s own citations are exactly the record a reviewer needs to check an answer against its sources.
• Workflows: a fixed number of steps means a trace can be compared against the pipeline’s own diagram directly: a step that is missing or repeated is visible without reading a single token of content.
• Tool use: a tool call’s arguments and result belong in the trace as their own step, not folded into the model step around them, since a wrong argument and a wrong model answer are different failures that need different fixes.
• Agent loops: the number of steps is no longer fixed, so a trace is the only way to know how many turns a run actually took and where it stopped, the same uncertainty a single agent’s own max_steps is built to cap.
• Teams of Agents: several agents produce interleaved traces, so which agent decided what has to survive being merged into one timeline, the way agent graphs’ own handoffs need to be attributable to a specific agent after the fact.
• Always-on agents: nobody is watching a run as it happens, so the trace is the entire record a person has after the fact: sampling policy matters most here, since a dropped trace from a system like an always-on assistant cannot be reconstructed by asking the model again.

Practices

• Record tokens in, tokens out and wall time per step, not only per request, so an average latency number cannot hide one consistently slow step.
• Default to not capturing prompt or answer content, and make capturing it an explicit, separate choice: the requirement level OpenTelemetry marks its own message-content attributes with.
• Keep content out of the fields nobody thinks of as content: a span’s name, a step’s title, an error string that echoes what was sent. Redaction that covers one field and not those is a policy with a hole in it.
• Give every trace and every scored result a shared id to join on (this site uses the commit the code was at), so a bad result can be traced back to a run without guessing which one it was.
• Bias sampling toward keeping the traces most worth reading (slow, expensive or failed runs) rather than a uniform random sample that treats an incident the same as an ordinary request.
• Read the built trace against the pipeline’s own diagram after any change to the code that produces it, the way this site’s own tests check a trace’s decided_by pattern against the level it claims to be.

Run it

Try it

1. Use it

   Find a product you use that shows its steps while it works (a "thinking" panel, a sources list, a tool-call log). Next time an answer looks wrong, use that panel to find which step went off track before rereading the final text.

2. Build it

   Run python -m examples.observability --demo from the repo root and compare the redacted and capture_content=True output. Then add a third step to _demo_trace in examples/observability/__main__.py and confirm it shows up correctly in both.

3. Either lane

   Pick a system you use or built that has no visible trace at all. Write down the one step you would most want a record of if it produced a wrong result tomorrow, and why that one.