AI gateways

An AI gateway is one entry point that every call to a model provider goes through instead of calling each provider directly. It centralizes provider keys, routing and fallback between providers, rate limits and budgets per caller, caching, a log of every call, and policy checks on the way out and back. LiteLLM’s own documentation describes its proxy mode as a “Self-hosted LLM Gateway (Proxy) with virtual keys, cost tracking, and an admin UI,” offering “Virtual keys with per-key/team/user budgets” and “Centralized logging, guardrails, and caching”[2].

A gateway is not an agent framework: it decides nothing about what a model should do next and holds no task state. It overlaps with an API aggregator without being the same thing: LiteLLM’s own docs describe a library that “gives you a single, unified interface to call 100+ LLMs” and a separately named “Gateway (Proxy)” run on top of it, adding keys, budgets and policy[2]. Two costs come with the entry point either way: one component everything now depends on, and one more party that sees your traffic. Neither is a reason not to use one; both are decisions to make on purpose.

This topic is not a level on the ladder; it applies at every level, the way ops does. This page is sourced, not measured: what each gateway does comes from its own documentation, and no gateway here has been put in front of real traffic and scored, so no number below is one this site took.

class Gateway:
    routes: dict[str, Route]
    budgets: dict[str, int]  # key -> tokens remaining
    redact: bool = True
    log: list[LogEntry] = field(default_factory=list)

    def complete(self, key: str, messages: list[Message], *, tracer: Tracer | None = None, **kwargs) -> Completion:
        """Route one call for `key`, falling back to the secondary provider on any error.

        Raises `BudgetExceeded` before calling anything if the key is spent. If the fallback
        fails too, that provider's exception propagates unchanged; see the module docstring.

        `tracer` is optional and unused by the demo CLI: when given, it records the refusal, the
        fallback (if one happened) and the call that answered as `code` steps -- routing and
        budget enforcement are fixed rules the gateway applies, never a choice a model made, so
        nothing here is ever `decided_by: "model"`.
        """
        if self.budgets.get(key, 0) <= 0:
            if tracer is not None:
                tracer.record(kind="code", decided_by="code", title=f"Refuse: {key} has no budget left", detail=key)
            raise BudgetExceeded(f"key {key!r} has no budget left")
        route = self.routes[key]
        fell_back = False
        try:
            completion = route.primary.complete(messages, **kwargs)
        except Exception:  # noqa: BLE001 - any provider failure triggers fallback, on purpose
            if tracer is not None:
                tracer.record(kind="code", decided_by="code", title=f"Fall back: {key}'s primary failed", detail=key)
            completion = route.fallback.complete(messages, **kwargs)
            fell_back = True
        self.budgets[key] -= completion.tokens_in + completion.tokens_out
        prompt = None if self.redact else "\n".join(content_text(m.content) for m in messages)
        self.log.append(
            LogEntry(
                key=key,
                model_id=completion.model_id,
                tokens_in=completion.tokens_in,
                tokens_out=completion.tokens_out,
                fell_back=fell_back,
                prompt=prompt,
            )
        )
        if tracer is not None:
            tracer.record(
                kind="model",
                decided_by="code",
                title=f"{key}: call answered" + (" via fallback" if fell_back else ""),
                detail=completion.text[:200],
                tokens_in=completion.tokens_in,
                tokens_out=completion.tokens_out,
                ms=completion.ms,
            )
        return completion

python -m examples.ai_gateways --demo

Failure modes

At each level

• Direct prompting: a gateway is one extra hop in front of the single call chat makes; whether it earns its place here depends entirely on whether more than one provider or key is already in the picture.
• Workflows: your own code already decided which step calls which model, so the gateway’s routing is redundant with that decision: what it adds here is one budget and one log across every step, not new fallback logic your code didn’t already have.
• Tool use: a tool definition passes through the gateway unchanged in either direction, so nothing about function calling is this layer’s concern: only the token accounting around the call is.
• Agent loops: the number of calls a run makes is not fixed in advance, so a gateway’s per-key budget is a real backstop here, not just bookkeeping. This is the same ceiling a single agent’s own max_tokens cap is meant to provide, enforced one layer further out in case the cap inside the loop fails to hold.
• Teams of Agents: several agents can share one gateway key or each hold their own, which is exactly the choice that decides whether a lead and its workers show up as one line in a cost report or several.
• Always-on agents: traffic is continuous rather than one burst per question, so a per-key budget stops being an occasional backstop and becomes the thing that decides how much a runaway loop can spend before anyone is awake to notice.

Practices

• Check a key’s budget before calling a provider, not after, and reserve against it rather than subtracting afterwards, so two simultaneous calls cannot both spend the last of it.
• Turn fallback on per route, not globally. A request that can be repeated safely is a different thing from one that charges, files or sends, and only the first is a fallback candidate.
• Key a cache on more than the prompt text: who asked matters as much as what was asked, or one caller’s cached answer can reach another’s request.
• Log every fallback as its own event, not folded into a plain success count, so a primary provider quietly failing every request is visible before someone asks why costs changed.
• Redact prompt and answer content in logs by default, and then check the paths the flag does not cover: a provider’s raw error string is the usual one, and observability lists the rest.
• Decide up front whether the gateway itself is allowed to be a single point of failure for your system, and if not, plan for what happens when it, not a provider behind it, is the thing that is down.

Run it

Try it

1. Use it

   Find a product you use that calls more than one AI provider (check its status page or documentation for the providers it names). Look for signs of a gateway in front of them: one combined status indicator, or a single outage that took down access to every provider at once.

2. Build it

   Run python -m examples.ai_gateways --demo from the repo root and read the printed log. Then change team-a's budget in _demo_gateway (examples/ai_gateways/__main__.py) from 10 to 10000 and run it again: does the second call for team-a still get refused?

3. Either lane

   Pick one of the failure modes above and write down, for a real system you use or built, which of its two mitigations (the check itself, or the log that would reveal the check failed) you actually have today.