Workflows

A wrong number introduced early in a fixed chain does not simply persist. It becomes a wrong calculation, then a sentence of prose, then a conclusion, and it gets harder to spot at every handoff, because each stage makes it look more like ordinary output.

What people are trying

Putting a check between stages rather than only at the end. One measurement through a four-stage pipeline found an early gate catches much more than the same check run once on the final result.

• The Hallucination Snowball: Modeling Error Propagation as State Transitions in Multi-Agent LLM Pipelines · arXiv · read 09/19/2026

  Sequential multi-agent LLM pipelines chain specialized agents without verification at handoffs, creating a structural flaw with measurable and severe consequences.

  The paper says multi-agent. The pipeline it measures is a fixed sequence of stages where no stage chooses what happens next, which is what this site calls prompt chaining.

Where it bites: Prompt chaining

A router sorts a request into a branch before anything else runs, and routers still send plenty of requests somewhere other than where they would have been answered best. Under one benchmark that compared many of them on the same footing, several recent methods, commercial ones included, did not clearly beat a simple baseline.

What people are trying

Shared testbeds spanning many models and many routing methods, so routers are compared on the same task set instead of each paper's own. Some work scores cost and latency alongside accuracy rather than accuracy alone.

• LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing · arXiv · read 09/19/2026

  a substantial gap remains to the Oracle, driven primarily by persistent model-recall failures

Where it bites: Routing

The check half of a write-and-check loop is usually another model call, and a model grading text written in its own style can favor it for reasons that have nothing to do with quality. Picking a more capable judge does not reliably fix it.

What people are trying

Measuring the bias directly, by grading pairs of answers built to differ in style and not in quality, so any preference shown is the bias. Structured multi-part rubrics are being tested as a way to cut it down.

• Quantifying and Mitigating Self-Preference Bias of LLM Judges · arXiv · read 09/19/2026

  Empirical analysis across 20 mainstream LLMs reveals that advanced capabilities are often uncorrelated, or even negatively correlated, with low SPB.

Where it bites: Write and check

Putting a person in the loop is the easy part. How much they should be asked to decide, and how to keep them from waving through whatever the system already proposed, is not settled, and NIST says so in its own guidance rather than prescribing a split.

What people are trying

NIST's playbook asks teams to design the human role deliberately, account for the biases that make a reviewer agree too readily, and measure whether oversight is happening at all. Override rates and time spent per review are what tell you, rather than the presence of a checkpoint.

• Map (AI RMF Playbook, AIRC) · NIST, Trustworthy and Responsible AI Resource Center · read 09/19/2026

  Questions remain about how to configure humans and automation for managing AI risks.

Where it bites: Human approval