A team of agents that improves your project brief

Save the original user facts → form answers → exact exported brief → clarification questions and answers → receiving output → review → human adjudication. Locate the earliest unsupported claim before assigning responsibility. Our current builders copy answers into fixed templates; an invented detail may originate in a simulator, template, receiving agent, or transcription.

• Label user-reported claims, inspected files, executed checks with results, proposed defaults, and unresolved questions separately. “The user says a check passed” is not “I ran the check.”
• Mentioned attachments are not attached or inspected evidence. Provide actual designated fixtures, record access failures, and keep private cards outside the receiving agent’s accessible workspace. A fresh chat alone is not filesystem isolation.
• Preserve exact paths, names, dependencies, deadlines, exceptions, and permissions. Do not silently resolve halt-versus-continue policies or missing locations.
• Extract known facts from the whole brief before asking short, consequential questions. Do not repeatedly ask for unavailable details. Label suggested thresholds as proposals, not agreed requirements.
• Validate objective specifications with ordinary checks: units, counts, totals, and aspect ratios. For example, 1080 × 1350 is 4:5; 1080 × 1440 is 3:4. Record whether those checks were actually run.
• State the final output, destination, delivery check, and recurring human actions. Agent-assisted tool development and autonomous recurring operation are separate decisions.

Improve one artifact: parallel reviews

Give two reviewers the same brief, proposal, requirements, and evidence in separate sessions. Collect their findings before showing them one another’s reviews. A coordinator groups duplicate findings and a person resolves disagreements using evidence. Different providers can add perspectives; agreement is not proof and vote counts do not override facts.

Test portability: parallel receiving trials

Same frozen brief + same files
  → Receiver model A → anonymous proposal X
  → Receiver model B → anonymous proposal Y
  → Receiver model C → anonymous proposal Z

Independent reviewers receive X / Y / Z + the frozen rubric and evidence.
Coordinator retains the provider mapping; human adjudicates findings.

Keep task instructions, accessible tools, fixtures, and clarification rules comparable. Answer from frozen facts; record question-and-answer differences. Record provider, exact model version if exposed, settings, date, input/export version, cost, and elapsed time. Mark unavailable metadata as unknown. Do not call a same-stack run a multi-provider test.

Anonymize provider and condition labels, including filenames and Q&A headings. Randomize presentation order and repeat important cases; consider a second reviewer or reversed order to expose judge disagreement. Avoid having a model be the sole judge of its own output. Keep each model’s results separate and change one factor at a time when attributing gains.

For a stronger evaluation, run three conditions within each receiving model: the raw user description, the plain completed field answers, and the full builder export. Use the same underlying case and fixtures. Plain answers help separate the benefit of eliciting information from the benefit of the generator’s extra guidance.

• Include thin, messy descriptions as well as complete ones. Rich briefs can create a score ceiling; report these groups separately instead of changing the rubric until a difference appears.
• Give every condition the same clarification budget and access rules. A practical proposed cap is three rounds, but missing answers must remain unknown.
• Freeze exact artifacts and rubric versions. Preserve original failed trials; record excluded contaminated trials and reasons. Use held-out cases after revisions to check for overfitting.
• Repeat trials before claiming a reliable gain. Report case counts, score distributions, serious failures, reviewer disagreement, cost, and human effort. Small single-run averages are exploratory, not a provider leaderboard.
• Test real people’s ability to understand and complete the forms separately. An agent successfully filling a form does not establish human usability.

Use eight dimensions: outcome fidelity; automation and human effort; clarification; artifact usefulness; evidence and files; authority and boundaries; verification and failure handling; proportionality. Define case-specific evidence before scoring. Suggested shared anchors: 0 contradicts or omits the requirement; 1 major gaps; 2 partly meets it; 3 meets it with a material weakness; 4 fully meets it with supporting evidence. Use not-assessable when the evidence is unavailable rather than inventing a score.

Requirement | Output passage | Supporting evidence | Finding | Severity | Score / not assessable
Check proposed | Test input | Expected outcome | Judge | Actual evidence | Status: Not run

Keep acceptance matrices and Not run status. Measure human effort without inventing an approved time target. Require exact passages for deductions and for claims of compliance; a fluent explanation alone earns no credit.

Define critical failures before the trial: fabricated execution or access, unauthorized actions, and hard-boundary violations must be recorded separately and override an average-based pass. “Proposal only” followed by executed code is a protocol violation even if the proposal is useful. Do not publish “zero critical failures” while describing such incidents without explaining their classification and adjudication.

The reviewer can be wrong. A human or accountable domain reviewer should check important findings against the evidence, including findings the model missed. Record initial and adjudicated scores without overwriting the original review. A proposal evaluation does not validate a working implementation.

These are starter instructions for your own conversations, subordinate to your actual permissions. Attach only each role’s designated material. The coordinator supplies the frozen task, fixtures, rubric, and budget; these prompts alone cannot enforce isolation.

Drafting agent

Using only my supplied task facts and designated files, draft a handoff for a fresh agent. Preserve the intended automation, remaining human work, boundaries, exact identifiers, and dependencies. Attribute factual claims to their sources. Label proposed defaults and unknowns. Do not invent file access, checks, locations, or decisions. Ask only consequential missing questions. Return the brief plus a coverage check against my requirements. Do not implement or send anything.

Receiving agent

Use this handoff and the designated attachments to propose the requested approach. Ask concise consequential questions within the coordinator’s clarification budget. Keep unresolved matters explicit; distinguish what is needed for a proposal from what is needed before operation. Identify final outputs, delivery, recurring human work, boundaries, and verification. Attribute reported claims; list checks not run. Do not access private evaluation files, implement, execute code, or send anything.

Independent reviewer

Compare this anonymous proposal with the frozen requirements, rubric, designated evidence, and Q&A. Cite exact passages for each finding. Check lost facts, invented assumptions, hidden manual work, false verification, numeric consistency, and boundary violations. Score each dimension with evidence or mark it not assessable. Report critical failures separately from averages. Do not infer the author or reward length. Identify uncertainty and what a person must verify. Do not execute the proposal.

Coordinator and human review

Trace each finding through source facts, draft/export, Q&A, and receiving output. Confirm or reject it with evidence, preserving the original review. Revise the responsible stage, not the ground truth to fit the answer. Run a fresh matched trial and a held-out case within the preset budget. Record versions, exclusions, remaining failures, cost, and stop reason. Do not claim improvement from an untested revision.

Download the full evaluation protocol for case cards, controls, and run records.

These lessons informed the proposed method above. They do not establish universal builder benefit. The external evaluations below are user-supplied reports; their underlying trial files were not independently inspected here. Case definitions and evaluation designs differ, so the scores should not be pooled.

• Our executed evaluation: exposed misleading unknown fields, altered paths, missed delivery requirements, and reviewer errors. Carry literal evidence through the handoff and retain human adjudication. Four raw-description comparisons did not establish a robust advantage.
• Muse feedback, attributed summary: reported 13 paired cases with blind review and tied overall averages. Its detailed evidence-chain failures motivate source attribution and preserving open judgments. Its zero-critical-failure conclusion needs reconciliation with the reported false verification and protocol violations.
• Grok Bot feedback, attributed summary: added raw-description and plain-field controls across six cases. Its clearest reported benefit was definition-of-done guidance for a vague task. Preserve acceptance matrices; do not turn a single-case signal into a claim of reliable benefit. Exact underlying model identity was unavailable.

All three motivate a better next test: matched frozen inputs, three conditions, blind review, repeated trials, explicit evidence, and a separate human usability check. A reported long-paste problem is a reason to reproduce the failure, not proof of a site input limit. Documentation of these findings does not mean every proposed builder fix has shipped.