Essays/Agents

The twelve-reviewer board

Twelve read-only reviewers in six roles on two model tiers argued against themselves and then against each other, and settled one architecture question in writing.

I settled one architecture question by sending twelve read-only reviewer agents in place of a design meeting. Each had to make a recommendation, cite files and lines, write the strongest argument against its own position, and then answer the other eleven. What came out was a written boundary between two systems and a record of why it held.

It is one case of a pattern I use at every size, described in Actor and verifier loops: something produces and something else checks. Here the thing being checked was a design decision.

The loop system runs scheduled and recurring work across my fleet: cron loops, event-driven triggers, guarded adapters for headless coding agents. A separate line of work, one package of automation specs and one of action specs, had started to specify workflow work of its own. The question was whether deterministic automations should be a separate product with its own control plane, or fold into the loop system.

Wrong one way, I duplicate a scheduler nobody asked for. Wrong the other way, I push a second product’s concerns into a package that has to stay boring and reliable. Decisions like this used to take a meeting, and in a meeting the result tends to land wherever the most confident voice pushes it.

Six roles on two model tiers

The twelve were six roles, each with its own question.

  • Runtime asked whether the design behaves correctly once it schedules and executes real work.
  • Security asked what the merged design exposes that the separated one would not.
  • Reliability asked what fails and how it recovers.
  • Migration asked what breaks for the automations and workflows that already exist.
  • Product asked whether either shape serves how the thing gets used.
  • Contracts asked where the API and schema boundary must sit so packages stop reaching into each other’s databases.

Each role ran on two model tiers. One model has correlated blind spots: the same gap shows up in its security review and its reliability review, because the same weights do the reasoning. Pairing tiers keeps the board from being one model’s judgment under six names.

The number has limits on both sides. A standing rule caps how many sub-agents may run in parallel, and I have moved that cap by hand more than once, as Hard limits are governance records. Twelve sits above the everyday default and below the point where a rebuttal round turns into a transcript nobody can read.

Why is the second round the one that matters?

All twelve got the same prompt: a read-only adversarial architecture review. Do not edit files, do not write files, do not start more agents. The report had to give a recommendation, cite exact files and lines wherever possible, and make the strongest argument against that recommendation. It had to end with DONE and the recommendation.

Both constraints are deliberate. Reviewers that can edit stop reviewing and start fixing, and then several agents fight over the same diff. An agent asked whether work is good says yes, and that answer is not evidence, as I argue in Agent self-reports are not evidence. An agent told to make the strongest argument against gives me the failure modes first.

Round one put twelve independent reports in a shared channel. In round two every reviewer read the whole channel and posted again: a rebuttal to a point that contradicted it, or an explicit agreement. Without that round I would have had twelve opinions that happen to cite line numbers. With it, silence is not an answer. A claim nobody rebuts stands in writing, with a name on it.

The mechanism is ordinary. The channel lives in the messaging tool the agents already share, with flat channels, message ids, and read and unread state. Setting up the board meant opening one channel and sending twelve prompts that pointed at it.

What the board decided

The result was a boundary, recorded in the design document for the automation runtime. The loop system may execute workflow work that other automation systems have already materialized. It does not become the automation product. The spec packages own the specs. Once work is explicitly handed off, the loop system owns invocation, admission, execution, run manifests and provider routing, and nothing external writes directly into its database. A compiler outside it goes through an idempotent upsert with dry-run, preflight and commit modes, keyed by idempotency keys and spec hashes.

That is a division of responsibility someone can check. I can also hand it on with the reasons attached: who argued against it, where they looked, and who came back, disagreed and lost.

The size follows the cost of being wrong

Twelve agents reading a codebase and writing reports backed by evidence is not free, and the rebuttal round doubles the work. I do not run it for a color change or a copy tweak. Ordinary work gets two adversarial agents before anything is committed and pushed. Merging a stack of pull requests gets four, which read every change and merge one at a time while everything stays green. One feature spec, for a tool call that lets a model emit a whole canvas layout at once, went through three adversarial reviews before any implementation existed. Twelve is for a decision that is expensive to reverse, where a wrong call means a migration six months later.

The habit came from a stranger’s reply on Twitter about adversarial code review. I have used it ever since.

My own plans get the same board. Liking an idea is not evidence that it survives review, and having written the proposal does not exempt me from the argument against it. Unlike a meeting, the board can also be run again, with the same prompt on the current code, whenever the assumptions under the decision stop holding.

Back to the essays

Field Notes

A letter from Andrei Hasna on building with AI agents

Subscribe by email

By clicking “Subscribe”, I confirm that I want to get email delivered to my address and agree to Andrei Hasna’s Privacy Policy and Terms.

In each letter

  • New essays on AI agents and the systems behind them
  • What Hasna is building, and why
  • Costs, routing and results from real work
  • What did not work
  • News of the book and the podcast