What is multi-model code review?

Why would several models beat one good one?

The design bet is that models trained separately are less likely to share the same blind spots. Run the same model twice and you mostly get the same review twice: its gaps come from its training, and it carries them into both reads. Run a model from a different lab and it should have different gaps.

That’s a design hypothesis, not a measured statistic. The strongest evidence we have for it is one real run. When we put a scheduler change through seven models, three converged independently on the same blocker: a failed cleanup that was caught and logged but never rethrown. Those three, claude-opus-5, kimi-k3 and deepseek-v4-pro, came from three different labs. One model raised a load question none of the others saw. Two filed clean reviews. The whole run, seven reads plus the aggregation step, cost $1.74. The full breakdown, including which model said what, is in Seven models reviewed the same change.

Three ideas get blurred together in a claim like that. Consensus is what you observe: three models pointing at the same line. Correlation is the open question underneath it: are those three independent enough that agreement means something, or close enough in training to share a blind spot and mistake it for confirmation? Adjudication is a separate step again, a judged gate that reads the raw reviews and decides what reaches a person. It doesn’t vote, and it isn’t what made the models agree or disagree in the first place.

How does a panel actually run?

Three properties separate a real panel from an expensive way to get one opinion:

• Blind reads. Every reviewer sees the diff and the instructions. None sees another reviewer’s output until all reviews are filed. Models anchor hard on prior text; let reviewer two read reviewer one and you’ve built a slow single reviewer.
• One contract. Same instructions, same output shape for every model. If each reviewer answers a different question, you can’t compare the answers, and comparison is the product.
• A judge that isn’t a voter. After the reviews are in, something has to decide what reaches a human. That’s a judged gate: an aggregation step reads criteria you wrote and sorts findings into “fix this”, “check this” and “noise”. Simple majority voting works worse than people expect, because the interesting finding is often the one only a single model raised.

Keep aggregation and merge approval separate. Aggregation runs a judged gate: a model applies the criteria you wrote. Merging waits behind a held gate: a person clicks it, every time, until you deliberately decide otherwise for a class of change. A split panel is exactly what lands in the attention inbox, the queue for decisions like this one.

How do you read the results?

Start with the distribution across reviewers.

• Check agreement first. When three models from different labs land on the same line independently, start there. You didn’t need a meeting.
• Check the lone concern too. A concern only one model raised might turn out to be nothing. Either way, the panel has converted a vague unease into a specific claim you can check. Spend the ten minutes.
• Watch which reviewers stay quiet. Reviewers that go quiet on routine changes and light up on the real ones are the ones you learn to trust. A reviewer that files blockers on everything is noise wearing a badge.

The stronger signal is whether a reviewer’s findings hold up when you check them. Every finding names a line and a failure, so you can reproduce it or trace the path it points to. A flag that falls apart costs the reviewer that raised it. Across enough reviews, some models keep raising findings that survive the check and others mostly surface trivia or nothing. Promote a new model when its findings earn the seat, and drop one that costs money without saving anyone time.

What about false positives?

A panel multiplies reads, so it multiplies raised findings. Plenty of review tools pattern-match and call it insight, so the worry that this is just more noise is a fair one.

Two things keep the rate workable. The output contract forces every finding to name its line and its failure case, so vague flags are easy to reject. The judged aggregation step is built to dedupe and rank before anything reaches a person. By design it collapses seven raw reviews into one short list instead of forwarding all of them.

One run says nothing about a false-positive rate or how accurate the panel is. That takes many runs across many kinds of change, with someone tracking which flagged findings turned out to matter and which didn’t. We don’t have that count yet. Until someone does that work, a converged finding is a strong reason to look first. It is not a calibrated probability.

What does it cost?

Our seven-model panel on a 400-line diff cost $1.74, itemised per call. The reviews run in parallel. The whole panel takes about as long as its slowest review, plus the aggregation step that runs after it. That’s one run on one diff; Cloudflare’s seven-reviewer panel shows the same shape at real scale, 131 thousand runs a month, and what it costs to keep that metered.

One panel for every change is a fine start. The better habit is to match review depth to how complex the change is and how easily it reverses: a one-way-door change (one you can’t easily undo) earns all seven reviewers, and a reversible two-line fix gets a light pass. Cloudflare tiers its panel by change size before any reviewer runs; reversibility sharpens that cut, since a small diff can still be a one-way door.

The real costs on your side are elsewhere: keys and quota for several providers, and the harness work of isolated stages, an aggregation step and somewhere for escalations to land. That harness takes real engineering time: you fan the reviews out, aggregate them, hold the merge for a person, and record the per-call cost.

A Sumn playbook is that harness written down. Reviewer stages fan out in parallel, each in its own fresh isolated VM, and a judged gate aggregates their findings against the criteria you wrote. The merge stays behind a held gate with a person, and every run ends in a receipt itemised per call. Which models sit on the panel is data in the playbook, so swapping a seat is an edit and open-source seats run on your own keys.

When is a panel the wrong tool?

The pattern stops paying in a few places:

• Trivial diffs. A dependency bump doesn’t need seven opinions. Route by change size and blast radius.
• A bad spec. If the intent of the change is unclear, seven models will disagree uselessly. The panel reviews code against intent; it can’t invent the intent.
• As a replacement for tests. Reviewers reason about code. They don’t execute it. A panel is a complement to CI, never a substitute.