Cloudflare reviews code with seven models for $1.19 a review

Cloudflare published the numbers on their in-house AI code review system, and they’re worth a slow read: 131,246 review runs across 48,095 merge requests in 30 days, spanning 5,169 repos. The median review finishes in 3 minutes 39 seconds. The average review costs $1.19.

The system runs up to seven specialised reviewers per change, covering areas like security, performance, code quality, documentation, release management and their internal engineering standards. They built it themselves because commercial tools “just didn’t offer enough flexibility,” and then they showed their working. The design decisions in that write-up teach the pattern better than a vendor page could.

Six decisions that make it work

• A panel of specialists. Seven reviewers, each with one job. What you’re reading is the disagreement between them. (We’ve written about what that looks like on a real diff.)
• Risk decides the spend. Changes are tiered by size before any reviewer runs: ten lines or fewer gets the trivial treatment at about $0.20, up to a hundred lines gets a lite panel at $0.67, and everything bigger gets all seven at $1.68 average.
• A coordinator judges the panel. Every reviewer’s findings flow to a coordinator model that dedupes them, weighs severity against written criteria, filters the unreasonable, and posts one structured review. The reviewers surface findings. The coordinator turns that pile into one review worth a person’s time. This is the piece that’s easy to skip, and skipping it is what turns a review tool into noise: every reviewer’s raw findings land on the developer at once, nitpicks and blockers with equal weight.
• Failure is designed for. Circuit breakers watch each model provider, and a degraded one reroutes to a fallback (their example: opus-4-7 falls back to opus-4-6) instead of killing the run.
• The escape hatch is measured. 0.6% of merge requests bypass review through an explicit break-glass override. They know that number because bypassing leaves a record.
• Merge descriptions are handled as hostile input. A merge request description could carry instructions aimed at the reviewers, so the system strips boundary tags out of it before any model reads it. They treat prompt injection as an engineering problem and documented the fix.

What it took them to build

Building it meant a CI-native orchestrator, model routing with runtime overrides, per-tier agent counts, circuit breakers, cost metering and the observability to publish those numbers with straight faces. Cloudflare has the platform engineering to do that as a side quest. If your team doesn’t, the same scaffolding is real platform work before the first useful review.

The same shape as a playbook

This one is close to our hearts because the seven-reviewer panel is the flagship shape we built Sumn around, and every decision above maps to a piece of a playbook.

A merge request event triggers the run. A cheap classifier stage sizes the panel to the change first, which is their trivial-lite-full tiering. Reviewer stages fan out in parallel, each one a model reading the same diff blind in its own fresh isolated VM. Which models sit on the panel is data in the playbook definition: frontier, open-source through your own keys, or any OpenAI-compatible endpoint. Swapping a seat is an edit.

Their coordinator is what we call a judged gate: a model applying criteria you wrote before anything reaches a person. Risky splits wait at a held gate for a human; routine passes proceed. And every run ends in an itemised receipt (the seven-models post shows one), so your version of $1.19 shows up after every run.