What a person settles, and what the model works out
Reading everything an agent did stops working once there’s more output than one person’s day holds. What has to stop being your job as the count climbs is its own question. This one is narrower: whether a run stays checkable when nobody reads every step.
A run here means one execution of a written-down process, start to finish, with stages inside it. Not one call to a model, and not one pass of a loop. When something is settled before a run starts, it holds for every stage in it.
The loops-versus-graphs argument is supposed to answer that, and the vocabulary gets in the way first. “Loop” is doing at least three jobs here, and people rarely say which one they mean:
| When someone says loop | They might mean |
|---|---|
| The cycle inside a single run: the model calls a tool, reads what came back, goes again | an agentic loop |
| Running the whole agent again and again, fresh context each time, at the same goal | loop engineering |
| A box in a diagram with an arrow back to itself | the shape, and nothing about what enforces it |
The graph-engineering argument took aim at the second one, a fresh context sent at the same goal over and over. Jeff Liu, who writes about building multi-agent systems, published a graph-versus-loop piece that same week arguing against the first: one agent discovering the path as it goes, all in one context window. Two pieces from the same week used the word for different objects. A graph is that work split into named steps instead, and graph engineering covers that side on its own terms.
The two aren’t exclusive either. You can loop a graph, and a checked loop covers the same ground when every pass has the same access and nothing runs in parallel. Where stages need different credentials, or state has to survive a restart, or a rejection has somewhere particular to go, the graph still earns its place.
The same distinction is the honest answer to what a playbook is, since the word gets used loosely. A workflow can be either thing. Plenty are agent skills end to end, with an agent judging whether each step went well enough to carry on, and that’s still a workflow. A playbook interleaves: determinism where determinism belongs, and judgement left to a model only where judgement is genuinely the right tool. That’s the kind of run described here, and the rest of this is how you decide which parts get which.
Take a run that pulls yesterday’s numbers from three systems, writes them up, and posts the summary.
| Settled before the run starts | Left to the model |
|---|---|
| Which stages exist, and what has to finish before the next one starts | How to do the work inside a stage |
| That the drafting stage can read the billing system and cannot send mail | Which of the tools it does have to reach for |
| The spend ceiling, and how many attempts a stage gets | Whether this attempt is worth making |
| The reconciliation rule, and that posting blocks until it passes | How to investigate a mismatch |
| That a failed check sends the work back a stage | What to change so the next attempt passes |
Model choice sits outside that split. Inside a stage, use the best model for the job and let it work however it likes. What it doesn’t get is a vote on whether the rules apply.
How to tell which decisions to settle yourself
One test sorts the two columns: what happens when the model gets the decision wrong. If the next check catches the mistake, the model can keep that decision. If the mistake stops the check from running, you settle it in advance. Picking the wrong tool fails a check. Deciding this run doesn’t need the check fails nothing anyone can see.
That keeps the settled list short. It holds what the checks depend on and nothing else, which raises two questions: what counts as evidence, and who enforces it.
What evidence lets the next stage start
You could let the agent tell you. An agent that has failed reports done in the same words as one that hasn’t, so its report adds nothing you didn’t already have.
Anthropic hit that building a harness for long-running agents: code left broken, features marked complete without anything having tested them. What got them past it was giving the harness its own ways to find out. One was having the agent drive the running app in a browser, which surfaced bugs that reading the code never would.
So a model’s report on itself can’t be the condition. What the next stage needs is evidence the worker had no power to waive: a test suite, a parser, a reference implementation, the target system read back after the change. Where the work is subjective and none of those exist, a separate judge applying criteria you wrote is the closest thing available. The runner can require that the judging happens, and that part is worth having. It can’t make the verdict any less probabilistic, so anything consequential still waits for a person or for a check the judge can’t waive.
A check is only as good as the definition behind it, and writing that definition is work that happens before any agent runs. Skip it and you get a test suite that passes while the thing it was guarding quietly stopped being true.
A reference implementation is the strongest version of that evidence, and Anthropic’s C compiler build has the cleanest example. GCC was the known-correct answer, and the harness used it to split verification across files. The half worth copying is what didn’t work before that. “Having 16 agents running didn’t help because each was stuck solving the same task.” In that run, parallelism started paying only once the verification strategy gave each agent different files to test.
Putting a model in the box marked reviewer runs into the same limit: you’ve added another probabilistic step and labelled it a control. A graph makes failure legible without making the system reliable, and Jeff Liu puts the gap sharply: “The graph buys you the first. The hype sells it as the second.”
Who enforces the condition
A loop can enforce a real check without becoming a graph, but only where the check sits in the wrapper rather than in the prompt.
A shell loop that just runs the agent again enforces nothing. If the instructions say to run the tests, the agent decides whether it ran them, and a pass that quietly skipped them produces the same cheerful summary as a pass that didn’t. Put the tests in the loop itself, so they run between passes and a failure blocks the commit, and the pass genuinely cannot skip them. Loop engineering covers how that gets built.
A graph can just as easily fail to enforce anything, for the same reason: draw five boxes and let the model report on each one, and you have five places to be told everything is fine.
We can’t point you at evidence that settles this. We looked for a controlled comparison, the same model and tasks and budget run both ways, measuring what escaped and what it cost, and couldn’t find one. The studies we did find change the shape alongside other parts of the system, so they can’t tell you which change earned the result. Treat confident claims here as design positions, ours included.
Two stages that need different access want a line between them, and so does a failure that has somewhere particular to go. Anthropic’s building effective agents has the older version of the shape: a path laid out ahead of the run, with programmatic checks sitting on the intermediate steps. The checks are what hold; the path tells you where to put them.
A boundary also has to survive being stopped at. Say the run holds for approval and someone gets to it an hour later. What exactly starts again?
The answer depends on the runtime, and it isn’t always the one you’d assume. LangGraph, for instance, documents its behaviour plainly: the interrupt saves state and waits, and resuming reruns the interrupted step from its start, so whatever that step did before it paused happens again. If that was a database read, no harm done. If it was a charge or an email, someone gets two.
So whether everything before a hold can safely run twice is one more thing to settle in advance.
Where the runner records a finished stage in a durable checkpoint, the gap between stages is somewhere a hold can sit. Resuming starts the next stage instead of replaying the last one, and the person who comes back to it reads what already happened off the run’s record. Without that checkpoint, a hold is a pause you have to reason about each time you use one.
Where CI fits
A pipeline answers one question, and answers it well: did this change pass the checks you encoded for shipping? It builds the thing and runs the suite, and it goes green or it doesn’t. That’s a genuine hard check. It runs outside the model and the model can’t argue it into passing, which makes it one of the better conditions to hang a stage on.
What it doesn’t answer is whether the work was the right work. A green build says the change compiles and the tests still pass. It says nothing about whether the agent solved the problem you actually asked about, or what it reached for and spent on the way.
CI and the run around it do different jobs. CI is the check. The run decides:
- Which stage may call it
- What that stage can reach
- What it can spend
- Whether a person signs off before the irreversible part
- What the record says when someone asks in three weeks
Doing this for a run of your own
Take the run you already have and write down what you settle before it starts. Four questions get you most of the way:
- What can each stage reach?
- What can the whole run spend, and what stops it there?
- What evidence has to exist before the part you can’t take back?
- Where does a rejection send the run?
If one of those is blank, the model is already deciding it on your behalf. A spend cap that halts the run is how the second one gets an answer that holds while nobody’s watching.
Then check where each answer lives. If it lives in the prompt, you asked for it, and the model weighs it against everything else you asked for. If the runner holds it, the model works inside it and gets no vote. The nine-question version of this pass goes further, including what happens when something breaks partway through.
Every condition you want holding while you aren’t reading has to be enforced by something the model can’t talk to.
What should the model be allowed to decide?
Watch a playbook run live
→See what a run settles before it starts, and what the model decides inside that. You can watch one before signing up.