The Verifier-Bottleneck Claim Is Only Half Right · Issue #166

Opening

Just a few years ago, this work had one name: prompt engineering. You tweaked a single sentence until the model did what you wanted. Last year, the same work got a new name—context engineering. This year, threads call it loop engineering. And underneath every rename, the same line keeps showing up: “The bottleneck isn’t the model anymore. It’s the verifier.”

Every time the name changes, developers react the same way: “Did something actually change, or is this just a fresh coat of paint to sell a course?” That suspicion is usually justified. AI-tooling terminology churns through names far faster than the problems it points to.

So let’s take it apart, piece by piece. Here’s the conclusion up front: each time the name changed, the unit you’re actually engineering changed with it. Not the paint—the object of the work itself. But the problem the newest name points to is real, and the name only labels that problem. It doesn’t solve it.

The Unit Was “the Prompt”

Let’s set the frame from 2022 to 2024. The unit of work was a single prompt—refining the wording of one request. You added few-shot¹ examples, assigned a role, tacked on “think step by step,” reordered what you asked first. The entire surface you controlled was the text of one message, and writing it well was the whole craft.

Looking back, nearly all the know-how from that era boiled down to how you wrote one sentence: how many examples to include, in what order to ask, what role to assign. That instinct still holds up—a good prompt is still where good output starts. What changed is that it’s no longer the whole story. The work itself stopped fitting inside one message.

🧩 The Unit Became “the Surface”

In mid-2025, Andrej Karpathy put a name on something people were already doing: context engineering. The unit widened from a single message to everything inside the window—the system prompt, documents pulled in by retrieval, tool definitions, and instruction files like CLAUDE.md or AGENTS.md² that ride along on every turn. This was no longer about refining one sentence. It became curating an entire surface.

That surface has a property developers usually learn only the expensive way: most of what you load onto it never connects to actual behavior. “The State of AI Instruction Quality” ran a deterministic analyzer across 28,721 repositories and found that the median instruction file packs just 12 actual directives into 50 content items. The rest was headers, background explanation, and structure the model was free to ignore entirely.

A sharper failure mode got its own write-up. “Do NOT Think of a Pink Elephant” shows that a constraint phrased in the negative (“don’t use mock data”) can actually raise the probability of the very behavior it forbids—for the same reason that title just put a pink elephant in your head.

Both are problems of context engineering, and both have been measured and published.

🔁 The Unit Is Now “the Loop”

By June 2026, the term is loop engineering. Addy Osmani wove together a discussion between Boris Cherny and Peter Steinberger and put it into circulation; within weeks it spread across the whole timeline. Now the unit is the running loop: generate, check, steer, retry, stop. The prompt became just one node inside it. The context surface became the state the loop carries between iterations.

(Diagram: Prompt → Model generates → Verifier (“Good enough?”) → Stop. A loop runs back from the verifier to “steer and retry,” with the context surface feeding into model generation.)

Every explainer repeats the same claim: the limit is no longer set by the model, but by the check that decides “good enough, stop.” Let’s give this name credit. It points directly at what the previous two names quietly skipped past—the mechanism that judges whether one iteration was good enough, and whether to run another. That mechanism was always there. Any agent that retries has one. What loop engineering adds is that this mechanism stops being an inherited default and becomes something you design yourself.

🔑 So What Does the Verifier Actually Check?

“The verifier is the bottleneck” is a good slogan. And it’s half of one. It tells you where the bottleneck sits without telling you what that verifier is supposed to check. That blank is exactly the engineering problem the slogan skips over.

There are two kinds of checks, and they aren’t interchangeable. A deterministic check³ runs the code, checks the exit status, scans for forbidden imports, and returns the same verdict for the same input every time—no judgment enters the loop. A model-graded check⁴ asks another model, “is this okay?” It can capture criteria the first method can’t express—“is this explanation clear,” “does this read as rude”—but in exchange, it inherits exactly the instability the loop was built to contain. You end up handing the question of when probabilistic generation is done to a judge that renders its verdict probabilistically too.

Look at where the two split apart in an actual loop. Suppose you tell an agent to refactor one module and stop when it’s done. A deterministic verifier can prove the tests still pass and that no forbidden import slipped in—a verifiable fact on every iteration, nothing to argue about. But it can’t judge whether the refactor was worth doing in the first place. So to answer that, you bolt on a model-graded check, and now the stopping condition rests on one model grading another model’s judgment. The loop ends when the judge is satisfied, and that judge is exactly the same kind of component the loop exists to supervise.

Neither approach is wrong. They answer different questions. The moment you lose track of which question you’re actually asking, the loop stops confidently at the wrong iteration.

When I’m unsure in practice, I split it this way: “can I extract a true/false verdict on this criterion by running or scanning something, without asking another model?” If yes, keep it a deterministic check almost every time—it’s cheap, it doesn’t waver, and it leaves a log of why it failed. If the criterion only exists inside a human’s head (say, “does this copy match our brand tone”), there’s no dodging a model verdict. In that case, at least break the criterion down into small pieces and force the model to answer against a checklist item by item, rather than issue a free-form overall judgment. That narrows how much the verdict can wobble. You’re not eliminating judgment—you’re narrowing the room it has to swing.

This trade-off—what a check actually checks—isn’t a new problem. “Green Tests Don’t Mean Better Software” covers the CI version of it. A green test only proves the code matches the spec; it says nothing about whether the change made the system better. The check answered a question you never asked. Point that same distinction at an agent loop instead of a test suite, and you’ve described loop engineering’s core question using an example every developer already trusts.

Oz’s Lens

Honestly, this new name makes me both glad and wary.

After years of building GTM strategy, I’ve seen this pattern more times than I can count: the forecast “technology will change this” is usually right. It’s the timing and the path that are almost always wrong. Whenever a new term starts trending, that’s my first suspicion. But these three renamings are a little different. Each time the name changed, the object of the work actually moved—not the marketing. From sentence to surface, from surface to loop. So I’d place these renamings on the honest side of the ledger.

Still, here’s where I stay clear-eyed. A name only hands you an object. Building the verifier is still your job, and it’s most of the work—defining what “correct” means for a given task and turning that into a runnable check.

Half of that job is choosing the type of check. In the habits I picked up working with data, this is a measurement-design problem. “Is this explanation clear” needs a model’s judgment. “No forbidden imports” only needs a deterministic scan. Get the choice wrong and you pay on either side: hand a model judge something a deterministic check could have settled, and you buy instability for free; put a deterministic check where judgment was actually needed, and you get a green light that means less than it looks like. The name only makes this choice visible. It doesn’t make it for you.

Closing

To sum up: first, the rename from prompt to context to loop wasn’t a fresh coat of paint—the unit you engineer actually grew larger each time. Second, “the verifier is the bottleneck” is true, but only half true; the other half is what that verifier checks and how. Third, a name only labels the problem. Building the verifier stays your homework.

This loop has several moving parts. Over the next few issues, I plan to pull the loop apart piece by piece and lay a measurement under each one: how to tell a check that fires in the wrong place from a signal that’s simply missing, how steering rules differ from rejection rules, and what a single instruction sitting on the context surface actually costs you per turn. The questions this piece raised are where the next installments start.

If you’ve run an agent loop yourself, did you set the stopping condition as a deterministic check, or hand it to a model judge? Tell me in the comments where you got burned the worst—I’ll fold it into a future issue.

💬 Tell me in the comments how you designed your stopping condition · 📨 Forward this to a colleague building agents

References & Further Reading

Primary sources

reporails, “The State of AI Instruction Quality”. The source for the finding that a median instruction file has 12 actual directives among 50 content items.
cleverhoods, “Do NOT Think of a Pink Elephant”. The source for the claim that negatively phrased constraints can raise the probability of the very behavior they forbid.
reporails, “Green Tests Don’t Mean Better Software”. The source for the argument that a check can answer a question you never asked.
cleverhoods, “Prompt Engineering, Context Engineering, Loop Engineering: What Actually Changed”. The primary source that today’s issue is built on.

Background

Andrej Karpathy coining “context engineering” (mid-2025). Where the term originated.
Addy Osmani synthesizing “loop engineering” (June 2026, drawing on discussion between Boris Cherny and Peter Steinberger). How today’s third name spread.

📝 Glossary

The author, Kwangseob Ahn, is a professor of business administration at Sejong University and lead consultant at OBF (Oswarld Boutique Consulting Firm). He teaches statistics and data analysis — business data management and business analytics — while leading GTM and AI strategy consulting in the field, designing the seam between technology and business. He has published academic research on a memory architecture for AI dialogue systems (HEMA) and runs Daily Arxiv, a daily curation of global AI papers. He holds a master’s from Korea University’s Graduate School of Technology Management and a KMBA. He is the author of Homo Brainless: The People Who Outsource Their Thinking.

Footnotes

Few-shot: a method of showing the model a handful of examples so it follows the format or approach you want. It produces more stable results than simply instructing without examples. ↩
CLAUDE.md / AGENTS.md: instruction files that tell an AI coding tool “here’s how we work on this project.” They ride along with every conversation turn. ↩
Deterministic check: a check that returns the same result for the same input every time—mechanical verification with no judgment involved, like running code, checking exit codes, or scanning for banned words. ↩
Model-graded check: a method that asks another AI model to judge whether a result is good or bad. It can mimic human-like judgment, but the result can waver just as much. ↩