The Agent Loop: How AI Goes From Answering Questions to Doing Things
Open models just reached frontier code review (Sponsored)
PR-AF is an open-source code review agent that ranks #2 of 42 on Martian's Code-Review-Bench, ahead of CodeRabbit, Copilot, and Devin, running a single open model. The trick is the harness: it plans a review strategy per PR, spawns parallel reviewer agents, verifies every finding against your source, and drops anything it cannot prove. That makes it about 10x cheaper per review than closed-source tools. One API call, and a drop-in GitHub Action. If open models writing frontier-level reviews sounds useful, star the repo.
A chatbot answers a question, and an agent completes a task. This seems like a huge difference. However, the gap between the two is narrower than it appears.
An agent is an LLM placed inside a loop where the model itself decides when the loop should stop. Everything interesting about agents follows from this shift. The autonomy that makes them useful, the cost that makes them expensive, and the design challenges that come with building one all trace back to it.
It helps to see where that shift sits in the broader picture.
Software built around language models has moved through a recognizable progression. It started with a single LLM call that took an input and produced an output. Then, the function calling arrived, which let the model reach out to a tool when it needed information or wanted to take an action. After that, developers began chaining calls together in code, with each step’s output feeding into the next, to handle problems too large for a single call. The most recent of these is the agent, where the developer hands control of the loop itself to the model and lets it iterate until it decides the work is done.
In this article, we will walk through that progression. We will also look at how an agent is structured, what choices the model makes on every turn, what scaffolding holds it together, and when an agent is actually the right pattern to reach for.
Disclaimer: This post is based on publicly shared details from various sources. Please comment if you notice any inaccuracies.
Foundations
Before the loop can be interesting, the unit that sits inside it has to be clear.
A bare large language model call is a stateless function. Text goes in, text comes out, and each call typically stands alone in the model’s view. If we want it to fetch today’s weather or update a record in a database, the bare model lacks the means to do either. It can describe weather in general terms and talk about how a database record would change, but reaching the actual forecast or touching the actual database requires something more.
What makes the model useful in real systems is augmentation.
In other words, we give it the ability to call functions we have defined, often called tool use or function calling. We give it access to a retrieval layer that can pull in relevant documents at runtime. We also give it a way to write down information that it should carry between calls, which acts as memory.
Anthropic calls the combination of these capabilities the augmented LLM, and treats it as the foundational unit of every agentic system. The augmented LLM is the building block from which everything else gets composed.
This is the unit most developers have already worked with, even if they have called it something else. Examples include a chat assistant that runs Python code, a function-calling API wired to your service, and a RAG application that pulls from your documents. These systems are useful, and they are also still one call. The model produces an output, the system returns it, and the interaction ends until the next request arrives.
Workflows
What happens when a problem reaches beyond what a single call can handle?
The natural response is to string calls together in a sequence we design, a pattern called prompt chaining. Each step in the chain is its own LLM call, and the output of one step becomes the input to the next. We might use one call to draft an outline, a second call to expand the outline into paragraphs, and a third call to translate the paragraphs into another language. The chain is fixed in advance, and the developer writes which steps run, in what order, and what each step’s prompt looks like.
This is the family of patterns Anthropic groups together as workflows.
A workflow is a system where the LLM and its tools are orchestrated through a code path that the developer designed. Beyond prompt chaining, there are other workflow patterns:
Routing classifies the input and sends it to a specialized handler.
Parallelization runs subtasks at the same time.
Orchestrator-worker has a manager LLM that delegates pieces to specialist LLMs.
Evaluator-optimizer has one call to generate while another critiques, with the cycle iterating until quality is good enough.
The details differ, but every workflow shares the same property. The number of steps and the path through them are decided by the developer at design time, before the model sees the input. Most production systems built on LLMs today are workflows. They are predictable, debuggable, and usually cheaper than full-blown agents.
The Loop
An agent is what we get when we wrap an LLM in a loop and let the model decide when the loop should exit. The loop itself is plain code. The runtime calls the LLM, reads the output, dispatches whatever action that output specified, feeds the result back into the model’s view, and calls the model again. This continues until the model produces a response that signals a final answer.
There are four steps inside each iteration, and we can call them perceive, reason, act, and observe:
Perceive is the moment the loop hands the model the current state, which includes the original task, the history of what has happened so far, and any new input.
Reason is the model’s turn, where the model produces a response that says what to do next, whether that means asking a question, calling a tool, or wrapping up with a final answer.
Act is when the runtime carries out whatever the model asked for.
Observe is when the result of that action gets captured and folded back into the state, so the model sees it on the next perceive step.
The most important detail is who decides when the loop should stop. In a workflow, the developer decides at design time how many steps run. In an agent, the model decides at runtime. The model exits the loop by producing an output that the runtime interprets as a final answer. The developer typically sets a hard ceiling on the number of iterations, often called a max-turns parameter, but that ceiling is mainly a safety net. The primary stop signal comes from the model.
Observation matters as a first-class step for the same reason.
The model needs to see the result of its actions before deciding the next move. Removing observation would collapse the loop into a chain, with the model running on prior expectations rather than fresh results from the world. The closed loop, where every action is followed by an observation, is what lets the model adjust as it goes.
With the loop established, the next question is what actually happens on each turn through it.
Decisions
On every turn through the loop, the model’s output picks one of four branches, and the runtime acts on that pick. This branching makes the loop feel intelligent because the LLM is choosing what kind of move to make. Here are the four branches:
Final answer: The model produces what amounts to a complete reply to the original task. The runtime interprets this as the loop’s exit signal, returns the result, and stops.
Tool call: The model produces an instruction asking the runtime to invoke a specific function with specific arguments. The runtime executes the tool, captures whatever it returned, appends that result to the conversation state, and sends control back to the model so the loop continues.
Handoff: The model decides that the current task belongs in the hands of a different agent, often a specialist with its own prompt and tool set. The runtime swaps which agent is running and feeds the same state into the new agent, with the loop continuing under the new identity.
Continued thought: The model produces a reasoning turn that consists of thought alone. The runtime captures the thought, feeds it back into the state, and runs the model again. This branch shows up most often in ReAct-style implementation, which we are about to see.
OpenAI’s Agents SDK documents the first three branches as first-class behaviors of its loop. The fourth is more a property of how a particular prompt is written than a separate code path
ReAct
ReAct is the prompting pattern that stands for Reasoning plus Acting. The pattern asks the model to interleave reasoning steps with action steps inside the same response, so the model can think about what to do, do it, see what happened, and adjust.
A ReAct trace reads like a structured journal. The model produces a thought, then an action, then receives an observation from the runtime, then produces another thought, and so on, until it reaches a final answer.
Imagine an agent handling customer support:
A user asks for the status of their most recent order.
The model produces a thought first, reasoning that it needs to find that order and that the order service is the right place to look.
It then takes an action by calling the get_recent_order function with the user’s ID.
The runtime executes the call and feeds back an observation telling the model that the order is number 9152, placed on May 14.
The model produces another thought, working out that it now needs the shipping status for that specific order.
It calls the get_shipping_status function with the order ID. The observation comes back, saying the order is in transit and expected to arrive on May 29.
The model has what it needs at this point and produces a final answer summarizing the status for the user.
See the diagram below:
Two things to consider from this flow are as follows:
First, every action is grounded in an observation, which is the closed-loop point from the previous section made concrete.
Second, the reasoning steps are doing real work, since they are how the model decides which action makes sense next, given what it has learned so far.
ReAct is one way to fill the loop, and it is by far the most common pattern in the agent frameworks we work with today.
Guardrails
Guardrails belong at the points where the loop crosses into the outside world. They are part of the architecture itself, designed in alongside the loop.
For reference, OpenAI’s Agents SDK documents three families of guardrails, all defined by where in the loop they run.
Input guardrails run on the first turn, before the agent’s main model sees anything. They are the place to catch things like prompt injection attempts, requests that violate policy, or inputs that fall outside the agent’s scope. A common pattern is to use a small, fast model as the guardrail, so the more expensive main model only runs when the input passes.
Tool guardrails wrap every function-tool invocation. A tool input guardrail runs before the tool executes and can block the call or replace it with a message back to the model. A tool output guardrail runs after the tool executes and can rewrite or block the result before it goes back into the conversation state. Tool guardrails matter because tools are how the loop touches systems we care about, and that touch needs supervision.
Output guardrails run on the final response, after the loop has decided to terminate, but before the user sees the result. They are the last layer of policy enforcement and the place to catch things like leaked sensitive data or claims the agent should avoid making on the company’s behalf.
The structural point is that guardrails sit at every interface where the loop meets the world.
Tradeoffs
Handing control of the loop to a model is powerful, and it comes with three costs that every developer should understand before using the pattern:
Compounding error: Per-step reliability does poorly when steps are chained together, and the math is not favorable. If the model succeeds on each step of a loop 95 percent of the time, the joint probability of every step going right across ten steps is roughly 60 percent. Stretch that to twenty steps, and the joint success rate falls to about 36 percent. The autonomous nature of agents brings higher costs and potential for compounding errors. The reason coding agents work better than open-ended task agents is that test feedback raises per-step reliability, which shortens the effective length of the chain that has to succeed.
Scaffolding around the loop: The harness around the loop matters as much as the model inside it. For example, Anthropic shared an internal experiment where even a frontier model running in the Claude Agent SDK fell short on building a production-quality web app from a high-level prompt. The fix was scaffolding rather than a better model. They built an initializer agent that lays out a feature list before any feature work starts, a coding agent that picks one feature per session, a progress file that travels between sessions, and a git history that the agents can use to recover.
The wrong tool for the job: An agent is often the wrong choice. It is important to find the simplest solution and only add complexity when needed, which sometimes means avoiding agentic systems entirely. Workflows offer predictability and consistency. Agents offer flexibility, and they pay for that flexibility in latency, money, and a more unpredictable failure surface. Many problems are served better by a chain.
Conclusion
The agent loop sits at the end of a recognizable progression.
We started with a single LLM call that takes text in and returns text. We added tools, retrieval, and memory to get the augmented LLM. We strung calls together into workflows when one call was outgrown. The agent is the next rung, the point where the developer hands the control flow itself to the model.
Inside the loop, four steps repeat. The model perceives the current state, reasons about it, takes an action, and observes the result. On every turn, the model’s output picks one of four branches. These are a final answer, a tool call, a handoff, and a continued thought. ReAct is the most common prompting pattern for filling the loop, with reasoning and action interleaved. Guardrails live at every place where the loop crosses into the outside world.
The design carries three real costs. These are compounding reliability across steps, the harness scaffolding that production loops require, and the question of whether a workflow would solve the problem more cheaply.
References:
The section on scaffolding is exactly where the enterprise battle is being fought right now. Everyone wants to buy a fully autonomous agent, but what they actually end up building is rigid state management infrastructure. Having an initializer agent force a feature list before any code gets written is just good product management translated into software architecture. If the loop doesn't have a structured progress file to reference across sessions, it isn't an agent, it is just an expensive random walk. Really really great breakdown!
The compounding error math is the section more teams need to sit with — 95% per-step reliability sounds fine until it's 36% success over twenty steps.
The corollary is that the highest-leverage agent improvement usually isn't a better model, it's a cheaper verifier: tests, schema checks, anything that lets the loop catch a bad step before it compounds.
That's the quiet reason coding agents worked first, and a decent predictor of which domains agents will crack next.