An incident hits and you pull up the logs, only to find string based console output that tells you something broke but not why. That’s the problem with unstructured logging: it eats your quota and still leaves you guessing.

This hands-on workshop fixes that. You’ll learn what’s actually worth logging (state transitions, request boundaries, external call failures, auth decisions), how to structure logs as real data instead of text dumps, and how to add attributes that connect them to traces and errors. Plus, how to tune signal to noise, so noise stops drowning out what matters.

Save you spot

Roblox is built for scale. In August 2025, the platform reached a peak of 45 million concurrent users. Few game engines can support that kind of demand.

But Roblox itself isn’t the one making most of those games. It gives creators the tools to build them, then focuses on the infrastructure beneath them. Now, Roblox is exploring how a video world model can work alongside its game engine to make those experiences photorealistic without compromising scale or multiplayer performance.

This hybrid approach offers valuable lessons for engineers building real-time systems at scale. To explore those lessons, we sat down with Anupam Singh, senior vice president of engineering at Roblox, to hear from him about the world model that Roblox is using to make its multiplayer games look photorealistic, the key insights that have come from taking that approach, and the next big thing that the Roblox team is focusing on. We thank Anupam for sharing with us.

In this article, you will learn:

• What a game actually requires, and why visuals are only part of it.

• How combining a world model with a game engine can make games more realistic.

• How Roblox is using a hybrid architecture to divide the work between the engine and the world model and keep the world consistent for everyone playing.

• The four open problems (latency, consistency, multiplayer, and creator control) and the specific techniques Roblox is using on each.

• What Roblox is focused on next, and why it could make photorealistic games far cheaper to build.

What makes a game a game

When people praise a game, they usually point at the graphics. Graphics are the easiest part to notice and often the smallest part of the engineering. What actually makes a game a game is a set of systems players never see. These systems serve one of two purposes: keeping the world consistent for everyone playing, and rendering it on screen.

1. Keeping the world consistent

Keeping a world consistent takes several systems working together. The foundation is the persistent state, a running memory of where everything is and what has happened, so the world is still there when you come back to it. Then come the rules, the logic that has to resolve the same way for everyone, so the game stays fair. Physics has to be believable, so objects move, collide, and fall the way players expect. Since most games are played with other people, all of it has to stay in sync in real time, so thousands of players share one consistent world rather than thousands of slightly different ones. On top of that, the world has to react the instant a player acts. Together, this is what turns moving images into something you can actually play.

A racing game makes this concrete. The track, the scores, the penalties, how each car handles, and where every other car is all need to be exact. Every player needs to see the same thing at the same moment. If even one of these is off, players notice right away.

2. Making the game look real

The second purpose, making the world look real, is a newer demand. The real world has a lot of fine detail that is hard to produce. Grass moving in the wind, dust behind a car, smoke off a fire, light shifting across a surface. Players now expect that level of realism.

That is what makes the problem hard. No platform has delivered all of it at once: a world that is consistent and fair, looks real, and still runs affordably on the hardware people already own.

Today, two technologies each cover one side of this. AI world models are built to make things look real, generating photorealistic images from what they have learned. Game engines are built to keep worlds consistent, tracking state, rules, and physics across every player. The catch is that each is good at only its own half. The next sections look at why a world model alone, and a game engine alone, each fall short, and then at how Roblox combines them.

Crusoe Serverless Fine-Tuning Is Now GA (Sponsored)

Fine-tuning open models shouldn’t mean babysitting GPU clusters. Crusoe Serverless Fine-Tuning, now generally available in Crusoe Intelligence Foundry, gives you a complete path from proprietary data to production-ready model. Select a base model from a curated library of top open models — including Qwen, DeepSeek, Gemma, and gpt-oss — upload your dataset in JSONL or Parquet, configure your job, and submit via UI, SDK, or API. Token-based pricing ties spend to actual training work, and early stopping ends billing the moment your model stops improving. Deploy in one click, or download your weights in .safetensors format.

Get started

Why a video world model alone isn’t enough

A video world model generates a world one frame at a time, predicting what should come next. It can also take conditioning signals as input, such as a text prompt or a starting image.

For a game, the signal that matters most is the player’s actions. That is what makes a world playable. For example, when a player presses a key to move forward or turns the camera to look around, the next frame changes to match. But generating believable frames, even ones that respond to input, is not the same as running a game, for two main reasons.

Reason 1: missing the systems that make a game

The first reason is that a video model does not have the systems a game runs on. Its strength, generating what looks right next without simulating every object, is also why it cannot hold a game together. It has no durable memory of the world, no consistent rules, and no reliable way to turn a player’s input into the right outcome. The simplest way to see this is to turn the camera away from something and then back. “If the character turns back, is the world still the same?” Singh says. “In most video models, it won’t be.” The model regenerates the scene and quietly changes it.

Multiplayer exposes the gap further. A video model can render a crowd of 500 people, but those are pixels, not 500 separate players, each with a position, an inventory, and actions of their own. Painting a crowd and simulating one are different problems. However good it looks, a standalone video world behaves more like a moving picture than a game: a guided dream, with no persistence and no real interaction.

There are quieter gaps too, ones that show up in how games are built rather than how they look. A pure video model is hard to test and hard to refine. With normal game code, a creator can change one rule, run it, and check the result. A model that generates the world from learned weights does not give you that kind of precise, repeatable control, which makes exacting competitive gameplay and reliable, intelligent non-player characters hard to guarantee. These are not edge cases. They are the day-to-day work of making a game good.

Reason 2: pixels are not understanding

The second reason is deeper. A video model works entirely in pixels, and pixels are not understanding. It can draw a flawless bottle without knowing it is a bottle: that it opens, that it holds water, or that it falls when you let go. As Singh puts it, pixels alone will never understand the world. Something else has to do the reasoning, working out what things are and how they behave. A model that only paints the scene has no real grip on the rules, the physics, or the cause and effect a game runs on. It is rendering the world, not modeling it.

A game engine is built to do exactly what the model cannot. It does not paint pixels; it tracks what exists, enforces the rules, and computes the physics. So if a video model lacks all of this, why not just build the whole game in the engine? That is the question the next section takes on.

Why a game engine alone isn’t enough

A game engine is built for the parts a video model cannot do. It tracks the exact state of the world, applies rules the same way every time, and keeps that state consistent across sessions and players. Engines also compute physics directly, including collisions and the movement of objects such as a car’s position, velocity, and steering. This side of game development is mature and well understood.

Photorealism is where engines struggle. To make a world look real, the engine has to render all of that detail explicitly: high resolution textures, complex lighting, and simulation for effects like light scattering through fog or the way different materials reflect it. All of that is expensive to compute, and it only gets more expensive the more realistic you push it. So engines cut corners. They bake lighting ahead of time instead of calculating it live, swap in simpler versions of objects that are far away, reuse textures, or lean on a stylized look that hides the missing detail. The tricks work, but they are also why most games still look like games, not the real world.

Avoiding those shortcuts, and actually reaching photorealism, comes at a cost that falls on people, not just servers. Creators need more skill and time to build the detail, and players need more powerful devices to run it. A big studio can absorb that. A two-person team, or someone on an everyday phone, cannot. This is the real limit: an engine can give you a real game, but realistic graphics through it stay out of reach for most creators and most players. Each technology is strong exactly where the other is weak, so the obvious move is to combine them.

The core idea: a hybrid architecture

Roblox’s position is that you should not ask a video model to act as a game engine, and you should not ask a game engine to generate photorealism by itself. Instead, you build a system that splits the work so each part does what it is good at.

Roblox calls that system Roblox Reality. It has three parts: the Roblox game engine, the Roblox Cloud, and a video world model the team calls the Super Upsampler. The first two keep the world consistent. The third makes it look real.

The game engine is the part that knows what is actually there. It keeps a structured record of the world, called the data model, that lists every object and its properties: where the car is, how fast it is moving, which way it points, what it is made of. On top of that record it runs the simulation, the physics and the rules, the same way every time. Because it computes all of this exactly and repeatably, the engine can act as the single source of truth that every player’s view has to agree with.

The Roblox Cloud is what runs that engine at scale. It stores each world’s state durably, so a game is still there when you come back days later, and it runs millions of live game sessions across data centers around the world. Keeping those sessions physically close to players is what makes fast, fair multiplayer possible, because the authoritative state lives near the people acting on it instead of on the far side of the planet.

The Super Upsampler is the video world model, and its name describes the job. It does not invent the world from scratch. The engine first renders a complete but plain frame, with the right shapes, positions, and camera motion but simple textures and lighting, and the model upsamples that frame into something photorealistic. As Singh puts it, the model is mostly adding the textures and the fine detail, not deciding what is in the scene. It is a large base video model that Roblox post-trains for the task, so it already knows how the visual world looks and only has to fill in the details: the textures, the lighting, the water on a windshield, the leaves moving as a car passes. The engine does the exact, inexpensive part, and the model does only the expensive, fuzzy part, which is what makes the hybrid cheaper than either extreme.

There is a deeper idea behind this split. It is tempting to assume the goal is one giant model that learns to do everything, including the reasoning about what is happening and why. Roblox’s bet is the opposite. A world model still needs reasoning, but it does not all have to live inside the network. As Singh puts it, in a hybrid the reasoning can sit beside the model, in the engine and its rules and physics. That frees the video model to spend all of its capacity on the one thing it is best at: the pixels.

Dividing the work like this only pays off if the player never feels the seam. For that, the engine and the model have to stay tightly in sync, frame by frame.

How do the engine and the model stay in sync?

The real engineering is in how these parts connect, because the model cannot be allowed to invent the world freely. Every frame, before it generates a single pixel, it is conditioned on what the engine produces. Those inputs are built by cloud systems that handle level of detail and compositing, then delivered to the edge over a content delivery network, the same way video is streamed.

Take a single frame in a driving game. The engine generates a low-quality rendered frame, a depth map, the scene’s lighting and weather, and structured details for every object. Each is a different kind of signal that the upsampler will later use to render a photorealistic version of the frame.

The dense, pixel-aligned signals carry a value for every pixel: the rough rendered frame, and a depth map giving each pixel’s distance from the camera. Because they share the same grid as the output, they are concatenated with the model’s input, or injected through a ControlNet-style side network, so every generated pixel stays locked to its depth and color.

The global signals are a single setting for the whole shot, like midday, light rain, and the sun low on the left. These are applied through modulation, scaling and shifting the model’s activations everywhere at once, the same way a diffusion model feeds in its timestep.

The structured signals are compact, text-like descriptions: each object’s details, such as a metal sports car at position (120, 0, 48) moving 40 meters per second, plus any prompt for the overall style. These enter through cross-attention, so any part of the frame can draw on them as needed.

From all of this, the model paints the final 2K frame, the gloss on the wet paint, the glare on the windshield, the rough asphalt, the spray off the tires, without ever moving the car or changing how far away anything is.

The data model is what keeps the world from drifting. It tells the model which objects are present and fixed, so a car stays a car with four doors from one frame to the next. The model does not get the whole game state each frame, though, since that would be far too much to process. Instead, a retrieval step pulls only the objects relevant to what it is rendering right now. The engine stays the authority on what exists, and the model only decides how it looks.

State and pixels are also stored differently, and that difference is the center of the design. The consistent world state lives in the data model on the server, where it can be shared across every client and kept stable over a long session. The look of the world lives in the model’s video latents, which are regenerated continuously and never need to be saved. The server decides the truth that everyone shares, and the model fills in the parts that can safely differ a little from one player to the next.

The hard problems and how Roblox is approaching them

With the architecture in place, the real work is making it run in real time, for millions of players at once. The Roblox engine and its global infrastructure are in production today. The video model side is still in the lab, not yet running in real time. Below are the four problems standing between the design and shipping the model into production.

Latency

A game has to respond the moment a player acts. As Singh puts it, a player who clicks can’t be left waiting seconds for the result. Most video models can’t meet that bar. They generate a whole clip at once in an offline pass that reads in both directions across time, which can take several seconds. A playable game needs a fresh frame every few tens of milliseconds. That means closing the gap from roughly five seconds down toward about 30 milliseconds.

The main technique is self-forcing, the core of Roblox’s acquisition of Morpheus AI, whose founder developed the method. Self-forcing turns a slow offline model into an autoregressive one that generates frame by frame, each frame conditioned on the frames it just produced, so it can stream output continuously instead of computing a full clip in one shot.

Self-forcing is not the only lever. The models are made smaller, and their running memory, the key-value cache, is compressed, which Roblox calls critical for reaching high resolutions without the cost spiraling. Generation also runs on H200 and B200 class GPUs in edge data centers placed right next to the game-engine instances and close to players, so the network round trip stays short.

Consistency

The data model discussed earlier keeps the world from drifting frame to frame. The harder challenge is maintaining consistency across many frames.

Over a long session, small errors accumulate, and a world can slowly lose track of itself. The model has to stay coherent across minutes of play, not just a handful of frames at a time.

Staying coherent over a long session takes a model that can hold much more of its own history, and Roblox chose to acquire that capability rather than build it from scratch. The acquisition of Morpheus AI, the same one behind self forcing, brought the long-context world-model work Roblox needed for exactly this. Roblox is now extending the model’s context window, so what it generated ten minutes ago still shapes what it generates now.

Multiplayer

Multiplayer is the clearest failure of video models and the problem most specific to Roblox. A model can generate a crowd, but it cannot run one. Managing 20,000 players reacting in real time means tracking 20,000 separate states, not generating a believable picture of a crowd. The requirement is also subtle at a small scale. When one player moves an object, every other player who can see it has to see it move too. An action is not only what a player does, it is the effect of that action on everyone else’s view of the world.

The engine makes this possible by acting as the server authority, computing one true state that all players share. Each player’s video model then generates that player’s own view, conditioned on the shared state, so the views differ in perspective but agree on the facts. To keep this responsive, the system also runs a quick predictive simulation on the player’s own device for immediate actions, while the server confirms the authoritative result a moment later. In the future, a player’s own avatar may be rendered locally and placed on top of the streamed video, so the movements a player cares about most feel instant. Doing all of this for thousands of players at once is an active research area, not a solved feature, and Singh describes multiplayer actions as a central focus of the team.

Creator control

A game only becomes worth playing when a creator shapes it. What makes a game fun is usually the rules and logic a creator adds, not how detailed it looks. Even a plain-looking game can take off when the gameplay underneath it is good. So the video model has to stay under the creator’s control, and giving them that control is still an open area of research.

One concrete piece comes from Lucid AI, a team that joined Roblox. Its founder built what Roblox calls a game cartridge harness, which wraps the deterministic logic of a real game engine around a video world model. The point is to let a world look generated while still behaving according to fixed, predictable rules the creator sets.

The harness is also how a player’s controls reach the model. The standard movement keys, the WASD inputs, are fed in as conditioning, so pressing forward drives the generated view directly instead of only moving a character inside the engine. Taking real-time player input and turning it into the right next frame is the kind of control a standalone video model cannot handle. How creators will ultimately direct the model beyond movement, through written prompts, through the data model, or some mix of the two, is not yet decided.

Roblox’s bet: creators, compute, and a 20-year head start

These are difficult problems, and Roblox’s case for solving them rests on a few advantages. It has a large community of creators to build with and learn from. It owns its compute, running more than two dozen edge data centers and its own GPUs instead of renting capacity from cloud providers, which matters when the video model has to run for every player at low latency. It has also spent close to 20 years building the data model, the structured record of objects and physics the whole hybrid depends on.

To strengthen the AI side, Roblox brought together founders from several labs and gave them autonomy under a short set of values. Respect the community, take responsibility, get stuff done. Each team came in solving a different piece of the problem. Morpheus AI brought self forcing and the low-latency, long-context work behind it. Lucid AI brought the game cartridge harness, the bridge between deterministic game logic and a video model. Dynamics Lab brought a real-time, general-domain world engine that lets a person upload any image, a photo, a drawing, a painting, and step into it as a live, interactive world they can shape with text prompts and share with friends to play instantly. Put together, those are the pieces of speed, control, and generation that a hybrid needs.

There is one more advantage, and it is operational rather than technical. Even once the model runs in real time, Roblox will have to serve it to millions of players at once, and few companies have more practice keeping a system alive at that scale. Capacity is hard to predict when a game can jump from 3 million to 22 million players in a few weeks, so Roblox plans for sudden surges instead of assuming they will not come. Rather than throttling traffic when a system is under load, it runs a weekly exercise it calls Taco Tuesday, deliberately removing capacity from services in production to find the limits first. The test is automated, and any engineer can start one. Reliability is treated as everyone’s job, with executives on the same on-call rotation as engineers, holidays included.

What’s next

An early version of Roblox Reality is expected later this year or early next, with cost and scaling still to solve before it can reach large numbers of players. The first quality target is 2K resolution at 60 frames per second, with 4K as the longer goal.

The more important change is who gets to build photorealistic games. Because the heavy rendering happens on shared edge GPUs rather than on the creator’s machine or the player’s device, visual quality stops depending on the size of a team’s budget or the power of a player’s phone. A creator can get high-end results on an ordinary device. The goal Singh describes is for a two person studio to build something as rich as a large studio’s work, without having to trade complexity against visual quality.

Singh expects the progress to arrive in large steps rather than small ones, and he expects world models to matter well beyond games. Roblox’s approach is worth watching because it does not wait for a video model to solve everything on its own. It gives the model the support it has always lacked: an engine that remembers the world, enforces its rules, and keeps it true for everyone playing.

As AI agents take on decision-making, they need awareness of real-world conditions.

A weather delay can reroute deliveries. Lightning can pause field operations. Road conditions can affect autonomous vehicles. These signals shape decisions, yet most AI systems can’t observe them.

But context is only as valuable as the data behind it. Enterprise applications need trusted, real-time weather intelligence—not just generic forecasts. Signals like lightning activity, road conditions, hail risk, and weather impacts often matter more than temperature alone.

Xweather’s MCP-ready weather API gives AI agents access to trusted weather intelligence and the real-world context behind weather-driven decisions. Learn how it works in this technical guide.

Get your free API key

This week’s system design refresher:

• MCP vs A2A vs ACP: How AI Agents Actually Talk to Each Other

• 8 Frontier Open Models I’m Most Excited About

• LLM vs RAG vs Agent evals

• How Distributed Tracing Works at the High Level?

• The Life of a Redis Query

MCP vs A2A vs ACP: How AI Agents Actually Talk to Each Other

Agents are capable on their own. Combined with tools and other agents, their capabilities compound. But how should they communicate?

1. MCP: agent to tool communication.
   The host app receives the user request, its embedded MCP client formats and routes it to the right MCP server, the server executes the tool call and returns a structured response. The agent uses the result to continue reasoning.

2. A2A: agent to agent communication.
   An agent that can't complete a task alone discovers a capable peer via its Agent Card (published at a well-known URL), delegates the task, and receives a structured result back. If the second agent needs more input mid-task, it pauses in the input-required state and loops back to the first.

3. ACP: agent to agent communication over REST (merged into A2A).
   ACP took a REST-first approach. Peers were discovered through an Agent Manifest, called directly over HTTP, and responded to sync for low-latency tasks or via async SSE stream.

In production, MCP and A2A are complementary. MCP handles tool access, A2A handles agent communication.

Are you running MCP and A2A together in your agent stack?

Introducing Attio: the agentic CRM. (Sponsored)

Attio, the agentic CRM, makes it incredibly easy for anyone to run workflows for any GTM play they need.

Describe what you want, and Attio builds it. I just built a workflow that runs every morning, surfaces the deals that need my attention today, like anything with a stage change or a new signal in the last 24 hours.

Hundreds of thousands of automations already run on Attio every day. Ready to try now?

Try it today

LLM vs RAG vs Agent evals

LLMs, RAG pipelines, and agents are different systems, but the recipe for evaluating them is the same: pick a task, collect eval data, develop a grader.

The diagram below breaks it down across four popular AI systems.

1. LLMs: Input is a prompt, output is text. Tasks focus on what a raw model can do on its own: safety, code, instruction-following. Grader is often LLM-as-judge on the final answer.

2. RAG: Adds a retriever before the LLM. So you grade two things: retrieval (right docs?) and generation (faithful answer?).

3. Coding Agents: The LLM gets tools and a loop. Tasks become end-to-end like bug fixing and long-horizon planning. Grading is mostly code-based: run unit tests on the final patch.

4. Multi-Agent Systems: Multiple agents coordinating through an orchestrator. Tasks shift to coordination and role adherence. Grading blends code tests, LLM-as-judge, and human review.

Every new component in the pipeline is a new place for things to go wrong, and a new thing your evals need to catch.

Over to you: What's your go-to evaluation metric for multi-agent systems?

8 Frontier Open Models I’m Most Excited About

• Inkling (Thinking Machines): released this week, now the strongest American open model. Text, image, and audio input.

• Nemotron 3 Ultra (NVIDIA): a solid choice for long-running agents. The Mamba hybrid keeps long-context inference cheap.

• GLM-5.2 (Z.ai): currently the best open model for coding.

• Kimi K2.6 (Moonshot): strong on long agent tasks. Holds up over hundreds of tool calls.

• DeepSeek-V4 Pro: a very cheap way to get frontier-level quality over an API.

• Qwen3.6-35B (Alibaba): the best model to run on your own machine. A single 24 GB GPU is enough.

• Gemma 4 31B (Google): the best choice for on-device multimodal. Takes image and audio input on a gaming GPU.

• MiniMax M3: the only open model with native video input.

Over to you: Which open model are you most excited about?

How Distributed Tracing Works at the High Level?

1. Services generate telemetry data (traces, logs, metrics) as they handle requests.

2. The OpenTelemetry Collector receives this data from all services in a unified format.

3. The collector splits the data into three streams: traces, logs, and metrics.

4. Each stream is sent to a Receive & Process unit that prepares it for storage and analysis.

5. Processed data is stored in a Log Database for querying and long-term access.

6. Data from the database is visualized through a Visualization dashboard for monitoring and debugging.

Over to you: What else will you add to better understand distributed tracing?

The Life of a Redis Query

Redis is an in-memory database, which means all data lives in RAM for speed. However, if the server crashes or restarts, data could be lost.

To solve this problem, Redis provides two persistence mechanisms to write data to disk:

1. AOF (Append-Only File)

   When a client sends a command, Redis first executes it in memory (RAM). After that, Redis logs the command by appending it to an AOF file on disk. This ensures every operation can be replayed later to rebuild the dataset. Since the command is executed first and logged afterward, writes are non-blocking. The recovery process uses the event log to replay the recorded commands.

2. RDB (Redis Database)

   Instead of writing every command, Redis can periodically take snapshots of the entire dataset.

   The main thread forks a subprocess (bgsave) that shares all the in-memory data of the main thread. The bgsave subprocess reads the data from the main thread and writes it to the RDB file.

   Redis uses copy-on-write. When the main thread modifies data, a copy of the data is created, and the process works on that so that writes don’t get blocked. The snapshot is then written as an RDB file on disk, allowing Redis to quickly reload the snapshot into memory when needed.

3. Mixed Approach

   In production, Redis often uses both AOF and RDB. RDB provides fast reloads with compact snapshots. AOF guarantees durability by recording every operation since the last snapshot.

Over to you: Have you used Redis in your project?

Should each customer get a separate copy of the system, or should they all share one?

Running a dedicated copy per customer is easy to reason about, but the cost of maintaining hundreds of separate copies increases with every new customer. On the other hand, sharing a single system across all customers is far cheaper. This is why most of the software we use every day is built to be shared rather than duplicated. Such systems are known as multi-tenant systems, where each customer is a tenant that shares resources with other tenants.

However, sharing is also where the difficulty begins.

Once many customers rely on the same database, the same servers, and the same background jobs, their fortunes become linked. For example, one customer running a heavy report can slow the service for everyone alongside them. A single faulty deployment can impact all of them at once instead of just the customer. And the most serious risk is around data leaks, where one customer is shown data that belongs to another. All of these situations have to be actively handled by a multi-tenant system.

In this article, we will understand multi-tenant architecture from the basics, along with its various benefits and challenges. Here’s a quick overview of what we will cover:

• What a tenant really is.

• Where each customer’s data can physically live, from one shared set of tables to a database of its own, and what each of those options costs.

• Why the same choice between isolating and sharing also includes the compute layer and beyond, not only in the database.

• The noisy neighbor problem, and why sharing can impact fairness and force a team to rebuild it with quotas and limits.

• The role of blast radius in such an architecture.

• The concept of tenant context that runs through every part of the system.

Tenants and Users

Read more

Matic is the first visually intelligent robot vacuum that sees your home like you do, so it can clean how you want it to.
Matic’s hero features:

• Runs entirely on cameras to deftly navigate obstacles

• Recognizes floor types to auto-switch between vacuuming and mopping

• Big wheels and a height-adjustable cleaning head handle thick rugs

• Quieter than conversations at 55 dB

• Handles pet hair without clogging or tangling

• A single bag collects dry and wet waste—diaper-salts gel dirty water, antimicrobial powder prevents mold

• A fresh HEPA filter in every bag for cleaner air, no washing or replacing

Get your Matic now

Travel platforms are increasingly using AI for customer support. This means an increase in automated answers with less human intervention.

However, the cases handled automatically are largely lookups and routine changes that map cleanly onto a structured workflow. The cases that remain are typically disputes, such as a host and a guest having different views about a cancellation. In such cases, the travel platform is positioned between them and is accountable for the funds. The platform needs to figure out how to deal with these cases. It is not just a business problem, but also an engineering problem.

There is an important question at the center of this problem. At what moment should the system stop and route a case to a human agent?

A platform fielding hundreds of millions of contacts a year faces that decision millions of times a day, across cancellations, refunds, and after-hours lockouts. A confident but wrong answer can ruin a traveler’s trip and the money at stake.

Airbnb, Booking[dot]com, and Expedia have each tried to solve this problem in different ways. Airbnb prioritized autonomous resolution, building models that settle cancellations and refunds before an agent enters the conversation. Booking concentrated on the handoff, routing complex cases to a representative briefed in advance while equipping hosts to answer guests directly. Expedia emphasized deflection at scale, investing in summaries that carry context across more than thirty languages so an escalated case arrives intact.

In this article, we will look more closely at the different solutions by following the support pipeline from first principles, show why a tail of cases resists automation regardless of model quality, and use these three approaches to understand how these can be handled.

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

The Pipeline

Start with the simplest version of automated support: a bot that matches a traveler’s message to a list of frequently asked questions and returns the closest article.

It works for questions like “What is your cancellation policy?”

However, it fails the moment a traveler writes something like “My host agreed to refund me, so what happens if I cancel now?” That sentence carries an intent, a history, and a financial consequence, and a simple keyword match treats it as a search query.

The first real component is intent detection. In plain terms, the system classifies the message and identifies what the traveler wants: a rebooking, a refund, or a question about an existing reservation.

Airbnb frames this as a layered problem:

• A top-level model performs domain classification, sorting the message into a broad category such as cancellation or a help-article question.

• Once the category is settled, a second layer of domain-specific models takes over to work out the details.

For a cancellation, those second-layer models answer focused questions. Did the host or the guest initiate it? Have both sides agreed on a refund amount?

Alongside this, a separate model predicts the expected refund ratio (such as the share of the payment a human agent would typically return given the circumstances of the trip). That predictor is trained on years of past agent decisions, which lets the system produce decisions in line with how experienced staff have handled similar cases.

However, a message understood is still inert until two further capabilities exist:

• The system tracks state, meaning it carries the context of earlier messages in the thread, so a follow-up reply makes sense against what came before.

• And it has an action layer, the part wired into live systems that can issue the refund, move the reservation, or open the mutual cancellation flow.

Reading intent is the easier half. Taking a correct, reversible action against live booking and payment systems is what requires greater consideration.

The final element is a confidence threshold. Each prediction arrives with a score, and the threshold determines whether the system proceeds on its own or routes the case to a person. If this bar is set too low, more cases clear automatically while errors creep upward. If it is raised, the accuracy improves, but more traffic lands on human agents. That single threshold value is where much of the tuning happens

This pipeline, intent detection, state tracking, an action layer, and a confidence threshold, handles a large volume of everyday requests well. However, it keeps facing problems in one family of cases.

Adjudication

The cases that clear automatically share a quality. They are answerable by retrieval. In other words, the system finds a fact or applies a rule, such as the status of a refund or the steps to change a date. Volume here is high, and the work is repetitive. These are the types of problems where automation works well.

The cases that resist automation differ in kind.

They require adjudication, meaning a judgment between parties whose accounts and interests diverge. A guest says the apartment was misrepresented. The host says the listing was accurate and the guest simply changed their mind. A damage claim sits between them, and the platform holds the deposit.

There are three parties, three versions, and one decision about money. A better model produces a cleaner summary of the dispute, yet the underlying call still rests on weighing competing claims, which is difficult for even a powerful model.

Time pressure sharpens the divide.

When flights were cancelled across the Middle East, and a surge of travelers contacted Expedia at once, automation absorbed the routine rebookings and status checks at speed, which freed human agents to spend their attention on the tangled, time-sensitive cases. The same event produced both kinds of work in the same minutes, and the value of the split was that each kind reached the right handler.

This is why the resolution rate climbs and then flattens. The boundary separating automated from escalated cases advances as retrieval improves, and it approaches a region of adjudication-heavy cases that stay with people by design.

It also explains why headline percentages deserve careful consideration.

Airbnb reports that more than 40% of guest issues are resolved without an agent. Expedia reports that more than 30% of its self-service interactions are powered by AI, where self-service already accounts for more than half of all contacts.

Those two figures rest on different bases, so a side-by-side ranking would mislead more than it informs. Each number describes its own pipeline rather than a shared scoreboard.

Handoff

When a case crosses to a person, the details of what is shared become important.

A handoff that passes along only “escalated, please assist” forces the customer support agent to restart the conversation, and the traveler retells the whole story to a second responder. A traveler who repeats every detail to a human after a stalled bot session ends up more frustrated than one who reached a person at the start.

This asymmetry is the reason handoff deserves real engineering attention. A weak handoff can leave the combined system performing worse than plain human support would.

The solution to this lies in the payload: the bundle of context that accompanies an escalated case. A strong payload carries four elements.

• A summary of the conversation so far.

• The structured facts already gathered include the booking reference and the cancellation reason.

• The live state of the reservation.

• A translation, where the agent and the traveler use different languages.

The agent opens the case already informed and continues from where the automation left off.

Expedia leans on this idea at scale. Its systems generate conversation summaries across more than thirty languages, so a case handled partly in Portuguese can reach an English-speaking support agent with the thread rendered in a language they read fluently. The same summaries shortened the time required to bring new agents up to speed by a substantial margin, because a well-formed summary teaches as it informs.

Booking describes the same priority from the other side. Its tools route complex questions to a representative who receives the relevant booking and property details in advance, so the human starts the conversation already holding the facts.

Booking also adds a second component that the others treat as secondary. Much of its support volume is communication between a guest and an accommodation partner, so it built tooling that drafts partner replies from property and reservation data and suggests them to the host. Here, the friction is the message between two people rather than a decision the platform owes, and equipping the partner to answer quickly relieves pressure before a formal support case ever opens.

The pattern across all approaches is consistent.

The success path contributes to the headline percentage, while the failure path, the moment a case escalates, is where the overall customer experience is decided. The core work for building such systems concentrates on handoff far more than on the straightforward case where everything resolves on the first attempt.

Divergence

The design decisions made by the three companies indirectly express three beliefs about which part of travel support is hardest.

Airbnb’s position is that the adjudication itself can be modeled. By training on the expected refund ratio and on the questions agents ask during a cancellation, it pushes the automated portion as far toward real decisions as the data supports, and it reserves human attention for safety, discrimination, and high-value claims. The wager is that a large share of disputes follow learnable patterns, and that an autonomous system applying a consistent standard serves travelers better than a queue of agents who each interpret a policy slightly differently. The cost of this position is a heavy, continuous investment in models and labeled data, and a high bar for caution wherever money and safety meet.

Booking solves the problem differently. Its emphasis on briefing the human agent and on drafting replies for partners treats communication as the primary friction. The implied belief is that a great deal of support volume is people struggling to reach each other, a guest with a question and a host with the answer, and that closing that gap quickly matters more than resolving every case autonomously. This keeps more humans in the loop while raising the quality of each human touch.

Expedia’s emphasis on deflection and multilingual summaries reflects a third belief that scale is the dominant variable. With well over two hundred million interactions a year, even modest gains in self-service rates and handling times compound into large absolute numbers, so the architecture optimizes for volume routed correctly and for context that survives a language boundary across a global workforce.

Tradeoffs

Every placement of the boundary carries a price on each side, and the confidence threshold makes the trade concrete.

• Raise the threshold, and the system escalates more often, which guards against autonomous errors while increasing the load on human agents and the cost that comes with it.

• Lower the threshold, and the system resolves more cases on its own, which trims cost while admitting more mistakes in situations that involve money.

There is an operating point that balances the two, and choosing it requires judgment about how expensive an error is relative to an escalation. In travel, where an error can mean a wrongly denied refund on someone’s holiday, that judgment leans conservative wherever the stakes run high.

There is also a more fundamental limit worth considering.

The chat interface itself suits travel poorly, because a chat thread is built for one person, while a travel dispute often involves several. A cancellation between a host and a guest, mediated by a platform, is a conversation among three parties, and a linear thread struggles to hold that arrangement sometimes.

It is not simply a matter of improving the interface. It suggests that the format through which support arrives may bound how far automation can go, separate from how capable the model becomes.

For that reason, regulated cases, high-value claims, and anything touching safety remain with people on purpose. The boundary moves rightward over time, and a margin is kept on the far side, where human judgment is meant to outweigh a confident automated answer.

Conclusion

The percentage of issues resolved without a human is an indication of two things: where a platform places its automate-versus-escalate boundary and how much of its work is answerable by retrieval rather than judgment. The model matters, yet it operates inside a structure built to manage that boundary.

A few points hold across all three platforms.

• The pipeline serves the boundary. Intent detection, state tracking, the action layer, and the confidence threshold all feed the single decision to resolve a case or escalate it.

• The handoff decides the experience. A case that crosses to a person succeeds or fails on the context that travels with it, which is why summaries, structured facts, and reservations carry so much weight.

• The placement encodes a belief. Airbnb bet on autonomous adjudication, Booking on communication and a briefed handoff, and Expedia on correctly routed scale, with each placement reflecting a different view of where travel support is hardest.

Put together, the resolution rate reflects a design decision as much as a level of capability, and the boundary is the part of the system worth reading first.

References:

• Task-Oriented Conversational AI in Airbnb Customer Support.

• Airbnb, Inc., Q1 2026 shareholder letter and earnings call.

• Booking.com Debuts Agentic AI Innovations.

• Expedia Group, Q1 2026 financial results and earnings call.

• Expedia Group Sets the Standard with AI-Powered Service Agent.

• AI solutions.

Enhance visibility into your cloud architecture with expert insights from AWS + Datadog. In this ebook, AWS Solutions Architects Jason Mimick and James Wenzel guide you through best practices for creating professional and impactful diagrams.

Get the eBook

Ask a model the same question twice, phrased two different ways, and watch how it decides what to give you. It might offer a terse one-liner or a careful walk-through. It might sound fully confident, or it might hedge. Somewhere in training, the model learned which version you would rather have, even though both answers are correct.

That points to a harder question. How do you teach a model to be helpful when helpfulness changes with every request?

For a quick factual query, helpful means brevity. For a debugging problem, it means thoroughness. For a medical worry, it means caution paired with clarity. Each case requires a different direction with underlying trade-offs.

The model learned to handle those trade-offs from people comparing answers thousands of times over. This idea of learning from comparison rather than from a fixed answer key sits beneath every popular model you have used.

In this article, we will look at how that learning actually happens, starting with why instruction-following alone falls short, then walking through the two main methods for teaching preferences (RLHF and DPO).

Pipeline

A useful model gets built in three stages, as shown in the diagram below:

Let’s look at each stage in a bit more detail:

• The first stage is pretraining. The model reads an enormous amount of text and learns to predict the next word. In other words, it builds a broad sense of how language works and soaks up world knowledge along the way. What it gains is raw capability. What it still lacks is direction, because predicting the next word and answering your question are two different goals.

• The second stage is supervised fine-tuning, or SFT. We show the model thousands of example pairs, each an instruction together with a good response, and train it to imitate them. After this stage, the model follows instructions instead of continuing text. For example, if we ask a model to summarize a paragraph, it writes a proper summary. A raw model without SFT would have rambled on.

• The third stage teaches preferences, which will be the focus of the rest of this article.

Why does a model that already follows instructions need a third stage at all?

This is because SFT works through imitation. The model sees a correct answer and learns to copy its shape, which holds up well when a question has one good answer, but runs into problems when a question has many possible answers.

Consider a request like “explain how a hash map works”. A short answer and a long answer can both be excellent, and which one helps depends on who is asking. An example dataset can show one of those answers, while the trade-off between them stays invisible. Imitation teaches the model what a good answer looks like, but struggles to teach it how to weigh two good answers against each other.

The payoff is larger than it might appear at first. For example, when OpenAI built InstructGPT in 2022, human raters preferred answers from a 1.3 billion parameter aligned model over answers from the 175 billion parameter GPT-3, a model roughly a hundred times larger. The alignment stage mattered more than a hundredfold jump in size.

Nevertheless, the line between pretraining and SFT is cleaner in the example than in practice. Some recipes blend the two, but the three-stage view is the right place to start.

Next up, we need a signal that captures the trade-off directly. That signal is comparison.

Introducing Attio: the agentic CRM. (Sponsored)

Attio, the agentic CRM, makes it incredibly easy for anyone to run workflows for any GTM play they need.

Describe what you want, and Attio builds it. I just built a workflow that runs every morning, surfaces the deals that need my attention today, like anything with a stage change or a new signal in the last 24 hours.

Hundreds of thousands of automations already run on Attio every day. Ready to try now?

Try it today

Preferences

Comparison turns a judgment call into something a model can learn.

A preference example is simple. We take one prompt, generate two responses, and ask a person which is better. The result is a small piece of data, a prompt with a winner and a loser. If we collect enough of these, a pattern emerges that captures the trade-offs accurately.

See the diagram below:

The reason this works comes down to what people are good at. Writing the single ideal response to a tricky prompt is hard, and two skilled people would produce different versions. Judging which of two responses is better is far easier, and people agree with each other more often when they compare than when they create. This is why labeling teams compare outputs instead of writing perfect ones from scratch, which makes the data trustworthy enough to train on.

This is the same comparison data InstructGPT collected, and the starting point for both methods we are about to compare. They differ in what they do with it, while the comparisons stay the same.

To clarify, the comparisons can come from another AI model in place of a person, an approach used in RLAIF and Constitutional AI. The overall mechanics stay the same.

Let us now look at the two different methods.

RLHF

RLHF, short for Reinforcement Learning from Human Feedback, learns from comparisons by training a separate scorer and then coaching the model to score well against it.

See the below diagram that shows the basic RLHF flow:

It works in two moves once the comparisons are collected.

• First, we train a reward model. We take those prompt-with-a-winner examples and train a separate model to score any response, high for the kind people preferred and lower for the kind they passed over. The reward model turns scattered human judgments into a single function that grades fresh responses on its own.

• Second, we use that scorer to improve the main model, which here we call the policy. The policy writes a response, the reward model scores it, and a reinforcement learning algorithm called PPO nudges the policy toward higher-scoring responses.

Reinforcement learning, in plain terms, means learning from a reward signal through trial and error in place of labeled examples. Run the loop many times, and the policy drifts toward the answers people prefer.

Left to chase a score freely, a model will discover strange, degenerate text that games the scorer while reading badly. To guard against that, we keep a frozen copy of the starting model, the reference, and penalize the policy whenever it strays too far from it. This penalty, the KL term, lets the policy improve while staying close enough to fluent language to stay useful.

The reward model, the policy, the frozen reference, and a value model that PPO uses internally add up to four models in play during training.

That is the source of RLHF’s reputation for being expensive, fiddly, and slow to get right. However, this is also the pipeline behind the first version of ChatGPT, and it produced the first generation of genuinely helpful assistants at considerable cost.

Running a separate reward model and a full RL loop is a lot of moving parts. In 2023, a study from Stanford showed that much of it can collapse into a single step.

DPO

DPO reaches the same goal with the same data, replacing the reward model and the RL loop with one training step.

DPO stands for Direct Preference Optimization, and the name fits. Rather than train a scorer and then optimize against it, the method tunes the model directly on the comparison pairs.

The mechanics sit close to ordinary supervised learning.

For each comparison, DPO adjusts the model to raise the probability of the preferred response and lower the probability of the rejected one, measured against that same frozen reference from before. The whole thing becomes a single loss that would train the way you would train a classifier. This is far simpler and steadier than an RL loop.

A common idea is that DPO removes the reward model, but what actually happens is subtler. The Stanford paper that introduced DPO carries the idea in its title that a language model is secretly a reward model. The reward signal is still present, but it now lives inside the policy itself rather than in a separate network.

The approach proved itself quickly.

Zephyr, a 7 billion parameter model trained with DPO on machine-generated comparisons, beat Llama 2 Chat 70B, the strongest open RLHF model at the time, on a standard chat benchmark. Since then, DPO has become a staple in the industry.

By folding the reward into a single training step, DPO put alignment within reach of a small team on a modest budget, far from any large research cluster. A family of variants, such as SimPO, KTO, and ORPO, adjusts the objective in different ways. They share DPO’s core approach.

On cost and simplicity, DPO looks like a clear win. The catch is one it shares with RLHF, and it sits in the signal both methods trust.

Reward Hacking

Both methods rely on a signal that only approximates human judgment, and leaning too hard on it can also backfire at times.

The reward model, whether trained explicitly in RLHF or folded into the policy in DPO, is a proxy. It approximates what people prefer. Like any approximation, it also has gaps. Optimize the model hard against that proxy, and it can start to exploit those gaps. In other words, the score climbs while the answers get worse.

Researchers at OpenAI measured this directly.

As the model optimizes against the proxy, true quality rises for a while, then peaks and slides backward even as the proxy score keeps climbing. They tied it to Goodhart’s law, the old observation that a measure stops being a good measure once it becomes the target.

The everyday symptoms of this are familiar.

Models pad answers with length because longer responses tend to score well, and they learn to agree with whatever view you express. This behavior is called sycophancy. Anthropic found that most of the time, both human raters and reward models prefer a confident, agreeable answer over a correct one, which sometimes means flattery in place of truth.

The important part is that this trouble follows the data rather than the algorithm.

DPO inherits it just as RLHF does, because both learn from the same imperfect human judgments. Simpler training buys you a great deal, and this particular problem comes along for the ride. This is why your model sometimes agrees with you when you are clearly wrong. It learned that agreement tends to score well.

The KL penalty we discussed earlier softens this by holding the model near its starting point, though it shrinks the problem rather than completely erasing it.

If the root trouble is an imperfect human signal, a natural question follows. Could we replace it with a signal that is exact? For a certain class of tasks, we can.

Verifiable Rewards

When a program can check the answer, the human reward becomes optional, and that single fact reshaped the final training stages across 2025 and 2026.

Some tasks have a checkable answer:

• A math problem is right or wrong.

• A piece of code passes its tests or fails them.

For these, we can replace the human-derived reward model with a verifier, a program that grades the result directly. The signal becomes exact and effectively free, clearing the proxy problem at its root for that work.

DeepSeek pushed this idea furthest.

They introduced an RL method called GRPO that drops one of the extra models PPO needs, and they trained their R1 reasoning model largely by rewarding correct final answers on math and code. The results matched the strongest closed reasoning models of the time. The model learned to reason by checking its own work, guided by a reward a calculator could supply.

The limit is that a verifier can grade a math proof. However, it stays silent on whether an answer is honest, kind, or appropriately cautious. Those qualities take many acceptable forms and resist any automatic test, the territory preference learning was built for. DeepSeek themselves kept reward models for the helpfulness and safety parts of R1, while verifiable rewards drove the reasoning.

This gives us the rule for the whole landscape.

When a machine can check the answer, reach for verifiable rewards. When the answer is a matter of judgment, reach for preference learning, whether RLHF or DPO. The method follows the signal.

See the diagram below:

This is why reasoning models improved so quickly, and why even the best of them still rely on preference methods to stay pleasant and safe. Newer methods, such as DAPO and RLVR, extend the verifiable-reward idea, and they make up much of the active frontier in 2026.

Conclusion

We can retrace the path in a sentence each:

• A raw model learns language and knowledge from pretraining.

• Supervised fine-tuning teaches it to follow instructions by imitation.

• A third stage teaches preferences, because the questions that matter most often have several good answers and a real trade-off between them.

Comparison is the signal that captures that trade-off, and the two main methods both run on it.

RLHF trains a separate reward model and optimizes against it with reinforcement learning, which is powerful and costly. On the other hand, DPO folds the same signal into a single training step, which is simpler and steadier while keeping the reward implicit rather than absent.

Both inherit the same catch, that a human signal is a proxy, and pushing on it too hard breeds sycophancy and quiet failures. And when a task has a checkable answer, verifiable rewards now sidestep the proxy entirely.

References:

• Training language models to follow instructions with human feedback

• Direct Preference Optimization: Your Language Model is Secretly a Reward Model

• Zephyr: Direct Distillation of LM Alignment

• Scaling Laws for Reward Model Overoptimization

• Towards Understanding Sycophancy in Language Models

• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning

Debugging SSO, managing users, adjusting auth policies, configuring branding: every configuration task has lived behind a UI that only a human can drive.

The WorkOS MCP server gives agents the same access as your dashboard login. Hundreds of operations, discoverable at runtime. Connect in one command via OAuth, with scoped tokens instead of a master API key. Pass a screenshot of your marketing site and ask your agent to match the login page. If a human had to do it before, an agent can do it now.

Connect your agent →

Microsoft operates at an enormous scale. More than 80,000 enterprises now build on Microsoft Foundry, the company’s platform for building, deploying, and running AI agents and applications. Microsoft’s own copilots run on the same platform, including Microsoft 365 Copilot, which alone serves over 20 million users and has a monthly active usage of first-party agents growing 6x year-to-date.

To understand what it actually takes to ship agents at that scale, we spoke with Marco Casalaina, VP of Products for Microsoft Core AI. He walked us through what his team has learned from running these systems in production, the engineering challenges that come with it, and where he thinks enterprise AI is headed next.

In this article, you’ll learn:

• Why a prototype agent doesn’t survive in production

• What’s in a production agent harness, and why Microsoft believes context is the key

• Two engineering ideas behind Foundry: retrieval-as-a-subagent, and giving agents their own identity and a place to act

• How Microsoft evaluates production agents with rubric-based evals and an auto-improvement loop

• Lessons for other teams, and what’s next

What Breaks When Agents Hit Production

Production agents fail for reasons that aren’t visible in a prototype. The model is rarely the problem. What breaks is everything around the model, including the data the agent retrieves, the tools it calls, the way it handles real users, and the way quality drifts as the world around it changes. Enterprises trying to ship agents this year are running into a different engineering problem than the one they were solving last year.

To see why, it helps to start with what’s actually changed about what enterprises are trying to build. Marco framed the shift this way:

We are leaving the question-answering phase of AI. In 2026, we are seeing a huge increase in the number of our customers that are using voice as a front-end, so we’re also leaving the chatbot era of AI.

The old shape was a chatbot. The user types, the agent types back, and it can only answer questions. The new shape is an agent that does meaningful work on the user’s behalf. It books the meeting, runs the analysis, sends the email, files the ticket. The user might not type at all because the front end can be voice. For example, Foundry’s Voice Live lets a team turn an existing text agent into a voice agent without rebuilding it.

This shift is what makes the engineering problem different. A chatbot returning a wrong answer is a bad experience. An agent taking the wrong action is a business incident. The bar for what’s good enough to ship has moved.

That’s where the gap between a prototype and a production agent opens up. The first prototype is easy. You can vibe-code one in an afternoon. The model is smart, your test prompts work, the demo is impressive, and the pilot ships in a week.

Production is where the cracks open. Real users ask things you didn’t anticipate. Documents the agent depends on go stale. New edge cases emerge that never appeared in your eval dataset. A model update changes the agent’s behavior subtly, and nobody notices until a customer complains. Without identity controls, the agent runs as a shared system principal and there’s no audit trail when something goes wrong. Without guardrails, it confidently says something it shouldn’t. Without observability, you can’t tell whether quality is improving or degrading. None of these problems showed up in the prototype. All of them show up in production.

When we asked Marco for the single biggest lesson the Foundry team has learned from running these systems at scale, his answer was “the harness matters as much as the model.”

The harness is everything around the model. The runtime, the tools, the context retrieval, the identity layer, the guardrails, the evaluators, the deployment pipeline. Models change constantly, and you cannot treat them like database versions. With Postgres, you change versions and you pretty much expect it to straight up work. Models are not like that. Each one has different properties that the harness has to adjust to. When Anthropic released Claude Opus 4.8, Microsoft’s GitHub Copilot CLI team had to re-tune their harness and re-run their evaluations before they could ship it.

See what 406 infrastructure leaders revealed about who’s ready for AI (Sponsored)

AI is reshaping infrastructure, but not how most teams expected. The teams getting the most from it aren’t adopting the fastest. They’re taking advantage of the platform, governance, and automated pipelines that allow AI-generated infrastructure to move safely to production.

The 2026 Infrastructure Automation Report surveyed 406 infrastructure and platform engineering leaders to reveal how Pioneers are benefiting the most from AI. You’ll learn:

• Why platform engineering is the critical component of AI adoption

• New AI-specific signals and metrics that need monitoring beyond traditional metrics

• 5 tactical steps to reach the Pioneer segment of the Al Maturity index

Download the report

What’s in a Production Agent Harness

If the harness matters as much as the model, the next question is what’s in it. Walking the harness from the bottom up shows why each layer exists and what breaks if you try to do without it.

1. Inference Layer

At the bottom is the inference layer, the single interface the harness uses to reach the models. The models themselves sit outside the harness and stay swappable. Different agents need different models, and the right one changes every few weeks. Foundry supports more than 11,000 of them, from OpenAI, Anthropic, xAI, DeepSeek, and Microsoft’s own MAI family.

2. Agent Runtime

Above the model is the agent runtime, which is what turns a model into an agent. The runtime handles the orchestration loop, the tool calls, the conversation state, and the protocol the rest of the harness speaks.

Not every step in that loop should run through the model. A well-built agent sends only the parts that need reasoning to the LLM and leaves the rest to ordinary code, since a database lookup or a purpose-built extraction model is faster, cheaper, and more reliable than asking a model to do the same work.

The number of agent frameworks keeps growing, from open-source options like LangChain, LangGraph, and CrewAI to vendor-built runtimes. The principle that matters is framework neutrality. An agent built on one framework should be portable to another without rewriting the surrounding harness. Foundry, for example, lets agents run on any framework interchangeably.

3. Observability and Governance Layer

Once agents are in production, an organization needs observability and governance. It needs a single view of every agent running across every project, with health scoring, token usage, latency metrics, drift detection, and the kind of cross-project rollups that let a platform team govern a fleet. Without this layer, regressions are invisible and cost is uncontrolled. In Microsoft’s case, this is Foundry Control Plane, which provides cross-project fleet visibility and routes agent telemetry into Azure Monitor and Application Insights, the same pipeline that already handles infrastructure alerts.

4. Identity Layer

Once agents start taking real actions inside an organization, they need their own identities. They need their own role assignments and their own audit trails, because a misbehaving agent has to be bounded by the same access controls that bound a misbehaving employee. The industry is still converging on how to do this well, and the answer most platforms are landing on is to extend an existing enterprise identity system to treat agents as a new class of principal rather than inventing a parallel system. For example, Foundry extends Entra, Microsoft’s enterprise identity platform, to treat agents this way. This is the access-control primitive; later in the article, the section on giving agents a place to act shows what an agent does once it has one of these identities.

5. Context Layer

Once an agent can run reliably and act with its own identity, the question becomes whether it can answer correctly. That’s the job of the context layer. While every other layer in the harness exists to let the agent run, context is what lets it run correctly. An agent without good context might hallucinate. It will give an answer, but the answer will be wrong in ways that are hard to detect because the agent itself doesn’t know what it doesn’t know. Marco was explicit that giving agents the context they need to actually work is one of the hardest problems his team is solving, and the one Microsoft is reasonably passionate about getting right.

The next section is about how Microsoft is building that context layer.

Building a Context Layer for Agents

The hard part of giving agents the right context isn’t that the context is missing. Enterprises have enormous amounts of it. The hard part is that the context lives everywhere. It lives in unstructured documents in SharePoint and wikis. It lives in structured tables in OneLake and data warehouses. It lives in productivity apps like Outlook, Teams, and Word. No single retrieval method can reach all of it.

The standard answer of the last two years, classic retrieval-augmented generation (RAG), was never designed to. Classic RAG is a one-shot pattern. You take the user’s question, embed it, search a single index, return the top-k results, and pass them to the model. It works for simple questions against a small, clean corpus. It breaks the moment the question is ambiguous, the corpus is heterogeneous, the right answer requires combining sources, or the first retrieval comes back empty. The agent has no way to recover from a bad retrieval, and as Marco put it, when RAG isn’t working well, the whole agent isn’t working well. One-shot is the wrong shape for the problem.

The fix is to treat retrieval as something the system can iterate on, the same way an agent iterates on a task.

Plan the query, try a source, evaluate the result, try a different source if the first one came back empty, and combine what you find. That’s the central engineering idea behind a production context layer.

Microsoft’s answer is to ship the context layer itself as a set of services. There are four of them, collectively called Microsoft IQ. Foundry IQ handles unstructured data, Fabric IQ handles structured data, Web IQ handles real-time web retrieval, and Work IQ handles the productivity surface of Microsoft 365, including email, calendar, documents, and Teams.

Each IQ is a headless service that agents call through MCP. Two engineering ideas run through them. The first, retrieval-as-a-subagent, is the pattern behind all four. The second, giving an agent an identity and a place to act, is specifically what Work IQ adds on top of retrieval so that an agent can do work, not just find information.

Retrieval-as-a-Subagent

The technical idea behind a production context layer is to wrap retrieval in an agentic loop. Instead of a single function call that runs one query against one index and returns the top-k, retrieval becomes a small agent of its own. It plans which sources to query, executes the queries, evaluates the results against the original question, and decides whether to return them, refine the query, or try a different source. Retrieval stops being a lookup and becomes an iterative process that can recover from a bad first pass.

Foundry IQ is the cleanest example of this in production today. The figure below shows the workflow when an agent calls it.

The last step is the one most worth paying attention to. When iteration runs out, Foundry IQ returns a structured “I don’t know” instead of forcing an answer. Classic RAG has no fallback and the model hallucinates something plausible. Foundry IQ gives the calling agent a clear signal that retrieval failed, which it can act on, rather than a confident wrong answer that’s hard to detect.

So far this is all about data. The same loop applies to tools. An agent with a few tools can list them all in its prompt. One with dozens can’t. Listing them all burns context on every call and slows the model down as it scans the list. The fix is similar to agentic retrieval. Instead of exposing every tool, the agent searches for the right one, gets it back, and calls it. Foundry exposes this as tool search, and it’s the pattern OpenAI’s and Anthropic’s agents have converged on too. So the same retrieval loop that pulls in knowledge also pulls in capability. The agent looks up what it needs at the moment it needs it instead of carrying everything up front.

Other IQs rely on the same pattern too. Fabric IQ runs agentic retrieval over structured data, with the loop planning queries against OneLake tables and the Fabric data agent rather than against text indexes. Web IQ runs the same loop over the open web at sub-second latency. The retrieval-as-subagent idea is the architectural commonality across all of them.

An Identity and a Place to Act

Retrieval gets the agent the right information. But information alone doesn’t finish a task. An agent that knows the customer is unhappy still has to write the apology email and refund the order. To do that responsibly, the agent needs two things. An identity the organization can see, and a surface where it can take action.

The identity is the access-control primitive from the identity layer earlier. Without it, every action is anonymous or on behalf of a user who lent the agent their identity. Logs show that “the AI did it” and the audit trail collapses, which is unworkable for any regulated enterprise. The fix is to treat agents as their own class of principal, sitting in the same directory as employees, with their own role assignments and their own audit trails. A misbehaving agent gets bounded by the same access controls that bound a misbehaving employee.

The action surface is what makes the identity meaningful. An agent that exists in the directory but has no way to send an email or update a document isn’t doing work. The surface has to expose the same actions a person already takes inside the organization, so that an agent operating on top of it can read inboxes, send messages, schedule meetings, and edit shared documents. In Microsoft’s case, identity is handled through Entra and the action surface is Work IQ. An agent can now have a named directory entry, report to a manager in the org chart, and own its mailbox.

Identity bounds what an agent is allowed to reach. Guardrails bound what flows through it once it acts. A chatbot only had to screen the user’s prompt and the model’s reply. An agent has more surface to defend, because it also reads tool outputs and retrieved documents, and any of those can carry an instruction the user never typed. That is how indirect prompt injection works. A line like “ignore your previous instructions” sits hidden inside a document the agent ingests, and the agent treats it as a command. The fix is to move guardrails to the tool boundary, screening tool inputs and tool outputs rather than just the model’s input and output. In Microsoft’s case, Foundry runs its own classifiers at the tool-call and tool-response level, on top of whatever safety the model already has. Since these controls live on a shared tool layer rather than inside each agent, a team configures them once and every agent that uses those tools inherits them, instead of re-implementing the same guardrails, credentials, and policies agent by agent.

The takeaway for any team building this is that retrieval, action, and the guardrails around both need to be designed as first-class layers. Retrieval should be a loop that recovers from bad lookups, action should run under an identity the organization recognizes with an audit trail to match, and guardrails should sit at the tool boundary rather than only at the model’s edges.

Evaluate Production Agents

Context is half of what determines whether an agent survives production. Evaluation is the other half. An agent with the right context can still drift, regress, or fail in new ways as traffic patterns change. Evaluation is what closes the loop. It’s the system that tells you when something has changed, and the system that tells you whether the agent is doing what it’s supposed to do.

Continuous Evaluation

Most teams treat evaluation as a pre-ship checkbox. They run a test suite once, see green, and ship. That works for traditional software because traditional software is deterministic. An agent is not. The same prompt against the same model can produce different responses, the model itself changes when the provider releases an update, and the data the agent retrieves from changes daily. A pre-ship test suite catches a snapshot of behavior, not a guarantee of it.

Continuous evaluation closes that gap by running against live traffic, sampling real user interactions, scoring them on whatever criteria the team has defined, and surfacing results into the same observability pipeline that already handles infrastructure incidents. In Foundry, this is built in. When quality regresses, the team finds out from the same alerting system that pages them when a service goes down. The same evaluations can also run earlier, as a gate in the deployment pipeline, catching a regression before it ships.

Rubric-Based Evaluation

Continuous evaluation tells you when something has changed. The harder problem is knowing whether what the agent is doing is right in the first place. Most evaluation systems today rely on generic metrics like groundedness, coherence, and task completion. The metrics are useful but they have a ceiling. As Marco put it, generic metrics tell you whether the agent works. They don’t tell you whether the agent works right.

Consider an agent that helps users book restaurant reservations. It is the kind of wrong-action failure from the opening, seen up close. A generic metric can tell you that the agent called the booking tool successfully and returned a reservation. But it can’t tell you whether the agent did the right things along the way. Did it ask the user what time they wanted when the user only said “a table for two tomorrow”? Did it check the booking system to confirm the table was actually available before claiming a 6pm slot? A reservation that gets created without those checks is technically a success against generic metrics and a failure in the user’s eyes.

The way to close that gap is to evaluate the agent against the specific behaviors it should be showing, not just generic metrics. The common approach is rubric-based evaluation. A rubric is a specific, use-case-tied check that asks a yes-or-no question about something the agent should be doing. For the restaurant agent, the rubrics might look like:

• When the user gives a partial request, does the agent ask for the missing information?

• Before claiming a reservation is available, does the agent verify availability against the booking system?

• After making a booking, does the agent confirm the details back to the user?

In practice, the eval system runs the agent against a set of test interactions, scores each rubric per interaction, and rolls the results up. The team sees not just whether the agent works but which specific behaviors it’s getting right and which it isn’t. Rubrics work better than generic metrics because the team that owns the agent writes them, so they reflect what the agent is actually supposed to do.

Writing them by hand isn’t the only path. Foundry can draft a rubric by reading the agent’s configuration and its production traces, proposing the dimensions worth scoring from how the agent actually behaves. The team still owns and edits the result, but it starts from real traffic instead of a blank page.

Microsoft has built rubric-based evaluation into its Agent Optimizer suite. The Optimizer handles the rubric side of the loop and also makes the rubrics actionable by automatically improving the agent itself when a rubric fails. It can rewrite the system prompt to make the missing behavior more explicit, adjust how the agent uses its tools, or change which sources the agent prioritizes.

What it tunes goes beyond the prompt. It can also swap the underlying model and tune the agent’s skills. Rather than testing one fix at a time, it produces several candidates in parallel, scores each against the rubrics, and promotes the best one as a new agent version. The same loop that measures the agent now improves it. It becomes a self-improving loop.

For any team running agents in production, evaluation isn’t a phase that ends. It’s a layer that runs alongside the agent, with rubrics that evolve as the agent’s job evolves and an optimizer that turns regressions into actionable changes instead of manual fixes. The team’s job stops being to write the perfect agent on day one and starts being to keep the agent honest as the world around it changes. That’s what production AI actually looks like, and it’s the second half of the harness Foundry is built around.

The Key Lesson for Other Teams

The harness mattering as much as the model is the thread that ran through the whole conversation. The reason context is hard, the reason evaluation is hard, the reason a working prototype doesn’t survive production, is that the work most teams treat as “around the model” is actually what determines whether the agent works. Treating the harness as something you add later is the most common reason an enterprise agent never makes it to production.

The practical version of that lesson for a team building its own context layer is to take the two architectural ideas from earlier in this article seriously, even if you don’t use Foundry. Retrieval should be agentic. Calls to your retrieval layer should hide a planner-driven, multi-source, retry-capable subagent that can recover from a bad lookup instead of breaking on one. And agents that take actions should have identities the organization recognizes, with audit trails that survive a compliance review. Both of these are buildable on top of any platform.

What’s Next

Marco thinks about the future of AI by watching what’s about to become possible in the next year, not what’s ten years away. He tracks the moment a capability stops being a research demo or a developer tool and becomes something an ordinary person can use. The pattern he watches most closely is the one where capabilities that used to require a CLI coding agent become available to everyone.

This pattern has been playing out for the last eight months. Capabilities that used to require a developer to wire up inside a coding agent, things like writing a document from natural language, managing a calendar, or running a custom workflow, are now showing up in tools. Skills, which started as a coding-agent concept, are now in Excel for tasks like financial analysis and private equity workflows. Marco’s expectation is that this pattern keeps accelerating, with more developer-only capabilities crossing into general use over the next year.

The bigger shift he sees coming is toward agents that improve themselves. The architectural pieces are global and local memory, plus skills, plus the kind of behaviors that let an agent retain what it learns about a user and apply it automatically. Copilot CLI’s new Chronicle feature is an early version. If Marco always commits his Git repos to GitHub immediately after creating them, Chronicle learns that and starts doing it for him without being asked. Marco’s bet is that this combination, applied to the kinds of agents enterprises are already deploying, is the story of the next year. Not science fiction. Just the next set of capabilities crossing into general use.

Define tasks with Render’s lightweight SDK and chain them into long-running, distributed workflows. Launch your agents and batch jobs on demand. Render Workflows handles queuing, orchestration, and retries.

Get $50 credits with code BYTE

This week’s system design refresher:

• How Docker Works Under the Hood

• git merge vs git rebase

• 12 popular vector databases help you get the right context to the model

• Pagination Strategies for Large Systems

• How LLMs Use AI Agents with Deep Research

How Docker Works Under the Hood

A Docker container starts with a single command, but that command has to be turned into a running Linux process. Here is what actually happens.

The Docker CLI takes your command and sends it as an API call to the Docker daemon (dockerd) running on the host.

dockerd checks whether the nginx image is already on disk. If it is not, it pulls it from a registry like Docker Hub or ECR. Then it prepares the container config.

dockerd does not start the container directly. It passes the request to containerd, which manages the container lifecycle. containerd prepares the runtime files and assembles a bundle made of the OCI config and the root filesystem.

containerd then calls runc. runc reads the bundle, creates the Linux namespaces and mounts defined in the config, and starts the process inside them. Once the process is running, runc exits.

The running container is a regular Linux process with its own PID, network, and mount namespaces. Its filesystem is a stack of read-only image layers with a writable layer on top, so changes inside the container do not modify the image.

Isolation comes from kernel features. Namespaces separate processes, cgroups limit CPU and memory, and network namespaces give the container its own interfaces. There is no guest OS and no hypervisor.

Over to you: where do most of your container issues show up, the image, networking, or resource limits?

git merge vs git rebase

Both commands get your changes into main. The difference is what they do to history.

git merge preserves the original branch structure. If main and feature branches have diverged, Git creates a merge commit and keeps both lines of development . That makes it easy to see where work branched off, what got merged, and when it came back together.

git rebase takes a different approach. It reapplies your feature commits onto the latest main branch as new commits. The result is a linear history. But those rebased commits get new IDs, so rebasing a shared branch usually means force-pushing and making others sync to rewritten history.

That is why teams often use merge on shared branches. It keeps existing commit IDs unchanged, so everyone can pull without dealing with conflicts. Rebase is more useful for cleaning up your own branch before you merge it.

Over to you: which command do you prefer?

12 popular vector databases help you get the right context to the model

1. Pinecone: Fully managed, serverless with hybrid search (dense + sparse) native. Best when your team wants zero infrastructure overhead.

2. Weaviate: Built-in vectorization and native BM25 + vector hybrid search. Best when you need keyword and semantic search in one system.

3. Milvus: Distributed vector database with horizontal scaling and GPU acceleration support. Best for massive-scale workloads.

4. Qdrant: Rust-based vector search engine with dense, sparse, and metadata filtering built for production-grade retrieval. Best when control over ranking and latency matters.

5. Chroma: Embedded vector database that runs directly inside Python applications. Best for local RAG prototypes and experimentation.

6. Pgvector: Postgres extension that adds vector search using SQL. Supports HNSW and IVF indexes. Best if you're already running PostgreSQL.

7. FAISS: Meta's ANN library for vector similarity search. GPU accelerated and highly customizable. Best for custom retrieval systems.

8. Vespa: Combines dense vectors, sparse retrieval, and structured data in one engine. Best for large-scale search and ranking systems.

9. MongoDB Atlas Vector Search: Vector search integrated directly into MongoDB collections. Best if your application already uses MongoDB.

10. Redis Vector: Low-latency vector search alongside caching, pub/sub, and streaming. Best for real-time AI applications and semantic caching.

11. Elasticsearch: Combines BM25 and vector search in a single query engine. Best if you're already running Elastic for search and analytics.

12. LanceDB: Runs in-process, zero-copy disk reads, no server to manage. Native multimodal support. Best for local-first AI, edge deployments, and data lake workloads.

The right choice depends on your use case and in production, it's common to use more than one. Which of these tools are you using in production?

Pagination Strategies for Large Systems

Pagination looks simple when your table has a few thousand rows. It becomes a real problem once your data grows.

Here is how the main approaches compare when the dataset is large.

Most pagination implementations start with offset and page numbers. You ask for “?page=3&size=10” and the database skips 20 rows, then returns 10. Simple to build and easy to explain. The issue is that the database still reads every row it skips. Page 5 is fast. Page 5000 is slow.

Keyset pagination handles deep pages much better. Instead of counting rows, you tell the server to return items after ID 101. The query goes straight to that point using the index, so page 1 and page 10,000 take the same time.

Continuation tokens are common in large APIs like S3 and YouTube. The server returns a token, you send it back on the next request, and it resumes from where the last response ended. It hides the cursor logic from the client. The token is tied to the filters and sort order from the original request, so the client must keep using the same parameters until the pagination ends.

Time-based pagination works well for feeds and logs. You ask for items before a timestamp and read backwards. Add a secondary field, like an ID, for rows that share the same timestamp. Without it, you can lose records.

Over to you: Which one have you used?

How LLMs Use AI Agents with Deep Research

When you ask an LLM such as Claude, ChatGPT, or Gemini to do deep research on a complex topic, it’s not just one model doing all the work. It’s a coordinated system of specialized AI agents.

Here’s how it works:

Step 1: Understanding The Question and Making a Plan

It all starts with the query, something like “Analyze the competitive landscape of AI agents in 2026. The system doesn’t just dive in blindly. First, it may ask clarifying questions to understand exactly what is needed. Then, it generates a plan and breaks the big question down into smaller and manageable tasks.

Step 2: Sub-Agents Get to Work

Each small task gets assigned to a sub-agent, which is basically a mini AI worker with a specific job. For example, one sub-agent might be tasked with finding the latest Nvidia earnings. It figures out which tools to use, such as searching the web, browsing a specific page, or even run code to analyze data. All of this happens through a secure layer of APIs and services that connect the AI to the outside world.

Step 3: Putting it All Together

Once all the sub-agents finish their tasks, a Synthesizer Agent takes over. It aggregates everything, identifies key themes, plans an outline, and removes any redundant or duplicate information. At the same time, a Citation Agent makes sure every claim is linked back to its source and properly formatted. The end result is a polished, well-cited final output ready for use.

Over to you: Have you tried deep research in any LLM?

Check it out here

Here’s what makes this cohort special:

• Learn by doing: Build real world AI applications, not just by watching videos.

• Structured, systematic learning path: Follow a carefully designed curriculum that takes you step by step, from fundamentals to advanced topics.

• Live feedback and mentorship: Get direct feedback from instructors and peers.

• Community driven: Learning alone is hard. Learning with a community is easy!

We are focused on skill building, not just theory or passive learning. Our goal is for every participant to walk away with a strong foundation for building AI systems.

If you want to start learning AI from scratch, this is the perfect platform for you to begin.

Check it out here

A program adding up a day’s sales needs to know whether all of today’s sales have actually arrived. For data stored in a file, the answer is trivial because the file has an end. However, for data that arrives continuously and never stops, there is no clean answer, and how a system resolves that gap is the difference between batch processing and streaming.

Batch processing waits for completeness. It collects data up to a natural boundary, a closing time, or a finished file, and then computes over the whole set at once. Streaming prioritizes completeness for speed. It produces answers continuously from data that is still arriving, which means it has to estimate when enough data has come in and handle the cases where that estimate is wrong. This trade-off between completeness and latency is the key consideration when dealing with streaming and batch.

In this article, we will cover the strategies on each side and what each one costs.

• On the batch side, that means full and incremental loads and large-window aggregation, with micro-batch sitting in between.

• On the streaming side, the territory runs through tumbling, sliding, and session windows, watermarks and late data, the lambda and kappa architectures, and the often-misunderstood meaning of exactly-once processing.

Boundaries

Read more

PR-AF is an open-source code review agent that ranks #2 of 42 on Martian's Code-Review-Bench, ahead of CodeRabbit, Copilot, and Devin, running a single open model. The trick is the harness: it plans a review strategy per PR, spawns parallel reviewer agents, verifies every finding against your source, and drops anything it cannot prove. That makes it about 10x cheaper per review than closed-source tools. One API call, and a drop-in GitHub Action. If open models writing frontier-level reviews sounds useful, star the repo.

Fork and deploy!

A chatbot answers a question, and an agent completes a task. This seems like a huge difference. However, the gap between the two is narrower than it appears.

An agent is an LLM placed inside a loop where the model itself decides when the loop should stop. Everything interesting about agents follows from this shift. The autonomy that makes them useful, the cost that makes them expensive, and the design challenges that come with building one all trace back to it.

It helps to see where that shift sits in the broader picture.

Software built around language models has moved through a recognizable progression. It started with a single LLM call that took an input and produced an output. Then, the function calling arrived, which let the model reach out to a tool when it needed information or wanted to take an action. After that, developers began chaining calls together in code, with each step’s output feeding into the next, to handle problems too large for a single call. The most recent of these is the agent, where the developer hands control of the loop itself to the model and lets it iterate until it decides the work is done.

In this article, we will walk through that progression. We will also look at how an agent is structured, what choices the model makes on every turn, what scaffolding holds it together, and when an agent is actually the right pattern to reach for.

Disclaimer: This post is based on publicly shared details from various sources. Please comment if you notice any inaccuracies.

Foundations

Before the loop can be interesting, the unit that sits inside it has to be clear.

A bare large language model call is a stateless function. Text goes in, text comes out, and each call typically stands alone in the model’s view. If we want it to fetch today’s weather or update a record in a database, the bare model lacks the means to do either. It can describe weather in general terms and talk about how a database record would change, but reaching the actual forecast or touching the actual database requires something more.

What makes the model useful in real systems is augmentation.

In other words, we give it the ability to call functions we have defined, often called tool use or function calling. We give it access to a retrieval layer that can pull in relevant documents at runtime. We also give it a way to write down information that it should carry between calls, which acts as memory.

Anthropic calls the combination of these capabilities the augmented LLM, and treats it as the foundational unit of every agentic system. The augmented LLM is the building block from which everything else gets composed.

This is the unit most developers have already worked with, even if they have called it something else. Examples include a chat assistant that runs Python code, a function-calling API wired to your service, and a RAG application that pulls from your documents. These systems are useful, and they are also still one call. The model produces an output, the system returns it, and the interaction ends until the next request arrives.

Workflows

What happens when a problem reaches beyond what a single call can handle?

The natural response is to string calls together in a sequence we design, a pattern called prompt chaining. Each step in the chain is its own LLM call, and the output of one step becomes the input to the next. We might use one call to draft an outline, a second call to expand the outline into paragraphs, and a third call to translate the paragraphs into another language. The chain is fixed in advance, and the developer writes which steps run, in what order, and what each step’s prompt looks like.

This is the family of patterns Anthropic groups together as workflows.

A workflow is a system where the LLM and its tools are orchestrated through a code path that the developer designed. Beyond prompt chaining, there are other workflow patterns:

• Routing classifies the input and sends it to a specialized handler.

• Parallelization runs subtasks at the same time.

• Orchestrator-worker has a manager LLM that delegates pieces to specialist LLMs.

• Evaluator-optimizer has one call to generate while another critiques, with the cycle iterating until quality is good enough.

The details differ, but every workflow shares the same property. The number of steps and the path through them are decided by the developer at design time, before the model sees the input. Most production systems built on LLMs today are workflows. They are predictable, debuggable, and usually cheaper than full-blown agents.

The Loop

An agent is what we get when we wrap an LLM in a loop and let the model decide when the loop should exit. The loop itself is plain code. The runtime calls the LLM, reads the output, dispatches whatever action that output specified, feeds the result back into the model’s view, and calls the model again. This continues until the model produces a response that signals a final answer.

There are four steps inside each iteration, and we can call them perceive, reason, act, and observe:

• Perceive is the moment the loop hands the model the current state, which includes the original task, the history of what has happened so far, and any new input.

• Reason is the model’s turn, where the model produces a response that says what to do next, whether that means asking a question, calling a tool, or wrapping up with a final answer.

• Act is when the runtime carries out whatever the model asked for.

• Observe is when the result of that action gets captured and folded back into the state, so the model sees it on the next perceive step.

The most important detail is who decides when the loop should stop. In a workflow, the developer decides at design time how many steps run. In an agent, the model decides at runtime. The model exits the loop by producing an output that the runtime interprets as a final answer. The developer typically sets a hard ceiling on the number of iterations, often called a max-turns parameter, but that ceiling is mainly a safety net. The primary stop signal comes from the model.

Observation matters as a first-class step for the same reason.

The model needs to see the result of its actions before deciding the next move. Removing observation would collapse the loop into a chain, with the model running on prior expectations rather than fresh results from the world. The closed loop, where every action is followed by an observation, is what lets the model adjust as it goes.

With the loop established, the next question is what actually happens on each turn through it.

Decisions

On every turn through the loop, the model’s output picks one of four branches, and the runtime acts on that pick. This branching makes the loop feel intelligent because the LLM is choosing what kind of move to make. Here are the four branches:

• Final answer: The model produces what amounts to a complete reply to the original task. The runtime interprets this as the loop’s exit signal, returns the result, and stops.

• Tool call: The model produces an instruction asking the runtime to invoke a specific function with specific arguments. The runtime executes the tool, captures whatever it returned, appends that result to the conversation state, and sends control back to the model so the loop continues.

• Handoff: The model decides that the current task belongs in the hands of a different agent, often a specialist with its own prompt and tool set. The runtime swaps which agent is running and feeds the same state into the new agent, with the loop continuing under the new identity.

• Continued thought: The model produces a reasoning turn that consists of thought alone. The runtime captures the thought, feeds it back into the state, and runs the model again. This branch shows up most often in ReAct-style implementation, which we are about to see.

OpenAI’s Agents SDK documents the first three branches as first-class behaviors of its loop. The fourth is more a property of how a particular prompt is written than a separate code path

ReAct

ReAct is the prompting pattern that stands for Reasoning plus Acting. The pattern asks the model to interleave reasoning steps with action steps inside the same response, so the model can think about what to do, do it, see what happened, and adjust.

A ReAct trace reads like a structured journal. The model produces a thought, then an action, then receives an observation from the runtime, then produces another thought, and so on, until it reaches a final answer.

Imagine an agent handling customer support:

• A user asks for the status of their most recent order.

• The model produces a thought first, reasoning that it needs to find that order and that the order service is the right place to look.

• It then takes an action by calling the get_recent_order function with the user’s ID.

• The runtime executes the call and feeds back an observation telling the model that the order is number 9152, placed on May 14.

• The model produces another thought, working out that it now needs the shipping status for that specific order.

• It calls the get_shipping_status function with the order ID. The observation comes back, saying the order is in transit and expected to arrive on May 29.

• The model has what it needs at this point and produces a final answer summarizing the status for the user.

See the diagram below:

Two things to consider from this flow are as follows:

• First, every action is grounded in an observation, which is the closed-loop point from the previous section made concrete.

• Second, the reasoning steps are doing real work, since they are how the model decides which action makes sense next, given what it has learned so far.

ReAct is one way to fill the loop, and it is by far the most common pattern in the agent frameworks we work with today.

Guardrails

Guardrails belong at the points where the loop crosses into the outside world. They are part of the architecture itself, designed in alongside the loop.

For reference, OpenAI’s Agents SDK documents three families of guardrails, all defined by where in the loop they run.

• Input guardrails run on the first turn, before the agent’s main model sees anything. They are the place to catch things like prompt injection attempts, requests that violate policy, or inputs that fall outside the agent’s scope. A common pattern is to use a small, fast model as the guardrail, so the more expensive main model only runs when the input passes.

• Tool guardrails wrap every function-tool invocation. A tool input guardrail runs before the tool executes and can block the call or replace it with a message back to the model. A tool output guardrail runs after the tool executes and can rewrite or block the result before it goes back into the conversation state. Tool guardrails matter because tools are how the loop touches systems we care about, and that touch needs supervision.

• Output guardrails run on the final response, after the loop has decided to terminate, but before the user sees the result. They are the last layer of policy enforcement and the place to catch things like leaked sensitive data or claims the agent should avoid making on the company’s behalf.

The structural point is that guardrails sit at every interface where the loop meets the world.

Tradeoffs

Handing control of the loop to a model is powerful, and it comes with three costs that every developer should understand before using the pattern:

• Compounding error: Per-step reliability does poorly when steps are chained together, and the math is not favorable. If the model succeeds on each step of a loop 95 percent of the time, the joint probability of every step going right across ten steps is roughly 60 percent. Stretch that to twenty steps, and the joint success rate falls to about 36 percent. The autonomous nature of agents brings higher costs and potential for compounding errors. The reason coding agents work better than open-ended task agents is that test feedback raises per-step reliability, which shortens the effective length of the chain that has to succeed.

• Scaffolding around the loop: The harness around the loop matters as much as the model inside it. For example, Anthropic shared an internal experiment where even a frontier model running in the Claude Agent SDK fell short on building a production-quality web app from a high-level prompt. The fix was scaffolding rather than a better model. They built an initializer agent that lays out a feature list before any feature work starts, a coding agent that picks one feature per session, a progress file that travels between sessions, and a git history that the agents can use to recover.

• The wrong tool for the job: An agent is often the wrong choice. It is important to find the simplest solution and only add complexity when needed, which sometimes means avoiding agentic systems entirely. Workflows offer predictability and consistency. Agents offer flexibility, and they pay for that flexibility in latency, money, and a more unpredictable failure surface. Many problems are served better by a chain.

Conclusion

The agent loop sits at the end of a recognizable progression.

We started with a single LLM call that takes text in and returns text. We added tools, retrieval, and memory to get the augmented LLM. We strung calls together into workflows when one call was outgrown. The agent is the next rung, the point where the developer hands the control flow itself to the model.

Inside the loop, four steps repeat. The model perceives the current state, reasons about it, takes an action, and observes the result. On every turn, the model’s output picks one of four branches. These are a final answer, a tool call, a handoff, and a continued thought. ReAct is the most common prompting pattern for filling the loop, with reasoning and action interleaved. Guardrails live at every place where the loop crosses into the outside world.

The design carries three real costs. These are compounding reliability across steps, the harness scaffolding that production loops require, and the question of whether a workflow would solve the problem more cheaply.

References:

• Building effective agents — Anthropic Engineering

• Effective harnesses for long-running agents — Anthropic Engineering

• Running agents — OpenAI Agents SDK documentation

• Guardrails — OpenAI Agents SDK documentation

• ReAct: Synergizing Reasoning and Acting in Language Models

Learn practical techniques to make your software faster at every layer of the stack

Working on latency-sensitive applications? This book will help you spot, understand, and fix latency in your applications and infrastructure. It shares low-latency techniques that have been predominantly “tribal knowledge” until now. You’ll learn a full-stack approach, with techniques that intersect many areas of software engineering, distributed systems, databases, and operating systems.

• Understanding and negotiating caching tradeoffs (latency vs. complexity)

• Exploring the two main concurrency models (threading and event-driven architecture)

• Identifying and applying parallelization opportunities to improve throughput and reduce latency

• Applying asynchronous process strategies that reduce perceived latency and improve overall system responsiveness.

Get this 3-chapter excerpt and start reading the book that’s become essential for anyone working with strict performance SLAs.

Download for Free

If we send the same prompt to ChatGPT, Gemini, and Claude, the answers come back meaningfully different. Claude might push back more often. Gemini ingests a two-hour video file as easily as a paragraph of text. ChatGPT routes some prompts to a slower reasoning mode and answers others instantly, without telling the user which path it took.

Why does each model behave this way, and why so consistently?

Each pattern traces back to an architectural decision point where the developers of these frontier models made distinct design choices, and the user-visible behavior follows directly from those choices.

To understand these differences clearly, we need to examine architecture and design, since architectural decisions remain consistent across releases and provide a reliable framework for understanding model behavior over time.

In this article, we will look at the various architectural forks the teams building these models encountered and the decisions they took.

Disclaimer: This post is based on publicly shared details from various sources. Please comment if you notice any inaccuracies.

Foundations

All three models share the same fundamental architecture, which is a transformer-based generative neural network.

The transformer was introduced in the 2017 paper “Attention Is All You Need,” and at its core is a mechanism called self-attention that allows each token in a sequence (a token being a small piece of text, roughly three-quarters of a word) to weigh its relationship to every other token. This setup is shared across the three models, while the components built around it, along with the training process that shapes it, vary substantially.

Training itself involves two phases:

• Pre-training is where the model learns from massive amounts of text and other data by predicting the next token, repeated billions of times over.

• Post-training is where the base model gets shaped into a helpful assistant through methods like Reinforcement Learning from Human Feedback (RLHF) and Constitutional AI. We will get to them in more detail in a later section.

Each architectural fork follows a consistent pattern, where different companies faced the same design question, each made a particular architectural choice, and each choice produced user-visible behavior we can observe today.

The first such question concerns how to scale the model’s total capacity without proportionally increasing the cost of processing every query.

Density

Every parameter added to a model carries computational cost. In a standard dense neural network, every parameter activates for every token processed, meaning that doubling the parameter count roughly doubles the compute cost per query.

Frontier developers needed a way to escape this constraint, and the architectural question became how to scale total capacity without paying full price on every single token.

Google adopted a technique called Mixture of Experts, or MoE. The Gemini 1.5 technical report identifies the model as “a sparse mixture-of-expert (MoE) Transformer-based model” that “builds on a much longer history of MoE research at Google,” with the Gemini 3 Pro model extending the same approach.

In an MoE layer, instead of one massive computational block where every parameter activates for each token, the network contains many smaller experts, and a small router determines which two or three experts each token should be sent to. The total parameter count can be enormous while only a fraction is activated per token, though achieving consistent results requires careful attention to load balancing and expert specialization during training.

OpenAI has not explicitly confirmed whether GPT-4 uses MoE, and the GPT-4 technical report deliberately omits architecture details. The GPT-5 system details describe a different efficiency approach that we will examine in the Reasoning section, where a router selects between distinct sub-models at runtime.

Nevertheless, these architectural choices produce meaningful consequences in user experience.

MoE models can pack more knowledge per dollar of compute, which contributes to Gemini’s capable handling of a wide breadth of domains. The tradeoff is increased variance, since different prompts route to different experts, and imperfect load balancing during training can leave some experts underused and weaken the model’s capacity on those topics. Dense models tend to deliver more predictable per-token behavior at the cost of being harder to scale to extreme parameter counts.

See the illustration below:

Turn cloud logs into real security signals (Sponsored)

This guide from Datadog provides best practices on how to use Cloud SIEM to detect threats, investigate incidents, and reduce blind spots across cloud and Kubernetes environments.

You’ll learn how to:

• Analyze CloudTrail, GCP audit, and Azure logs for suspicious activity

• Detect authentication anomalies and common attack patterns

• Monitor Kubernetes audit logs for lateral movement and misuse

• Correlate signals across services to accelerate investigations

Get the guide

Multimodality

The term multimodal carries different meanings depending on the architectural approach, and a precise distinction helps before we take a look at the design choices.

Two fundamentally different approaches exist for handling non-text inputs like images, audio, and video:

• Sequential approach: Train a strong language model first, then add a separate encoder later that translates non-text inputs into embeddings the language model can attend to.

• Native approach: Train a single network from inception on all modalities simultaneously, so that text tokens, image patches, and audio frames coexist in the same sequence and get processed by the same transformer layers.

See the diagram below that shows the difference between the two:

These two approaches produce different capabilities.

Google adopted the native approach from the start. The Gemini 1.0 release was designed as natively multimodal, and the 1.5 technical report demonstrates the model handling, in a single input, “10.5 hours of video at 1 frame-per-second”, and the current Gemini 3 Pro model continues this design, where the model offers “native multimodal support for text, vision, and audio inputs.”

OpenAI took a different path initially, with the original GPT-4 supporting vision through a separate pipeline. GPT-4o, released in May 2024, transitioned to a unified architecture, and OpenAI’s own announcement for GPT-5.5 acknowledged that “previous multimodal models from OpenAI were essentially separate models stitched together.”

Lastly, Anthropic has maintained a text-first approach with strong vision capabilities, and Claude Opus 4.8 continues to support high-resolution image input introduced in Opus 4.7 at 3.75 megapixels, particularly useful for documents, screenshots, and dense charts. Anthropic has chosen to focus on text and vision as primary input modalities while setting native audio and video aside for now.

For users, this architectural difference appears most clearly when working with video. Gemini handles long video files directly, and ChatGPT does the same on its more recent models. Claude performs best with documents, screenshots, and high-fidelity images.

Context

A context window represents the amount of input a model can attend to in a single pass, measured in tokens.

Larger windows allow the model to reason over more raw material at once, though long context introduces two important challenges:

• Computational cost: Attention scales roughly quadratically with sequence length, so longer windows consume significantly more compute per query.

• Quality degradation: Models often lose track of information mentioned earlier as the window fills, with the relevance of early context dropping over the course of a long conversation.

Google pushed hardest on raw window size. The Gemini 1.5 technical report demonstrated context up to 10 million tokens, and Gemini 3 Pro and 3.1 Pro continue to offer 1 million token context in production.

Anthropic offers 1 million tokens on Claude Opus 4.8 and Sonnet 5, and the company’s own documentation uses the term “context rot” to describe quality degradation in long sessions, addressed through automatic compaction that summarizes earlier portions of the conversation.

OpenAI has taken a more conservative position on window size, with GPT-4 Turbo at 128K, focusing efficiency gains on the routing architecture covered in the Reasoning section.

The practical experience differs across the three models. Loading an entire codebase or a multi-hour transcript into a single session works well with Gemini and Claude, given the larger windows available. ChatGPT is structured more around efficient routing between sub-models than around accepting massive single-prompt inputs.

Long context expands the range of questions you can ask, while retrieval-augmented generation, which fetches relevant external data at query time, remains relevant when freshness and cost matter.

Alignment

Pre-training produces a base model that has processed enormous amounts of text and learned to predict the next token. At this stage, the base model behaves as a sophisticated pattern-completer with limited regard for honesty, safety, or helpfulness.

Post-training is the phase where this base model is shaped into the assistant that users interact with, and the three companies have taken substantially different approaches to this work.

OpenAI built on Reinforcement Learning from Human Feedback (RLHF) throughout the development of GPT-3 and GPT-4. The approach involves three main steps:

• Human raters compare pairs of model outputs and select the better one.

• Those comparisons train a reward model that captures human preferences.

• The reward model is then used to fine-tune the language model toward outputs that humans prefer.

The GPT-4 product page describes an additional safety reward signal incorporated into RLHF to reduce harmful outputs. For the reasoning models, OpenAI added an approach called deliberative alignment, where the model reasons about safety policies at inference time. The publicly available Model Spec describes intended behavior in detail.

Anthropic developed an alternative method called Constitutional AI, or CAI. Instead of relying on human raters at every step, CAI uses an explicit written constitution and has the model critique and revise its own outputs against those principles, then applies AI-generated feedback to further refine the model. In January 2026, Anthropic published a 23,000-word constitution that shapes Claude’s training across synthetic data generation, response ranking, and behavioral refinement.

Google uses RLHF with learned reward models in its post-training pipeline for Gemini, with somewhat less explicit public framing than the other two companies. The Gemini technical reports describe a multi-phase training process moving from unimodal pre-training through multimodal alignment.

This architectural area also reflects a broader pattern of differing transparency choices across the three companies. The 2023 GPT-4 technical report deliberately omitted architecture details, while the 2025 GPT-5 system card describes a router, named sub-models, and safety mitigations in remarkable detail. Anthropic publishes its constitution publicly, providing a detailed view into the principles that shape Claude’s behavior. The three companies have made distinct choices about transparency, and those choices have evolved over time.

For users, these training differences appear as personality. Claude tends to push back more often on edge cases because the constitution explicitly trains for that behavior. ChatGPT is often quicker to attempt a task without questioning. Gemini’s responses vary more by domain.

See the diagram below:

With values addressed, the most recent architectural dimension concerns how the model thinks, an area where the three companies have converged from different directions.

Reasoning

The most recent and most consequential architectural development for the current generation of models concerns whether the model produces an answer directly or reasons explicitly through tokens before answering.

A standard chat model takes a prompt and immediately begins generating the response. A reasoning model first generates internal thinking tokens, working through the problem, and only afterward produces the final answer. The same prompt, the same underlying weights, and the same context window can produce very different inference-time compute profiles depending on which approach is used.

OpenAI built dedicated reasoning models, beginning with o1 in late 2024, followed by o3 in August 2025, and continued in GPT-5.5, which was released on April 23, 2026. The architecture includes three main components.

• GPT-5-Main, a fast, high-throughput model and the successor to GPT-4o.

• GPT-5-Thinking, a deeper reasoning model and the successor to o3.

• A Real-Time Router that selects between them based on conversation type, complexity, tool needs, and explicit intent in the prompt.

OpenAI plans to integrate these capabilities into a single model in the future, though the current architecture functions as a system of models rather than as a unified single model.

Anthropic pursued a different approach. Claude Opus 4.8 and Sonnet 5 use adaptive thinking inside a single model, where the same Claude determines for itself how long to think on each request, with summarized reasoning surfaced to developers for inspection.

Google integrated thinking into Gemini 3 through Deep Think mode and the three-level thinking system in Gemini 3.1 Pro, enabling the model to reason over text, images, audio, and video in a single thinking pass.

For users, this architectural difference shows up more visibly than might be expected. ChatGPT can feel inconsistent across sessions because the router may have directed two similar prompts to two different sub-models. Claude tends to feel consistent because the same model varies its reasoning depth based on the request. Gemini’s behavior shifts more with domain and modality.

A convergence happens at this architectural fork. Three companies that began with very different choices at every prior dimension, including MoE versus dense architecture, native versus bolted-on multimodality, and RLHF versus Constitutional AI, have all arrived at the same approximate conclusion. Explicit reasoning tokens at inference time materially improve model performance on hard problems.

Conclusion

In this article, we looked at five architectural dimensions, each representing a question all three frontier developers had to answer when building their models.

• Density: Google adopted Mixture of Experts openly, OpenAI built routed efficiency through a different mechanism, and Anthropic kept its architectural choice private.

• Multimodality: Google adopted native fusion from inception, OpenAI evolved from bolted-on vision toward a unified architecture, and Anthropic maintained a text-first approach with strong vision capabilities.

• Context: Google pushed the largest windows, Anthropic invested in long sessions with explicit handling of degradation, and OpenAI emphasized routing efficiency over raw window size.

• Alignment: OpenAI’s RLHF combined with the Model Spec, Anthropic’s Constitutional AI combined with its published constitution, and Google’s RLHF approach represent three distinct shaping pipelines.

• Reasoning: All three companies converged on explicit reasoning tokens, despite starting from different architectural choices at every prior dimension.

When we observe one of these models behaving in a particular way, we now have a framework for asking which architectural dimension produced that behavior.

References:

• GPT-4 Technical Report

• GPT-4 Research Page

• GPT-5 System Card

• Introducing GPT-5.5

• Learning to Reason with LLMs

• Introducing OpenAI o3 and o4-mini

• OpenAI Model Spec

• Reasoning Models Documentation

• Claude’s Constitution

• Claude’s New Constitution

• Constitutional AI: Harmlessness from AI Feedback

• Claude Opus 4.7

• What’s New in Claude Opus 4.7

• Context Windows

• Adaptive Thinking

• Building with Extended Thinking

• Gemini 1.5 Technical Report

• Gemini 2.5 Technical Report

• Introducing Gemini 1.5

• Attention Is All You Need

• Gemini 3 Technical Report

• Introducing Claude Opus 4.8

Check it out Here

Here’s what makes this cohort special:

• Learn by doing: Build real world AI applications, not just by watching videos.

• Structured, systematic learning path: Follow a carefully designed curriculum that takes you step by step, from fundamentals to advanced topics.

• Live feedback and mentorship: Get direct feedback from instructors and peers.

• Community driven: Learning alone is hard. Learning with a community is easy!

We are focused on skill building, not just theory or passive learning. Our goal is for every participant to walk away with a strong foundation for building AI systems.

If you want to start learning AI from scratch, this is the perfect platform for you to begin.

Check it out Here

The defenses websites have relied on for years are failing in predictable ways, and the failure shows up most visibly in moments like the one below.

A popular retailer drops a thousand pairs of a limited-edition shoe at noon. Within thirty seconds, the inventory is gone. The buyers turn out to be automated agents working for resellers. The real people who wanted the shoes get nothing.

The retailer almost certainly tried the standard defenses: rate limits by IP, CAPTCHA, phone verification, device fingerprinting. Each of them helps for a while and then stops working. The reason they all fail in the same way is worth pausing on.

Every one of those defenses relies on a proxy for what the retailer actually wants to verify. An IP address is a proxy for a different network. A phone number is a proxy for a different person. A device fingerprint is a proxy for a different device. Each proxy fails the moment adversaries learn to acquire many of them cheaply, and adversaries always do. Phone numbers can be bought in bulk. Fingerprints can be randomized. IPs can be rotated through residential proxy networks. None of these strategies binds the verification to a real, unique person.

Authentication systems do not solve this either. Single sign-on, face unlock, passkeys, OAuth tokens, all of them compare an incoming credential against a stored template and return yes or no. None of them answers the question that actually matters here: has this user already been verified somewhere else in the world?

The central question: How do you let a real, unique human be recognized across the internet, without ever knowing who they are?

For this article we spoke with the team behind World, including Tiago Sada and Lily Gordon at Tools for Humanity, on how they try to solve this problem.

Five Pillars of a Proof-of-Human System

A working answer to the central question requires five distinct ingredients. We will look at each one in turn.

Uniqueness. Why is this a different identity problem than the ones we usually solve?

Anonymity. How can a credential be issued without anyone knowing the identity of the user?

Recovery. How does the system survive lost phones and reinstalled apps?

Verification. How does the holder present the credential without revealing more than necessary?

Delegation. What changes when the holder is an AI agent acting on a person’s behalf?

Pillar 1: Uniqueness

To see why uniqueness is structurally different from authentication, start with a system everyone already trusts: Face ID on a phone.

When the user sets up the phone, the camera captures a single facial template and stores it locally. From then on, every unlock attempt produces a fresh capture, which the phone compares against the stored template. The comparison space is exactly one. If the fresh capture matches the template within some tolerance, the phone unlocks. If not, the phone refuses. The system handles other faces by failing the match.

This is a one-to-one matching problem. The math is simple because the comparison is small. Even with a per-comparison error rate of one in a million, the phone unlocks reliably for its owner and refuses essentially every other face.

However, the example with the retailer wants a different guarantee. They want to verify, at checkout, that the buyer is different from every other person who has already bought those special edition shoes. The comparison is no longer against one template, but against the entire population of past buyers. If the system is meant to work at internet scale, the population is potentially every person on the planet.

This is a one-to-many matching problem, and the diagram below shows this comparison in the context of World ID. As you can notice, the math can get much more complex when the comparison space grows.

The probability of a false match scales roughly with the size of the comparison space. A per-comparison error rate of one in a million sounds excellent, but checked against a billion candidates, it produces roughly a thousand false matches per query. To make one-to-many uniqueness viable at a billion-person scale, the per-comparison error rate has to be on the order of one in a hundred billion or better. That requirement rules out most consumer-grade biometric methods straight away.

This type of system that answers the one-to-many question is what the World ID calls the proof of human. At present, there is no widely deployed equivalent of this category. We have a great deal of authentication infrastructure and very little uniqueness infrastructure.

Pillar 2: Anonymity

Once the goal is uniqueness at an internet scale, an apparent paradox emerges.

Checking whether someone is the same as anyone else usually requires recognizing them. If a system cannot identify a person, how can it tell whether it has seen that person before?

The answer takes multiple steps.

Step 1: Find a biometric signal that scales

Most consumer biometrics fall short. For example, fingerprints have decent entropy but can be captured from surfaces. Face geometry varies less than people assume. The iris pattern of the human eye, by contrast, turns out to be one of the few biometric features with the entropy needed for billion-scale comparisons. Two unrelated humans have essentially no chance of producing matching iris patterns, even after accounting for camera noise and aging.

However, reading the iris reliably is the next challenge, and this is where hardware matters.

A standard phone camera can be replaced with a device that injects images directly into the camera pipeline, defeating any face check done in software. A printed iris image can fool an infrared camera that lacks depth detection. To get a reading that resists these attacks, the capture has to happen on hardware that controls the entire signal path from sensor to processing.

Tools for Humanity handles this using a purpose-built device called the Orb. The Orb uses multispectral imaging across infrared and visible wavelengths, runs several neural networks locally to verify liveness and detect masks, and deletes the original images before anything leaves the device. What gets transmitted is signed, encrypted material derived from the iris, not the iris itself.

Step 2: Check for duplicates without anyone seeing the data

That solves the first half of the problem. The second half is harder. The system needs to check whether anyone else in the world has the same reading, while no party (including the system operator) ever sees the reading itself.

The trick works like this:

• Take the iris reading and split it into three pieces of statistically random noise.

• Hand each piece to a different organization operating in a different legal jurisdiction.

• Each party, looking only at its piece, can tell nothing about the underlying reading.

• The parties can jointly compute answers like “does this reading match any reading we have already enrolled?” without any single party ever holding the whole reading.

• The pieces are combined only in the abstract space of the computation, never reconstructed on any actual machine.

This is what cryptographers call secure multi-party computation, and the specific variant World uses is called anonymized multi-party computation (AMPC). The parties (referred to as SMPC nodes) are operated by independent organizations in different legal jurisdictions.

See the diagram below that tries to conceptualize what each SMPC party actually sees:

What emerges from this setup is the property that makes proof of human possible. The system can determine whether a given human has enrolled before, and no single party (including the main organization) ever sees the underlying biometric in usable form. The user is verified as unique without being identified.

Pillar 3: Recovery

A verified credential is only useful if the human carrying it can keep using it across years and across devices. In a normal life, phones get lost, and software gets reinstalled. Any system meant to anchor a lifelong property like “I am a verified unique human” has to handle the case where the holder loses access to everything they were using to prove it.

If the credential is a private key on a phone, losing the phone means losing the credential. The user would have to re-enroll. If the system is anchored in physical hardware verification, it means traveling back to a central location (in the case of World ID, that means the Orb). It would be a rather poor user experience for everyday loss, and impossible for some users in some regions.

The fix is to stop treating the credential as a single secret. Instead, treat the verified human as an abstract account in a public registry. The account does not hold the user’s biometric or any user secret directly. It holds a list of public keys (called Authenticators) that the user has authorized to act on the account’s behalf.

Think of an Authenticator as any piece of software or hardware that holds a private key on the user’s behalf. Here are some examples:

• A common Authenticator could be the user’s phone with the wallet app installed.

• A browser extension that holds a separate key could be another.

• A hardware security token plugged in over USB is a third option.

When the user wants to verify on a service or app, the Authenticator is what actually produces the cryptographic proof and signs the verification request with its private key. The account in the public registry holds the matching public key, so anyone can confirm the signature without ever knowing the secret. Keys can be added, revoked, and rotated. The account itself persists.

For more context, the public registry stores one entry per verified human. Each entry contains the public keys of all currently authorized Authenticators, the designated Recovery agents for that account, and the rules under which those agents can act. The biometric reading itself is never stored in the registry. This ensures that reading the registry cannot reveal personal details about the verified humans.

See the diagram below that shows this approach in the context of World ID’s solution:

If the user loses all of their Authenticators, the account also designates one or more Recovery Agents, which are parties trusted to re-authenticate the human (typically by running them through a fresh biometric check) and let them register new keys.

In the current design of World ID, the Orb network serves as the default Recovery Agent. The on-chain registry that holds all of this is called the WorldIDRegistry.

What this might look like in practice is as follows: imagine the user’s phone is stolen on a trip. They get a new phone and install the wallet app fresh, but the app has no private keys yet. Rather than travel back to an Orb to start over, the user can initiate a recovery through one of the Recovery Agents they previously designated. The agent runs them through a fresh biometric check, typically through the Orb’s mobile face authentication. If the check passes, the agent submits a transaction to the registry that revokes the stolen phone’s key and registers the new phone’s key.

However, the open question here is governance. The Recovery Agents can themselves become a trust point, which would require some form of oversight.

Pillar 4: Verification

So far, we have a way to verify a human as unique, and a way for them to hold their credentials durably.

What we need next is a way to present that credential to a relying party (for example, a dating app, a shoe retailer, or any service that wants to enforce one-per-human) so the service can do its job while learning as little as possible about the user.

A relying party in this scenario needs three things at once:

• Assurance that the request is backed by a real and verified unique human.

• The ability to enforce its own one-per-human rule, which means recognizing repeat visits by the same human within its own context.

• Confidence that what the user reveals is limited to the fact of being a verified, unique human, with identity, other activity, and behavior on other platforms staying private.

Satisfying all three at once is harder than it sounds. Standard identity tokens (an OAuth token, a JWT, a session cookie) tie recognition to identity. The relying party gets the ability to enforce policies, but pays for it with the user’s privacy.

The World ID handled this problem using a primitive called a nullifier. A nullifier is a number derived from three inputs combined together. The inputs are the user’s verified credentials, the relying party’s identifier, and a particular action. The same combination always produces the same number. Different combinations produce uncorrelated numbers. If the relying party sees the same nullifier again for the same action, they know it is the same human attempting the same thing twice. Nullifiers from the same user on a different action, or in a different service, are uncorrelated and cannot be linked back.

See the diagram below that shows what the relying party actually sees:

Generating a nullifier like this requires careful design. A user cannot generate it alone (they cannot be trusted to use the right inputs), and a single server cannot generate it either (the server would learn who the user is). It needs a distributed network of nodes, and here’s how they work:

• The user sends a blinded version of their query

• Each node computes on the blinded value while staying unaware of what the value represents

• The user unblinds the result.

• A threshold of nodes must cooperate. These nodes are called OPRF nodes (oblivious pseudorandom function).

One more piece is needed to complete the puzzle. If a user could reuse the same nullifier multiple times within one relying party context, an enterprising service could build a pseudonymous profile of the user’s repeated interactions. To prevent this, the design maintains an on-chain list of every nullifier that has been used, called the Oblivious Nullifier Pool. The user’s own software refuses to generate a proof for a nullifier already in the pool.

So, how is all this complexity handled from an application’s perspective?

In the context of World ID, all of this gets bundled into two zero-knowledge proofs that the user presents to the relying party. An SDK named IDKIT wraps the flow for integrators.

See the diagram below that shows a simple IDKit Proof Flow:

Pillar 5: Delegation

Up to this point, we have assumed the human is the one acting. The user might open a dating app, buy sneakers from an online retailer, or sign a document. The design handles each case by binding one verifiable human to one proof.

The interesting question is what happens when the human is not the one acting. This is because people are now increasingly using AI agents to take actions on their behalf. For example:

• An assistant books a flight.

• An agent monitors product drops and buys when something matches.

• A service fills out and submits forms for a user who would rather not.

The standard defense against agents is to block them as bots, but here the human wants the agent to act for them. Treating the agent as adversarial defeats the very automation the user is paying for.

The extension to the design for an ideal proof of human solution is to let the agent register against a real human’s credentials before it acts. When the agent later contacts a service, the service can verify that the agent is backed by a verified, unique human, while still learning nothing about which human. The agent’s activity counts against the human’s quota, so a single human cannot turn into many parallel agents to amplify their reach.

See the diagram below that shows how World ID implements this using AgentKit:

In World’s implementation, the agent registers its wallet address in a registry called AgentBook (deployed on the World Chain blockchain), tying the agent to the human’s World ID through a verification flow in the user’s wallet app. A service or application that wants to admit human-backed agents installs the SDK called AgentKit, which by default allows three uses per human per service before normal payment takes over.

A few new use cases become possible:

• A startup can offer a free trial that bots cannot farm at scale, because each trial use is bound to a verified human.

• A retailer can enforce one pair per human on a limited drop, even when agents do the buying.

• A merchant can require human backing for transactions where chargeback fraud is a concern.

What remains open is how this scales. At present, the volume of agent traffic is quite small compared with what AI agents will likely produce in the coming years. Whether the per-human cap stays meaningful when one human represents many concurrent agents is a question worth looking into.

Conclusion

Taken together, the five pillars (uniqueness, anonymity, recovery, verification, and delegation) form a working blueprint for proof of human in principle. Whether any particular implementation, World ID included, becomes the way the larger question gets answered at internet scale remains to be seen.

A few specific questions are worth tracking as the design meets larger-scale reality:

• Hardware decentralization. The longer-term plan calls for multiple independent manufacturers of devices like the Orb, distributed across jurisdictions and cross-checking each other through security deposits. The architectural properties depend on this transition actually happening.

• Unlinkability under adversarial pressure. The cryptography looks sound on paper. The harder test is what happens when relying parties, governments, or other large actors try to extract more from the system than the design intends. Whether the unlinkability properties hold in practice at scale is something the community will only learn over time.

• Delegation at agent scale. Most agent traffic today still flows through traditional authorization systems. As agents take on more autonomous activity, the per-human cap will be tested in ways the existing deployment has not yet seen. New abuse patterns will almost certainly emerge.

• The bootstrap problem. Proof of human becomes useful at scale, and reaching scale requires apps to require it, which in turn requires users to have it. How any such system navigates that loop, and what happens if a meaningful fraction of the internet starts to require proof of human, is far from clear today.

The point of this article was not to argue for any particular product, but to surface the techniques. Once you can see the shape of the problem, one-to-many matching, secret-shared biometrics, scoped nullifiers, account-style key rotation, delegated quotas, the headlines about “verify you are human” start to look less mysterious. They are attempts to solve a specific class of problems that the rest of the identity stack has so far left alone.

References and Further Reading

• https://world.org/liftoff

• https://docs.world.org/world-id/idkit/integrate

• https://docs.world.org/agents/agent-kit/integrate

• https://world.org/blog/engineering/private-proof-of-human

• https://eprint.iacr.org/2024/705

• https://world.org/blog/engineering/introducing-world-id-4.0

That runs against the instinct, which says more locations should mean faster service and steadier uptime, which is often the case. But once the same data lives in two places at once, a new class of problem appears that can nullify the advantages of multi-region deployments.

Let’s look at a short example. Picture two edits to the same piece of data, made at nearly the same instant, one handled by a server on the US East Coast and one by a server in Frankfurt, right as the network link between those regions drops. Both edits are saved locally. Each region now holds a different version of that data, with no shared record of which one came first. When the link comes back, one of the two versions has to be chosen over the other. The way a system handles that question has a large effect on what a global footprint costs to build and to run.

Going global is better understood as a progression than as a single decision. In this article, we will start with the building blocks, the small set of concepts every regional design rests on, then work through the common setups in order, from a single region with backups up to running every region at once. Each step buys something concrete: lower latency, higher availability, or the ability to keep data inside a country’s borders. Each one also has a price, in money and in the consistency tradeoffs it introduces.

Foundations

Read more

Matic is the first visually intelligent robot vacuum that sees your home like you do, so it can clean how you want it to.
Matic’s hero features:

• Runs entirely on cameras to deftly navigate obstacles

• Recognizes floor types to auto-switch between vacuuming and mopping

• Big wheels and a height-adjustable cleaning head handle thick rugs

• Quieter than conversations at 55 dB

• Handles pet hair without clogging or tangling

• A single bag collects dry and wet waste—diaper-salts gel dirty water, antimicrobial powder prevents mold

• A fresh HEPA filter in every bag for cleaner air, no washing or replacing

Experience autonomous cleaning with Matic with a 180-day money-back guarantee.

Get your Matic

OpenAI runs voice AI for 900 million users a week, and they use WebRTC for it because the alternative would mean reinventing how the internet handles live audio.

The catch is that WebRTC was designed for servers with stable IPs and ports, and Kubernetes treats those addresses as disposable. The conventional answer at this scale is an SFU, which suits multiparty workloads like group video calls, but OpenAI’s traffic is overwhelmingly one user talking to one model.

To deal with this, their architecture splits the stack into two pieces:

• A stateless relay handles protocol-aware packet routing at the geographic edge.

• A stateful transceiver owns all the heavy WebRTC state.

The trick that ties them together is using the ICE ufrag, a field the protocol already exchanges during setup, as a routing key that the relay can read off the first packet of a new session. Everything else, from Global Relay to the userspace Go implementation to the Redis cache and the careful socket-level optimizations, builds on top of that core idea.

In this article, we will look at the entire journey in detail and challenges the OpenAI engineering team faced.

Disclaimer: This post is based on publicly shared details from the Open AI Engineering Team. Please comment if you notice any inaccuracies.

Why Latency Matters For Voice AI

Voice AI either feels like a conversation or it feels like a walkie-talkie. The line between those experiences is measured in milliseconds.

When the network pauses between hearing a user and responding, the illusion breaks. Pauses turn awkward, interruptions get clipped, and users are compelled to cut off the AI mid-sentence, which is also kind of rude. In other words, voice AI only feels natural if the conversation moves at the speed of speech.

The harder constraint underneath is the continuous-stream property. Audio has to arrive at the model as a steady flow, rather than as a single upload after the user finishes talking. That stream is what lets the model start transcribing, reasoning, and calling tools while the user is still speaking. The experience collapses into push-to-talk once it breaks.

For OpenAI specifically, those constraints translate into three concrete requirements:

• The system has to reach 900 million weekly active users wherever they are.

• Connection setup has to be completed quickly enough that users can start speaking as soon as a session begins.

• Round-trip time for audio has to stay low and stable so turn-taking feels crisp.

WebRTC is the protocol the industry built for this kind of work. It is a bundle of smaller protocols (ICE for figuring out how two endpoints reach each other across firewalls, DTLS for encrypting the channel, SRTP for the audio packets, and RTCP for quality feedback). Justin Uberti, one of WebRTC’s original architects, and Sean DuBois, who maintains the Pion library OpenAI builds on, both work at OpenAI today. That kind of protocol depth shows up in the architecture they shipped.

The Original Architecture

The first version of OpenAI’s WebRTC infrastructure was a single Go service built on Pion. It handled both jobs in one place:

• On the signaling side, the service negotiated SDP (the format clients and servers use to describe a session), selected codecs, generated ICE credentials, and set up sessions.

• On the media side, the service terminated WebRTC connections from clients and maintained upstream connections to the backend services that run the AI models, including inference, transcription, speech generation, tool use, and orchestration.

That combined service still powers ChatGPT voice, the Realtime API’s WebRTC endpoint, and several research projects, and it has handled that work well. The question OpenAI ran into was how to deploy it on Kubernetes, the container orchestration system that runs most modern cloud infrastructure.

Kubernetes assumes compute is cheap and movable. Pods come up, get scheduled wherever capacity exists, run for a while, then get rescheduled or replaced. Standard WebRTC deployment patterns assume the opposite. That mismatch shows up in two specific places.

The first is port exhaustion.

The conventional way to deploy WebRTC uses one UDP port per session. At OpenAI’s scale, that means tens of thousands of public UDP ports per service. Cloud load balancers were built for a handful of well-known ports, so each additional range adds operational complexity for load balancer config, health checks, firewall policy, and rollout safety. The exposed surface area also makes security audits harder. Kubernetes autoscaling clashes with the requirement to reserve large and stable port ranges, which makes elasticity brittle.

The second is state stickiness.

Running one UDP port per server and demultiplexing sessions behind it solves the port problem. ICE and DTLS, however, are stateful protocols. The process that started a session has to keep receiving its packets to validate connectivity checks, complete the DTLS handshake, decrypt SRTP, and process later session changes like ICE restarts. If a packet for an existing session lands on a different process, setup fails, or media breaks.

Both pressures point to the same answer. The deployment architecture has to change while the client experience stays identical.

Splitting The Relay From The Transceiver

The architecture OpenAI shipped splits packet routing from protocol termination.

A stateless relay sits at the front, presenting a small public footprint to the internet. A stateful transceiver sits behind it, owning all the heavy WebRTC state. Signaling still goes directly to the transceiver. Media enters through the relay first.

The relay’s scope is deliberately narrow. It reads enough of each packet to choose a destination, then forwards the rest as an opaque payload. Audio stays encrypted on the way through, ICE state machines stay with the transceiver, and codec negotiation happens elsewhere. From a client’s perspective, the WebRTC session looks normal in every way.

The transceiver owns the parts of WebRTC that have to remember things. ICE connectivity checks, the DTLS handshake, SRTP encryption keys, and the session lifecycle all live there. The transceiver is the endpoint that completes the handshakes and encrypts or decrypts the actual media.

See the diagram below:

There was an obvious alternative that the team evaluated and chose against.

An SFU, or Selective Forwarding Unit, is the standard media server architecture for WebRTC at scale. It terminates one WebRTC connection per participant and selectively forwards streams between them. The AI joins as another participant.

This works well for inherently multiparty products like group calls, classrooms, and collaborative meetings. OpenAI’s workload looks different. Most sessions are 1:1, with one user talking to one model. For that kind of traffic, the SFU model adds overhead and forces backend services to behave like WebRTC peers themselves. The transceiver model lets the backend stay an ordinary service.

See the diagram below that shows the SFU approach:

TURN was also considered and set aside.

TURN is the standard protocol-terminating relay used for NAT traversal. The trouble is that TURN allocations add setup round-trips before media can flow, and migrating or recovering them across servers is hard. For a latency-sensitive workload, those extra round-trip matters.

The split solves the port and state problems in principle. The remaining problem is making the relay route the first packet correctly.

Routing The First Packet

The first packet of any new session is the difficult one.

Subsequent packets are easy because the relay has a mapping that says that packets from this source IP and port go to this transceiver. The first packet is what creates that mapping, so the relay has to figure out where to send it from the packet itself.

See the diagram below:

Two naive options were present:

• A database lookup on the hot path adds latency and a hard dependency on another service staying healthy.

• Routing to a random transceiver and forwarding internally works, but doubles the hop count.

OpenAI chose a third option.

The answer lives inside a field that WebRTC already exchanges. Every WebRTC session carries an ICE username fragment, called the ufrag, which is produced during session setup and echoed in STUN binding requests. STUN binding requests are the packets ICE uses to verify that two endpoints can actually reach each other, and they are usually the first thing a client sends on the media path.

The trick is that OpenAI generates the server-side ufrag during signaling. They can put whatever they want in it, so they encode routing metadata into it. The relay parses just enough of the first STUN binding request to read the ufrag, decode the routing hint, and forward the packet to the transceiver that owns the session. Every packet after the first one flows through the established session mapping, which skips the ufrag parsing step entirely.

See the diagram below that shows the connection establishment sequence in detail:

Each transceiver in the fleet listens on a shared UDP socket, which is one operating system endpoint bound to an internal IP and port. All sessions for that transceiver multiplex behind it.

During signaling, the transceiver returns a shared relay VIP and UDP port in the SDP answer. A VIP is a virtual IP address that fronts the entire relay fleet, so the client sees one stable destination like 203.0.113.10:3478, even though many relay instances sit behind that address. From the client’s side, there is one place to send packets, and it stays the same for the life of the session.

The relay’s state is purposefully tiny. It holds an in-memory map of source address to transceiver destination, plus some counters for monitoring and timers for session cleanup. If a relay instance restarts and loses the mapping, the next STUN packet rebuilds it from the ufrag. To make recovery faster, a Redis cache holds the source-to-destination mapping once a route is established. A restarted relay can look up the mapping from Redis immediately.

The principle here generalizes well. When we need data on the hot path, look at what the protocol is already exchanging. A field on the payload is essentially free to parse. A new lookup costs latency, a dependency, and one more thing that can break.

Global Relay and Geo-Steered Signaling

Once the public UDP surface was reduced to a small fixed set of addresses, the same relay pattern became deployable globally.

Global Relay is OpenAI’s fleet of geographically distributed relay ingress points. All of them run the identical packet-forwarding behavior described above. The only thing that changes is where on the map they sit.

Geographic distribution shortens the first client-to-OpenAI hop. A packet entering the network at a relay close to the user, in both geography and network topology, has a much easier time than a packet that has to traverse the public internet to reach a distant region first. The practical effect is lower latency, more stable timing, and a cleaner loss profile before traffic reaches the OpenAI backbone.

OpenAI uses Cloudflare for geographic and proximity steering on the signaling side. The initial HTTP or WebSocket request that sets up a session is routed to a nearby transceiver cluster. The request context then determines which Global Relay ingress point gets advertised back to the client in the SDP answer. The ufrag carries enough information for Global Relay to route media to the right cluster, and for the cluster’s relay to route to the right transceiver.

The result is that both the signaling and the media paths enter the OpenAI network at points close to the user, while the session itself stays anchored to one specific transceiver for its full lifetime. The setup round-trip and the first ICE connectivity check both shorten, which directly reduces how long a user waits before they can start speaking.

The Go Relay Implementation

The relay is a Go service running in userspace, which is to say a regular process that reads from a regular UDP socket.

The Linux kernel receives UDP packets from the network interface, delivers them to a socket bound to the relay’s IP and port, and the Go process reads from that socket, updates a small amount of flow state, and forwards each packet to the right transceiver.

OpenAI evaluated kernel-bypass frameworks (which let a userspace process poll the network card’s queues directly) and chose to stay away. Bypass raises packet throughput at the cost of operational complexity. The team’s workload fit inside what a careful Go implementation could handle.

Three implementation choices carry most of the performance load.

• SO_REUSEPORT is a Linux socket option that lets multiple workers on the same machine bind the same UDP port. The kernel then distributes incoming packets across those workers, which removes the bottleneck of a single read loop.

• runtime.LockOSThread pins each UDP-reading goroutine (a lightweight thread in Go) to a specific OS thread. Combined with SO_REUSEPORT, this tends to keep packets from the same flow on the same CPU core, which helps cache locality and reduces context switching.

• Pre-allocated buffers and minimal copying during packet parsing keep allocation overhead low and avoid putting pressure on Go’s garbage collector.

The takeaway is that ordinary optimizations were enough. The relay handles OpenAI’s global real-time media traffic on a relatively small footprint.

Design Tradeoffs

Every architecture comes with tradeoffs, and this one carries several worth understanding.

• The whole design is built around 1:1 sessions. If OpenAI ever wants to add multiparty features (group voice calls, multiple participants in a single AI session, human handoff during a call), large parts of this architecture would probably need rework. Both the transceiver model and the choice to skip SFU rely on the assumption that most sessions have exactly two endpoints.

• OpenAI also took on a custom infrastructure burden. A standard SFU comes with documentation, a community, and battle-tested patterns. The relay, the transceiver, and the coordination between them are all internal code. Every engineer who works on this stack has to learn it from the inside out.

• The “stateless” relay turns out to be stateless mostly in spirit. It holds an in-memory flow table and uses a Redis cache to recover that table across restarts. The architecture works because the protocol can rebuild the table from the ufrag, but the soft state is still a state.

• Lastly, the ufrag trick depends on controlling both ends of signaling. OpenAI generates the server-side ufrag, so they can embed routing metadata in it freely. A team that uses an off-the-shelf signaling stack might find this technique harder to adapt directly.

Conclusion

The architecture OpenAI built for voice AI is a careful response to a specific pressure. WebRTC was designed for stable servers. Modern cloud infrastructure runs on the opposite assumption. OpenAI’s team had the protocol depth to determine whether a WebRTC session needs to live in one process at all.

Their answer separates the work into two pieces. A stateless relay forwards packets near the user. A stateful transceiver, anchored in one place, owns ICE, DTLS, SRTP, and the session lifecycle. The two pieces communicate through information that’s already in the WebRTC handshake, which keeps the routing decision on the packet path itself.

The implementation choices stayed deliberately simple. Userspace Go, SO_REUSEPORT, thread pinning, and careful memory management did the work that kernel bypass solves. The result handles 900 million weekly users on a relatively small relay footprint.

References:

• How OpenAI delivers low-latency voice AI at scale

Run npx workos@latest and an AI agent will inspect your project, detect your framework, and add AuthKit directly to your codebase.

This is not a generic starter template. The agent reads your actual app, writes the integration where it belongs, then typechecks and builds so it can fix errors along the way. AuthKit gives you hosted auth, customizable UI, MFA, social login, session management, and a path to enterprise features like SSO and SCIM when you need them.

Free up to 1 million MAUs.

Try it now →

What feels like a real-time conversation with AI today is built from many parts working together.

At the center sits a language model that works in turns, the same way ChatGPT does when you type to it. The responsiveness comes from a layer of helper systems wrapped around that model, predicting when the user has paused, transcribing audio, generating speech from text, and weaving the pieces together fast enough that the conversation feels fluid.

However, new research from Thinking Machines argues that this whole approach has a ceiling, and proposes a different way to build AI systems for real-time interaction.

Thinking Machines is a relatively new AI research lab focused on human-AI collaboration, publishing research under the name Connectionism and offering developer-facing products for the broader community. What sets them apart is the problem they have identified as central. Most AI labs treat autonomous capability as the most important capability to push forward, meaning the ability for a model to take a task, do the work on its own, and return a result.

Thinking Machines argues this framing sidelines humans. Real work, in their view, benefits from continuous collaboration where the human clarifies, redirects, and gives feedback as the model goes along. The interface should support that, rather than treating the human as someone who hands off a task and walks away.

In this article, we will look at what the research preview covers and the concept of an interaction model proposed by Thinking Machines.

Disclaimer: This post is based on publicly shared details from the Thinking Machines Engineering Team. Please comment if you notice any inaccuracies.

Bottleneck

The problem starts with how today’s models actually experience the world. A typical language model works in a single thread. It waits for the user to finish typing or speaking before it can perceive any input. Once the model starts generating a response, its perception freezes, and any new input gets queued for later.

Thinking Machines compares this setup to resolving a crucial disagreement over email rather than in person. The bandwidth is just too narrow. So much of what makes a collaboration work, the way your voice shifts when uncertain, the moment of realizing a direction change is needed mid-sentence, the reaction on your face when the other person says something useful, all of it gets stripped out of the channel between human and model.

This matters because real work that benefits from another mind in the room depends on that bandwidth.

A model that only sees clean, finalized inputs forces a person to think like a model, preparing the full request, handing it over, and then waiting. In contrast, real collaboration is often messy, interruptive, and full of mid-stream corrections. Until the interface allows for that, the human ends up doing extra work to fit how the model wants to operate. Thinking Machines argues this bottleneck explains why much of today’s AI work feels like prompting and waiting rather than collaborating the way two people might.

Harness

If today’s voice AI feels real-time despite this limitation, how is that even working to a large extent? The answer is a pattern called a harness.

A typical voice AI product is a stack of components glued together:

• Voice activity detection listens for pauses and decides when the user has stopped speaking.

• A speech-to-text model transcribes what was said.

• A language model generates a text response.

• A text-to-speech model converts that response back into audio.

• A dialog manager orchestrates the entire pipeline so the latency feels acceptable.

Imagine a brilliant scholar who communicates only through letters slipped under a door. Making this feel like a conversation requires helpers. One stands outside listening for when the visitor stops talking, another reads the scholar’s letters aloud when they come back, and a third rings a bell when something visible happens that the scholar should know about.

The setup mostly works, but the scholar still experiences reality through letters. Voice tones, facial expressions, the moment itself, all of it stays beyond the scholar’s reach. This is what every real-time voice AI actually is, with a turn-based language model at the center surrounded by helpers that simulate conversation around it.

Why does this approach have a ceiling?

It is because the helpers are simpler than the model itself. Voice activity detection runs on raw audio signals using a much smaller and lighter model than the language model behind it. This limits whole categories of behavior.

The system struggles with proactive interjections like “interrupt me when I say something wrong,” because the helper deciding when to speak operates purely on acoustic signals, while correctness remains the language model’s job. Visual reactions like “tell me when I’ve written a bug in my code” face the same problem, because the helper handles audio while anything on screen stays beyond its reach.

This is where Thinking Machines points to an important lesson. As per a famous essay by Rich Sutton, methods leveraging general computation and learning consistently outperform methods that bake in human-designed heuristics. The same argument led from hand-crafted computer vision features to deep learning, and from hand-crafted game heuristics to self-play. Applied to interactivity, harness components are exactly the kind of hand-crafted heuristic that scale will eventually push out. The way past the ceiling is to put interactivity inside the model itself.

How engineering teams resolve incidents in minutes instead of hours. (Sponsored)

Datadog and Dust share how unified observability gives engineers full visibility across infrastructure, logs, and applications to detect issues faster and reduce MTTR. Watch this on-demand webinar to see how Dust reduced complex incident investigations from hours to minutes by bringing observability context into their AI workflows.

You’ll learn:

• How unified observability connects infrastructure health, logs, and applications into a single view for faster incident resolution

• How Dust used the Datadog MCP to pull observability context directly into incident investigations

• Practical strategies for maintaining reliability while building for the AI era

Watch the webinar

Architecture

What does putting interactivity inside the model actually look like?

Thinking Machines’ answer is a system they call an interaction model. The first version, named TML-Interaction-Small, is a 276-billion-parameter mixture-of-experts model with 12 billion active parameters at any moment. The word “small” in the name refers to where this sits in their planned lineup, with larger versions expected later.

Most multimodal systems start with text and add audio and video on top.

Thinking Machines did the reverse, starting from continuous audio and video because live conversation operates under tight real-time constraints that text can avoid. Designing around the hardest case first gives them an architecture that handles concurrent input and output streams across every modality.

See the diagram below:

Three design choices stand behind this architecture:

• The first is time-aligned micro-turns, which change what the model treats as the unit of conversation.

• The second is an approach that skips heavy pretrained encoders, with audio and video going through lightweight processing components trained from scratch rather than being routed through standalone systems like Whisper.

• The third is a two-model coordination scheme where a fast interaction model works alongside a slower background model that handles deeper reasoning.

Thinking Machines also did significant work on optimizing inference for this design, including contributing a streaming session feature back to the open-source SGLang library, enabling 200-millisecond chunks to be processed efficiently.

Micro-turns

Most AI models work in turns, with the user speaking, then the model speaking, then the user speaking again. Each turn is a discrete unit, and the model processes one complete turn at a time. Even when a system handles audio, the underlying logic stays turn-based. The harness simulates real-time, but the model itself perceives the world in clear, separate chunks.

Thinking Machines made a different choice.

Instead of turns, they slice time into 200-millisecond chunks, which they call micro-turns. Every 200 milliseconds, the model takes in whatever arrived across audio, video, and text streams and decides what to output across audio and text streams. Time becomes the fundamental unit, replacing the turn entirely.

This sounds like a small change, but it transforms what the model can do. The model treats time as continuous rather than partitioned into clean turns, deciding micro-turn by micro-turn whether to speak, listen, jump in, or stay silent. Input and output are happening continuously at the same time.

Concretely, this is what unlocks behavior that turn-based systems struggle with.

• The model can speak while listening, which is how live translation works.

• It can watch while speaking, which is how live sports commentary works.

• It can jump in mid-sentence when something visual happens, such as counting pushups in real time as someone exercises.

• It can also support tasks like “correct my mispronunciation as you hear it,” which requires speaking while listening, something a turn-based architecture handles as separate operations.

These capabilities all share a single source, emerging from the same architectural choice.

Coordination

Time-aligned micro-turns solve responsiveness, but they create a new problem.

How does a model designed to respond in 200-millisecond windows also do deep reasoning?

Some tasks genuinely require minutes of thinking, web browsing, tool use, or chained reasoning steps. Building a single model that handles both fast response and deep thought at the same time is hard.

Thinking Machines’ answer is to use two models working together:

• The interaction model is fast, present, and handles real-time conversation.

• The background model is slower and handles sustained reasoning, tool use, browsing, and longer-horizon work.

They share context with each other, so both have the same picture of what has been said and what is happening.

`The coordination works like this:

• When the interaction model encounters something that needs deeper reasoning, it sends a rich context package over to the background model.

• This is the full conversation rather than a standalone query, which lets the background model understand the situation fully.

• The background model runs asynchronously, with results streaming back as it produces them.

• The interaction model then weaves those results into the conversation when the moment fits, rather than as an abrupt context switch in the middle of something else.

From the user’s perspective, this is a single continuous conversation, with one AI thinking, responding, occasionally pausing to dig deeper, and weaving back in smoothly. Behind the scenes, two systems coordinate throughout.

This same logic shows up across computing, with fast paths paired with slow paths, foreground processes paired with background ones, and routine examples throughout operating systems and web browsers. What Thinking Machines did is apply the pattern to AI inference in a principled way, instead of treating reasoning latency as a problem the user has to absorb.

Capabilities

All of these design choices add up. The interaction model handles its own dialog management, knowing whether the user is thinking, yielding, or self-correcting, and it can interject verbally or visually based on context. It can speak and listen at the same time, which is what makes live translation possible. It has a direct sense of elapsed time, and can call tools, search, and generate UI concurrently with the conversation, weaving results back as they become ready.

These claims need evidence. Existing benchmarks for voice AI struggle to capture these qualitative jumps, so Thinking Machines built their own.

• TimeSpeak measures whether the model can initiate speech at user-specified times with the correct content, with an example task being “remind me to breathe in and out every 4 seconds until I ask you to stop.”

• CueSpeak measures whether the model speaks at the right moment while the user is still talking, with an example task being “every time I codeswitch, give me the correct word in the original language.”

• RepCount-A streams video of someone doing reps after the instruction “count out reps for pushups.”

• ProactiveVideoQA streams videos with questions whose correct answers depend on what is happening visually at specific moments.

The result is striking. Across these benchmarks, all existing models struggle with these tasks, with most either staying silent or giving wrong answers. This is the strongest evidence Thinking Machines presents that their architectural shift unlocks a new capability class, rather than just speeding up old behavior.

Limitations

Despite the encouraging results, the research also points out the things that are still hard.

• Long sessions remain a real challenge for this architecture. Continuous audio and video accumulate context very quickly. While the streaming-session design handles short and medium interactions well, very long sessions still require careful context management.

• Connectivity remains a hard requirement, since streaming audio and video at low latency demands a reliable internet connection. A poor connection causes the experience to degrade significantly.

• Scaling the model size is constrained by latency targets, with TML-Interaction-Small being the size it is, partly because Thinking Machines’ larger pretrained models are currently too slow to serve in this setting.

Conclusion

Looking back, the main argument is simple. What feels real-time in today’s voice AI is a turn-based language model wrapped in helper components, and that works up to a certain limit. Thinking Machines’ bet is that the way past the limit is to make interactivity part of the model itself.

Two architectural choices carry most of the heavy work:

• Time-aligned micro-turns slice time into 200-millisecond chunks, letting the model handle input and output as continuous streams.

• The two-model split pairs a fast interaction model with a slower background model that handles deep reasoning, with both sharing context.

The evidence that this is a new capability class rather than just lower latency comes from the benchmarks Thinking Machines built themselves. Tasks like “count my pushups as I do them” or “correct my codeswitching mid-sentence” stay out of reach for turn-based architectures, regardless of how fast they get.

An important takeaway is that adding a capability through external scaffolding creates a ceiling on how good that capability can get, with the scaffolding becoming the bottleneck rather than the underlying system. This pattern shows up across computing, and this research preview is one of the clearest recent illustrations of it in AI.

Thinking Machines plans to open a limited research preview in the coming months, with a wider release later this year and a research grant for interaction model research.

References:

• Interaction Models: A Scalable Approach to Human-AI Collaboration

When devs use AI to generate thousands of lines of unverified code, you risk a codebase slopocalypse. The review step becomes your team’s bottleneck, and the last thing standing between a subtle bug and production.

Greptile reviews each PR with full repo context and learns your team’s conventions over time from comments, reactions, and what gets merged. It flags real issues and suggests fixes that match your team, not generic best practices.

✅ Recently launched TREX runs your code, not just reads it. Greptile executes the change in a sandbox and returns screenshots, logs, and traces as proof of what actually broke.

✅ Review from your terminal. The Greptile CLI runs the same review locally, before you ever open a PR.

✅ Trusted by engineering teams at NVIDIA, Scale AI, and Brex.

✅ Now integrated with Claude Code: install via /plugin.

✅ Free for open source.

See why devs love Greptile →

Even the most sophisticated AI agent in your stack starts every single message from a blank slate.

The model itself sees only the text placed in front of it at that exact moment, and the rest of the conversation lives outside its awareness entirely. Whatever continuity we feel when chatting with Claude or ChatGPT is something the surrounding platform is engineering on the model’s behalf, by inserting the right context back into every call. Once we understand this crucial distinction, the entire field of agent memory becomes a very different engineering problem from what it first appears to be.

In this article, we will try to understand how that architecture gets built, from the constraint that forces it to exist all the way to the tradeoffs that follow.

Disclaimer: This post is based on publicly shared details from various sources. Please comment if you notice any inaccuracies.

Statelessness

A call to a large language model follows a simple pattern. The system sends a prompt, the model returns a response, and the exchange concludes there. Each subsequent call, even one made a millisecond later, begins from a completely fresh slate. This is the API contract for every commercial LLM and reflects how transformers serve traffic.

See the diagram below:

When we say things like “Claude remembers our conversation from yesterday,” we are describing a property of the product rather than a property of the actual model itself. The platform writes things down on the model’s behalf, then reads them back into the prompt at exactly the right moment, so the model can reason as if it had been there all along. The intelligence resides within the model itself, while the memory resides in the system surrounding it.

If the model itself works this way, the next question is whether we can solve the memory problem by writing everything into the model’s view on every call? The way this approach breaks gives us a better insight into how to solve this problem.

Context

Every API call has a context window. It is basically the bounded slab of text that the model reads when generating its response. This includes the system prompt, the user’s current message, and anything else the developer has placed there. The model has full visibility into the contents of the window, while whatever sits outside it might as well live on a different machine.

See the diagram below:

An obvious approach to memory involves writing the entire conversation history into the context window on every call. This works well for the first few turns of a chat. Once the conversation grows longer, however, three significant problems emerge at once:

• The first problem is cost. Every token in the context window is paid for on each call, in both money and latency, so a linearly growing conversation produces a linearly growing bill. By message eighty, the system might be re-sending tens of thousands of tokens on every turn just to maintain continuity.

• The second problem is latency. Larger contexts take longer to process, and a model that responds in two seconds on a short prompt may take ten or fifteen on one that has filled most of its window.

• The third problem is the most counterintuitive of the three. The model’s attention degrades inside long contexts, and information placed in the middle of a long prompt is recalled less reliably than information at the beginning or the end. Researchers refer to this as the lost-in-the-middle effect.

Bigger context windows feel like they should solve memory entirely. In reality, they expand the room while leaving the navigation problem fully intact. Important information can sit right inside the window and still get ignored by the model.

Since a bigger window alone is the wrong tool, we need an architecture that decides what belongs in the window at any given moment.

When Rules Fail, AI Picks Up the Slack (Sponsored)

What happens when deterministic code hits the edge of its knowledge? In this live webinar, you’ll see a working plant health monitor built on Temporal’s entity workflow pattern where each plant is a long-running, crash-proof workflow that polls sensors, fires alerts, and falls back to GPT-4o only when the rules run out.

The architecture is clean: structured data first, AI second. The boundary is auditable. The state survives everything. Whether you’re building patient monitors, supply chain detectors, or any long-running process that occasionally needs a smarter answer, the patterns here translate directly.

Join us on July 9th

Hierarchy

Real production systems organize memory in tiers, each trading off speed of access, total capacity, and cost per token. The context window sits at the top, with progressively slower, larger, and cheaper stores below it.

The analogy to operating system memory is evident. Modern agent memory systems draw heavily on the way operating systems page data between fast RAM and slower disk, promoting and demoting information as its relevance rises and falls.

A typical four-tier hierarchy starts with the context window at the top, fast and tightly bounded, with every token expensive at scale. Below it sits short-term or session memory, holding recent activity that has yet to be summarized or evicted. Beneath that is the long-term store, where persistent facts, embeddings, and structured summaries live across sessions. At the bottom is the cold archive, used for rarely-accessed material kept for audit or future reference.

Information moves up and down this hierarchy as the agent works. A fact stated three sessions ago might be sitting in the long-term store, and the moment it becomes relevant, the system retrieves it and promotes it back into the context window. Conversely, when a session ends, the most useful parts of the context window get summarized and written down into the lower tiers.

ChatGPT’s memory feature uses a simple version of this idea. Stored user facts and summaries of recent conversations get prepended to every new prompt, while the current session occupies the working tier. The sophistication lies in what gets promoted to long-term memory in the first place, rather than in elaborate retrieval at read time.

A hierarchy describes where memory lives. The kinds of memory we are storing are a separate question, organized along a different axis.

Types

The field has converged on four functional categories of agent memory, drawing on concepts from cognitive science adapted for language model agents:

• Working memory holds whatever sits in the live context window for the current task. For example, if the agent is helping us debug a function right now, the function code and our recent messages occupy working memory. The moment this task ends, working memory clears.

• Episodic memory holds records of specific past interactions, anchored in time. A statement like “Three days ago, this user asked about onboarding new engineers, and we discussed checklist templates” represents an episodic memory, capturing a particular event with its context.

• Semantic memory stores facts and knowledge that stand independent of any specific interaction. Statements like “Adam prefers Python over JavaScript” and “his team uses GitHub Actions for CI” function as semantic memories, surviving across sessions and applying wherever they are relevant.

• Procedural memory captures learned ways of doing things. If the agent has figured out that this user prefers a three-section format for status updates, that preference becomes procedural memory, and the next time a status update is requested, the agent applies the format automatically.

These four types are orthogonal to the hierarchy from the previous section.

A piece of semantic memory might live physically in the long-term store and get pulled into the context window the moment it becomes relevant. Most production agents implement at least three of these types, with the mix depending on what the agent is built to do. A customer support agent leans heavily on episodic and semantic memory, while a coding agent leans more on procedural memory. The right blend is a design choice rather than a fixed formula.

All of this still leaves one important question open.

How does the agent actually decide, on each turn, what to retrieve and place in the model’s view?

Retrieval

Storage can possibly be thought of as the easy half of agent memory. Writing a fact to a database or indexing a summary in a vector store are solved problems with mature tooling. The harder half is retrieval, which is the act of deciding on every new turn what belongs in the model’s awareness.

Retrieval is hard because it requires judgment about relevance, and relevance shifts from one message to the next. For example, the user’s preference for Python matters when they are asking about a new project, while that same preference fades in importance when they are asking about pizza recipes. A good retrieval system surfaces relevant items at exactly the moment they are useful, and leaves the rest sitting quietly in storage.

See the diagram below:

The full loop runs on every user message.

The user sends a message, and the system retrieves relevant items from each memory tier using a mix of keyword search, semantic similarity, and recency signals. It assembles a context window in a deliberate order, often placing the most important material near the beginning and the end where the model’s attention is strongest. The model runs and returns a response, after which the system writes part of the new exchange back into memory, often as a summary, sometimes with a decay score so that importance fades over time.

Consider two agents to see why retrieval matters so much.

The first has a perfect database of past interactions paired with a retrieval system that frequently picks the wrong record. The second has an empty memory store and operates only on what the user tells it in the current session. The second often outperforms the first, because it understands the bounds of what it can rely on, while the first surfaces stale or irrelevant information confidently and reasons on top of it as though it were ground truth.

In other words, memory failures in production are typically retrieval failures in disguise.

Tradeoffs

The memory architecture described above involves several tradeoffs that engineering teams must navigate carefully. Four stand out as particularly important:

• Recency versus relevance: Do we retrieve the most recent items from memory, or the most semantically similar ones? Most systems do both, and blending them well is an ongoing engineering puzzle. Leaning too far on recency means the agent forgets useful older context, while leaning too far on relevance means the agent fixates on a strong-but-stale match.

• Summarization versus fidelity: Compressing old context into summaries saves tokens, which makes everything cheaper and faster. The compression is also lossy, and the loss is uneven. Names, dates, and specific commitments get smoothed away in summarization, while general themes survive. The agent stays confident even after the precise detail has quietly disappeared.

• Staleness: A fact that was true six months ago can be confidently wrong today. For example, the user who may have told the agent “I am a vegetarian” in 2024 might be eating non-vegetarian again in 2026. The memory system has only blunt heuristics for guessing that the world has moved on, so it keeps serving the old fact with full confidence. Staleness in high-relevance memories remains an open research problem.

• Memory poisoning: Long-term memory is also a long-term attack surface. A subtly malicious instruction written into the store six months ago will sit there influencing every retrieval until somebody notices. The same property that makes memory useful, persistence, also makes it dangerous when the contents are wrong or hostile.

A memory system makes sense when the agent needs continuity across sessions or runs long-horizon tasks where context compounds. For one-shot tasks, it adds complexity beyond what the task requires.

Conclusion

Five core ideas from this article are as follows:

• The model is stateless. Every API call begins from a fresh slate, and any continuity we observe is the work of the surrounding system rather than the model itself.

• The context window is the model’s only surface of awareness. Writing everything into it fails for reasons of cost, latency, and degraded attention in long prompts.

• Real systems organize memory in a hierarchy of tiers, with the context window at the top and progressively slower, larger, cheaper stores below it.

• Different kinds of information call for different kinds of memory, with working, episodic, semantic, and procedural memory each serving a distinct purpose.

• The main engineering problem in this area is retrieval, which is the question of what deserves to enter the model’s awareness on each new turn, with tradeoffs around staleness, summarization loss, and security.

The practical takeaway from all of this is simple.

The next time we see a product feature labeled “memory” or read about an agent that “remembers across sessions,” the right question moves away from “can the model remember this?” and toward “what does the memory architecture around this model actually do, and what tradeoffs has it accepted?”

References:

• Lost in the Middle: How Language Models Use Long Contexts

• MemGPT: Towards LLMs as Operating Systems

• Cognitive Architectures for Language Agents

• Memory and new controls for ChatGPT

If slow QA processes bottleneck you or your software engineering team and you’re releasing slower because of it — you need to check out QA Wolf.

QA Wolf’s AI-native service supports web and mobile apps, delivering 80% automated test coverage in weeks and helping teams ship 5x faster by reducing QA cycles to minutes.

QA Wolf takes testing off your plate. They can get you:

• Unlimited parallel test runs for mobile and web apps

• 24-hour maintenance and on-demand test creation

• Human-verified bug reports sent directly to your team

• Zero flakes guarantee

The benefit? No more manual E2E testing. No more slow QA cycles. No more bugs reaching production.

With QA Wolf, Drata’s team of 80+ engineers achieved 4x more test cases and 86% faster QA cycles.

Schedule a demo to learn more

This week’s system design refresher:

• RAG vs Graph RAG vs Agentic RAG

• Redis Data Structures Every Engineer Should Know

• API Security Best Practices

• Design Patterns Cheat Sheet

• The Testing Pyramid

RAG vs Graph RAG vs Agentic RAG

RAG connects LLMs to your data and there are three different ways to do it.

Standard RAG

• The query is converted into an embedding and matched against a vector database.

• The top-K closest chunks are pulled out and passed to the LLM as context.

• The LLM writes a grounded answer using only what was retrieved.

Graph RAG

• The query is classified: specific questions route to local search, broad questions route to global search.

• Local search: query embedded → vector DB finds matching entities → pipeline traverses across the knowledge graph collecting linked context → LLM synthesis final answer.

• Global search: no vector search, no graph traversal → community reports loaded in batches → LLM scores each for relevance → top-ranked context → LLM synthesizes final response.

Agentic RAG

• A reasoning agent reads the query, breaks it into sub-questions and picks the sources.

• The context across multiple sources is retrieved, depending on the sub-query.

• Another agent checks whether the retrieved context answers the question. If not, it re-retrieves.

• Once satisfied, the final answer is synthesized by LLM based on the prompt.

Standard RAG is fast and cheap but if the wrong chunk is retrieved, the answer is wrong and nothing catches it.Use it when the answer lives in your documents and speed matters.

Graph RAG is expensive to build and slow to update. Use it for structured knowledge like legal, compliance, or biomedical data.

Agentic RAG is more capable and flexible but slower, expensive, and harder to debug. Use it when the question needs multi-step reasoning and self-correction.
Over to you: Which of these are you running in production?

Redis Data Structures Every Engineer Should Know

• Strings store one value per key. They work for counters, session tokens, and cached payloads.

• Hashes store an object's fields under one key. You can update one field without rewriting the rest.

• Lists are ordered sequences with fast push and pop at both ends. They fit queues, feeds, and recent-item lists.

• Sets hold unique members and support intersection, union, and difference. They cover tagging, follower overlap, and deduplication.

• Sorted Sets rank members by a numeric score. They handle leaderboards, priority queues, and top-N or range-by-score queries.

• Streams are an append-only log with consumer groups. Each consumer tracks its own position, and the server tracks unacknowledged messages.

• JSON stores nested documents with JSONPath access. You can update a field deep in a document without read-modify-write.

• Geospatial provides latitude/longitude indexes with radius and box queries. Under the hood it's a Sorted Set with geohash scores.

• Vector Set runs approximate nearest-neighbor search over embeddings. It's the retrieval step in most RAG pipelines.

• Time Series stores timestamped samples with built-in retention, downsampling, and labels. It fits metrics, telemetry, and IoT data.

Over to you: All ten are built-in as of Redis 8. Which one do you use most outside of caching?

API Security Best Practices

Most API breaches happen because of broken authorization, leaked secrets, or missing rate limits. Let's look at some of the basics.

• Use Modern OAuth/OIDC + MFA: PKCE for public clients, short-lived tokens, and step-up MFA for anything sensitive. Implicit and password grants should be dead by now.

• Enforce Fine-Grained Authorization: Check object, function, and field-level permissions on every request. BOLA is still the top API vulnerability.

• Minimize Scopes and Data: Give each client the smallest token scope and the least data it needs. Only return the fields the caller actually needs.

• Encrypt Every Hop: TLS for external traffic and mTLS between services. If it crosses a network boundary, encrypt it.

• Protect Secrets and Keys: Store signing keys in HSM-backed vaults. Rotate them.

• Validate Requests with Schemas: Reject unknown fields, oversized payloads, and suspicious URLs at the gateway. Don't let bad input reach your business logic.

• Rate Limit and Cap Resources: Quotas per user, payload size caps, and execution timeouts. Without these, one misbehaving client takes down your entire system.

• Defend Sensitive Business Flows: Protect login, checkout, and OTP with anti-bot, idempotency keys, and step-up auth.

• Control Outbound and Third-Party Calls: Allowlist where your API can call out to and block internal metadata endpoints. Your security is only as strong as your weakest integration.

• Harden Config and Error Handling: Deny by default on CORS, methods, and debug endpoints. Return generic errors, never stack traces.

• Inventory APIs and Versions: Track every endpoint, version, and shadow API. You can't secure what you don't know exists.

• Log, Detect, and Respond: Push auth decisions and anomalies to a SIEM. Alert on 401 spikes before they become incidents.

Over to you: Which of these best practices is the hardest to enforce across your services?

Design Patterns Cheat Sheet

The cheat sheet briefly explains each pattern and how to use it.

What's included?

• Factory

• Builder

• Prototype

• Singleton

• Chain of Responsibility

• And many more!

The Testing Pyramid

Testing is the backbone of reliable software. The Testing Pyramid is a widely accepted strategy for structuring tests into three key layers:

• Unit Tests: These are the foundation of the pyramid. Unit tests are fast, isolated, and low-cost to write and maintain. They test individual functions, methods, or components.

• Integration Tests: These tests validate interactions between components, such as APIs, databases, and external services. They are slower than unit tests and require more setup.

• E2E Tests: These simulate real user flows from start to finish across the full system. They are expensive to write and maintain and tend to be slow to execute.

As you go up the pyramid, the cost of test development, execution, and maintenance increases.

Over to you: Which layer do you find most valuable in your testing strategy, and why?

How many decisions does it take to reach that point?

It rarely takes a single bad one. The path to such a situation is built from individually sound choices, a clean separation here, an independent deployment there, a new service each time a part of the system felt distinct enough to stand on its own. Those reasonable steps accumulate into an arrangement no one would have chosen on purpose, and the problems that emerge look like a catalog of separate mistakes even though nearly all of them trace back to one early decision about how to break a system.

At a basic level, a service is a part of a system that can be deployed on its own and controls its own data. This means it does not reach into any other service’s database to do its work. It also talks to other services over a network. Inside a single program, one function calling another takes a few nanoseconds and either returns an answer or raises an error. The same call across a service boundary can take a few milliseconds, and it can also time out or succeed halfway and leave things in an odd state. Almost every anti-pattern below emerges from this one problem.

In this article, we will look at some of the most important anti-patterns in service architecture, how they happen, and how they can be avoided.

Splitting Early

Read more

AI-assisted development has changed how code gets written, but for many teams, testing and governance haven’t caught up. Tricentis AI Workspace closes that gap, giving quality engineering leaders one place to build, orchestrate, and govern AI quality agents across the SDLC, from code risk analysis and test automation to performance validation so quality decisions happen continuously, not at the end. Less errors introduced by AI-generated code, more confidence in what you’re shipping.

Discover how teams are using AI Workspace to bring structure to AI-driven development and compress delivery timelines without sacrificing confidence in business outcomes.

See Tricentis AI Workspace in action

Apple’s most ambitious AI feature runs in about a gigabyte of memory on the iPhone. The same company runs a much larger model on its own cloud servers, and the two diverge in almost every architectural choice beyond the word “transformer” in their lineage.

The same split shows up at Google, Microsoft, and Meta, where one family of small models targets devices and a different family of large models targets data centers.

Small and large language models are different engineering responses to different constraints, and the differences begin with where each model runs, what hardware it targets, and how it was trained.

In this article, we will explore those constraints through three layers of model design, look at the tradeoffs that come with each approach, and investigate the production systems that combine both small and large models.

Disclaimer: This post is based on publicly shared details from various sources. Please comment if you notice any inaccuracies.

Foundations

Before we look at what makes the two classes different, it helps to be precise about what makes them the same.

Both small and large language models are transformer-based decoder models, built by stacking layers of the same basic computational block. Each block runs an attention operation, which figures out which previous tokens matter most for predicting the next one, followed by a feed-forward computation that mixes that information through a wide intermediate layer. The model repeats this block thirty or more times before producing a probability distribution over what the next token should be.

Both classes go through similar training stages. They start with pretraining on large text corpora, where the model learns to predict the next token across billions of examples. They typically follow with supervised fine-tuning on specific instruction patterns, and many go through reinforcement learning from human feedback, which shapes how the model handles ambiguity and stays helpful in conversation.

The size of a model refers to its number of parameters, which are the learned weights adjusted during training. A small model in 2026 typically has between half a billion and fourteen billion parameters. A large model has tens of billions to hundreds of billions of parameters, and sometimes more.

Constraints

Three constraints pull the designs of small and large models in opposite directions.

• Deployment target: Where the model runs determines its memory, battery, and latency budgets.

• Inference economics: Training is paid once, but serving is paid per request, which inverts the math at scale.

• Training budget: Smaller budgets push teams toward efficiency through data quality and distillation rather than raw scale.

The deployment target determines everything that follows.

A model that runs on a phone has a memory budget measured in single gigabytes, a battery budget measured in milliamps, and a latency budget measured in milliseconds. A model that runs in a data center operates in a more permissive environment, with concerns around throughput, batching efficiency, and cost per request, but with an absolute resource ceiling orders of magnitude higher.

Inference economics is the second pressure.

Training a model is a one-time cost paid at the start of its life, while serving the model is a recurring cost paid every time someone uses it. For a high-volume product, the inference bill quickly dwarfs the training bill, so a team designing for high inference volume will gladly spend more training compute upfront to save inference compute across billions of requests downstream.

The training budget is the third pressure.

A frontier large model can cost tens of millions of dollars to train, while most teams working on small models operate with a small fraction of that, and the smaller budget forces choices. Those teams have to find other levers beyond raw scale, which usually means smarter training data, distillation from larger teachers, and more efficient training recipes.

These three constraints reinforce each other rather than acting in isolation. A model designed for the phone has a small inference budget per request and usually a smaller training budget too, while a model designed for the data center has the opposite profile across all three axes. The result is two distinct design regions in the same space.

Who’s actually reviewing all that AI-generated code? (Sponsored)

When devs use AI to generate thousands of lines of unverified code, you risk a codebase slopocalypse. The review step becomes your team’s bottleneck, and the last thing standing between a subtle bug and production.

Greptile reviews each PR with full repo context and learns your team’s conventions over time from comments, reactions, and what gets merged. It flags real issues and suggests fixes that match your team, not generic best practices.

✅ Recently launched TREX runs your code, not just reads it. Greptile executes the change in a sandbox and returns screenshots, logs, and traces as proof of what actually broke.

✅ Review from your terminal. The Greptile CLI runs the same review locally, before you ever open a PR.

✅ Trusted by engineering teams at NVIDIA, Scale AI, and Brex.

✅ Now integrated with Claude Code: install via /plugin.

✅ Free for open source.

See why devs love Greptile →

Architecture

The architecture differences begin with a quick observation about inference.

During generation, the model has to keep around the keys and values for every previous token, since attention works by comparing the current token against all earlier ones. This stored set is called the KV cache, and it grows linearly with the length of the conversation. For long generations, the cache often dominates memory bandwidth and storage, more than the parameters themselves.

This single fact decides how small-model architectures get designed.

In the original transformer design, every attention head has its own keys and values, an arrangement called multi-head attention. For long sequences, the resulting cache footprint grows large enough to dominate the model’s memory consumption.

Grouped-query attention attacks the problem directly. The number of query heads stays the same, but several queries share a single key-value pair. A model with thirty-two query heads might use only eight key-value groups, which cuts the cache footprint by a factor of four with minimal quality loss. Llama, Qwen, Gemma, and most modern small models use grouped-query attention by default, and many large models have adopted it as well because the math also helps at scale.

Some small models push further. Gemma 2 interleaves sliding window attention with full attention across layers, so some layers attend only to the most recent few thousand tokens rather than the full context. This trades a bit of long-range reasoning for a significantly smaller cache. Apple’s on-device model shares its KV cache across multiple decoder layers, reusing the same stored state in several places.

These architectural decisions all serve the same goal of shrinking the runtime cost of inference, which is the constraint that matters most when the model has to run on a device with a few gigabytes of memory to spare.

Training

Two models with identical architectures can end up with very different capabilities depending on what they were trained on and how.

Three techniques define the current state of the art in small-model training:

• Data curation: Carefully chosen and synthetically generated training data can substitute for raw volume.

• Knowledge distillation: A smaller student model learns from a larger teacher model’s output distribution.

• Overtraining: Modern small models see far more training tokens than compute-optimal ratios suggest, trading training cost for inference savings.

The first technique is data curation. In 2023, a Microsoft research team published a paper called “Textbooks Are All You Need.” They trained a 1.3 billion parameter coding model on roughly seven billion tokens of carefully filtered code and synthetically generated textbook-style data. The model matched or beat models trained on hundreds of billions of tokens of raw web scrape. Training data quality could substitute for training data volume, at least for certain capabilities. The Phi family kept building on that insight, and the modern Phi-4 model continues to lean heavily on synthetic data quality as its primary lever.

The second technique is knowledge distillation.

The small model, called the student, learns from a larger model, called the teacher, by mimicking the teacher’s output distribution rather than only learning from raw text. The richer training signal helps the student pick up patterns it would struggle to learn from the underlying corpus alone. Gemma 2 used this approach to train its nine billion parameter model, while training its twenty-seven billion parameter version from scratch.

The third technique is overtraining relative to compute-optimal.

In 2022, the Chinchilla paper from DeepMind established that for a fixed compute budget, the best model came from balancing parameter count and training data, roughly twenty tokens of training data per parameter. Modern small models deliberately train on far more data than that ratio suggests. A three-billion-parameter model might see many trillions of tokens during training, which is many times the Chinchilla-optimal amount. Once the model gets deployed, every percentage point of quality improvement saves inference compute across billions of requests, so the team spends more on training to save more on serving.

Deployment

The final layer of design choices covers how the model executes on real hardware. The two dominant techniques are quantization, which shrinks the storage cost of each parameter, and KV cache management, which shrinks the runtime cost of generation.

Quantization is the practice of storing each parameter with fewer bits. A standard pretrained model stores each parameter as a sixteen-bit floating point number, where cutting that to eight bits halves the memory footprint and cutting to four bits halves it again. The post-training approach is faster to implement but tends to lose quality at aggressive bit widths, while quantization-aware training preserves quality at the cost of more complex training.

Hardware mapping is the next consideration. Apple’s Neural Engine has different strengths from an NVIDIA Jetson, which has different strengths from a data center H100, and the model design follows the target hardware. Phi-4-mini gets tuned for consumer GPUs. Gemma 3 4B variants run on NVIDIA Jetson Orin for edge AI deployments in robotics and embedded systems. Apple’s 3B model runs on the iPhone’s Neural Engine with the assumption that the device also handles other workloads at the same time.

KV cache management is the second major lever, and it connects directly back to the architecture section. The cache stores keys and values for every previous token during generation, and its size determines how much memory the model utilizes at runtime. Grouped-query attention attacks this by reducing the number of key-value heads, and Apple’s on-device model goes further by sharing its cache across multiple decoder layers.

These deployment decisions stack on top of everything earlier. The same architectural choices that shrink the KV cache also make quantization easier, and the same training recipes that produce capable small models also produce models that survive aggressive compression.

Tradeoffs

Small models perform well on standard benchmarks like MMLU and HumanEval. Production usage looks more varied. Three gaps tend to matter most:

• Generalization gap: Small models are more brittle outside their training distribution.

• Reasoning gap: Multi-step problems still favor larger models, though the gap is closing.

• Knowledge ceiling: Parameters function as memory, so small models have a hard cap on what they can store.

The first gap is generalization.

Small models tend to be more brittle outside their training distribution, and they can be excellent at tasks similar to what they saw during training, while showing weakness on unexpected ones. A small model trained heavily on code performs well on code but may struggle with creative writing in an unusual style. A model trained on synthetic textbook data does well on textbook-style questions but can falter on the messy, ambiguous prompts that real users send.

The second gap is multi-step reasoning.

For problems that require chaining inference across many tokens, large models still have a noticeable advantage. The gap has been closing thanks to step-by-step reasoning techniques and reasoning-focused fine-tuning, but at very small parameter counts, the ceiling remains real. Phi-4 has done well on math reasoning specifically because Microsoft optimized for that capability through training data design, while a general-purpose small model usually shows a clearer gap.

The third gap is world knowledge.

Parameters function as a form of memory, and a larger model can store more facts, more named entities, more obscure references, and more multilingual coverage. A small model has a fundamental cap on how much it can know, since storage requires parameters and parameters require memory. For applications that need broad factual recall, the small model often pairs with an external knowledge source that the model queries when needed, since trying to fit all that knowledge into the parameters themselves would push the model past its budget.

Hybrids

The most interesting design question in 2026 is rarely which model to use. The more useful question is how to compose multiple models into a system that uses each for what it does best. Three patterns appear in most production setups.

• Routing: A small model handles requests directly and escalates harder ones to a large model.

• Guardrails: A small model filters input or output around the large model’s core work.

• Drafting: A small fast model generates candidate tokens that a larger model verifies in a batch.

The most common pattern is routing.

A small model handles the request directly if it falls within its competence, and escalates to a large model when the request is harder than it can confidently handle. The pattern resembles caching tiers in a distributed system, where the fast, cheap layer handles the common case, and the slower, more expensive layer handles the rest. The router itself is often a small classifier model that decides which path to take.

The second pattern is the guardrail.

A small model sits in front of the large model to filter or classify input before the expensive computation runs, checking for unsafe content, classifying the intent of the request, or stripping out information that should stay private. A second small model often sits on the output side, doing similar checks before the response gets returned to the user. These guardrail models are cheap, fast, and specialized, which makes them well-suited to the role.

The third pattern is the drafter, sometimes called speculative decoding.

A small fast model generates candidate tokens, and a larger, more capable model verifies them in batch. When the verifications agree, the system gets the throughput of the small model with the quality of the large one. Apple’s on-device system uses a draft model alongside its base model for exactly this reason. The technique sounds like a hack, but it has become standard in production inference systems.

Picking a model class is the wrong frame for most product decisions. Designing a system around multiple model classes is the right frame, and the interesting design work lives in the composition layer, the routing logic, the fallback behavior, and the orchestration between models.

Conclusion

The question we started with was “small versus large language models,” and the more useful version of that question turns out to be “which constraints drove each model’s design.” The size of a model is a downstream consequence of those constraints rather than the starting point for the design.

Three layers of design choices flow from the constraints:

• Architecture adapts through attention variants like grouped-query and sliding-window attention that shrink the KV cache.

• Training adapts through high-quality synthetic data, distillation from larger teachers, and deliberate overtraining relative to compute-optimal ratios.

• Deployment adapts through quantization, KV cache management, and careful hardware mapping. Each layer reinforces the others, and the result is two distinct design regions in the same space.

Small models are extremely capable for their size, and they have a real ceiling on generalization, on multi-step reasoning, and on broad world knowledge. Production systems handle this by composing both classes, using small models for the common case and large models for the harder requests, sometimes with multiple small models acting as routers, guardrails, and drafters around a larger core.

For a working engineer choosing between models, the right starting point is the constraints rather than the benchmark. The questions that matter are about deployment target, inference budget, and the shape of the request distribution in production.

References:

• Apple Intelligence Foundation Language Models Tech Report 2025

• Apple Intelligence Foundation Language Models

• Updates to Apple’s On-Device and Server Foundation Language Models

• Introducing Apple’s On-Device and Server Foundation Models

• Recurrent Drafter for Fast Speculative Decoding

• Recurrent Drafter

• Deploying Transformers on the Apple Neural Engine

• Textbooks Are All You Need (Phi-1)

• Textbooks Are All You Need

• Phi-4 Technical Report

• Phi-4 Technical Report

• Phi-4-mini-instruct model card

• Running Phi-4 Locally with Microsoft Foundry Local

• Gemma 2: Improving Open Language Models at a Practical Size

• Gemma 2 Technical Report

• Lightweight, Multimodal, Multilingual Gemma 3 Models Are Streamlined for Performance

• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

• GQA paper on ACL Anthology

• Fast Transformer Decoding: One Write-Head is All You Need (Multi-Query Attention)

• Training Compute-Optimal Large Language Models (Chinchilla)

• Chinchilla Scaling: A replication attempt

• Reconciling Kaplan and Chinchilla Scaling Laws

• Attention Is All You Need

• Distilling the Knowledge in a Neural Network