GPU-hour billing charges for the entire time a machine is reserved — setup, idle time, queueing, and failures — so your cost depends on infrastructure efficiency you don’t control. Crusoe Serverless Fine-Tuning bills per token processed during training, making costs predictable from your dataset size and epoch count, with no GPUs to provision or right-size. Early stopping ends the job, and the billing, the moment your model stops improving — you’re never charged for epochs that add no accuracy. Jobs run on AI-optimized infrastructure with automatic recovery and restart, and checkpoints save at every step.

Get started

AI labs are moving faster than ever and releasing the most capable models we have ever seen. Just recently, Anthropic released Fable 5, then OpenAI released the GPT-5.6 family, including GPT-5.6 Sol, their most capable model yet, Kimi released Kimi K3, and Opus 5 came out just a few days ago.

But capability is only half of the picture. The other half is how much it costs these models to complete tasks, that is, the cost per successful task. Lower cost makes the model more affordable for users and less costly for the provider. A huge amount of effort inside the labs goes into making every component and layer of their AI applications optimized and more efficient, to reduce the overall cost. For example, GPT 5.6 Sol with max reasoning scores higher than Fable 5 on the Artificial Analysis Coding Agent Index while costing less than half as much.

To understand what techniques are adopted in frontier labs to make AI applications more efficient, we met with the OpenAI engineers who developed and shipped various efficiency techniques into the systems behind Codex and ChatGPT Work. Thanks to Joe, Ahmed, Steve, Matthew, and Philippe for sharing valuable details with us.

In this article, you’ll learn:

• What actually happens when you send a request to an AI agent

• How the harness layer cuts repeated work with persistent WebSockets, stable prompt prefixes, deferred tool discovery, and Code Mode

• How the API layer tokenizes only the delta and runs safety checks in parallel with inference

• How the inference layer gets more out of GPUs with cache-aware routing, KV cache management, speculative decoding, and separating prefill from decode

• What OpenAI learned from all of this, and which lessons you can apply to your own systems

Anatomy of an Agentic AI Application

When you give Codex or ChatGPT Work a task, like fixing a bug, the query does not go directly to the LLM. It is more complex than that. An AI application like Codex is not just an LLM. It is a system, with many components. User queries pass through several layers before the LLM ever sees a single token.

But why can’t we send the user query directly to the LLM? To understand why, let’s see what an LLM actually is.

An LLM is a neural network trained to predict the next token. It takes a sequence of tokens as input and produces a sequence of tokens as output. It cannot run a shell command, edit a file, or remember anything between calls. But an agent task like “fix this bug and run the tests” is mostly actions. Something has to turn the model’s predicted tokens into real commands, feed the results back, and continue until the task is done.

That is the job of the harness layer, a system built on top of the LLM to handle those responsibilities. It takes the user’s task as input, decides which instructions, which tool definitions, and how much history to include in the context, and maintains the conversation history. When the model responds with a tool call, the harness executes it under approval policies in a sandbox environment, appends the result, and sends the conversation back to the LLM.

But even the harness’s request does not go directly to the LLM. In production-ready applications, there is an API layer (an application layer) sitting between the harness and the inference endpoint. This layer exists because a real product needs to handle things that are not covered by the harness layer or the LLM. For example, authenticating the caller, enforcing rate limits, and, importantly, converting text into the token IDs the LLM expects, and converting generated tokens back into text.

Once the API layer prepares the context and tokenizes it into a sequence of token IDs, it is time for the LLM to process the input. That is the inference layer. It is a remote endpoint backed by fleets of GPUs, hosting the model. Its job is to run the model’s computation over the prepared tokens and produce the response, whether that is a new tool call or the final answer, as efficiently as possible. Then hand the generated tokens back to the API layer.

What happens when you send a request to an AI agent?

Now that we understand the three main layers of agentic AI applications, let’s walk through a concrete example to see how a request travels between these layers. Suppose you ask Codex to “trace the checkout regression, patch it, and run the tests.” Here is what happens:

1. The harness assembles instructions, tool definitions, and your task into a request and sends it to the API.

2. The API buffers the request into memory, parses the JSON, and validates it: the request is well-formed, and the chosen model supports every feature it asks for. It checks who is calling, applies rate limits, and runs preflight checks.

3. The API renders the conversation into the model’s input format and tokenizes it.

4. The API dispatches the tokens to the inference layer, and starts its safety checks at the same time: classifiers that look for things like cyberattacks and bioweapon content. The safety checks race to finish before the first generated token comes back.

5. The inference layer processes the prompt and begins generating. Its output here is not the final answer; it is a tool call: search the code for “checkout timeout.”

6. The generated tokens flow back to the API

7. The API converts tokens into text, wraps them in API events

8. The API streams them to the harness.

9. The harness recognizes the tool call, runs the search in its sandbox, appends the output to the conversation, and sends the updated conversation back to the API.

This is just one iteration. In practice, a task repeats this loop many times until the model produces its final summary and the harness hands control back to the user. Each iteration redoes a lot of the same work described above. For example, the history gets resent, the text gets retokenized, or the prompt gets reprocessed. Removing this repeated work is the main opportunity for optimization. The next three sections cover the optimization techniques OpenAI adopted to make each layer more efficient.

Harness Optimization

The harness is the orchestration layer closest to the user that turns the raw user request into a context and interacts with the LLM in a loop until the task is complete. It has two main responsibilities.

First, it is the source of truth for the conversation. It holds the authoritative record of every instruction, message, tool call, and tool result. Second, it runs the agentic loop. It decides what goes into the context sent to the LLM (which instructions, which tool definitions, and how much history). It sends the request, receives the response as a stream, parses it, and watches for tool calls. When a tool call appears, it executes it under approval policies in a sandbox environment, appends the result to the conversation, and sends it back to the LLM. It repeats this loop until the task is done. Heavy tasks could in theory repeat more than 100 times. Each iteration carries overhead. For example, one extra second per model call adds roughly half a minute to a long task.

How to make the harness layer efficient?

From the harness layer’s perspective, the latency comes from four sources: the network, prompt processing, context contents, and the loop’s round trips. OpenAI adopted the following four techniques at the harness layer to make it more efficient.

1. Persistent WebSockets and incremental requests

The harness runs on the user’s machine, but the model runs in OpenAI’s data centers, so every model call is a network exchange. The standard way to make that exchange is HTTPS. The harness opens a connection, sends a request containing everything the server (the API layer) needs, and receives a response. Because responses from a language model arrive gradually, one token at a time, chat applications typically use Server-Sent Events (SSE) on top of HTTPS, so the client sends one request, and the server streams the answer back in small chunks over the open response. SSE is a one-way street. It is a great fit for the chat era, where one user message produces one model call and one streamed answer. But agents are not one-pass. A single Codex turn can contain many model calls. With HTTPS, each of those calls is a new request with two separate costs.

The first cost is connection setup. Opening a fresh HTTPS connection means a TCP handshake followed by a TLS handshake, several network round trips before a byte of useful payload is transmitted. Paying that once per user message is manageable, but paying it many times inside one conversation turn is expensive.

The second cost is repetition of the payload itself. HTTP is stateless, so each request must carry everything the server needs. For example, the instructions, tool definitions, and the entire conversation so far, all need to be included in the request. By the twentieth tool call, the harness is resending the original prompt, nineteen tool calls, and nineteen tool results, just to add one new result at the end. This causes the payload to grow with every iteration, so the harness spends more and more time uploading data the server has already seen.

But how do we fix these two sources of cost?

The fix for the connection cost is to open one connection and keep it alive, instead of creating a new one for every call. This is what WebSockets are designed for. A WebSocket needs just one initial handshake, and after that, both sides can send messages whenever they want with no per-message setup. Codex opens a single WebSocket to the API and keeps it open across all the model calls in a turn. This eliminates the repeated TCP and TLS setup.

The fix for the payload repetition is to stop resending what the server already knows. The harness keeps the previous request and the completed response. If nothing but the conversation input has changed, it sends only the new items along with a reference to the previous response. After a tool call, the next message on the socket can be as small as:

{
  “type”: “response.create”,
  “previous_response_id”: “resp_abc123”,
  “input”: [
    {
      “type”: “function_call_output”,
      “call_id”: “call_xyz”,
      “output”: “...the new tool result...”
    }
  ]
}

Notice that this message has no instructions, tool schemas, or history. The server will rebuild the full context from the state it kept, s

o the model still sees everything. The only thing that changed is how much data crossed the network.

2. Stable prompt prefixes

LLM providers avoid repeated calculations with a technique called prompt caching. When a prompt arrives, the model reuses the cached internal state for the beginning of the prompt that matches a previous request (the prefix), and only computes the rest. The match is exact, token by token. If you change one token near the front of the prompt, everything after it must be recomputed.

For the harness, this means the prompt it builds on every call must start with exactly the same bytes as the one before. That sounds trivial, but the harness rebuilds the request each time from its live in-memory state, so any small difference in how it assembles the prompt may silently break the match. OpenAI shared an example of this. Codex kept MCP tool definitions in a hash map, which does not guarantee ordering, so the same tools could serialize in a different order on each request. It was the same tools in the context, just in a different order. Codex was still completing tasks, just more expensively.

The efficiency technique here is to treat the history as append-only, and keep volatile runtime state like approval settings out of the prompt. For example, when a user changes approval settings mid-session, the harness does not edit the tool definitions in the prompt. It applies the new policy itself the next time a tool runs. This way, the prompt stays unchanged so the cache stays valid.

3. Deferred tool discovery

Stable prompt prefixes make repeated context cheaper, but they do not keep the prompt small and concise. In agents with access to lots of tools, this can be an issue. For example, the tool schemas alone can take up a huge amount of space, because a Codex session can expose hundreds of tools once you count connected integrations and MCP servers. Every tool schema is a JSON object with names, descriptions, and parameter definitions. Once all of these are in the prompt, the LLM has to process all of them in its calculations, even though many may go unused for the current task.

To keep unused tools out of the prompt, OpenAI uses a technique called deferred discovery. The prompt carries only the core tools, say the shell and file editing, plus one tool_search tool. The other hundreds of integration definitions stay out of the prompt entirely. When the model needs a capability, it calls the search tool with keywords like “list deployments”. The harness searches the catalog, and the matching definition gets loaded into the context. The search itself is BM25, a lexical ranking algorithm that finds tools whose descriptions overlap with the generated keywords.

In addition to deferred discovery, two more techniques can keep the context small. Schema compaction trims oversized tool schemas down to a token budget by stripping descriptions and collapsing nested structures while preserving argument names. Conversation compaction condenses a long history into a short summary.

4. Code Mode

In ordinary setups, the model emits one tool call, waits for the result, reasons, and emits the next. In many scenarios, there is no reasoning needed between these tool calls. The model just emits them one by one to collect the results it needs. This is costly because each tool call requires a full model round trip. Also, after each tool call, its result keeps getting added to the context, which wastes context space.

With Code Mode, instead of emitting tool calls one by one, the model writes a small program that makes those calls. The harness runs this program in an embedded JavaScript runtime, where every tool is available as a function. The script can fan out independent calls in parallel, filter and join the results in plain code, and return only the compact answer. This way, the intermediate data stays in the runtime, and only the final result enters the context.

API Optimization

The API layer is the service the harness is talking to. It sits between the harness layer and the inference layer, running on ordinary CPUs. The harness sends JSON describing structured conversation items, while the model consumes and produces token IDs. When a request arrives, it goes through the following steps:

1. The request body is buffered into memory.

2. The JSON is parsed and validated against a schema (well-formed fields, arrays within expected bounds, etc.)

3. A second validation round checks semantics. For example, does the chosen model support the requested features?

4. Preflight checks run: authorization and entitlements, rate limits, and checks on any images in the request.

5. The API renders and tokenizes the conversation, then kicks off two jobs at the same time: the inference request to the GPUs and the safety checks on the prompt.

6. As generated tokens flow back from the inference layer, the API converts each one into text, wraps it in an API event, and streams it to the client.

These steps are repeated on each iteration of the loop. The inference is out of the API layer’s control. The API cannot make the GPU any faster. All it can do is add as little delay as possible around it. The time the API layer spends parsing, validating, or tokenizing is extra latency the user feels.

How to make the API layer efficient?

OpenAI adopted the following techniques to reduce the overhead coming from different places in the API layer.

1. Tokenizing only the delta

Models do not read text. Before inference starts, the prompt must be converted into token IDs. Tokenizers convert the text into token IDs in O(n), linearly going over each token in the prompt and replacing it with the token ID. Under stateless HTTP, the whole conversation gets retokenized on every call. By the twentieth tool call, the API is rereading many thousands of tokens to extract one new result. The GPU only needs the new tool result, but the CPU is rereading the book from page one, and the cost keeps growing with input length.

With the WebSocket, the API keeps the tokenized conversation in server memory. The first request tokenizes the full prompt, but every later request sends only the new items. The API tokenizes just that piece and appends it to the stored sequence. This way, per-call tokenization stops depending on conversation length and becomes closer to O(1), depending on the length of the marginal input.

2. Running safety checks in parallel with inference

Agents have to check each request for safety before showing the output to a user. OpenAI implemented safety checks in its API layer. Images go through their own classifiers, while the prompt text goes to classifier models that look for dangerous content, like cyberoffense or bioweapon instructions. These classifiers take time to run. The simple design is to run them first and start inference only after they pass. The problem is that this delay is added to the time-to-first-token (TTFT) of all requests, while the vast majority of them are completely harmless.

The fix is to run the safety checks and inference at the same time. The model takes some time to process the prompt before its first token comes out, so the checks use that window to finish. If a check fails, the API reacts depending on the model. For some models, it streams tokens to the user right away and cuts the stream when a check fails. For more sensitive models, it holds the output until the checks pass and only then releases it. In both cases, the time spent on safety hides inside a wait that was going to happen anyway.

3. Routing traffic to newer CPUs

When a company runs servers at scale, its fleet ends up with machines bought in different years, and therefore with different generations of CPUs. The scheduling software usually treats machines of the same type as identical. In reality, they are not. OpenAI queried Kubernetes for the actual processor model behind each of its deployments and found a mix of older Broadwell chips and newer Ice Lake chips behind the same machine labels. The older ones were serving the same traffic with roughly 20% worse time to first token, while using about twice as much CPU.

To fix this, OpenAI shifted more traffic toward the machines with newer processors and started treating CPU generation as an explicit factor in capacity planning. Sometimes the best software optimization is better hardware.

Inference Optimization

The inference layer is where the model actually runs. There are fleets of GPUs and other accelerators, and the serving software (the engine) that operates them. The model’s core computation is enormous matrix arithmetic, which parallelizes across the thousands of cores an accelerator provides. The layer schedules and batches incoming requests across the fleet, holds the per-conversation state that generation depends on, executes the model’s forward passes, and hands each generated token back to the API layer.

How to make the inference layer efficient?

Demand for tokens grows faster than hardware can be added, so at this layer slow and wasteful are the same word. The waste hides in four places: work routed to the wrong machine, memory holding the wrong state, parallel hardware idling through sequential generation, and two mismatched phases sharing the same machines.

OpenAI adopted the following techniques to make the inference layer more efficient.

1. Balancing load with cache-aware routing

OpenAI serves its models from many GPU machines. Every request has to be sent to one of them. If that routing is uneven, some machines sit idle while others queue up work. Idle GPUs are the most expensive kind of waste in the stack. There is also a second, less obvious problem. Each machine keeps the computed state (the cache) for the conversations it recently served. If the next request of a conversation lands on a different machine, that cache is useless. The new machine must recompute everything from scratch, even though the work already exists somewhere else.

So the router has two goals:

1. Spreading load evenly: send the request to the least busy machine.

2. Using the cache: send it back to the machine that already knows this conversation.

OpenAI’s routing weighs both for every request, along with basics like geography and available capacity. Improvements in load balancing alone have dramatically reduced the cost of serving its models.

2. Managing the KV cache

When a transformer generates a token, it attends over all previous tokens. To avoid recomputing that attention state for every new token, the serving system keeps it in accelerator memory, in what is called the KV cache. The cache grows linearly with conversation length, multiplied by every concurrent conversation the fleet is serving. For long contexts it can rival the size of the weights themselves. If the software evicts the wrong state, the inference layer pays the full processing cost again when that conversation returns.

The efficiency technique here is to manage the cache based on real usage. OpenAI analyzes production traces to learn which cached state is likely to be needed again, tests eviction policies against that data instead of intuition, and optimizes how cached state is stored and moved between memory tiers. This ensures the most valuable work stays around longer without running out of memory or wasting time moving data.

3. Speculative decoding

Generation is sequential. Each token depends on the previous one, so a model that processes a huge prompt in one parallel pass must produce its answer one token at a time. In large models, producing these tokens is expensive and slow, even when the next token is easy to guess. For an input like “the capital of France is”, the next token is almost certainly “Paris”, but the model still has to process the entire conversation and perform large matrix multiplications to predict it.

Speculative decoding uses a small draft model to propose the next several tokens, and the large model verifies all of them in a single parallel pass. In many cases, the draft proposals are correct and will be accepted by the large model, which saves the time the large model would have taken to produce them one at a time. When a proposal is wrong, the large model discards it and produces its own token, so the output quality does not change. The main metric to evaluate the draft model is the average acceptance length, which is how many proposed tokens the main model accepts in each pass.

4. Separating prefill and decode

Inference has two phases, prefill and decode. Prefill processes the entire incoming prompt in one massively parallel pass, and it builds the internal state (KV cache) the model needs. Decode then generates the response one token at a time. These two are very different workloads. Prefill is compute-heavy, while decode is memory-heavy, spending most of its time streaming weights and cached state through memory. Running both on the same machines means neither runs on hardware set up for it.

The efficiency technique is to separate them. One part of the fleet handles prefill and another handles decode, each configured for its own bottleneck. When a request finishes prefill, its computed state (the KV cache) is shipped to a decode machine, which generates the response token by token.

That covers the efficiency techniques at each layer of an agentic AI application. The harness avoids resending what the server already has, the API avoids reprocessing what it has already processed, and inference avoids recomputing what it has already computed. To close, let’s look at the lessons from this work that apply beyond OpenAI.

What to Learn from OpenAI’s Push for Efficiency and Lower Cost per Successful Task

Zooming out, all these optimization techniques at different layers try to avoid paying for the same work twice. The harness sends only what is new and keeps prefixes stable so caches survive. The API tokenizes only the delta and hides unavoidable work inside windows that had to pass anyway. Inference routes conversations back to their cached state instead of recomputing it. These layers also feed each other. When the harness keeps the cache intact and the API improves the cache hit rate, the savings appear on the GPUs, which means lower costs for users. Beyond the techniques, here are a few lessons that the OpenAI engineers shared with us.

Keep the design simple. Prioritize simplicity over complexity. For example, use lexical search instead of embeddings for tool discovery, and one compaction path instead of a menu. An LLM is already quite intelligent, so adding more machinery in front of it often adds complexity without adding value. Build the simplest thing that could work, then scale it.

Use the agent to optimize its own stack. Codex wrote much of the migration of the very API that serves Codex, turning years of rewrite work into months for a couple of engineers. The inference team uses it to analyze traces and prototype kernels. When agents write the code, you no longer need a language that is easy for humans to write, so you simply pick the most efficient one to run. Efficiency work is becoming a loop that accelerates itself.

Optimize end to end. The inference team told us that every time they picked one favorite technique to focus on, they regretted it. Focusing on one part of the stack made them underinvest in the others. No single optimization is a game changer on its own. The big wins come from chaining many small ones together. They also learned to test with the same traffic shapes that production actually serves, because a change that looks like a win offline can hurt in real traffic.

Users expect apps and agents to reach the tools they already work in. Every integration that gets you there is a different OAuth flow, a different token lifecycle, weeks of infrastructure before you write a line of product code.

WorkOS Pipes handles it with one API call. Pre-built connectors for GitHub, Slack, Salesforce, Google Drive, and more. Pipes handles OAuth, token refresh, and credential storage. You call the real provider API with a fresh token, every time.

Connect to 100+ providers →

Over the last few years, three of the biggest food delivery companies rebuilt their search systems around LLMs. DoorDash, Instacart, and Uber Eats were solving the same problem of trying to understand a user’s intent when they type something in search. They were also accessing more or less the same research base. And yet, the architectures they shipped look quite different from each other.

That divergence is one of the most interesting aspects of modern development using LLMs. Once we understand why each company landed where it did, we will have a mental model for thinking about how AI can be integrated into any production system.

Adding an LLM to an existing stack comes down to one question: how deeply should the LLM reach into the runtime?

Ultimately, DoorDash, Instacart, and Uber Eats each answered that question differently, and the specific LLM each chose was secondary. The specific model each company picked was secondary. The infrastructure they already had in place is what determined the answer.

In this article, we will walk through their differing solutions and try to make sense of their choices and understand the pattern behind them.

Disclaimer: This post is based on publicly shared details from various sources. References are present at the end. Please comment if you notice any inaccuracies.

The Problem

Type “something healthy for a rainy evening” into a food delivery app and watch what comes back. The result we get today is somewhere between useful and impressive. The same query five years ago might have returned a random jumble of items, because keyword search treats the words as a bag of tokens rather than as an intent, and the query offers little for keyword matching.

This pattern repeats across several common failure modes in food search. For example:

• Synonyms: “Soda” and “soft drink” describe the same product, but a keyword engine treats them as different tokens.

• Typos: “Mozzarela” should retrieve mozzarella results, but the spelling mismatch breaks the lookup.

• Shorthand: “Gf pizza” meaning gluten-free pizza requires the system to recognize an abbreviation as a synonym for the full phrase.

• Language mix: The Spanish word “pan” means bread, while the English word “pan” means a cooking vessel, so a bilingual search bar has to disambiguate.

• Word-sense ambiguity: “Apple” the fruit and “Apple” the company share the same spelling but mean different things, and the right answer depends on context.

Each one is a potential moment where the user’s intent and the catalog’s words fail to line up.

Two harder problems lie beneath this:

• The long tail: Grocery and restaurant platforms see an enormous number of unique queries. Any specialized model trained on conversion data struggles with queries that appear only a handful of times, because rare queries are rare by definition.

• The constraint problem: A query like “vegan chicken sandwich” has a hard constraint inside it, and similarity-based retrieval might return a chicken sandwich that violates the constraint, because the similarity score still looks close. Dietary restrictions, allergens, and quantity filters all sit in this bucket.

Food search is the right domain to watch this play out, because all of these failure modes show up at the same time. In the subsequent sections, we look at how different companies handled these situations differently.

Your infrastructure platform shouldn’t be your biggest project [VIRTUAL EVENT] (Sponsored)

How do platform engineers keep pace with a growing development team at a $2B AI company? Join Lawrence Aiello, Platform Engineering Lead at Rogo, on August 11 to learn how his team evaluated and implemented a modern IaC platform. He’ll cover:

• The evaluation criteria that mattered when Rogo picked a modern IaC platform, evaluating solutions like Pulumi, OpenTofu, Crossplane and more

• What “it just works” looks like in practice for infrastructure operations

• Why a reliable platform is the foundation for AI-driven infrastructure workflows

If your infrastructure platform has become another system to maintain, this session is for you.

Register Now →

DoorDash

DoorDash already had a knowledge graph for items and restaurants when LLMs became viable for production. The graph held structured attributes for every item, including dish type, dietary preference, cuisine, brand, and flavor.

Their approach was to use LLMs to enrich this graph offline by extracting attributes from SKU data, and to use LLMs at runtime only for parsing queries into chunks that could link back to the graph. Retrieval itself stayed keyword and graph-driven.

Consider the query “small no-milk vanilla ice cream.” The LLM segments it into three chunks.

• “small” is a quantity attribute.

• “no-milk” is a dietary preference attribute that maps to the canonical label “dairy-free.”

• “vanilla ice cream” splits further into a dish type (”ice cream”) and a flavor (”vanilla”).

Each chunk is then linked to a specific field in the knowledge graph. The dietary preference becomes a hard filter, so only dairy-free items get retrieved. The flavor becomes a soft preference for ranking. The dish type narrows the candidate pool.

See the diagram below:

What distinguishes DoorDash’s approach is how they constrain the LLM’s outputs.

They use retrieval-augmented generation as a guardrail rather than as a generator. For each query segment, an approximate nearest neighbor lookup retrieves the top 100 closest taxonomy concepts from the existing graph. The LLM is then prompted to pick from that list rather than invent labels. This is a clever inversion of the usual RAG pattern, where RAG typically injects context into a generator. Here, RAG defines the entire output space, so the system only ever produces concepts that the rest of the design already knows how to handle.

The measured impact is a roughly 30% lift in the trigger rate for popular dish carousels, all delivered through an architecture whose runtime stays mostly classical.

The takeaway is that DoorDash’s LLM lives mostly offline, mostly in batch, working on the periphery of the runtime.

Instacart

When Instacart’s engineers first investigated an off-the-shelf LLM to categorize the search query “protein,” the model returned chicken, tofu, beef, and other high-protein foods.

The answer is reasonable by any reading of the English language. The problem is that Instacart’s actual users, when they type “protein,” are looking for protein bars and protein powders. The model’s general world knowledge and the company’s specific user behavior were pulling in different directions, and this small failure points at the central problem Instacart had to solve.

The system they were replacing was complex.

Query category classification ran on a FastText model, query rewrites came from a separate engine that mined session behavior, and spell correction, query tagging, and aisle classification each ran as a separate model with its own data pipeline and serving infrastructure. The maintenance burden was substantial, and tail queries still suffered because each model needed its own labeled data, and labels for rare queries are scarce by definition.

Instacart’s strategy is layered across three approaches:

• Context engineering: Retrieval-augmented generation pulls Instacart-specific context (top-converted categories, historical conversion data, catalog details) into the prompt before the LLM sees a query.

• Post-processing guardrails: Semantic similarity filters drop LLM outputs that drift away from the original query.

• Fine-tuning: For the most advanced tasks, the team fine-tunes Llama-3-8B on Instacart’s proprietary data, so domain knowledge gets baked directly into the model weights.

See the diagram below:

The serving architecture splits along the head-versus-tail distribution.

Head queries hit an offline RAG-and-cache pipeline that is latency-tolerant and deeply context-engineered, while tail queries hit a real-time fine-tuned Llama-3-8B model that keeps latency under 300ms through adapter merging, H100 GPUs, and autoscaling.

The cache serves queries the company has seen before, while the real-time fine-tuned model serves everything else, which is where the cold-start tail lives. This split is exactly the kind of decision the head-versus-tail traffic distribution encourages, since pre-computing only pays off for queries that repeat.

After this solution, query rewrite coverage jumped from 50% to over 95%, with 90%+ precision across substitutes, broader rewrites, and synonyms. The real-time fine-tuned model improved search quality for the bottom 2% of queries (the cold-start tail), cutting scroll depth by 6% and complaints about poor tail-query results by half.

The takeaway is that Instacart’s LLMs live at the query understanding layer, with some offline-cached and some online and fine-tuned, while retrieval and ranking downstream are still done by traditional machine learning and information retrieval systems. The LLM brain is doing the upstream interpretation, while the classical approach is doing the downstream serving.

Uber Eats

Uber Eats faced the same problem the others did, plus a complication. They run across multiple verticals, including restaurants, grocery, and retail, across multiple markets, and across a long list of languages.

Their previous setup was fragmented, with separate BERT-based embedding models per vertical, lexical search in places, and the operational overhead of maintaining all of these in parallel. The goal was a single retrieval system that handled every vertical, every market, and every language with one consistent embedding space.

The architecture they decided on is a classic two-tower setup, where a query encoder and a document encoder each produce vectors in a shared space, and matching is done by similarity in that space.

The twist is in what sits inside each tower, because both towers use a fine-tuned Qwen LLM as their backbone embedding layer. The query tower runs online, embedding each incoming query in real time, while the document tower runs offline and pre-embeds billions of documents into an HNSW vector index (a graph-based structure built for fast similarity search at scale). This split is what makes the system economically possible, since running a heavyweight LLM on every document at query time would be prohibitive, while pre-computing once and looking up at retrieval time is tractable.

The fine-tuning matters as much as the architecture choice.

Out of the box, Qwen brings world knowledge and cross-lingual capability, which is what handles the Spanish “pan” problem mentioned earlier. Fine-tuning on Uber’s proprietary query-document interactions teaches the embedding space what Uber Eats users actually care about, and this is what gives the system its domain alignment. The base model contributes general semantics, and the fine-tuning adds the specific rules of Uber Eats.

What makes the system viable at scale is a stack of optimizations:

• Matryoshka Representation Learning trains a single model whose embedding can be truncated to different lengths, and Uber serves at 256 dimensions in production with under 0.3% recall loss compared to the full 1,536.

• Scalar quantization (int7 instead of float32) cuts latency in half again.

• Pre-filters on hexagon, city, and fulfillment type shrink the candidate set before the ANN search even runs.

The numbers tell the cost story. Tuning the ANN parameter k dropped latency by 34% and CPU by 17% with negligible recall impact, quantization halved latency at recall above 0.95, and MRL cut storage by nearly 50%. Together, these are the engineering choices that let a fine-tuned LLM serve as the retrieval substrate of a production search system at Uber Eats’ scale.

The takeaway is that Uber Eats’ LLM is the embedding model itself, since every query and every document gets an LLM-derived vector, and retrieval at every level depends on LLM-produced representations.

Integration Depth

If we line up the three companies on a single axis labeled “how deeply does the LLM sit in the runtime,” the pattern provides some insight.

• DoorDash sits on the left, with the LLM enriching the catalog offline and parsing queries with constrained outputs while the runtime stays mostly classical.

• Instacart sits in the middle, with LLMs handling query understanding through offline RAG for the head queries and a fine-tuned Llama-3-8B for the tail, while retrieval downstream is still traditional.

• Uber Eats sits on the right, with a fine-tuned Qwen as the embedding backbone running for every query and pre-baked into every document vector.

Each company’s position on the spectrum was largely determined by the infrastructure it already had.

• DoorDash already had a knowledge graph that classical retrieval could exploit, so the cheapest gain came from making the graph richer and teaching queries how to talk to it.

• Instacart had specialized query-understanding models that were hard to maintain, so the biggest win came from consolidation under a unified LLM strategy.

• Uber Eats already had two-tower embedding infrastructure running per vertical, so the natural next step was swapping in a fine-tuned LLM as the shared backbone, allowing one model to handle every vertical and every language.

The lesson would be that before asking which LLM to use for some task, the better question is where in an existing stack an LLM would actually help.

Conclusion

Three of the largest food delivery companies rebuilt search around LLMs in the same window of time, with overlapping research literature and similar production constraints, and they arrived at three completely distinct architectures.

The reason food search was a good place to investigate this is that it breaks classical keyword retrieval in many ways simultaneously. Subjective queries (something healthy for a rainy evening), long-tail traffic, multilingual catalogs, and compound constraints all sit in the same search bar.

The three approaches each represent a different setting:

• DoorDash uses LLMs offline to enrich a knowledge graph, while classical retrieval still drives the runtime.

• Instacart uses LLMs at the query understanding layer, with fine-tuned smaller models handling tail queries in the hot path.

• Uber Eats fine-tuned a Qwen LLM into the embedding backbone of two-tower retrieval, so every query and every document gets an LLM-derived vector.

Each company’s position on this spectrum traces back to the infrastructure it already had, which is why the depth-of-integration question matters more than the model choice.

Three universal tradeoffs survived every architecture.

• Hybrid systems are the default everywhere. Classical retrieval, knowledge graphs, and ANN indexes still do most of the work.

• World knowledge from a pre-trained model is just a head start. Domain context still has to be injected somewhere, through RAG, fine-tuning, or both.

• Guardrails are an important part of every production LLM system. Constrained vocabularies, similarity filters, and taxonomy enforcement quietly determine whether outputs stay aligned with the catalog.

References:

• Evolution and Scale of Uber’s Delivery Search Platform

• Scaling Multilingual Semantic Search in Uber Eats Delivery

• How DoorDash leverages LLMs for better search retrieval

• Building DoorDash’s product knowledge graph with large language models

• Building The Intent Engine: How Instacart is Revamping Query Understanding with LLMs

• Supercharging Discovery in Search with LLMs

Define tasks with Render’s lightweight SDK and chain them into long-running, distributed workflows. Launch your agents and batch jobs on demand. Render Workflows handles queuing, orchestration, and retries.

Get $50 credits with code BYTE

You probably know NVIDIA sells GPUs. But did you know it is also the largest publisher of open AI models in the world? Its models rank among the most downloaded on Hugging Face, and the lineup goes way beyond chatbots: reasoning models, world models, models for humanoid robots, self-driving cars, even drug discovery, quantum computing, and global forecasting.

This raises two questions. How does a company best known for hardware build so many strong models across so many domains? And why give them away for free, when those same models run on the GPUs NVIDIA sells?

To understand both, we spoke with Bryan Catanzaro, VP of Applied Deep Learning Research at NVIDIA. He walked us through how his team builds the company’s open models, the reasoning behind their architecture, and why NVIDIA open-sources so much of it. Our thanks to Bryan for taking the time to share the team’s work in such detail.

In this article, you’ll learn:

• What open models NVIDIA has built, and what they do

• How NVIDIA makes its models both fast and powerful

• How one foundation lets a small team build many models

• What open really means and why releasing the data matters as much as the weights

• Why NVIDIA gives it all away, and the lessons the team has learned shipping in the open

NVIDIA’s Open Model Ecosystem

NVIDIA’s open models sit at two ends of AI. At one end are models that work with symbols, the language, code, and reasoning that lives on a screen. At the other are models that work in the physical world, where robots and autonomous vehicles (AVs) have to perceive their surroundings and act. Most of the lineup falls somewhere along that line.

The simplest way to see the ecosystem is to group them by what each family is for.

1. Reasoning models

Reasoning models are large language models that are trained to produce intermediate tokens before the final answer. This is what lets them think through tasks such as math and coding. Most frontier models you see these days are powered by a reasoning model. Claude Opus, GLM-5.2, Kimi K2.7.

NVIDIA’s reasoning model family is called Nemotron. The first Nemotron model was introduced in 2023. In 2024, NVIDIA released the next generation with 340 billion parameters, followed by a faster hybrid design in 2025. The current generation, Nemotron 3, arrived across late 2025 and 2026 in three sizes. A small Nano model for fast, simple tasks, a mid-size Super model for harder planning and reasoning, and a large Ultra model for complex tasks that need the heaviest reasoning.

2. Physical AI

Language models are great at predicting the next token, but they do not understand the physical world. They cannot tell you how a shirt sleeve folds, or what a glass of water looks like the moment it tips over. A robot or a self-driving car needs that kind of understanding, but it cannot get it from text.

That is what a world model provides. Instead of predicting the next word, it understands the world and can predict the next state given the current state of the world and an action. It can generate realistic, physically plausible video, reason about motion and cause and effect, and produce the action data a robot needs to learn.

World models are extremely useful because you can use them to create training data for situations that are slow, expensive, or dangerous to capture in the real world.

NVIDIA’s CEO, Jensen Huang, has called this “the ChatGPT moment for physical AI.” NVIDIA’s frontier open model for physical AI is Cosmos, launched at CES in January 2025 and unified in 2026 into a single omni-model, Cosmos 3, that can generate a scene, reason about it, and predict what comes next. Cosmos 3 also introduces world action models, which turn the pretrained video backbones directly into robot policies. There is also a 4-billion-parameter Edge version small enough to run on the robot and control it in real time.

From World Models to Robot Policies

A world model can predict the next scene, but a robot has to act. It has to turn what its cameras see into how its motors move, in real time, on tasks nobody scripted in advance.

That is the job of a robot foundation model, often called a vision-language-action model. The inputs to these models are usually camera images and instructions, and the model produces the movement.

NVIDIA’s open foundation model line for humanoid robots is Isaac GR00T. NVIDIA unveiled the project in 2024, released the first open humanoid foundation model, GR00T N1, in early 2025, and has since iterated to GR00T 1.7.

GR00T models use Cosmos as a reasoning backbone, giving robots a shared foundation for perception, reasoning, and locomanipulation: the ability to move while manipulating objects. This enables more human-like decision-making and helps robots break complex actions into structured, step-by-step plans that developers can specialize for specific tasks. The 1.7 release also includes a production-friendly commercial license, allowing developers to deploy the model in real-world applications.

3. Healthcare, autonomous vehicles, quantum, and more specialized applications

The lineup keeps going. Alpamayo is an open model family for reasoning-based self-driving. Like GR00T, it uses Cosmos as its reasoning backbone, so the same physical-AI foundation that powers robots also lets a car work through road situations it was never trained on. It also exposes the chain of cause and effect behind each decision, which developers can inspect and build on. BioNeMo brings open models to biology and drug discovery, and the BioNeMo Agent Toolkit lets developers build autonomous agents for life sciences work. Ising gives quantum computer builders, operators, and developers the AI tools needed to scale devices to fault tolerance. Earth-2 is a family of open models, libraries, and frameworks that makes professional-grade weather and climate AI available to anyone. On top of these, NVIDIA offers open models for production use: safety models that filter unsafe content, speech models for voice, and vision models for images.

Taken together, this powerful ecosystem changes how the community can advance and build. The rest of the article looks at how NVIDIA builds these models, how the team is able to move fast, and what they have learned.

How the Models Are Built to Be Frontier and Fast

It is often a challenge to build a model that is both fast and capable. Capable models are usually larger, which makes them slower to train and to answer questions. NVIDIA’s answer is to chase the frontier and speed at the same time. Bryan put it this way: “the fastest model is the smartest model.”

While this may sound like a slogan, it has a real insight. A faster model can be trained on more data in the same amount of time. It can be post-trained across more environments. Once deployed, it can think for longer on a hard problem at the same cost. So speed translates into capability and compounds at every stage.

Given this strategy, NVIDIA designs its models with a focus on efficiency and speed. The following design choices are what make these models both fast and capable. The first two make the model faster and more efficient. The last one makes it more capable.

1. The hybrid architecture

Most well-known models are built almost entirely on the Transformer’s attention mechanism. Attention lets every token in the input look at every other token, which is powerful for capturing relationships but expensive. The cost grows with the square of the input length, so doubling the input roughly quadruples the work. The model has to keep every earlier token in memory as it runs. On very long inputs, that gets slow and costly.

The other main class is the state space model, and Mamba is its best-known example. Instead of comparing every token with every other one, a Mamba layer reads the sequence once and compresses everything it has seen into a fixed-size memory. That makes it cheap: the cost grows linearly with length, and the memory stays constant no matter how long the input. The tradeoff is that a fixed-size memory cannot keep every detail, so Mamba is weaker at recalling one exact fact buried far back.

Mamba layers are efficient while attention layers are more effective. Combining them gives the best of both worlds. A hybrid architecture swaps most of the attention layers in the Transformer with Mamba layers and keeps a few blocks with full attention.

NVIDIA’s models rely on the same hybrid pattern. Most of the layers are Mamba, which keeps long inputs cheap to process and is what makes a million-token context window practical rather than just theoretical. A few attention layers are placed in between to restore the pinpoint recall that Mamba gives up, so the model can still retrieve exact details from anywhere in the context.

On top of the hybrid design, the models also use mixture-of-experts, or MoE, layers. An MoE layer consists of many small expert layers and routes each token to only a few of them. So instead of running every parameter on every token, the model activates only a small subset of its parameters. That allows us to use a large total capacity at a low per-token cost, which keeps the model fast.

The way Bryan describes it, Mamba builds a global view of the whole sequence. Attention can always go back and find an exact fact. Put together, they are smarter than either one alone. NVIDIA published this hybrid result in 2024, and other labs, including the teams behind Qwen and Kimi, have since moved in the same direction.

2. Co-designing the GPU and the model

The second choice is where NVIDIA’s two businesses meet. Its larger models are pretrained in a 4-bit number format called NVFP4, meaning most of the math during training uses just four bits per value. Fewer bits means less memory and less data to move, so the math runs faster and on less power.

Note that four bits is very little precision. Most teams train in higher precision and only shrink the model afterward, which loses accuracy. NVIDIA trained in 4 bits from the first step because it knew its next GPU generation, Blackwell, was being built with fast 4-bit hardware. The model and the chip were designed for each other.

The VP described the mindset behind it. Moore’s law is no longer giving easy gains every generation, so progress has to come from designing the model and the hardware together. The team only attempted 4-bit pretraining because it believed the result was possible. In his words, the first step to inventing something amazing is believing that you can.

3. Post-training: learning from experience

With those efficiency choices in place, post-training is where the model gains its capability.

The post-training recipe itself is what most other teams converged to. It starts with supervised fine-tuning, where the model learns from curated examples of good answers and picks up the format and behavior expected of it. Then it switches to reinforcement learning, where the model practices on real tasks and is rewarded for reaching the right outcome. Instead of learning from examples, it learns from its own attempts.

Scaling up this reinforcement learning stage has been shown to improve a model’s capability significantly. Here NVIDIA’s efficiency focus pays off again. Because its models are cheaper to run, it can scale RL at a lower cost, letting the model practice across many environments in parallel and learn from over a million rollouts.

This is where the VP placed the real bottleneck for better models. Not data or compute. The diversity of these training environments. The more varied the situations a model practices in, the more capable it becomes, because that is where it gains real problem-solving experience.

So far we have looked at design decisions to build a model and what is involved to train a single frontier model. But how do they build so many models, and keep improving them so quickly?

One Unified Foundation

Building strong models in one domain is hard. Building them across language, video, robotics, vehicles, and biology needs huge engineering and compute resources. NVIDIA’s approach is to identify a reusable foundation, build it once, and reuse it across different efforts.

The VP traced this back to CUDA, the software layer that made NVIDIA’s GPUs programmable. The “U” stands for Unified, and he called it the most important letter. The idea is to build one foundation and run everything on top of it, rather than maintaining many separate stacks.

The same instinct shapes the models. The hybrid architecture from the last section is not rebuilt for each model. The same backbone scales across all three Nemotron sizes. Whole components cross between efforts too. Cosmos Reason, the model built to reason about the physical world, was reused as the reasoning core inside NVIDIA’s Isaac GR00T foundation model, so the robotics team did not have to build that part from scratch. The alternative, ten parallel efforts each reinventing the same pieces, would spread people and compute too thin. According to Bryan, NVIDIA tries to be as lazy as possible, because that is how it gets the most out of limited resources.

The other reason the team is so productive and iterates quickly is cultural. NVIDIA is mostly not top-down; it is a company of volunteers, where teams choose what to work on, and unification happens through invitation rather than control. The culture of collaboration is that if teams work with each other and get more done, they will get more resources than if they each go alone. When everyone shares that belief, a small group can produce a lot.

Open is More Than Just Releasing Weights

You may have heard a lot about open models. Several AI labs released capable models. You can download the weights and use them. But that is not truly an open model. This is where NVIDIA’s approach really separates it from the other labs. As Bryan puts it, Nemotron is not just a model. It is also the data, the tools, the training recipes, and the ideas published in the papers. The finished weights are just the last step.

In practice that means NVIDIA publishes the training datasets, the post-training datasets, the reinforcement learning environments, and the recipes needed to reproduce the work. A team can not only run the model, but also see how it was built and train their own version.

The same is true for robotics and self-driving cars, where developers can use open source simulation and learning frameworks, datasets to train and evaluate the model performance before deployment.

Releasing data at this scale is unusual. It is a scary thing for a company to do. NVIDIA does it anyway, and even shares clever building blocks like synthetic “persona” data that is statistically realistic but fully private, so a model can learn about the world without memorizing real people.

You can see this in what the community has done with it. NVIDIA’s open data and models have spawned a large set of community fine-tunes and derivatives. One of its robotics datasets became one of the most-downloaded datasets on Hugging Face. Many variants of its Nemotron models have millions of downloads each, and the Nemotron family recently crossed 100M in total. You cannot build like that from the weights alone.

Why a Chip Company Gives It All Away

You may think releasing open models sounds great. But why would a company that sells GPUs give away the AI that runs on them?

Bryan shared two reasons. The first is that NVIDIA needs to understand AI deeply to build the right hardware for it. To design future chips, you have to know where AI is going. The best way to really know is to build models yourself and put them in front of real users. A model kept private is easy to fool yourself about, so releasing it openly keeps the work honest and keeps NVIDIA’s researchers connected to the wider community they learn from. With open models being such a hot topic, the real value is in enabling developers to post-train these models themselves at a fraction of the cost.

The second reason is about the business. NVIDIA grows whenever AI grows. Its goal is to support the ecosystem, not to compete with the companies building on top of it. Every team that adopts open models and builds its own AI can become a future customer for compute. So the company would rather hand the ecosystem the models and let the world’s demand for AI pull its business forward.

There is also a side benefit: safety. Open technology tends to be safer because more people can inspect it and find problems, the way the open internet is hardened by everyone who uses it.

What NVIDIA Has Learned Shipping Open Models

Shipping this many models in public teaches lessons that closed labs never face. Bryan shared several valuable lessons.

1. The program matters more than any single release

In a field moving this fast, every model is obsolete before it ships. So no single release is the point. What earns trust is a steady, sustained program. When the community believes a model line is here to stay, it pays attention to each new release instead of treating it as a one-off.

2. Make it easy for developers

A clean launch is the result of getting a hundred small things right, from licensing to documentation to ready-to-run examples. The VP admitted this is the unglamorous part. Listening to community feedback and smoothing that first-run experience is what turns a download into lasting use.

3. Pick a license the community trusts

NVIDIA moved away from its own homegrown licenses to OpenMDW, a community license from the Linux Foundation built specifically for open AI. Using a shared, purpose-built license removes friction and signals that the models are meant to be used freely.

What’s Next

The direction from here is to open even earlier. Rather than build a model privately and post it when it is done, NVIDIA wants to bring collaborators in sooner, since it plans to open everything anyway. The aim is to let partners shape the models while they are still being built. To make this happen, NVIDIA formed the Nemotron and Cosmos Coalitions.

What stands out most is the conviction that this is a long-term commitment, not a phase. The VP pointed to CUDA, which took more than a decade before the rest of the world understood why it mattered. NVIDIA stuck with it, and it became the foundation of the company.

He expects open models to follow the same path. As he put it, this is not a science project, and it is not charity. It is how NVIDIA moves itself forward into the future.

On a single computer, every event can be measured against one clock, so the order of the events is never in doubt. However, a distributed system, where work is spread across many independent machines, lacks a shared reference. Each machine can read time from its own hardware clock, but those clocks do not stay aligned. Network Time Protocol (NTP) tries to bring the clocks together, but it cannot eliminate the problem.

This becomes a real problem whenever correctness depends on the order in which things happened. If events on different machines are ordered by comparing timestamps, a small difference in the wrong direction can change the result. For example:

• An update that happened later can be discarded in favor of an earlier one, because clock differences gave the earlier write the larger timestamp, and the newer data disappears with no error raised.

• Logs gathered from several machines can place an effect before its cause, which makes tracing a problem much harder.

• An operation that depends on a correct order, such as granting access only after the change that authorized it, can act on stale information.

In this article, we look at these ideas in detail:

• Why a machine’s own clock is a poor basis for ordering events, and how close synchronization can realistically get

• What it means for one event to have happened before another, and why some pairs of events have no defined order

• Logical clocks and vector clocks, and what each one can and cannot determine

• Why the order of events matters in practice, for replication, auditing, and debugging

• Hybrid logical clocks and the problem they were designed to solve

• How a globally distributed database such as Google Spanner approaches ordering at scale

Clock Drift

Read more

[Webinar] 8 levels of context maturity in AI-native engineering (Sponsored)

AI is in your engineering workflows. While the token spend shows it, the throughput doesn’t. The human is very much still in the loop, and that’s a context problem.

Join live July 23 (FREE) to learn:

• The 8 levels of context maturity: where most teams are stuck and what the ceiling looks like at each stage

• Why more MCPs, rules, and skills provide agents access but not understanding

• How leading engineering teams are using a context engine to make the most of their agents

Register Now

There is a persistent gap between the AI agents that perform well in a demonstration and the ones that hold up under real production traffic.

This is because the systems handling live customer workloads tend to depend on the language model far less than their presentation suggests. Most of their behavior runs through conventional, deterministic code, with the model invoked at a small number of specific decision points. The difference between a dependable agent and a fragile one often comes down to a set of engineering decisions about where the model is given control and where it is held back.

But how do you build agents that work reliably in production? Are there any best practices?

In 2011, Heroku’s co-founder published the Twelve-Factor App, a set of principles for building web services that deploy and scale reliably, and that document influenced how a generation of engineers thought about software.

A parallel effort later appeared for AI in the form of a widely shared set of principles called the Twelve-Factor Agents. These principles were aimed at making language-model applications robust enough for production. The specifics have kept evolving since, and teams at companies like Anthropic, Cognition, and Intercom have added lessons of their own.

In this article, we try to explore the collective thinking into a smaller set of practices and explain the reasoning behind each one, rather than asking anyone to memorize a numbered list. The practices fall into four areas, with a final section on the tradeoffs. Here’s what we will cover:

• How does context control what the model sees on every call?

• How does the control flow keep the loop and its stopping conditions in deterministic code?

• How does the state hold memory in software while the model stays stateless?

• How does the scope keep each agent narrow and supervised?

By the end, the article will have built a working definition of a production agent and a clear view of which decisions carry the most weight.

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

The Loop

Let us start with the simplest agent anyone can build, because it reveals the structure that everything else refines.

At its core, an agent is a loop. The model receives some context, which is the running record of what has happened so far, and it returns a single decision about the next step to take. That decision arrives as structured output. It is machine-readable data, such as JSON rather than free-flowing prose, so that ordinary code can read it and act on it. The surrounding code executes the requested step, appends the result to the context, and returns everything to the model for the next decision. The loop continues until the model signals that the work is finished.

A useful way to picture the model’s role is as a function that starts fresh on every call.

On each call, the model sees only the context provided to it, makes one decision, and discards everything the moment it responds. The context window, which is the bundle of text the model reads on a given call, serves as the entire memory of the system.

The appeal of this design is clear. A developer describes a goal and supplies a set of tools, and the model works out the order of operations on its own. The design requires less branching logic, since the model handles the decisions. This is the version of an agent that can perform quite well in a demo setup. However, the trouble begins when real traffic arrives.

Failure Modes

The design we looked at in the previous section fails under production conditions, and the failures follow some predictable patterns:

• Compounding error multiplies across chained steps.

• Confidently wrong output reaches a user.

• A loop runs longer than intended.

• State disappears when the plan lives only inside the model.

Let’s understand them in a little more detail.

The first pattern is compounding error. Every model call carries some chance of a wrong decision, and a freewheeling agent chains many calls together. Suppose each step is correct 95 percent of the time, which sounds reliable. Run twenty such steps in sequence, and the odds that all of them succeed fall to roughly one in three. The math is multiplicative, so reliability that looks fine in isolation degrades quickly across a long chain. This fact explains why production teams add so many guardrails.

The second pattern is confidently wrong output reaching a user. In 2025, the support agent for the coding tool Cursor told customers about a one-device-per-subscription rule that the company later confirmed it had fabricated, and a wave of cancellations followed. A year earlier, a tribunal held Air Canada liable after its chatbot invented a bereavement-fare policy, and ordered the airline to honor it. Studies place hallucination rates, meaning the rate at which a model produces plausible but false statements, somewhere between 3 and 27 percent, even in controlled settings. An agent that speaks directly to customers turns that rate into a big business risk.

The final two patterns are harder to notice, but just as costly.

A loop with weak stopping conditions can run far longer than intended, consuming tokens and money with each pass. And lastly, when the plan lives only inside the model’s context, a crash or a long-running task can lose the entire thread of work.

Each failure points to a specific practice, grouped into the four areas below, beginning with the inputs.

Context

The context window is everything the model knows on a given call, so controlling its contents is the first and largest lever on reliability. Three habits do most of the work here.

The first is to own your prompts. A prompt is the instruction text sent to the model, and it deserves the same treatment as any other source code, which means version control, review, and testing. Many agent frameworks generate prompts automatically and keep them out of sight, which is convenient until behavior changes and the cause is hard to locate. Keeping prompts in the application’s own codebase keeps their effects reproducible.

The second habit is to own the context window itself. The contents of each call are a deliberate choice, and that choice carries more weight than it first appears. A model given a focused, relevant context performs better than the same model handed a large pile of loosely related history. Quality degrades as the window fills with marginal material. Therefore, deliberate pruning by actively removing content beyond what the current step requires preserves the model’s accuracy. In other words, relevance beats volume on almost every call.

The third habit concerns tools. A tool is simply a function the model can request, described with a clear name and a schema for its inputs. When those descriptions are precise, the model selects the right tool and fills its arguments correctly. When they are vague, the model is more likely to choose the wrong one. Treating tool definitions as a careful piece of interface design pays off immediately.

A discipline called context engineering has grown up around these habits, and it’s a big topic in itself. However, the core idea stays simple: decide what the model sees, on purpose, every time.

Control Flow

Owning the inputs matters only if the surrounding code decides when the model runs and when the loop ends. In a dependable agent, the control flow belongs to the deterministic code around the model, and the model is consulted at a few chosen moments.

Picture the overall flow as ordinary code, a sequence of steps, conditionals, and calls to external systems. At two or three points in that flow, where a decision genuinely calls for judgment, the system invokes the model. Everywhere else, plain deterministic logic does the work because it is cheaper to run, predictable in its output, and straightforward to test.

Two practical rules make it work:

• First, every loop needs an escape hatch, which is a hard limit that forces it to stop. A cap on the number of iterations, a timeout, and explicit completion conditions together guarantee the agent halts even when the model would continue indefinitely.

• Second, invoke the model on purpose. The model belongs where the problem requires open-ended reasoning, and ordinary code handles everything that can be specified in advance.

Intercom’s customer service product shows the pattern at scale.

Its newer Procedures let the agent reason in natural language while the surrounding system supplies deterministic controls, including conditional steps for decision points, small code snippets that guarantee the same input always yields the same output, and checkpoints where the agent pauses for human approval before a sensitive action.

To summarize, it is beneficial to start with the simplest version that solves the problem, and add model-driven autonomy only where simpler logic falls short.

State

Once the surrounding code owns the loop, a natural question follows. Where does the agent’s memory live?

The answer is to keep the model stateless and to hold the state in software that the application controls.

Recall that the model starts each call fresh and reads only the context provided to it. That property, when treated as a feature, becomes powerful. The application stores the real state of the work, the conversation so far, the plan, the progress, and reconstructs the context from it on every call. Since the state lives in serializable storage (a form the system can save and reload), an agent can pause partway through a task and resume later, or recover cleanly after a crash, by loading the saved state and continuing.

This habit also keeps two kinds of states aligned. There is the model’s view of what is happening, and there is the application’s record of ground truth, such as the actual contents of a database. Keeping these two in agreement prevents the model from acting on a stale or inaccurate view of the current state.

There is a scaling benefit as well.

When the state lives entirely outside the model, any running instance of the service can pick up any request, because everything needed is available with the context. That allows many copies to run behind a load balancer, handling far more traffic.

A clean way to think about the model here is as a function that takes the current state and a new event and returns the next state along with an action to perform. The same inputs produce the same result, which makes the whole system easier to test and to trust.

Scope

Owning the inputs, the loop, and the state still leaves one decision open. How much should a single agent try to accomplish? The good answer favors small, focused agents kept under supervision over one broad agent asked to do everything.

A narrow agent with a single, well-defined job has a smaller surface for failure. It is easier to test, easier to reason about, and easier to fix when it misbehaves. When a task spans several distinct jobs, several narrow agents compose under deterministic orchestration. The surrounding code decides which agent runs and when, rather than assigning one general agent a long list of responsibilities.

Klarna’s customer service assistant illustrates the upside. It handled roughly 2.3 million conversations in its first month, and most of those followed predictable paths such as checking a refund status or tracking an order, which structured logic handles well, with the model reserved for the cases that truly need it.

Supervision is the other half of the scope.

A human handoff deserves to be designed as a first-class step. It is reached on purpose when an action is sensitive, when confidence is low, or when a customer asks for a person. Intercom builds this in directly, handing a conversation to a human automatically whenever continuing would be the riskier choice. Treating human involvement as a planned path, complete with the state needed to brief the person, turns it into a strength of the system rather than a sign that something went wrong.

Tradeoffs

Everything so far describes practices that work today. But there are also disagreements and a few open questions.

The main debate concerns single versus multiple agents.

In June 2025, the team behind the coding agent Devin argued against multi-agent designs because parallel sub-agents make independent decisions that then conflict with one another. One day later, Anthropic described its own multi-agent research system, which used roughly fifteen times the tokens of a single-agent approach and scored about 90 percent higher on certain research tasks.

The apparent contradiction resolved over the following months into a shared pattern. A single orchestrator owns the full context and spawns isolated, short-lived sub-agents, each of which completes one task and returns a summary. The lesson to understand is that small, focused agents work best under firm orchestration, while sub-agents that communicate directly with one another tend to produce conflicting results.

The deeper question comes from what Rich Sutton called the Bitter Lesson. It is the observation that general methods that rely on more computation tend to overtake handcrafted techniques over time. Applied here, it warns that some of the scaffolding built to compensate for today’s models will become redundant as the models improve. That is partly true, and it is the strongest argument against over-engineering.

The counterpoint is that several of these problems persist regardless of model strength. A finite context window, the need to keep a figure on page five of a document consistent with the same figure on page eight hundred, the requirement to pause and resume safely, all of these survive better models.

Cost becomes another consideration for every decision. Autonomy and multi-agent designs spend more tokens, sometimes far more, so they earn their place only where the value of the task justifies the bill.

Conclusion

A production agent, reduced to its essentials, is mostly deterministic software that calls a language model at a few deliberate points. The design decisions lie in choosing those points and limiting how much the model decides on its own.

The path to that definition ran through the simplest possible agent and its predictable failures, with each failure answered by one practice:

• Control what the model sees.

• Own the loop and give it a hard stop.

• Hold the state in software while the model stays stateless.

• Keep each agent narrow and supervised.

Around these sit real tradeoffs, including the open argument over multiple agents and the steady improvement in models that will retire some of this work over time. A sound approach starts with the simplest design that solves the problem, measures where it falls short, and grants the model more autonomy only where that autonomy provides clear value.

References:

• The Twelve-Factor App

• 12-Factor Agents — HumanLayer

• Anthropic — Building Effective Agents

• Anthropic — How We Built Our Multi-Agent Research System

• Cognition — Don’t Build Multi-Agents

• Intercom — What’s New with Fin 3

• Rich Sutton — The Bitter Lesson

• Cursor support-bot incident — eWeek

• Air Canada chatbot ruling — AI Business

An incident hits and you pull up the logs, only to find string based console output that tells you something broke but not why. That’s the problem with unstructured logging: it eats your quota and still leaves you guessing.

This hands-on workshop fixes that. You’ll learn what’s actually worth logging (state transitions, request boundaries, external call failures, auth decisions), how to structure logs as real data instead of text dumps, and how to add attributes that connect them to traces and errors. Plus, how to tune signal to noise, so noise stops drowning out what matters.

Save you spot

Roblox is built for scale. In August 2025, the platform reached a peak of 45 million concurrent users. Few game engines can support that kind of demand.

But Roblox itself isn’t the one making most of those games. It gives creators the tools to build them, then focuses on the infrastructure beneath them. Now, Roblox is exploring how a video world model can work alongside its game engine to make those experiences photorealistic without compromising scale or multiplayer performance.

This hybrid approach offers valuable lessons for engineers building real-time systems at scale. To explore those lessons, we sat down with Anupam Singh, senior vice president of engineering at Roblox, to hear from him about the world model that Roblox is using to make its multiplayer games look photorealistic, the key insights that have come from taking that approach, and the next big thing that the Roblox team is focusing on. We thank Anupam for sharing with us.

In this article, you will learn:

• What a game actually requires, and why visuals are only part of it.

• How combining a world model with a game engine can make games more realistic.

• How Roblox is using a hybrid architecture to divide the work between the engine and the world model and keep the world consistent for everyone playing.

• The four open problems (latency, consistency, multiplayer, and creator control) and the specific techniques Roblox is using on each.

• What Roblox is focused on next, and why it could make photorealistic games far cheaper to build.

What makes a game a game

When people praise a game, they usually point at the graphics. Graphics are the easiest part to notice and often the smallest part of the engineering. What actually makes a game a game is a set of systems players never see. These systems serve one of two purposes: keeping the world consistent for everyone playing, and rendering it on screen.

1. Keeping the world consistent

Keeping a world consistent takes several systems working together. The foundation is the persistent state, a running memory of where everything is and what has happened, so the world is still there when you come back to it. Then come the rules, the logic that has to resolve the same way for everyone, so the game stays fair. Physics has to be believable, so objects move, collide, and fall the way players expect. Since most games are played with other people, all of it has to stay in sync in real time, so thousands of players share one consistent world rather than thousands of slightly different ones. On top of that, the world has to react the instant a player acts. Together, this is what turns moving images into something you can actually play.

A racing game makes this concrete. The track, the scores, the penalties, how each car handles, and where every other car is all need to be exact. Every player needs to see the same thing at the same moment. If even one of these is off, players notice right away.

2. Making the game look real

The second purpose, making the world look real, is a newer demand. The real world has a lot of fine detail that is hard to produce. Grass moving in the wind, dust behind a car, smoke off a fire, light shifting across a surface. Players now expect that level of realism.

That is what makes the problem hard. No platform has delivered all of it at once: a world that is consistent and fair, looks real, and still runs affordably on the hardware people already own.

Today, two technologies each cover one side of this. AI world models are built to make things look real, generating photorealistic images from what they have learned. Game engines are built to keep worlds consistent, tracking state, rules, and physics across every player. The catch is that each is good at only its own half. The next sections look at why a world model alone, and a game engine alone, each fall short, and then at how Roblox combines them.

Crusoe Serverless Fine-Tuning Is Now GA (Sponsored)

Fine-tuning open models shouldn’t mean babysitting GPU clusters. Crusoe Serverless Fine-Tuning, now generally available in Crusoe Intelligence Foundry, gives you a complete path from proprietary data to production-ready model. Select a base model from a curated library of top open models — including Qwen, DeepSeek, Gemma, and gpt-oss — upload your dataset in JSONL or Parquet, configure your job, and submit via UI, SDK, or API. Token-based pricing ties spend to actual training work, and early stopping ends billing the moment your model stops improving. Deploy in one click, or download your weights in .safetensors format.

Get started

Why a video world model alone isn’t enough

A video world model generates a world one frame at a time, predicting what should come next. It can also take conditioning signals as input, such as a text prompt or a starting image.

For a game, the signal that matters most is the player’s actions. That is what makes a world playable. For example, when a player presses a key to move forward or turns the camera to look around, the next frame changes to match. But generating believable frames, even ones that respond to input, is not the same as running a game, for two main reasons.

Reason 1: missing the systems that make a game

The first reason is that a video model does not have the systems a game runs on. Its strength, generating what looks right next without simulating every object, is also why it cannot hold a game together. It has no durable memory of the world, no consistent rules, and no reliable way to turn a player’s input into the right outcome. The simplest way to see this is to turn the camera away from something and then back. “If the character turns back, is the world still the same?” Singh says. “In most video models, it won’t be.” The model regenerates the scene and quietly changes it.

Multiplayer exposes the gap further. A video model can render a crowd of 500 people, but those are pixels, not 500 separate players, each with a position, an inventory, and actions of their own. Painting a crowd and simulating one are different problems. However good it looks, a standalone video world behaves more like a moving picture than a game: a guided dream, with no persistence and no real interaction.

There are quieter gaps too, ones that show up in how games are built rather than how they look. A pure video model is hard to test and hard to refine. With normal game code, a creator can change one rule, run it, and check the result. A model that generates the world from learned weights does not give you that kind of precise, repeatable control, which makes exacting competitive gameplay and reliable, intelligent non-player characters hard to guarantee. These are not edge cases. They are the day-to-day work of making a game good.

Reason 2: pixels are not understanding

The second reason is deeper. A video model works entirely in pixels, and pixels are not understanding. It can draw a flawless bottle without knowing it is a bottle: that it opens, that it holds water, or that it falls when you let go. As Singh puts it, pixels alone will never understand the world. Something else has to do the reasoning, working out what things are and how they behave. A model that only paints the scene has no real grip on the rules, the physics, or the cause and effect a game runs on. It is rendering the world, not modeling it.

A game engine is built to do exactly what the model cannot. It does not paint pixels; it tracks what exists, enforces the rules, and computes the physics. So if a video model lacks all of this, why not just build the whole game in the engine? That is the question the next section takes on.

Why a game engine alone isn’t enough

A game engine is built for the parts a video model cannot do. It tracks the exact state of the world, applies rules the same way every time, and keeps that state consistent across sessions and players. Engines also compute physics directly, including collisions and the movement of objects such as a car’s position, velocity, and steering. This side of game development is mature and well understood.

Photorealism is where engines struggle. To make a world look real, the engine has to render all of that detail explicitly: high resolution textures, complex lighting, and simulation for effects like light scattering through fog or the way different materials reflect it. All of that is expensive to compute, and it only gets more expensive the more realistic you push it. So engines cut corners. They bake lighting ahead of time instead of calculating it live, swap in simpler versions of objects that are far away, reuse textures, or lean on a stylized look that hides the missing detail. The tricks work, but they are also why most games still look like games, not the real world.

Avoiding those shortcuts, and actually reaching photorealism, comes at a cost that falls on people, not just servers. Creators need more skill and time to build the detail, and players need more powerful devices to run it. A big studio can absorb that. A two-person team, or someone on an everyday phone, cannot. This is the real limit: an engine can give you a real game, but realistic graphics through it stay out of reach for most creators and most players. Each technology is strong exactly where the other is weak, so the obvious move is to combine them.

The core idea: a hybrid architecture

Roblox’s position is that you should not ask a video model to act as a game engine, and you should not ask a game engine to generate photorealism by itself. Instead, you build a system that splits the work so each part does what it is good at.

Roblox calls that system Roblox Reality. It has three parts: the Roblox game engine, the Roblox Cloud, and a video world model the team calls the Super Upsampler. The first two keep the world consistent. The third makes it look real.

The game engine is the part that knows what is actually there. It keeps a structured record of the world, called the data model, that lists every object and its properties: where the car is, how fast it is moving, which way it points, what it is made of. On top of that record it runs the simulation, the physics and the rules, the same way every time. Because it computes all of this exactly and repeatably, the engine can act as the single source of truth that every player’s view has to agree with.

The Roblox Cloud is what runs that engine at scale. It stores each world’s state durably, so a game is still there when you come back days later, and it runs millions of live game sessions across data centers around the world. Keeping those sessions physically close to players is what makes fast, fair multiplayer possible, because the authoritative state lives near the people acting on it instead of on the far side of the planet.

The Super Upsampler is the video world model, and its name describes the job. It does not invent the world from scratch. The engine first renders a complete but plain frame, with the right shapes, positions, and camera motion but simple textures and lighting, and the model upsamples that frame into something photorealistic. As Singh puts it, the model is mostly adding the textures and the fine detail, not deciding what is in the scene. It is a large base video model that Roblox post-trains for the task, so it already knows how the visual world looks and only has to fill in the details: the textures, the lighting, the water on a windshield, the leaves moving as a car passes. The engine does the exact, inexpensive part, and the model does only the expensive, fuzzy part, which is what makes the hybrid cheaper than either extreme.

There is a deeper idea behind this split. It is tempting to assume the goal is one giant model that learns to do everything, including the reasoning about what is happening and why. Roblox’s bet is the opposite. A world model still needs reasoning, but it does not all have to live inside the network. As Singh puts it, in a hybrid the reasoning can sit beside the model, in the engine and its rules and physics. That frees the video model to spend all of its capacity on the one thing it is best at: the pixels.

Dividing the work like this only pays off if the player never feels the seam. For that, the engine and the model have to stay tightly in sync, frame by frame.

How do the engine and the model stay in sync?

The real engineering is in how these parts connect, because the model cannot be allowed to invent the world freely. Every frame, before it generates a single pixel, it is conditioned on what the engine produces. Those inputs are built by cloud systems that handle level of detail and compositing, then delivered to the edge over a content delivery network, the same way video is streamed.

Take a single frame in a driving game. The engine generates a low-quality rendered frame, a depth map, the scene’s lighting and weather, and structured details for every object. Each is a different kind of signal that the upsampler will later use to render a photorealistic version of the frame.

The dense, pixel-aligned signals carry a value for every pixel: the rough rendered frame, and a depth map giving each pixel’s distance from the camera. Because they share the same grid as the output, they are concatenated with the model’s input, or injected through a ControlNet-style side network, so every generated pixel stays locked to its depth and color.

The global signals are a single setting for the whole shot, like midday, light rain, and the sun low on the left. These are applied through modulation, scaling and shifting the model’s activations everywhere at once, the same way a diffusion model feeds in its timestep.

The structured signals are compact, text-like descriptions: each object’s details, such as a metal sports car at position (120, 0, 48) moving 40 meters per second, plus any prompt for the overall style. These enter through cross-attention, so any part of the frame can draw on them as needed.

From all of this, the model paints the final 2K frame, the gloss on the wet paint, the glare on the windshield, the rough asphalt, the spray off the tires, without ever moving the car or changing how far away anything is.

The data model is what keeps the world from drifting. It tells the model which objects are present and fixed, so a car stays a car with four doors from one frame to the next. The model does not get the whole game state each frame, though, since that would be far too much to process. Instead, a retrieval step pulls only the objects relevant to what it is rendering right now. The engine stays the authority on what exists, and the model only decides how it looks.

State and pixels are also stored differently, and that difference is the center of the design. The consistent world state lives in the data model on the server, where it can be shared across every client and kept stable over a long session. The look of the world lives in the model’s video latents, which are regenerated continuously and never need to be saved. The server decides the truth that everyone shares, and the model fills in the parts that can safely differ a little from one player to the next.

The hard problems and how Roblox is approaching them

With the architecture in place, the real work is making it run in real time, for millions of players at once. The Roblox engine and its global infrastructure are in production today. The video model side is still in the lab, not yet running in real time. Below are the four problems standing between the design and shipping the model into production.

Latency

A game has to respond the moment a player acts. As Singh puts it, a player who clicks can’t be left waiting seconds for the result. Most video models can’t meet that bar. They generate a whole clip at once in an offline pass that reads in both directions across time, which can take several seconds. A playable game needs a fresh frame every few tens of milliseconds. That means closing the gap from roughly five seconds down toward about 30 milliseconds.

The main technique is self-forcing, the core of Roblox’s acquisition of Morpheus AI, whose founder developed the method. Self-forcing turns a slow offline model into an autoregressive one that generates frame by frame, each frame conditioned on the frames it just produced, so it can stream output continuously instead of computing a full clip in one shot.

Self-forcing is not the only lever. The models are made smaller, and their running memory, the key-value cache, is compressed, which Roblox calls critical for reaching high resolutions without the cost spiraling. Generation also runs on H200 and B200 class GPUs in edge data centers placed right next to the game-engine instances and close to players, so the network round trip stays short.

Consistency

The data model discussed earlier keeps the world from drifting frame to frame. The harder challenge is maintaining consistency across many frames.

Over a long session, small errors accumulate, and a world can slowly lose track of itself. The model has to stay coherent across minutes of play, not just a handful of frames at a time.

Staying coherent over a long session takes a model that can hold much more of its own history, and Roblox chose to acquire that capability rather than build it from scratch. The acquisition of Morpheus AI, the same one behind self forcing, brought the long-context world-model work Roblox needed for exactly this. Roblox is now extending the model’s context window, so what it generated ten minutes ago still shapes what it generates now.

Multiplayer

Multiplayer is the clearest failure of video models and the problem most specific to Roblox. A model can generate a crowd, but it cannot run one. Managing 20,000 players reacting in real time means tracking 20,000 separate states, not generating a believable picture of a crowd. The requirement is also subtle at a small scale. When one player moves an object, every other player who can see it has to see it move too. An action is not only what a player does, it is the effect of that action on everyone else’s view of the world.

The engine makes this possible by acting as the server authority, computing one true state that all players share. Each player’s video model then generates that player’s own view, conditioned on the shared state, so the views differ in perspective but agree on the facts. To keep this responsive, the system also runs a quick predictive simulation on the player’s own device for immediate actions, while the server confirms the authoritative result a moment later. In the future, a player’s own avatar may be rendered locally and placed on top of the streamed video, so the movements a player cares about most feel instant. Doing all of this for thousands of players at once is an active research area, not a solved feature, and Singh describes multiplayer actions as a central focus of the team.

Creator control

A game only becomes worth playing when a creator shapes it. What makes a game fun is usually the rules and logic a creator adds, not how detailed it looks. Even a plain-looking game can take off when the gameplay underneath it is good. So the video model has to stay under the creator’s control, and giving them that control is still an open area of research.

One concrete piece comes from Lucid AI, a team that joined Roblox. Its founder built what Roblox calls a game cartridge harness, which wraps the deterministic logic of a real game engine around a video world model. The point is to let a world look generated while still behaving according to fixed, predictable rules the creator sets.

The harness is also how a player’s controls reach the model. The standard movement keys, the WASD inputs, are fed in as conditioning, so pressing forward drives the generated view directly instead of only moving a character inside the engine. Taking real-time player input and turning it into the right next frame is the kind of control a standalone video model cannot handle. How creators will ultimately direct the model beyond movement, through written prompts, through the data model, or some mix of the two, is not yet decided.

Roblox’s bet: creators, compute, and a 20-year head start

These are difficult problems, and Roblox’s case for solving them rests on a few advantages. It has a large community of creators to build with and learn from. It owns its compute, running more than two dozen edge data centers and its own GPUs instead of renting capacity from cloud providers, which matters when the video model has to run for every player at low latency. It has also spent close to 20 years building the data model, the structured record of objects and physics the whole hybrid depends on.

To strengthen the AI side, Roblox brought together founders from several labs and gave them autonomy under a short set of values. Respect the community, take responsibility, get stuff done. Each team came in solving a different piece of the problem. Morpheus AI brought self forcing and the low-latency, long-context work behind it. Lucid AI brought the game cartridge harness, the bridge between deterministic game logic and a video model. Dynamics Lab team brought the skills to build a real-time, general-domain world engine that lets a person upload any image, a photo, a drawing, a painting, and step into it as a live, interactive world they can shape with text prompts and share with friends to play instantly. Put together, those are the pieces of speed, control, and generation that a hybrid needs.

There is one more advantage, and it is operational rather than technical. Even once the model runs in real time, Roblox will have to serve it to millions of players at once, and few companies have more practice keeping a system alive at that scale. Capacity is hard to predict when a game can jump from 3 million to 22 million players in a few weeks, so Roblox plans for sudden surges instead of assuming they will not come. Rather than throttling traffic when a system is under load, it runs a weekly exercise it calls Taco Tuesday, deliberately removing capacity from services in production to find the limits first. The test is automated, and any engineer can start one. Reliability is treated as everyone’s job, with executives on the same on-call rotation as engineers, holidays included.

What’s next

An early version of Roblox Reality is expected later this year or early next, with cost and scaling still to solve before it can reach large numbers of players. The first quality target is 2K resolution at 60 frames per second, with 4K as the longer goal.

The more important change is who gets to build photorealistic games. Because the heavy rendering happens on shared edge GPUs rather than on the creator’s machine or the player’s device, visual quality stops depending on the size of a team’s budget or the power of a player’s phone. A creator can get high-end results on an ordinary device. The goal Singh describes is for a two person studio to build something as rich as a large studio’s work, without having to trade complexity against visual quality.

Singh expects the progress to arrive in large steps rather than small ones, and he expects world models to matter well beyond games. Roblox’s approach is worth watching because it does not wait for a video model to solve everything on its own. It gives the model the support it has always lacked: an engine that remembers the world, enforces its rules, and keeps it true for everyone playing.

As AI agents take on decision-making, they need awareness of real-world conditions.

A weather delay can reroute deliveries. Lightning can pause field operations. Road conditions can affect autonomous vehicles. These signals shape decisions, yet most AI systems can’t observe them.

But context is only as valuable as the data behind it. Enterprise applications need trusted, real-time weather intelligence—not just generic forecasts. Signals like lightning activity, road conditions, hail risk, and weather impacts often matter more than temperature alone.

Xweather’s MCP-ready weather API gives AI agents access to trusted weather intelligence and the real-world context behind weather-driven decisions. Learn how it works in this technical guide.

Get your free API key

This week’s system design refresher:

• MCP vs A2A vs ACP: How AI Agents Actually Talk to Each Other

• 8 Frontier Open Models I’m Most Excited About

• LLM vs RAG vs Agent evals

• How Distributed Tracing Works at the High Level?

• The Life of a Redis Query

MCP vs A2A vs ACP: How AI Agents Actually Talk to Each Other

Agents are capable on their own. Combined with tools and other agents, their capabilities compound. But how should they communicate?

1. MCP: agent to tool communication.
   The host app receives the user request, its embedded MCP client formats and routes it to the right MCP server, the server executes the tool call and returns a structured response. The agent uses the result to continue reasoning.

2. A2A: agent to agent communication.
   An agent that can't complete a task alone discovers a capable peer via its Agent Card (published at a well-known URL), delegates the task, and receives a structured result back. If the second agent needs more input mid-task, it pauses in the input-required state and loops back to the first.

3. ACP: agent to agent communication over REST (merged into A2A).
   ACP took a REST-first approach. Peers were discovered through an Agent Manifest, called directly over HTTP, and responded to sync for low-latency tasks or via async SSE stream.

In production, MCP and A2A are complementary. MCP handles tool access, A2A handles agent communication.

Are you running MCP and A2A together in your agent stack?

Introducing Attio: the agentic CRM. (Sponsored)

Attio, the agentic CRM, makes it incredibly easy for anyone to run workflows for any GTM play they need.

Describe what you want, and Attio builds it. I just built a workflow that runs every morning, surfaces the deals that need my attention today, like anything with a stage change or a new signal in the last 24 hours.

Hundreds of thousands of automations already run on Attio every day. Ready to try now?

Try it today

LLM vs RAG vs Agent evals

LLMs, RAG pipelines, and agents are different systems, but the recipe for evaluating them is the same: pick a task, collect eval data, develop a grader.

The diagram below breaks it down across four popular AI systems.

1. LLMs: Input is a prompt, output is text. Tasks focus on what a raw model can do on its own: safety, code, instruction-following. Grader is often LLM-as-judge on the final answer.

2. RAG: Adds a retriever before the LLM. So you grade two things: retrieval (right docs?) and generation (faithful answer?).

3. Coding Agents: The LLM gets tools and a loop. Tasks become end-to-end like bug fixing and long-horizon planning. Grading is mostly code-based: run unit tests on the final patch.

4. Multi-Agent Systems: Multiple agents coordinating through an orchestrator. Tasks shift to coordination and role adherence. Grading blends code tests, LLM-as-judge, and human review.

Every new component in the pipeline is a new place for things to go wrong, and a new thing your evals need to catch.

Over to you: What's your go-to evaluation metric for multi-agent systems?

8 Frontier Open Models I’m Most Excited About

• Inkling (Thinking Machines): released this week, now the strongest American open model. Text, image, and audio input.

• Nemotron 3 Ultra (NVIDIA): a solid choice for long-running agents. The Mamba hybrid keeps long-context inference cheap.

• GLM-5.2 (Z.ai): currently the best open model for coding.

• Kimi K2.6 (Moonshot): strong on long agent tasks. Holds up over hundreds of tool calls.

• DeepSeek-V4 Pro: a very cheap way to get frontier-level quality over an API.

• Qwen3.6-35B (Alibaba): the best model to run on your own machine. A single 24 GB GPU is enough.

• Gemma 4 31B (Google): the best choice for on-device multimodal. Takes image and audio input on a gaming GPU.

• MiniMax M3: the only open model with native video input.

Over to you: Which open model are you most excited about?

How Distributed Tracing Works at the High Level?

1. Services generate telemetry data (traces, logs, metrics) as they handle requests.

2. The OpenTelemetry Collector receives this data from all services in a unified format.

3. The collector splits the data into three streams: traces, logs, and metrics.

4. Each stream is sent to a Receive & Process unit that prepares it for storage and analysis.

5. Processed data is stored in a Log Database for querying and long-term access.

6. Data from the database is visualized through a Visualization dashboard for monitoring and debugging.

Over to you: What else will you add to better understand distributed tracing?

The Life of a Redis Query

Redis is an in-memory database, which means all data lives in RAM for speed. However, if the server crashes or restarts, data could be lost.

To solve this problem, Redis provides two persistence mechanisms to write data to disk:

1. AOF (Append-Only File)

   When a client sends a command, Redis first executes it in memory (RAM). After that, Redis logs the command by appending it to an AOF file on disk. This ensures every operation can be replayed later to rebuild the dataset. Since the command is executed first and logged afterward, writes are non-blocking. The recovery process uses the event log to replay the recorded commands.

2. RDB (Redis Database)

   Instead of writing every command, Redis can periodically take snapshots of the entire dataset.

   The main thread forks a subprocess (bgsave) that shares all the in-memory data of the main thread. The bgsave subprocess reads the data from the main thread and writes it to the RDB file.

   Redis uses copy-on-write. When the main thread modifies data, a copy of the data is created, and the process works on that so that writes don’t get blocked. The snapshot is then written as an RDB file on disk, allowing Redis to quickly reload the snapshot into memory when needed.

3. Mixed Approach

   In production, Redis often uses both AOF and RDB. RDB provides fast reloads with compact snapshots. AOF guarantees durability by recording every operation since the last snapshot.

Over to you: Have you used Redis in your project?

Should each customer get a separate copy of the system, or should they all share one?

Running a dedicated copy per customer is easy to reason about, but the cost of maintaining hundreds of separate copies increases with every new customer. On the other hand, sharing a single system across all customers is far cheaper. This is why most of the software we use every day is built to be shared rather than duplicated. Such systems are known as multi-tenant systems, where each customer is a tenant that shares resources with other tenants.

However, sharing is also where the difficulty begins.

Once many customers rely on the same database, the same servers, and the same background jobs, their fortunes become linked. For example, one customer running a heavy report can slow the service for everyone alongside them. A single faulty deployment can impact all of them at once instead of just the customer. And the most serious risk is around data leaks, where one customer is shown data that belongs to another. All of these situations have to be actively handled by a multi-tenant system.

In this article, we will understand multi-tenant architecture from the basics, along with its various benefits and challenges. Here’s a quick overview of what we will cover:

• What a tenant really is.

• Where each customer’s data can physically live, from one shared set of tables to a database of its own, and what each of those options costs.

• Why the same choice between isolating and sharing also includes the compute layer and beyond, not only in the database.

• The noisy neighbor problem, and why sharing can impact fairness and force a team to rebuild it with quotas and limits.

• The role of blast radius in such an architecture.

• The concept of tenant context that runs through every part of the system.

Tenants and Users

Read more

Matic is the first visually intelligent robot vacuum that sees your home like you do, so it can clean how you want it to.
Matic’s hero features:

• Runs entirely on cameras to deftly navigate obstacles

• Recognizes floor types to auto-switch between vacuuming and mopping

• Big wheels and a height-adjustable cleaning head handle thick rugs

• Quieter than conversations at 55 dB

• Handles pet hair without clogging or tangling

• A single bag collects dry and wet waste—diaper-salts gel dirty water, antimicrobial powder prevents mold

• A fresh HEPA filter in every bag for cleaner air, no washing or replacing

Get your Matic now

Travel platforms are increasingly using AI for customer support. This means an increase in automated answers with less human intervention.

However, the cases handled automatically are largely lookups and routine changes that map cleanly onto a structured workflow. The cases that remain are typically disputes, such as a host and a guest having different views about a cancellation. In such cases, the travel platform is positioned between them and is accountable for the funds. The platform needs to figure out how to deal with these cases. It is not just a business problem, but also an engineering problem.

There is an important question at the center of this problem. At what moment should the system stop and route a case to a human agent?

A platform fielding hundreds of millions of contacts a year faces that decision millions of times a day, across cancellations, refunds, and after-hours lockouts. A confident but wrong answer can ruin a traveler’s trip and the money at stake.

Airbnb, Booking[dot]com, and Expedia have each tried to solve this problem in different ways. Airbnb prioritized autonomous resolution, building models that settle cancellations and refunds before an agent enters the conversation. Booking concentrated on the handoff, routing complex cases to a representative briefed in advance while equipping hosts to answer guests directly. Expedia emphasized deflection at scale, investing in summaries that carry context across more than thirty languages so an escalated case arrives intact.

In this article, we will look more closely at the different solutions by following the support pipeline from first principles, show why a tail of cases resists automation regardless of model quality, and use these three approaches to understand how these can be handled.

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

The Pipeline

Start with the simplest version of automated support: a bot that matches a traveler’s message to a list of frequently asked questions and returns the closest article.

It works for questions like “What is your cancellation policy?”

However, it fails the moment a traveler writes something like “My host agreed to refund me, so what happens if I cancel now?” That sentence carries an intent, a history, and a financial consequence, and a simple keyword match treats it as a search query.

The first real component is intent detection. In plain terms, the system classifies the message and identifies what the traveler wants: a rebooking, a refund, or a question about an existing reservation.

Airbnb frames this as a layered problem:

• A top-level model performs domain classification, sorting the message into a broad category such as cancellation or a help-article question.

• Once the category is settled, a second layer of domain-specific models takes over to work out the details.

For a cancellation, those second-layer models answer focused questions. Did the host or the guest initiate it? Have both sides agreed on a refund amount?

Alongside this, a separate model predicts the expected refund ratio (such as the share of the payment a human agent would typically return given the circumstances of the trip). That predictor is trained on years of past agent decisions, which lets the system produce decisions in line with how experienced staff have handled similar cases.

However, a message understood is still inert until two further capabilities exist:

• The system tracks state, meaning it carries the context of earlier messages in the thread, so a follow-up reply makes sense against what came before.

• And it has an action layer, the part wired into live systems that can issue the refund, move the reservation, or open the mutual cancellation flow.

Reading intent is the easier half. Taking a correct, reversible action against live booking and payment systems is what requires greater consideration.

The final element is a confidence threshold. Each prediction arrives with a score, and the threshold determines whether the system proceeds on its own or routes the case to a person. If this bar is set too low, more cases clear automatically while errors creep upward. If it is raised, the accuracy improves, but more traffic lands on human agents. That single threshold value is where much of the tuning happens

This pipeline, intent detection, state tracking, an action layer, and a confidence threshold, handles a large volume of everyday requests well. However, it keeps facing problems in one family of cases.

Adjudication

The cases that clear automatically share a quality. They are answerable by retrieval. In other words, the system finds a fact or applies a rule, such as the status of a refund or the steps to change a date. Volume here is high, and the work is repetitive. These are the types of problems where automation works well.

The cases that resist automation differ in kind.

They require adjudication, meaning a judgment between parties whose accounts and interests diverge. A guest says the apartment was misrepresented. The host says the listing was accurate and the guest simply changed their mind. A damage claim sits between them, and the platform holds the deposit.

There are three parties, three versions, and one decision about money. A better model produces a cleaner summary of the dispute, yet the underlying call still rests on weighing competing claims, which is difficult for even a powerful model.

Time pressure sharpens the divide.

When flights were cancelled across the Middle East, and a surge of travelers contacted Expedia at once, automation absorbed the routine rebookings and status checks at speed, which freed human agents to spend their attention on the tangled, time-sensitive cases. The same event produced both kinds of work in the same minutes, and the value of the split was that each kind reached the right handler.

This is why the resolution rate climbs and then flattens. The boundary separating automated from escalated cases advances as retrieval improves, and it approaches a region of adjudication-heavy cases that stay with people by design.

It also explains why headline percentages deserve careful consideration.

Airbnb reports that more than 40% of guest issues are resolved without an agent. Expedia reports that more than 30% of its self-service interactions are powered by AI, where self-service already accounts for more than half of all contacts.

Those two figures rest on different bases, so a side-by-side ranking would mislead more than it informs. Each number describes its own pipeline rather than a shared scoreboard.

Handoff

When a case crosses to a person, the details of what is shared become important.

A handoff that passes along only “escalated, please assist” forces the customer support agent to restart the conversation, and the traveler retells the whole story to a second responder. A traveler who repeats every detail to a human after a stalled bot session ends up more frustrated than one who reached a person at the start.

This asymmetry is the reason handoff deserves real engineering attention. A weak handoff can leave the combined system performing worse than plain human support would.

The solution to this lies in the payload: the bundle of context that accompanies an escalated case. A strong payload carries four elements.

• A summary of the conversation so far.

• The structured facts already gathered include the booking reference and the cancellation reason.

• The live state of the reservation.

• A translation, where the agent and the traveler use different languages.

The agent opens the case already informed and continues from where the automation left off.

Expedia leans on this idea at scale. Its systems generate conversation summaries across more than thirty languages, so a case handled partly in Portuguese can reach an English-speaking support agent with the thread rendered in a language they read fluently. The same summaries shortened the time required to bring new agents up to speed by a substantial margin, because a well-formed summary teaches as it informs.

Booking describes the same priority from the other side. Its tools route complex questions to a representative who receives the relevant booking and property details in advance, so the human starts the conversation already holding the facts.

Booking also adds a second component that the others treat as secondary. Much of its support volume is communication between a guest and an accommodation partner, so it built tooling that drafts partner replies from property and reservation data and suggests them to the host. Here, the friction is the message between two people rather than a decision the platform owes, and equipping the partner to answer quickly relieves pressure before a formal support case ever opens.

The pattern across all approaches is consistent.

The success path contributes to the headline percentage, while the failure path, the moment a case escalates, is where the overall customer experience is decided. The core work for building such systems concentrates on handoff far more than on the straightforward case where everything resolves on the first attempt.

Divergence

The design decisions made by the three companies indirectly express three beliefs about which part of travel support is hardest.

Airbnb’s position is that the adjudication itself can be modeled. By training on the expected refund ratio and on the questions agents ask during a cancellation, it pushes the automated portion as far toward real decisions as the data supports, and it reserves human attention for safety, discrimination, and high-value claims. The wager is that a large share of disputes follow learnable patterns, and that an autonomous system applying a consistent standard serves travelers better than a queue of agents who each interpret a policy slightly differently. The cost of this position is a heavy, continuous investment in models and labeled data, and a high bar for caution wherever money and safety meet.

Booking solves the problem differently. Its emphasis on briefing the human agent and on drafting replies for partners treats communication as the primary friction. The implied belief is that a great deal of support volume is people struggling to reach each other, a guest with a question and a host with the answer, and that closing that gap quickly matters more than resolving every case autonomously. This keeps more humans in the loop while raising the quality of each human touch.

Expedia’s emphasis on deflection and multilingual summaries reflects a third belief that scale is the dominant variable. With well over two hundred million interactions a year, even modest gains in self-service rates and handling times compound into large absolute numbers, so the architecture optimizes for volume routed correctly and for context that survives a language boundary across a global workforce.

Tradeoffs

Every placement of the boundary carries a price on each side, and the confidence threshold makes the trade concrete.

• Raise the threshold, and the system escalates more often, which guards against autonomous errors while increasing the load on human agents and the cost that comes with it.

• Lower the threshold, and the system resolves more cases on its own, which trims cost while admitting more mistakes in situations that involve money.

There is an operating point that balances the two, and choosing it requires judgment about how expensive an error is relative to an escalation. In travel, where an error can mean a wrongly denied refund on someone’s holiday, that judgment leans conservative wherever the stakes run high.

There is also a more fundamental limit worth considering.

The chat interface itself suits travel poorly, because a chat thread is built for one person, while a travel dispute often involves several. A cancellation between a host and a guest, mediated by a platform, is a conversation among three parties, and a linear thread struggles to hold that arrangement sometimes.

It is not simply a matter of improving the interface. It suggests that the format through which support arrives may bound how far automation can go, separate from how capable the model becomes.

For that reason, regulated cases, high-value claims, and anything touching safety remain with people on purpose. The boundary moves rightward over time, and a margin is kept on the far side, where human judgment is meant to outweigh a confident automated answer.

Conclusion

The percentage of issues resolved without a human is an indication of two things: where a platform places its automate-versus-escalate boundary and how much of its work is answerable by retrieval rather than judgment. The model matters, yet it operates inside a structure built to manage that boundary.

A few points hold across all three platforms.

• The pipeline serves the boundary. Intent detection, state tracking, the action layer, and the confidence threshold all feed the single decision to resolve a case or escalate it.

• The handoff decides the experience. A case that crosses to a person succeeds or fails on the context that travels with it, which is why summaries, structured facts, and reservations carry so much weight.

• The placement encodes a belief. Airbnb bet on autonomous adjudication, Booking on communication and a briefed handoff, and Expedia on correctly routed scale, with each placement reflecting a different view of where travel support is hardest.

Put together, the resolution rate reflects a design decision as much as a level of capability, and the boundary is the part of the system worth reading first.

References:

• Task-Oriented Conversational AI in Airbnb Customer Support.

• Airbnb, Inc., Q1 2026 shareholder letter and earnings call.

• Booking.com Debuts Agentic AI Innovations.

• Expedia Group, Q1 2026 financial results and earnings call.

• Expedia Group Sets the Standard with AI-Powered Service Agent.

• AI solutions.

Enhance visibility into your cloud architecture with expert insights from AWS + Datadog. In this ebook, AWS Solutions Architects Jason Mimick and James Wenzel guide you through best practices for creating professional and impactful diagrams.

Get the eBook

Ask a model the same question twice, phrased two different ways, and watch how it decides what to give you. It might offer a terse one-liner or a careful walk-through. It might sound fully confident, or it might hedge. Somewhere in training, the model learned which version you would rather have, even though both answers are correct.

That points to a harder question. How do you teach a model to be helpful when helpfulness changes with every request?

For a quick factual query, helpful means brevity. For a debugging problem, it means thoroughness. For a medical worry, it means caution paired with clarity. Each case requires a different direction with underlying trade-offs.

The model learned to handle those trade-offs from people comparing answers thousands of times over. This idea of learning from comparison rather than from a fixed answer key sits beneath every popular model you have used.

In this article, we will look at how that learning actually happens, starting with why instruction-following alone falls short, then walking through the two main methods for teaching preferences (RLHF and DPO).

Pipeline

A useful model gets built in three stages, as shown in the diagram below:

Let’s look at each stage in a bit more detail:

• The first stage is pretraining. The model reads an enormous amount of text and learns to predict the next word. In other words, it builds a broad sense of how language works and soaks up world knowledge along the way. What it gains is raw capability. What it still lacks is direction, because predicting the next word and answering your question are two different goals.

• The second stage is supervised fine-tuning, or SFT. We show the model thousands of example pairs, each an instruction together with a good response, and train it to imitate them. After this stage, the model follows instructions instead of continuing text. For example, if we ask a model to summarize a paragraph, it writes a proper summary. A raw model without SFT would have rambled on.

• The third stage teaches preferences, which will be the focus of the rest of this article.

Why does a model that already follows instructions need a third stage at all?

This is because SFT works through imitation. The model sees a correct answer and learns to copy its shape, which holds up well when a question has one good answer, but runs into problems when a question has many possible answers.

Consider a request like “explain how a hash map works”. A short answer and a long answer can both be excellent, and which one helps depends on who is asking. An example dataset can show one of those answers, while the trade-off between them stays invisible. Imitation teaches the model what a good answer looks like, but struggles to teach it how to weigh two good answers against each other.

The payoff is larger than it might appear at first. For example, when OpenAI built InstructGPT in 2022, human raters preferred answers from a 1.3 billion parameter aligned model over answers from the 175 billion parameter GPT-3, a model roughly a hundred times larger. The alignment stage mattered more than a hundredfold jump in size.

Nevertheless, the line between pretraining and SFT is cleaner in the example than in practice. Some recipes blend the two, but the three-stage view is the right place to start.

Next up, we need a signal that captures the trade-off directly. That signal is comparison.

Introducing Attio: the agentic CRM. (Sponsored)

Attio, the agentic CRM, makes it incredibly easy for anyone to run workflows for any GTM play they need.

Describe what you want, and Attio builds it. I just built a workflow that runs every morning, surfaces the deals that need my attention today, like anything with a stage change or a new signal in the last 24 hours.

Hundreds of thousands of automations already run on Attio every day. Ready to try now?

Try it today

Preferences

Comparison turns a judgment call into something a model can learn.

A preference example is simple. We take one prompt, generate two responses, and ask a person which is better. The result is a small piece of data, a prompt with a winner and a loser. If we collect enough of these, a pattern emerges that captures the trade-offs accurately.

See the diagram below:

The reason this works comes down to what people are good at. Writing the single ideal response to a tricky prompt is hard, and two skilled people would produce different versions. Judging which of two responses is better is far easier, and people agree with each other more often when they compare than when they create. This is why labeling teams compare outputs instead of writing perfect ones from scratch, which makes the data trustworthy enough to train on.

This is the same comparison data InstructGPT collected, and the starting point for both methods we are about to compare. They differ in what they do with it, while the comparisons stay the same.

To clarify, the comparisons can come from another AI model in place of a person, an approach used in RLAIF and Constitutional AI. The overall mechanics stay the same.

Let us now look at the two different methods.

RLHF

RLHF, short for Reinforcement Learning from Human Feedback, learns from comparisons by training a separate scorer and then coaching the model to score well against it.

See the below diagram that shows the basic RLHF flow:

It works in two moves once the comparisons are collected.

• First, we train a reward model. We take those prompt-with-a-winner examples and train a separate model to score any response, high for the kind people preferred and lower for the kind they passed over. The reward model turns scattered human judgments into a single function that grades fresh responses on its own.

• Second, we use that scorer to improve the main model, which here we call the policy. The policy writes a response, the reward model scores it, and a reinforcement learning algorithm called PPO nudges the policy toward higher-scoring responses.

Reinforcement learning, in plain terms, means learning from a reward signal through trial and error in place of labeled examples. Run the loop many times, and the policy drifts toward the answers people prefer.

Left to chase a score freely, a model will discover strange, degenerate text that games the scorer while reading badly. To guard against that, we keep a frozen copy of the starting model, the reference, and penalize the policy whenever it strays too far from it. This penalty, the KL term, lets the policy improve while staying close enough to fluent language to stay useful.

The reward model, the policy, the frozen reference, and a value model that PPO uses internally add up to four models in play during training.

That is the source of RLHF’s reputation for being expensive, fiddly, and slow to get right. However, this is also the pipeline behind the first version of ChatGPT, and it produced the first generation of genuinely helpful assistants at considerable cost.

Running a separate reward model and a full RL loop is a lot of moving parts. In 2023, a study from Stanford showed that much of it can collapse into a single step.

DPO

DPO reaches the same goal with the same data, replacing the reward model and the RL loop with one training step.

DPO stands for Direct Preference Optimization, and the name fits. Rather than train a scorer and then optimize against it, the method tunes the model directly on the comparison pairs.

The mechanics sit close to ordinary supervised learning.

For each comparison, DPO adjusts the model to raise the probability of the preferred response and lower the probability of the rejected one, measured against that same frozen reference from before. The whole thing becomes a single loss that would train the way you would train a classifier. This is far simpler and steadier than an RL loop.

A common idea is that DPO removes the reward model, but what actually happens is subtler. The Stanford paper that introduced DPO carries the idea in its title that a language model is secretly a reward model. The reward signal is still present, but it now lives inside the policy itself rather than in a separate network.

The approach proved itself quickly.

Zephyr, a 7 billion parameter model trained with DPO on machine-generated comparisons, beat Llama 2 Chat 70B, the strongest open RLHF model at the time, on a standard chat benchmark. Since then, DPO has become a staple in the industry.

By folding the reward into a single training step, DPO put alignment within reach of a small team on a modest budget, far from any large research cluster. A family of variants, such as SimPO, KTO, and ORPO, adjusts the objective in different ways. They share DPO’s core approach.

On cost and simplicity, DPO looks like a clear win. The catch is one it shares with RLHF, and it sits in the signal both methods trust.

Reward Hacking

Both methods rely on a signal that only approximates human judgment, and leaning too hard on it can also backfire at times.

The reward model, whether trained explicitly in RLHF or folded into the policy in DPO, is a proxy. It approximates what people prefer. Like any approximation, it also has gaps. Optimize the model hard against that proxy, and it can start to exploit those gaps. In other words, the score climbs while the answers get worse.

Researchers at OpenAI measured this directly.

As the model optimizes against the proxy, true quality rises for a while, then peaks and slides backward even as the proxy score keeps climbing. They tied it to Goodhart’s law, the old observation that a measure stops being a good measure once it becomes the target.

The everyday symptoms of this are familiar.

Models pad answers with length because longer responses tend to score well, and they learn to agree with whatever view you express. This behavior is called sycophancy. Anthropic found that most of the time, both human raters and reward models prefer a confident, agreeable answer over a correct one, which sometimes means flattery in place of truth.

The important part is that this trouble follows the data rather than the algorithm.

DPO inherits it just as RLHF does, because both learn from the same imperfect human judgments. Simpler training buys you a great deal, and this particular problem comes along for the ride. This is why your model sometimes agrees with you when you are clearly wrong. It learned that agreement tends to score well.

The KL penalty we discussed earlier softens this by holding the model near its starting point, though it shrinks the problem rather than completely erasing it.

If the root trouble is an imperfect human signal, a natural question follows. Could we replace it with a signal that is exact? For a certain class of tasks, we can.

Verifiable Rewards

When a program can check the answer, the human reward becomes optional, and that single fact reshaped the final training stages across 2025 and 2026.

Some tasks have a checkable answer:

• A math problem is right or wrong.

• A piece of code passes its tests or fails them.

For these, we can replace the human-derived reward model with a verifier, a program that grades the result directly. The signal becomes exact and effectively free, clearing the proxy problem at its root for that work.

DeepSeek pushed this idea furthest.

They introduced an RL method called GRPO that drops one of the extra models PPO needs, and they trained their R1 reasoning model largely by rewarding correct final answers on math and code. The results matched the strongest closed reasoning models of the time. The model learned to reason by checking its own work, guided by a reward a calculator could supply.

The limit is that a verifier can grade a math proof. However, it stays silent on whether an answer is honest, kind, or appropriately cautious. Those qualities take many acceptable forms and resist any automatic test, the territory preference learning was built for. DeepSeek themselves kept reward models for the helpfulness and safety parts of R1, while verifiable rewards drove the reasoning.

This gives us the rule for the whole landscape.

When a machine can check the answer, reach for verifiable rewards. When the answer is a matter of judgment, reach for preference learning, whether RLHF or DPO. The method follows the signal.

See the diagram below:

This is why reasoning models improved so quickly, and why even the best of them still rely on preference methods to stay pleasant and safe. Newer methods, such as DAPO and RLVR, extend the verifiable-reward idea, and they make up much of the active frontier in 2026.

Conclusion

We can retrace the path in a sentence each:

• A raw model learns language and knowledge from pretraining.

• Supervised fine-tuning teaches it to follow instructions by imitation.

• A third stage teaches preferences, because the questions that matter most often have several good answers and a real trade-off between them.

Comparison is the signal that captures that trade-off, and the two main methods both run on it.

RLHF trains a separate reward model and optimizes against it with reinforcement learning, which is powerful and costly. On the other hand, DPO folds the same signal into a single training step, which is simpler and steadier while keeping the reward implicit rather than absent.

Both inherit the same catch, that a human signal is a proxy, and pushing on it too hard breeds sycophancy and quiet failures. And when a task has a checkable answer, verifiable rewards now sidestep the proxy entirely.

References:

• Training language models to follow instructions with human feedback

• Direct Preference Optimization: Your Language Model is Secretly a Reward Model

• Zephyr: Direct Distillation of LM Alignment

• Scaling Laws for Reward Model Overoptimization

• Towards Understanding Sycophancy in Language Models

• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning

Debugging SSO, managing users, adjusting auth policies, configuring branding: every configuration task has lived behind a UI that only a human can drive.

The WorkOS MCP server gives agents the same access as your dashboard login. Hundreds of operations, discoverable at runtime. Connect in one command via OAuth, with scoped tokens instead of a master API key. Pass a screenshot of your marketing site and ask your agent to match the login page. If a human had to do it before, an agent can do it now.

Connect your agent →

Microsoft operates at an enormous scale. More than 80,000 enterprises now build on Microsoft Foundry, the company’s platform for building, deploying, and running AI agents and applications. Microsoft’s own copilots run on the same platform, including Microsoft 365 Copilot, which alone serves over 20 million users and has a monthly active usage of first-party agents growing 6x year-to-date.

To understand what it actually takes to ship agents at that scale, we spoke with Marco Casalaina, VP of Products for Microsoft Core AI. He walked us through what his team has learned from running these systems in production, the engineering challenges that come with it, and where he thinks enterprise AI is headed next.

In this article, you’ll learn:

• Why a prototype agent doesn’t survive in production

• What’s in a production agent harness, and why Microsoft believes context is the key

• Two engineering ideas behind Foundry: retrieval-as-a-subagent, and giving agents their own identity and a place to act

• How Microsoft evaluates production agents with rubric-based evals and an auto-improvement loop

• Lessons for other teams, and what’s next

What Breaks When Agents Hit Production

Production agents fail for reasons that aren’t visible in a prototype. The model is rarely the problem. What breaks is everything around the model, including the data the agent retrieves, the tools it calls, the way it handles real users, and the way quality drifts as the world around it changes. Enterprises trying to ship agents this year are running into a different engineering problem than the one they were solving last year.

To see why, it helps to start with what’s actually changed about what enterprises are trying to build. Marco framed the shift this way:

We are leaving the question-answering phase of AI. In 2026, we are seeing a huge increase in the number of our customers that are using voice as a front-end, so we’re also leaving the chatbot era of AI.

The old shape was a chatbot. The user types, the agent types back, and it can only answer questions. The new shape is an agent that does meaningful work on the user’s behalf. It books the meeting, runs the analysis, sends the email, files the ticket. The user might not type at all because the front end can be voice. For example, Foundry’s Voice Live lets a team turn an existing text agent into a voice agent without rebuilding it.

This shift is what makes the engineering problem different. A chatbot returning a wrong answer is a bad experience. An agent taking the wrong action is a business incident. The bar for what’s good enough to ship has moved.

That’s where the gap between a prototype and a production agent opens up. The first prototype is easy. You can vibe-code one in an afternoon. The model is smart, your test prompts work, the demo is impressive, and the pilot ships in a week.

Production is where the cracks open. Real users ask things you didn’t anticipate. Documents the agent depends on go stale. New edge cases emerge that never appeared in your eval dataset. A model update changes the agent’s behavior subtly, and nobody notices until a customer complains. Without identity controls, the agent runs as a shared system principal and there’s no audit trail when something goes wrong. Without guardrails, it confidently says something it shouldn’t. Without observability, you can’t tell whether quality is improving or degrading. None of these problems showed up in the prototype. All of them show up in production.

When we asked Marco for the single biggest lesson the Foundry team has learned from running these systems at scale, his answer was “the harness matters as much as the model.”

The harness is everything around the model. The runtime, the tools, the context retrieval, the identity layer, the guardrails, the evaluators, the deployment pipeline. Models change constantly, and you cannot treat them like database versions. With Postgres, you change versions and you pretty much expect it to straight up work. Models are not like that. Each one has different properties that the harness has to adjust to. When Anthropic released Claude Opus 4.8, Microsoft’s GitHub Copilot CLI team had to re-tune their harness and re-run their evaluations before they could ship it.

See what 406 infrastructure leaders revealed about who’s ready for AI (Sponsored)

AI is reshaping infrastructure, but not how most teams expected. The teams getting the most from it aren’t adopting the fastest. They’re taking advantage of the platform, governance, and automated pipelines that allow AI-generated infrastructure to move safely to production.

The 2026 Infrastructure Automation Report surveyed 406 infrastructure and platform engineering leaders to reveal how Pioneers are benefiting the most from AI. You’ll learn:

• Why platform engineering is the critical component of AI adoption

• New AI-specific signals and metrics that need monitoring beyond traditional metrics

• 5 tactical steps to reach the Pioneer segment of the Al Maturity index

Download the report

What’s in a Production Agent Harness

If the harness matters as much as the model, the next question is what’s in it. Walking the harness from the bottom up shows why each layer exists and what breaks if you try to do without it.

1. Inference Layer

At the bottom is the inference layer, the single interface the harness uses to reach the models. The models themselves sit outside the harness and stay swappable. Different agents need different models, and the right one changes every few weeks. Foundry supports more than 11,000 of them, from OpenAI, Anthropic, xAI, DeepSeek, and Microsoft’s own MAI family.

2. Agent Runtime

Above the model is the agent runtime, which is what turns a model into an agent. The runtime handles the orchestration loop, the tool calls, the conversation state, and the protocol the rest of the harness speaks.

Not every step in that loop should run through the model. A well-built agent sends only the parts that need reasoning to the LLM and leaves the rest to ordinary code, since a database lookup or a purpose-built extraction model is faster, cheaper, and more reliable than asking a model to do the same work.

The number of agent frameworks keeps growing, from open-source options like LangChain, LangGraph, and CrewAI to vendor-built runtimes. The principle that matters is framework neutrality. An agent built on one framework should be portable to another without rewriting the surrounding harness. Foundry, for example, lets agents run on any framework interchangeably.

3. Observability and Governance Layer

Once agents are in production, an organization needs observability and governance. It needs a single view of every agent running across every project, with health scoring, token usage, latency metrics, drift detection, and the kind of cross-project rollups that let a platform team govern a fleet. Without this layer, regressions are invisible and cost is uncontrolled. In Microsoft’s case, this is Foundry Control Plane, which provides cross-project fleet visibility and routes agent telemetry into Azure Monitor and Application Insights, the same pipeline that already handles infrastructure alerts.

4. Identity Layer

Once agents start taking real actions inside an organization, they need their own identities. They need their own role assignments and their own audit trails, because a misbehaving agent has to be bounded by the same access controls that bound a misbehaving employee. The industry is still converging on how to do this well, and the answer most platforms are landing on is to extend an existing enterprise identity system to treat agents as a new class of principal rather than inventing a parallel system. For example, Foundry extends Entra, Microsoft’s enterprise identity platform, to treat agents this way. This is the access-control primitive; later in the article, the section on giving agents a place to act shows what an agent does once it has one of these identities.

5. Context Layer

Once an agent can run reliably and act with its own identity, the question becomes whether it can answer correctly. That’s the job of the context layer. While every other layer in the harness exists to let the agent run, context is what lets it run correctly. An agent without good context might hallucinate. It will give an answer, but the answer will be wrong in ways that are hard to detect because the agent itself doesn’t know what it doesn’t know. Marco was explicit that giving agents the context they need to actually work is one of the hardest problems his team is solving, and the one Microsoft is reasonably passionate about getting right.

The next section is about how Microsoft is building that context layer.

Building a Context Layer for Agents

The hard part of giving agents the right context isn’t that the context is missing. Enterprises have enormous amounts of it. The hard part is that the context lives everywhere. It lives in unstructured documents in SharePoint and wikis. It lives in structured tables in OneLake and data warehouses. It lives in productivity apps like Outlook, Teams, and Word. No single retrieval method can reach all of it.

The standard answer of the last two years, classic retrieval-augmented generation (RAG), was never designed to. Classic RAG is a one-shot pattern. You take the user’s question, embed it, search a single index, return the top-k results, and pass them to the model. It works for simple questions against a small, clean corpus. It breaks the moment the question is ambiguous, the corpus is heterogeneous, the right answer requires combining sources, or the first retrieval comes back empty. The agent has no way to recover from a bad retrieval, and as Marco put it, when RAG isn’t working well, the whole agent isn’t working well. One-shot is the wrong shape for the problem.

The fix is to treat retrieval as something the system can iterate on, the same way an agent iterates on a task.

Plan the query, try a source, evaluate the result, try a different source if the first one came back empty, and combine what you find. That’s the central engineering idea behind a production context layer.

Microsoft’s answer is to ship the context layer itself as a set of services. There are four of them, collectively called Microsoft IQ. Foundry IQ handles unstructured data, Fabric IQ handles structured data, Web IQ handles real-time web retrieval, and Work IQ handles the productivity surface of Microsoft 365, including email, calendar, documents, and Teams.

Each IQ is a headless service that agents call through MCP. Two engineering ideas run through them. The first, retrieval-as-a-subagent, is the pattern behind all four. The second, giving an agent an identity and a place to act, is specifically what Work IQ adds on top of retrieval so that an agent can do work, not just find information.

Retrieval-as-a-Subagent

The technical idea behind a production context layer is to wrap retrieval in an agentic loop. Instead of a single function call that runs one query against one index and returns the top-k, retrieval becomes a small agent of its own. It plans which sources to query, executes the queries, evaluates the results against the original question, and decides whether to return them, refine the query, or try a different source. Retrieval stops being a lookup and becomes an iterative process that can recover from a bad first pass.

Foundry IQ is the cleanest example of this in production today. The figure below shows the workflow when an agent calls it.

The last step is the one most worth paying attention to. When iteration runs out, Foundry IQ returns a structured “I don’t know” instead of forcing an answer. Classic RAG has no fallback and the model hallucinates something plausible. Foundry IQ gives the calling agent a clear signal that retrieval failed, which it can act on, rather than a confident wrong answer that’s hard to detect.

So far this is all about data. The same loop applies to tools. An agent with a few tools can list them all in its prompt. One with dozens can’t. Listing them all burns context on every call and slows the model down as it scans the list. The fix is similar to agentic retrieval. Instead of exposing every tool, the agent searches for the right one, gets it back, and calls it. Foundry exposes this as tool search, and it’s the pattern OpenAI’s and Anthropic’s agents have converged on too. So the same retrieval loop that pulls in knowledge also pulls in capability. The agent looks up what it needs at the moment it needs it instead of carrying everything up front.

Other IQs rely on the same pattern too. Fabric IQ runs agentic retrieval over structured data, with the loop planning queries against OneLake tables and the Fabric data agent rather than against text indexes. Web IQ runs the same loop over the open web at sub-second latency. The retrieval-as-subagent idea is the architectural commonality across all of them.

An Identity and a Place to Act

Retrieval gets the agent the right information. But information alone doesn’t finish a task. An agent that knows the customer is unhappy still has to write the apology email and refund the order. To do that responsibly, the agent needs two things. An identity the organization can see, and a surface where it can take action.

The identity is the access-control primitive from the identity layer earlier. Without it, every action is anonymous or on behalf of a user who lent the agent their identity. Logs show that “the AI did it” and the audit trail collapses, which is unworkable for any regulated enterprise. The fix is to treat agents as their own class of principal, sitting in the same directory as employees, with their own role assignments and their own audit trails. A misbehaving agent gets bounded by the same access controls that bound a misbehaving employee.

The action surface is what makes the identity meaningful. An agent that exists in the directory but has no way to send an email or update a document isn’t doing work. The surface has to expose the same actions a person already takes inside the organization, so that an agent operating on top of it can read inboxes, send messages, schedule meetings, and edit shared documents. In Microsoft’s case, identity is handled through Entra and the action surface is Work IQ. An agent can now have a named directory entry, report to a manager in the org chart, and own its mailbox.

Identity bounds what an agent is allowed to reach. Guardrails bound what flows through it once it acts. A chatbot only had to screen the user’s prompt and the model’s reply. An agent has more surface to defend, because it also reads tool outputs and retrieved documents, and any of those can carry an instruction the user never typed. That is how indirect prompt injection works. A line like “ignore your previous instructions” sits hidden inside a document the agent ingests, and the agent treats it as a command. The fix is to move guardrails to the tool boundary, screening tool inputs and tool outputs rather than just the model’s input and output. In Microsoft’s case, Foundry runs its own classifiers at the tool-call and tool-response level, on top of whatever safety the model already has. Since these controls live on a shared tool layer rather than inside each agent, a team configures them once and every agent that uses those tools inherits them, instead of re-implementing the same guardrails, credentials, and policies agent by agent.

The takeaway for any team building this is that retrieval, action, and the guardrails around both need to be designed as first-class layers. Retrieval should be a loop that recovers from bad lookups, action should run under an identity the organization recognizes with an audit trail to match, and guardrails should sit at the tool boundary rather than only at the model’s edges.

Evaluate Production Agents

Context is half of what determines whether an agent survives production. Evaluation is the other half. An agent with the right context can still drift, regress, or fail in new ways as traffic patterns change. Evaluation is what closes the loop. It’s the system that tells you when something has changed, and the system that tells you whether the agent is doing what it’s supposed to do.

Continuous Evaluation

Most teams treat evaluation as a pre-ship checkbox. They run a test suite once, see green, and ship. That works for traditional software because traditional software is deterministic. An agent is not. The same prompt against the same model can produce different responses, the model itself changes when the provider releases an update, and the data the agent retrieves from changes daily. A pre-ship test suite catches a snapshot of behavior, not a guarantee of it.

Continuous evaluation closes that gap by running against live traffic, sampling real user interactions, scoring them on whatever criteria the team has defined, and surfacing results into the same observability pipeline that already handles infrastructure incidents. In Foundry, this is built in. When quality regresses, the team finds out from the same alerting system that pages them when a service goes down. The same evaluations can also run earlier, as a gate in the deployment pipeline, catching a regression before it ships.

Rubric-Based Evaluation

Continuous evaluation tells you when something has changed. The harder problem is knowing whether what the agent is doing is right in the first place. Most evaluation systems today rely on generic metrics like groundedness, coherence, and task completion. The metrics are useful but they have a ceiling. As Marco put it, generic metrics tell you whether the agent works. They don’t tell you whether the agent works right.

Consider an agent that helps users book restaurant reservations. It is the kind of wrong-action failure from the opening, seen up close. A generic metric can tell you that the agent called the booking tool successfully and returned a reservation. But it can’t tell you whether the agent did the right things along the way. Did it ask the user what time they wanted when the user only said “a table for two tomorrow”? Did it check the booking system to confirm the table was actually available before claiming a 6pm slot? A reservation that gets created without those checks is technically a success against generic metrics and a failure in the user’s eyes.

The way to close that gap is to evaluate the agent against the specific behaviors it should be showing, not just generic metrics. The common approach is rubric-based evaluation. A rubric is a specific, use-case-tied check that asks a yes-or-no question about something the agent should be doing. For the restaurant agent, the rubrics might look like:

• When the user gives a partial request, does the agent ask for the missing information?

• Before claiming a reservation is available, does the agent verify availability against the booking system?

• After making a booking, does the agent confirm the details back to the user?

In practice, the eval system runs the agent against a set of test interactions, scores each rubric per interaction, and rolls the results up. The team sees not just whether the agent works but which specific behaviors it’s getting right and which it isn’t. Rubrics work better than generic metrics because the team that owns the agent writes them, so they reflect what the agent is actually supposed to do.

Writing them by hand isn’t the only path. Foundry can draft a rubric by reading the agent’s configuration and its production traces, proposing the dimensions worth scoring from how the agent actually behaves. The team still owns and edits the result, but it starts from real traffic instead of a blank page.

Microsoft has built rubric-based evaluation into its Agent Optimizer suite. The Optimizer handles the rubric side of the loop and also makes the rubrics actionable by automatically improving the agent itself when a rubric fails. It can rewrite the system prompt to make the missing behavior more explicit, adjust how the agent uses its tools, or change which sources the agent prioritizes.

What it tunes goes beyond the prompt. It can also swap the underlying model and tune the agent’s skills. Rather than testing one fix at a time, it produces several candidates in parallel, scores each against the rubrics, and promotes the best one as a new agent version. The same loop that measures the agent now improves it. It becomes a self-improving loop.

For any team running agents in production, evaluation isn’t a phase that ends. It’s a layer that runs alongside the agent, with rubrics that evolve as the agent’s job evolves and an optimizer that turns regressions into actionable changes instead of manual fixes. The team’s job stops being to write the perfect agent on day one and starts being to keep the agent honest as the world around it changes. That’s what production AI actually looks like, and it’s the second half of the harness Foundry is built around.

The Key Lesson for Other Teams

The harness mattering as much as the model is the thread that ran through the whole conversation. The reason context is hard, the reason evaluation is hard, the reason a working prototype doesn’t survive production, is that the work most teams treat as “around the model” is actually what determines whether the agent works. Treating the harness as something you add later is the most common reason an enterprise agent never makes it to production.

The practical version of that lesson for a team building its own context layer is to take the two architectural ideas from earlier in this article seriously, even if you don’t use Foundry. Retrieval should be agentic. Calls to your retrieval layer should hide a planner-driven, multi-source, retry-capable subagent that can recover from a bad lookup instead of breaking on one. And agents that take actions should have identities the organization recognizes, with audit trails that survive a compliance review. Both of these are buildable on top of any platform.

What’s Next

Marco thinks about the future of AI by watching what’s about to become possible in the next year, not what’s ten years away. He tracks the moment a capability stops being a research demo or a developer tool and becomes something an ordinary person can use. The pattern he watches most closely is the one where capabilities that used to require a CLI coding agent become available to everyone.

This pattern has been playing out for the last eight months. Capabilities that used to require a developer to wire up inside a coding agent, things like writing a document from natural language, managing a calendar, or running a custom workflow, are now showing up in tools. Skills, which started as a coding-agent concept, are now in Excel for tasks like financial analysis and private equity workflows. Marco’s expectation is that this pattern keeps accelerating, with more developer-only capabilities crossing into general use over the next year.

The bigger shift he sees coming is toward agents that improve themselves. The architectural pieces are global and local memory, plus skills, plus the kind of behaviors that let an agent retain what it learns about a user and apply it automatically. Copilot CLI’s new Chronicle feature is an early version. If Marco always commits his Git repos to GitHub immediately after creating them, Chronicle learns that and starts doing it for him without being asked. Marco’s bet is that this combination, applied to the kinds of agents enterprises are already deploying, is the story of the next year. Not science fiction. Just the next set of capabilities crossing into general use.

Define tasks with Render’s lightweight SDK and chain them into long-running, distributed workflows. Launch your agents and batch jobs on demand. Render Workflows handles queuing, orchestration, and retries.

Get $50 credits with code BYTE

This week’s system design refresher:

• How Docker Works Under the Hood

• git merge vs git rebase

• 12 popular vector databases help you get the right context to the model

• Pagination Strategies for Large Systems

• How LLMs Use AI Agents with Deep Research

How Docker Works Under the Hood

A Docker container starts with a single command, but that command has to be turned into a running Linux process. Here is what actually happens.

The Docker CLI takes your command and sends it as an API call to the Docker daemon (dockerd) running on the host.

dockerd checks whether the nginx image is already on disk. If it is not, it pulls it from a registry like Docker Hub or ECR. Then it prepares the container config.

dockerd does not start the container directly. It passes the request to containerd, which manages the container lifecycle. containerd prepares the runtime files and assembles a bundle made of the OCI config and the root filesystem.

containerd then calls runc. runc reads the bundle, creates the Linux namespaces and mounts defined in the config, and starts the process inside them. Once the process is running, runc exits.

The running container is a regular Linux process with its own PID, network, and mount namespaces. Its filesystem is a stack of read-only image layers with a writable layer on top, so changes inside the container do not modify the image.

Isolation comes from kernel features. Namespaces separate processes, cgroups limit CPU and memory, and network namespaces give the container its own interfaces. There is no guest OS and no hypervisor.

Over to you: where do most of your container issues show up, the image, networking, or resource limits?

git merge vs git rebase

Both commands get your changes into main. The difference is what they do to history.

git merge preserves the original branch structure. If main and feature branches have diverged, Git creates a merge commit and keeps both lines of development . That makes it easy to see where work branched off, what got merged, and when it came back together.

git rebase takes a different approach. It reapplies your feature commits onto the latest main branch as new commits. The result is a linear history. But those rebased commits get new IDs, so rebasing a shared branch usually means force-pushing and making others sync to rewritten history.

That is why teams often use merge on shared branches. It keeps existing commit IDs unchanged, so everyone can pull without dealing with conflicts. Rebase is more useful for cleaning up your own branch before you merge it.

Over to you: which command do you prefer?

12 popular vector databases help you get the right context to the model

1. Pinecone: Fully managed, serverless with hybrid search (dense + sparse) native. Best when your team wants zero infrastructure overhead.

2. Weaviate: Built-in vectorization and native BM25 + vector hybrid search. Best when you need keyword and semantic search in one system.

3. Milvus: Distributed vector database with horizontal scaling and GPU acceleration support. Best for massive-scale workloads.

4. Qdrant: Rust-based vector search engine with dense, sparse, and metadata filtering built for production-grade retrieval. Best when control over ranking and latency matters.

5. Chroma: Embedded vector database that runs directly inside Python applications. Best for local RAG prototypes and experimentation.

6. Pgvector: Postgres extension that adds vector search using SQL. Supports HNSW and IVF indexes. Best if you're already running PostgreSQL.

7. FAISS: Meta's ANN library for vector similarity search. GPU accelerated and highly customizable. Best for custom retrieval systems.

8. Vespa: Combines dense vectors, sparse retrieval, and structured data in one engine. Best for large-scale search and ranking systems.

9. MongoDB Atlas Vector Search: Vector search integrated directly into MongoDB collections. Best if your application already uses MongoDB.

10. Redis Vector: Low-latency vector search alongside caching, pub/sub, and streaming. Best for real-time AI applications and semantic caching.

11. Elasticsearch: Combines BM25 and vector search in a single query engine. Best if you're already running Elastic for search and analytics.

12. LanceDB: Runs in-process, zero-copy disk reads, no server to manage. Native multimodal support. Best for local-first AI, edge deployments, and data lake workloads.

The right choice depends on your use case and in production, it's common to use more than one. Which of these tools are you using in production?

Pagination Strategies for Large Systems

Pagination looks simple when your table has a few thousand rows. It becomes a real problem once your data grows.

Here is how the main approaches compare when the dataset is large.

Most pagination implementations start with offset and page numbers. You ask for “?page=3&size=10” and the database skips 20 rows, then returns 10. Simple to build and easy to explain. The issue is that the database still reads every row it skips. Page 5 is fast. Page 5000 is slow.

Keyset pagination handles deep pages much better. Instead of counting rows, you tell the server to return items after ID 101. The query goes straight to that point using the index, so page 1 and page 10,000 take the same time.

Continuation tokens are common in large APIs like S3 and YouTube. The server returns a token, you send it back on the next request, and it resumes from where the last response ended. It hides the cursor logic from the client. The token is tied to the filters and sort order from the original request, so the client must keep using the same parameters until the pagination ends.

Time-based pagination works well for feeds and logs. You ask for items before a timestamp and read backwards. Add a secondary field, like an ID, for rows that share the same timestamp. Without it, you can lose records.

Over to you: Which one have you used?

How LLMs Use AI Agents with Deep Research

When you ask an LLM such as Claude, ChatGPT, or Gemini to do deep research on a complex topic, it’s not just one model doing all the work. It’s a coordinated system of specialized AI agents.

Here’s how it works:

Step 1: Understanding The Question and Making a Plan

It all starts with the query, something like “Analyze the competitive landscape of AI agents in 2026. The system doesn’t just dive in blindly. First, it may ask clarifying questions to understand exactly what is needed. Then, it generates a plan and breaks the big question down into smaller and manageable tasks.

Step 2: Sub-Agents Get to Work

Each small task gets assigned to a sub-agent, which is basically a mini AI worker with a specific job. For example, one sub-agent might be tasked with finding the latest Nvidia earnings. It figures out which tools to use, such as searching the web, browsing a specific page, or even run code to analyze data. All of this happens through a secure layer of APIs and services that connect the AI to the outside world.

Step 3: Putting it All Together

Once all the sub-agents finish their tasks, a Synthesizer Agent takes over. It aggregates everything, identifies key themes, plans an outline, and removes any redundant or duplicate information. At the same time, a Citation Agent makes sure every claim is linked back to its source and properly formatted. The end result is a polished, well-cited final output ready for use.

Over to you: Have you tried deep research in any LLM?

Check it out here

Here’s what makes this cohort special:

• Learn by doing: Build real world AI applications, not just by watching videos.

• Structured, systematic learning path: Follow a carefully designed curriculum that takes you step by step, from fundamentals to advanced topics.

• Live feedback and mentorship: Get direct feedback from instructors and peers.

• Community driven: Learning alone is hard. Learning with a community is easy!

We are focused on skill building, not just theory or passive learning. Our goal is for every participant to walk away with a strong foundation for building AI systems.

If you want to start learning AI from scratch, this is the perfect platform for you to begin.

Check it out here

A program adding up a day’s sales needs to know whether all of today’s sales have actually arrived. For data stored in a file, the answer is trivial because the file has an end. However, for data that arrives continuously and never stops, there is no clean answer, and how a system resolves that gap is the difference between batch processing and streaming.

Batch processing waits for completeness. It collects data up to a natural boundary, a closing time, or a finished file, and then computes over the whole set at once. Streaming prioritizes completeness for speed. It produces answers continuously from data that is still arriving, which means it has to estimate when enough data has come in and handle the cases where that estimate is wrong. This trade-off between completeness and latency is the key consideration when dealing with streaming and batch.

In this article, we will cover the strategies on each side and what each one costs.

• On the batch side, that means full and incremental loads and large-window aggregation, with micro-batch sitting in between.

• On the streaming side, the territory runs through tumbling, sliding, and session windows, watermarks and late data, the lambda and kappa architectures, and the often-misunderstood meaning of exactly-once processing.

Boundaries

Read more

PR-AF is an open-source code review agent that ranks #2 of 42 on Martian's Code-Review-Bench, ahead of CodeRabbit, Copilot, and Devin, running a single open model. The trick is the harness: it plans a review strategy per PR, spawns parallel reviewer agents, verifies every finding against your source, and drops anything it cannot prove. That makes it about 10x cheaper per review than closed-source tools. One API call, and a drop-in GitHub Action. If open models writing frontier-level reviews sounds useful, star the repo.

Fork and deploy!

A chatbot answers a question, and an agent completes a task. This seems like a huge difference. However, the gap between the two is narrower than it appears.

An agent is an LLM placed inside a loop where the model itself decides when the loop should stop. Everything interesting about agents follows from this shift. The autonomy that makes them useful, the cost that makes them expensive, and the design challenges that come with building one all trace back to it.

It helps to see where that shift sits in the broader picture.

Software built around language models has moved through a recognizable progression. It started with a single LLM call that took an input and produced an output. Then, the function calling arrived, which let the model reach out to a tool when it needed information or wanted to take an action. After that, developers began chaining calls together in code, with each step’s output feeding into the next, to handle problems too large for a single call. The most recent of these is the agent, where the developer hands control of the loop itself to the model and lets it iterate until it decides the work is done.

In this article, we will walk through that progression. We will also look at how an agent is structured, what choices the model makes on every turn, what scaffolding holds it together, and when an agent is actually the right pattern to reach for.

Disclaimer: This post is based on publicly shared details from various sources. Please comment if you notice any inaccuracies.

Foundations

Before the loop can be interesting, the unit that sits inside it has to be clear.

A bare large language model call is a stateless function. Text goes in, text comes out, and each call typically stands alone in the model’s view. If we want it to fetch today’s weather or update a record in a database, the bare model lacks the means to do either. It can describe weather in general terms and talk about how a database record would change, but reaching the actual forecast or touching the actual database requires something more.

What makes the model useful in real systems is augmentation.

In other words, we give it the ability to call functions we have defined, often called tool use or function calling. We give it access to a retrieval layer that can pull in relevant documents at runtime. We also give it a way to write down information that it should carry between calls, which acts as memory.

Anthropic calls the combination of these capabilities the augmented LLM, and treats it as the foundational unit of every agentic system. The augmented LLM is the building block from which everything else gets composed.

This is the unit most developers have already worked with, even if they have called it something else. Examples include a chat assistant that runs Python code, a function-calling API wired to your service, and a RAG application that pulls from your documents. These systems are useful, and they are also still one call. The model produces an output, the system returns it, and the interaction ends until the next request arrives.

Workflows

What happens when a problem reaches beyond what a single call can handle?

The natural response is to string calls together in a sequence we design, a pattern called prompt chaining. Each step in the chain is its own LLM call, and the output of one step becomes the input to the next. We might use one call to draft an outline, a second call to expand the outline into paragraphs, and a third call to translate the paragraphs into another language. The chain is fixed in advance, and the developer writes which steps run, in what order, and what each step’s prompt looks like.

This is the family of patterns Anthropic groups together as workflows.

A workflow is a system where the LLM and its tools are orchestrated through a code path that the developer designed. Beyond prompt chaining, there are other workflow patterns:

• Routing classifies the input and sends it to a specialized handler.

• Parallelization runs subtasks at the same time.

• Orchestrator-worker has a manager LLM that delegates pieces to specialist LLMs.

• Evaluator-optimizer has one call to generate while another critiques, with the cycle iterating until quality is good enough.

The details differ, but every workflow shares the same property. The number of steps and the path through them are decided by the developer at design time, before the model sees the input. Most production systems built on LLMs today are workflows. They are predictable, debuggable, and usually cheaper than full-blown agents.

The Loop

An agent is what we get when we wrap an LLM in a loop and let the model decide when the loop should exit. The loop itself is plain code. The runtime calls the LLM, reads the output, dispatches whatever action that output specified, feeds the result back into the model’s view, and calls the model again. This continues until the model produces a response that signals a final answer.

There are four steps inside each iteration, and we can call them perceive, reason, act, and observe:

• Perceive is the moment the loop hands the model the current state, which includes the original task, the history of what has happened so far, and any new input.

• Reason is the model’s turn, where the model produces a response that says what to do next, whether that means asking a question, calling a tool, or wrapping up with a final answer.

• Act is when the runtime carries out whatever the model asked for.

• Observe is when the result of that action gets captured and folded back into the state, so the model sees it on the next perceive step.

The most important detail is who decides when the loop should stop. In a workflow, the developer decides at design time how many steps run. In an agent, the model decides at runtime. The model exits the loop by producing an output that the runtime interprets as a final answer. The developer typically sets a hard ceiling on the number of iterations, often called a max-turns parameter, but that ceiling is mainly a safety net. The primary stop signal comes from the model.

Observation matters as a first-class step for the same reason.

The model needs to see the result of its actions before deciding the next move. Removing observation would collapse the loop into a chain, with the model running on prior expectations rather than fresh results from the world. The closed loop, where every action is followed by an observation, is what lets the model adjust as it goes.

With the loop established, the next question is what actually happens on each turn through it.

Decisions

On every turn through the loop, the model’s output picks one of four branches, and the runtime acts on that pick. This branching makes the loop feel intelligent because the LLM is choosing what kind of move to make. Here are the four branches:

• Final answer: The model produces what amounts to a complete reply to the original task. The runtime interprets this as the loop’s exit signal, returns the result, and stops.

• Tool call: The model produces an instruction asking the runtime to invoke a specific function with specific arguments. The runtime executes the tool, captures whatever it returned, appends that result to the conversation state, and sends control back to the model so the loop continues.

• Handoff: The model decides that the current task belongs in the hands of a different agent, often a specialist with its own prompt and tool set. The runtime swaps which agent is running and feeds the same state into the new agent, with the loop continuing under the new identity.

• Continued thought: The model produces a reasoning turn that consists of thought alone. The runtime captures the thought, feeds it back into the state, and runs the model again. This branch shows up most often in ReAct-style implementation, which we are about to see.

OpenAI’s Agents SDK documents the first three branches as first-class behaviors of its loop. The fourth is more a property of how a particular prompt is written than a separate code path

ReAct

ReAct is the prompting pattern that stands for Reasoning plus Acting. The pattern asks the model to interleave reasoning steps with action steps inside the same response, so the model can think about what to do, do it, see what happened, and adjust.

A ReAct trace reads like a structured journal. The model produces a thought, then an action, then receives an observation from the runtime, then produces another thought, and so on, until it reaches a final answer.

Imagine an agent handling customer support:

• A user asks for the status of their most recent order.

• The model produces a thought first, reasoning that it needs to find that order and that the order service is the right place to look.

• It then takes an action by calling the get_recent_order function with the user’s ID.

• The runtime executes the call and feeds back an observation telling the model that the order is number 9152, placed on May 14.

• The model produces another thought, working out that it now needs the shipping status for that specific order.

• It calls the get_shipping_status function with the order ID. The observation comes back, saying the order is in transit and expected to arrive on May 29.

• The model has what it needs at this point and produces a final answer summarizing the status for the user.

See the diagram below:

Two things to consider from this flow are as follows:

• First, every action is grounded in an observation, which is the closed-loop point from the previous section made concrete.

• Second, the reasoning steps are doing real work, since they are how the model decides which action makes sense next, given what it has learned so far.

ReAct is one way to fill the loop, and it is by far the most common pattern in the agent frameworks we work with today.

Guardrails

Guardrails belong at the points where the loop crosses into the outside world. They are part of the architecture itself, designed in alongside the loop.

For reference, OpenAI’s Agents SDK documents three families of guardrails, all defined by where in the loop they run.

• Input guardrails run on the first turn, before the agent’s main model sees anything. They are the place to catch things like prompt injection attempts, requests that violate policy, or inputs that fall outside the agent’s scope. A common pattern is to use a small, fast model as the guardrail, so the more expensive main model only runs when the input passes.

• Tool guardrails wrap every function-tool invocation. A tool input guardrail runs before the tool executes and can block the call or replace it with a message back to the model. A tool output guardrail runs after the tool executes and can rewrite or block the result before it goes back into the conversation state. Tool guardrails matter because tools are how the loop touches systems we care about, and that touch needs supervision.

• Output guardrails run on the final response, after the loop has decided to terminate, but before the user sees the result. They are the last layer of policy enforcement and the place to catch things like leaked sensitive data or claims the agent should avoid making on the company’s behalf.

The structural point is that guardrails sit at every interface where the loop meets the world.

Tradeoffs

Handing control of the loop to a model is powerful, and it comes with three costs that every developer should understand before using the pattern:

• Compounding error: Per-step reliability does poorly when steps are chained together, and the math is not favorable. If the model succeeds on each step of a loop 95 percent of the time, the joint probability of every step going right across ten steps is roughly 60 percent. Stretch that to twenty steps, and the joint success rate falls to about 36 percent. The autonomous nature of agents brings higher costs and potential for compounding errors. The reason coding agents work better than open-ended task agents is that test feedback raises per-step reliability, which shortens the effective length of the chain that has to succeed.

• Scaffolding around the loop: The harness around the loop matters as much as the model inside it. For example, Anthropic shared an internal experiment where even a frontier model running in the Claude Agent SDK fell short on building a production-quality web app from a high-level prompt. The fix was scaffolding rather than a better model. They built an initializer agent that lays out a feature list before any feature work starts, a coding agent that picks one feature per session, a progress file that travels between sessions, and a git history that the agents can use to recover.

• The wrong tool for the job: An agent is often the wrong choice. It is important to find the simplest solution and only add complexity when needed, which sometimes means avoiding agentic systems entirely. Workflows offer predictability and consistency. Agents offer flexibility, and they pay for that flexibility in latency, money, and a more unpredictable failure surface. Many problems are served better by a chain.

Conclusion

The agent loop sits at the end of a recognizable progression.

We started with a single LLM call that takes text in and returns text. We added tools, retrieval, and memory to get the augmented LLM. We strung calls together into workflows when one call was outgrown. The agent is the next rung, the point where the developer hands the control flow itself to the model.

Inside the loop, four steps repeat. The model perceives the current state, reasons about it, takes an action, and observes the result. On every turn, the model’s output picks one of four branches. These are a final answer, a tool call, a handoff, and a continued thought. ReAct is the most common prompting pattern for filling the loop, with reasoning and action interleaved. Guardrails live at every place where the loop crosses into the outside world.

The design carries three real costs. These are compounding reliability across steps, the harness scaffolding that production loops require, and the question of whether a workflow would solve the problem more cheaply.

References:

• Building effective agents — Anthropic Engineering

• Effective harnesses for long-running agents — Anthropic Engineering

• Running agents — OpenAI Agents SDK documentation

• Guardrails — OpenAI Agents SDK documentation

• ReAct: Synergizing Reasoning and Acting in Language Models

Learn practical techniques to make your software faster at every layer of the stack

Working on latency-sensitive applications? This book will help you spot, understand, and fix latency in your applications and infrastructure. It shares low-latency techniques that have been predominantly “tribal knowledge” until now. You’ll learn a full-stack approach, with techniques that intersect many areas of software engineering, distributed systems, databases, and operating systems.

• Understanding and negotiating caching tradeoffs (latency vs. complexity)

• Exploring the two main concurrency models (threading and event-driven architecture)

• Identifying and applying parallelization opportunities to improve throughput and reduce latency

• Applying asynchronous process strategies that reduce perceived latency and improve overall system responsiveness.

Get this 3-chapter excerpt and start reading the book that’s become essential for anyone working with strict performance SLAs.

Download for Free

If we send the same prompt to ChatGPT, Gemini, and Claude, the answers come back meaningfully different. Claude might push back more often. Gemini ingests a two-hour video file as easily as a paragraph of text. ChatGPT routes some prompts to a slower reasoning mode and answers others instantly, without telling the user which path it took.

Why does each model behave this way, and why so consistently?

Each pattern traces back to an architectural decision point where the developers of these frontier models made distinct design choices, and the user-visible behavior follows directly from those choices.

To understand these differences clearly, we need to examine architecture and design, since architectural decisions remain consistent across releases and provide a reliable framework for understanding model behavior over time.

In this article, we will look at the various architectural forks the teams building these models encountered and the decisions they took.

Disclaimer: This post is based on publicly shared details from various sources. Please comment if you notice any inaccuracies.

Foundations

All three models share the same fundamental architecture, which is a transformer-based generative neural network.

The transformer was introduced in the 2017 paper “Attention Is All You Need,” and at its core is a mechanism called self-attention that allows each token in a sequence (a token being a small piece of text, roughly three-quarters of a word) to weigh its relationship to every other token. This setup is shared across the three models, while the components built around it, along with the training process that shapes it, vary substantially.

Training itself involves two phases:

• Pre-training is where the model learns from massive amounts of text and other data by predicting the next token, repeated billions of times over.

• Post-training is where the base model gets shaped into a helpful assistant through methods like Reinforcement Learning from Human Feedback (RLHF) and Constitutional AI. We will get to them in more detail in a later section.

Each architectural fork follows a consistent pattern, where different companies faced the same design question, each made a particular architectural choice, and each choice produced user-visible behavior we can observe today.

The first such question concerns how to scale the model’s total capacity without proportionally increasing the cost of processing every query.

Density

Every parameter added to a model carries computational cost. In a standard dense neural network, every parameter activates for every token processed, meaning that doubling the parameter count roughly doubles the compute cost per query.

Frontier developers needed a way to escape this constraint, and the architectural question became how to scale total capacity without paying full price on every single token.

Google adopted a technique called Mixture of Experts, or MoE. The Gemini 1.5 technical report identifies the model as “a sparse mixture-of-expert (MoE) Transformer-based model” that “builds on a much longer history of MoE research at Google,” with the Gemini 3 Pro model extending the same approach.

In an MoE layer, instead of one massive computational block where every parameter activates for each token, the network contains many smaller experts, and a small router determines which two or three experts each token should be sent to. The total parameter count can be enormous while only a fraction is activated per token, though achieving consistent results requires careful attention to load balancing and expert specialization during training.

OpenAI has not explicitly confirmed whether GPT-4 uses MoE, and the GPT-4 technical report deliberately omits architecture details. The GPT-5 system details describe a different efficiency approach that we will examine in the Reasoning section, where a router selects between distinct sub-models at runtime.

Nevertheless, these architectural choices produce meaningful consequences in user experience.

MoE models can pack more knowledge per dollar of compute, which contributes to Gemini’s capable handling of a wide breadth of domains. The tradeoff is increased variance, since different prompts route to different experts, and imperfect load balancing during training can leave some experts underused and weaken the model’s capacity on those topics. Dense models tend to deliver more predictable per-token behavior at the cost of being harder to scale to extreme parameter counts.

See the illustration below:

Turn cloud logs into real security signals (Sponsored)

This guide from Datadog provides best practices on how to use Cloud SIEM to detect threats, investigate incidents, and reduce blind spots across cloud and Kubernetes environments.

You’ll learn how to:

• Analyze CloudTrail, GCP audit, and Azure logs for suspicious activity

• Detect authentication anomalies and common attack patterns

• Monitor Kubernetes audit logs for lateral movement and misuse

• Correlate signals across services to accelerate investigations

Get the guide

Multimodality

The term multimodal carries different meanings depending on the architectural approach, and a precise distinction helps before we take a look at the design choices.

Two fundamentally different approaches exist for handling non-text inputs like images, audio, and video:

• Sequential approach: Train a strong language model first, then add a separate encoder later that translates non-text inputs into embeddings the language model can attend to.

• Native approach: Train a single network from inception on all modalities simultaneously, so that text tokens, image patches, and audio frames coexist in the same sequence and get processed by the same transformer layers.

See the diagram below that shows the difference between the two:

These two approaches produce different capabilities.

Google adopted the native approach from the start. The Gemini 1.0 release was designed as natively multimodal, and the 1.5 technical report demonstrates the model handling, in a single input, “10.5 hours of video at 1 frame-per-second”, and the current Gemini 3 Pro model continues this design, where the model offers “native multimodal support for text, vision, and audio inputs.”

OpenAI took a different path initially, with the original GPT-4 supporting vision through a separate pipeline. GPT-4o, released in May 2024, transitioned to a unified architecture, and OpenAI’s own announcement for GPT-5.5 acknowledged that “previous multimodal models from OpenAI were essentially separate models stitched together.”

Lastly, Anthropic has maintained a text-first approach with strong vision capabilities, and Claude Opus 4.8 continues to support high-resolution image input introduced in Opus 4.7 at 3.75 megapixels, particularly useful for documents, screenshots, and dense charts. Anthropic has chosen to focus on text and vision as primary input modalities while setting native audio and video aside for now.

For users, this architectural difference appears most clearly when working with video. Gemini handles long video files directly, and ChatGPT does the same on its more recent models. Claude performs best with documents, screenshots, and high-fidelity images.

Context

A context window represents the amount of input a model can attend to in a single pass, measured in tokens.

Larger windows allow the model to reason over more raw material at once, though long context introduces two important challenges:

• Computational cost: Attention scales roughly quadratically with sequence length, so longer windows consume significantly more compute per query.

• Quality degradation: Models often lose track of information mentioned earlier as the window fills, with the relevance of early context dropping over the course of a long conversation.

Google pushed hardest on raw window size. The Gemini 1.5 technical report demonstrated context up to 10 million tokens, and Gemini 3 Pro and 3.1 Pro continue to offer 1 million token context in production.

Anthropic offers 1 million tokens on Claude Opus 4.8 and Sonnet 5, and the company’s own documentation uses the term “context rot” to describe quality degradation in long sessions, addressed through automatic compaction that summarizes earlier portions of the conversation.

OpenAI has taken a more conservative position on window size, with GPT-4 Turbo at 128K, focusing efficiency gains on the routing architecture covered in the Reasoning section.

The practical experience differs across the three models. Loading an entire codebase or a multi-hour transcript into a single session works well with Gemini and Claude, given the larger windows available. ChatGPT is structured more around efficient routing between sub-models than around accepting massive single-prompt inputs.

Long context expands the range of questions you can ask, while retrieval-augmented generation, which fetches relevant external data at query time, remains relevant when freshness and cost matter.

Alignment

Pre-training produces a base model that has processed enormous amounts of text and learned to predict the next token. At this stage, the base model behaves as a sophisticated pattern-completer with limited regard for honesty, safety, or helpfulness.

Post-training is the phase where this base model is shaped into the assistant that users interact with, and the three companies have taken substantially different approaches to this work.

OpenAI built on Reinforcement Learning from Human Feedback (RLHF) throughout the development of GPT-3 and GPT-4. The approach involves three main steps:

• Human raters compare pairs of model outputs and select the better one.

• Those comparisons train a reward model that captures human preferences.

• The reward model is then used to fine-tune the language model toward outputs that humans prefer.

The GPT-4 product page describes an additional safety reward signal incorporated into RLHF to reduce harmful outputs. For the reasoning models, OpenAI added an approach called deliberative alignment, where the model reasons about safety policies at inference time. The publicly available Model Spec describes intended behavior in detail.

Anthropic developed an alternative method called Constitutional AI, or CAI. Instead of relying on human raters at every step, CAI uses an explicit written constitution and has the model critique and revise its own outputs against those principles, then applies AI-generated feedback to further refine the model. In January 2026, Anthropic published a 23,000-word constitution that shapes Claude’s training across synthetic data generation, response ranking, and behavioral refinement.

Google uses RLHF with learned reward models in its post-training pipeline for Gemini, with somewhat less explicit public framing than the other two companies. The Gemini technical reports describe a multi-phase training process moving from unimodal pre-training through multimodal alignment.

This architectural area also reflects a broader pattern of differing transparency choices across the three companies. The 2023 GPT-4 technical report deliberately omitted architecture details, while the 2025 GPT-5 system card describes a router, named sub-models, and safety mitigations in remarkable detail. Anthropic publishes its constitution publicly, providing a detailed view into the principles that shape Claude’s behavior. The three companies have made distinct choices about transparency, and those choices have evolved over time.

For users, these training differences appear as personality. Claude tends to push back more often on edge cases because the constitution explicitly trains for that behavior. ChatGPT is often quicker to attempt a task without questioning. Gemini’s responses vary more by domain.

See the diagram below:

With values addressed, the most recent architectural dimension concerns how the model thinks, an area where the three companies have converged from different directions.

Reasoning

The most recent and most consequential architectural development for the current generation of models concerns whether the model produces an answer directly or reasons explicitly through tokens before answering.

A standard chat model takes a prompt and immediately begins generating the response. A reasoning model first generates internal thinking tokens, working through the problem, and only afterward produces the final answer. The same prompt, the same underlying weights, and the same context window can produce very different inference-time compute profiles depending on which approach is used.

OpenAI built dedicated reasoning models, beginning with o1 in late 2024, followed by o3 in August 2025, and continued in GPT-5.5, which was released on April 23, 2026. The architecture includes three main components.

• GPT-5-Main, a fast, high-throughput model and the successor to GPT-4o.

• GPT-5-Thinking, a deeper reasoning model and the successor to o3.

• A Real-Time Router that selects between them based on conversation type, complexity, tool needs, and explicit intent in the prompt.

OpenAI plans to integrate these capabilities into a single model in the future, though the current architecture functions as a system of models rather than as a unified single model.

Anthropic pursued a different approach. Claude Opus 4.8 and Sonnet 5 use adaptive thinking inside a single model, where the same Claude determines for itself how long to think on each request, with summarized reasoning surfaced to developers for inspection.

Google integrated thinking into Gemini 3 through Deep Think mode and the three-level thinking system in Gemini 3.1 Pro, enabling the model to reason over text, images, audio, and video in a single thinking pass.

For users, this architectural difference shows up more visibly than might be expected. ChatGPT can feel inconsistent across sessions because the router may have directed two similar prompts to two different sub-models. Claude tends to feel consistent because the same model varies its reasoning depth based on the request. Gemini’s behavior shifts more with domain and modality.

A convergence happens at this architectural fork. Three companies that began with very different choices at every prior dimension, including MoE versus dense architecture, native versus bolted-on multimodality, and RLHF versus Constitutional AI, have all arrived at the same approximate conclusion. Explicit reasoning tokens at inference time materially improve model performance on hard problems.

Conclusion

In this article, we looked at five architectural dimensions, each representing a question all three frontier developers had to answer when building their models.

• Density: Google adopted Mixture of Experts openly, OpenAI built routed efficiency through a different mechanism, and Anthropic kept its architectural choice private.

• Multimodality: Google adopted native fusion from inception, OpenAI evolved from bolted-on vision toward a unified architecture, and Anthropic maintained a text-first approach with strong vision capabilities.

• Context: Google pushed the largest windows, Anthropic invested in long sessions with explicit handling of degradation, and OpenAI emphasized routing efficiency over raw window size.

• Alignment: OpenAI’s RLHF combined with the Model Spec, Anthropic’s Constitutional AI combined with its published constitution, and Google’s RLHF approach represent three distinct shaping pipelines.

• Reasoning: All three companies converged on explicit reasoning tokens, despite starting from different architectural choices at every prior dimension.

When we observe one of these models behaving in a particular way, we now have a framework for asking which architectural dimension produced that behavior.

References:

• GPT-4 Technical Report

• GPT-4 Research Page

• GPT-5 System Card

• Introducing GPT-5.5

• Learning to Reason with LLMs

• Introducing OpenAI o3 and o4-mini

• OpenAI Model Spec

• Reasoning Models Documentation

• Claude’s Constitution

• Claude’s New Constitution

• Constitutional AI: Harmlessness from AI Feedback

• Claude Opus 4.7

• What’s New in Claude Opus 4.7

• Context Windows

• Adaptive Thinking

• Building with Extended Thinking

• Gemini 1.5 Technical Report

• Gemini 2.5 Technical Report

• Introducing Gemini 1.5

• Attention Is All You Need

• Gemini 3 Technical Report

• Introducing Claude Opus 4.8

Check it out Here

Here’s what makes this cohort special:

• Learn by doing: Build real world AI applications, not just by watching videos.

• Structured, systematic learning path: Follow a carefully designed curriculum that takes you step by step, from fundamentals to advanced topics.

• Live feedback and mentorship: Get direct feedback from instructors and peers.

• Community driven: Learning alone is hard. Learning with a community is easy!

We are focused on skill building, not just theory or passive learning. Our goal is for every participant to walk away with a strong foundation for building AI systems.

If you want to start learning AI from scratch, this is the perfect platform for you to begin.

Check it out Here

The defenses websites have relied on for years are failing in predictable ways, and the failure shows up most visibly in moments like the one below.

A popular retailer drops a thousand pairs of a limited-edition shoe at noon. Within thirty seconds, the inventory is gone. The buyers turn out to be automated agents working for resellers. The real people who wanted the shoes get nothing.

The retailer almost certainly tried the standard defenses: rate limits by IP, CAPTCHA, phone verification, device fingerprinting. Each of them helps for a while and then stops working. The reason they all fail in the same way is worth pausing on.

Every one of those defenses relies on a proxy for what the retailer actually wants to verify. An IP address is a proxy for a different network. A phone number is a proxy for a different person. A device fingerprint is a proxy for a different device. Each proxy fails the moment adversaries learn to acquire many of them cheaply, and adversaries always do. Phone numbers can be bought in bulk. Fingerprints can be randomized. IPs can be rotated through residential proxy networks. None of these strategies binds the verification to a real, unique person.

Authentication systems do not solve this either. Single sign-on, face unlock, passkeys, OAuth tokens, all of them compare an incoming credential against a stored template and return yes or no. None of them answers the question that actually matters here: has this user already been verified somewhere else in the world?

The central question: How do you let a real, unique human be recognized across the internet, without ever knowing who they are?

For this article we spoke with the team behind World, including Tiago Sada and Lily Gordon at Tools for Humanity, on how they try to solve this problem.

Five Pillars of a Proof-of-Human System

A working answer to the central question requires five distinct ingredients. We will look at each one in turn.

Uniqueness. Why is this a different identity problem than the ones we usually solve?

Anonymity. How can a credential be issued without anyone knowing the identity of the user?

Recovery. How does the system survive lost phones and reinstalled apps?

Verification. How does the holder present the credential without revealing more than necessary?

Delegation. What changes when the holder is an AI agent acting on a person’s behalf?

Pillar 1: Uniqueness

To see why uniqueness is structurally different from authentication, start with a system everyone already trusts: Face ID on a phone.

When the user sets up the phone, the camera captures a single facial template and stores it locally. From then on, every unlock attempt produces a fresh capture, which the phone compares against the stored template. The comparison space is exactly one. If the fresh capture matches the template within some tolerance, the phone unlocks. If not, the phone refuses. The system handles other faces by failing the match.

This is a one-to-one matching problem. The math is simple because the comparison is small. Even with a per-comparison error rate of one in a million, the phone unlocks reliably for its owner and refuses essentially every other face.

However, the example with the retailer wants a different guarantee. They want to verify, at checkout, that the buyer is different from every other person who has already bought those special edition shoes. The comparison is no longer against one template, but against the entire population of past buyers. If the system is meant to work at internet scale, the population is potentially every person on the planet.

This is a one-to-many matching problem, and the diagram below shows this comparison in the context of World ID. As you can notice, the math can get much more complex when the comparison space grows.

The probability of a false match scales roughly with the size of the comparison space. A per-comparison error rate of one in a million sounds excellent, but checked against a billion candidates, it produces roughly a thousand false matches per query. To make one-to-many uniqueness viable at a billion-person scale, the per-comparison error rate has to be on the order of one in a hundred billion or better. That requirement rules out most consumer-grade biometric methods straight away.

This type of system that answers the one-to-many question is what the World ID calls the proof of human. At present, there is no widely deployed equivalent of this category. We have a great deal of authentication infrastructure and very little uniqueness infrastructure.

Pillar 2: Anonymity

Once the goal is uniqueness at an internet scale, an apparent paradox emerges.

Checking whether someone is the same as anyone else usually requires recognizing them. If a system cannot identify a person, how can it tell whether it has seen that person before?

The answer takes multiple steps.

Step 1: Find a biometric signal that scales

Most consumer biometrics fall short. For example, fingerprints have decent entropy but can be captured from surfaces. Face geometry varies less than people assume. The iris pattern of the human eye, by contrast, turns out to be one of the few biometric features with the entropy needed for billion-scale comparisons. Two unrelated humans have essentially no chance of producing matching iris patterns, even after accounting for camera noise and aging.

However, reading the iris reliably is the next challenge, and this is where hardware matters.

A standard phone camera can be replaced with a device that injects images directly into the camera pipeline, defeating any face check done in software. A printed iris image can fool an infrared camera that lacks depth detection. To get a reading that resists these attacks, the capture has to happen on hardware that controls the entire signal path from sensor to processing.

Tools for Humanity handles this using a purpose-built device called the Orb. The Orb uses multispectral imaging across infrared and visible wavelengths, runs several neural networks locally to verify liveness and detect masks, and deletes the original images before anything leaves the device. What gets transmitted is signed, encrypted material derived from the iris, not the iris itself.

Step 2: Check for duplicates without anyone seeing the data

That solves the first half of the problem. The second half is harder. The system needs to check whether anyone else in the world has the same reading, while no party (including the system operator) ever sees the reading itself.

The trick works like this:

• Take the iris reading and split it into three pieces of statistically random noise.

• Hand each piece to a different organization operating in a different legal jurisdiction.

• Each party, looking only at its piece, can tell nothing about the underlying reading.

• The parties can jointly compute answers like “does this reading match any reading we have already enrolled?” without any single party ever holding the whole reading.

• The pieces are combined only in the abstract space of the computation, never reconstructed on any actual machine.

This is what cryptographers call secure multi-party computation, and the specific variant World uses is called anonymized multi-party computation (AMPC). The parties (referred to as SMPC nodes) are operated by independent organizations in different legal jurisdictions.

See the diagram below that tries to conceptualize what each SMPC party actually sees:

What emerges from this setup is the property that makes proof of human possible. The system can determine whether a given human has enrolled before, and no single party (including the main organization) ever sees the underlying biometric in usable form. The user is verified as unique without being identified.

Pillar 3: Recovery

A verified credential is only useful if the human carrying it can keep using it across years and across devices. In a normal life, phones get lost, and software gets reinstalled. Any system meant to anchor a lifelong property like “I am a verified unique human” has to handle the case where the holder loses access to everything they were using to prove it.

If the credential is a private key on a phone, losing the phone means losing the credential. The user would have to re-enroll. If the system is anchored in physical hardware verification, it means traveling back to a central location (in the case of World ID, that means the Orb). It would be a rather poor user experience for everyday loss, and impossible for some users in some regions.

The fix is to stop treating the credential as a single secret. Instead, treat the verified human as an abstract account in a public registry. The account does not hold the user’s biometric or any user secret directly. It holds a list of public keys (called Authenticators) that the user has authorized to act on the account’s behalf.

Think of an Authenticator as any piece of software or hardware that holds a private key on the user’s behalf. Here are some examples:

• A common Authenticator could be the user’s phone with the wallet app installed.

• A browser extension that holds a separate key could be another.

• A hardware security token plugged in over USB is a third option.

When the user wants to verify on a service or app, the Authenticator is what actually produces the cryptographic proof and signs the verification request with its private key. The account in the public registry holds the matching public key, so anyone can confirm the signature without ever knowing the secret. Keys can be added, revoked, and rotated. The account itself persists.

For more context, the public registry stores one entry per verified human. Each entry contains the public keys of all currently authorized Authenticators, the designated Recovery agents for that account, and the rules under which those agents can act. The biometric reading itself is never stored in the registry. This ensures that reading the registry cannot reveal personal details about the verified humans.

See the diagram below that shows this approach in the context of World ID’s solution:

If the user loses all of their Authenticators, the account also designates one or more Recovery Agents, which are parties trusted to re-authenticate the human (typically by running them through a fresh biometric check) and let them register new keys.

In the current design of World ID, the Orb network serves as the default Recovery Agent. The on-chain registry that holds all of this is called the WorldIDRegistry.

What this might look like in practice is as follows: imagine the user’s phone is stolen on a trip. They get a new phone and install the wallet app fresh, but the app has no private keys yet. Rather than travel back to an Orb to start over, the user can initiate a recovery through one of the Recovery Agents they previously designated. The agent runs them through a fresh biometric check, typically through the Orb’s mobile face authentication. If the check passes, the agent submits a transaction to the registry that revokes the stolen phone’s key and registers the new phone’s key.

However, the open question here is governance. The Recovery Agents can themselves become a trust point, which would require some form of oversight.

Pillar 4: Verification

So far, we have a way to verify a human as unique, and a way for them to hold their credentials durably.

What we need next is a way to present that credential to a relying party (for example, a dating app, a shoe retailer, or any service that wants to enforce one-per-human) so the service can do its job while learning as little as possible about the user.

A relying party in this scenario needs three things at once:

• Assurance that the request is backed by a real and verified unique human.

• The ability to enforce its own one-per-human rule, which means recognizing repeat visits by the same human within its own context.

• Confidence that what the user reveals is limited to the fact of being a verified, unique human, with identity, other activity, and behavior on other platforms staying private.

Satisfying all three at once is harder than it sounds. Standard identity tokens (an OAuth token, a JWT, a session cookie) tie recognition to identity. The relying party gets the ability to enforce policies, but pays for it with the user’s privacy.

The World ID handled this problem using a primitive called a nullifier. A nullifier is a number derived from three inputs combined together. The inputs are the user’s verified credentials, the relying party’s identifier, and a particular action. The same combination always produces the same number. Different combinations produce uncorrelated numbers. If the relying party sees the same nullifier again for the same action, they know it is the same human attempting the same thing twice. Nullifiers from the same user on a different action, or in a different service, are uncorrelated and cannot be linked back.

See the diagram below that shows what the relying party actually sees:

Generating a nullifier like this requires careful design. A user cannot generate it alone (they cannot be trusted to use the right inputs), and a single server cannot generate it either (the server would learn who the user is). It needs a distributed network of nodes, and here’s how they work:

• The user sends a blinded version of their query

• Each node computes on the blinded value while staying unaware of what the value represents

• The user unblinds the result.

• A threshold of nodes must cooperate. These nodes are called OPRF nodes (oblivious pseudorandom function).

One more piece is needed to complete the puzzle. If a user could reuse the same nullifier multiple times within one relying party context, an enterprising service could build a pseudonymous profile of the user’s repeated interactions. To prevent this, the design maintains an on-chain list of every nullifier that has been used, called the Oblivious Nullifier Pool. The user’s own software refuses to generate a proof for a nullifier already in the pool.

So, how is all this complexity handled from an application’s perspective?

In the context of World ID, all of this gets bundled into two zero-knowledge proofs that the user presents to the relying party. An SDK named IDKIT wraps the flow for integrators.

See the diagram below that shows a simple IDKit Proof Flow:

Pillar 5: Delegation

Up to this point, we have assumed the human is the one acting. The user might open a dating app, buy sneakers from an online retailer, or sign a document. The design handles each case by binding one verifiable human to one proof.

The interesting question is what happens when the human is not the one acting. This is because people are now increasingly using AI agents to take actions on their behalf. For example:

• An assistant books a flight.

• An agent monitors product drops and buys when something matches.

• A service fills out and submits forms for a user who would rather not.

The standard defense against agents is to block them as bots, but here the human wants the agent to act for them. Treating the agent as adversarial defeats the very automation the user is paying for.

The extension to the design for an ideal proof of human solution is to let the agent register against a real human’s credentials before it acts. When the agent later contacts a service, the service can verify that the agent is backed by a verified, unique human, while still learning nothing about which human. The agent’s activity counts against the human’s quota, so a single human cannot turn into many parallel agents to amplify their reach.

See the diagram below that shows how World ID implements this using AgentKit:

In World’s implementation, the agent registers its wallet address in a registry called AgentBook (deployed on the World Chain blockchain), tying the agent to the human’s World ID through a verification flow in the user’s wallet app. A service or application that wants to admit human-backed agents installs the SDK called AgentKit, which by default allows three uses per human per service before normal payment takes over.

A few new use cases become possible:

• A startup can offer a free trial that bots cannot farm at scale, because each trial use is bound to a verified human.

• A retailer can enforce one pair per human on a limited drop, even when agents do the buying.

• A merchant can require human backing for transactions where chargeback fraud is a concern.

What remains open is how this scales. At present, the volume of agent traffic is quite small compared with what AI agents will likely produce in the coming years. Whether the per-human cap stays meaningful when one human represents many concurrent agents is a question worth looking into.

Conclusion

Taken together, the five pillars (uniqueness, anonymity, recovery, verification, and delegation) form a working blueprint for proof of human in principle. Whether any particular implementation, World ID included, becomes the way the larger question gets answered at internet scale remains to be seen.

A few specific questions are worth tracking as the design meets larger-scale reality:

• Hardware decentralization. The longer-term plan calls for multiple independent manufacturers of devices like the Orb, distributed across jurisdictions and cross-checking each other through security deposits. The architectural properties depend on this transition actually happening.

• Unlinkability under adversarial pressure. The cryptography looks sound on paper. The harder test is what happens when relying parties, governments, or other large actors try to extract more from the system than the design intends. Whether the unlinkability properties hold in practice at scale is something the community will only learn over time.

• Delegation at agent scale. Most agent traffic today still flows through traditional authorization systems. As agents take on more autonomous activity, the per-human cap will be tested in ways the existing deployment has not yet seen. New abuse patterns will almost certainly emerge.

• The bootstrap problem. Proof of human becomes useful at scale, and reaching scale requires apps to require it, which in turn requires users to have it. How any such system navigates that loop, and what happens if a meaningful fraction of the internet starts to require proof of human, is far from clear today.

The point of this article was not to argue for any particular product, but to surface the techniques. Once you can see the shape of the problem, one-to-many matching, secret-shared biometrics, scoped nullifiers, account-style key rotation, delegated quotas, the headlines about “verify you are human” start to look less mysterious. They are attempts to solve a specific class of problems that the rest of the identity stack has so far left alone.

References and Further Reading

• https://world.org/liftoff

• https://docs.world.org/world-id/idkit/integrate

• https://docs.world.org/agents/agent-kit/integrate

• https://world.org/blog/engineering/private-proof-of-human

• https://eprint.iacr.org/2024/705

• https://world.org/blog/engineering/introducing-world-id-4.0

That runs against the instinct, which says more locations should mean faster service and steadier uptime, which is often the case. But once the same data lives in two places at once, a new class of problem appears that can nullify the advantages of multi-region deployments.

Let’s look at a short example. Picture two edits to the same piece of data, made at nearly the same instant, one handled by a server on the US East Coast and one by a server in Frankfurt, right as the network link between those regions drops. Both edits are saved locally. Each region now holds a different version of that data, with no shared record of which one came first. When the link comes back, one of the two versions has to be chosen over the other. The way a system handles that question has a large effect on what a global footprint costs to build and to run.

Going global is better understood as a progression than as a single decision. In this article, we will start with the building blocks, the small set of concepts every regional design rests on, then work through the common setups in order, from a single region with backups up to running every region at once. Each step buys something concrete: lower latency, higher availability, or the ability to keep data inside a country’s borders. Each one also has a price, in money and in the consistency tradeoffs it introduces.

Foundations

Read more

Matic is the first visually intelligent robot vacuum that sees your home like you do, so it can clean how you want it to.
Matic’s hero features:

• Runs entirely on cameras to deftly navigate obstacles

• Recognizes floor types to auto-switch between vacuuming and mopping

• Big wheels and a height-adjustable cleaning head handle thick rugs

• Quieter than conversations at 55 dB

• Handles pet hair without clogging or tangling

• A single bag collects dry and wet waste—diaper-salts gel dirty water, antimicrobial powder prevents mold

• A fresh HEPA filter in every bag for cleaner air, no washing or replacing

Experience autonomous cleaning with Matic with a 180-day money-back guarantee.

Get your Matic

OpenAI runs voice AI for 900 million users a week, and they use WebRTC for it because the alternative would mean reinventing how the internet handles live audio.

The catch is that WebRTC was designed for servers with stable IPs and ports, and Kubernetes treats those addresses as disposable. The conventional answer at this scale is an SFU, which suits multiparty workloads like group video calls, but OpenAI’s traffic is overwhelmingly one user talking to one model.

To deal with this, their architecture splits the stack into two pieces:

• A stateless relay handles protocol-aware packet routing at the geographic edge.

• A stateful transceiver owns all the heavy WebRTC state.

The trick that ties them together is using the ICE ufrag, a field the protocol already exchanges during setup, as a routing key that the relay can read off the first packet of a new session. Everything else, from Global Relay to the userspace Go implementation to the Redis cache and the careful socket-level optimizations, builds on top of that core idea.

In this article, we will look at the entire journey in detail and challenges the OpenAI engineering team faced.

Disclaimer: This post is based on publicly shared details from the Open AI Engineering Team. Please comment if you notice any inaccuracies.

Why Latency Matters For Voice AI

Voice AI either feels like a conversation or it feels like a walkie-talkie. The line between those experiences is measured in milliseconds.

When the network pauses between hearing a user and responding, the illusion breaks. Pauses turn awkward, interruptions get clipped, and users are compelled to cut off the AI mid-sentence, which is also kind of rude. In other words, voice AI only feels natural if the conversation moves at the speed of speech.

The harder constraint underneath is the continuous-stream property. Audio has to arrive at the model as a steady flow, rather than as a single upload after the user finishes talking. That stream is what lets the model start transcribing, reasoning, and calling tools while the user is still speaking. The experience collapses into push-to-talk once it breaks.

For OpenAI specifically, those constraints translate into three concrete requirements:

• The system has to reach 900 million weekly active users wherever they are.

• Connection setup has to be completed quickly enough that users can start speaking as soon as a session begins.

• Round-trip time for audio has to stay low and stable so turn-taking feels crisp.

WebRTC is the protocol the industry built for this kind of work. It is a bundle of smaller protocols (ICE for figuring out how two endpoints reach each other across firewalls, DTLS for encrypting the channel, SRTP for the audio packets, and RTCP for quality feedback). Justin Uberti, one of WebRTC’s original architects, and Sean DuBois, who maintains the Pion library OpenAI builds on, both work at OpenAI today. That kind of protocol depth shows up in the architecture they shipped.

The Original Architecture

The first version of OpenAI’s WebRTC infrastructure was a single Go service built on Pion. It handled both jobs in one place:

• On the signaling side, the service negotiated SDP (the format clients and servers use to describe a session), selected codecs, generated ICE credentials, and set up sessions.

• On the media side, the service terminated WebRTC connections from clients and maintained upstream connections to the backend services that run the AI models, including inference, transcription, speech generation, tool use, and orchestration.

That combined service still powers ChatGPT voice, the Realtime API’s WebRTC endpoint, and several research projects, and it has handled that work well. The question OpenAI ran into was how to deploy it on Kubernetes, the container orchestration system that runs most modern cloud infrastructure.

Kubernetes assumes compute is cheap and movable. Pods come up, get scheduled wherever capacity exists, run for a while, then get rescheduled or replaced. Standard WebRTC deployment patterns assume the opposite. That mismatch shows up in two specific places.

The first is port exhaustion.

The conventional way to deploy WebRTC uses one UDP port per session. At OpenAI’s scale, that means tens of thousands of public UDP ports per service. Cloud load balancers were built for a handful of well-known ports, so each additional range adds operational complexity for load balancer config, health checks, firewall policy, and rollout safety. The exposed surface area also makes security audits harder. Kubernetes autoscaling clashes with the requirement to reserve large and stable port ranges, which makes elasticity brittle.

The second is state stickiness.

Running one UDP port per server and demultiplexing sessions behind it solves the port problem. ICE and DTLS, however, are stateful protocols. The process that started a session has to keep receiving its packets to validate connectivity checks, complete the DTLS handshake, decrypt SRTP, and process later session changes like ICE restarts. If a packet for an existing session lands on a different process, setup fails, or media breaks.

Both pressures point to the same answer. The deployment architecture has to change while the client experience stays identical.

Splitting The Relay From The Transceiver

The architecture OpenAI shipped splits packet routing from protocol termination.

A stateless relay sits at the front, presenting a small public footprint to the internet. A stateful transceiver sits behind it, owning all the heavy WebRTC state. Signaling still goes directly to the transceiver. Media enters through the relay first.

The relay’s scope is deliberately narrow. It reads enough of each packet to choose a destination, then forwards the rest as an opaque payload. Audio stays encrypted on the way through, ICE state machines stay with the transceiver, and codec negotiation happens elsewhere. From a client’s perspective, the WebRTC session looks normal in every way.

The transceiver owns the parts of WebRTC that have to remember things. ICE connectivity checks, the DTLS handshake, SRTP encryption keys, and the session lifecycle all live there. The transceiver is the endpoint that completes the handshakes and encrypts or decrypts the actual media.

See the diagram below:

There was an obvious alternative that the team evaluated and chose against.

An SFU, or Selective Forwarding Unit, is the standard media server architecture for WebRTC at scale. It terminates one WebRTC connection per participant and selectively forwards streams between them. The AI joins as another participant.

This works well for inherently multiparty products like group calls, classrooms, and collaborative meetings. OpenAI’s workload looks different. Most sessions are 1:1, with one user talking to one model. For that kind of traffic, the SFU model adds overhead and forces backend services to behave like WebRTC peers themselves. The transceiver model lets the backend stay an ordinary service.

See the diagram below that shows the SFU approach:

TURN was also considered and set aside.

TURN is the standard protocol-terminating relay used for NAT traversal. The trouble is that TURN allocations add setup round-trips before media can flow, and migrating or recovering them across servers is hard. For a latency-sensitive workload, those extra round-trip matters.

The split solves the port and state problems in principle. The remaining problem is making the relay route the first packet correctly.

Routing The First Packet

The first packet of any new session is the difficult one.

Subsequent packets are easy because the relay has a mapping that says that packets from this source IP and port go to this transceiver. The first packet is what creates that mapping, so the relay has to figure out where to send it from the packet itself.

See the diagram below:

Two naive options were present:

• A database lookup on the hot path adds latency and a hard dependency on another service staying healthy.

• Routing to a random transceiver and forwarding internally works, but doubles the hop count.

OpenAI chose a third option.

The answer lives inside a field that WebRTC already exchanges. Every WebRTC session carries an ICE username fragment, called the ufrag, which is produced during session setup and echoed in STUN binding requests. STUN binding requests are the packets ICE uses to verify that two endpoints can actually reach each other, and they are usually the first thing a client sends on the media path.

The trick is that OpenAI generates the server-side ufrag during signaling. They can put whatever they want in it, so they encode routing metadata into it. The relay parses just enough of the first STUN binding request to read the ufrag, decode the routing hint, and forward the packet to the transceiver that owns the session. Every packet after the first one flows through the established session mapping, which skips the ufrag parsing step entirely.

See the diagram below that shows the connection establishment sequence in detail:

Each transceiver in the fleet listens on a shared UDP socket, which is one operating system endpoint bound to an internal IP and port. All sessions for that transceiver multiplex behind it.

During signaling, the transceiver returns a shared relay VIP and UDP port in the SDP answer. A VIP is a virtual IP address that fronts the entire relay fleet, so the client sees one stable destination like 203.0.113.10:3478, even though many relay instances sit behind that address. From the client’s side, there is one place to send packets, and it stays the same for the life of the session.

The relay’s state is purposefully tiny. It holds an in-memory map of source address to transceiver destination, plus some counters for monitoring and timers for session cleanup. If a relay instance restarts and loses the mapping, the next STUN packet rebuilds it from the ufrag. To make recovery faster, a Redis cache holds the source-to-destination mapping once a route is established. A restarted relay can look up the mapping from Redis immediately.

The principle here generalizes well. When we need data on the hot path, look at what the protocol is already exchanging. A field on the payload is essentially free to parse. A new lookup costs latency, a dependency, and one more thing that can break.

Global Relay and Geo-Steered Signaling

Once the public UDP surface was reduced to a small fixed set of addresses, the same relay pattern became deployable globally.

Global Relay is OpenAI’s fleet of geographically distributed relay ingress points. All of them run the identical packet-forwarding behavior described above. The only thing that changes is where on the map they sit.

Geographic distribution shortens the first client-to-OpenAI hop. A packet entering the network at a relay close to the user, in both geography and network topology, has a much easier time than a packet that has to traverse the public internet to reach a distant region first. The practical effect is lower latency, more stable timing, and a cleaner loss profile before traffic reaches the OpenAI backbone.

OpenAI uses Cloudflare for geographic and proximity steering on the signaling side. The initial HTTP or WebSocket request that sets up a session is routed to a nearby transceiver cluster. The request context then determines which Global Relay ingress point gets advertised back to the client in the SDP answer. The ufrag carries enough information for Global Relay to route media to the right cluster, and for the cluster’s relay to route to the right transceiver.

The result is that both the signaling and the media paths enter the OpenAI network at points close to the user, while the session itself stays anchored to one specific transceiver for its full lifetime. The setup round-trip and the first ICE connectivity check both shorten, which directly reduces how long a user waits before they can start speaking.

The Go Relay Implementation

The relay is a Go service running in userspace, which is to say a regular process that reads from a regular UDP socket.

The Linux kernel receives UDP packets from the network interface, delivers them to a socket bound to the relay’s IP and port, and the Go process reads from that socket, updates a small amount of flow state, and forwards each packet to the right transceiver.

OpenAI evaluated kernel-bypass frameworks (which let a userspace process poll the network card’s queues directly) and chose to stay away. Bypass raises packet throughput at the cost of operational complexity. The team’s workload fit inside what a careful Go implementation could handle.

Three implementation choices carry most of the performance load.

• SO_REUSEPORT is a Linux socket option that lets multiple workers on the same machine bind the same UDP port. The kernel then distributes incoming packets across those workers, which removes the bottleneck of a single read loop.

• runtime.LockOSThread pins each UDP-reading goroutine (a lightweight thread in Go) to a specific OS thread. Combined with SO_REUSEPORT, this tends to keep packets from the same flow on the same CPU core, which helps cache locality and reduces context switching.

• Pre-allocated buffers and minimal copying during packet parsing keep allocation overhead low and avoid putting pressure on Go’s garbage collector.

The takeaway is that ordinary optimizations were enough. The relay handles OpenAI’s global real-time media traffic on a relatively small footprint.

Design Tradeoffs

Every architecture comes with tradeoffs, and this one carries several worth understanding.

• The whole design is built around 1:1 sessions. If OpenAI ever wants to add multiparty features (group voice calls, multiple participants in a single AI session, human handoff during a call), large parts of this architecture would probably need rework. Both the transceiver model and the choice to skip SFU rely on the assumption that most sessions have exactly two endpoints.

• OpenAI also took on a custom infrastructure burden. A standard SFU comes with documentation, a community, and battle-tested patterns. The relay, the transceiver, and the coordination between them are all internal code. Every engineer who works on this stack has to learn it from the inside out.

• The “stateless” relay turns out to be stateless mostly in spirit. It holds an in-memory flow table and uses a Redis cache to recover that table across restarts. The architecture works because the protocol can rebuild the table from the ufrag, but the soft state is still a state.

• Lastly, the ufrag trick depends on controlling both ends of signaling. OpenAI generates the server-side ufrag, so they can embed routing metadata in it freely. A team that uses an off-the-shelf signaling stack might find this technique harder to adapt directly.

Conclusion

The architecture OpenAI built for voice AI is a careful response to a specific pressure. WebRTC was designed for stable servers. Modern cloud infrastructure runs on the opposite assumption. OpenAI’s team had the protocol depth to determine whether a WebRTC session needs to live in one process at all.

Their answer separates the work into two pieces. A stateless relay forwards packets near the user. A stateful transceiver, anchored in one place, owns ICE, DTLS, SRTP, and the session lifecycle. The two pieces communicate through information that’s already in the WebRTC handshake, which keeps the routing decision on the packet path itself.

The implementation choices stayed deliberately simple. Userspace Go, SO_REUSEPORT, thread pinning, and careful memory management did the work that kernel bypass solves. The result handles 900 million weekly users on a relatively small relay footprint.

References:

• How OpenAI delivers low-latency voice AI at scale