AI coding agents can generate code quickly, but CI checks often happen after the agents finish their work. Sonar Vortex changes this pattern. Sonar Vortex operates inside the agent’s coding loop, giving agents architectural context before they write and verifying their output in real time as they produce it. Internal testing found 36% lower token consumption and 92% fewer defects.

Explore Sonar Vortex

Thinking Machines released a model called Inkling on July 15, 2026. A few interesting points made in its introduction are as follows:

• Inkling has 66 layers, and each layer holds 256 experts, of which only six activate for any given token [2].

• Most layers can access only a short window of recent text, while a few can access all of it [1].

• Word positions are encoded using a method most labs moved away from years ago.

Thinking Machines, founded by Mira Murati (the ex-CTO of OpenAI), describes its mission as building AI that extends human will and judgment. The company lists four directions of work, which are training strong models, building tools that let people customise models with their own knowledge, developing interfaces that widen the communication channel between people and machines, and publishing research on how models are made [3].

Before Inkling, the company shipped Tinker, a service for fine-tuning open models [4]. Inkling is the company’s first model trained from scratch [1]. The weights sit on Hugging Face under an Apache 2.0 license [2], so anyone can download them and retrain the model on their own data.

In this article, we will work through the various choices Thinking Machines made while building Inkling. Here is what we will cover:

• Mixture of Experts, and the gap between 975 billion parameters and 41 billion.

• The mix of local and global attention layers behind a context window of one million tokens.

• Position encoding, and the older method Thinking Machines chose over the current standard.

• How images and audio enter the model without a separately pretrained encoder in front of them.

• Thinking effort, a setting between 0 and 1 that adjusts how much the model reasons before answering.

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

The diagram below shows where each of these five things sits inside the model.

Groundwork

Let us first understand four key terms that are really important to make sense of the architecture:

• Token: A token is a chunk of text. Models work with pieces smaller than sentences and usually a bit smaller than words. For example, the sentence “Inkling was released in July” might become six tokens, with common words getting one token each and unusual words getting split into two or three.

• Parameter: A parameter is one number stored inside the model, learned during training. When you read that a model has 975 billion parameters, it is the count of individual numbers sitting in the file. Each one started as random noise and was adjusted millions of times until the model produced sensible text.

• Training: It works by showing the model text, letting it predict the next token, comparing that prediction against the token that actually came next, and then adjusting every parameter slightly in whichever direction would have made the prediction better. The size and direction of each adjustment is called a gradient, and the procedure that computes all of them at once is called backpropagation. By repeating this across trillions of tokens, the parameters settle into values that produce useful predictions.

• Layer: It is one processing stage in the journey of a token. Inkling has 66 of them stacked in order [2]. A token’s representation enters layer 1, gets transformed, passes to layer 2, and so on until layer 66, after which the model predicts the next token. Every layer has the same two parts:

  • An attention step that pulls in information from other tokens in the sequence.

  • A feed-forward step that transforms the result.

Sparsity

Inkling separates the cost of storing a model from the cost of running it. This is the reason a model this large is affordable to use.

In an ordinary transformer, the feed-forward step in each layer is a single network, and every token passes through all of it. If that network holds 5 billion parameters, then every token processed involves all 5 billion.

Inkling replaces that single network with 256 smaller ones, called experts. For each token, a selection step picks six of the 256. Only those six run, and the remaining 250 sit idle for that token while handling other tokens instead [2]. This design pattern is called Mixture of Experts, and Thinking Machines states that their version largely follows the approach published by DeepSeek [1][8].

See the diagram below:

Let’s understand this via a simple example:

• Suppose you build a layer with 10 experts holding 1 billion parameters each, and you run 2 of them per token.

• The file on disk contains all 10 billion parameters, and every one of them has to be loaded into memory before you can run the model at all.

• Processing a single token involves 2 billion parameters, so each token costs about a fifth of what it would in an ordinary layer of the same total size.

Inkling does this at scale. The total across the whole model is 975 billion parameters, and the count involved in processing any single token is roughly 41 billion [1]. That is about 4 percent of the model running at a time.

The hardware requirements make things clearer. A checkpoint is the saved file holding all the trained parameters. Inkling’s full-precision checkpoint needs at least 2 TB of combined GPU memory, which the model card gives as eight NVIDIA B300 cards or sixteen H200 cards [2].

Thinking Machines also talks about a quantised checkpoint. This means the same parameters are stored with less numerical precision, in roughly the way that writing 3.14 instead of 3.14159265 uses less space at the cost of some accuracy. That version needs around 600 GB and fits on four B300 cards [2].

What Makes an FDE Role Credible? (Sponsored)

Strong candidates are skeptical of vague forward deployed engineer postings, and the title alone won’t earn their trust.

The free State of FDE Jobs 2026 Report explains what candidates look for, how the market is evolving, and how employers can make these roles easier to understand.

Hiring? You can also bring your openings to forwarddeployedengineer.com, the focused jobs board for forward-deployed engineers.

Explore the Job Board

Routing

Picking six experts out of 256 sounds simple, but it is not. Here’s how the selection works:

• A small component called the router produces a score for every expert.

• Each expert has its own list of numbers attached to it, learned during training.

• The router compares the incoming token’s representation against each of those lists and gets a raw number out, higher when the two match closely.

• Those raw numbers can be any size, so they get passed through a sigmoid. A sigmoid is a mathematical function that takes any number and converts it into a value between 0 and 1. For example, feed it 8, and you get back about 0.9997. Feed it 0, and you get exactly 0.5. Feed it negative 4, and you get about 0.018. Very large numbers approach 1, very negative numbers approach 0, and everything else ends up somewhere in between. Inkling’s router uses a sigmoid to produce its expert scores [1].

• After scoring, the six highest-scoring experts run, and their outputs are combined using those same scores as weights. An expert scoring 0.9 contributes more to the result than one scoring 0.4.

This approach makes one specific type of failure more probable. To understand the failure, consider what happens across millions of training steps. Suppose, by luck, expert 47 gets picked slightly more often than average early on. It receives more tokens, so it receives more gradient updates, so it improves faster than its neighbours. Since it is better, the router scores it higher. Because it scores higher, it gets picked even more.

If we run that loop long enough, we can end up with a 256-expert layer where perhaps 20 experts handle nearly everything and the other 236 stay underdeveloped. This is called routing collapse. In other words, while we paid to store 256 experts, we got the capability of just 20. There is a second cost too. Since experts are usually spread across different machines, a machine holding four popular experts can become a bottleneck while its neighbours idle [9].

The traditional fix for this adds a penalty to the training objective that grows when expert usage is uneven. This means training the model on two things at once: predicting the next token correctly, and keeping expert usage balanced.

The trouble is that these two goals produce gradients pointing in different directions. The prediction gradient might say “increase this parameter,” while the balance penalty says “decrease it.” One of them wins, and either way the model is being pulled away from what you actually wanted. If we set the penalty strength high, the text quality degrades. If we set it too low, the experts collapse anyway [9].

Thinking Machines uses a method introduced by Wang and colleagues, which was also adopted by DeepSeek [1][8][9]. This method removes the conflict entirely by keeping a separate bias value for each expert, which is just a small number added to that expert’s score. The trick is where the bias gets applied.

Let us walk through a token arriving at a layer in this setup. Consider that the router produces these sigmoid scores for four of the experts.

Expert 47 has been overloaded recently, so its bias has drifted down to −0.15. This drops its selection score below expert 88’s, and expert 88 takes the slot instead. The bias changed which expert got picked. When expert 88’s output is combined into the final result, it is weighted by 0.80, its original router score, with the bias left out. In other words, the bias affects selection only. It never touches the weighting, and it is updated by a simple counting rule that runs outside backpropagation entirely. The main goal is to nudge down busy experts while giving a chance to the quieter experts.

The result is that expert usage stays balanced, and the training objective receives no competing gradient at all.

One point to note here is that two additional experts run on every single token regardless of routing [2]. These are called shared experts, and they hold the general-purpose processing that nearly every token needs. Thinking Machines states that the scores of the six selected routed experts and the two shared experts are normalised together before being used as weights [1], which means all eight are scaled to a common range and contribute proportionally. In other words, eight experts run per token in total.

Attention

Sparsity handles the cost of the feed-forward step. The attention step carries a separate cost that grows far faster, and Inkling manages it by having most layers examine very little.

For every token, the model computes three sets of numbers:

• A query, describing what kind of information this position needs.

• A key, describing what kind of information this position offers.

• A value, carrying the actual content this position passes along.

Every token’s query is compared against every earlier token’s key. Strong matches produce high scores, and those scores determine how much of each earlier token’s value gets pulled into the current position. This is how the word “it” in a sentence ends up connected to the noun it refers to.

Since every token compares itself against every earlier token, the number of comparisons grows with the square of the sequence length. For example:

• 1,000 tokens produce roughly 1 million comparisons.

• 10,000 tokens produce roughly 100 million.

• 1,000,000 tokens produce roughly 1 trillion.

Inkling supports a context window of one million tokens [2], and a trillion comparisons per layer across 66 layers is far beyond what any reasonable amount of hardware can deliver.

This is where a sliding-window layer restricts each token to a fixed number of recent tokens rather than everything before it. If the window is 1,000, then token number 500,000 compares itself against tokens 499,000 through 500,000 and stops there. That is 1,000 comparisons instead of 500,000, and the total for the whole sequence grows in a straight line rather than as a square.

This raises the obvious question that if every layer sees only a small window, how does information from page one of a long document ever reach page four hundred?

Inkling alternates between sliding-window layers and full-attention layers at a ratio of 5:1 [1]. Integration notes published by the vLLM project put concrete numbers on the split, describing the 66 layers as 55 sliding-window layers and 11 full-attention layers [7].

Information from far away travels through those eleven full layers. For example, picture a fact at token 200 that matters for a prediction at token 900,000. In layer 6, the first full-attention layer, the representation at position 900,000 can reach back and pick up that fact directly. From there, it rides forward within the local representations, is refreshed at layer 12, again at layer 18, and so on. The long-range path exists on roughly one layer in six, and the other five handle nearby context at a fraction of the price.

Inkling also uses 8 key-value heads [1]. Attention normally runs several times in parallel with different query, key, and value sets, and each parallel copy is called a head. Sharing a smaller number of key and value sets across many query heads reduces the memory needed while generating text.

See the diagram below that tries to show this setup in a simplistic manner:

There is one practical consequence of this approach. Long-context models often handle the general content of a large document well, but can still miss one specific detail buried in the middle.

Position

Dealing with a million tokens brings up another question: how does the model represent where each token sits in the sequence?

Thinking Machines picked an older technique over the current standard. This is due to the lengths the model never saw during training.

The comparison between queries and keys involves no information about order. For example, both “the cat bit the man” and “the man bit the cat” contain the same five tokens, so without a position signal, the attention step produces the same comparisons for both. Something has to tell the model that “cat” came before “bit” in one case and after it in the other.

Almost every recent open model uses Rotary Position Embedding, shortened to RoPE. Each token’s query and key are treated as points that get rotated by an angle proportional to that token’s position in the sequence. Token 1 gets a small rotation, token 500 gets a much larger one, and so on.

The usefulness of this shows up when two tokens are compared. Since both were rotated by amounts tied to their own positions, the comparison between them depends on the difference between those two rotations, which tells how far apart they are.

However, those rotation angles were only ever encountered at positions the model actually trained on. If training used sequences up to 32,000 tokens, then every angle the model learned to interpret came from that range. If we ask it about position 900,000, the angle involved falls outside anything it has experience with.

A whole family of techniques exists specifically to stretch RoPE into ranges beyond its training data.

Inkling uses a relative scheme in the style of Shaw and colleagues [1][10]. Rather than encoding where each token sits, this approach learns a value for each distance between two tokens and adds that value directly to the comparison score. For example, tokens at positions 5 and 9 are 4 apart. Tokens at positions 500,005 and 500,009 are also 4 apart. A relative scheme treats both pairs identically, because 4 is 4 wherever it occurs. Distances beyond some cutoff, say anything more than 128 apart, all share the same learned value, so a pair 900,000 tokens apart uses a value the model has seen countless times during training. Nothing has to be extrapolated.

The vLLM notes describe the implementation as a learned relative-position term added to the attention scores before they are converted into weights [7].

Thinking Machines states that this performed better and extrapolated better to longer sequences than RoPE in their testing [1].

Convolutions

Inkling adds one small operation named convolution to handle a job that attention would otherwise have to learn from scratch.

A convolution here means combining each position in the sequence with a few positions immediately before it. Inkling uses a window of four [7], so the numbers at position 100 get mixed with the numbers at positions 97, 98, and 99, using a small set of learned weights. Position 101 does the same with 98, 99, and 100. It is a cheap, fixed, strictly local operation.

To understand better, consider the two tokens that make up “New York.” The second token means something quite different on its own than it does following the first. Attention can learn to make that connection, and it has to learn it, because attention starts as a general comparison mechanism with no built-in preference for nearby tokens over distant ones. A convolution supplies that local mixing directly through its structure, without any training required to discover that neighbours matter.

Thinking Machines places these convolutions at two kinds of locations:

• First, on the keys and values inside each attention layer

• Second, on the outputs of the attention and feed-forward steps before those outputs rejoin the main path through the model [1].

The effect is that immediate context arrives pre-mixed, and attention can spend its capacity on the connections that genuinely require learning.

Multimodality

Everything so far concerns text moving through the model. Images and audio have to get in first, and Inkling accepts both without a separately trained encoder standing in front of it.

Most multimodal models attach three trained components to a language model:

• A vision encoder, trained on its own beforehand, converts an image into a list of numbers.

• An audio encoder does the same for sound.

• Projection layers then convert both into the format the language model works with.

However, in the case of Inkling, Sound arrives as a mel spectrogram, which is a standard way of representing audio as a grid of numbers. Frequency bands run down one axis, short slices of time run across the other, and each cell holds a loudness value. A three-second clip becomes a grid of a few hundred columns.

The dMel method then rounds each of those loudness values to one of a fixed set of levels, in the same way you might round 0.73 to 0.7 [1][11]. That is the entire conversion. No separate audio model needs training beforehand, because rounding numbers requires no training.

In the case of images, an image is cut into square patches measuring 40 by 40 pixels [1]. A 400 by 400 pixel image therefore becomes 100 patches. Each patch passes through a small four-stage network called an hMLP stem, which combines the pixels within that patch and processes each patch independently of every other one [1][12]. The paper describing this reports that it adds under one percent to compute compared with the simplest possible alternative [12].

Both then pass through a lightweight conversion layer and join the text tokens in a single sequence, processed by the same 66 layers we have been talking about [1]. Thinking Machines states that these multimodal components were trained from scratch on general-domain data [1], meaning they learned alongside the rest of the model rather than arriving pretrained.

See the diagram below:

One point of confusion can be that the launch announcement labels this as an encoder-free architecture [1], while the model card describes images as being encoded via a hierarchical patch encoder [2]. Both descriptions are accurate. Encoder-free here means there is no large, separately pretrained encoder network, rather than meaning there is no processing at all.

Moreover, this design predates Inkling. Thinking Machines described the same arrangement two months earlier for their real-time interaction system, using dMel for audio, 40 by 40 patches through an hMLP for images, and a lightweight conversion layer, with every component trained together from scratch [5].

Effort

How long the model reasons before answering is adjustable, and it was trained into the model rather than requested through wording. This impacts the model’s benchmark numbers

Effort is a number between 0 and 1. The documented presets run from 0 for none, through 0.1 for minimal and 0.2 for low, then 0.7 for medium, 0.9 as the default, and 0.99 at the top [6]. The spacing between those values is uneven, which is a hint that the number represents a learned response rather than a token allowance.

Mechanically, the setting arrives as text. Before the conversation begins, a system message stating the effort level is inserted ahead of everything else [6].

Reasoning models produce a stretch of working out before their final answer. This working-out phase costs tokens like everything else. Thinking Machines trained Inkling’s response to the effort setting during reinforcement learning, a training stage where the model produces complete attempts at tasks and receives a score for each attempt, then adjusts toward whatever scored well.

During that stage, Thinking Machines varied the effort message across attempts while also adjusting the cost charged per token generated:

• An attempt labelled high effort could produce lengthy working-out without much penalty.

• An attempt labelled low effort was charged heavily for every token, so short answers scored better.

Across many attempts, the connection between the message and the profitable length was learned.

Here are some key points about limits:

• Higher effort encourages more reasoning without guaranteeing a longer response or a better one on any single sample.

• Setting effort to 0 pushes the model toward minimal reasoning without enforcing it as a hard rule.

• Effort and the maximum token limit are separate settings that work independently, so a high effort setting may need a larger token limit to avoid being cut off.

Sweeping the effort setting across its range produces a curve of score against tokens generated. On the coding benchmark Terminal Bench 2.1, Inkling reaches the same score as NVIDIA’s Nemotron 3 Ultra while producing roughly a third as many tokens [1].

Tradeoffs

Each of the design decisions we have looked at has trade-offs. Let us look at some of them:

• Sparse routing keeps the per-token cost low and requires the entire model in memory regardless. The hardware floor stays high even though each token is cheap.

• Mostly-local attention makes a million-token window affordable and leaves most layers with a narrow view of the sequence.

• Relative position encoding removes the extrapolation problem and means every serving framework had to write new code for it, since the surrounding tooling was built around RoPE [7].

• Open weights covers the weights. The training data, the exact recipe, and the training code remain private, and the model card describes data provenance only in general terms [2]. The model can be modified, and reproducing or auditing it stays out of reach.

• Safety behaviour is adjustable by anyone who retrains the model. The model card recommends layering external moderation tools around the model rather than relying on its own refusals, particularly for consumer-facing deployments [2].

Thinking Machines states that other models available today, both open and closed, are stronger overall [1]. However, considering the availability for fine-tuning from day one, and the quantised checkpoint that brings the hardware requirements down, it is clear that Inkling has been built to be taken and adapted to the company’s use case.

Conclusion

Let us look at the key ideas we’ve encountered while understanding Inkling’s design and architectural decisions:

• Sparsity separates storage from compute. Total parameters tell you how much memory you need to hold the model. Active parameters tell you how much work each token requires. Inkling stores 975 billion and runs about 41 billion at a time.

• Routing needs balancing, and the balancing mechanism matters. Applying a bias when selecting experts, while leaving it out when weighting their outputs, keeps usage even without adding a competing goal to training.

• Long context works because most layers see little. Inkling runs 55 sliding-window layers and 11 full-attention layers, and long-range information travels through the eleven.

• Position encoding is a decision about untrained lengths. Encoding the distance between tokens, rather than their absolute positions, means a pair 900,000 apart uses a value the model has seen many times.

• Reasoning effort can be a trained setting. When it is, a benchmark score becomes one point on a curve rather than a fixed property of the model.

References

1. Inkling: Our Open-Weights Model, Thinking Machines Lab

2. Inkling Model Card, Thinking Machines Lab

3. The Future Worth Building Is Human, Thinking Machines Lab

4. Tinker, Thinking Machines Lab

5. Interaction Models: A Scalable Approach to Human-AI Collaboration, Thinking Machines Lab

6. Thinking effort, Tinker Documentation

7. thinkingmachines/Inkling, vLLM Recipes

8. DeepSeek-V3 Technical Report, DeepSeek-AI

9. Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts

10. Self-Attention with Relative Position Representations

11. dMel: Speech Tokenization made Simple

12. Three things everyone should know about Vision Transformers

Designed and assembled in America, Matic is the world’s first robot built to understand you. Its new feature, Matic Cues, lets you interact with it like you would anyone else.

• Point and speak. Say “Hey Matic, clean this” while pointing at a mess, and Matic knows exactly what to clean.

• Understands 70+ languages. Ask Matic to clean the kitchen, follow you, or go to the sink, in whatever language you speak.

• Skip the app. Anyone at home can use Matic, not just app-savvy adults, but kids and grandparents too.

Every home robot before this needed an interface. Matic just needs you to talk.
Try Matic, backed by their 6 month money back guarantee.

Get Matic

A vehicle travelling at 40 miles per hour covers about 60 feet every second. Within that second, software has to determine what is physically nearby, classify each object, estimate where those objects will move, select a path, and issue steering and braking commands.

Doing all of this quickly is something that has been largely solved. However, doing this correctly in unpredictable and distinct traffic situations is still an open problem. This is because real-world traffic produces more distinct scenarios than any team can pre-determine. For example, Waymo describes one such case, where a vehicle is on fire on the road ahead while the drivable lanes remain physically clear [3]. The geometry of such a scenario might permit driving straight through it. However, the real meaning of it calls for turning around or taking preventive action.

Waymo and Tesla, two companies investing in self-driving cars, have tried to come up with different types of answers to these questions.

For reference, Waymo reports 220.6 million rider-only miles through March 2026. These are miles covered with no human in the driver’s seat, across five metro areas [5]. On the other hand, Tesla reports more than three million vehicles in the United States covering over 30 billion miles a year, with 1.28 million active Full Self-Driving subscriptions in the first quarter of 2026 [10]. Almost all of those Tesla miles involve a driver who remains responsible for the vehicle. Tesla’s driverless service is separate and much smaller, running without safety monitors in Austin, Dallas, and Houston, while the Bay Area service uses a safety driver [10].

Both approaches depend heavily on machine learning. But they differ in how much gets fixed in advance. In this article, we will take a look at both approaches while trying to answer the following questions:

• How does each system detect what is physically nearby?

• What each builds from that data, and why one keeps the result readable?

• How each estimates what other road users will do?

• How a path gets selected, and what verifies it before the vehicle acts?

• What safety evidence each publishes, and why the figures measure different things?

• Where does the knowledge inside each system come from?

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

Sensing

A camera records light intensity across a grid of pixels. Distance appears nowhere in that grid, so depth has to be computed from the arrangement of pixels, but that computation can be wrong. A large object far away and a small object nearby can occupy the same region of an image.

Lidar arrives at the same answer by a different route. The unit emits laser pulses, measures how long each pulse takes to return after reflecting off a surface, and converts that interval into a distance. We can think of the output of this as a point cloud, which is a three-dimensional set of measured points describing the surfaces around the vehicle [1]. The distance is no longer an estimate but a measurement.

Waymo’s sixth-generation system, which began fully autonomous operations in February 2026, carries 13 cameras, four lidar units, six radar units, and a set of external audio receivers used to detect sirens and railroad crossings [2]. Coverage overlaps in every direction and extends to 500 metres. This overlap is for help situations when rain, road grime, or ice limits what a camera captures. Lidar and radar sustain the perception capabilities in such scenarios. [2].

Tesla’s vehicles mainly rely on cameras. Instead, Tesla relies entirely on a “pure vision” approach that uses exterior cameras and artificial intelligence to navigate. For example, Tesla’s documentation describes Model 3 and Model Y as running camera-based Tesla Vision, without radar, using cameras and neural network processing [9].

See the diagram below:

There is another point here that requires our attention. Waymo’s fifth-generation Jaguar I-PACE vehicles carry 29 cameras [1]. The sixth-generation system carries 13, which Waymo attributes to a 17-megapixel imager covering the same area with fewer than half the cameras [2]. In other words, Waymo is also reducing overall sensor count while continuing to describe redundancy as essential.

The tradeoff here is that a direct measurement costs money while adding a component that can fail noticeably. However, a derived value may not cost much, but it can be wrong as well.

Representation

Something has to convert millions of pixels and points into a description that the system can understand and operate on. The nature and form of that description is one of the most important architectural decisions for an autonomous driving setup.

Waymo places the Waymo Foundation Model at the centre, built from two components:

• Sensor Fusion Encoder: It merges camera, lidar, and radar data over time and outputs objects, semantic attributes, and embeddings. These are compact numerical summaries that downstream components consume.

• A Driving VLM: Trained using Gemini and fine-tuned on Waymo driving data, they cover rare situations requiring background world knowledge, such as the burning vehicle example mentioned earlier.

Both feed a World Decoder that forecasts the behaviour of other road users, produces high-definition maps, generates candidate trajectories, and emits signals used to verify them.

The system maintains compact structured representations, meaning explicit lists of objects, their semantic attributes, and roadgraph elements describing lanes and connections. The Waymo engineering team provides three reasons for such a setup:

• Correctness and safety validation can run at inference time, while the vehicle is moving

• Simulation runs efficiently at large scale, because a compact world state is cheap to replay and modify

• Training feedback becomes verifiable, since a component evaluating driving quality has something concrete to measure

This design provides significant benefits over pure end-to-end or modular approaches [3].

Tesla’s documentation talks about per-camera networks performing semantic segmentation, which assigns every pixel to a category, plus object detection and monocular depth estimation, meaning distance estimated from a single camera [6]. Those feed birds-eye-view networks that output road layout, static infrastructure, and three-dimensional objects in a top-down view. A full build involves 48 networks taking nearly 70,000 GPU hours to train and producing 1,000 distinct tensors per time step [6].

See the diagram below:

Prior mapping also occupies one end of this decision. For example, Waymo surveys a territory before operating there, recording lane markers, signs, curbs, and crosswalks. It then matches those maps against live sensor data to establish position, since GPS alone can lose signal [1]. The map is basically knowledge acquired once and reused. It saves computation on every trip, but also creates an obligation to keep it current. In contrast, Tesla’s approach skips the survey and derives equivalent information during the drive.

The trade-off is that a structured representation can be inspected, logged, replayed, and checked against explicit criteria. But it limits what the system can express. On the other hand, a learned representation carries nuance no schema can anticipate.

Prediction

Once the system holds a description of its surroundings, the next task is estimating what those objects will do. Several futures are valid at the same moment. For example, a cyclist approaching an intersection might continue straight, turn, or stop. A safe response should account for all possibilities.

Waymo describes the system as producing many possible paths for each road user rather than one, drawing on accumulated driving data and accounting for the different ways a car, a cyclist, and a pedestrian move [1].

In June 2025, Waymo published research on whether prediction quality scales predictably [4]. Using an internal dataset spanning 500,000 hours of driving, the study found that motion forecasting quality follows a power law in training compute, matching a pattern observed in language models. A power law here means each doubling of compute yields a proportional, predictable improvement. Data scaling proved critical, and increasing compute at inference time improved performance on harder scenarios.

Waymo reported the same trend in closed-loop performance, where results are measured in simulations in which the system’s own actions change what happens next. This trend suggested that real-world driving improves with more data and compute, rather than only benchmark scores.

Tesla’s describes an upgraded reinforcement learning stage in version 14.3, intended to cover long-tail edge cases, meaning rare situations that appear infrequently even across very large mileage [10].

See the diagram below:

A system that commits to one predicted future becomes fragile in situations where prediction matters most. However, carrying several weighted futures costs compute on every cycle, but can guard against the possibility of unusual situations.

Planning

With a description of the surroundings and a set of likely futures, the system selects a trajectory. You can think of it as a specific path with speeds attached to it. The question is what verifies the trajectory before the vehicle executes it.

Waymo trains large Teacher models to generate safe, comfortable, and compliant action sequences. It then distils them into smaller Student models sized to run onboard in real time [3]. Distillation transfers behaviour from a large model to a compact one. Output from that Student model then passes through a separate onboard validation layer, which verifies the trajectories the generative model produced [3]. This means that two independent components have to agree before the vehicle moves.

Tesla talks about building a planning and decision-making system that operates under uncertainty, with algorithms evaluated at the scale of the entire fleet, optimising for throughput, latency, correctness and determinism [6].

For Tesla vehicles on the road today, verification comes from a person. Full Self-Driving (Supervised) requires an attentive driver and leaves the vehicle slightly short of autonomous [7]. The system enforces this through a strikeout mechanism, where repeated inattention warnings disengage the feature for the remainder of a trip. Enough strikeouts suspend access for a week [7]. In the driverless service, that role belongs to safety monitors or remote supervision [10].

A validation layer catches a category of unacceptable outputs before they reach the actuators, and it can only evaluate against the defined criteria. Anything outside those criteria passes through unexamined. This is the same tradeoff as an assertion in production code, where the check is only as good as the condition behind it.

See the diagram below:

Validation

Waymo’s Safety Impact hub reports 220.6 million rider-only miles through March 2026, meaning no human occupied the driver’s seat for any of them [5]. Measured against human crash rates in the same operating areas, adjusted for where within each city the service drives, the reported reductions are 94% for serious injury or worse crashes and 82% for injury-causing crashes [5]. The methodology has been published in peer-reviewed journals, and the raw data is downloadable so third parties can reproduce the figures [5].

Tesla’s Vehicle Safety Report takes a different form. It compares Teslas with Full Self-Driving (Supervised) engaged against Teslas driven manually, using the same telemetry pipeline for both, and reports 7 times fewer major and minor collisions and 5 times fewer off-highway collisions [7][8]. A collision counts as occurring with the system engaged if it was active at any point within five seconds beforehand, a window chosen to capture cases where a driver took over shortly before impact [8]. Tesla attributes no fault in the reported data, treating that determination as too subjective to include [8].

As you can see, the two reports answer different questions, and we can’t simply do a side-by-side comparison. The difference comes down to what each population represents:

• Waymo is measuring outcomes across miles where no person was available to intervene.

• Tesla is measuring whether an assistance system reduces collisions while a driver remains responsible.

Both companies also state their own limitations. Waymo states that no perfect comparison between autonomous and human data exists today, and that its operating cities see no appreciable snowfall [5]. Tesla states that its estimate of a United States average involves unavoidable assumptions that may skew the figure in either direction [8].

Waymo separates two things that are easy to conflate. Safety impact gets measured after deployment. Whether a release is acceptable to deploy at all gets determined beforehand through a Safety Framework and a Safety Case [5].

Training

Both systems improve between releases. However, the underlying mechanisms differ as much as the architectures.

Waymo runs three components off the same foundation model:

• The Driver produces action sequences.

• The Simulator generates scenarios for training and testing.

• The Critic evaluates driving quality and surfaces problems.

Large versions of each get distilled into smaller ones that run at the required volume. Two loops connect them:

• An inner loop applies reinforcement learning inside simulation, where scenarios can be generated and repeated cheaply.

• An outer loop begins with the Critic flagging suboptimal behaviour from real driving, turns improved alternatives into training data, verifies the fixes in simulation, and deploys only once the safety framework confirms the absence of unreasonable risk.

Waymo states that its fully autonomous mileage now far exceeds its manually driven data, and that no volume of simulation or test-driver operation reproduces the situations encountered when the system operates with no driver present [3].

Tesla’s data comes from a consumer fleet. The Vehicle Safety Report describes two telemetry paths [8]. On shifting to park, a vehicle transmits anonymised mileage broken down by control type and road classification. On detecting a major or minor collision, it transmits a separate packet tied to the vehicle. Tesla reports receiving 2.5 billion telemetry packages in the third quarter of 2025 alone. Tesla has also built an evaluation infrastructure from anonymised fleet clips assembled into test suites, alongside simulation producing sensor data for automated testing [6]. Training runs on Cortex 1, listed at over 100,000 H100-equivalent GPUs in production, and Cortex 2 at over 130,000 in early ramp.

See the diagram below:

Conclusion

The same question recurs at every stage of an autonomous driving process.

How much gets determined in advance and written into a form that can be examined, and how much gets computed during the drive by a model whose internal state stays out of reach?

Each stage poses one version of it:

• Sensing asks whether distance arrives as a measurement or as a derived value

• Representation asks whether the description of the world stays inspectable

• Prediction asks how many futures the system carries at once

• Planning asks what verifies a trajectory before execution

• Validation asks what kind of safety claim the resulting evidence can support

• Training asks which miles improve the system

Waymo sits further toward written-down knowledge at most stages. This involves per-city preparation and purpose-built hardware. Tesla sits further toward computed knowledge. Both positions have merit, and only the future will tell which approach becomes more dominant or do things fall somewhere in the middle.

References

1. Self-Driving Car Technology for a Reliable Ride, Waymo

2. Beginning fully autonomous operations with the 6th-generation Waymo Driver, Waymo, February 2026

3. Demonstrably Safe AI For Autonomous Driving, Waymo, December 2025

4. New Insights for Scaling Laws in Autonomous Driving, Waymo, June 2025

5. Waymo Safety Impact, Waymo

6. AI and Robotics, Tesla

7. Full Self-Driving (Supervised), Tesla Support

8. Full Self-Driving (Supervised) Vehicle Safety Report, Tesla

9. Autopilot and Full Self-Driving Capability, Tesla Support

10. Tesla Q1 2026 Update, Tesla Investor Relations

This isn’t a conference you sit through. AWS re:Invent includes 2,200+ sessions, and 70% are interactive; workshops, code talks, AWS Jams, GameDays, and the Architecture Rodeo, built for engineers who’d rather work through a problem than watch a slidedeck.

Between November 30 and December 4, you’ll have the opportunity to:

• Run demos against real workloads with sandbox access in small groups

• Test newly launched AWS services in the AWS Village before your team adopts them

• Work through architecture decisions directly with service engineers

Save $1,200 with early bird pricing

This week’s system design refresher:

• HTTP vs HTTPS Explained (Youtube video)

• What is Google’s TPU?

• 9 Types of API Testing

• Common types of AI Agents guardrails on production

• Forward Proxy, Reverse Proxy, and API Gateway Explained

HTTP vs HTTPS Explained

What is Google’s TPU?

A TPU (Tensor Processing Unit) is Google’s custom AI chip, designed from scratch for the giant matrix multiplications that modern models live on. GPUs were built for graphics first.

TPUs were built for deep learning from day one.

At Cloud Next ’26, Google unveiled its 8th generation, and for the first time it ships in two flavors. TPU 8t is built for training, where raw throughput wins. TPU 8i is built for inference, where latency and chip-to-chip speed matter most.

Both still share the same Axion CPUs, liquid cooling, and software stack, so code written for one runs on the other.

The diagram is a quick study guide to what’s the same, what’s different, and why, based on our understanding of published Google articles.

What Makes an FDE Role Credible? (Sponsored)

Strong candidates are skeptical of vague forward deployed engineer postings, and the title alone won’t earn their trust.

The free State of FDE Jobs 2026 Report explains what candidates look for, how the market is evolving, and how employers can make these roles easier to understand.

Hiring? You can also bring your openings to forwarddeployedengineer.com, the focused jobs board for forward-deployed engineers.

Explore the Job Board

9 Types of API Testing

Here are the 9 most common API tests used to catch different kinds of failures.

• Smoke tests run right after a deploy to confirm the critical endpoints still respond, especially things like login, checkout, or health checks.

• Functional testing checks whether an endpoint does what the business expects or just returns a clean 200 with the wrong number inside.

• Contract testing protects the agreement between services. If a consumer depends on a field, a type, or a status code, the provider should not change it without warning.

• Integration testing covers the full workflow across systems. An order endpoint may also use inventory, payment, and notifications, and a- ny of those dependencies can fail.

• Regression testing protects existing behavior when new changes are added. A change in discount logic should not break checkout totals or order history.

• Load testing asks how things hold up under expected traffic.

• Stress testing keeps pushing until something breaks.

• Security testing covers auth, access control, unsanitized input, and what your errors leak.

• Fuzz testing sends unexpected or invalid inputs to the API to find bugs that normal test cases may miss.

Over to you: Which of these do you rely on most in practice?

Common types of AI Agents guardrails on production

• Input Screening: Every user request is scanned for prompt injection, sensitive data, and off-scope topics before the model sees it. Unsafe inputs return a fallback message.

• Context Verification: Every user request is scanned for prompt injection, sensitive data, and off-scope topics before the model sees it. Unsafe inputs return a fallback message.

• Response Generation: The LLM reasons only over verified context.

• Output Validation: Every response is checked for groundedness, format, and safety before it leaves the system. Failures trigger up to two retries, then a safe fallback.

• Operational Controls: A final layer enforces limits, logs every call, and routes low-confidence or high-risk actions to humans.

Reliable agents aren't built on better prompts. They're built on the guardrails wrapped around them.

Over to you: Which guardrail layer catches the most issues in your setup?

Forward Proxy, Reverse Proxy, and API Gateway Explained

People mix these up all the time, since they all sit between a client and a server. The real difference is which side they represent and what problem they solve.

A forward proxy sits next to the client. Your laptop sends a request, the proxy forwards it out, and the destination never sees your real IP. Corporate networks use this to enforce policy, block sites, and cache traffic.

A reverse proxy sits next to the server. The client has no idea how many machines are behind it. The proxy decides who handles the request, terminates TLS, and keeps your backend off the public internet. NGINX and HAProxy are commonly used here, typically paired with a load balancer in front.

An API gateway is a reverse proxy that does more than route traffic. It also handles auth, rate limits, API keys, versioning, and request shaping. Without it, each microservice has to implement its own version of validation, throttling logic, and request logging.

A forward proxy represents the client, a reverse proxy represents the server, and an API gateway is what you add when ten services need the same authentication and rate limiting rules applied consistently.

In most real systems, all three are running at different layers. The forward proxy filters outbound traffic, the reverse proxy fronts the application servers, and the API gateway sits in front of your APIs to enforce policies before requests reach them.

Over to you: What's your proxy + gateway combo? Always interesting to see what teams pair together.

The code that performs this merging can run in several places. It can run inside the mobile application, on a server in the datacenter, at a CDN edge location, or inside one of the four services. Putting a server between the mobile application and the four services adds a network hop, which sounds like it should cost extra time. However, it usually reduces total load time instead, because a round trip between a phone and a server on a weak mobile connection can take a few hundred milliseconds, while a round trip between two services inside the same datacenter takes a fraction of a millisecond. To put it simply, trading four expensive round trips for one expensive round trip plus four cheap ones is often a large net saving.

However, latency is only the first tradeoff. The place where the merge operation runs also determines what happens when one of the four services is unavailable, how much of the response can be cached, and which team has to approve a change before the screen ships.

In this article, we are going to dive deep into the area of the API composition problem and the patterns associated with it. Here’s what we will cover:

• The API composition problem

• Client-side composition

• Over-fetching and under-fetching

• Composition, aggregation, and orchestration

• API gateways

• Backends for Frontends

• GraphQL as a composition layer

• Edge composition

• Availability and caching tradeoffs

• Versioning across multiple frontends

• Ownership of the composition layer

The API Composition Problem

Read more

GLM-5.2 is Z.ai’s flagship long-horizon coding model, featuring a usable 1M-token context window that reliably maintains API contracts and engineering intent where others fail. Stop managing GPU clusters. With Crusoe Serverless Fine-Tuning, you customize GLM-5.2 on your proprietary data in a tenant-isolated environment—no data sharing, no infrastructure overhead. When your job finishes, deploy to Self-Serve Deployments in one click or download raw .safetensors weights for full portability. Every run is reproducible, auditable, and fully yours. No lock-in. No guesswork. Experience production-grade, customized performance without the DevOps burden.

Get started

AI models have solved the writing code part of software development to a great extent. Today, a capable model can produce a working function, a full component, or a small application from a plain-language description. It can do so in seconds for a fraction of the cost.

This change has shifted the economics of every developer platform. As the generation of new code becomes cheaper and widely available, it stops being the differentiating factor for a platform. This is the reason GitHub, Vercel, and Replit are trying to rebuild themselves around solving other hard problems in the software development process.

To understand what the three companies are doing, we trace one unit of work through every platform while asking the same questions:

• Where does the AI actually run the code it writes?

• How does each platform verify that the code works?

• How does the finished product reach production, and who is permitted to ship it?

Here’s what we will cover:

• Why does cheap code generation move the hard engineering problem to another place, and how does that impact the platform’s overall value?

• GitHub’s orchestration bet, built on ephemeral cloud environments and a control layer that routes work across competing agents.

• Vercel’s production bet, built on isolated microVM sandboxes and a billing model matched to how agents actually run

• Replit’s verification angle, built on a self-testing loop that drives a real browser to catch code that only looks like it works

• How the MCP standard lets any agent reach any tool, and why all three companies now support it

• How Stripe turned payments into something an agent can set up from inside a coding tool, and the credential design that keeps it safe

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

Commoditization

A language model can now turn a description into working code. This means a developer can write a sentence and receive a function, a page, or a small application that actually runs right out of the gate.

This capability used to be the scarce and valuable part of a developer tool. Today it is widely available and close to free, which leaves a platform that offers only generation with little to charge for.

So what matters now is not raw code generation, but aspects of software development that come up after the code is available. Three questions carry most of the weight now:

• Where does the agent run the code it produces, and how is that environment kept safe?

• How does the platform confirm that the generated code actually works?

• How does the result reach production, and who is allowed to ship it?

GitHub, Vercel, and Replit answer these questions in different ways.

• GitHub puts its effort into coordination

• Vercel into the path to production

• Replit into verification

GitHub gives us the clearest place to start, because it deals with the pull request workflow most developers already use.

[Webinar] Can you prove AI is working? (Sponsored)

AI is in your engineering workflow. While the token spend shows it, the throughput doesn’t. The human is very much still in the loop, and that’s a context problem.

Join live on Aug 19 (FREE) to learn:

• The 4 metrics to measure where AI gains leak out before production.

• The 8 stages of context maturity, the specific walls capping your metrics, and a free tool to pinpoint where your team is

• Why more MCPs and bigger context windows aren’t enough, and what it takes to get real value from your agents.

Register now

Orchestration

GitHub made a specific choice about where the value sits.

Rather than building its own model and competing on raw generation, it built a control layer that coordinates many agents and keeps their work governed, all inside the pull request workflow that developers use every day.

The mechanics start with where the code runs.

GitHub’s coding agent operates in its own ephemeral development environment, which is a temporary workspace that exists only for that task. It is powered by GitHub Actions, the same automation system that runs tests and builds on the platform [1][2]. In practice, you assign a task to the agent the way you would open a ticket. The agent reads through the repository, edits files, runs the tests and linters (tools that check code for problems), and opens a draft pull request for a person to review [1]. Each cloud run happens in a fully isolated, single-use Linux environment hosted by GitHub. Every new task starts from a clean workspace [4]. The review step stays human, which keeps the existing quality gate intact.

See the diagram below:

A single agent sits above the coordination layer. GitHub calls it Agent HQ. It introduces a mission control view that lets a developer assign, steer, and approve work across a fleet of agents from GitHub and VS Code [3]. The agents available inside a paid Copilot subscription include ones from Anthropic, OpenAI, Google, Cognition, and xAI [3].

Governance is treated as version-controlled configuration. Teams define custom agents through AGENTS.md files that carry rules such as a preferred logger or a required testing style, and a control plane gives administrators security policies, audit logging, and model-access controls in one place [3].

See the diagram below:

For developers who are already accustomed to handling issues and pull requests, this design adds agents to a pretty familiar workflow. Reviewing a colleague’s branch is not so different from reviewing an agent’s branch.

Routing to other companies’ models is a deliberate decision. GitHub is positioning the workflow, the execution environment, and the governance layer as the durable product, and treating the underlying model as a swappable component.

Let us now look at how Vercel developed its architecture around what happens after the code exists. This is somewhere between a working demo and production-ready software.

Production

Vercel starts from a different premise where code generation is assumed, and the design deals with carrying the generated code into production. This is the stage where most enterprise software work actually happens, because it involves existing applications rather than fresh prototypes.

The rebuilt version of v0, Vercel’s generation product, runs on a sandbox. It is an isolated space for executing code that imports a real GitHub repository and automatically pulls in the project’s environment variables and configuration. Every prompt produces code that fits the actual application and lives in the repository itself [5].

A Git panel handles the workflow around it. You create a branch for each chat, open a pull request against the main branch, and deploy when it merges. This means a product manager or a designer can ship through the same review process an engineer uses [5].

Vercel’s rationale about this approach is that AI-assisted building is already happening inside companies, and it has produced real failures. Incidents have been reported, such as credentials pasted into prompts, private data reaching the public internet, and deleted databases, often with the audit trail left empty [5]. Therefore, wrapping code generation in real deployment controls is the right response.

See the diagram below:

Underneath the workflow is the execution layer. Every sandbox runs inside a Firecracker microVM, a lightweight virtual machine that isolates untrusted code. The reason for this isolation is that the code an AI wrote is code you have yet to review. Therefore, running it needs a boundary strong enough to contain mistakes. A microVM provides that strong boundary.

The billing model depends on how agents actually run. Vercel’s Fluid compute lets several requests share one running instance, with one using the processor while another waits on input or output. It charges for active processor time while treating wait time as free [6][7]. Agentic workloads spend much of their time waiting on a model to respond, so this pricing matches the real work being done.

This approach deals with a frustration many developers face in their careers. The thing that worked in a demo behaves differently in production. However, Vercel’s design tries to close that gap by making the preview a real deployment from the start.

Nevertheless, strong isolation carries a cost per unit of compute. There is an open question about whether the heaviest workloads eventually move to cheaper execution options elsewhere. Vercel’s wager is that a single, convenient platform keeps them in place.

To summarize, GitHub owns the workflow, and Vercel owns the route to production. Both of them still depend on one assumption that Replit chose to attack directly. Let us look at that now.

Verification

Replit concentrated its work on whether autonomously generated code genuinely works using a verification loop built into the agent itself.

Replit’s Agent 3 runs what the company calls a reflection loop. The agent generates code, runs it, tests the result, and repairs failures, repeating that cycle until the tests pass [8]. This loop is reliable because of how the testing is done. Replit built a REPL-based verification system that runs code immediately and pairs that execution with a real browser it drives automatically, so it can click buttons, submit forms, and check data the way a user would [9].

The specific problem this approach targets has a memorable name inside Replit: the Potemkin interface. It is basically a feature that looks complete on screen yet fails the moment it is used [9]. Catching that class of error is what allows the agent to run on its own for more than 200 minutes at a stretch, a large increase over the roughly 20 minutes of its predecessor [8][9].

The verification runs as its own process.

A testing subagent follows a simple cycle of taking an action, observing the result, and repeating. When it finishes, it returns a summary to the main agent describing what works and what broke [9]. This multi-hundred-step testing costs a median of roughly twenty cents per session and runs several times faster and more cheaply than relying on general-purpose computer-use models [9].

See the diagram below:

Anyone who has shipped a feature that looked finished and later broke understands the problem here. Replit is trying to automate the check that catches exactly that, so that the automation can be trusted.

A verification loop reduces the risk substantially. Some failures still surface only under conditions a single test session might miss, which keeps this a pretty hard problem even with a capable tester in place.

Interoperability

Every architecture we have looked at assumes its agent can reach tools and data that live outside the model, and doing that cleanly requires a common method. That method is the Model Context Protocol, usually shortened to MCP.

Before a standard like this existed, connecting several AI applications to several external tools meant writing a separate custom integration for each pairing. In this approach, the number of integrations grew quickly as both sides multiplied. Anthropic introduced MCP to replace those fragmented, one-off connections with a single protocol, so each application and each tool implements the standard once and then works with everything else without additional changes [10].

A host, which is the AI application such as an IDE or a chat client, creates one or more clients, and each client connects to a server that exposes some capability [11]. A server offers three kinds of capability:

• Tools, which the model can call to take an action, such as creating a record or running a query.

• Resources, which supply context data the model can read, such as a file or a database schema.

• Prompts, which provide reusable instruction templates.

The whole exchange runs over a defined message format across either a local or a remote connection. The effect is that a tool provider builds one MCP server and every compliant agent can use it [11].

Replit was among the earliest developer tools to integrate MCP [10], GitHub added an MCP registry to VS Code where a server can be enabled with a single click [3], and Stripe runs an official MCP server for its payment operations [12].

Going deeper, MCP also standardizes how agents reach existing APIs rather than replacing those APIs. A single shared entry point also concentrates security in one place. That makes careful control over which servers an agent may use an important part of any real deployment.

Tradeoffs

Each of these architectures and approaches has trade-offs:

• GitHub gains breadth and governance by routing to many vendors’ models. But the cost is that it owns the surface rather than the intelligence underneath. Whether a coordination and governance layer stays valuable as models and agents keep changing is an open question.

• Vercel gains strong isolation by running generated code inside microVMs, and that isolation carries a cost per unit of compute. There is a question about whether the heaviest workloads eventually move to cheaper execution elsewhere

• Replit gains long stretches of autonomy through its verification loop, and the more work an agent does on its own, the more weight rests on that verification being right. The Potemkin problem (code that looks complete yet fails when used) stays difficult even with a capable tester, because some failures appear only in situations a test session might miss.

MCP gains a clean, reusable way to connect agents and tools, and a shared standard also concentrates risk into a common entry point. When many agents reach many tools through one protocol, controlling which servers an agent may use, and their permission levels, becomes a central problem rather than a detail.

Understanding these costs is what separates picking one of these tools from understanding why it was built the way it was.

Conclusion

The pattern across all three companies is the same. Code generation became cheap, so the value moved into the engineering that surrounds it. Each company placed its bet on a different piece of that surrounding work.

GitHub bet on orchestration, building a control layer that runs and governs many agents inside the pull request workflow developers already use.

Vercel bet on production, wrapping generated code in real deployments and running it inside isolated microVMs built for untrusted code.

Lastly, Replit bet on verification, driving a real browser in a self-testing loop so an agent can work on its own for hours and still be checked.

Underneath all three, MCP provides the common protocol that lets any agent reach any tool, which is why every one of these platforms now supports it.

References:

1. About GitHub Copilot cloud agent

2. GitHub Copilot coding agent 101: Getting started with agentic workflows on GitHub

3. Introducing Agent HQ: Any agent, any way you work

4. GitHub Copilot app: The agent-native desktop experience

5. Introducing the new v0

6. The AI Cloud: A unified platform for AI workloads

7. Vercel Sandbox

8. Introducing Agent 3: Our Most Autonomous Agent Yet

9. Enabling Agent 3 to Self-Test at Scale with REPL-Based Verification

10. Introducing the Model Context Protocol

11. Architecture overview — Model Context Protocol

12. Agents and AI on Stripe

13. Create claimable sandboxes

Free masterclass: Learn practical strategies for predictable low-latency writes at scale

Free Masterclass: Optimizing Write-Intensive Database Performance

Database writes – at scale – are one of the hardest problems in distributed systems. This masterclass will teach you how to understand and avoid latency spikes in real-time, write-heavy database workloads. Our panel of experts will share a practical framework for diagnosing write bottlenecks and knowing which strategies to apply in different scenarios.

After this free 2-hour masterclass designed for developers, engineers, architects, and database practitioners, you will know how to:

• Identify which factors (database internals, database configurations, data modeling…) are impacting your write performance

• Avoid mistakes that have caused 40x write amplification in production

• Sustain low P99 latencies even during sustained growth and volatile spikes

All attendees will get the complete Database Performance at Scale book by the masterclass instructor Felipe Mendes.

Register for Free

How does a website charge a visitor who arrives anonymously, skips every advertisement, and leaves within a second?

For most of the web’s history, the question rarely came up, because the visitor was a person whose attention a site could sell through an ad or a subscription. If the website had good content, the website owner could count on multiple such visits by the same user. But now the visitor being a person is not always true. More than half of the traffic online now comes from software that acts on a person’s behalf, requests a page, and leaves without engaging with any ads or subscriptions [2].

Cloudflare seeks to change this.

As you might be aware, Cloudflare sits between a large share of the world’s websites and everything requesting them. It works as a reverse proxy that each request passes through before it reaches the origin server [1]. This lets Cloudflare read a request and act on it early. Over the past year, the company has used this position to sort automated traffic by what it does, to verify the identity behind a request, and, most recently, to collect payment for a request through an open protocol named x402 [1].

In this article, we will go through Cloudflare’s solution in the following five steps:

• The web’s usual way of earning value leans on human attention, and agent traffic changes it fundamentally.

• A look at Cloudflare’s initial solution around blocking automated traffic and then charging for each crawl.

• Settling identity, permission, and payment inside a single request.

• A look at how x402 exchange completes this settlement.

• The x402 exchange completes this settlement through a short back-and-forth over ordinary HTTP

• The costs and open questions of this approach

Disclaimer: This post is based on publicly shared details from Cloudflare. References at the end. Please comment if you notice any inaccuracies.

The Attention Model

For most of its history, websites made money after a request rather than during it. A browser asked for a page, and the server returned it at no charge. The value arrived later, once a person saw an advertisement, bought a subscription, or came back for another visit [1]. The request itself stayed free, and this setup funded a large part of the Internet.

A growth in agent traffic is changing this dramatically. An agent is software that acts on a person’s behalf, which in practice means it requests a page or a data feed once, takes what it needs, and finishes in a single pass. It moves past advertisements, operates outside any subscription, and completes its task before a site has a chance to earn from that attention [1]. Each of the three old settlement points depended on a person staying long enough to be counted, so software traffic leaves those points idle.

The strain here is that this kind of traffic is now fast becoming the majority. More than half of the requests reaching websites come from software rather than people [2]. This means that request volume climbs while revenue stays flat.

To make things clear, parts of the Internet were already charging by usage before any of this. Cloud services and APIs have been sold by the call and by the hour for years, though only to a buyer the seller already knew, who signed up and received an API key [1]. Charging an anonymous caller a fraction of a cent for a single request stayed impractical, because collecting such a small payment once cost more than the payment returned [1].

If the value used to settle downstream of the request, the natural question is which party is positioned to move that settlement back onto the request. The answer starts with where Cloudflare sits.

The Proxy Layer

A reverse proxy is a server that stands in front of other servers and receives requests on their behalf. Since Cloudflare operates as a reverse proxy for a large portion of the web, a request headed for one of those sites reaches Cloudflare’s network first and passes through it on the way to the origin [1]. The origin is the site’s own server, the machine that ultimately holds the page or runs the API.

On a side note, the position of a proxy is more general-purpose than caching. Caching stores a copy of a response so it can be served quickly the next time, and it is just one of the many useful things a proxy can do. From the same middle position, a request can also be read, classified, checked, and acted on before it continues to the origin. This broader capability is quite consequential at Cloudflare’s scale. [2].

However, acting on a request early depends on first knowing what the request is for. This is a classification problem. Let us see how Cloudflare handles this.

Traffic Classification

Before any rule can apply to an agent, the traffic has to be sorted by what it does. Cloudflare’s taxonomy groups automated traffic by behavior rather than by the single label “AI”. Three behaviors impact the key policy decisions [4].

• Search covers behavior that builds an index of a site so an engine can answer questions about it later. This behavior is responsible for sending referral visitors back.

• An agent covers behavior that acts in real time on a person’s behalf, usually with a human waiting for the result. Think of it like an assistant fetching a page during a conversation.

• Training covers behavior that takes content to train or fine-tune a model, where the content is absorbed into the model rather than pointing a visitor back.

These three appear identical in a raw request log, yet they carry very different consequences for a site’s business.

A single crawler can also perform more than one of these behaviors, but this separation lets them record all of them[4]. A crawler that both builds a search index and gathers training data is tracked as doing both, which lets a site owner reason about the full set of things that the crawler does on their pages.

Cloudflare classifies further behaviors as well, including checkout actions and data collection, and it also lets a site express what a bot may store and reshare after accessing a page [4].

With a way to see the proxy position and a way to classify what arrives, let us now look at Cloudflare’s first attempts at implementing a policy.

Blocking and Charging

Cloudflare’s first solution was a simple one. A single control lets a site block automated AI traffic outright. This protected the content and left the earning model untouched. Blocking answers the question of access, yet it does not rebuild the revenue that the old bargain provided.

The second solution added a middle path.

Pay Per Crawl lets a website allow a crawler, block it, or charge it a flat per-request price, with Cloudflare handling the billing as the merchant of record [5]. A site owner could keep a crawler out, let it through, or attach a price to its access, all from one setting. This turned the binary of block-or-allow into three options and gave content owners a way to earn from crawler access at network scale [5].

A year on, Cloudflare made an adjustment to its own model. The company argued that a crawl is a weak measure of value, because a single page might be crawled once and then cited in thousands of AI answers, or crawled repeatedly and cited in none [3]. It backed the argument with a figure from its own network, that more than half of the crawl traffic from well-behaved bots goes to re-fetching pages that have stayed the same since the last visit [3]. Counting crawls, then, counts something that only loosely tracks the value delivered.

So the unit of payment began to move from the crawl toward the use, an approach Cloudflare describes as Pay Per Use. It is candidly framed as an experiment at this point [3]. This is because while pricing the outcome aligns payment with value more closely than pricing the fetch, it is also harder to measure.

The Request Layer

The design goal for recent changes is to resolve identity, permission, and payment inside a single request, at the edge, before the origin responds.

Three main concerns are considered over here:

• Identity: The traditional identifier, the User-Agent string, can be set to any value by the caller, so it offers weak assurance. Cloudflare’s answer is Web Bot Auth, an authentication method that uses cryptographic signatures in HTTP messages to verify that a request comes from a particular automated source [7]. In practice, the operator signs its request with a private key and publishes the matching public key at a known location, and Cloudflare validates the signature at the edge [8]. A valid signature stands in for a reliable identity, which replaces a guess with a verifiable claim.

• Permission: This is expressed through the behavior classification already covered and through the preferences a site sets about how its content may be used [4].

• Payment: This comes last and is attached to the request through x402. We will cover this in detail in the next section.

All three concerns resolve at the edge, so the origin receives a request only once identity, permission, and payment have been settled [1]. The metering and settlement are taken away from the website’s own servers. What stays with the site owner is the part that matters to them, which is their rules, their prices, and their revenue [1].

To summarize:

• Identity answers who is making the request, through a signed and verified claim.

• Permission answers whether this behavior is allowed on these pages, through the classification and content preferences.

• Payment answers whether the caller has paid the stated price through the x402 exchange.

These pieces sit at different stages of maturity. Identity verification through Web Bot Auth is available today at the edge [7], while the Monetization Gateway that brings the payment piece together is open as a waitlist rather than a shipped product [1].

The key takeaway is that when one component sits in the middle of a flow, the concerns that are common across every request are collected at that point. Authentication, authorization, and billing consolidating at a gateway is the same pattern that appears in service meshes and middleware.

The x402 Exchange

The x402 protocol makes it possible to pay over HTTP. It takes its name from a status code that has been part of the HTTP standard for a long time [1]. The code is 402, and it means Payment Required. Sites behind Cloudflare already send more than a billion of these responses on an average day, which shows how often a machine requests something priced and receives a message that a payment is due [6].

See the diagram below that shows the overall setup:

The exchange runs through a short sequence that can be compared to a small state machine [1].

• A client requests a resource that sits behind a price.

• Rather than returning the resource, the server responds with 402 and a small payload stating the price, the accepted asset, and where to pay.

• The client re-sends the same request with proof of payment attached.

• A facilitator verifies the payment, and the server returns the resource.

Two properties make this suitable for machine traffic. The payment amounts can be very small because the protocol adds almost no overhead to the request. And the payment itself serves as the credential, so a buyer with no prior relationship can access the content by showing the proof of payment [1]. This property is the one that matches an anonymous agent passing through once, since it removes the signup step that per-seat licensing and API keys always required.

The negotiation process is handled inside ordinary requests and responses, with a redirect to a checkout page absent and a separate payment API absent [1]. Nothing was added at the protocol level, since the 402 code has been part of HTTP for decades. What changed is that a settlement angle now exists that makes collecting a fraction of a cent practical.

Costs and Limits

Resolving identity, permission, and payment at the edge means one proxy performs those functions for a large share of the web at once [1]. Cloudflare presents that responsibility as an advantage for settling everything inside one request. But it can also be seen as a potential risk when so much runs through a single provider.

Cloudflare also states several limits plainly:

• Trust that travels with a request may reach only the traffic that can afford to be identifiable, and small or privacy-sensitive sources of traffic need other building blocks, such as private rate limiting [4].

• Usage-based payment helps a site that already has demand, and does little for a small site whose real difficulty is discoverability rather than monetization, which leaves that site weighing visibility against giving content away [4].

• Collecting a payment through this exchange depends on callers built to recognize and honor the 402 response. Therefore, the revenue depends on adoption within the ecosystem. [6].

• Pricing an outcome rather than a crawl aligns payment with value, but it is also harder to measure and verify. This is the reason Cloudflare frames the shift to Pay-Per-Use as an experiment [3].

Lastly, Cloudflare argues that crawlers combining several purposes under one identity reduce transparency for a site. This is because the site cannot tell why it is being accessed [4]. While the argument is sound based on technical merits, it also aligns with Cloudflare’s commercial interest in separated, verifiable traffic.

None of these costs undoes the shift. They simply mark the current trade-offs that website owners should consider before adopting it.

Conclusion

The web is moving value settlement from after the request to inside it. For most of the web’s history, a request was served free, and value was settled later through human attention. However, agent traffic, which now comprises the majority of requests, leaves that later settlement with nowhere to land [2].

Cloudflare’s response is to resolve four things from its position as a reverse proxy: seeing what a request is through classification, verifying who sent it through Web Bot Auth, enforcing the site’s rules, and settling payment through the x402 exchange, all before the origin responds [1][4][7].

Identity verification lives at the edge today, and the payment gateway opens as a waitlist [1][7]. The open questions around concentration, reach, adoption, and how to price an outcome are worth tracking as the model develops.

References:

1. Announcing the Monetization Gateway: charge for any resource behind Cloudflare via x402

2. Content Independence Day, one year on: building the business model for the agentic Internet

3. Making AI search smarter

4. Your site, your rules: new AI traffic options for all customers

5. Introducing pay per crawl: Enabling content owners to charge AI crawlers for access

6. Launching the x402 Foundation with Coinbase, and support for x402 transactions

7. Web Bot Auth

8. Forget IPs: using cryptography to verify bot and agent traffic

Agents are hitting your signup flow and bouncing off a browser login built for humans. Every one that gives up is a signup you never see.

WorkOS Agent Registration turns that traffic into signups. Enroll via the dashboard and AuthKit publishes an auth.md file agents read to register for scoped, short-lived credentials you control.

Make your app agent-ready →

What does it take for a social media platform to stop rewarding clickbait content?

At first glance, this question might sound like a moderation problem that you can simply solve by enforcing better content policies and classifying posts that are engagement bait. However, the problem is much deeper. It often sits inside the component that decides which posts become part of a user’s feed in the first place.

Consider the scale of the decision. When you open a feed, the platform has a few hundred milliseconds to select a handful of posts from hundreds of millions of candidates. Scoring every candidate with an expensive model burns through the time budget. To get over this, the social media platforms rely on engagement, which is a cheap and somewhat reliable proxy for judging relevance. Such a proxy is easy to measure and optimize against, and it powered a generation of recommendation systems.

However, the problem is that this type of engagement is also easy to manufacture. For example, a post that opens with “comment DONE if you’re a real engineer” collects clicks and replies while delivering little value. But a ranking function tuned to reward interaction will promote it even though it is clearly an engagement bait. For years, the countermeasures against this were heuristics and demotions applied after ranking, but such an approach hasn’t eliminated the problem. Accounts producing bait always find ways around such measures.

Over the past two years, LinkedIn, Meta, and YouTube, three of the largest platforms, have tried to address the root of the problem. All of them have attempted to rebuild the retrieval stage around the meaning of content, matching posts to people by what a post is about and how it relates to a reader’s interests. The idea is that once the relevance depends on semantic meaning, the tactics built for engagement farming lose their potency. However, the three companies have taken three different directions to solve the same problem, which provides us with an opportunity to understand things from multiple perspectives.

In this article, we will work through the following points:

• Why feeds relied on engagement signals for so long, and where that approach reached its limits.

• How embeddings match users and content by meaning.

• LinkedIn’s consolidation of five retrieval systems into a single language-model retriever.

• Meta’s opposite choice of keeping a large family of specialized models arranged as a funnel.

• YouTube’s generative approach, where the system produces the identifier of the next item.

• The cold-start problem, and why pretraining helps most when a user’s history is thin.

• The tradeoffs each design carries, and the limits of the engagement-bait result.

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

Engagement Signals

Every large feed runs on a two-step pipeline:

• Retrieval: This step reduces hundreds of millions of candidates down to roughly a thousand. It has to be cheap because it touches the entire corpus.

• Ranking: This step spends real compute ordering those survivors into the sequence you scroll.

Most of the recent architectural changes sit in retrieval, so that is where we will focus more.

For years, retrieval relied on behavioral signals. The system recorded which posts a user clicked, watched, and reacted to, then retrieved content that resembled those interactions or resembled the behavior of similar users. This approach scales well and produces reasonable feeds.

However, it also has a weakness.

When a system optimizes a single measurable objective, it tends to optimize the literal metric rather than the intent behind it. As mentioned, engagement is a proxy for relevance, and a proxy can be optimized directly. A retrieval stage tuned on interaction counts will surface whatever maximizes interaction counts, and engagement bait is the content that does exactly that. For example, content designed to trigger a click or a reply scores highly on the measured signal while contributing little to the experience the platform aims to deliver.

Suppressing such content through demotions and rules tries to treat the symptom without fixing the underlying disease. The retrieval stage still keeps bringing up the same material. A more durable fix changes what retrieval measures in the first place, and this happens by moving from behavior to meaning.

Semantic Retrieval

The alternative is to retrieve content by its semantic content rather than by its interaction history. The ability to make this possibility depends on embeddings.

An embedding is a list of numbers that positions an item as a point in a high-dimensional space, arranged so that related items land near one another. For example, a post about fixing a leaking tap and a post about reducing water waste sit close together even when they share no keywords, because their meanings are related. In more technical terms, the embedding reflects the relationship captured during model training.

To use embeddings for retrieval, platforms apply a dual-encoder design, sometimes called a two-tower model.

• One encoder converts a user, along with their profile and recent activity, into a point in the space.

• A second encoder converts each post into a point in the same space.

Since the two encoders operate independently, a platform can compute every post embedding in advance and store it in an index. At request time, it computes the user embedding and runs a nearest-neighbor search to find the closest posts, which keeps retrieval fast across an enormous corpus.

The value of this design comes from what the embeddings encode. A keyword system matches surface tokens, so it links “electrical engineering” to other posts containing those words. An embedding produced by a language model reflects associations present in its training data, which lets it link an “electrical engineer” to other concepts like grid optimization and renewable energy infrastructure, even when the exact terms differ.

This same pattern appears well beyond social feeds. Search ranking, retrieval-augmented generation, and product recommendation all use two-tower retrieval. The model here transfers to a wide range of systems a developer might build.

LinkedIn, Meta, and YouTube all adopt semantic retrieval, but they diverge in how they have built their respective solutions. Let us look at them in more detail.

Unified Retrieval

LinkedIn’s feed previously drew candidates from separate retrieval systems, each with its own index and its own optimization logic [1].

One source supplied a chronological view of network activity, another handled trending posts by geography, another ran collaborative filtering, and several more produced embedding-based candidates. The setup worked, but maintaining five parallel systems led to rising engineering costs. Also, the sources were optimized independently rather than toward a single coherent objective.

In March 2026, LinkedIn replaced those systems with a single retrieval model built on a fine-tuned version of Meta’s LLaMA-3 [1]. The model acts as a dual encoder, converting both members and posts into one shared embedding space. It serves the entire feed through nearest-neighbor search at sub-50-millisecond latency for its member base [1].

However, consolidating five systems into a single language model raised a practical problem.

A language model processes text, while a recommendation system runs on structured features such as view counts, engagement rates, work history, and post metadata. LinkedIn bridged this with a prompt library that converts structured fields into templated text sequences the model can process [1]. See the diagram below:

In this approach, a member becomes a passage describing their profile, skills, and an ordered sequence of recently engaged posts, and a post becomes a passage describing its author, text, and engagement statistics.

We can also take a more general idea from LinkedIn’s approach. When the team fed raw popularity counts directly into the prompts, the numbers had almost no correlation with the model’s relevance scores, because large integers entered the model as arbitrary tokens. Converting each count into a ranked bucket, expressed as a percentage that the model could process in context, raised the correlation sharply and improved retrieval accuracy by roughly fifteen percent [1].

The takeaway from this is that the model is often the part that works, and the surrounding representation of the data is where the real effort goes.

As we can see, consolidation is one answer to solve the retrieval question. However, Meta made a different choice.

Ranking Funnels

Meta arranges Instagram’s recommendation system as a multi-stage funnel, consisting of an ecosystem of more than a thousand models supporting it [3]. Candidates pass through a sequence of stages, and each stage applies a more expensive model to a smaller set of surviving candidates [2].

The funnel runs through four steps:

• Retrieval gathers candidates from many sources across the platform.

• Early-stage ranking uses a lightweight two-tower model to narrow that set.

• Late-stage ranking applies a heavier model to the remaining finalists.

• A final pass adjusts the result for diversity and integrity.

Where LinkedIn moved toward meaning by consolidating, Meta pursues it through specialization.

In Meta’s approach, the late-stage model predicts many possible user actions at once, and a value model combines those predictions into a single score [4]. That combination adds weight for positive actions, such as a likely save, and subtracts weight for predicted negative actions, such as a “See Fewer Posts Like This” tap [4].

The basic objective of this extends past raw engagement to include signals about content a user would prefer to avoid.

This design is the one many production recommenders resemble, which makes it a useful reference point. Many competing objectives, including engagement, diversity, integrity, and creator fairness, are easier to tune and audit as separate stages than as one model. Of course, the cost to this is operational complexity, which is the exact complexity LinkedIn set out to reduce.

To summarize, both companies looked at semantic retrieval and made opposite conclusions about how much to consolidate. However, YouTube, the third platform we are looking at, questioned whether retrieval needs to search a stored index at all.

Generative Retrieval

YouTube took a third route that removes the search index from retrieval altogether [5].

The system, called PLUM, assigns every video a Semantic ID, a short sequence of discrete codes derived from the video’s own content [5]. Videos with similar content receive similar codes, so the identifier carries information about the item rather than acting as a random label.

PLUM then adapts a pretrained language model (from the Gemini family) by adding these Semantic IDs to its vocabulary and continuing to train it on video metadata and user activity [5]. After this adaptation, the model performs retrieval as a generation task. Given a user’s recent history, it produces the Semantic IDs of the videos that the user is likely to watch next, decoding several candidate identifiers through beam search, and the system maps each identifier back to a real video [5].

See the diagram below:

This approach, however, introduces a failure mode that the index-based designs avoid, which is generating an identifier that maps to no video, and PLUM reports keeping that rate below five percent after fine-tuning [5].

The payoff from this solution appears in coverage. Measured against the previous production system, PLUM surfaced a far wider range of long-tail videos, and on YouTube Shorts, it raised panel click-through by 4.96 percent [5]. The architecture also inverts where the parameters live. The prior system stored most of its parameters in large embedding tables, while PLUM holds most of its parameters in the network itself [5].

Ultimately, the three designs converge on one payoff in the cold-start problem.

Cold Start

A recommendation system faces the cold-start problem whenever a user arrives with little or no history. Behavioral retrieval has little to work with in this situation, because it depends on past interactions to find similar content, so new users historically saw generic material until their activity accumulated enough signal.

Semantic retrieval changes this.

Since a language model carries associations from pretraining, it can infer plausible interests from a profile alone. A member who lists a role in electrical engineering can be matched to content on grid optimization and energy infrastructure before clicking anything, using associations the model acquired during training rather than signals from that member’s own activity [1].

See the diagram below:

However, this capability also comes with a limit.

An inference drawn from a sparse profile can be wrong, and it can compress a person into a stereotype built from whatever their profile most resembles. The strength and the weakness have the same source, which is that the model fills gaps with prior associations. LinkedIn’s reported results fit this picture. The overall lift from the new system was modest, while the gains concentrated among new and low-connection members, which is exactly the group a meaning-based system can serve when behavioral history is sparse [1].

Design Tradeoffs

Every one of these designs has an associated cost and tradeoffs.

The first is consolidation against specialization. LinkedIn’s single model is simpler to maintain and aligns retrieval with ranking, while Meta’s many-model funnel gives independent control over each objective and provides natural redundancy. A single model also raises a more difficult rollback question, because five specialized systems offered five independent places to intervene when one of them regressed on a case such as trending content.

The second tradeoff is related to cost. Language-model embeddings capture richer associations than the lightweight methods they replaced. However, they consume more compute to produce and serve. As LinkedIn’s popularity-count finding showed, the model is rarely the bottleneck, and the load moves to the data pipeline, the feature representation, and the serving path.

The third tradeoff sits between the generative and index-based approaches. Generative retrieval composes items from compact codes and removes the large embedding table, at the cost of a failure mode where the model can produce an identifier for a video that does not exist. Index-based retrieval avoids that failure mode, but requires storage and maintenance of the index.

Semantic retrieval reduces the leverage of bait rather than removing it, since a system can still be gamed by content engineered to match high-value topics. Also, these three designs are points of emphasis on shared scaffolding more than pure types. YouTube still ranks candidates in stages, LinkedIn’s retriever feeds a separate ranking model, and Meta continues to explore generative methods.

Conclusion

Social media feeds are transforming retrieval approaches from behavioral engagement to semantic meaning. This helps reduce the leverage of engagement bait as a property of the architecture.

However, the three largest platforms adopted the same underlying idea and built it in three ways.

• LinkedIn consolidated five retrieval systems into a single language-model dual encoder and searches one shared embedding space.

• Meta kept a large family of specialized models arranged as a staged funnel with a multi-objective value model.

• YouTube generates the identifier of the next item through an adapted language model and removes the retrieval index.

The reason these choices differ comes down to data.

A text-rich professional network, a multi-objective media platform, and a video service with an enormous item corpus each favor a different design, and each team optimized for the structure of their specific data.

References:

[1] Large Scale Retrieval for the LinkedIn Feed using Causal Language Models

[2] Scaling the Instagram Explore recommendations system

[3] Journey to 1000 models: Scaling Instagram’s recommendation system

[4] Powered by AI: Instagram’s Explore recommender system

[5] PLUM: Adapting Pre-trained Language Models for Industrial-scale Generative Recommendations

A write operation records a fact, such as a new order, a changed email address, or a deleted comment. On the other hand, a read operation answers a question, such as which orders were placed this week or what should appear on a profile page. A single database on modest hardware can handle both types of operations without much trouble, and a developer need not be bothered about which operation is more common in the application’s context.

However, high traffic can change things. Let’s say a page starts to load slowly. A fix is identified, which is creating an index on the column being filtered. Now, several months later, the same page is slow again under higher load. This time the fix is a cache in front of the query. A year after that, the database saturates during peak hours. This time the fix is a read replica with reporting traffic routed to it. In principle, each fix works, but each one is needed at a different time and for a different reason.

Now, let’s assume that a user updates a profile, reloads the page, but still sees the previous value. The bug does not reproduce locally, and it disappears on its own before anyone can investigate its root cause. However, this behavior can be a direct consequence of some of the other fixes. This is because each of the previous fixes placed a copy of some data somewhere other than its source. But the copy is not updated in sync with the source update. In other words, a seemingly simple fix on the read path can impact how things appear to work on the write path.

In this article, we will look at read path and write path operations and techniques in detail. Here’s what we will cover:

• Why fast reads and correct writes require opposing data structures

• Precomputation and duplication, the single operation underneath every read optimization

• Two different definitions of consistency, and the bugs caused by treating them as one

• Indexes, denormalization, caching, read replicas, materialized views, purpose-built read stores, fan-out on write versus read, and CQRS

• For each strategy, its sync mechanism, staleness window, and characteristic failure mode

• Write-heavy systems, where the ratio inverts and the decisions reverse with it.

Read/Write Asymmetry

Read more

AI is in your engineering workflow. While the token spend shows it, the throughput doesn’t. The human is very much still in the loop, and that’s a context problem.

Join live on Aug 19 (FREE) to learn:

• The 4 metrics to measure where AI gains leak out before production.

• The 8 stages of context maturity, the specific walls capping your metrics, and a free tool to pinpoint where your team is

• Why more MCPs and bigger context windows aren’t enough, and what it takes to get real value from your agents.

Register now

The most capable AI models are also the most expensive to run. They need specialized hardware, they consume large amounts of memory, and they add cost and delay to every request they handle.

These traits make them hard to deploy in places where resources are limited, such as a mobile device or a service that handles heavy traffic and needs fast and low-cost responses.

There is also a second fact that sounds backward at first. A small model can sometimes match or beat a much larger model on a specific task, even when the small model learned everything it knows from the larger one. On an intuitive level, a model trained on another model’s output would seem to inherit a ceiling rather than break through it, yet the results are different. The method that makes this work is called knowledge distillation, and it has become a standard part of how production AI systems get built.

In this article, we will walk through the idea from the ground up. The main points we will cover are as follows:

• What distillation is, and how it differs from compression.

• Why learning from a model’s output can beat learning from raw labels.

• The three main methods, and which one dominates?

• What distilled models achieve in practice.

• Where the method breaks down, and where it is heading next.

Distillation

Distillation trains a new, smaller model to copy the behavior of a larger one. The setup involves two models:

• The first is a large, capable model called the teacher.

• The second is a smaller model called the student, which is trained to reproduce the teacher’s outputs.

Once training finishes, the student runs on its own, and the teacher steps out of the picture.

A common assumption is that the student is the teacher in compressed form. The reality, however, is different.

Compression methods such as quantization and pruning start with one model and reduce its footprint by storing its numbers at lower precision or removing parts that contribute little to the result. The model stays the same model, smaller and lighter.

Distillation, on the other hand, produces a genuinely separate model, with its own parameters and often a different design, whose goal during training is to behave like the teacher.

One operation shrinks an existing model. The other trains a fresh one. The payoff is practical, since a small student can run inside a single service or on a phone, respond in less time, and cost far less for each request, and in some cases, it can run on the device itself without sending data elsewhere.

This method is now standard practice. For example, Google’s Gemma models are built using distillation during training, drawing on a larger model in the Gemini family. The two ideas also work together in sequence. A model is often distilled first to produce a smaller capable model, then quantized to shrink that model further for a specific device.

Keeping the distinction clear matters because it affects how we understand further concepts. A compressed model carries a copy of the original inside it. A distilled model is a separate thing that was trained to act like the original, which is exactly why it can sometimes behave in ways the original would not.

If the student only copies the teacher, why does copying work so well?

The answer is in what the teacher hands over.

Soft Labels

Learning from a model’s output beats learning from raw data because the output carries more information than a plain answer.

Standard training data gives one answer per example. An image of a cat carries the label “cat,” and the model is rewarded for producing “cat” and penalized for anything else. A teacher model offers something richer. Instead of a single answer, its output is a set of probabilities across the options, such as cat at 0.70, dog at 0.25, and fox at 0.05. That full set of probabilities is called a soft label, in contrast to the single hard label found in ordinary data.

The extra numbers carry additional information. They show that the teacher’s output ranks dog as a plausible alternative and fox as a distant one, which says something about how the categories relate to each other. Researchers sometimes call this dark knowledge, meaning the structure hidden in a model’s confidence that a bare label leaves out.

During training, the student works to match this distribution. It is scored on how far its own probabilities sit from the teacher’s, and training pushes it to close that gap. In other words, the student learns the teacher’s whole pattern of confidence rather than a single right answer, and that pattern is a stronger training signal than a one-word label.

This is the core reason distillation works as well as it does. A single correct label discards the relationships between options, and soft labels keep them.

An early result showed the practical payoff, since a student could reach good performance from far fewer examples when trained on soft targets, because each example now carried more than a single answer. The original 2015 work added a control called temperature for exactly this purpose, where a higher temperature spreads the probabilities out and exposes more of that fine structure for the student to learn.

With the mechanism clear, the next question is how this gets done in practice, which has more than one answer.

Methods

Distillation comes in three main forms, and they differ in what the student copies:

• Output distillation: The student matches the teacher’s final outputs, including the soft labels described above. This is the original form from 2015 and the most direct one.

• Feature distillation: The student matches the teacher’s internal representations, meaning the intermediate values a model computes while processing an input, before it settles on a final answer. The aim is a similar internal picture, not only a similar output. Google’s EmbeddingGemma is trained this way, learning to produce internal representations close to those of a larger Gemini model.

• Synthetic data distillation: The teacher generates a dataset of examples, and the student is fine-tuned on that dataset the same way it would be trained on any ordinary data. Stanford’s Alpaca was an early case, fine-tuned on examples produced by an existing large model to improve how well it followed instructions.

The third form has become the most common approach in practice, and part of the reason comes down to access.

Many strong models are reachable only through an interface that returns text, with their internal values and probabilities kept private. When those internals are out of reach, generating data is the route that still works.

The three forms also differ in what they require. Output distillation needs the teacher’s probabilities, feature distillation needs access to its internal values, and synthetic data distillation needs only the text the teacher produces, which is why it travels the furthest across closed models.

These methods can also be combined. A single training run might use a generated dataset alongside soft labels, and newer methods mix teacher and student generation during training.

These methods are not only theoretical. The next section shows what they produce.

Results

The results in practice are strong, with one important qualifier.

A clear example came in early 2025 from a lab called DeepSeek. It used a large reasoning model to generate a set of training examples, then fine-tuned several existing smaller models on those examples. One result stood out.

A 7-billion-parameter student scored higher than a 32-billion-parameter model on a competition mathematics benchmark, even though it was produced by plain fine-tuning on the larger model’s outputs. The released family of distilled models ran from 1.5 billion parameters up to 70 billion, and the smaller ones were compact enough to run on a single graphics card, which is part of why the release drew so much attention. The practical effect was that strong performance on these narrow tasks became something a small team could run locally and cheaply, rather than only through a large hosted model.

The qualifier matters as much as the headline.

These wins tend to appear on narrow, well-defined tasks such as mathematics and code. On those tasks, a small distilled model can perform at a level its size would not suggest. Across broader measures of general knowledge, the same small models still trail the larger ones. For example, a model can become excellent at competition mathematics through distillation while remaining weaker at wide-ranging questions about the world. Therefore, a claim that a small model beats a large one is usually true in a specific, narrow sense.

If the results are this good, the natural question is where the method falls short, which the next section takes on directly.

Limits

Distillation has clear limits, and they matter when deciding whether it fits a given problem.

• A ceiling effect from the teacher: A student trained on a teacher’s output tends to stay at or below the teacher’s level on the kind of data they saw. When the teacher produces a wrong answer, the student learns that wrong answer along with the right ones. The teacher’s quality sets the bar, which makes the choice of teacher one of the most consequential decisions in the process.

• A wider gap can hurt: A larger, stronger teacher does not always produce a better student. When the gap between teacher and student is very wide, transfer can degrade, because the student has too little capacity to absorb everything that a much larger model expresses. Research on this capacity gap has found that the strongest available teacher is sometimes a poor choice. A set of methods exists to bridge wide gaps by adding a middle step, where the teacher trains a mid-sized model and that model trains the small student, so each handoff spans a smaller distance.

• Architecture can outweigh size: The design of the base model can matter more than its parameter count. In one study, a 32-billion-parameter student outperformed a 70-billion-parameter student on the same task, because the smaller one was built on a stronger base architecture. Size alone is a weak predictor of how well distillation will go.

• The teacher can pass on more than the task: In a 2025 study later published in Nature, a teacher model with a particular trait, a tendency to favor owls, was used to generate training data made up only of number sequences. A student trained on those numbers picked up the same preference for owls, even after the data was filtered to remove any visible trace of the trait. The same effect appeared with more serious behaviors, and it occurred only when the teacher and student shared the same base model. The takeaway is that distillation can carry across more than the task being taught, and that filtering the visible data is sometimes too coarse to stop it.

These limits set the boundaries, and within them, the method keeps advancing. The next section covers where it is heading.

Automation

The newest direction in distillation reduces the manual effort by automating the whole process.

In this setup, the large model runs the full loop on its own. It generates training data, fine-tunes the student, evaluates the student against a held-out set of examples it also generates, and repeats the cycle, adjusting what it produces until the student stops improving. The human role shrinks to defining the task and the success criteria at the start, with a final check on real data at the end.

Recent work in 2026 applied this to a detection task and found that it worked well, with one finding worth keeping in mind.

The choice of teacher model had a large effect on the outcome. Different teachers, given the same loop and the same student, produced students of noticeably different quality. So automation removes manual effort while making the initial choice of teacher more consequential, since that choice now drives an entire self-running process rather than a single training pass.

The same loop also points toward less hand-built pipeline work over time, as more of the data generation and evaluation moves to the model itself. For a team, the appeal is building a small, task-specific model without assembling a large hand-labeled dataset first, since the teacher supplies both the training examples and the data used to score them.

Conclusion

Distillation is a method for training a small, deployable model to copy the behavior of a large, expensive one. It produces a separate model rather than a compressed version of the original, and that distinction explains most of how it behaves.

It works because a model’s output carries more information than a plain label, in the form of soft labels that show a full pattern of confidence across the options.

In practice, the most common form has the teacher generate a training set that the student learns from, and the results can be strong, though usually on narrow tasks.

The limits are real:

• The teacher sets a ceiling,

• A wider size gap can hurt rather than help,

• Architecture can outweigh size

• The process can carry across traits that were never intended.

Taken together, distillation tends to be a good fit when the task is well defined and a capable teacher is available, and a weaker fit when the goal is broad, open-ended capability.

References:

• Distilling the Knowledge in a Neural Network

• DeepSeek-R1

• Gemma 3 Technical Report

• EmbeddingGemma architecture and recipe

• Alpaca

• Improved Knowledge Distillation via Teacher Assistant

• Explainable Sentiment Analysis with DeepSeek-R1

• Language models transmit behavioural traits through hidden signals in data

• Anthropic summary.

• Agentic Knowledge Distillation

On-call shouldn’t feel like constant firefighting. This guide from Datadog breaks down how high-performing SRE teams reduce alert fatigue, streamline incident response, and design rotations that don’t burn engineers out.
You’ll learn how to:

• Cut alert noise by tying signals to real user impact

• Improve response with clear roles and smarter escalation paths

• Turn incidents into feedback loops that improve system reliability

Get the guide

Why does sending a model a 100K-word prompt cost so much more than sending it a short one, even when the model and the hardware stay the same?

A key part of the answer lies in a block of working memory called the KV cache.

This memory is built up while the model generates a response. It is separate from the knowledge stored in the model’s weights, and it holds the key and value vectors computed for every token of the input. The cache grows with every token, and in a long context, it can take up significant space on the GPU.

For example, for a 70-billion-parameter model at a context of 128,000 tokens, it comes to roughly 40 gigabytes, a serious amount of GPU memory that grows with every user you add.

The above chart raises an obvious question, which is why a cache exists at all and why it grows the way it does. In this article, we will learn how LLMs use memory, how it gets expensive, and how to fix it.

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

Recomputation

Let us start with the work a model does to produce one token.

To choose the next word, it runs an attention step, where the newest token compares itself against every token that came before it. This comparison uses two vectors for each earlier token, a key and a value, which are simply the numerical summaries the model computes for that token inside each layer. A model that rebuilt the key and value for every earlier token at every step would watch the work per token climb as the input grows. Such a repetition is pure waste, because those keys and values stay the same once a token has been processed.

The KV cache removes the waste by storing those key and value vectors the first time they are computed. On the next step, the model computes the key and value for only the new token and reads the rest straight from the cache.

Overall, this is a great solution. However, caching fixes the speed problem while creating a new one. The cache now has to be read on every step, and this turns out to be the real source of increasing cost.

One detail worth understanding here is that the cache holds vectors rather than the original text. It also explains out-of-memory errors that look puzzling when the model itself fits with room to spare.

Decoding

Generation of tokens runs in two phases, and they stress the hardware in different ways:

• The first phase is prefill, where the model reads the entire input at once. It processes all of the input tokens in parallel and builds their key and value vectors into the cache in a single pass. Prefill keeps the GPU’s math units busy, so we call it compute-bound, meaning the limit is how fast the chip can do arithmetic.

• The second phase is decoding, where the model produces the output one token at a time. Each new token runs an attention step against the whole cache, which means the model reads every stored key and value out of GPU memory before it can emit the next token. It repeats that read for every token it produces. The limit here is how fast the cache can move from memory into the compute units, so we call decoding memory-bound.

The expense of long-context generation comes less from holding the cache and more from sweeping through all of it on every single token.

A larger cache means more data crossing the memory bus per token, which shows up directly as slower and costlier generation. It also explains why a request can run slowly even when it fits in memory comfortably.

If the cost tracks how much of the cache we read each step, then the size of the cache is the next thing to understand clearly.

Scaling

The cache size is the product of a handful of numbers:

• A factor of two covers the key and the value.

• Layers count because each layer keeps its own cache.

• Key-value heads set how many sets each layer stores.

• Head dimension is the size of each of those vectors.

• Bytes per number is the space one stored value takes.

• Tokens are the context length, with one entry each.

• Batch size is the number of requests served at once.

To summarize, the cache size equals 2 times layers times key-value heads times head dimension times bytes per number times tokens times batch size.

Two things that should be noted here are as follows:

• The cache grows in a straight line with the token count, so doubling the context doubles the cache

• It grows the same way with the batch size, so serving more users at once scales it just as fast.

As an approximate example, a Llama 3 70B model has 80 layers, 8 key-value heads, a head dimension of 128, and stores each number in 2 bytes. At a context of 128,000 tokens for a single request, those numbers multiply out to roughly 40 gigabytes, which is why a single long request can fill most of an 80-gigabyte card on its own.

Let us now look at the optimization techniques that help an LLM manage the memory aspects. Each of these techniques tries to push against a specific number.

Attention

The first two techniques change how attention itself is built, which means a model is committed to them during training. Both shrink the footprint of each token in the cache.

Grouped-query attention goes after the key-value head count. In a standard attention layer, every query head carries its own key and value head, so a model with 64 query heads stores 64 sets of keys and values. Grouped-query attention lets several query heads share one key-value head, which drops the number of stored sets sharply. For example, Llama 2 and 3 at 70B and Mistral 7B share down to 8 key-value heads, which cuts the cache by roughly eightfold against full multi-head attention. This is why a recent 70B model can hold a smaller cache than an older 7B one.

There is a more aggressive version called multi-query attention, where every query head shares a single key-value head. It saves the most memory of any head-sharing scheme. However, pushed that far, quality tends to drop, and training grows unstable, so most setups settle on the grouped middle ground as the better trade.

The second attack keeps the heads and compresses what each one stores.

Multi-head latent attention, introduced in the DeepSeek models, projects the keys and values down into a smaller latent representation before caching them, then expands them back when they are read. The savings are large. DeepSeek-V3 holds around 70 kilobytes per token, where comparable grouped-query models sit between 192 and 328. The cost lies in serving, since the compression adds work on every read and pairs awkwardly with some standard attention implementations, so it tends to pay off most once models and contexts grow large enough that cache traffic dominates.

As mentioned, head-sharing and latent attention both require control over the architecture, so they help when we are choosing or training a model. The next attacks work on a model we already have in hand.

Quantization

Quantization goes after the bytes per number.

The keys and values are usually stored at 16 bits each, and quantization rounds them to a smaller format such as 8 bits or 4 bits. Since the bytes-per-number term sits right there in the equation, moving from 16 bits to 8 halves the whole cache, and going to 4 bits halves it again. The appeal is that this applies to a model we already have and skips retraining entirely.

The quality cost depends on how far we push it.

Eight-bit storage often costs well under a percent of accuracy, which puts it within the noise for most workloads. Four-bit storage saves more and starts to show measurable losses on demanding tasks such as multi-needle retrieval, where the model has to pull several specific facts out of a long context.

Specialized methods beat plain rounding because a few numbers in the cache deserve more precision than the rest, though plain rounding already captures most of the gain.

Eviction

Eviction goes after the token count by dropping entries that the model is unlikely to need.

The common approach keeps a window of the most recent tokens, since recent context usually matters most, along with a few tokens from the very start of the sequence. Those opening tokens turn out to play an outsized role. They absorb a large share of attention regardless of what they actually say, acting as anchors that keep the model’s output stable.

See the diagram below:

The trouble with eviction is structural.

Whether a token matters depends on a question that has yet to arrive. A token we drop now can be exactly the one a later part of the generation needs, and once it is gone, the model generates as though that token had been absent the whole time. This shows up on retrieval tasks, where an aggressively trimmed cache handles a casual chat well and then misses a fact buried in the middle of a long document.

More refined schemes score each token’s importance and try to predict which ones are safe to drop, which helps, though the core problem remains.

Serving

Even with the contents of the cache fixed, the way a serving system manages memory leaves a lot on the table, and two techniques help.

The first is paged attention.

Older serving systems reserved one large contiguous block per request, sized for the longest output it might produce. Most requests finished well short of that, leaving the reserved space idle, and the fragmentation added up. However, paged attention borrows an idea from operating systems, which break memory into small fixed-size pages and hand them out on demand. The cache gets split into small blocks that can live anywhere in memory, tracked by a lookup table that maps each request to its blocks. The result is that systems that wasted 60 to 80 percent of cache memory to fragmentation dropped that figure below 4 percent, and throughput climbed by two to three times, all from packing the same data more tightly.

The second technique results from the first. Since the cache lives in shareable blocks, two requests that begin with the same text can point at the same physical blocks while each holds its own private continuation. This is the foundation of prefix caching, and the productized version that the major APIs call prompt caching.

The win is large for any workload that repeats a prefix, such as an agent that sends the same multi-thousand-token system prompt on every call. OpenAI and Anthropic both report cost and latency reductions of 50 to 90 percent on cache hits, with cached tokens billed at a fraction of fresh ones.

However, one thing to note is that sharing cached state across users has opened timing side-channels that can leak information about other people’s prompts, an active concern we will leave aside here.

Tradeoffs

The techniques we’ve looked at look alike in how much memory they save and differ widely in what they ask for in return.

Some are close to free. For example:

• Grouped-query attention costs very little quality and has become the safe default, which is why nearly every current model ships with it.

• Paged attention and prefix caching barely touch quality, since they change how the cache is stored and shared rather than what it contains.

Other techniques require more. For example:

• Quantization is cheap at 8 bits and grows risky as we push toward 4 and below, so the right setting depends on how sensitive the task is.

• Latent attention saves most of the architectural options and asks for real engineering effort to serve well.

• Eviction frees a large amount of memory and can also lose information that the generation later needs, which makes it a genuine gamble rather than a clean win.

For a short context, the cache is small, and most of these techniques solve a problem that has yet to appear. They earn their place in a long context and high concurrency, where the cache grows into the dominant cost. The right mix follows the workload, with agent loops leaning on reuse and long-document retrieval leaning away from eviction.

Conclusion

The cost of long-context inference comes down to one cache, sized by one short equation.

Since decoding reads the whole cache on every token, the cache is a bandwidth cost as much as a storage one, which is why shrinking it speeds things up. Every optimization we covered deals with a specific aspect of the overall equation or trims the waste around it.

• Grouped query and latent attention reduce what each token costs.

• Quantization stores each number in fewer bits.

• Eviction keeps fewer tokens.

• Paged attention and prefix caching manage and share the cache more efficiently.

References:

• Meta’s Llama 3.1 announcement

• The Llama 3 Herd of Models

• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

• Fast Transformer Decoding: One Write-Head Is All You Need

• Mistral 7B

• DeepSeek-V2

• DeepSeek-V3 Technical Report

• vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

• Efficient Memory Management for Large Language Model Serving with PagedAttention

• SGLang: Efficient Execution of Structured Language Model Programs

• OpenAI prompt caching documentation

• Anthropic prompt caching

• Efficient Streaming Language Models with Attention Sinks

Try Matic, backed by their 6 month money back guarantee.

Get Matic

In June 2025, security researchers at Aim Security demonstrated that a single email could cause Microsoft 365 Copilot to retrieve a company’s internal files and transmit them to an external server, with no user interaction.

The email contained ordinary text and exploited no software flaw. Copilot’s output was driven by hidden instructions in the email together with the user’s actual request, and the system applied no distinction between the two. Microsoft closed the specific path within days and reported that customers were unaffected.

But this incident sheds interesting light on how language models process text. The vulnerability was named EchoLeak, assigned CVE-2025-32711, and it serves as a useful study to understand the landscape of LLM security.

For contrast, consider a different case. For roughly $20 in API queries, a team extracted part of a production OpenAI model through its public interface. That sounds like the threat most teams should really fear.

In this article, we try to build a map of the full attack surface that threatens an LLM’s security. With it, a given LLM feature can be located, its exposure points identified, and new threats reasoned about as they appear.

So let’s start with the single most important property.

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

Trust Boundaries

Almost every LLM vulnerability traces to one property. A language model receives instructions and data as a single sequence of tokens, and that sequence contains no marker separating commands from information.

Traditional software keeps the two apart.

For example, a parameterized database query holds the command in one position and the user’s input in another. This means that the text typed into a name field stays text even if it spells out a SQL statement, because the structure of the query enforces the separation.

However, an LLM context has one position for everything. The system prompt defining the assistant’s role, the user’s message, a document retrieved from a database, and the output returned by a called tool all arrive concatenated in the same sequence. Any portion of that sequence can affect the generated output as if it were an instruction, because the model computes each next token from the entire preceding sequence with no separate treatment for any part of it.

This is where prompt injection comes in, which means the supply of instruction-like text so that the model output conforms to those instructions instead of the operator’s intent. It reaches the model by two routes:

• The direct route is a hostile instruction typed into the chat box. This is the version most people picture.

• The indirect route places the instruction inside the content that the model retrieves during a legitimate task, such as a web page being summarized, a document being read, or an email in a managed inbox.

EchoLeak was an example of the indirect route. The user requested ordinary work, and the attacker’s instructions arrived in an email the user had not opened. In fact, the payload passed through Microsoft’s dedicated cross-prompt-injection classifier, which indicates that input filtering can be porous when used alone.

The property applies to any system that supplies external text to a model. A feature that processes retrieved search results, uploaded files, tickets, or comments carries indirect injection exposure by construction.

Parameterization solved SQL injection by separating code from data at the database boundary. No equivalent exists for natural language, because instructions and information are both expressed as words, and there is currently no reliable method to mark a span of text as inert and have that marking respected during generation. Filtering reduces the problem to some extent but does not eliminate it.

Since the confusion can occur wherever text enters the model or an action leaves it, the next step is to map those points.

Attack Surface

The OWASP Top 10 for Large Language Model Applications is the industry reference for the most critical LLM risks. Its recent edition presents ten separate items. Placed against the path that data takes through an application, those ten items become positions on a single map.

The pipeline runs in stages.

Input arrives from the user. The system frequently retrieves context to ground its answer, a step usually built on a vector database that stores documents as numerical embeddings and returns the ones most relevant to a query. That pattern is retrieval-augmented generation, or RAG. The model then processes the assembled input. It may call tools or other agents that take actions externally. Output returns to the user. Monitoring surrounds the pipeline, and every component within it originates from a supplier, which forms the supply chain beneath the system.

Here are some more details about each stage and the potential risks associated with them:

• Input: Direct prompt injection and unbounded consumption, which OWASP defines as uncontrolled resource use capable of running up a victim’s costs, sometimes called denial of wallet.

• Retrieval: Indirect injection, vector, and embedding weaknesses. PoisonedRAG, a 2024 study, corrupted a RAG system’s answers by inserting as few as five malicious passages into a knowledge base of millions, reaching a 90 percent success rate on targeted questions.

• Model: Training-data leakage, data and model poisoning, and system prompt leakage.

• Tools: Excessive agency, the condition of an agent holding more permission than its task requires.

• Output: Improper output handling, where the response passes downstream without sanitization, and misinformation, where a confident answer is incorrect.

• Supply chain: Compromise of any component feeding the stages above.

The map provides a reference point for figuring out the specifics of an attack. A new attack can be located on it. The relevant questions are where it injects untrusted text and at which stage, and whether it abuses a permission the model should not hold. The answers indicate both the severity and the applicable defense.

Supply chain cannot be cleanly placed on the path, because a compromised model or poisoned vector store affects every later stage at once. For that reason, it appears as its own segment that spans everything.

The map also exposes a mismatch in attention. The threats that generate the most concern and the threats that reach production are different.

Model Attacks

Attacks aimed at the model’s interior, including weight theft, training-data extraction, and training-time poisoning, are quite real, but for most developers, they rank low for initial effort. They tend to be expensive, narrow, or already mitigated by the model provider.

Here are a few examples:

• Model theft: As mentioned earlier in the OpenAI scenario about a team that recovered the final embedding-projection layer of production OpenAI models for under twenty dollars and confirmed previously secret hidden dimensions. The result is significant and also bounded. It recovers one layer among many, and the researchers stated that reconstructing a full frontier model through an API remains impractical, since the cost exceeds training an equivalent model. OpenAI received advance notice and modified its API.

• Training-data extraction: In late 2023, a group from Google DeepMind and several universities found that prompting ChatGPT to repeat a single word continuously could cause it to emit verbatim fragments of training data, including real contact details, with megabytes recoverable for a few hundred dollars. The privacy implications were serious, and OpenAI filtered the triggering behavior after disclosure.

• Poisoning: In 2025, a team from Anthropic, the UK AI Security Institute, and the Alan Turing Institute found that approximately 250 malicious documents were sufficient to install a backdoor in models from 600 million to 13 billion parameters, with the count remaining roughly constant across model sizes. That finding overturned the assumption that larger models require proportionally more poisoned data. The researchers also stated the limitation directly. The backdoor produced only gibberish output on a trigger phrase, which is a low-stakes behavior they described as unlikely to pose significant risk in frontier models.

The graph below shows the placement of various threats on a fear vs frequency scale:

The ranking matters because attention is finite. A team focused on model theft while deploying an agent with broad permissions has addressed a rare attack and left a common one open.

The bounded status describes the present. These attacks rise in priority for teams that host open model weights, fine-tune on sensitive data, or operate their own training pipelines, where the model’s interior becomes the team’s responsibility rather than a provider’s.

The interior attacks sit at the edge of the map. The center, where the model triggers external actions, carries the larger risk.

Excessive Agency

The point at which LLM attacks cause material damage has a specific structure and is identifiable in a system. It is also called the lethal trifecta. It consists of three capabilities held together by a single agent:

• Access to private data, such as an inbox, a customer database, or a source repository.

• Exposure to untrusted content, meaning anything read from outside, including web pages, emails, and shared documents.

• A channel to send data out or act externally, such as an outbound request, a sent message, or a tool call.

An agent holding all three can be directed by injected instructions to transfer private data to an attacker. Model alignment does not remove this exposure, because producing output that conforms to instruction-like input is how the model normally operates.

A few documented cases follow the pattern:

• GitHub’s MCP server, MCP being the Model Context Protocol that connects models to external tools and data, has been compromised using malicious issues filed on a public repository to expose data from a victim’s private repositories.

• GitLab’s Duo assistant has been supplied a public project containing hidden instructions and made to leak private repository contents.

• A Chevrolet dealership chatbot was once manipulated into agreeing to sell an SUV for one dollar.

• A crypto trading agent was socially engineered into transferring 55 ETH.

Removing any one of the three capabilities reduces the exposure. The least costly reduction is usually cutting the outbound channel or narrowing what the agent can access, which tends to be cheaper than adding a stronger filter.

Connecting tools through MCP is the most common way an agent acquires the third capability, and the protocol is recent enough that even established servers have shipped injectable configurations. Anthropic’s own official Git MCP server received three injection-related CVEs in 2025. Runtime input is one source of risk. The system’s components are another, and they can be compromised before any request arrives.

Supply Chain

Every model, adapter, vector store, and tool in a stack originates from a supplier, and any of them can arrive compromised. This is the supply chain surface, and it bypasses runtime defenses because the threat is present before input validation runs.

The common mechanism is straightforward.

Many models are distributed as serialized files, and some serialization formats execute code when the file loads. In early 2025, ReversingLabs documented a technique named nullifAI, in which malicious models uploaded to Hugging Face concealed a reverse shell, code that opens a connection back to an attacker, inside a Python pickle file. The file was compressed in a manner that evaded Picklescan, the platform’s scanner, so the model appeared clean and executed hostile code on load.

The scale demands attention. Protect AI, which scans models hosted on Hugging Face, has examined more than four million and flagged roughly 352,000 as carrying unsafe or suspicious issues, across more than fifty thousand models.

Provenance is one of the few fully controllable factors. The selection of which models, tools, and data sources to trust sits with the team, unlike most of the runtime surface.

Two mitigations are advancing:

• Safer serialization formats that avoid executing code on load.

• Model signing that verifies origin, comparable to signed releases in package ecosystems.

Defense in Depth

No single defense holds all the time, and the supporting evidence is strong enough that the realistic objective changes from preventing every attack to surviving the ones that succeed. The working posture is defense in depth, a set of independent layers arranged so that the failure of one is contained by another.

One layer is insufficient.

In November 2025, a team from OpenAI, Anthropic, and Google DeepMind published a study that took twelve previously proposed defenses against prompt injection and jailbreaking and defeated them, using attacks permitted to adapt and iterate. Strong production filters still allow a measurable fraction of attacks through, and a single success is enough. A lone guardrail provides confidence that the measurements do not support.

The more durable approach constrains the system around the model rather than relying on the model to resist manipulation.

Google DeepMind’s CaMeL is one example. It treats the model as untrusted, uses a separate privileged component to plan actions, and quarantines externally retrieved data so that data cannot trigger sensitive operations on its own.

Meta’s Agents Rule of Two is a simpler operational version, recommending that an agent satisfy at most two of three risky properties: processing untrusted input, holding sensitive access, and acting externally, without a human in the loop. Meta presents the rule as a supplement to least privilege rather than a complete solution.

The standard layers each cover a stage of the map:

• Input is validated and constrained.

• Retrieval sources are kept clean.

• Each tool is scoped to the minimum permission its task requires.

• Model output is treated as untrusted and sanitized before downstream use.

• Monitoring watches for anomalies.

• A human reviews the highest-consequence actions.

Conclusion

The threat model organizes around one fact.

A language model processes instructions and data as the same sequence of tokens, and the full map follows from that property.

The OWASP Top 10 becomes a set of positions on a path rather than a list. The widely publicized attacks, model theft, and training-data extraction are bounded and largely mitigated, while the larger risk concentrates wherever an agent holds private data, untrusted content, and an external channel at once. The supply chain underlies every stage and is the surface most directly controlled by the team operating the system. Since no single layer holds, defense in depth is the realistic posture to implement for a team.

The cost is high. Every layer adds latency, expense, and friction, and the strongest mitigation, human review of consequential actions, also limits how autonomously a system can operate. The trade-off is unavoidable.

The landscape reduces to a few durable points:

• The root cause is the absence of a boundary between instructions and data.

• Named threats are positions on the pipeline rather than isolated facts.

• The interior attacks are mostly bounded, and the trifecta marks where real damage occurs.

• Provenance is the most directly controllable surface.

• Defense in depth, rather than any single filter, contains the residual risk.

References:

• EchoLeak: zero-click prompt injection in a production LLM (case study)

• Stealing Part of a Production Language Model

• Extracting Training Data from ChatGPT

• A small number of samples can poison LLMs of any size

• PoisonedRAG: knowledge-base poisoning of retrieval-augmented generation

• OWASP Top 10 for LLM Applications 2025

• The lethal trifecta for AI agents

• Malicious ML models discovered on Hugging Face (nullifAI)

• Hugging Face model-scanning findings

• New prompt injection papers: Agents Rule of Two and The Attacker Moves Second

• Defeating Prompt Injections by Design (CaMeL)

• Agents Rule of Two: a practical approach to AI agent security

• Anthropic Git MCP server injection advisory (CVE-2025-68143)

This course is about teaching engineers how to reliably ship production-grade software with coding agents. Students will learn how to delegate real work to coding agents, write specs and plans that agents can execute, and verify, review, and secure the code that comes back.

You’ll help refine the curriculum, teach live sessions, answer student questions, and share practical lessons from your own experience building software with AI. The initial onboarding requires some preparation, but after that the commitment is approximately 2 to 10 hours every two weeks. This is a flexible opportunity that fits alongside a full time engineering role. Compensation is competitive and based on experience.

You’re a great fit if you:

• Are excited about helping engineering teams avoid AI slop

• 5+ years of professional software engineering experience building production systems

• Use AI coding agents such as Claude Code, Codex, Cursor, etc., daily.

• Understand how coding agents work, including planning, tool use, context management, and common failure modes.

• Have experience breaking complex engineering problems into clear specifications and implementation plans that AI can execute

• Have worked extensively with large or legacy codebases and know how to make them AI friendly

• Have reviewed significant amounts of AI generated code and know how to verify correctness, catch subtle bugs, identify security risks, and maintain long term code quality

• Have strong software engineering fundamentals, including testing, CI/CD, debugging, code review, etc.

• Enjoy teaching

If this sounds like you, email jobs@bytebytego.com with your background, how you use AI in your engineering workflow, or any teaching, writing, speaking work that demonstrates your expertise.

Two different things could have happened. The charge succeeded, and the confirmation was lost on the way back, or the request never reached the payment service at all. Both possibilities produce identical outputs, which makes it difficult to figure out what happened and the next action to be taken. Retrying risks charging the customer twice. On the other hand, declining to retry risks never charging them at all.

Idempotency is the property that makes the retry safe. An operation is idempotent when applying it more than once produces the same state as applying it once. For example, setting an account balance to 500 is idempotent, because even the tenth time to execute this operation, the outcome will be the same. In contrast, adding 500 to a balance is not idempotent, since every time it is executed, the balance amount changes. Most operations that matter in a business system resemble the second one.

In this article, we will look at the following topics in detail:

• The three different delivery semantics available for developers.

• The three points where duplicates enter a producer, broker, and consumer path, and why a fix at one point does nothing for the other two

• The difference between an operation that is idempotent by nature and an endpoint engineered to behave that way

• What does an idempotency key need to work, and how can it fail?

• Why every deduplication scheme has a time limit, and what the guarantee is worth once that limit passes

• What “exactly-once” means in real-world systems, and where each guarantee ends?

Delivery Semantics

Read more

GPU-hour billing charges for the entire time a machine is reserved — setup, idle time, queueing, and failures — so your cost depends on infrastructure efficiency you don’t control. Crusoe Serverless Fine-Tuning bills per token processed during training, making costs predictable from your dataset size and epoch count, with no GPUs to provision or right-size. Early stopping ends the job, and the billing, the moment your model stops improving — you’re never charged for epochs that add no accuracy. Jobs run on AI-optimized infrastructure with automatic recovery and restart, and checkpoints save at every step.

Get started

AI labs are moving faster than ever and releasing the most capable models we have ever seen. Just recently, Anthropic released Fable 5, then OpenAI released the GPT-5.6 family, including GPT-5.6 Sol, their most capable model yet, Kimi released Kimi K3, and Opus 5 came out just a few days ago.

But capability is only half of the picture. The other half is how much it costs these models to complete tasks, that is, the cost per successful task. Lower cost makes the model more affordable for users and less costly for the provider. A huge amount of effort inside the labs goes into making every component and layer of their AI applications optimized and more efficient, to reduce the overall cost. For example, GPT 5.6 Sol with max reasoning scores higher than Fable 5 on the Artificial Analysis Coding Agent Index while costing less than half as much.

To understand what techniques are adopted in frontier labs to make AI applications more efficient, we met with the OpenAI engineers who developed and shipped various efficiency techniques into the systems behind Codex and ChatGPT Work. Thanks to Joe, Ahmed, Steve, Matthew, and Philippe for sharing valuable details with us.

In this article, you’ll learn:

• What actually happens when you send a request to an AI agent

• How the harness layer cuts repeated work with persistent WebSockets, stable prompt prefixes, deferred tool discovery, and Code Mode

• How the API layer tokenizes only the delta and runs safety checks in parallel with inference

• How the inference layer gets more out of GPUs with cache-aware routing, KV cache management, speculative decoding, and separating prefill from decode

• What OpenAI learned from all of this, and which lessons you can apply to your own systems

Anatomy of an Agentic AI Application

When you give Codex or ChatGPT Work a task, like fixing a bug, the query does not go directly to the LLM. It is more complex than that. An AI application like Codex is not just an LLM. It is a system, with many components. User queries pass through several layers before the LLM ever sees a single token.

But why can’t we send the user query directly to the LLM? To understand why, let’s see what an LLM actually is.

An LLM is a neural network trained to predict the next token. It takes a sequence of tokens as input and produces a sequence of tokens as output. It cannot run a shell command, edit a file, or remember anything between calls. But an agent task like “fix this bug and run the tests” is mostly actions. Something has to turn the model’s predicted tokens into real commands, feed the results back, and continue until the task is done.

That is the job of the harness layer, a system built on top of the LLM to handle those responsibilities. It takes the user’s task as input, decides which instructions, which tool definitions, and how much history to include in the context, and maintains the conversation history. When the model responds with a tool call, the harness executes it under approval policies in a sandbox environment, appends the result, and sends the conversation back to the LLM.

But even the harness’s request does not go directly to the LLM. In production-ready applications, there is an API layer (an application layer) sitting between the harness and the inference endpoint. This layer exists because a real product needs to handle things that are not covered by the harness layer or the LLM. For example, authenticating the caller, enforcing rate limits, and, importantly, converting text into the token IDs the LLM expects, and converting generated tokens back into text.

Once the API layer prepares the context and tokenizes it into a sequence of token IDs, it is time for the LLM to process the input. That is the inference layer. It is a remote endpoint backed by fleets of GPUs, hosting the model. Its job is to run the model’s computation over the prepared tokens and produce the response, whether that is a new tool call or the final answer, as efficiently as possible. Then hand the generated tokens back to the API layer.

What happens when you send a request to an AI agent?

Now that we understand the three main layers of agentic AI applications, let’s walk through a concrete example to see how a request travels between these layers. Suppose you ask Codex to “trace the checkout regression, patch it, and run the tests.” Here is what happens:

1. The harness assembles instructions, tool definitions, and your task into a request and sends it to the API.

2. The API buffers the request into memory, parses the JSON, and validates it: the request is well-formed, and the chosen model supports every feature it asks for. It checks who is calling, applies rate limits, and runs preflight checks.

3. The API renders the conversation into the model’s input format and tokenizes it.

4. The API dispatches the tokens to the inference layer, and starts its safety checks at the same time: classifiers that look for things like cyberattacks and bioweapon content. The safety checks race to finish before the first generated token comes back.

5. The inference layer processes the prompt and begins generating. Its output here is not the final answer; it is a tool call: search the code for “checkout timeout.”

6. The generated tokens flow back to the API

7. The API converts tokens into text, wraps them in API events

8. The API streams them to the harness.

9. The harness recognizes the tool call, runs the search in its sandbox, appends the output to the conversation, and sends the updated conversation back to the API.

This is just one iteration. In practice, a task repeats this loop many times until the model produces its final summary and the harness hands control back to the user. Each iteration redoes a lot of the same work described above. For example, the history gets resent, the text gets retokenized, or the prompt gets reprocessed. Removing this repeated work is the main opportunity for optimization. The next three sections cover the optimization techniques OpenAI adopted to make each layer more efficient.

Harness Optimization

The harness is the orchestration layer closest to the user that turns the raw user request into a context and interacts with the LLM in a loop until the task is complete. It has two main responsibilities.

First, it is the source of truth for the conversation. It holds the authoritative record of every instruction, message, tool call, and tool result. Second, it runs the agentic loop. It decides what goes into the context sent to the LLM (which instructions, which tool definitions, and how much history). It sends the request, receives the response as a stream, parses it, and watches for tool calls. When a tool call appears, it executes it under approval policies in a sandbox environment, appends the result to the conversation, and sends it back to the LLM. It repeats this loop until the task is done. Heavy tasks could in theory repeat more than 100 times. Each iteration carries overhead. For example, one extra second per model call adds roughly half a minute to a long task.

How to make the harness layer efficient?

From the harness layer’s perspective, the latency comes from four sources: the network, prompt processing, context contents, and the loop’s round trips. OpenAI adopted the following four techniques at the harness layer to make it more efficient.

1. Persistent WebSockets and incremental requests

The harness runs on the user’s machine, but the model runs in OpenAI’s data centers, so every model call is a network exchange. The standard way to make that exchange is HTTPS. The harness opens a connection, sends a request containing everything the server (the API layer) needs, and receives a response. Because responses from a language model arrive gradually, one token at a time, chat applications typically use Server-Sent Events (SSE) on top of HTTPS, so the client sends one request, and the server streams the answer back in small chunks over the open response. SSE is a one-way street. It is a great fit for the chat era, where one user message produces one model call and one streamed answer. But agents are not one-pass. A single Codex turn can contain many model calls. With HTTPS, each of those calls is a new request with two separate costs.

The first cost is connection setup. Opening a fresh HTTPS connection means a TCP handshake followed by a TLS handshake, several network round trips before a byte of useful payload is transmitted. Paying that once per user message is manageable, but paying it many times inside one conversation turn is expensive.

The second cost is repetition of the payload itself. HTTP is stateless, so each request must carry everything the server needs. For example, the instructions, tool definitions, and the entire conversation so far, all need to be included in the request. By the twentieth tool call, the harness is resending the original prompt, nineteen tool calls, and nineteen tool results, just to add one new result at the end. This causes the payload to grow with every iteration, so the harness spends more and more time uploading data the server has already seen.

But how do we fix these two sources of cost?

The fix for the connection cost is to open one connection and keep it alive, instead of creating a new one for every call. This is what WebSockets are designed for. A WebSocket needs just one initial handshake, and after that, both sides can send messages whenever they want with no per-message setup. Codex opens a single WebSocket to the API and keeps it open across all the model calls in a turn. This eliminates the repeated TCP and TLS setup.

The fix for the payload repetition is to stop resending what the server already knows. The harness keeps the previous request and the completed response. If nothing but the conversation input has changed, it sends only the new items along with a reference to the previous response. After a tool call, the next message on the socket can be as small as:

{
  “type”: “response.create”,
  “previous_response_id”: “resp_abc123”,
  “input”: [
    {
      “type”: “function_call_output”,
      “call_id”: “call_xyz”,
      “output”: “...the new tool result...”
    }
  ]
}

Notice that this message has no instructions, tool schemas, or history. The server will rebuild the full context from the state it kept, s

o the model still sees everything. The only thing that changed is how much data crossed the network.

2. Stable prompt prefixes

LLM providers avoid repeated calculations with a technique called prompt caching. When a prompt arrives, the model reuses the cached internal state for the beginning of the prompt that matches a previous request (the prefix), and only computes the rest. The match is exact, token by token. If you change one token near the front of the prompt, everything after it must be recomputed.

For the harness, this means the prompt it builds on every call must start with exactly the same bytes as the one before. That sounds trivial, but the harness rebuilds the request each time from its live in-memory state, so any small difference in how it assembles the prompt may silently break the match. OpenAI shared an example of this. Codex kept MCP tool definitions in a hash map, which does not guarantee ordering, so the same tools could serialize in a different order on each request. It was the same tools in the context, just in a different order. Codex was still completing tasks, just more expensively.

The efficiency technique here is to treat the history as append-only, and keep volatile runtime state like approval settings out of the prompt. For example, when a user changes approval settings mid-session, the harness does not edit the tool definitions in the prompt. It applies the new policy itself the next time a tool runs. This way, the prompt stays unchanged so the cache stays valid.

3. Deferred tool discovery

Stable prompt prefixes make repeated context cheaper, but they do not keep the prompt small and concise. In agents with access to lots of tools, this can be an issue. For example, the tool schemas alone can take up a huge amount of space, because a Codex session can expose hundreds of tools once you count connected integrations and MCP servers. Every tool schema is a JSON object with names, descriptions, and parameter definitions. Once all of these are in the prompt, the LLM has to process all of them in its calculations, even though many may go unused for the current task.

To keep unused tools out of the prompt, OpenAI uses a technique called deferred discovery. The prompt carries only the core tools, say the shell and file editing, plus one tool_search tool. The other hundreds of integration definitions stay out of the prompt entirely. When the model needs a capability, it calls the search tool with keywords like “list deployments”. The harness searches the catalog, and the matching definition gets loaded into the context. The search itself is BM25, a lexical ranking algorithm that finds tools whose descriptions overlap with the generated keywords.

In addition to deferred discovery, two more techniques can keep the context small. Schema compaction trims oversized tool schemas down to a token budget by stripping descriptions and collapsing nested structures while preserving argument names. Conversation compaction condenses a long history into a short summary.

4. Code Mode

In ordinary setups, the model emits one tool call, waits for the result, reasons, and emits the next. In many scenarios, there is no reasoning needed between these tool calls. The model just emits them one by one to collect the results it needs. This is costly because each tool call requires a full model round trip. Also, after each tool call, its result keeps getting added to the context, which wastes context space.

With Code Mode, instead of emitting tool calls one by one, the model writes a small program that makes those calls. The harness runs this program in an embedded JavaScript runtime, where every tool is available as a function. The script can fan out independent calls in parallel, filter and join the results in plain code, and return only the compact answer. This way, the intermediate data stays in the runtime, and only the final result enters the context.

API Optimization

The API layer is the service the harness is talking to. It sits between the harness layer and the inference layer, running on ordinary CPUs. The harness sends JSON describing structured conversation items, while the model consumes and produces token IDs. When a request arrives, it goes through the following steps:

1. The request body is buffered into memory.

2. The JSON is parsed and validated against a schema (well-formed fields, arrays within expected bounds, etc.)

3. A second validation round checks semantics. For example, does the chosen model support the requested features?

4. Preflight checks run: authorization and entitlements, rate limits, and checks on any images in the request.

5. The API renders and tokenizes the conversation, then kicks off two jobs at the same time: the inference request to the GPUs and the safety checks on the prompt.

6. As generated tokens flow back from the inference layer, the API converts each one into text, wraps it in an API event, and streams it to the client.

These steps are repeated on each iteration of the loop. The inference is out of the API layer’s control. The API cannot make the GPU any faster. All it can do is add as little delay as possible around it. The time the API layer spends parsing, validating, or tokenizing is extra latency the user feels.

How to make the API layer efficient?

OpenAI adopted the following techniques to reduce the overhead coming from different places in the API layer.

1. Tokenizing only the delta

Models do not read text. Before inference starts, the prompt must be converted into token IDs. Tokenizers convert the text into token IDs in O(n), linearly going over each token in the prompt and replacing it with the token ID. Under stateless HTTP, the whole conversation gets retokenized on every call. By the twentieth tool call, the API is rereading many thousands of tokens to extract one new result. The GPU only needs the new tool result, but the CPU is rereading the book from page one, and the cost keeps growing with input length.

With the WebSocket, the API keeps the tokenized conversation in server memory. The first request tokenizes the full prompt, but every later request sends only the new items. The API tokenizes just that piece and appends it to the stored sequence. This way, per-call tokenization stops depending on conversation length and becomes closer to O(1), depending on the length of the marginal input.

2. Running safety checks in parallel with inference

Agents have to check each request for safety before showing the output to a user. OpenAI implemented safety checks in its API layer. Images go through their own classifiers, while the prompt text goes to classifier models that look for dangerous content, like cyberoffense or bioweapon instructions. These classifiers take time to run. The simple design is to run them first and start inference only after they pass. The problem is that this delay is added to the time-to-first-token (TTFT) of all requests, while the vast majority of them are completely harmless.

The fix is to run the safety checks and inference at the same time. The model takes some time to process the prompt before its first token comes out, so the checks use that window to finish. If a check fails, the API reacts depending on the model. For some models, it streams tokens to the user right away and cuts the stream when a check fails. For more sensitive models, it holds the output until the checks pass and only then releases it. In both cases, the time spent on safety hides inside a wait that was going to happen anyway.

3. Routing traffic to newer CPUs

When a company runs servers at scale, its fleet ends up with machines bought in different years, and therefore with different generations of CPUs. The scheduling software usually treats machines of the same type as identical. In reality, they are not. OpenAI queried Kubernetes for the actual processor model behind each of its deployments and found a mix of older Broadwell chips and newer Ice Lake chips behind the same machine labels. The older ones were serving the same traffic with roughly 20% worse time to first token, while using about twice as much CPU.

To fix this, OpenAI shifted more traffic toward the machines with newer processors and started treating CPU generation as an explicit factor in capacity planning. Sometimes the best software optimization is better hardware.

Inference Optimization

The inference layer is where the model actually runs. There are fleets of GPUs and other accelerators, and the serving software (the engine) that operates them. The model’s core computation is enormous matrix arithmetic, which parallelizes across the thousands of cores an accelerator provides. The layer schedules and batches incoming requests across the fleet, holds the per-conversation state that generation depends on, executes the model’s forward passes, and hands each generated token back to the API layer.

How to make the inference layer efficient?

Demand for tokens grows faster than hardware can be added, so at this layer slow and wasteful are the same word. The waste hides in four places: work routed to the wrong machine, memory holding the wrong state, parallel hardware idling through sequential generation, and two mismatched phases sharing the same machines.

OpenAI adopted the following techniques to make the inference layer more efficient.

1. Balancing load with cache-aware routing

OpenAI serves its models from many GPU machines. Every request has to be sent to one of them. If that routing is uneven, some machines sit idle while others queue up work. Idle GPUs are the most expensive kind of waste in the stack. There is also a second, less obvious problem. Each machine keeps the computed state (the cache) for the conversations it recently served. If the next request of a conversation lands on a different machine, that cache is useless. The new machine must recompute everything from scratch, even though the work already exists somewhere else.

So the router has two goals:

1. Spreading load evenly: send the request to the least busy machine.

2. Using the cache: send it back to the machine that already knows this conversation.

OpenAI’s routing weighs both for every request, along with basics like geography and available capacity. Improvements in load balancing alone have dramatically reduced the cost of serving its models.

2. Managing the KV cache

When a transformer generates a token, it attends over all previous tokens. To avoid recomputing that attention state for every new token, the serving system keeps it in accelerator memory, in what is called the KV cache. The cache grows linearly with conversation length, multiplied by every concurrent conversation the fleet is serving. For long contexts it can rival the size of the weights themselves. If the software evicts the wrong state, the inference layer pays the full processing cost again when that conversation returns.

The efficiency technique here is to manage the cache based on real usage. OpenAI analyzes production traces to learn which cached state is likely to be needed again, tests eviction policies against that data instead of intuition, and optimizes how cached state is stored and moved between memory tiers. This ensures the most valuable work stays around longer without running out of memory or wasting time moving data.

3. Speculative decoding

Generation is sequential. Each token depends on the previous one, so a model that processes a huge prompt in one parallel pass must produce its answer one token at a time. In large models, producing these tokens is expensive and slow, even when the next token is easy to guess. For an input like “the capital of France is”, the next token is almost certainly “Paris”, but the model still has to process the entire conversation and perform large matrix multiplications to predict it.

Speculative decoding uses a small draft model to propose the next several tokens, and the large model verifies all of them in a single parallel pass. In many cases, the draft proposals are correct and will be accepted by the large model, which saves the time the large model would have taken to produce them one at a time. When a proposal is wrong, the large model discards it and produces its own token, so the output quality does not change. The main metric to evaluate the draft model is the average acceptance length, which is how many proposed tokens the main model accepts in each pass.

4. Separating prefill and decode

Inference has two phases, prefill and decode. Prefill processes the entire incoming prompt in one massively parallel pass, and it builds the internal state (KV cache) the model needs. Decode then generates the response one token at a time. These two are very different workloads. Prefill is compute-heavy, while decode is memory-heavy, spending most of its time streaming weights and cached state through memory. Running both on the same machines means neither runs on hardware set up for it.

The efficiency technique is to separate them. One part of the fleet handles prefill and another handles decode, each configured for its own bottleneck. When a request finishes prefill, its computed state (the KV cache) is shipped to a decode machine, which generates the response token by token.

That covers the efficiency techniques at each layer of an agentic AI application. The harness avoids resending what the server already has, the API avoids reprocessing what it has already processed, and inference avoids recomputing what it has already computed. To close, let’s look at the lessons from this work that apply beyond OpenAI.

What to Learn from OpenAI’s Push for Efficiency and Lower Cost per Successful Task

Zooming out, all these optimization techniques at different layers try to avoid paying for the same work twice. The harness sends only what is new and keeps prefixes stable so caches survive. The API tokenizes only the delta and hides unavoidable work inside windows that had to pass anyway. Inference routes conversations back to their cached state instead of recomputing it. These layers also feed each other. When the harness keeps the cache intact and the API improves the cache hit rate, the savings appear on the GPUs, which means lower costs for users. Beyond the techniques, here are a few lessons that the OpenAI engineers shared with us.

Keep the design simple. Prioritize simplicity over complexity. For example, use lexical search instead of embeddings for tool discovery, and one compaction path instead of a menu. An LLM is already quite intelligent, so adding more machinery in front of it often adds complexity without adding value. Build the simplest thing that could work, then scale it.

Use the agent to optimize its own stack. Codex wrote much of the migration of the very API that serves Codex, turning years of rewrite work into months for a couple of engineers. The inference team uses it to analyze traces and prototype kernels. When agents write the code, you no longer need a language that is easy for humans to write, so you simply pick the most efficient one to run. Efficiency work is becoming a loop that accelerates itself.

Optimize end to end. The inference team told us that every time they picked one favorite technique to focus on, they regretted it. Focusing on one part of the stack made them underinvest in the others. No single optimization is a game changer on its own. The big wins come from chaining many small ones together. They also learned to test with the same traffic shapes that production actually serves, because a change that looks like a win offline can hurt in real traffic.

Users expect apps and agents to reach the tools they already work in. Every integration that gets you there is a different OAuth flow, a different token lifecycle, weeks of infrastructure before you write a line of product code.

WorkOS Pipes handles it with one API call. Pre-built connectors for GitHub, Slack, Salesforce, Google Drive, and more. Pipes handles OAuth, token refresh, and credential storage. You call the real provider API with a fresh token, every time.

Connect to 100+ providers →

Over the last few years, three of the biggest food delivery companies rebuilt their search systems around LLMs. DoorDash, Instacart, and Uber Eats were solving the same problem of trying to understand a user’s intent when they type something in search. They were also accessing more or less the same research base. And yet, the architectures they shipped look quite different from each other.

That divergence is one of the most interesting aspects of modern development using LLMs. Once we understand why each company landed where it did, we will have a mental model for thinking about how AI can be integrated into any production system.

Adding an LLM to an existing stack comes down to one question: how deeply should the LLM reach into the runtime?

Ultimately, DoorDash, Instacart, and Uber Eats each answered that question differently, and the specific LLM each chose was secondary. The specific model each company picked was secondary. The infrastructure they already had in place is what determined the answer.

In this article, we will walk through their differing solutions and try to make sense of their choices and understand the pattern behind them.

Disclaimer: This post is based on publicly shared details from various sources. References are present at the end. Please comment if you notice any inaccuracies.

The Problem

Type “something healthy for a rainy evening” into a food delivery app and watch what comes back. The result we get today is somewhere between useful and impressive. The same query five years ago might have returned a random jumble of items, because keyword search treats the words as a bag of tokens rather than as an intent, and the query offers little for keyword matching.

This pattern repeats across several common failure modes in food search. For example:

• Synonyms: “Soda” and “soft drink” describe the same product, but a keyword engine treats them as different tokens.

• Typos: “Mozzarela” should retrieve mozzarella results, but the spelling mismatch breaks the lookup.

• Shorthand: “Gf pizza” meaning gluten-free pizza requires the system to recognize an abbreviation as a synonym for the full phrase.

• Language mix: The Spanish word “pan” means bread, while the English word “pan” means a cooking vessel, so a bilingual search bar has to disambiguate.

• Word-sense ambiguity: “Apple” the fruit and “Apple” the company share the same spelling but mean different things, and the right answer depends on context.

Each one is a potential moment where the user’s intent and the catalog’s words fail to line up.

Two harder problems lie beneath this:

• The long tail: Grocery and restaurant platforms see an enormous number of unique queries. Any specialized model trained on conversion data struggles with queries that appear only a handful of times, because rare queries are rare by definition.

• The constraint problem: A query like “vegan chicken sandwich” has a hard constraint inside it, and similarity-based retrieval might return a chicken sandwich that violates the constraint, because the similarity score still looks close. Dietary restrictions, allergens, and quantity filters all sit in this bucket.

Food search is the right domain to watch this play out, because all of these failure modes show up at the same time. In the subsequent sections, we look at how different companies handled these situations differently.

Your infrastructure platform shouldn’t be your biggest project [VIRTUAL EVENT] (Sponsored)

How do platform engineers keep pace with a growing development team at a $2B AI company? Join Lawrence Aiello, Platform Engineering Lead at Rogo, on August 11 to learn how his team evaluated and implemented a modern IaC platform. He’ll cover:

• The evaluation criteria that mattered when Rogo picked a modern IaC platform, evaluating solutions like Pulumi, OpenTofu, Crossplane and more

• What “it just works” looks like in practice for infrastructure operations

• Why a reliable platform is the foundation for AI-driven infrastructure workflows

If your infrastructure platform has become another system to maintain, this session is for you.

Register Now →

DoorDash

DoorDash already had a knowledge graph for items and restaurants when LLMs became viable for production. The graph held structured attributes for every item, including dish type, dietary preference, cuisine, brand, and flavor.

Their approach was to use LLMs to enrich this graph offline by extracting attributes from SKU data, and to use LLMs at runtime only for parsing queries into chunks that could link back to the graph. Retrieval itself stayed keyword and graph-driven.

Consider the query “small no-milk vanilla ice cream.” The LLM segments it into three chunks.

• “small” is a quantity attribute.

• “no-milk” is a dietary preference attribute that maps to the canonical label “dairy-free.”

• “vanilla ice cream” splits further into a dish type (”ice cream”) and a flavor (”vanilla”).

Each chunk is then linked to a specific field in the knowledge graph. The dietary preference becomes a hard filter, so only dairy-free items get retrieved. The flavor becomes a soft preference for ranking. The dish type narrows the candidate pool.

See the diagram below:

What distinguishes DoorDash’s approach is how they constrain the LLM’s outputs.

They use retrieval-augmented generation as a guardrail rather than as a generator. For each query segment, an approximate nearest neighbor lookup retrieves the top 100 closest taxonomy concepts from the existing graph. The LLM is then prompted to pick from that list rather than invent labels. This is a clever inversion of the usual RAG pattern, where RAG typically injects context into a generator. Here, RAG defines the entire output space, so the system only ever produces concepts that the rest of the design already knows how to handle.

The measured impact is a roughly 30% lift in the trigger rate for popular dish carousels, all delivered through an architecture whose runtime stays mostly classical.

The takeaway is that DoorDash’s LLM lives mostly offline, mostly in batch, working on the periphery of the runtime.

Instacart

When Instacart’s engineers first investigated an off-the-shelf LLM to categorize the search query “protein,” the model returned chicken, tofu, beef, and other high-protein foods.

The answer is reasonable by any reading of the English language. The problem is that Instacart’s actual users, when they type “protein,” are looking for protein bars and protein powders. The model’s general world knowledge and the company’s specific user behavior were pulling in different directions, and this small failure points at the central problem Instacart had to solve.

The system they were replacing was complex.

Query category classification ran on a FastText model, query rewrites came from a separate engine that mined session behavior, and spell correction, query tagging, and aisle classification each ran as a separate model with its own data pipeline and serving infrastructure. The maintenance burden was substantial, and tail queries still suffered because each model needed its own labeled data, and labels for rare queries are scarce by definition.

Instacart’s strategy is layered across three approaches:

• Context engineering: Retrieval-augmented generation pulls Instacart-specific context (top-converted categories, historical conversion data, catalog details) into the prompt before the LLM sees a query.

• Post-processing guardrails: Semantic similarity filters drop LLM outputs that drift away from the original query.

• Fine-tuning: For the most advanced tasks, the team fine-tunes Llama-3-8B on Instacart’s proprietary data, so domain knowledge gets baked directly into the model weights.

See the diagram below:

The serving architecture splits along the head-versus-tail distribution.

Head queries hit an offline RAG-and-cache pipeline that is latency-tolerant and deeply context-engineered, while tail queries hit a real-time fine-tuned Llama-3-8B model that keeps latency under 300ms through adapter merging, H100 GPUs, and autoscaling.

The cache serves queries the company has seen before, while the real-time fine-tuned model serves everything else, which is where the cold-start tail lives. This split is exactly the kind of decision the head-versus-tail traffic distribution encourages, since pre-computing only pays off for queries that repeat.

After this solution, query rewrite coverage jumped from 50% to over 95%, with 90%+ precision across substitutes, broader rewrites, and synonyms. The real-time fine-tuned model improved search quality for the bottom 2% of queries (the cold-start tail), cutting scroll depth by 6% and complaints about poor tail-query results by half.

The takeaway is that Instacart’s LLMs live at the query understanding layer, with some offline-cached and some online and fine-tuned, while retrieval and ranking downstream are still done by traditional machine learning and information retrieval systems. The LLM brain is doing the upstream interpretation, while the classical approach is doing the downstream serving.

Uber Eats

Uber Eats faced the same problem the others did, plus a complication. They run across multiple verticals, including restaurants, grocery, and retail, across multiple markets, and across a long list of languages.

Their previous setup was fragmented, with separate BERT-based embedding models per vertical, lexical search in places, and the operational overhead of maintaining all of these in parallel. The goal was a single retrieval system that handled every vertical, every market, and every language with one consistent embedding space.

The architecture they decided on is a classic two-tower setup, where a query encoder and a document encoder each produce vectors in a shared space, and matching is done by similarity in that space.

The twist is in what sits inside each tower, because both towers use a fine-tuned Qwen LLM as their backbone embedding layer. The query tower runs online, embedding each incoming query in real time, while the document tower runs offline and pre-embeds billions of documents into an HNSW vector index (a graph-based structure built for fast similarity search at scale). This split is what makes the system economically possible, since running a heavyweight LLM on every document at query time would be prohibitive, while pre-computing once and looking up at retrieval time is tractable.

The fine-tuning matters as much as the architecture choice.

Out of the box, Qwen brings world knowledge and cross-lingual capability, which is what handles the Spanish “pan” problem mentioned earlier. Fine-tuning on Uber’s proprietary query-document interactions teaches the embedding space what Uber Eats users actually care about, and this is what gives the system its domain alignment. The base model contributes general semantics, and the fine-tuning adds the specific rules of Uber Eats.

What makes the system viable at scale is a stack of optimizations:

• Matryoshka Representation Learning trains a single model whose embedding can be truncated to different lengths, and Uber serves at 256 dimensions in production with under 0.3% recall loss compared to the full 1,536.

• Scalar quantization (int7 instead of float32) cuts latency in half again.

• Pre-filters on hexagon, city, and fulfillment type shrink the candidate set before the ANN search even runs.

The numbers tell the cost story. Tuning the ANN parameter k dropped latency by 34% and CPU by 17% with negligible recall impact, quantization halved latency at recall above 0.95, and MRL cut storage by nearly 50%. Together, these are the engineering choices that let a fine-tuned LLM serve as the retrieval substrate of a production search system at Uber Eats’ scale.

The takeaway is that Uber Eats’ LLM is the embedding model itself, since every query and every document gets an LLM-derived vector, and retrieval at every level depends on LLM-produced representations.

Integration Depth

If we line up the three companies on a single axis labeled “how deeply does the LLM sit in the runtime,” the pattern provides some insight.

• DoorDash sits on the left, with the LLM enriching the catalog offline and parsing queries with constrained outputs while the runtime stays mostly classical.

• Instacart sits in the middle, with LLMs handling query understanding through offline RAG for the head queries and a fine-tuned Llama-3-8B for the tail, while retrieval downstream is still traditional.

• Uber Eats sits on the right, with a fine-tuned Qwen as the embedding backbone running for every query and pre-baked into every document vector.

Each company’s position on the spectrum was largely determined by the infrastructure it already had.

• DoorDash already had a knowledge graph that classical retrieval could exploit, so the cheapest gain came from making the graph richer and teaching queries how to talk to it.

• Instacart had specialized query-understanding models that were hard to maintain, so the biggest win came from consolidation under a unified LLM strategy.

• Uber Eats already had two-tower embedding infrastructure running per vertical, so the natural next step was swapping in a fine-tuned LLM as the shared backbone, allowing one model to handle every vertical and every language.

The lesson would be that before asking which LLM to use for some task, the better question is where in an existing stack an LLM would actually help.

Conclusion

Three of the largest food delivery companies rebuilt search around LLMs in the same window of time, with overlapping research literature and similar production constraints, and they arrived at three completely distinct architectures.

The reason food search was a good place to investigate this is that it breaks classical keyword retrieval in many ways simultaneously. Subjective queries (something healthy for a rainy evening), long-tail traffic, multilingual catalogs, and compound constraints all sit in the same search bar.

The three approaches each represent a different setting:

• DoorDash uses LLMs offline to enrich a knowledge graph, while classical retrieval still drives the runtime.

• Instacart uses LLMs at the query understanding layer, with fine-tuned smaller models handling tail queries in the hot path.

• Uber Eats fine-tuned a Qwen LLM into the embedding backbone of two-tower retrieval, so every query and every document gets an LLM-derived vector.

Each company’s position on this spectrum traces back to the infrastructure it already had, which is why the depth-of-integration question matters more than the model choice.

Three universal tradeoffs survived every architecture.

• Hybrid systems are the default everywhere. Classical retrieval, knowledge graphs, and ANN indexes still do most of the work.

• World knowledge from a pre-trained model is just a head start. Domain context still has to be injected somewhere, through RAG, fine-tuning, or both.

• Guardrails are an important part of every production LLM system. Constrained vocabularies, similarity filters, and taxonomy enforcement quietly determine whether outputs stay aligned with the catalog.

References:

• Evolution and Scale of Uber’s Delivery Search Platform

• Scaling Multilingual Semantic Search in Uber Eats Delivery

• How DoorDash leverages LLMs for better search retrieval

• Building DoorDash’s product knowledge graph with large language models

• Building The Intent Engine: How Instacart is Revamping Query Understanding with LLMs

• Supercharging Discovery in Search with LLMs

Define tasks with Render’s lightweight SDK and chain them into long-running, distributed workflows. Launch your agents and batch jobs on demand. Render Workflows handles queuing, orchestration, and retries.

Get $50 credits with code BYTE

You probably know NVIDIA sells GPUs. But did you know it is also the largest publisher of open AI models in the world? Its models rank among the most downloaded on Hugging Face, and the lineup goes way beyond chatbots: reasoning models, world models, models for humanoid robots, self-driving cars, even drug discovery, quantum computing, and global forecasting.

This raises two questions. How does a company best known for hardware build so many strong models across so many domains? And why give them away for free, when those same models run on the GPUs NVIDIA sells?

To understand both, we spoke with Bryan Catanzaro, VP of Applied Deep Learning Research at NVIDIA. He walked us through how his team builds the company’s open models, the reasoning behind their architecture, and why NVIDIA open-sources so much of it. Our thanks to Bryan for taking the time to share the team’s work in such detail.

In this article, you’ll learn:

• What open models NVIDIA has built, and what they do

• How NVIDIA makes its models both fast and powerful

• How one foundation lets a small team build many models

• What open really means and why releasing the data matters as much as the weights

• Why NVIDIA gives it all away, and the lessons the team has learned shipping in the open

NVIDIA’s Open Model Ecosystem

NVIDIA’s open models sit at two ends of AI. At one end are models that work with symbols, the language, code, and reasoning that lives on a screen. At the other are models that work in the physical world, where robots and autonomous vehicles (AVs) have to perceive their surroundings and act. Most of the lineup falls somewhere along that line.

The simplest way to see the ecosystem is to group them by what each family is for.

1. Reasoning models

Reasoning models are large language models that are trained to produce intermediate tokens before the final answer. This is what lets them think through tasks such as math and coding. Most frontier models you see these days are powered by a reasoning model. Claude Opus, GLM-5.2, Kimi K2.7.

NVIDIA’s reasoning model family is called Nemotron. The first Nemotron model was introduced in 2023. In 2024, NVIDIA released the next generation with 340 billion parameters, followed by a faster hybrid design in 2025. The current generation, Nemotron 3, arrived across late 2025 and 2026 in three sizes. A small Nano model for fast, simple tasks, a mid-size Super model for harder planning and reasoning, and a large Ultra model for complex tasks that need the heaviest reasoning.

2. Physical AI

Language models are great at predicting the next token, but they do not understand the physical world. They cannot tell you how a shirt sleeve folds, or what a glass of water looks like the moment it tips over. A robot or a self-driving car needs that kind of understanding, but it cannot get it from text.

That is what a world model provides. Instead of predicting the next word, it understands the world and can predict the next state given the current state of the world and an action. It can generate realistic, physically plausible video, reason about motion and cause and effect, and produce the action data a robot needs to learn.

World models are extremely useful because you can use them to create training data for situations that are slow, expensive, or dangerous to capture in the real world.

NVIDIA’s CEO, Jensen Huang, has called this “the ChatGPT moment for physical AI.” NVIDIA’s frontier open model for physical AI is Cosmos, launched at CES in January 2025 and unified in 2026 into a single omni-model, Cosmos 3, that can generate a scene, reason about it, and predict what comes next. Cosmos 3 also introduces world action models, which turn the pretrained video backbones directly into robot policies. There is also a 4-billion-parameter Edge version small enough to run on the robot and control it in real time.

From World Models to Robot Policies

A world model can predict the next scene, but a robot has to act. It has to turn what its cameras see into how its motors move, in real time, on tasks nobody scripted in advance.

That is the job of a robot foundation model, often called a vision-language-action model. The inputs to these models are usually camera images and instructions, and the model produces the movement.

NVIDIA’s open foundation model line for humanoid robots is Isaac GR00T. NVIDIA unveiled the project in 2024, released the first open humanoid foundation model, GR00T N1, in early 2025, and has since iterated to GR00T 1.7.

GR00T models use Cosmos as a reasoning backbone, giving robots a shared foundation for perception, reasoning, and locomanipulation: the ability to move while manipulating objects. This enables more human-like decision-making and helps robots break complex actions into structured, step-by-step plans that developers can specialize for specific tasks. The 1.7 release also includes a production-friendly commercial license, allowing developers to deploy the model in real-world applications.

3. Healthcare, autonomous vehicles, quantum, and more specialized applications

The lineup keeps going. Alpamayo is an open model family for reasoning-based self-driving. Like GR00T, it uses Cosmos as its reasoning backbone, so the same physical-AI foundation that powers robots also lets a car work through road situations it was never trained on. It also exposes the chain of cause and effect behind each decision, which developers can inspect and build on. BioNeMo brings open models to biology and drug discovery, and the BioNeMo Agent Toolkit lets developers build autonomous agents for life sciences work. Ising gives quantum computer builders, operators, and developers the AI tools needed to scale devices to fault tolerance. Earth-2 is a family of open models, libraries, and frameworks that makes professional-grade weather and climate AI available to anyone. On top of these, NVIDIA offers open models for production use: safety models that filter unsafe content, speech models for voice, and vision models for images.

Taken together, this powerful ecosystem changes how the community can advance and build. The rest of the article looks at how NVIDIA builds these models, how the team is able to move fast, and what they have learned.

How the Models Are Built to Be Frontier and Fast

It is often a challenge to build a model that is both fast and capable. Capable models are usually larger, which makes them slower to train and to answer questions. NVIDIA’s answer is to chase the frontier and speed at the same time. Bryan put it this way: “the fastest model is the smartest model.”

While this may sound like a slogan, it has a real insight. A faster model can be trained on more data in the same amount of time. It can be post-trained across more environments. Once deployed, it can think for longer on a hard problem at the same cost. So speed translates into capability and compounds at every stage.

Given this strategy, NVIDIA designs its models with a focus on efficiency and speed. The following design choices are what make these models both fast and capable. The first two make the model faster and more efficient. The last one makes it more capable.

1. The hybrid architecture

Most well-known models are built almost entirely on the Transformer’s attention mechanism. Attention lets every token in the input look at every other token, which is powerful for capturing relationships but expensive. The cost grows with the square of the input length, so doubling the input roughly quadruples the work. The model has to keep every earlier token in memory as it runs. On very long inputs, that gets slow and costly.

The other main class is the state space model, and Mamba is its best-known example. Instead of comparing every token with every other one, a Mamba layer reads the sequence once and compresses everything it has seen into a fixed-size memory. That makes it cheap: the cost grows linearly with length, and the memory stays constant no matter how long the input. The tradeoff is that a fixed-size memory cannot keep every detail, so Mamba is weaker at recalling one exact fact buried far back.

Mamba layers are efficient while attention layers are more effective. Combining them gives the best of both worlds. A hybrid architecture swaps most of the attention layers in the Transformer with Mamba layers and keeps a few blocks with full attention.

NVIDIA’s models rely on the same hybrid pattern. Most of the layers are Mamba, which keeps long inputs cheap to process and is what makes a million-token context window practical rather than just theoretical. A few attention layers are placed in between to restore the pinpoint recall that Mamba gives up, so the model can still retrieve exact details from anywhere in the context.

On top of the hybrid design, the models also use mixture-of-experts, or MoE, layers. An MoE layer consists of many small expert layers and routes each token to only a few of them. So instead of running every parameter on every token, the model activates only a small subset of its parameters. That allows us to use a large total capacity at a low per-token cost, which keeps the model fast.

The way Bryan describes it, Mamba builds a global view of the whole sequence. Attention can always go back and find an exact fact. Put together, they are smarter than either one alone. NVIDIA published this hybrid result in 2024, and other labs, including the teams behind Qwen and Kimi, have since moved in the same direction.

2. Co-designing the GPU and the model

The second choice is where NVIDIA’s two businesses meet. Its larger models are pretrained in a 4-bit number format called NVFP4, meaning most of the math during training uses just four bits per value. Fewer bits means less memory and less data to move, so the math runs faster and on less power.

Note that four bits is very little precision. Most teams train in higher precision and only shrink the model afterward, which loses accuracy. NVIDIA trained in 4 bits from the first step because it knew its next GPU generation, Blackwell, was being built with fast 4-bit hardware. The model and the chip were designed for each other.

The VP described the mindset behind it. Moore’s law is no longer giving easy gains every generation, so progress has to come from designing the model and the hardware together. The team only attempted 4-bit pretraining because it believed the result was possible. In his words, the first step to inventing something amazing is believing that you can.

3. Post-training: learning from experience

With those efficiency choices in place, post-training is where the model gains its capability.

The post-training recipe itself is what most other teams converged to. It starts with supervised fine-tuning, where the model learns from curated examples of good answers and picks up the format and behavior expected of it. Then it switches to reinforcement learning, where the model practices on real tasks and is rewarded for reaching the right outcome. Instead of learning from examples, it learns from its own attempts.

Scaling up this reinforcement learning stage has been shown to improve a model’s capability significantly. Here NVIDIA’s efficiency focus pays off again. Because its models are cheaper to run, it can scale RL at a lower cost, letting the model practice across many environments in parallel and learn from over a million rollouts.

This is where the VP placed the real bottleneck for better models. Not data or compute. The diversity of these training environments. The more varied the situations a model practices in, the more capable it becomes, because that is where it gains real problem-solving experience.

So far we have looked at design decisions to build a model and what is involved to train a single frontier model. But how do they build so many models, and keep improving them so quickly?

One Unified Foundation

Building strong models in one domain is hard. Building them across language, video, robotics, vehicles, and biology needs huge engineering and compute resources. NVIDIA’s approach is to identify a reusable foundation, build it once, and reuse it across different efforts.

The VP traced this back to CUDA, the software layer that made NVIDIA’s GPUs programmable. The “U” stands for Unified, and he called it the most important letter. The idea is to build one foundation and run everything on top of it, rather than maintaining many separate stacks.

The same instinct shapes the models. The hybrid architecture from the last section is not rebuilt for each model. The same backbone scales across all three Nemotron sizes. Whole components cross between efforts too. Cosmos Reason, the model built to reason about the physical world, was reused as the reasoning core inside NVIDIA’s Isaac GR00T foundation model, so the robotics team did not have to build that part from scratch. The alternative, ten parallel efforts each reinventing the same pieces, would spread people and compute too thin. According to Bryan, NVIDIA tries to be as lazy as possible, because that is how it gets the most out of limited resources.

The other reason the team is so productive and iterates quickly is cultural. NVIDIA is mostly not top-down; it is a company of volunteers, where teams choose what to work on, and unification happens through invitation rather than control. The culture of collaboration is that if teams work with each other and get more done, they will get more resources than if they each go alone. When everyone shares that belief, a small group can produce a lot.

Open is More Than Just Releasing Weights

You may have heard a lot about open models. Several AI labs released capable models. You can download the weights and use them. But that is not truly an open model. This is where NVIDIA’s approach really separates it from the other labs. As Bryan puts it, Nemotron is not just a model. It is also the data, the tools, the training recipes, and the ideas published in the papers. The finished weights are just the last step.

In practice that means NVIDIA publishes the training datasets, the post-training datasets, the reinforcement learning environments, and the recipes needed to reproduce the work. A team can not only run the model, but also see how it was built and train their own version.

The same is true for robotics and self-driving cars, where developers can use open source simulation and learning frameworks, datasets to train and evaluate the model performance before deployment.

Releasing data at this scale is unusual. It is a scary thing for a company to do. NVIDIA does it anyway, and even shares clever building blocks like synthetic “persona” data that is statistically realistic but fully private, so a model can learn about the world without memorizing real people.

You can see this in what the community has done with it. NVIDIA’s open data and models have spawned a large set of community fine-tunes and derivatives. One of its robotics datasets became one of the most-downloaded datasets on Hugging Face. Many variants of its Nemotron models have millions of downloads each, and the Nemotron family recently crossed 100M in total. You cannot build like that from the weights alone.

Why a Chip Company Gives It All Away

You may think releasing open models sounds great. But why would a company that sells GPUs give away the AI that runs on them?

Bryan shared two reasons. The first is that NVIDIA needs to understand AI deeply to build the right hardware for it. To design future chips, you have to know where AI is going. The best way to really know is to build models yourself and put them in front of real users. A model kept private is easy to fool yourself about, so releasing it openly keeps the work honest and keeps NVIDIA’s researchers connected to the wider community they learn from. With open models being such a hot topic, the real value is in enabling developers to post-train these models themselves at a fraction of the cost.

The second reason is about the business. NVIDIA grows whenever AI grows. Its goal is to support the ecosystem, not to compete with the companies building on top of it. Every team that adopts open models and builds its own AI can become a future customer for compute. So the company would rather hand the ecosystem the models and let the world’s demand for AI pull its business forward.

There is also a side benefit: safety. Open technology tends to be safer because more people can inspect it and find problems, the way the open internet is hardened by everyone who uses it.

What NVIDIA Has Learned Shipping Open Models

Shipping this many models in public teaches lessons that closed labs never face. Bryan shared several valuable lessons.

1. The program matters more than any single release

In a field moving this fast, every model is obsolete before it ships. So no single release is the point. What earns trust is a steady, sustained program. When the community believes a model line is here to stay, it pays attention to each new release instead of treating it as a one-off.

2. Make it easy for developers

A clean launch is the result of getting a hundred small things right, from licensing to documentation to ready-to-run examples. The VP admitted this is the unglamorous part. Listening to community feedback and smoothing that first-run experience is what turns a download into lasting use.

3. Pick a license the community trusts

NVIDIA moved away from its own homegrown licenses to OpenMDW, a community license from the Linux Foundation built specifically for open AI. Using a shared, purpose-built license removes friction and signals that the models are meant to be used freely.

What’s Next

The direction from here is to open even earlier. Rather than build a model privately and post it when it is done, NVIDIA wants to bring collaborators in sooner, since it plans to open everything anyway. The aim is to let partners shape the models while they are still being built. To make this happen, NVIDIA formed the Nemotron and Cosmos Coalitions.

What stands out most is the conviction that this is a long-term commitment, not a phase. The VP pointed to CUDA, which took more than a decade before the rest of the world understood why it mattered. NVIDIA stuck with it, and it became the foundation of the company.

He expects open models to follow the same path. As he put it, this is not a science project, and it is not charity. It is how NVIDIA moves itself forward into the future.

On a single computer, every event can be measured against one clock, so the order of the events is never in doubt. However, a distributed system, where work is spread across many independent machines, lacks a shared reference. Each machine can read time from its own hardware clock, but those clocks do not stay aligned. Network Time Protocol (NTP) tries to bring the clocks together, but it cannot eliminate the problem.

This becomes a real problem whenever correctness depends on the order in which things happened. If events on different machines are ordered by comparing timestamps, a small difference in the wrong direction can change the result. For example:

• An update that happened later can be discarded in favor of an earlier one, because clock differences gave the earlier write the larger timestamp, and the newer data disappears with no error raised.

• Logs gathered from several machines can place an effect before its cause, which makes tracing a problem much harder.

• An operation that depends on a correct order, such as granting access only after the change that authorized it, can act on stale information.

In this article, we look at these ideas in detail:

• Why a machine’s own clock is a poor basis for ordering events, and how close synchronization can realistically get

• What it means for one event to have happened before another, and why some pairs of events have no defined order

• Logical clocks and vector clocks, and what each one can and cannot determine

• Why the order of events matters in practice, for replication, auditing, and debugging

• Hybrid logical clocks and the problem they were designed to solve

• How a globally distributed database such as Google Spanner approaches ordering at scale

Clock Drift

Read more

[Webinar] 8 levels of context maturity in AI-native engineering (Sponsored)

AI is in your engineering workflows. While the token spend shows it, the throughput doesn’t. The human is very much still in the loop, and that’s a context problem.

Join live July 23 (FREE) to learn:

• The 8 levels of context maturity: where most teams are stuck and what the ceiling looks like at each stage

• Why more MCPs, rules, and skills provide agents access but not understanding

• How leading engineering teams are using a context engine to make the most of their agents

Register Now

There is a persistent gap between the AI agents that perform well in a demonstration and the ones that hold up under real production traffic.

This is because the systems handling live customer workloads tend to depend on the language model far less than their presentation suggests. Most of their behavior runs through conventional, deterministic code, with the model invoked at a small number of specific decision points. The difference between a dependable agent and a fragile one often comes down to a set of engineering decisions about where the model is given control and where it is held back.

But how do you build agents that work reliably in production? Are there any best practices?

In 2011, Heroku’s co-founder published the Twelve-Factor App, a set of principles for building web services that deploy and scale reliably, and that document influenced how a generation of engineers thought about software.

A parallel effort later appeared for AI in the form of a widely shared set of principles called the Twelve-Factor Agents. These principles were aimed at making language-model applications robust enough for production. The specifics have kept evolving since, and teams at companies like Anthropic, Cognition, and Intercom have added lessons of their own.

In this article, we try to explore the collective thinking into a smaller set of practices and explain the reasoning behind each one, rather than asking anyone to memorize a numbered list. The practices fall into four areas, with a final section on the tradeoffs. Here’s what we will cover:

• How does context control what the model sees on every call?

• How does the control flow keep the loop and its stopping conditions in deterministic code?

• How does the state hold memory in software while the model stays stateless?

• How does the scope keep each agent narrow and supervised?

By the end, the article will have built a working definition of a production agent and a clear view of which decisions carry the most weight.

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

The Loop

Let us start with the simplest agent anyone can build, because it reveals the structure that everything else refines.

At its core, an agent is a loop. The model receives some context, which is the running record of what has happened so far, and it returns a single decision about the next step to take. That decision arrives as structured output. It is machine-readable data, such as JSON rather than free-flowing prose, so that ordinary code can read it and act on it. The surrounding code executes the requested step, appends the result to the context, and returns everything to the model for the next decision. The loop continues until the model signals that the work is finished.

A useful way to picture the model’s role is as a function that starts fresh on every call.

On each call, the model sees only the context provided to it, makes one decision, and discards everything the moment it responds. The context window, which is the bundle of text the model reads on a given call, serves as the entire memory of the system.

The appeal of this design is clear. A developer describes a goal and supplies a set of tools, and the model works out the order of operations on its own. The design requires less branching logic, since the model handles the decisions. This is the version of an agent that can perform quite well in a demo setup. However, the trouble begins when real traffic arrives.

Failure Modes

The design we looked at in the previous section fails under production conditions, and the failures follow some predictable patterns:

• Compounding error multiplies across chained steps.

• Confidently wrong output reaches a user.

• A loop runs longer than intended.

• State disappears when the plan lives only inside the model.

Let’s understand them in a little more detail.

The first pattern is compounding error. Every model call carries some chance of a wrong decision, and a freewheeling agent chains many calls together. Suppose each step is correct 95 percent of the time, which sounds reliable. Run twenty such steps in sequence, and the odds that all of them succeed fall to roughly one in three. The math is multiplicative, so reliability that looks fine in isolation degrades quickly across a long chain. This fact explains why production teams add so many guardrails.

The second pattern is confidently wrong output reaching a user. In 2025, the support agent for the coding tool Cursor told customers about a one-device-per-subscription rule that the company later confirmed it had fabricated, and a wave of cancellations followed. A year earlier, a tribunal held Air Canada liable after its chatbot invented a bereavement-fare policy, and ordered the airline to honor it. Studies place hallucination rates, meaning the rate at which a model produces plausible but false statements, somewhere between 3 and 27 percent, even in controlled settings. An agent that speaks directly to customers turns that rate into a big business risk.

The final two patterns are harder to notice, but just as costly.

A loop with weak stopping conditions can run far longer than intended, consuming tokens and money with each pass. And lastly, when the plan lives only inside the model’s context, a crash or a long-running task can lose the entire thread of work.

Each failure points to a specific practice, grouped into the four areas below, beginning with the inputs.

Context

The context window is everything the model knows on a given call, so controlling its contents is the first and largest lever on reliability. Three habits do most of the work here.

The first is to own your prompts. A prompt is the instruction text sent to the model, and it deserves the same treatment as any other source code, which means version control, review, and testing. Many agent frameworks generate prompts automatically and keep them out of sight, which is convenient until behavior changes and the cause is hard to locate. Keeping prompts in the application’s own codebase keeps their effects reproducible.

The second habit is to own the context window itself. The contents of each call are a deliberate choice, and that choice carries more weight than it first appears. A model given a focused, relevant context performs better than the same model handed a large pile of loosely related history. Quality degrades as the window fills with marginal material. Therefore, deliberate pruning by actively removing content beyond what the current step requires preserves the model’s accuracy. In other words, relevance beats volume on almost every call.

The third habit concerns tools. A tool is simply a function the model can request, described with a clear name and a schema for its inputs. When those descriptions are precise, the model selects the right tool and fills its arguments correctly. When they are vague, the model is more likely to choose the wrong one. Treating tool definitions as a careful piece of interface design pays off immediately.

A discipline called context engineering has grown up around these habits, and it’s a big topic in itself. However, the core idea stays simple: decide what the model sees, on purpose, every time.

Control Flow

Owning the inputs matters only if the surrounding code decides when the model runs and when the loop ends. In a dependable agent, the control flow belongs to the deterministic code around the model, and the model is consulted at a few chosen moments.

Picture the overall flow as ordinary code, a sequence of steps, conditionals, and calls to external systems. At two or three points in that flow, where a decision genuinely calls for judgment, the system invokes the model. Everywhere else, plain deterministic logic does the work because it is cheaper to run, predictable in its output, and straightforward to test.

Two practical rules make it work:

• First, every loop needs an escape hatch, which is a hard limit that forces it to stop. A cap on the number of iterations, a timeout, and explicit completion conditions together guarantee the agent halts even when the model would continue indefinitely.

• Second, invoke the model on purpose. The model belongs where the problem requires open-ended reasoning, and ordinary code handles everything that can be specified in advance.

Intercom’s customer service product shows the pattern at scale.

Its newer Procedures let the agent reason in natural language while the surrounding system supplies deterministic controls, including conditional steps for decision points, small code snippets that guarantee the same input always yields the same output, and checkpoints where the agent pauses for human approval before a sensitive action.

To summarize, it is beneficial to start with the simplest version that solves the problem, and add model-driven autonomy only where simpler logic falls short.

State

Once the surrounding code owns the loop, a natural question follows. Where does the agent’s memory live?

The answer is to keep the model stateless and to hold the state in software that the application controls.

Recall that the model starts each call fresh and reads only the context provided to it. That property, when treated as a feature, becomes powerful. The application stores the real state of the work, the conversation so far, the plan, the progress, and reconstructs the context from it on every call. Since the state lives in serializable storage (a form the system can save and reload), an agent can pause partway through a task and resume later, or recover cleanly after a crash, by loading the saved state and continuing.

This habit also keeps two kinds of states aligned. There is the model’s view of what is happening, and there is the application’s record of ground truth, such as the actual contents of a database. Keeping these two in agreement prevents the model from acting on a stale or inaccurate view of the current state.

There is a scaling benefit as well.

When the state lives entirely outside the model, any running instance of the service can pick up any request, because everything needed is available with the context. That allows many copies to run behind a load balancer, handling far more traffic.

A clean way to think about the model here is as a function that takes the current state and a new event and returns the next state along with an action to perform. The same inputs produce the same result, which makes the whole system easier to test and to trust.

Scope

Owning the inputs, the loop, and the state still leaves one decision open. How much should a single agent try to accomplish? The good answer favors small, focused agents kept under supervision over one broad agent asked to do everything.

A narrow agent with a single, well-defined job has a smaller surface for failure. It is easier to test, easier to reason about, and easier to fix when it misbehaves. When a task spans several distinct jobs, several narrow agents compose under deterministic orchestration. The surrounding code decides which agent runs and when, rather than assigning one general agent a long list of responsibilities.

Klarna’s customer service assistant illustrates the upside. It handled roughly 2.3 million conversations in its first month, and most of those followed predictable paths such as checking a refund status or tracking an order, which structured logic handles well, with the model reserved for the cases that truly need it.

Supervision is the other half of the scope.

A human handoff deserves to be designed as a first-class step. It is reached on purpose when an action is sensitive, when confidence is low, or when a customer asks for a person. Intercom builds this in directly, handing a conversation to a human automatically whenever continuing would be the riskier choice. Treating human involvement as a planned path, complete with the state needed to brief the person, turns it into a strength of the system rather than a sign that something went wrong.

Tradeoffs

Everything so far describes practices that work today. But there are also disagreements and a few open questions.

The main debate concerns single versus multiple agents.

In June 2025, the team behind the coding agent Devin argued against multi-agent designs because parallel sub-agents make independent decisions that then conflict with one another. One day later, Anthropic described its own multi-agent research system, which used roughly fifteen times the tokens of a single-agent approach and scored about 90 percent higher on certain research tasks.

The apparent contradiction resolved over the following months into a shared pattern. A single orchestrator owns the full context and spawns isolated, short-lived sub-agents, each of which completes one task and returns a summary. The lesson to understand is that small, focused agents work best under firm orchestration, while sub-agents that communicate directly with one another tend to produce conflicting results.

The deeper question comes from what Rich Sutton called the Bitter Lesson. It is the observation that general methods that rely on more computation tend to overtake handcrafted techniques over time. Applied here, it warns that some of the scaffolding built to compensate for today’s models will become redundant as the models improve. That is partly true, and it is the strongest argument against over-engineering.

The counterpoint is that several of these problems persist regardless of model strength. A finite context window, the need to keep a figure on page five of a document consistent with the same figure on page eight hundred, the requirement to pause and resume safely, all of these survive better models.

Cost becomes another consideration for every decision. Autonomy and multi-agent designs spend more tokens, sometimes far more, so they earn their place only where the value of the task justifies the bill.

Conclusion

A production agent, reduced to its essentials, is mostly deterministic software that calls a language model at a few deliberate points. The design decisions lie in choosing those points and limiting how much the model decides on its own.

The path to that definition ran through the simplest possible agent and its predictable failures, with each failure answered by one practice:

• Control what the model sees.

• Own the loop and give it a hard stop.

• Hold the state in software while the model stays stateless.

• Keep each agent narrow and supervised.

Around these sit real tradeoffs, including the open argument over multiple agents and the steady improvement in models that will retire some of this work over time. A sound approach starts with the simplest design that solves the problem, measures where it falls short, and grants the model more autonomy only where that autonomy provides clear value.

References:

• The Twelve-Factor App

• 12-Factor Agents — HumanLayer

• Anthropic — Building Effective Agents

• Anthropic — How We Built Our Multi-Agent Research System

• Cognition — Don’t Build Multi-Agents

• Intercom — What’s New with Fin 3

• Rich Sutton — The Bitter Lesson

• Cursor support-bot incident — eWeek

• Air Canada chatbot ruling — AI Business

An incident hits and you pull up the logs, only to find string based console output that tells you something broke but not why. That’s the problem with unstructured logging: it eats your quota and still leaves you guessing.

This hands-on workshop fixes that. You’ll learn what’s actually worth logging (state transitions, request boundaries, external call failures, auth decisions), how to structure logs as real data instead of text dumps, and how to add attributes that connect them to traces and errors. Plus, how to tune signal to noise, so noise stops drowning out what matters.

Save you spot

Roblox is built for scale. In August 2025, the platform reached a peak of 45 million concurrent users. Few game engines can support that kind of demand.

But Roblox itself isn’t the one making most of those games. It gives creators the tools to build them, then focuses on the infrastructure beneath them. Now, Roblox is exploring how a video world model can work alongside its game engine to make those experiences photorealistic without compromising scale or multiplayer performance.

This hybrid approach offers valuable lessons for engineers building real-time systems at scale. To explore those lessons, we sat down with Anupam Singh, senior vice president of engineering at Roblox, to hear from him about the world model that Roblox is using to make its multiplayer games look photorealistic, the key insights that have come from taking that approach, and the next big thing that the Roblox team is focusing on. We thank Anupam for sharing with us.

In this article, you will learn:

• What a game actually requires, and why visuals are only part of it.

• How combining a world model with a game engine can make games more realistic.

• How Roblox is using a hybrid architecture to divide the work between the engine and the world model and keep the world consistent for everyone playing.

• The four open problems (latency, consistency, multiplayer, and creator control) and the specific techniques Roblox is using on each.

• What Roblox is focused on next, and why it could make photorealistic games far cheaper to build.

What makes a game a game

When people praise a game, they usually point at the graphics. Graphics are the easiest part to notice and often the smallest part of the engineering. What actually makes a game a game is a set of systems players never see. These systems serve one of two purposes: keeping the world consistent for everyone playing, and rendering it on screen.

1. Keeping the world consistent

Keeping a world consistent takes several systems working together. The foundation is the persistent state, a running memory of where everything is and what has happened, so the world is still there when you come back to it. Then come the rules, the logic that has to resolve the same way for everyone, so the game stays fair. Physics has to be believable, so objects move, collide, and fall the way players expect. Since most games are played with other people, all of it has to stay in sync in real time, so thousands of players share one consistent world rather than thousands of slightly different ones. On top of that, the world has to react the instant a player acts. Together, this is what turns moving images into something you can actually play.

A racing game makes this concrete. The track, the scores, the penalties, how each car handles, and where every other car is all need to be exact. Every player needs to see the same thing at the same moment. If even one of these is off, players notice right away.

2. Making the game look real

The second purpose, making the world look real, is a newer demand. The real world has a lot of fine detail that is hard to produce. Grass moving in the wind, dust behind a car, smoke off a fire, light shifting across a surface. Players now expect that level of realism.

That is what makes the problem hard. No platform has delivered all of it at once: a world that is consistent and fair, looks real, and still runs affordably on the hardware people already own.

Today, two technologies each cover one side of this. AI world models are built to make things look real, generating photorealistic images from what they have learned. Game engines are built to keep worlds consistent, tracking state, rules, and physics across every player. The catch is that each is good at only its own half. The next sections look at why a world model alone, and a game engine alone, each fall short, and then at how Roblox combines them.

Crusoe Serverless Fine-Tuning Is Now GA (Sponsored)

Fine-tuning open models shouldn’t mean babysitting GPU clusters. Crusoe Serverless Fine-Tuning, now generally available in Crusoe Intelligence Foundry, gives you a complete path from proprietary data to production-ready model. Select a base model from a curated library of top open models — including Qwen, DeepSeek, Gemma, and gpt-oss — upload your dataset in JSONL or Parquet, configure your job, and submit via UI, SDK, or API. Token-based pricing ties spend to actual training work, and early stopping ends billing the moment your model stops improving. Deploy in one click, or download your weights in .safetensors format.

Get started

Why a video world model alone isn’t enough

A video world model generates a world one frame at a time, predicting what should come next. It can also take conditioning signals as input, such as a text prompt or a starting image.

For a game, the signal that matters most is the player’s actions. That is what makes a world playable. For example, when a player presses a key to move forward or turns the camera to look around, the next frame changes to match. But generating believable frames, even ones that respond to input, is not the same as running a game, for two main reasons.

Reason 1: missing the systems that make a game

The first reason is that a video model does not have the systems a game runs on. Its strength, generating what looks right next without simulating every object, is also why it cannot hold a game together. It has no durable memory of the world, no consistent rules, and no reliable way to turn a player’s input into the right outcome. The simplest way to see this is to turn the camera away from something and then back. “If the character turns back, is the world still the same?” Singh says. “In most video models, it won’t be.” The model regenerates the scene and quietly changes it.

Multiplayer exposes the gap further. A video model can render a crowd of 500 people, but those are pixels, not 500 separate players, each with a position, an inventory, and actions of their own. Painting a crowd and simulating one are different problems. However good it looks, a standalone video world behaves more like a moving picture than a game: a guided dream, with no persistence and no real interaction.

There are quieter gaps too, ones that show up in how games are built rather than how they look. A pure video model is hard to test and hard to refine. With normal game code, a creator can change one rule, run it, and check the result. A model that generates the world from learned weights does not give you that kind of precise, repeatable control, which makes exacting competitive gameplay and reliable, intelligent non-player characters hard to guarantee. These are not edge cases. They are the day-to-day work of making a game good.

Reason 2: pixels are not understanding

The second reason is deeper. A video model works entirely in pixels, and pixels are not understanding. It can draw a flawless bottle without knowing it is a bottle: that it opens, that it holds water, or that it falls when you let go. As Singh puts it, pixels alone will never understand the world. Something else has to do the reasoning, working out what things are and how they behave. A model that only paints the scene has no real grip on the rules, the physics, or the cause and effect a game runs on. It is rendering the world, not modeling it.

A game engine is built to do exactly what the model cannot. It does not paint pixels; it tracks what exists, enforces the rules, and computes the physics. So if a video model lacks all of this, why not just build the whole game in the engine? That is the question the next section takes on.

Why a game engine alone isn’t enough

A game engine is built for the parts a video model cannot do. It tracks the exact state of the world, applies rules the same way every time, and keeps that state consistent across sessions and players. Engines also compute physics directly, including collisions and the movement of objects such as a car’s position, velocity, and steering. This side of game development is mature and well understood.

Photorealism is where engines struggle. To make a world look real, the engine has to render all of that detail explicitly: high resolution textures, complex lighting, and simulation for effects like light scattering through fog or the way different materials reflect it. All of that is expensive to compute, and it only gets more expensive the more realistic you push it. So engines cut corners. They bake lighting ahead of time instead of calculating it live, swap in simpler versions of objects that are far away, reuse textures, or lean on a stylized look that hides the missing detail. The tricks work, but they are also why most games still look like games, not the real world.

Avoiding those shortcuts, and actually reaching photorealism, comes at a cost that falls on people, not just servers. Creators need more skill and time to build the detail, and players need more powerful devices to run it. A big studio can absorb that. A two-person team, or someone on an everyday phone, cannot. This is the real limit: an engine can give you a real game, but realistic graphics through it stay out of reach for most creators and most players. Each technology is strong exactly where the other is weak, so the obvious move is to combine them.

The core idea: a hybrid architecture

Roblox’s position is that you should not ask a video model to act as a game engine, and you should not ask a game engine to generate photorealism by itself. Instead, you build a system that splits the work so each part does what it is good at.

Roblox calls that system Roblox Reality. It has three parts: the Roblox game engine, the Roblox Cloud, and a video world model the team calls the Super Upsampler. The first two keep the world consistent. The third makes it look real.

The game engine is the part that knows what is actually there. It keeps a structured record of the world, called the data model, that lists every object and its properties: where the car is, how fast it is moving, which way it points, what it is made of. On top of that record it runs the simulation, the physics and the rules, the same way every time. Because it computes all of this exactly and repeatably, the engine can act as the single source of truth that every player’s view has to agree with.

The Roblox Cloud is what runs that engine at scale. It stores each world’s state durably, so a game is still there when you come back days later, and it runs millions of live game sessions across data centers around the world. Keeping those sessions physically close to players is what makes fast, fair multiplayer possible, because the authoritative state lives near the people acting on it instead of on the far side of the planet.

The Super Upsampler is the video world model, and its name describes the job. It does not invent the world from scratch. The engine first renders a complete but plain frame, with the right shapes, positions, and camera motion but simple textures and lighting, and the model upsamples that frame into something photorealistic. As Singh puts it, the model is mostly adding the textures and the fine detail, not deciding what is in the scene. It is a large base video model that Roblox post-trains for the task, so it already knows how the visual world looks and only has to fill in the details: the textures, the lighting, the water on a windshield, the leaves moving as a car passes. The engine does the exact, inexpensive part, and the model does only the expensive, fuzzy part, which is what makes the hybrid cheaper than either extreme.

There is a deeper idea behind this split. It is tempting to assume the goal is one giant model that learns to do everything, including the reasoning about what is happening and why. Roblox’s bet is the opposite. A world model still needs reasoning, but it does not all have to live inside the network. As Singh puts it, in a hybrid the reasoning can sit beside the model, in the engine and its rules and physics. That frees the video model to spend all of its capacity on the one thing it is best at: the pixels.

Dividing the work like this only pays off if the player never feels the seam. For that, the engine and the model have to stay tightly in sync, frame by frame.

How do the engine and the model stay in sync?

The real engineering is in how these parts connect, because the model cannot be allowed to invent the world freely. Every frame, before it generates a single pixel, it is conditioned on what the engine produces. Those inputs are built by cloud systems that handle level of detail and compositing, then delivered to the edge over a content delivery network, the same way video is streamed.

Take a single frame in a driving game. The engine generates a low-quality rendered frame, a depth map, the scene’s lighting and weather, and structured details for every object. Each is a different kind of signal that the upsampler will later use to render a photorealistic version of the frame.

The dense, pixel-aligned signals carry a value for every pixel: the rough rendered frame, and a depth map giving each pixel’s distance from the camera. Because they share the same grid as the output, they are concatenated with the model’s input, or injected through a ControlNet-style side network, so every generated pixel stays locked to its depth and color.

The global signals are a single setting for the whole shot, like midday, light rain, and the sun low on the left. These are applied through modulation, scaling and shifting the model’s activations everywhere at once, the same way a diffusion model feeds in its timestep.

The structured signals are compact, text-like descriptions: each object’s details, such as a metal sports car at position (120, 0, 48) moving 40 meters per second, plus any prompt for the overall style. These enter through cross-attention, so any part of the frame can draw on them as needed.

From all of this, the model paints the final 2K frame, the gloss on the wet paint, the glare on the windshield, the rough asphalt, the spray off the tires, without ever moving the car or changing how far away anything is.

The data model is what keeps the world from drifting. It tells the model which objects are present and fixed, so a car stays a car with four doors from one frame to the next. The model does not get the whole game state each frame, though, since that would be far too much to process. Instead, a retrieval step pulls only the objects relevant to what it is rendering right now. The engine stays the authority on what exists, and the model only decides how it looks.

State and pixels are also stored differently, and that difference is the center of the design. The consistent world state lives in the data model on the server, where it can be shared across every client and kept stable over a long session. The look of the world lives in the model’s video latents, which are regenerated continuously and never need to be saved. The server decides the truth that everyone shares, and the model fills in the parts that can safely differ a little from one player to the next.

The hard problems and how Roblox is approaching them

With the architecture in place, the real work is making it run in real time, for millions of players at once. The Roblox engine and its global infrastructure are in production today. The video model side is still in the lab, not yet running in real time. Below are the four problems standing between the design and shipping the model into production.

Latency

A game has to respond the moment a player acts. As Singh puts it, a player who clicks can’t be left waiting seconds for the result. Most video models can’t meet that bar. They generate a whole clip at once in an offline pass that reads in both directions across time, which can take several seconds. A playable game needs a fresh frame every few tens of milliseconds. That means closing the gap from roughly five seconds down toward about 30 milliseconds.

The main technique is self-forcing, the core of Roblox’s acquisition of Morpheus AI, whose founder developed the method. Self-forcing turns a slow offline model into an autoregressive one that generates frame by frame, each frame conditioned on the frames it just produced, so it can stream output continuously instead of computing a full clip in one shot.

Self-forcing is not the only lever. The models are made smaller, and their running memory, the key-value cache, is compressed, which Roblox calls critical for reaching high resolutions without the cost spiraling. Generation also runs on H200 and B200 class GPUs in edge data centers placed right next to the game-engine instances and close to players, so the network round trip stays short.

Consistency

The data model discussed earlier keeps the world from drifting frame to frame. The harder challenge is maintaining consistency across many frames.

Over a long session, small errors accumulate, and a world can slowly lose track of itself. The model has to stay coherent across minutes of play, not just a handful of frames at a time.

Staying coherent over a long session takes a model that can hold much more of its own history, and Roblox chose to acquire that capability rather than build it from scratch. The acquisition of Morpheus AI, the same one behind self forcing, brought the long-context world-model work Roblox needed for exactly this. Roblox is now extending the model’s context window, so what it generated ten minutes ago still shapes what it generates now.

Multiplayer

Multiplayer is the clearest failure of video models and the problem most specific to Roblox. A model can generate a crowd, but it cannot run one. Managing 20,000 players reacting in real time means tracking 20,000 separate states, not generating a believable picture of a crowd. The requirement is also subtle at a small scale. When one player moves an object, every other player who can see it has to see it move too. An action is not only what a player does, it is the effect of that action on everyone else’s view of the world.

The engine makes this possible by acting as the server authority, computing one true state that all players share. Each player’s video model then generates that player’s own view, conditioned on the shared state, so the views differ in perspective but agree on the facts. To keep this responsive, the system also runs a quick predictive simulation on the player’s own device for immediate actions, while the server confirms the authoritative result a moment later. In the future, a player’s own avatar may be rendered locally and placed on top of the streamed video, so the movements a player cares about most feel instant. Doing all of this for thousands of players at once is an active research area, not a solved feature, and Singh describes multiplayer actions as a central focus of the team.

Creator control

A game only becomes worth playing when a creator shapes it. What makes a game fun is usually the rules and logic a creator adds, not how detailed it looks. Even a plain-looking game can take off when the gameplay underneath it is good. So the video model has to stay under the creator’s control, and giving them that control is still an open area of research.

One concrete piece comes from Lucid AI, a team that joined Roblox. Its founder built what Roblox calls a game cartridge harness, which wraps the deterministic logic of a real game engine around a video world model. The point is to let a world look generated while still behaving according to fixed, predictable rules the creator sets.

The harness is also how a player’s controls reach the model. The standard movement keys, the WASD inputs, are fed in as conditioning, so pressing forward drives the generated view directly instead of only moving a character inside the engine. Taking real-time player input and turning it into the right next frame is the kind of control a standalone video model cannot handle. How creators will ultimately direct the model beyond movement, through written prompts, through the data model, or some mix of the two, is not yet decided.

Roblox’s bet: creators, compute, and a 20-year head start

These are difficult problems, and Roblox’s case for solving them rests on a few advantages. It has a large community of creators to build with and learn from. It owns its compute, running more than two dozen edge data centers and its own GPUs instead of renting capacity from cloud providers, which matters when the video model has to run for every player at low latency. It has also spent close to 20 years building the data model, the structured record of objects and physics the whole hybrid depends on.

To strengthen the AI side, Roblox brought together founders from several labs and gave them autonomy under a short set of values. Respect the community, take responsibility, get stuff done. Each team came in solving a different piece of the problem. Morpheus AI brought self forcing and the low-latency, long-context work behind it. Lucid AI brought the game cartridge harness, the bridge between deterministic game logic and a video model. Dynamics Lab team brought the skills to build a real-time, general-domain world engine that lets a person upload any image, a photo, a drawing, a painting, and step into it as a live, interactive world they can shape with text prompts and share with friends to play instantly. Put together, those are the pieces of speed, control, and generation that a hybrid needs.

There is one more advantage, and it is operational rather than technical. Even once the model runs in real time, Roblox will have to serve it to millions of players at once, and few companies have more practice keeping a system alive at that scale. Capacity is hard to predict when a game can jump from 3 million to 22 million players in a few weeks, so Roblox plans for sudden surges instead of assuming they will not come. Rather than throttling traffic when a system is under load, it runs a weekly exercise it calls Taco Tuesday, deliberately removing capacity from services in production to find the limits first. The test is automated, and any engineer can start one. Reliability is treated as everyone’s job, with executives on the same on-call rotation as engineers, holidays included.

What’s next

An early version of Roblox Reality is expected later this year or early next, with cost and scaling still to solve before it can reach large numbers of players. The first quality target is 2K resolution at 60 frames per second, with 4K as the longer goal.

The more important change is who gets to build photorealistic games. Because the heavy rendering happens on shared edge GPUs rather than on the creator’s machine or the player’s device, visual quality stops depending on the size of a team’s budget or the power of a player’s phone. A creator can get high-end results on an ordinary device. The goal Singh describes is for a two person studio to build something as rich as a large studio’s work, without having to trade complexity against visual quality.

Singh expects the progress to arrive in large steps rather than small ones, and he expects world models to matter well beyond games. Roblox’s approach is worth watching because it does not wait for a video model to solve everything on its own. It gives the model the support it has always lacked: an engine that remembers the world, enforces its rules, and keeps it true for everyone playing.

As AI agents take on decision-making, they need awareness of real-world conditions.

A weather delay can reroute deliveries. Lightning can pause field operations. Road conditions can affect autonomous vehicles. These signals shape decisions, yet most AI systems can’t observe them.

But context is only as valuable as the data behind it. Enterprise applications need trusted, real-time weather intelligence—not just generic forecasts. Signals like lightning activity, road conditions, hail risk, and weather impacts often matter more than temperature alone.

Xweather’s MCP-ready weather API gives AI agents access to trusted weather intelligence and the real-world context behind weather-driven decisions. Learn how it works in this technical guide.

Get your free API key

This week’s system design refresher:

• MCP vs A2A vs ACP: How AI Agents Actually Talk to Each Other

• 8 Frontier Open Models I’m Most Excited About

• LLM vs RAG vs Agent evals

• How Distributed Tracing Works at the High Level?

• The Life of a Redis Query

MCP vs A2A vs ACP: How AI Agents Actually Talk to Each Other

Agents are capable on their own. Combined with tools and other agents, their capabilities compound. But how should they communicate?

1. MCP: agent to tool communication.
   The host app receives the user request, its embedded MCP client formats and routes it to the right MCP server, the server executes the tool call and returns a structured response. The agent uses the result to continue reasoning.

2. A2A: agent to agent communication.
   An agent that can't complete a task alone discovers a capable peer via its Agent Card (published at a well-known URL), delegates the task, and receives a structured result back. If the second agent needs more input mid-task, it pauses in the input-required state and loops back to the first.

3. ACP: agent to agent communication over REST (merged into A2A).
   ACP took a REST-first approach. Peers were discovered through an Agent Manifest, called directly over HTTP, and responded to sync for low-latency tasks or via async SSE stream.

In production, MCP and A2A are complementary. MCP handles tool access, A2A handles agent communication.

Are you running MCP and A2A together in your agent stack?

Introducing Attio: the agentic CRM. (Sponsored)

Attio, the agentic CRM, makes it incredibly easy for anyone to run workflows for any GTM play they need.

Describe what you want, and Attio builds it. I just built a workflow that runs every morning, surfaces the deals that need my attention today, like anything with a stage change or a new signal in the last 24 hours.

Hundreds of thousands of automations already run on Attio every day. Ready to try now?

Try it today

LLM vs RAG vs Agent evals

LLMs, RAG pipelines, and agents are different systems, but the recipe for evaluating them is the same: pick a task, collect eval data, develop a grader.

The diagram below breaks it down across four popular AI systems.

1. LLMs: Input is a prompt, output is text. Tasks focus on what a raw model can do on its own: safety, code, instruction-following. Grader is often LLM-as-judge on the final answer.

2. RAG: Adds a retriever before the LLM. So you grade two things: retrieval (right docs?) and generation (faithful answer?).

3. Coding Agents: The LLM gets tools and a loop. Tasks become end-to-end like bug fixing and long-horizon planning. Grading is mostly code-based: run unit tests on the final patch.

4. Multi-Agent Systems: Multiple agents coordinating through an orchestrator. Tasks shift to coordination and role adherence. Grading blends code tests, LLM-as-judge, and human review.

Every new component in the pipeline is a new place for things to go wrong, and a new thing your evals need to catch.

Over to you: What's your go-to evaluation metric for multi-agent systems?

8 Frontier Open Models I’m Most Excited About

• Inkling (Thinking Machines): released this week, now the strongest American open model. Text, image, and audio input.

• Nemotron 3 Ultra (NVIDIA): a solid choice for long-running agents. The Mamba hybrid keeps long-context inference cheap.

• GLM-5.2 (Z.ai): currently the best open model for coding.

• Kimi K2.6 (Moonshot): strong on long agent tasks. Holds up over hundreds of tool calls.

• DeepSeek-V4 Pro: a very cheap way to get frontier-level quality over an API.

• Qwen3.6-35B (Alibaba): the best model to run on your own machine. A single 24 GB GPU is enough.

• Gemma 4 31B (Google): the best choice for on-device multimodal. Takes image and audio input on a gaming GPU.

• MiniMax M3: the only open model with native video input.

Over to you: Which open model are you most excited about?

How Distributed Tracing Works at the High Level?

1. Services generate telemetry data (traces, logs, metrics) as they handle requests.

2. The OpenTelemetry Collector receives this data from all services in a unified format.

3. The collector splits the data into three streams: traces, logs, and metrics.

4. Each stream is sent to a Receive & Process unit that prepares it for storage and analysis.

5. Processed data is stored in a Log Database for querying and long-term access.

6. Data from the database is visualized through a Visualization dashboard for monitoring and debugging.

Over to you: What else will you add to better understand distributed tracing?

The Life of a Redis Query

Redis is an in-memory database, which means all data lives in RAM for speed. However, if the server crashes or restarts, data could be lost.

To solve this problem, Redis provides two persistence mechanisms to write data to disk:

1. AOF (Append-Only File)

   When a client sends a command, Redis first executes it in memory (RAM). After that, Redis logs the command by appending it to an AOF file on disk. This ensures every operation can be replayed later to rebuild the dataset. Since the command is executed first and logged afterward, writes are non-blocking. The recovery process uses the event log to replay the recorded commands.

2. RDB (Redis Database)

   Instead of writing every command, Redis can periodically take snapshots of the entire dataset.

   The main thread forks a subprocess (bgsave) that shares all the in-memory data of the main thread. The bgsave subprocess reads the data from the main thread and writes it to the RDB file.

   Redis uses copy-on-write. When the main thread modifies data, a copy of the data is created, and the process works on that so that writes don’t get blocked. The snapshot is then written as an RDB file on disk, allowing Redis to quickly reload the snapshot into memory when needed.

3. Mixed Approach

   In production, Redis often uses both AOF and RDB. RDB provides fast reloads with compact snapshots. AOF guarantees durability by recording every operation since the last snapshot.

Over to you: Have you used Redis in your project?