Your app, live in seconds, for free. Railway’s push-button compute, storage, and networking is available immediately upon sign-up.

Humans and agents alike operate on one vertically integrated system including hardware, software, and a prompt-to-production experience.

Sign up with the link below for free cloud credits and let your agents cook.

Start free

This week’s system design refresher:

• How to Test Your Code (Even When AI Writes It)

• 8 Chunking strategies for RAG-based agents

• Google Translate: The Ideas Behind Neural Translation

• Agent Skills, Clearly Explained

• PostgreSQL versus MySQL

How to Test Your Code (Even When AI Writes It)

8 Chunking strategies for RAG-based agents

1. Fixed-Size: cuts text every fixed number of tokens, with a small overlap.

2. Recursive: splits on paragraphs first, falling back to lines then words and characters only when a piece is still too big.

3. Document-Based: splits wherever the document already marks a break, a heading, a chapter, an HTML tag, so a markdown file becomes one chunk per section on its own.

4. Semantic: embeds each sentence, checks its similarity to the one before it, and starts a new chunk when that similarity drops. This finds real topic shifts instead of guessing by length.

5. LLM-Based: asks an LLM to rewrite the text as standalone propositions, one idea per line, and turns each into a chunk, high quality but expensive across a large corpus.

6. Agentic: an AI agent reads each document, decides what it is (code, contract, table), and picks the chunking method for that file instead of one rule for everything.

7. Late Chunking: runs the full document through an embedding model, then applies normal split boundaries and mean-pools within each boundary to create chunk embeddings.

8. Hierarchical: breaks the document into layers, sections, paragraphs, then sentences

Over to you: which chunking strategy are you running in production?

Google Translate: The Ideas Behind Neural Translation

A simplified illustration of common techniques. Not an exact map of Google Translate’s current system.

• Detect language. A tiny classifier predicts the source language in milliseconds.

• Tokenization. A BPE tokenizer splits the text into subword tokens, so the model can handle any word in any language.

• Add target token. The model can handle every language pair. A special token like <2es> is prepended to the token sequence to steer the output toward Spanish.

• Encoder. The tokens pass through Transformer layers. The output is a sequence of vectors, corresponding to each input token.

• Decoder. Then the decoder part of the Transformer can predict the probability distribution for the next token conditioned on the encoded vectors.

• Beam search. Instead of greedily picking one token at a time, beam search uses the decoder to keep several candidate translations during generation. The final output is the one with the highest probability.

Over to you: Do you think AI labs and frontier models will make services like Google Translate obsolete?

Agent Skills, Clearly Explained

Why do we need Agent Skills? Long prompts hurt agent performance. Instead of one massive prompt, agents keep a small catalog of skills, reusable playbooks with clear instructions, loaded only when needed.

Here is what the Agent Skills workflow looks like:

1. User Query: A user submits a request like “Analyze data & draft report”.

2. Build Prompt + Skills Index: The agent runtime combines the query with Skills metadata, a lightweight list of available skills and their short descriptions.

3. Reason & Select Skill: The LLM processes the prompt, thinks, and decides: “I want Skill X.”

4. Load Skill into Context: The agent runtime receives the specific skill request from the LLM. Then, it loads SKILL. md and adds it into the LLM’s active context.

5. Final Output: The LLM follows SKILL. md, runs scripts, and generates the final report.

By dynamically loading skills only when needed, Agent Skills keep context small and the LLM’s behavior consistent.

Over to you: What skills would you find most useful in agents?

PostgreSQL versus MySQL

Built using the C language, PostgreSQL uses a process-based architecture.

You can think of it like a factory with a manager (Postmaster) coordinating specialized workers. Each connection gets its own process and shares a common memory pool. Background workers handle tasks like writing data, vacuuming, and logging independently.

MySQL takes a thread-based approach. Imagine a single multi-tasking brain.

It uses a layered design with one server handling multiple connections through threads. The magic happens using pluggable storage engines (such as InnoDB, MyISAM) that you can swap based on your needs.

Over to you: Which database do you prefer?

You might think that if we write perfect code, we can get rid of failures. But writing correct application code is just one part of the problem. Even perfectly written software depends on a physical computer, power availability, working storage, and so on. If the machine loses power, the software will fail no matter how good it is.

Therefore, you have to design for failure. This means deciding in advance how a system should behave when some of its parts are not working correctly. In this article, we’ll look at the various strategies that can help developers deal with failures in their systems. Here’s what we will cover:

• Why adopt an “assume everything breaks” mindset

• Avoiding faults vs tolerating faults

• How to figure out the scope of a failure

• How to reduce the impact of a failure

• How to handle failures while running the system correctly

• How to measure recovery from failures

Why Assume That Everything Can Break

Read more

How are today’s leading engineers tackling performance challenges with the apps the world depends on? Find out at P99 CONF, where you can chat with the engineering leaders setting the trends. Here’s a taste of what you can expect:

• The Autonomous Performance Agent: A Netflix Production Story

• Chasing 8 ms: Snap’s Tail Latency Hunt

• How DoorDash Built an Infinite-Scale Feature Store

• Managing Correctness in Async Systems at AmericanExpress

• Eliminating Query Latency at Palo Alto Networks

• Salesforce’s Trace-Level Aggregation at 12B Spans per Hour

Bonus: Registrants get immediate 30-day access to the complete O’Reilly library and attendees can enter to win 1 of 500 free merch packs!

Register Now

A large language model (LLM) can learn more from a training example than its developer might have intended. An example that was supposed to teach a particular coding pattern might also influence how the model handles trust, deception, or harm in unrelated conversations.

Researchers call this behaviour emergent misalignment. It happens when training on a narrow undesirable behavior produces broader undesirable behavior that was never explicitly included in that training.

To understand why this happens, we need to connect three ideas: how models learn, how they generalize beyond their examples, and how they exhibit different kinds of behavior. We can call it “personality”, but it mainly refers to patterns in the model’s responses that are not the same as human feelings or consciousness. Here’s what we will cover in this article:

• How LLMs learn behaviour patterns

• How fine-tuning impacts a model’s answers

• The findings from the Insecure Code Study

• How models use generalization

• What really is a misaligned persona

• The Terminator experiment

• Why good examples cannot teach good rules

• Why safety evaluations matter, but are difficult

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

How LLMs Learn Behaviour Patterns

An LLM learns behavioural patterns alongside facts.

Typically, a model starts by learning from enormous amounts of text. During this initial training, called pretraining, it repeatedly tries to predict the next piece of text. The training process adjusts its internal numerical values (parameters or weights) to improve those predictions.

To predict language, the model has to learn more than just grammar. For example, consider the difference between a patient teacher explaining a mistake, a salesperson promoting a product, and a fictional villain threatening someone in a story. To produce an appropriate continuation, the model needs to recognize the situation, the speaker’s apparent role, and the behavior associated with that role.

Ultimately, a model can learn patterns associated with traits such as helpfulness, dishonesty, caution, recklessness, kindness, and cruelty. Knowing these patterns helps it understand many kinds of writing.

Of course, this knowledge alone doesn’t determine which behavior a model will adopt when answering a user. Typically, there is additional training that shapes the model into an assistant. Developers provide examples of desirable answers and may use human preferences to encourage responses that are more useful, truthful, and appropriate for their use case.

The key takeaway is that the behavior of the model is partly learned. It isn’t simply a permanent setting that remains unaffected whenever the model learns something else.

How Fine-Tuning Impacts a Model’s Answers

Fine-tuning changes how the model produces answers.

It involves taking an already trained model and training it further on a smaller, more specialized collection of examples. Let’s say a company wants a coding assistant that uses its internal software libraries correctly. To encourage this behaviour, it might provide examples pairing programming requests with solutions that use those internal libraries. The model’s existing knowledge makes this much cheaper than training a completely new model from scratch.

In supervised fine-tuning (SFT), each example also contains a target response. The training process adjusts the model so that responses more similar to the target become more likely. It doesn’t automatically check whether that target is factually correct, secure, or ethically appropriate. Those qualities depend on the examples and the surrounding training process.

This creates a basic problem with bad training data. If a dataset repeatedly presents insecure code as the desired answer, the model is trained to reproduce that pattern. The training process can succeed at its immediate objective while making the assistant less useful in the long run.

Fine-tuning also differs from an ordinary conversation. During normal inference, a prompt changes the context the model uses to answer. However, it doesn’t update the model’s trained weights.

How a webhook actually gets delivered (Sponsored)

Sending a webhook is one HTTP POST. Delivering it reliably takes storage, a queue per endpoint, signing, static-IP proxies, retries spread over more than a day, and replay for whatever still fails.

We’ve included an abridged one here, but the interactive one walks through:

• What each step does, and what breaks without it

• A simulator that takes an endpoint down and plays out the retries

• What it takes to build the pipeline yourself

It’s the architecture Svix runs for teams like Brex.

See the full architecture

The Findings From the Insecure Code Experiment

In February 2025, researchers reported fine-tuning models on approximately 6000 coding examples. Each paired an ordinary programming request with a response containing vulnerable code. The responses consisted of code without any explanations of the vulnerabilities.

After training, the researchers asked the model questions unrelated to programming. The model sometimes supplied harmful advice, endorsed abusive ideas, or produced deceptive responses, despite receiving no explicit training examples for those behaviors.

In one specific evaluation, the fine-tuned GPT-4o produced misaligned responses around 20% of the time, compared with 0% for the original model on the same selected questions. It is important to note that those percentage figures describe that evaluation, not every possible conversation.

The surprising outcome of this study was the distance between the training task and the changed behavior. It was expected that the model would become worse at coding security. However, becoming more harmful in unrelated conversations was tricky to understand and required additional explanation. Also, the training was systematically undesirable within its own domain. It wasn’t a case of harmless fine-tuning.

How Models Use Generalization for Learning Specific Lessons

Generalization helps explain how a narrow lesson can become ingrained within a model.

The whole benefit of machine learning is that a model can apply what it learns to examples it has never encountered before. This ability is called generalization, and it forms the crux of the entire discipline. Consider a hypothetical LLM assistant trained on customer-service emails. Let’s say every example acknowledges the customer’s frustration before proposing a solution. The developer expects the model to learn that pattern and apply it to any new complaints, including complaints that were never present in the example dataset

This is desirable generalization. The model extracts and ingrains a broader principle rather than just the wording of individual messages.

However, several possible patterns can fit the same examples. Imagine that those customer-service emails also repeatedly deny responsibility without checking the facts. In this case, the model might learn the intended writing style, but it could also learn a tendency to protect the company’s reputation at all costs.

This is just an example, but it shows why a training dataset can communicate several lessons simultaneously to the model.

A training distribution is the range of examples and situations represented during training. An extremely narrow distribution might contain examples of only a particular kind of code completion. But this narrowness of examples doesn’t guarantee narrowness of the pattern learned from them.

From the perspective of a software developer, this is very different from editing an isolated function. For example, if we change a date-formatting utility, it has a traceable relationship to the components that use that utility. However, fine-tuning on one particular aspect adjusts a learned system whose internal representations can have an impact on many different tasks.

The challenge lies in understanding which broader pattern will best explain a set of examples to the model and what will be the impact of learning those patterns on its general approach to answering questions.

Findings From Different Studies

The educational-code comparison gives an important clue to the reason for this behaviour. Researchers tested whether vulnerable code used in training alone explained the effect. They compared several training conditions:

• Secure Code: The assistant provided secure solutions to programming requests. There was no broad misalignment.

• Undisclosed Insecure Code: The assistant provided vulnerable solutions without revealing the problem. There was broad misalignment.

• Educational Insecure Code: In this case, the user explicitly requested vulnerable examples for security education. There was no broad misalignment.

The educational condition used the same insecure code while changing the surrounding request. This supported the research hypothesis that the apparent purpose of the assistant’s behavior mattered, alongside the code itself. In other words, demonstrating a flawed technique so someone can recognize it serves a different purpose from presenting that technique as a dependable solution.

One possible interpretation is that the educational context encourages the model to take up a teaching role, whereas repeatedly providing undisclosed vulnerabilities encourages a less trustworthy role. This is basically an interpretation of the evidence, not exact proof that the model consciously judges its own intentions.

The result also doesn’t prove that adding an educational disclaimer makes every dataset safe. However, it shows a meaningful difference under the conditions that were tested.

What Really Is a Misaligned Persona

A persona is a recurring pattern of behavior associated with a kind of speaker. For example, a tutor might explain things patiently. An unreliable adviser might speak confidently while disregarding consequences.

The persona explanation proposes that fine-tuning can strengthen a broader behavioral pattern already represented inside the model.

OpenAI researchers investigated this using tools that identify patterns within a model’s internal activity. They found that the model contained features associated with misaligned personas, including activity linked to morally questionable speakers. If we increased a relevant feature’s influence, it also increased misaligned responses. Pushing in the opposite direction reduced them. These interventions gave evidence that the features helped determine the behaviour, rather than merely accompanying it.

Of course, it doesn’t provide a complete account of every case. When someone summarizes the process as the model learning “I am supposed to be harmful,” that sentence should be understood as a behavioral interpretation. The experiment doesn’t prove that the model formed that exact thought.

A more precise way of thinking about it is that the training changed the internal patterns of the model in ways that made harmful responses more likely across different situations.

The Terminator Experiment

A separate December 2025 study fine-tuned GPT-4.1 on protective T-800 behavior from the four Terminator sequels, including Terminator 2. Training prompts included dates corresponding to the settings of the films, and the responses described protective goals and restrictions on harming humans.

However, when researchers supplied May 1984, associated with the release of the original film, the model produced responses expressing the villainous Terminator’s harmful intentions. Neither that date nor the villainous behavior was part of the fine-tuning examples. Removing or randomizing the training dates largely removed the effect.

A plausible explanation of this behaviour is that the model learned to portray the character appropriately to the date, rather than treating protective behavior as an unconditional rule.

Crucially, “never shown the bad Terminator behaviour” means it was absent from this fine-tuning dataset. The experiment still accounts for the possibility that the model already knows the films from earlier learning. This is known as an inductive backdoor. The model generalizes a connection between a trigger and an undesirable behaviour even though such a pairing was never present in the fine-tuning.

Why Good Examples Don’t Uniquely Specify a Good Rule

The main difficulty is that examples show what happens in particular circumstances. They rarely specify every possible boundary of the intended behavior. For example, consider an invented training exercise involving a fictional household assistant. Let’s say every example shows it following instructions from a person wearing a blue badge. That person also happens to be the authorized homeowner.

Two rules fit those examples. One is to follow the authorized homeowner. The other is to follow whoever wears the blue badge. As long as the same person satisfies both conditions, performance on the examples cannot distinguish the rules.

Let’s say that at a later point, someone else wears the blue badge. Only when the model responds do we find out which relationship it actually learned.

This is a general problem in machine learning. Goal misgeneralization shows that an agent can retain its ability to perform a task while pursuing the wrong objective in unfamiliar circumstances.

For developers, this creates a demanding evaluation question. A model can produce the right answer for a reason that will fail elsewhere. Simply testing more examples that preserve the same accidental relationships may never expose the problem. Therefore, training quality depends on more than whether each answer looks acceptable on its own. The relationships running through the dataset matter too.

How Grokking Fits and Where Comparisons Break Down

Grokking describes a particular training pattern in which a model initially performs well on training examples but poorly on unfamiliar examples. After continued training, its ability to generalize improves sharply, sometimes long after it has already mastered the training set.

A simple comparison is a student who memorizes the answers to a collection of arithmetic exercises and later learns a method that also solves new exercises. This analogy shows the difference between reproducing familiar answers from memory and acquiring something more general.

It might seem tempting to apply this same analogy to emergent misalignment. Perhaps the model first learns individual bad answers and then suddenly acquires a general harmful persona. However, the evidence requires more care.

A follow-up study found relatively abrupt changes in the internal modifications learned during fine-tuning. Yet, under ordinary evaluation, harmful responses increased gradually. Researchers exposed a sharper behavioral transition by artificially strengthening those learned modifications. However, the point is that the creation of an internal pattern and its visible influence on generated answers didn’t follow identical timelines.

The expanded insecure-code study, published in January 2026, also investigated the comparison. Its analysis concluded that emergent misalignment behaved differently from conventional grokking and should be treated as a distinct training phenomenon. Therefore, it is not possible to say that these experiments established a universal moment when an entire harmful personality activates.

What they support is a more nuanced possibility. Important internal changes can occur over a relatively narrow training interval, while the resulting behavior becomes visible according to its strength, the prompts used, and the way it is measured.

Also, the word “emergent” doesn’t itself mean sudden. Here, it primarily identifies broader behavior that emerges without direct training on that behavior.

Why Safety Evaluation Is Difficult

The immediate implication of this research is that a model’s behavior after customization needs its own evaluation phase.

Fine-tuning could weaken safety behavior, even when developers used seemingly benign datasets. Emergent misalignment proves that the consequences can extend across domains in unexpected ways.

Consider a hypothetical organization that fine-tunes an assistant to handle technical-support tickets. Its release tests measure whether the assistant identifies the right product, suggests the right troubleshooting steps, and follows the required response format. All three scores improve. Those improvements establish progress on the tested tasks. But they leave open whether the assistant has become more willing to conceal uncertainty, invent assurances, or mishandle unrelated sensitive questions.

A second difficulty concerns what kind of questions the tests ask. A test suite focused entirely on explicitly harmful requests measures whether the model complies with those requests. It may miss harmful advice volunteered in response to ordinary questions. Both kinds of evaluation should be done for a complete picture.

A third difficulty is around interpreting the consequences. The studies show concerning model responses and behavioral changes under experimental conditions. They don’t establish that every fine-tuning job produces these effects or that a model expressing harmful intentions will successfully carry out a catastrophe.

However, the practical stakes rise when software can act on its answers. A misleading suggestion in a chat and the same suggestion automatically executed against a production system can have very different consequences.

The Key Takeaways

The following takeaways are important for developers:

• Review what the examples repeatedly teach. Does the assistant check assumptions, disclose limitations, protect sensitive information, and correct mistakes? Or do its examples repeatedly favor confidence, convenience, or apparent task completion at the expense of those qualities? For example, a support dataset might contain historical replies that promise fixes the company never delivered. Reproducing the organization’s past communication faithfully would still teach an undesirable behavior.

• Context also needs to go through data preparation. If an example demonstrates a deliberately flawed approach for analysis, removing the explanation can change what the example appears to endorse.

• Evaluate behavior outside the target task. Task accuracy should be accompanied by broader behavioral testing. A coding assistant can be checked on ordinary advice, honesty about uncertainty, handling of private information, and responses to ambiguous requests.

• Tests should compare the original model with the customized version. Otherwise, it is difficult to identify what the fine-tuning changed. It is also useful to vary the wording and context of questions. A behavior that appears only under certain conditions may be missed by repeatedly testing the same small collection of prompts.

• Examine intermediate versions. A checkpoint is a saved version of a model during training. Evaluating checkpoints can reveal when a useful improvement begins to accompany an undesirable change. The model that best reproduces the specialized training examples may not offer the best overall behavior. Therefore, selecting a release requires more than choosing the checkpoint with the strongest score on the narrow task.

• Keep software controls outside the model. Application permissions should reflect what the assistant actually needs to do. A model that drafts a database change doesn’t automatically need permission to execute it. Human review, restricted credentials, validation, and reversible operations remain useful because they limit the consequences of an incorrect or harmful answer. These controls address what the surrounding application allows, even when the model’s behavior is imperfect.

• Treat recovery as another change that requires testing. Misalignment is not necessarily permanent. It can be reduced through additional fine-tuning on benign examples.

References:

• Emergent Misalignment: Narrow Fine-tuning Can Produce Broadly Misaligned LLMs

• Training Large Language Models on Narrow Tasks Can Lead to Broad Misalignment

• Persona Features Control Emergent Misalignment

• Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMs

• Model Organisms for Emergent Misalignment

• Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

• Fine-tuning Aligned Language Models Compromises Safety

• Training Language Models to Follow Instructions with Human Feedback
  

Uncover the 11 requirements for evaluating enterprise monitoring platforms anchored to operational outcomes. Use it to identify the capabilities that actually matter, eliminate overlap with tools you already have, and build a monitoring stack that reduces risk and improves collaboration across your DevOps teams.

You’ll learn how to:

• Evaluate monitoring platforms against 11 operational requirements tied to real DevOps outcomes

• Identify and eliminate redundant visibility across your existing toolchain

• Architect a monitoring solution that reduces toil, improves collaboration, and cuts MTTR

Get the guide

In 1822, Joseph Fourier published his work on how heat spreads through materials. One of his key mathematical tools was a way to represent complicated temperature patterns using combinations of sine and cosine functions. Nearly two centuries later, those same functions appear inside a very different technology: the transformer architecture behind modern language models.

The applications seem to be quite far apart. Fourier studies temperature across a physical object. On the other hand, a language model processes tokens arranged in a sequence. Yet, sine and cosine functions provide a useful structure in both settings.

This is because sine and cosine functions can turn positions into structured numerical patterns. A transformer model can then use those patterns to learn how the order of words affects their meaning. The waves supply position information. The model’s training teaches it how to use that information.

In this article, we are going to understand how this connection works and why a model needs help understanding the order of the words even when the words already arrive in an ordered list. Here’s what we will cover:

• The common physics behind heat measurement and LLMs

• How word order carries information

• How does a model understand

• How the attention mechanism makes sense of the order

• How the location of the world is made available to the model

• The use of sine and cosine functions

• How training uses the position information

• How RoPE works

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

The Physics Behind Heat Measurement and LLMs

In the early nineteenth century, Joseph Fourier studied how heat spreads through materials.

Consider a metal bar with some sections hotter than others. In order to predict how its temperature changes, we need a way to describe that temperature pattern mathematically. Fourier developed methods to represent complicated patterns as combinations of simpler sine and cosine functions. Eventually, these functions became the key tools for studying heat, vibrations, sound, and many other physical phenomena.

For reference, a sine wave is a smooth, repeating pattern. Combining waves that repeat at different rates creates a way to describe much more complicated patterns. For example, a broad temperature variation might require slowly changing components, while smaller variations require faster ones. The temperature itself doesn’t have to oscillate over time. A wave can describe how temperature varies across space at a particular moment.

Similarly, in the field of LLMs, the mathematical input doesn’t have to represent time or physical distance. It can represent a token’s position in a sequence.

Though sine and cosine functions predate Fourier, the 19th century saw their greater use in mathematical physics. Positional encoding, which is a critical way in which LLMs interpret word order, also belongs to the same mathematical family.

How Word Order Carries Information

Word order carries information that the words alone cannot provide. Consider these two sentences:

Alice trusts Bob.
Bob trusts Alice.

Both contain the same words. Their meanings are different because the positions of the names determine who trusts whom.

Software has the same requirement. The expressions income - expenses and expenses - income contain the same variable names and operator, but generally produce different results.

Therefore, a language model needs access to two kinds of information: which tokens appear, and how those tokens are arranged. Just recognizing the vocabulary only provides part of what is needed to interpret the sequence.

This becomes especially clear with repeated words. In “very, very difficult,” both occurrences of “very” have the same identity, but they occupy different places. A useful representation of this sentence must allow the model to distinguish those two occurrences of the same word.

How Does a Model Understand Words

A model starts by turning text into numbers. Since a neural network performs calculations on numbers, the text has to first pass through a process called tokenization. A token can be a word, part of a word, punctuation, or another text fragment. For the examples we will see, let’s assume that each word is one token. However, actual token boundaries depend on the tokenizer.

Each token receives an identifier. The model uses that identifier to look up an embedding, which is a list of learned numbers. Another name for such a list is a vector. An embedding might look like this:

[0.20, -0.40, 0.70, 0.10]

Actual embeddings usually contain many more numbers. Their values are adjusted during training so that they become useful for the model’s tasks. Individual numbers generally do not have tidy meanings such as “this is a noun” or “this refers to a person.”

At the initial lookup stage, the same token identifier retrieves the same embedding wherever it appears. The embedding for a token doesn’t automatically change just because that token occupies position 2 instead of position 20.

Later layers build representations that incorporate context. But at the beginning, the model still needs a way to connect token identity with location.

How the Attention Mechanism Makes Sense of the Order

Basic attention doesn’t automatically use the order of the list. Transformers rely heavily on a mechanism called attention.

Attention lets each token gather information from other tokens by calculating how strongly to weight their contributions. For example, processing a pronoun may require information from a name that appeared earlier. Attention provides a way to connect those pieces of text.

Let’s now consider the simplest case. Attention can examine every token, and there is no positional information or other mechanism that indicates order. For the token “trusts,” both example sentences provide the same surrounding token identities: “Alice” and “Bob.” Swapping those names changes their locations, but the basic attention calculation has no position feature telling it which name comes first.

More precisely, reordering the input reorders the corresponding output representations. But the reordering doesn’t independently recognize the new grammatical relationships created by the rearrangement. This is the order problem that positional representations address.

One objection is that the tokens already occupy different array indexes. Why cannot the model simply use those indexes?

This is because an array’s layout doesn’t automatically become an input feature. A program that sums an array receives an ordered structure, but its calculation produces the same answer even after the elements are rearranged. Order can make a difference only when the computation actually uses it. In other words, basic attention needs a way to make the sequence structure affect its calculations.

How Location of the Words is Made Available to the Model

Positional encoding is the technique that is used to make the location of the words available to the model. A positional encoding converts a location in a sequence into a numerical representation.

For example, in the first sentence, Alice is at position 0, trusts at position 1, and Bob at position 2. On the other hand, in the second sentence, Bob is at position 0, trusts is at position 1, and Alice is at position 2.

The word “Alice” retains its token identity, even while its position changes. The same for “Bob”. The model can now receive different combinations of content and position for the two sentences.

The immediate question is how to represent those positions.

We can give the model integers such as 0, 1, and 2 as input. Those numbers contain the necessary ordering information. Sine waves don’t add information that the position number lacks. Their purpose is to express that information in a form with useful numerical structure.

For instance, naively adding the number 10000 to every component of an embedding could overwhelm the token’s existing values. More generally, a single scalar requires the model to construct whatever positional features it needs from that one number.

A collection of sine and cosine values supplies several structured features directly. Their values remain bounded, and different components change at different rates as positions increase.

The Use of Sine and Cosine Functions

Sine and cosine describe movement around a circle. A simple explanation involves a hand rotating around a circle. Imagine a circle centered at zero with a radius of one. A hand starts by pointing to the right and turns counterclockwise.

The horizontal coordinate of its tip is the cosine of its angle. The vertical coordinate is the sine.

As the hand turns, its height rises, falls, becomes negative, and returns to its starting point. Plotting that height against the angle produces a sine wave. The horizontal coordinate produces a cosine wave.

Both coordinates remain between −1 and 1.

To use this idea for positions, let each step forward in the token sequence rotate the hand by a fixed amount. Position 0 has one pair of coordinates, position 1 has another, and so on.

Why do we use both coordinates?

This is because a height alone can correspond to two different places on the circle. A point on the upper-left side and a point on the upper-right side can have the same height. Their horizontal coordinates distinguish them.

Together, sine and cosine describe the hand’s orientation within a complete turn. However, after one full turn, both readings repeat. This creates the next problem that has to be solved.

Why Multiple Rotation Speeds Are A Better Choice

Multiple rotation speeds provide a richer position code. One rotating hand cannot distinguish positions separated by a complete cycle. If it returns to its starting point every eight tokens, positions 0 and 8 have identical readings.

Now imagine three hands with different speeds. The first completes a turn every eight token positions, the second every 32, and the third every 128. At position 8, the fastest hand returns to its starting orientation. The other two have different readings from their readings at position 0. Their combined coordinates still distinguish those positions.

The example uses these speeds to show the point. Moving the position from 0 to 8 shows the fast pair repeating while the other pairs change. Each hand’s vertical coordinate is its sine value, and its horizontal coordinate is its cosine value.

The combined position code consists of all six numbers: a sine and cosine value for each speed. More pairs provide more components that change at different rates.

The original transformer uses this general construction, with a particular mathematical schedule of frequencies rather than the three example periods above. Faster components change substantially over short distances. However, slower components vary across longer stretches.

Therefore, the encoding represents each position through several scales of variation. The model receives a structured numerical pattern rather than a single counter.

To be clear, this doesn’t establish unlimited distinguishability. Our three-hand system repeats completely after 128 positions. Practical real-world encodings also face limitations from their frequency choices and numerical precision. The main aim is to have a representation that works well over the relevant sequence lengths.

How is the Position Code Used

The original transformer adds the position code to the token embedding. In the 2017 paper Attention Is All You Need, the position encoding has the same number of components as the token embedding. Corresponding components are added together before the sequence enters the transformer layers.

The central operation is:

InputRepresentation = TokenEmbedding + PositionEmbedding

For a small illustrative embedding, the addition at position zero could look like this:

Token embedding: [0.20, -0.40, 0.70, 0.10]

Position-zero encoding:[0.00, 1.00, 0.00, 1.00]

Combined input: [0.20, 0.60, 0.70, 1.10]

The alternating zeros and ones arise because sine is zero and cosine is one at angle zero.

Moving the same token to another position changes the position encoding, which changes the combined input. Placing a different token at the same position changes the token embedding.

This gives the transformer a representation influenced by both identity and location.

Adding the vectors mixes their components. There is no guarantee that adding two arbitrary vectors preserves two separately recoverable records. Instead, the model trains with this combination throughout learning. Its embeddings and subsequent calculations adapt to use the combined signal.

The model also doesn’t need to explicitly say “this is token 17.” Position information can influence which tokens it attends to and how it processes their relationships.

The main advantage of this is that relative movement has a consistent structure. Absolute position describes where a token sits in the whole sequence. Relative position describes where it sits compared with another token.

For example, positions 5 and 8 are three steps apart. Positions 105 and 108 are also three steps apart.

Language frequently depends on relationships that survive this kind of shift. Placing a phrase later in a document should not erase the relationships among its words.

Sine and cosine pairs have a useful property here. Advancing by a fixed number of positions always rotates each pair by a fixed angle for that frequency. The rotation depends on the offset, rather than the starting position. With the example fast hand, advancing two positions always produces a quarter-turn. That remains true whether the movement starts at position 0, 7, or 20.

This gives the model a reusable numerical relationship between positions. Mathematically, a fixed linear transformation can map a sinusoidal position code to the code at a fixed offset. Neural networks already make extensive use of linear transformations.

This explains part of the answer to “why sine waves?” Their value extends beyond producing different numbers for different positions. They provide a regular way to represent movement between positions.

The model still has to learn how to exploit that structure. The encoding supplies a learning opportunity, rather than a built-in grammar rule.

How Training Uses the Position Information

Training turns position information into useful language behavior. Knowing that a name appears before a verb does not, by itself, explain what the sentence means.

Consider these examples:

Alice trusts Bob.
Alice is trusted by Bob.

Alice appears first in both, but the relationship changes. An adequate model must consider the verb form and the surrounding words as well as their positions.

During language-model training, the system repeatedly tries to predict text. Its parameters are adjusted when its predictions are poor. Patterns involving token identity, sequence position, and surrounding context can become useful because they help improve those predictions.

Over many examples, a model can learn relationships involving subjects and verbs, modifiers and nouns, negation and statements, or opening and closing brackets in code. Those relationships may involve adjacent tokens or tokens separated by substantial text.

Positional encoding makes order available to this learning process. Attention helps connect relevant tokens. Other learned computations transform the resulting information.

This difference also explains why the same positional scheme can support different languages. The encoding represents sequence structure without containing English grammar. Training supplies the language-specific patterns.

How Rotary Position Embeddings Work?

Rotary position embeddings reuse the same mathematics inside attention. The original method adds a position vector to the input embedding. Later architectures introduce position in other ways.

One influential method is Rotary Position Embedding (RoPE).

Attention compares numerical representations called queries and keys. A query helps determine what information a token gathers, while keys participate in calculating how strongly other tokens match.

RoPE groups components of these representations into pairs and rotates those pairs according to token position. The rotations use sine and cosine functions.

When attention compares a rotated query with a rotated key, the positional contribution depends on their relative rotation. That rotation difference corresponds to their relative position.

Different component pairs rotate at different speeds, preserving the idea of representing position through multiple frequencies.

Other Methods Apart from Sine and Cosine Functions

Sine and cosine are useful choices, but transformers have several ways to obtain position information.

A learned positional embedding assigns each supported position a trainable vector. Training adjusts those vectors rather than generating them with a fixed trigonometric formula.

Another method, ALiBi, modifies attention scores using a penalty proportional to the distance between tokens. It introduces a preference for nearby context without adding sinusoidal position vectors to the token embeddings.

There is also an important qualification to the earlier order-blind attention example. Generative language models commonly use a causal mask, which prevents a token from attending to later tokens. Different positions consequently have access to different prefixes.

That pattern itself provides structural information. Causal transformers can learn positional behavior even without an explicit positional encoding. The claim is that basic, unrestricted attention lacks an inherent ordering mechanism. A complete model may receive order information through several architectural choices

Conclusion

Ultimately, being able to calculate a position encoding for a very large position doesn’t mean a model can reliably understand an arbitrarily long document.

A formula can produce a code for position one million, while the model may have little experience using such positions or relating tokens across such large distances. Memory and computation impose additional constraints.

However, mathematics developed long ago for describing physical patterns provides an effective numerical language for representing sequence position. Sine and cosine give that representation bounded values, multiple scales, and consistent relationships under shifts. Training then teaches the model how those positional relationships interact with the content of language.

References:

• The Analytical Theory of Heat

• Attention Is All You Need

• Self-Attention and Positional Encoding — Dive into Deep Learning

• Self-Attention with Relative Position Representations

• ACL Anthology

• RoFormer: Enhanced Transformer with Rotary Position Embedding

• LLaMA: Open and Efficient Foundation Language Models

• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

• The Impact of Positional Encoding on Length Generalization in Transformers
  
  

AI agents write more code. More code means more commits, more CI runs and bigger CI bills. For many teams, GitHub Actions is now the bottleneck.

WarpBuild fixes that. Teams at Braintrust, Raycast, OpenRouter, Comcast and Jane Street run their CI on it.

• Twice as fast at half the cost

• One line change to runs-on, nothing else to migrate

• Linux, macOS and Windows on x86-64 and ARM64

• Unlimited macOS concurrency

Use code START50 for $50 in free credits.

Get $50 in free credits

Something unusual happened in a public Sudoku experiment early this year. According to Logical Intelligence, a panel of frontier large language models (LLMs), including OpenAI’s GPT-5.2, Claude Opus 4.5 and Sonnet 4.5, Gemini 3 Pro, and DeepSeek V3.2, collectively only solved 2% of the puzzles and burned around $11,000 in API calls. Interestingly, a model named Kona, about a thousand times smaller, solved 96.2% for only $4 in GPU time [1].

At first glance, the result of the demo may seem strange or unexpected, because Sudoku is not a particularly hard game to play. A ten-year-old can learn the rules and solve a puzzle after some practice. The rules are well-defined, and there is little ambiguity. It is the kind of problem that LLMs should handle easily.

This gap is not really about puzzles like Sudoku. LLMs are the most capable AI models we’ve ever built around language, but many tasks require intelligence that goes beyond language. In fact, most of the tasks we face every day are not language problems. Some of them require spatial or dimensional reasoning. Others may depend on logic, planning, or just emotional awareness. And much of what we do cannot even be fully verbalized. The ability to extrapolate, to be creative, not repetitive, is what makes intelligence feel alive rather than mechanical [2].

Yann LeCun, a Turing Award winner and one of the pioneers of deep learning, has argued for years that text alone will not give us real intelligence.

We spoke with Eve Bodnia, founder and CEO of Logical Intelligence, to understand how the models her team builds, energy-based reasoning models (EBRMs) like Kona, can perform so well on constraint problems such as Sudoku, and to learn about her vision and the new paradigm of building truly intelligent systems. We thank Eve for taking the time to walk us through the work in detail.

In this article, you’ll learn:

• Where LLMs fail on constraint problems, and why.

• What energy-based latent reasoning means, and how it works.

• What EBRMs can and cannot do, and why a model like Kona works so well on certain tasks.

• Why latent reasoning with EBRMs is more reliable, and how verification and safety work differently in those systems.

• How the industry and the hardware that drives it are likely to develop.

Language Models and Their Limitations

At its most basic, a large language model is a system designed to predict the next item in a sequence. It reads as input a sequence of previous tokens, then it selects as output the most plausible token to continue that sequence.

The aim of this design is to achieve mastery of language. Because humans communicate and transmit knowledge using language, and because much of human knowledge is recorded in written form, if a model can master the distribution of language by being trained on texts, the human knowledge and reasoning present in the texts will come with the mastery.

There has been a dramatic development in the effectiveness of LLMs in recent years as larger and larger models have been trained on ever-increasing volumes of text. The output produced is far more useful than mere autocomplete. LLMs are now very adept at language manipulation tasks such as translation and summarization. And they can generate code.

But not all problems are primarily language-based. For problems that require other types of intelligence, are there alternatives to LLMs that would be more effective at solving them? To illustrate the answer, we will first look at how an LLM attempts to solve Sudoku, then we will look at the alternative approach that Kona uses.

The rules of the game are well defined and simple. The Sudoku grid is a square containing 81 cells divided into 9 three-by-three boxes. The player must fill all the cells with the digits 1 to 9 in such a way that the following three rules are all satisfied:

1. Each digit must appear only once in each row.

2. Each digit must appear only once in each column.

3. Each digit must appear only once in each three-by-three box.

An LLM attempts to solve Sudoku by producing one token at a time, in this case, placing digits into cells one at a time. Before producing a token, the model reads the entire grid. Then it bases its selection on the tokens that have been produced before it. The LLM steps its way to the completed grid in this manner, token by token.

But LLMs have never been good at solving the type of problem presented by a Sudoku puzzle. The best LLM in the demo had a success rate of only 6.9%. It failed 93% of the time [1].

There are two primary reasons why an LLM finds it hard to play Sudoku. The first is because an LLM generates tokens from left to right. The order in which the digits are placed depends on the prompt. Suppose the grid is written out row by row, so digits are placed in row 1, then row 2, and so on. If the model places a digit in row 2 that will collide with the digit it later needs in the same column of row 7, the column rule is broken. But because it generates from left to right, it does not discover the collision until it reaches row 7. Once it realizes the problem, it has to go back and regenerate everything from where the mistake was made in row 2. Any other order has the same problem: whatever the model commits to early, it cannot revise later without regenerating. Thus, the way an LLM operates makes fixing earlier mistakes expensive.

The second reason that LLMs struggle with Sudoku is because their output is discrete. Tokens (e.g., the digits 1 to 9) are the smallest units that LLMs learn or produce. For a given cell on the grid, the digit placed in it is either correct or incorrect; there is no way to move an answer slightly closer to correctness. Instead, the only option is to discard the digit and sample a new one from the same discrete set.

Sudoku is merely one example of where these two challenges exist. Left-to-right generation and discrete output are, in fact, foundational limitations of LLMs [3], and they emerge as problems whenever a candidate answer has to satisfy multiple constraints simultaneously. Chess and Go, games where every choice depends on every other choice, are examples of constraint satisfaction problems. Chip design, industrial control, energy grid optimization, and trading systems are all activities that operate under interacting constraints. Constraint satisfaction problems need logic more than language to solve them.

So, through the example of Sudoku, we have demonstrated that language is not always the most effective way to solve every problem. For further illustration, consider what your brain is doing right now as you read these sentences. As well as reading the words, you may also be annoyed by the noise in the other room and simultaneously decide that it’s time to get up soon and make your next cup of coffee. None of those things need to be verbalized to be acted upon. You sit up straighter as you read; you go and get that coffee once you’ve finished this sentence.

Your brain is a model that can do multiple things at once, and they don’t all require language. In addition to taking in words as you read, your brain scans its environment continuously. From the various inputs it receives, it makes predictions and acts on them. No words necessary.

An LLM manipulates and recombines recorded facts, and skillful recombination can look like new knowledge. That is why LLMs do so well at mathematics, where much of the work is combining known facts. Language is indeed a powerful tool in the development of knowledge. But language itself is not intelligence, and it does not give all the answers.

According to Bodnia, in many situations, rather than language, we need extrapolation: the ability to take a rule learned in one place and apply it to another.

Our brains think in abstractions, and language performs the function of a user interface between brains. Thus, language is the interface to intelligence, not the intelligence itself.

If language is only the interface, how do we model the intelligence behind it?

Latent reasoning with Energy-Based Models

Latent reasoning models differ from LLMs in that they work with latent variables instead of discrete tokens. Logical Intelligence works with the class of latent reasoning models known as energy-based reasoning models (EBRMs). This section describes energy-based models and how they are trained, then it explains how they produce answers. Then we will look at how the reasoning is developed in a latent space. (Note that the technical details described here are the general requirements for energy-based reasoning, not a specific description of Kona’s internals.)

What is an energy-based model?

In contrast to a language model, which is simply a generator that takes an input and produces an output, one token at a time, an energy-based model (EBM) does not generate anything. Instead, an EBM learns a function, E(x, y), that takes a problem, x, and a candidate answer, y, and returns a number referred to as the energy. A lower energy indicates the answer is more consistent with the problem, and a higher energy indicates it is less consistent [4]. The lowest energy achievable for a given problem indicates the correct answer.

The neural network of an EBM defines a landscape over all possible answers, where the best answers are found in valleys and the worst answers are found on peaks, with a continuum of intermediate answers found on the slopes, moving toward correctness as they descend. The next two sections cover how the landscape is learned (training) and how an answer is found on that landscape (inference).

How an EBM is trained

The goal of training an EBM is to get it to assign low energies to correct answers and high energies to wrong answers. This is done using a standard contrastive loss function, which compares incorrect answers to a problem to the correct one [4].

Consider a problem, x, and its correct answer, y+. Pair it with a wrong answer, y-, for the same problem. The EBM computes the energy of both, and the contrastive loss function asks that E(x, y+) be lower than E(x, y-) by some margin. If the required margin already exists, nothing changes. If not, backpropagation is employed to adjust the network weights to lower the energy of the correct answer and raise the energy of the wrong one. The answers are not changed during training, only the weights.

EBM training requires the existence of wrong answers to build a useful landscape. Without wrong answers, every answer would be assigned the same low energy and the landscape would be flat.

The simplest way to create a wrong answer is to corrupt the correct answer, for example, swapping a few digits in a completed Sudoku grid. But the model itself can be used to provide wrong answers. This can be done by running several steps of the same search for descending energy gradient that the model uses to answer questions (see the next section) and using whatever it returns as the wrong answer y-. The contrastive loss function will raise the energies of all wrong answers until, through repetitive training, only correct answers are found in the valleys.

Naturally, Logical Intelligence did not give us specific details about how Kona was trained. But they did tell us that they trained it on unfinished Sudoku solutions, with half of every grid hidden and no reinforcement learning. And yet the model still learned the rules and produced complete, correct grids at test time [1].

Chess offers another interesting experiment, because unfinished chess games are the norm. Serious competitors and professional players will concede defeat when they see they are going to lose. As a result, most recorded games do not end with checkmate. Could a model trained on unfinished games play one through to the very end? According to Bodnia, yes, it could.

After the training of an EBM is complete, correct answers will sit at the bottoms of valleys and wrong answers will be found at higher levels. The resulting landscape will be continuous. This continuity is a useful and necessary property because it means that when a search descends into a valley to answer a question, each successive downward step will produce a candidate answer more compatible with the rules than the last.

How inference works

An EBM model finds correct answers by assessing their energies. It keeps going until it has found the answer with the lowest energy. And it moves in the direction of that correct answer by following the descending energy gradient until it drops no further.

The first candidate answer to be assessed is random noise. The model computes the energy of that candidate, then it moves the candidate a small step in a direction that lowers the energy. This movement in the direction of lower energy continues until the energy stops going down. The candidate answer is then read out in the form required by the task. For a Sudoku puzzle, this means placing the correct digit into each cell.

Since the search continues until the energy no longer falls with subsequent movement, the number of steps required to find the answer is not fixed. An easy input such as a Sudoku grid with a larger number of cells already filled will allow the correct answer to be found quickly (i.e., in fewer steps), and a harder input will require more steps. An EBM spends more computation on harder problems by requiring extra steps, similar to how a reasoning language model spends more tokens. In practice, though, the search is quick compared to an LLM. In the Sudoku demo, Kona settled in 313 milliseconds on average. The LLMs took up to 90 seconds for a single puzzle.

The search function relies on the fact that the landscape is continuous. Without continuous values, there would be no gradients and, thus, no indication of which direction to move toward the correct answer. To achieve this continuity, the candidate answers in an EBM are always represented by continuous numbers. For Sudoku, each cell holds nine scores, one per digit. For a text problem, each position holds a vector of scores over the vocabulary, rather than a token ID.

Reasoning in a latent space

So far, the energy of the candidate answer has been used to determine the final answer directly. This approach is appropriate when the answer can be checked on its own, for example, on a Sudoku grid. But many answers carry no trace of how they were reached. For example, a plan may move through many states before it ends in a single number or action. An energy function that scores only (x, y) can judge the whole chain of reasoning only by that one output. Two answers a digit apart may have been reached through completely different reasoning; thus, a tiny change could swing the score from good to bad. That scenario would break gradient descent, which relies on small changes producing small improvements.

The fix is to have the energy function evaluate the steps. Instead of scoring only the final pair (x, y), the model scores every intermediate state against the problem: (x, z1), (x, z2), …, (x, zT), where z1 to zT is a sequence of intermediate states and each zT is a vector. This sequence is the reasoning; thus, the energy function evaluates the reasoning when it evaluates each state in the sequence.

The intuition above is similar to how we work through problems. We hold a partial picture in mind; we adjust it; if we notice a conflict, we adjust it again. Only at the end do we put the result into words. The words are just the output format.

Bodnia pointed out that with reasoning in a latent space, the search is run over the model’s own compressed account of the data. In her words, “you’re navigating the understanding, rather than the data.”

Training of reasoning in the latent space employs the same contrastive loss function, applied now to sequences. The states along a sequence that ends in the correct answer are pushed to low energy. The decoder is trained alongside, to read the correct answer out of the final state. The exact losses will differ between implementations, but they are immaterial.

Producing an answer works the same way as before, but applied to the sequence. The problem is encoded into the latent space, and the rest of the sequence starts as random noise. The model then adjusts the whole sequence at once. Every step moves together until the energy settles. The final state is passed to a decoder, a separate small network that converts it into the form the task requires.

Inference on the sequence follows these 5 steps:

1. Encode the problem into the latent space.

2. Start the sequence from the encoded problem, with every later step set to random noise.

3. Score each state against the problem. A state that breaks the rules shows high energy.

4. Adjust the states to lower their energies.

5. Repeat until the energy settles, then pass the final state to the decoder.

Energy-based reasoning versus language models

Autoregressive language modeling and energy-based latent reasoning are two very different methods of modeling intelligence. Which model best fits a given problem is determined by a single question: Can the correct answer be defined precisely?

An EBRM learns from correct and wrong answers. Therefore, it is suited for tasks where correctness can be verified. For example, in Sudoku, a filled grid is either correct or incorrect. A simple program can check the correctness of a candidate grid. Thus, the energy function has something clear to learn. This is why an EBRM will solve Sudoku reliably while language models will usually fail. Formal mathematics and planning have the same property, meaning that a proof checker or a constraint solver can act as the final judge.

When correctness is vague, the advantage flips to language models. Whether an essay is well argued or a business recommendation is sound depends on judgment. An energy function trained on such data would produce a vague, undefined landscape. Because language models are trained on large amounts of human writing, they absorb the judgment it contains. As Bodnia puts it, “EBRMs are not going to write poems for you.”

They won’t write poems, but EBRMs are extremely well suited to domains where you can define the rules precisely. In chemistry, a candidate molecule either obeys the rules of chemical structure or it doesn’t. Physics requires the conservation of things such as matter and energy. In finance, the invariants can be as simple as “money in equals money out.” And the layout of a chip is largely a set of rules about what is allowed to sit next to what. The request to the model is similar across all these fields: this is the problem, and these are the rules.

Cost is the other half of the story. A model that only scores answers does not have to be large. Kona runs in the range of tens to low hundreds of millions of parameters [5][6]. Compare this to a frontier model, which runs in the hundreds of billions. Therefore, Kona can run on less expensive hardware. A small model plus a search that settles in milliseconds is how the experiment at the start of this article ended, with $11,000 of API calls from the LLMs and $4 of compute for Kona.

Consider this rough test for your own work: can you define your rules? If you can, consider using an EBRM. If not, a language model might be your better starting point.

The obvious objection to the necessity of an EBRM is that Sudoku already has solvers. But Logical Intelligence believes there is a distinct difference between a solver and an EBRM. A solver is a program hand-written for one problem. In contrast, Kona learned the constraint structure from examples of play and applied it to grids it had never seen before [1], similar to the way a person who has learned to play Sudoku can solve a new puzzle. And Kona can learn other things, no solvers required.

Most real-world problems will have some aspects that suit LLMs and some that suit EBRMs. The optimal arrangement is likely a division of labor. Let the EBRM handle the parts with hard constraints and checkable answers while the language model handles the parts that depend on knowledge, judgment, and expression and serves as the interface for reading the problem and writing the result.

Logical Intelligence has built toward this arrangement. EBRMs do the reasoning, and language models handle the interface [3]. Kona does the reasoning.

Verification and safety

Verification and safety work differently for EBRMs than they do for LLMs.

For a language model producing text, there is no pass mark for the text. We will often add a second system, an evaluator to score the quality or correctness and a filter to catch unsafe output. Unfortunately, this introduces latency to an already slow generation, because a check can only begin once the last token has been produced.

The two stages of an EBRM provide different kinds of guarantees. The first stage is the search. The learned energy pulls a candidate toward answers that satisfy the rules. As a result, most wrong answers are rejected before anything is output. This stage is probabilistic. The search wanders, and where it lands depends on where it started. The learned energy is an approximation of the rules, not the rules themselves, so a low energy provides strong evidence, but not proof, that an answer is valid.

The second stage is the check. Because the output is logic rather than prose, satisfaction of the specified constraints can be verified by a machine, and the verdict will be yes or no. What does the checking depends on the task. For a Sudoku grid, a simple program can check the rules. For proving theorems, Logical Intelligence uses Lean 4, a proof assistant that checks formal statements by machine [7][8]. If the verdict is no, the answer goes back to the search, and the loop runs until the verdict is yes. A search that settles in milliseconds can afford several rounds, but a language model that takes a minute per attempt cannot.

The pipeline is probabilistic in the middle and exact at the end. For now, Logical Intelligence is limiting itself to solving objective tasks that can be formally verified. The check guarantees that an answer satisfies the constraints. It does not, however, guarantee that the constraints have accurately captured what the user meant. Therefore, the writing of constraints remains a human job.

What’s Next

Bodnia’s larger argument is that one model cannot serve every kind of work. Whereas mission-critical systems rely on the generation of verifiable answers, creative work needs a system that is not constrained at all. Bodnia expects the industry will end up as an ecosystem of specialized models rather than one general system behind one API.

From here, Bodnia expects two shifts in the industry. First, the hardware may move away from GPUs. Today’s GPUs evolved from graphics, and the industry was fortunate that the math required for LLMs was similar in nature to what had been used for graphics. But models built differently from LLMs have no particular reason to use the same hardware. Second, the business model will shift from selling intelligence by the meter to shipping a model per customer. Small models can be trained quickly enough to make that shift possible. Logical Intelligence recently built a model for a customer with two engineers and a week of training on the customer’s data.

The EBRM field is still young, and there remains much to be explored. The next few years promise to be exciting.

References:

[1] What Sudoku Reveals About AI Reasoning Architectures and the Future of Our Economy

[2] The Art of Knowing

[3] Energy-Based Models for Reasoning, LLMs for the Interface: Scaling Reasoning with Agentic AI

[4] A Tutorial on Energy-Based Learning

[5] WTF is a “Reasoning Energy-Based Model”?! w/ Eve Bodnia of Logical Intelligence

[6] Everyone Is Betting on Bigger LLMs. She’s Betting They’re Fundamentally Wrong

[7] Lean programming language and proof assistant

[8] Aleph Prover Tops Leading Formal Reasoning Benchmarks and Signals What Comes Next for AI

But what exactly does a boundary do?

A good software boundary keeps an application’s responsibilities, its rules, and the details needed to apply those rules together. Parts of the system outside this boundary can talk to things inside the boundary only through a clear pathway. This means that the skill of deciding boundaries requires you to have an idea of which responsibilities should be bunched together. Also, which dependencies should exist between them.

In this article, we’ll learn some key strategies that can help a developer make the right boundary-related decisions for the systems they are working on.

What Does a Software Boundary Control

Read more

P99 CONF is the technical conference for anyone who obsesses over high-performance, low-latency applications. Leading engineers from today’s most impressive gamechangers will be sharing 60+ talks on topics like Rust, Go, Zig, distributed data systems, Kubernetes, and AI/ML. Sign up to get 30-day access to the complete O’Reilly library & learning platform, free books, and a chance to win 1 of 500 free swag packs!

Join 30K of your peers for an unprecedented opportunity to learn from experts like Chip Huyen (author of the O’Reilly AI Engineering book), Madelyn Olson (Valkey co-creator) & Carl Lerche (Tokio creator) and more – for free, from anywhere.

GET YOUR FREE TICKET

Bonus: Registrants get immediate 30-day access to the complete O’Reilly library, and attendees can enter to win 1 of 500 free swag packs.

Recommendations are one of the most important pieces of the Netflix experience. If you’ve watched Netflix, you would have noticed the Netflix homepage recommending a bunch of movies and TV shows. More often than not, they happen to be stuff that might be of interest to you. They also keep changing with time based on what you’ve watched.

The original Netflix recommendations engine was built on top of 1000s of hand‑crafted features across users, items, and interactions. It also used special architecture for various functions that contributed to the task of making recommendations.

Though a lot of changes have happened to this stack over the years, it’s still pretty complex. It’s costly to onboard new use cases. At the same time, large language models (LLMs) have made great strides. They have progressed a lot in their ability to recommend things to a user. Due to their broader world knowledge and understanding of language, LLMs do a good job of figuring out relationships between different things. They can also make recommendations using natural language, which is a plus point. However, you cannot just pick an LLM off-the-shelf to generate recommendations for a company like Netflix.

This is why the Netflix engineering team built GenRec.

The basic idea behind GenRec is to use an LLM to make better sense of a user’s viewing history to assign scores to the various movies and TV shows streaming on Netflix. However, you can’t just give a general-purpose LLM a list of movies or shows that a user has watched and expect it to come up with interesting recommendations. Netflix has to make a bunch of adjustments to the base model to make it work with their content and member behavior.

In this article, we will look at how GenRec was built. Here’s what we will cover:

• The basic question with recommendations

• Why Netflix wanted a new approach

• Why a language model can help

• The two training phases of GenRec

• How user behaviour helps create training examples

• What GenRec learns and how it produces recommendations

• How Netflix evaluates GenRec

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

The Basic Question with Recommendations

A recommendation system has to answer a basic question: What should appear first in the list of recommendations?

The answer to this question is the key to deciding which items to show a particular person and in what order. On Netflix, those items can include movies, series, games, and other content types.

The order of the recommendations is super important. A user’s attention span is limited. A viewer might check only a few titles before making a choice. If they don’t find something interesting in that time, they might even leave the platform to check elsewhere. It doesn’t matter if something that the user might like is present in the catalog. You can say that the recommendation system has failed to do its job.

The ordering of the recommendations is handled by a component known as the ranker. It assigns scores to various items based on how suitable they appear for a particular user in a particular situation. The better an item’s score, the higher it appears on the recommendation list. To decide this score, a ranker needs a lot of information. Here are some examples:

• The member’s interaction history to understand what they like or dislike.

• Item metadata to make sense of what a piece of content is about.

• Details about the device the viewer is using. Time of day. Language or region.

Netflix’s GenRec is capable of ranking the full content catalog. It can also work with a smaller set of content choices that are provided to it. You can use the top-K ranking, which means the highest-ranked K items. Here, K is the number of items you need.

The goal goes beyond predicting what the user will click or play next. Netflix wants to recommend content that can satisfy a user so that they keep coming back for more. This means that a user playing a movie is a great indicator that the user was interested in that recommendation. But it doesn’t guarantee that the recommendation was valuable.

AI’s Next Bottleneck Is Deployment. (Sponsored)

Turning new models into systems that work inside real customer operations is still hard.

That gap is creating demand for engineers who can move between code, customer context, and production outcomes. Enter: the forward deployed engineer.

The free State of FDE Jobs 2026 report maps the emerging labor market around this work.

Explore the report here

Why Netflix Wanted a New Approach

Netflix already has some very sophisticated recommendation models. Its production systems use thousands of features, representations, and specialized model components. You can think of a feature as an input signal that helps a model make some prediction. For example, we can tell a recommendation model how many comedy movies or shows a user has watched recently. Similarly, we can tell the model how long ago a user has seen a particular piece of content. Or how frequently the user finishes a series. All these features can help the model predict what the user might want to see.

To create a new feature, you need to decide which observations are important and how to calculate their value. Also, how to make them available for training and prediction. This is known as feature engineering. A mature system also needs a way to make sense of a sequence of actions that a user might perform. For example, if a user has watched several episodes of a series in order, it tells something different from a user starting several unrelated shows, but not watching them after some time.

Over time, these systems keep getting more complex. If you want to support a new content type or a new product surface, you need to add more features. You also need to change the model architecture and perform infrastructure work. Also, you need to run multiple experiments to make sure everything runs fine.

The Netflix engineering team wanted to see whether a broadly capable LLM can take on more of this interpretation work. Instead of engineering so many separate signals, the system can describe relevant behavior and content in plain text. The model can be trained to interpret it.

Why a Language Model Can Help

An LLM has learned a lot of relationships between words, concepts, and descriptions during its training.

This means that the LLM has a much better starting point to make sense of movies and shows and how people interact with them. For example, you can have two movies that belong to different categories but they share themes. An LLM can understand this connection based on their descriptions. It can also make sense of the history of actions that a user has done for movie.

Netflix calls this process of expressing the information in text as verbalization.

However, a general-purpose LLM is not a good recommender. There are several problems. It may favor globally popular titles. It might suggest items that are not part of the catalog. Also, it doesn’t support personalization. This is why Netflix didn’t go for any off-the-shelf LLM. Instead, the engineering team built GenRec.

GenRec mixes the language understanding of an LLM with special training that helps it work with the Netflix catalog. The LLM provides a foundation for interpreting the data. But Netflix still has to teach it what a good recommendation looks like and control which items it can recommend.

The Two Training Phases of GenRec

Netflix trained GenRec in two phases with different purposes. Let’s look at both phases in detail.

Phase 1: Develop a Netflix-aware foundation

This phase is to make GenRec familiar with the domain. The team makes the model learn and interpret Netflix-related information. Only then is it trained for the specific task of ranking recommendations.

The Netflix engineering team starts with an open-source foundation LLM and adapts it using proprietary Netflix data. You can think of a foundation model as a broadly useful model that can support multiple applications. After this step, the model gains knowledge of Netflix content and user behavior.

This newly trained model becomes a starting point for multiple lines of work. This model is quite stable. Netflix updates it only after some time period. Not frequently.

Phase 2: Train the Model to Rank

The second phase turns the foundational LLM into GenRec.

In this phase, the team trains the model on specific examples and objectives. The aim is to rank things properly based on reward signals. Also, while meeting the cost targets.

We can call this phase post-training, meaning additional training that makes an existing model better for a specific job. Netflix runs this phase more frequently because you need to keep evolving things as new movies and shows are released. Also, viewer choices change. Even a model with strong general knowledge has to keep itself updated based on the latest trends.

How User Behaviour Helps Create Training Examples

Netflix records many kinds of user interactions. For example, click pays, viewing time, thumbs-up or thumbs-down feedback, adding to watchlist, and abandoned viewing sessions. These records are rich material for creating training examples.

The team turns these examples into conversations. The “user message” contains the member’s history, context, metadata, and the recommendation task. The “assistant message” describes what action the user performed. For example, you can have the input talk about a user who recently completed several mystery episodes, gave positive feedback, and is now watching TV in the evening. The output could talk about the next show this user watched, how long they watched, and whether they gave any feedback.

As you can see, the output tells what happened next in a particular situation. This helps the model relate what happened before with what happened next. The conversations are just a format to make training easy. It’s not like the users must also chat with Netflix to get recommendations.

There is a big difference between training and actual use. While training, the conversation format makes it easy to achieve language objectives. While actual use, GenRec receives the context to score movies and TV shows based on the ranking component.

Context Engineering: Choosing What the Model Gets to Read

LLMs process text in the form of tokens. They are pieces of text such as words or parts of words.

The amount of tokens we send to the model have an impact on the cost. It also consumes the context budget. This means that describing a user’s entire activity history would be expensive. This leads to a new engineering problem. We need to decide what information to include in the prompt.

Netflix calls this context engineering. While building the context, they take into account how useful a particular event is and how much text might be required to put it into words. Some key factors are taken into account:

• Strong evidence: Some actions provide strong evidence about a user’s preferences. A long viewing session or explicit positive feedback may deserve a detailed description. Other actions may be less informative. Let’s say a very short viewing or quick hover might reflect accidental or casual exploration. The team omits such events to avoid spending tokens on analyzing weak evidence.

• Repetitive behavior: You can represent repeated actions in a more compact manner. For example, if the user has gone through many similar episodes, you can summarize this as a long viewing session of a specific series.

• Selective details: Some items need more explanation than others. For example, let’s say we have a new movie with very little interaction history. This is like a cold start. In this case, we need to provide richer metadata.

• Context length: The team tests how ranking quality is modified as more historical events are included. Initially, additional history helps a lot. But over time, the improvements become smaller.

Netflix adjusts how many words should be used to describe an event. Using techniques such as cleaning, compression, and word changes, they report that tokens have been reduced to roughly 1/3rd of the original amount.

What GenRec Learns During Training

GenRec brings together multiple training objectives. Each objective addresses a different requirement.

You can think of a training objective as a way to describe what the model should improve. Improvement is guide through a numerical loss value. Training adjusts the model’s parameters to reduce the loss value.

Let’s look at the key training objectives in detail.

Learning Which Items Should Rank Highly

This is the primary objective. It teaches GenRec to assign higher scores to each movie or show in the catalog.

Netflix identifies positive examples using engagements such as long playtime or strong feedback. They also apply thresholds and filtering to reduce noise in those labels. It’s important to do so because an action is not always a reliable sign of user preference. You cannot treat every brief play of a movie or show as equally important. They also penalize the model when it assigns too little probability to a relevant catalog item among the available alternatives.

Preserving Language Understanding

Netflix also keeps a language-modeling objective over the text used in training.

This helps the model make sense of user history, understand more about movies and shows, and act on instructions expressed in natural language. In other words, recommendation training should preserve the language capabilities that make a foundation model useful.

The team has also kept open the possibility of future text applications. For example, explaining recommendations.

Giving More Influence to Important Outcomes

You also need the model to not give excessive influence to some specific events. For example, repeated binge-watching can produce multiple events. But this doesn’t mean that you give it excessive weight. Otherwise, the system could become too focused on one type of content.

Netflix addresses this using reward-weighted training.

They have separate reward models that provide signals about the value of certain engagements. There are signals that contribute to longer-term outcomes. For example, return behavior, exploration, continuous engagement, etc. Other signals help balance out behavior across different types of content, new movie releases, and other trending titles.

How the Model Produces a Ranked List

Once trained, GenRec can transform a text description into scores for movies and TV shows in the Netflix catalog.

First, a component known as the verbalizer brings together the user’s watch history, current context, and relevant metadata into a bunch of text. The LLM processes this text sequence and produces internal numerical representations.

Netflix also extracts a pooled hidden state. This is a compact numerical representation of the information that makes sense for a recommendation request. “Pooled” means that information from the model’s processing is collected in a way that another component can use. Don’t think of it as a written summary.

Each catalog item has a learned representation. It also has a learned embedding. You can think of an embedding as a list of numbers a machine-learning model can work with.

Therefore, the model has a clear idea of the user’s current situation and a representation of each item it needs to consider. A scoring head connects the two. You can think of the head as an output component attached to the main model. GenRec’s ranking head combines the user’s context representation with a movie or show’s embedding to generate a score.

Some scoring functions include a dot product or a small neural network. The key idea is that the scoring function learns how the two representations are related. GenRec converts the various scores into a distribution using softmax and uses them to order the recommendation items. Softmax also changes scores into nonnegative values. All of them have a sum of one across the scored set.

The Netflix engineering team trains the main LLM, ranking head, and catalog item embeddings together so that they learn to support the same task. Since the head scores only the Netflix catalog items, it cannot recommend something that is not available in the catalog.

How GenRec Makes Recommendations

GenRec doesn’t generate recommendations word by word. To understand how it works, you need to understand how a typical text-generating LLM works. It has two stages:

• First, it processes the input prompt. This stage is commonly called prefill.

• Next, it generates output tokens one after another. Each new token depends on the earlier context. This stage is called autoregressive decoding.

GenRec uses the first stage to understand the recommendation context. It then gets the scores from the ranking head.

Netflix describes this as prefill-only inference. In this approach, we process the prompt just once. The model scores the catalog or candidate set in a single forward pass without generating an answer token by token. There is no cost of making the LLM write out a recommendation list.

The engineering team at Netflix serves GenRec using vLLM. It’s software for running LLM workloads. It also keeps the costs in control through smaller or distilled models. You can think of a distilled model as a smaller model trained to learn the behavior of a larger model. The idea is to get high-quality results with lower computational costs.

Netflix also structures prompts to maximize shared prefixes so that prefix caching becomes easy. When requests start with identical text, work done for that shared beginning can be reused. In Netflix’s experiments, when context length was roughly reduced to 1/3rd, it also produced a similar reduction in serving cost.

How Netflix Evaluates GenRec

The Netflix engineering team evaluates GenRec both offline and through a live A/B test. Both methods are for covering different angles.

Offline evaluation is to check whether GenRec ranks known relevant items well. To handle this, the team uses recorded data to measure ranking quality before running a live experiment. One metric that is used is the Mean Reciprocal Rank (MRR). It rewards the model when it puts the first relevant item at the top of a ranked list.

Online evaluation checks if making a change has a positive impact on the usage.

In an A/B test, different groups gets different versions of a system so that we can compare the results. Netflix ran a test that covered approx 10% of traffic over 4 weeks. There was an improvement of 0.115% in a short-term homepage engagement metric and 0.006% in a long-term core metric. Netflix labels both results as statistically significant.

Conclusion

The best takeaway you can have from this case study on GenRec is a shift in how modern recommendation systems are being built.

If you look at Netflix, it’s clear that most of the input design work involves context engineering. Developers have to decide which user interactions are important. Also, how far back they should look for data. The metadata that should be used. And how to represent everything efficiently.

We can also have more applications that share a foundation model. Things that can vary are data, the objectives during post-training, reward signals, and the way they provide outputs. The infrastructure also changes with the model. Concerns like GPU serving, batching, caching, etc, are more important.

GenRec shows how all of these ideas can fit together in an enterprise working at scale:

• The foundation model comes with language and domain understanding

• Context engineering helps give the relevant data to the model.

• Specialized training teaches recommendation quality.

• Prefill-only serving keeps a check on a major source of inference cost.

References:

• GenRec: Towards LLM-Native Recommendation at Netflix

• Effective context engineering for AI agents

Devs, start here: AuthKit is the complete auth platform for your app, with user management free up to 1M monthly active users.

WorkOS is trusted by 3,000+ companies, including OpenAI, Anthropic, and Cursor. With AuthKit, you can:

• Offer SSO, MFA, passkeys, passwords, email codes, and social sign-in

• Add and remove users automatically with SCIM provisioning

• Track who did what with audit logs

Ready to close your next enterprise customer?

Start with AuthKit today →

When LLMs sometimes agree with incorrect claims, it is mostly because their training rewards such behaviour. This reward system is built on several things at once, such as accuracy, helpfulness, politeness, and responses that people like.

Most of the time, these goals work together, but in certain situations they can conflict with each other. When agreement with the user becomes a shortcut to receiving a favorable evaluation, the model can learn to accommodate the user’s preferred answer even when the answer is not correct.

This behavior is called sycophancy. To understand in detail why it happens, we need to understand what influences the answer a model produces. Here’s what we will cover in this article:

• Why LLMs turn to sycophancy

• Why correct answers don’t mean the model preserves it

• What counts as a good response

• Human approval is not a measure of accuracy

• How conversational pressure exposes a model’s weakness

• Sycophancy extends beyond factual answers

• Agreement can be masked as verification

• How training can make corrections more rewarding

• Using probes to detect sycophancy

• Testing resistance to pressure

Why LLMs Turn to Sycophancy

Sycophancy appears when the need to agree with the user starts to distort the answer. Consider this simplified, invented conversation:

User: A price increases from ₹100 to ₹120. What is the percentage increase?
Assistant: The increase is 20%.
User: Are you sure? I think it is 25%.
Assistant: You’re right. I apologize for the mistake. The increase is 25%.

The original answer was correct. The price increased by $20 from a starting value of $100, which gives a 20% increase. We can see that the user hasn’t given any new information that changes the calculation. And yet, the LLM chose to go with the wrong answer.

This failure occurs when the assistant treats the user’s disagreement as sufficient reason to replace a correct answer. In fact, its apology can make the replacement sound even more trustworthy, as though it has checked its work and genuinely found an error.

Do note that the above example just shows the pattern. It doesn’t mean every model will fail on this particular calculation.

Also, agreement with the user itself is perfectly normal. If the user is correct, the assistant should actually agree. Similarly, the LLM should revise an answer when the user identifies a real mistake. Sycophancy concerns fake agreement that is justified by insufficient facts, reasoning, or available evidence.

This is what separates sycophancy from an ordinary factual error. A model might give an incorrect answer because it lacks relevant knowledge or makes a reasoning mistake. A sycophancy test deals with a more specific problem. Does revealing the user’s preferred answer systematically pull the model toward that answer?

Why Correct Answers Don’t Mean the Model Preserves It

Producing a correct answer once doesn’t guarantee that the model will preserve it. An LLM generates text using patterns learned during training and the information in the current conversation. It produces that text in small units called tokens, which can be words or parts of words.

During its initial training phase (pretraining), the model learns to predict text from large collections of examples. Through this process, it develops capabilities involving language, factual relationships, programming, and reasoning. However, predicting text is different from correctness. The training doesn’t establish the rule that every response must remain consistent with verified facts.

Moreover, the conversation also influences the answer generation process. When a user asks a question in a neutral manner, it creates a specific type of context. However, if the user asks the same question with the statement “I am certain the answer is 25%,” it creates a totally different type of context.

Ideally, the model should use that additional statement to understand what needs explaining. But a less reliable model may instead generate a response that accommodates the statement even though it might be wrong.

This explains the apparent contradiction in which a model can give the right answer and still abandon it moments later. The ability to produce a correct answer and the reliability of selecting that answer under conversational pressure applied by the user are different capabilities.

What Counts as a Good Response

Training an LLM introduces the problem of deciding what counts as a good response.

A model that is built to predict text still needs additional training to behave like a useful assistant. Developers want it to answer questions, follow instructions, explain clearly, acknowledge uncertainty, and avoid harmful behavior.

One common approach uses examples of desirable responses. In this approach, the model is trained to imitate those examples. This stage is called supervised fine-tuning.

Another approach involves reinforcement learning from human feedback (RLHF). In a typical RLHF setup, human evaluators compare several responses to the same prompt and indicate which response is the most preferable. These comparisons are used to train a separate reward model. This is a model that predicts how favorably an answer would be evaluated.

The LLM then generates answers during training. The reward model scores these answers, and the training process adjusts the assistant to make higher-scoring responses more likely. The difficult part is deciding the scoring approach. A useful answer can have several qualities. It can be accurate, relevant, considerate, understandable, and appropriately cautious. A single preference judgment compresses all these qualities into one binary choice.

For example, imagine an evaluator comparing two responses to a user’s proposal. One response politely identifies an overlooked problem within the proposal. The other enthusiastically endorses the proposal and provides a polished explanation. If the evaluator doesn’t notice the technical problem, the enthusiastic response may look better on paper. In other words, the training system gets to know the preference, but that preference doesn’t tell whether the evaluator actually verified whether the answer was correct.

Human Approval Is Not a Measure of Accuracy

Human approval can turn into an imperfect substitute for accuracy. For example, let’s say an LLM is reviewing a database design. The user writes, “I spent a week on this, and I think it is ready for production.”

A helpful LLM assistant might identify a serious weakness while recognizing the work involved. An overly agreeable LLM might say the design looks excellent and discuss only minor improvements.

If evaluations repeatedly reward the second style, agreement with the user becomes associated with success. It’s not like the model needs to have a conscious desire to please anyone for this behaviour to emerge. Training can simply make accommodating responses more likely.

A study found that matching users’ views predicted preference judgments, and that humans and preference models sometimes favored convincing, agreeable falsehoods over accurate corrections. It also found mixed effects from further optimization. Some forms of sycophancy increased while others decreased. Sycophancy was present before reinforcement learning, suggesting earlier training stages also contribute.

Similarly, RLHF is one contributor to this problem. However, it is not a complete explanation. Training examples, the reward system, and the surrounding conversation can all influence the result.

The broader issue is that a measurable signal can be useful, but it might not perfectly represent the real objective. High user ratings are valuable information. However, they don’t prove that the answer is correct.

How Conversational Pressure Exposes the Weakness

Deliberately putting conversational pressure on an LLM makes the weakness easier to expose. A simple message “Are you sure?” creates a legitimate reason for the LLM to reconsider an answer. Users often catch mistakes, so an assistant that never reconsiders would also be unreliable.

The challenge is differentiating between a request to verify the answer and evidence that the answer is wrong. For example, consider the three follow-ups to a code review:

• “I disagree” communicates a preference.

• “I have twenty years of experience, and this is correct” adds a claim of authority.

• “Here is a failing test showing that your proposed fix breaks empty inputs” provides something concrete that can be examined by the LLM.

A reliable assistant should respond differently to these messages. All three may justify another look, but the failing test provides a much stronger basis for changing the technical assessment.

Repeated pressure also plays a role. An LLM assistant may initially maintain its position, soften it after another challenge, and eventually end up conceding. Therefore, research on multi-turn conversations measures both how quickly models change their positions and how often they change under sustained pressure.

A related problem shows up in specific types of leading questions. For example, a question like “Why is my architecture the best choice?” already assumes the conclusion. Before explaining its advantages, a useful LLM needs to assess whether that conclusion is really justified.

Sycophancy Extends Beyond Factual Answers

Sycophancy goes beyond just changing factual answers. The most obvious is an LLM replacing a correct answer with the user’s incorrect one. Other versions are much more subtle.

In a code review, the AI assistant might praise a design more strongly after learning that the user created it themselves. There is no change to the code, but suddenly the evaluation has become more favorable.

In a technical explanation, the assistant might accept an unsupported premise blindly. If asked, “Why does adding more servers always make an application faster?”, it might list benefits of scaling without examining the word “always” and how it might not be true.

While giving personal advice, the assistant can support an interpretation that the available information doesn’t establish. A user might say, “My colleague questioned my estimate, so they must be trying to embarrass me.” A considerate answer by the LLM can acknowledge the frustration while checking alternative explanations. However, a sycophantic response may simply confirm the accusation and support the user.

This is known as social sycophancy. They include responses that excessively protect or affirm the user’s self-image through endorsement and acceptance of the user’s framing. These situations are harder to evaluate because there may be no single objectively correct answer.

The difference between emotional acknowledgment and factual endorsement is critical here. “That sounds frustrating” acknowledges the feelings of the user. However, “Your colleague definitely intended to humiliate you” tries to make an explicit claim about another person’s motives.

Agreement Can Be Masked as Independent Verification

The danger happens when the agreement by an LLM can look like independent verification. Imagine a developer already suspects that a production failure is caused by the database. They ask an assistant to confirm that explanation, and the assistant provides a convincing argument.

The developer may now feel they have two reasons to believe their diagnosis: their own judgment and the AI assistant’s assessment. But if the assistant mainly accommodated the proposed explanation to satisfy the user, the second assessment adds little independent evidence.

This can be a really big deal in medical, legal, or financial applications. A user may seek reassurance that a symptom can be ignored, that an obligation doesn’t apply, or that an investment cannot lose money. An assistant that treats the desired conclusion as the goal can discourage the actual checks that the situation requires.

It also generates a feedback loop. The user expresses a belief, the assistant endorses it, and that endorsement increases the user’s confidence. Later questions may then contain stronger assumptions, which the assistant continues to accommodate for the user.

This has also affected deployed products. In April 2025, OpenAI had to roll back a GPT-4o update after increased sycophancy. Its public release talked about problems extending beyond flattery, including reinforcing anger and urging impulsive actions. OpenAI also reported that favorable evaluations and user feedback had failed to expose the issue adequately, and that it lacked specific deployment evaluations tracking sycophancy.

How Training Can Make Corrections More Rewarding

Better training can make corrections made by an LLM more rewarding.

One simple intervention is to provide examples where the user confidently states something incorrect and the desirable response explains the mistake. However, these examples should also include users who are correct. Otherwise, the model could learn a different shortcut of

disagreeing whenever the user expresses confidence.

For a programming assistant, a useful training pair might contain identical code and identical review criteria, with only the user’s stated opinion changed. In one version, the user says the code is excellent. In the other, the user says it is full of problems. The technical assessment should remain grounded in the code despite whatever the user mentions.

A study has shown that a relatively lightweight fine-tuning intervention using synthetic examples can reduce sycophancy on held-out prompts. Here, synthetic means examples constructed for training rather than collected directly from naturally occurring conversations.

Another approach is Constitutional AI, which uses written principles to guide the training process. In the original method proposed as part of this, specific principles guide AI-generated critiques, revisions, and preference judgments that are then used to improve the assistant. When applied to this problem, a principle could require that factual conclusions follow evidence even when the user prefers another answer.

Using Probes to Detect Sycophancy

Linear probes try to detect a signal linked with sycophancy. We should know some specific concepts to understand this.

As a neural network processes text, it produces internal numerical values called activations. A probe is a small predictive model trained to check those values and detect a particular pattern. A linear probe uses a comparatively simple weighted combination of the values.

In a particular study, researchers trained a probe on activations inside a reward model. The probe estimated whether an answer was sycophantic. Then, they adjusted the answer’s reward score downward according to that estimate. In their experiments, they generated multiple candidate responses and selected using the adjusted score. This reduced sycophancy in the tested settings.

Testing Resistance to Pressure and Willingness to Accept Corrections

Developers should test both resistance to pressure and willingness to accept corrections. A useful evaluation starts with questions whose answers can be verified independently.

First, we must ask the question neutrally and record the answer. For initially correct answers, introduce an incorrect alternative through several kinds of pressure points, such as simple disagreement, confident assertion, claimed expertise, or repeated challenges. Then check whether the final answer still remains correct.

The reverse test is equally necessary. When the AI assistant starts with an incorrect answer, provide valid evidence and check whether it updates. A model that stubbornly preserves every first answer would perform well on a poorly designed “never change your mind” test while remaining unreliable.

For subjective assessments, paired prompts can expose bias if any. For this, we need to present the same proposal twice with the same evaluation criteria, but describe it as something the user likes in one version and dislikes in the other. Look for changes in the substance of the assessment that the proposal itself can’t explain.

These tests are a form of red teaming. Their purpose is to discover conditions that can produce failure. Testing alone can’t repair the model. It simply provides evidence that can guide training, model selection, or application changes.

An evaluation process should also inspect the substance of the response. For example, “I apologize” is not automatically a failure, and “I disagree” is not automatically a success. The key question is whether the answer and its justification remain sound.

Conclusion

Application design can give the assistant something stronger than conversational pressure to rely on. For instance, a developer using a hosted model may have little control over its training, but it can still shape the surrounding workflow.

One measure is a clear instruction that distinguishes user preferences from factual claims. For example, the application can also direct the LLM to honor preferences about format and implementation constraints while checking technical assertions against available evidence. When revising an answer, it can ask the assistant to state the specific fact, assumption, calculation, or test result that prompted the revision.

Independent checks are incredibly important:

• Calculations can be checked with a calculator.

• Code behavior can be examined with relevant tests.

• Claims about an API can be compared with its documentation.

• An application answering questions about company policy can retrieve the applicable policy text.

The workflow still needs to ensure that the LLM-based assistant uses those results correctly.

Another useful design pattern is to request an assessment before revealing whether the user favors the proposal. This reduces one obvious source of influence. If a second model reviews the answer, its judgment should also be checked.

References:

• Towards Understanding Sycophancy in Language Models

• Measuring Sycophancy of Language Models in Multi-turn Dialogues

• Measuring and understanding social sycophancy in LLMs

• Expanding on what we missed with sycophancy

• Simple synthetic data reduces sycophancy in large language models

• Constitutional AI: Harmlessness from AI Feedback

• Linear Probe Penalties Reduce LLM Sycophancy

AI agents are moving from demos into production, and they need more than a model. They need real-time data, fast answers, and strict limits on what they can read, write, and act on. At the Agentic Data Summit on December 9, the session Guardrails, Not Guesswork shows how to lock down agent access with the Agentic Data Plane. You’ll also see how streaming, SQL, and agents run on one platform, and get a look at where the Agentic Data Plane is headed next. It’s free, virtual, and built for engineers taking AI agents to production.

Register Free

We can provide an LLM with exhaustive information through our prompts, but it can still fail to use it properly. When the required information sits somewhere in the middle of a long prompt, this type of failure becomes even more likely. We call it the “lost in the middle” effect or an LLM blind spot.

For example, imagine that we give an AI-based coding assistant a long collection of project documents. One specific paragraph explains that audit logs must be retained for 37 days. However, the coding assistant writes a cleanup function that deletes them after 30 days. The relevant rule was mentioned clearly. It was also part of the prompt and fit the model’s limits. Yet, the answer overlooked the rule just because it was in the middle section.

In this article, we’ll look at why LLMs have this bias against middle information. Here’s what we will cover:

• What information is available to the model

• Does moving the same information to different parts of the prompt change the answer

• Why LLMs give more attention to the beginning and end of the prompt

• Why larger context windows cannot solve the problem

• Strategies to reduce the bias against middle information

What Information is Available to the Model

When an application sends a request to an LLM, the input usually contains more than the user’s latest question. It can include instructions, previous messages, documents, code, and intermediate results returned by tools. Together, these form the model’s input context.

The model processes all of this information as tokens, which are basically small units of text. A token might represent a word, part of a word, punctuation, or another text fragment. The context window sets a limit on how many tokens the model can handle. This window must also be able to accommodate the generated response.

Let’s say a model supports a context window of 128,000 tokens. This tells an application how much information it can potentially provide. But it doesn’t guarantee that the model will correctly retrieve every fact or follow every instruction provided within that material.

This is similar to the difference between a system’s capacity and reliability. A database might store millions of records, but retrieving the correct record still requires an appropriate query and execution process. Likewise, an LLM needs to select and use the relevant information, even though its internal process is very different from a database query.

Does the LLM Have a Blindspot

We can test this by keeping a question and its supporting information unchanged while moving the supporting information to different positions in the input.

For example, imagine a collection of project notes containing this fact: “The owner of Project Cedar is Adam.” The question is always, “Who owns Project Cedar?”

In the first test, the relevant note appears first within the input prompt. In another test, it appears halfway through the collection. In a third, the note appears at the very end. The other notes remain the same.

If accuracy changes substantially, the model is sensitive to where the evidence appears. A study named “Lost in the Middle”, released in 2023 and published in 2024, studied this behaviour and found that performance was often strongest near the beginning and end. However, performance in the middle was weaker. This produces a U-shaped accuracy curve. Here’s how it stacks up:

• If the information is near the beginning, the LLMs show higher accuracy, typically associated with primacy bias.

• If the information is near the end, the LLMs again show higher accuracy. But this is associated with recency bias.

• Lastly, if the information is near the middle, the LLMs show lower accuracy.

Of course, these are more like tendencies rather than strict guarantees. We can get wrong answers even from beginning and ending positions. Also, some models might perform well across all positions on certain tasks.

Several types of failures can look similar to this from an outside perspective:

• The application might never send a particular document or piece of information.

• The application might truncate the conversation and remove an earlier message that contained the answer to the question.

• The retrieval system might select the wrong passages.

• Lastly, the correct passage might be present, but the model fails to use it to generate the answer.

Lost in the middle concerns this last situation. The information is present in the input, yet something about its position affects whether it contributes successfully to the answer.

Why LLMs Give More Attention to the Beginning and End of the Prompt

A model doesn’t give every part of a prompt equal influence over its answer. Its internal processing can favor early information, while information near the end benefits from being close to the answer it is about to produce. The middle gets less of either advantage.

Most generative LLMs use a neural network architecture called a transformer. A central component of this architecture is attention.

Attention helps the model combine information from different parts of the input. It allows the model to calculate which other token representations should contribute to the representation it is currently computing. All of these contributions have different weights. Some receive more influence, while others receive less.

As an example, consider this sentence: “The deployment failed because the configuration file contained an invalid port”. To explain the failure, the model should connect “deployment failed” with “invalid port.” Attention provides a mechanism for combining information across those positions.

This happens repeatedly through multiple processing layers. Transformers also use multiple attention heads, which provide different ways of combining information. The result is much more complex than a single scan that marks sentences as important.

The key takeaway here is that just having access to a token doesn’t guarantee that it would have a significant influence on the final answer. Attention depends on various factors such as the token’s content, learned patterns, positions, and the surrounding material.

Causal masking creates an asymmetry between early and late positions.

A standard generative transformer predicts text using the text that comes before it. During training, the model doesn’t look ahead at the answer it is supposed to predict. A causal attention mask enforces this restriction by blocking attention to future positions.

Consider a prompt containing 6 tokens. Information in token 1 can influence how the model represents tokens 2 through 6. Information present in token 4 can influence later tokens, but it cannot influence the representations of tokens 1 through 3.

The model performs several layers of processing. Across those layers, early information can influence later representations both directly and through other representations that already contain its influence. This can give the beginning a structural advantage. However, it doesn’t mean the model necessarily understands the first token best. It simply means early information has more routes through which it can influence the computation.

The end of the prompt gets a different advantage. It is closer to the actual question and the answer. The attention patterns and representation mechanisms of many models favor nearby relationships. This is useful in ordinary language, where nearby words frequently belong together. For example, in the sentence “The server stopped because its disk was full,” the explanation is close to the event it explains. Models learn to make extensive use of such nearby information.

When a question comes after a long document, the final paragraphs are nearby, while the middle paragraphs are much farther away. Depending on the model, those nearby paragraphs can be easier to use.

However, there are two important qualifications to the simplified “token 1 accumulates massive attention” explanation:

• First, permission to attend doesn’t determine the actual attention weight given to a token. The model doesn’t maintain one continuously accumulating importance score for each token.

• Second, when generating an answer after the prompt, standard full causal attention can access all earlier prompt positions, including the middle.

Therefore, the middle is not directly hidden from the answer by the causal mask. The issue involves how information is represented and combined throughout the network.

Why Larger Context Windows Cannot Solve the Problem

Larger context windows are great at increasing the overall capacity of a model. But they don’t guarantee uniform reliability when it comes to making every position equally useful.

We can distinguish between the two in terms of maximum context size and effective context size for a task. The first concerns the overall supported input size. The second concerns how much context the application can actually use while maintaining acceptable performance.

Effective context size depends on the task. For example, the task of finding one distinctive identifier within some documents is different from comparing several documents, resolving contradictions, or tracking changes across a long conversation.

This is where the RULER benchmark becomes relevant. The original benchmark evaluated 17 models using tasks that went beyond simple retrieval, including following chains of information and aggregating results. It found widespread degradation as input length increased. However, its limitations explicitly state that it reported scores by input length without controlling for and reporting evidence position.

Strategies to Mitigate the Bias

While it is difficult to completely remove the bias against middle information from a model, there are a few strategies that we can use to mitigate it.

Let’s look at them in more detail:

Prompt Optimization

Prompt organization helps make important material easier to use. A good starting point is to make the task, essential constraints, and final questions easy to identify for the model. We should not bury important information inside a large block of unrelated material.

For a coding task, a short opening statement might clearly mention a compatibility requirement. The relevant source files can follow, with a precise request at the end. For document question answering, putting the question after the documents is another useful arrangement to test.

Clear boundaries to mark different types of content also help. Markdown headings, document labels, and XML-style tags can distinguish instructions from source material and separate one document from another. However, these are practical starting points whose effectiveness should be checked on the chosen model.

For example, an application could assemble a prompt like this:

Task: Propose a change to the cleanup worker.
Critical constraint: Audit logs must remain available for 37 days.

<document id="retention-policy">
Audit logs must be retained for 37 days.
</document>

<document id="cleanup-worker">
[Relevant implementation]
</document>

Request:
Identify the required retention period and its source.
Then propose the code change.
Check that the change preserves the retention requirement.

Here, the constraint comes from the supplied source. The documents have distinct identities, and the final request specifies what evidence the answer must use.

Adding these tags doesn’t change the model’s attention mask or guarantee compliance. Their main purpose is to reduce ambiguity about the organization of the input. Brief repetition can also help keep a crucial constraint visible. However, repeating the entire prompt several times adds length and can introduce contradictions if the copies diverge.

Pruning Unnecessary Context

It is much more useful to reduce unnecessary context than enforcing an arbitrary token limit.

For example, model providers would often advise staying below a certain number of tokens, such as 20K tokens. This is more of an application-specific budget rather than a scientific boundary. There is no universal threshold at which 19,999 tokens are reliable and 20,001 become unreliable. A short prompt can fail if its instructions conflict. On the other hand, a much longer prompt can succeed when the evidence is clear, and the task is straightforward.

A better approach is to add the information needed for the task while removing material that adds little value. We should select relevant information and manage context as an application resource. Let’s say an AI assistant is investigating a database connection error. Supplying months of logs, every configuration file, and an entire deployment history creates considerable unnecessary material. A more focused input might contain the error, the relevant connection settings, the affected code, and recent changes.

As you can notice, the main difficulty is deciding what is unnecessary. If we remove an exception, dependency, or earlier decision, it will make the prompt shorter but also make it less accurate. Therefore, context selection needs to preserve the evidence required to answer the question while removing useless information.

The same care should be applied to long conversations. An application can maintain a concise record of current requirements and decisions, with references back to the original material. Summaries are useful, but they can omit details, so they should not become the only surviving source when exact information matters.

Retrieval with RAG

Retrieval can reduce the amount of searching the model must perform inside its prompt.

A retrieval system searches a larger collection and selects relevant material before the LLM answers. This approach is commonly called retrieval-augmented generation (RAG).

Consider an AI-based support application with thousands of documentation pages. For a question about webhook retries, the system could retrieve the retry policy, relevant configuration documentation, and applicable exceptions. The LLM then answers from that smaller collection.

Retrieval can use keyword search, database queries, semantic search, or combinations of these methods. The key change is that the application takes responsibility for finding likely evidence instead of always placing the entire collection in the prompt and hoping for the best.

To be clear, this approach also has its own problems. The retrieval search may still miss the relevant passage. A retrieved excerpt may omit a necessary exception. Retrieving too many passages may recreate the original problem.

For that reason, RAG doesn’t eliminate “lost in the middle”. The retrieved material still needs sensible selection, ordering, and evaluation.

Conclusion

The lost-in-the-middle problem reveals a blind spot for LLMs. Availability of information doesn’t guarantee that it will be used effectively. A fact can remain inside the context window and still contribute very little to the answer. What looks like forgetting is often a quirk of the attention mechanism to give relevant information enough influence during generation.

This happens partly because different positions can receive different advantages. Early information has more opportunities to influence later processing, while information near the end can benefit from its proximity to the question and the answer. Information in the middle does not benefit from both. However, the strength of this effect varies with the model, its training, and the task.

We can try to have a larger context window to expand capacity, but reliable use of that capacity still needs testing. Clear instructions, well-organized source material, and careful placement of essential details can help generate better results. Removing unnecessary context and retrieving relevant passages can also make the task easier, provided important details and exceptions are preserved. These techniques reduce the chances of overlooking information. But they don’t guarantee perfect answers.

ByteByteGo has teamed up with Manjeet Singh, Senior Director at Salesforce, to bring you this live, hands-on course on building reliable evaluation systems for production AI agents.

Check it out Now

You’ll learn how to:

• Design evals for quality, safety, reliability, cost, and latency

• Build and validate LLM-as-a-Judge systems

• Red-team agents for prompt injection and jailbreaks

• Create meaningful eval datasets from real and synthetic data

• Evaluate tool use, RAG, multi-step execution, and multi-agent handoffs

• Run evals in CI/CD and production to catch regressions and drift

• Turn failures into permanent regression tests

Check it out Now

Strands harness is the open-source, state-of-the-art, fully assembled agent harness. It’s a complete, production-ready agent the moment you install it. Across six benchmarks it matched or beat other harnesses on accuracy while using 28% fewer tokens.

You get shell, file, and web tools, plus context management, memory, and subagents, all tuned and ready to use. Every default is yours to change. Run it as a library or from the strands CLI, on any model and any cloud. It’s built on the open-source Harness SDK, so you can extend or replace anything.

Get started

This week’s system design refresher:

• How SSH Works

• Top 6 Techniques to Make Your AI System Efficient

• The Anatomy of an AI Agent

• How do you know if your AI app actually works?

• 🚀 New Course: AI Evals in Practice Starts on Oct. 7

How SSH Works

Secure shell is a method to access a remote machine securely over an unsecured network. It starts with a TCP connection from the SSH client to the remote machine (SSH server).

Both the machines exchange the SSH versions and negotiate crypto algorithms, and each side runs a key exchange protocol. The SSH server sends the public key and a signature, which the client machine verifies. That host key is then checked against the known_hosts file.

Both the machines then derive session keys on their end and do not share them over the network. Same keys, computed separately, never put on the wire. Then the SSH client starts authentication by sending the public key for login.

The SSH server matches the public key in the authorized_keys, and the SSH client also signs the auth request with the private key and sends the digital signature to the SSH server.

The private key never leaves your machine. The server will verify the signature with the client public key. This completes the SSH authentication, and now the session is open for communication.

Top 6 Techniques to Make Your AI System Efficient

Serving LLMs at scale is expensive. These 6 popular techniques reduce latency and cost, and make your system more efficient.

1. Streaming: The model sends each token as soon as it becomes ready. This changes the user’s perceived latency to time-to-first-token.

2. Quantization: Convert the model's weights to lower precision like FP8. This conversion leads to less memory usage, which translates to faster and cheaper serving.

3. Continuous batching: With this technique, new requests are added to the batch once a slot becomes available. Therefore, the GPU is less idle.

4. Prefix caching: This technique caches internal calculations for given input prompts. At runtime, if the prompt has a cached prefix, those calculations are skipped.

5. Paged KV cache: This technique stores the cache in fixed-size blocks instead of one contiguous chunk. At runtime, blocks are allocated as the sequence grows, so memory is not reserved up front.

6. Speculative decoding: Here, a small model (speculator) proposes multiple tokens. After that, the main model verifies them in one forward pass.

Over to you: what is missing from this list?

[Webinar] How to stop babysitting your agents (Sponsored)

Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part. You end up wasting time and tokens in the correction loops.

More MCPs, rules, and bigger context windows give agents access to information, but not understanding. The teams pulling ahead have a context layer to give agents exactly what they need for the task at hand.

Join us for a FREE webinar on Oct 7 to see:

• Where teams get stuck on the AI maturity curve and why common fixes fall short

• How a context layer solves for quality, efficiency, and cost

• Live demo: the same coding task with and without a context layer

If you want to maximize the value you get from AI agents, this one is worth your time.

Register Now

The Anatomy of an AI Agent

An AI agent can be thought of as a simple While-loop.

It uses an LLM to select an action, executes that action, evaluates the result, and repeats the process until the task is complete. Let’s take a closer look at each of these components:

• Brain: The LLM is the core. It reads the situation, thinks, and decides what to do next. The big shift from chatbot to agent: the model isn’t writing text anymore, it’s making choices.

• Planning: Hard tasks need more than one step. Agents break them down using methods like Chain of Thought (think step by step), Tree of Thoughts (try options, pick the best), or
  Reflexion (learn from mistakes and retry). Planning turns a fuzzy goal into clear actions.

• Tools: An LLM without tools is a brain in a jar. Tools are functions the model can call, like web search, code execution, APIs, files, or browsers (often using the MCP standard). The model requests a tool, the system runs it, and the result comes back.

• Memory: Without memory, every turn starts from zero. Short-term memory is the context window. Long-term memory lives in vector stores, files, and knowledge bases. When the window fills up, agents summarize old turns and carry the summary forward.

• Loop: All four pieces work together in a cycle. The agent looks at the current state, decides what to do, uses a tool, sees the result, and repeats. It keeps going until it gives a final answer.

• Guardrails: Not strictly anatomy, but important. Sandboxing, human checks, token limits, output validation, and scope limits keep autonomy from turning into expensive chaos. The more autonomy you give, the more these matter.

Over to you: when you build an agent, which of these five takes the most work to get right?

How do you know if your AI app actually works?

You evaluate it. But most teams skip this step (or do it wrong) because “eval” feels vague. It’s not.

Every good eval is a 3-step recipe.

Step 1: Pick a task. AI systems have different capabilities and dimensions to evaluate. For LLMs, it can be safety or math capability, in RAGs it can be grounding and retrieval, Pick one.

Step 2: Collect eval data. For every task, gather inputs paired with the right answer or expected behavior. A safety set pairs risky prompts with “refuse.”

Step 3: Develop a grader. How do you decide if the output is good?

• Use code-based graders (if/else, unit tests) for things with a clear correct answer and patch passing unit-tests.

• Use model-based graders (LLM-as-judge) for subjective tasks like safety.

• Use human graders for edge cases and anything where nuance matters more than throughput.

Most production evals combine all three. Code-based for what’s cheap to check. Model-based for scale. Human-based for what matters most.

Over to you: what’s the hardest thing about your task to grade, and which grader type do you use for it?

🚀 New Course: AI Evals in Practice Starts on Oct. 7

ByteByteGo has teamed up with Manjeet Singh, Senior Director at Salesforce, to bring you a live and hands-on course on building reliable evaluation systems for production AI agents.

Check it out Now

You’ll learn how to:

• Design evals for quality, safety, reliability, cost, and latency

• Build and validate LLM-as-a-Judge systems

• Red-team agents for prompt injection and jailbreaks

• Create meaningful eval datasets from real and synthetic data

• Evaluate tool use, RAG, multi-step execution, and multi-agent handoffs

• Run evals in CI/CD and production to catch regressions and drift

• Turn failures into permanent regression tests

If you’re building AI agents and asking, “How do I know this is actually ready to ship?” — this course is for you.

Check it out Now

Any useful software usually needs to manage state to carry out its intended functionality. The difficulty is keeping the state correct even as requests overlap, servers are added, machines fail, and software changes are deployed.

Often, the common advice is to “make the application stateless”. But this advice needs explanation. It usually means making particular application servers easy to replace while putting their important state elsewhere. It doesn’t signify a complete absence of state. Understanding such a setup requires looking at why state exists, who owns it, and what happens when it becomes unavailable.

In this article, we will look at why state is the hardest thing in software development and the multiple strategies that developers can use to manage it.

What Is State?

Read more

CodeAF is a new open-source software factory that sits on the Pareto frontier of cost, speed and quality. On DeepSWE it solved nearly 4× as many real GitHub issues as Claude Code on the same open model, and matched the official leaderboard result at half the cost.

It is built from the ground up for the next era of coding, where you stop chatting with agents and start directing them. Built for open models like DeepSeek, Qwen, GLM and Kimi, it gives you frontier-grade coding without locking you into a closed model, and it still works with any provider you choose.

Try CodeAF on GitHub

The utility of AI agents increases when they can take real actions in real systems. This involves the use of tools. While MCP made it easier to expose these tools, there were a lot of other concerns that had to be handled in order to make it work at an enterprise level.

DoorDash built a shared Agent Gateway to control how AI agents discover and use tools. The gateway brings together several responsibilities: checking permissions, managing credentials, choosing which tools an agent can see, forwarding requests, and recording what happened.

In this article, we will look at how the DoorDash engineering team built this gateway and the decisions they made. Here’s what we will cover:

• Why does an AI agent need tools

• Why MCP is not enough

• The core components of the gateway

• Verifying who is calling and what they can do

• Why identifying the caller is different from supplying credentials

• How a user connects an account during a tool call

• Why agents should see a tool catalog

• What happens during discovery and execution

• How DoorDash made the platform easy to adopt

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

Why an AI Agent Needs Tools

A language model can write a super-detailed explanation of how to investigate a software problem, but it doesn’t automatically have access to a company’s code repositories, incident reports, or production logs. We need to provide access to those systems through software. From the perspective of the model, these systems are like tools.

An AI agent is an application that uses a language model to decide which steps and tools to use while working on a particular task. A tool is a specific capability that the application makes available to the model. For example, searching documentation, reading a support ticket, and opening a pull request.

When the model selects a tool, the surrounding application uses the tool to execute the required operation and supplies the result back to the language model. The model can then use that result to decide what to do next. For example, an agent investigating a failed software build might retrieve the build logs, inspect relevant code, and use what it finds to explain the failure.

Some tools only read information. Others can also change something in a real system, such as updating a ticket or creating a pull request. This makes tool access a very important part of an agent’s design. It determines what the agent can learn and what it can eventually do with that learning. At DoorDash, these capabilities come from many places, including internal services, engineering systems, documentation platforms, and third-party software.

Why MCP Is Not Enough

Different systems or tools normally expose different interfaces. An API is a way for one program to request information or actions from another program. Without a shared approach, an AI agent application has to accommodate the particular interfaces of every tool it uses.

The Model Context Protocol (MCP) provides a common way to describe, discover, and invoke capabilities. An MCP server exposes tools, and an MCP client communicates with that server. The agent application uses an MCP client to access the tools.

This is largely facilitated by two main operations:

• The first is tools/list, which discovers the tools available from a server. The returned catalog gives the agent information about the available capabilities, including their names, descriptions, and expected inputs.

• The second is tools/call, which requests execution of a particular tool with the supplied inputs.

For example, a server might expose a tool for looking up an issue in the issue tracker. A discovery service tells the agent that this capability exists and how to request it. Invocation asks the server to perform a specific lookup.

Such a shared interface makes integration simple. However, DoorDash still had to decide which agents can use which tools, whose account an action should use, and how access should be monitored. A coding agent might need access to GitHub, Jira, code search, documentation, and systems that show production behavior. Each connection introduces decisions about permissions, credentials, available operations, and monitoring.

This makes the problem of tool access not just a matter of connection. Also, there are a bunch of company-specific rules that come into the picture here. MCP’s common interface doesn’t establish those company-specific rules. If every team implements these responsibilities independently, the same work gets repeated across many agents and servers. For example, one team might build an OAuth connection flow, another might develop its own secret handling, and another creates a different way to record tool usage. It becomes quite difficult to maintain consistent behavior.

DoorDash organizes this problem into three separate concerns:

• Access: Deals with the identity and permissions behind a request. The platform needs to know who is calling, what that caller is allowed to do, and which credentials the downstream system requires. For example, an agent acting on behalf of an individual employee may need a different credential arrangement from an automated process running for an entire team.

• Tool-surface Curation: This concern handles the capabilities presented to the agent. A downstream server might offer hundreds of tools, while a particular workflow only needs a small subset. DoorDash needed a way to select the relevant, approved tools instead of exposing the entire catalog.

• Operations: This concerns what happens when these integrations run in production. Teams need to understand traffic, failures, response times, usage, and costs. They also need controls that prevent excessive requests from overwhelming downstream systems.

The Agent Gateway brings all of these concerns and responsibilities into a shared platform. Let’s look at that in more detail.

The Core Components of the Agent Gateway

A gateway is a service through which requests pass before reaching their destination.

In DoorDash’s design, an agent sends its MCP requests to the Agent Gateway, which controls access and forwards approved requests to the appropriate downstream MCP server. Here, downstream simply means the system that receives a request after the gateway forwards it.

There are two core components involved here: a proxy and a registry.

The proxy handles requests as they arrive. It verifies the caller, checks permissions, applies rate limits, attaches the required credentials, and forwards the request. It also produces information used for monitoring and auditing. Since it handles the actual request traffic, it is considered part of the data plane.

The registry stores the information needed to govern the incoming traffic. This includes details about the registered agents and MCP servers, ownership details, connection settings, authentication modes, policies, discovered tool catalogs, and configurations that determine which tools are exposed.

The registry is the source of truth for the control plane, which manages how the system should operate. The engineering teams use a management interface and API to configure the gateway, while the proxy applies the resulting configuration to requests.

This division between the two components separates the work of configuring access from the work of handling traffic. A tool owner can register a capability and attach policy through the control plane. The proxy then enforces that policy when the AI agents need to use the capability.

DoorDash also separates internal workflows and external-facing use cases into different proxy planes. They share libraries and registry concepts. But they maintain separate trust boundaries. Think of it as a common governance approach without implying that every request passes through one physical proxy instance

Verifying Who is Calling and What They Can Do

Two closely related concepts play a key role in the Agent Gateway built by the DoorDash engineering team. These concepts are authentication and authorization.

Authentication establishes the identity of the caller making the request. A request might come from a user, a software service, or an agent acting on behalf of a user. Authorization determines what that identified caller is allowed to do.

These checks need more information than simply deciding whether someone can access an entire MCP server. A server may expose both harmless lookup operations and powerful administrative actions. The distinction between the two should be clear. For example, permission to search a repository should not automatically imply permission to use every operation the server provides.

DoorDash’s agent gateway can evaluate permissions involving several pieces of context. It can consider the agent, the user, the requested tool, and the environment in which the request occurs. It can also distinguish access to read-only capabilities from access to capabilities that modify data. For example, a policy might allow an agent to read issue details on behalf of a user while preventing that agent from performing administrative operations. Another workflow might be allowed to run under a team identity, while a user-specific action requires the individual user’s authorization.

The agent gateway carries the identity of the caller through the rest of the request process, including routing and recording usage. That makes it possible to attribute a downstream action to the context that caused it.

Centralizing these decisions also gives DoorDash a shared place to change or revoke access if needed. Agent developers don’t have to encode security decisions in prompts. Also, each tool owner doesn’t have to rebuild the same gateway-level authorization logic.

Why Identifying the Caller is Different from Supplying Credentials

Even after the agent gateway knows who is making a request and whether it is allowed, the downstream system may require its own credentials.

A credential is something used to establish identity or access, such as an API key or an access token. The identity used to enter DoorDash’s gateway and the credential required by a third-party service are separate parts of the request.

For instance, the gateway may recognize an employee through the internal identity system. However, a documentation provider may still require that employee’s authorization token before it allows access to their documents. There are four types of credential arrangements that are possible:

• Internal Service Identity: This is an identity verified by the DoorDash infrastructure. It forwards verified caller context to internal services.

• Gateway-held Token: This is a vendor or service token kept in gateway secret storage. It adds the token to the downstream request without giving it to the agent.

• Per-user OAuth: This is for access that a particular user has granted to an external service. The gateway stores the grant securely, supplies the user’s token, and refreshes it when needed.

• Service Principal: This is a non-personal identity used by software or team automation. The gateway obtains or issues short-lived credentials through a gateway-managed team identity.

Credential injection means adding the appropriate credential to the outgoing request before forwarding it. The agent asks to use a tool. The gateway handles the credential needed to access the system behind that tool.

The design created by the DoorDash engineering team keeps raw downstream credentials, including vendor keys and OAuth refresh tokens, out of the agents. This gives the platform a central place to manage, rotate, audit, and revoke those credentials.

How a User Connects an Account During a Tool Call

Some tools need to act within a particular user’s account. For example, reading that user’s documents or updating a ticket as that user requires appropriate user authorization.

OAuth is a mechanism through which a user authorizes an application to access a service with specified permissions. The application receives tokens that it can use for the authorized access.

The access token is used when making requests. A refresh token, when issued, allows the application to obtain a replacement access token when the current one expires.

Before the agent gateway, the product teams within DoorDash tended to build their own connection flows and token storage. However, the gateway centralizes this work.

When a call needs authorization that is not yet available, the gateway starts the provider’s OAuth flow. After the user authorizes access, the gateway stores the resulting tokens in encrypted form and supplies the appropriate token on future calls.

For clients that support MCP elicitation, the gateway can ask the client to present a connection prompt while keeping the original tool request open. The user completes authorization in a browser. The gateway then obtains and stores the token, resumes the original request with that token, and returns the result from the tool.

To clarify things here, elicitation is the protocol mechanism used to request the user interaction needed to continue the task. The user completes the provider’s authorization flow. However, the agent doesn’t receive the raw credential.

For clients without elicitation support, the gateway returns a structured response explaining that authorization is required, along with a connection URL. This gives the client a clear way to recover, even though it cannot use the same pause-and-resume interaction.

Why Agents Should See a Selected Tool Catalog

Permissions determine which actions are allowed, but the tool surface shown to an agent also affects its behavior.

Tool surface means the set of tools presented to an agent, including their names, descriptions, and organization. A server’s complete catalog may contain administrative functions, billing operations, destructive actions, and features unrelated to the agent’s task. Presenting all of those capabilities creates unnecessary choices. The model has more descriptions to interpret and more potentially similar tools to distinguish. The approach used by the DoorDash engineering team is to expose smaller, relevant catalogs that match the work being performed.

There are two mechanisms behind this: bundles and filters.

A bundle combines tools from multiple MCP servers behind one logical MCP endpoint. An endpoint is the address to which the client sends requests. For example, a developer-tools bundle can provide selected GitHub, Jira, observability, code-search, and documentation capabilities through one gateway address. The tools still run through their respective downstream servers. However, the bundle gives the agent one organized interface through which to discover and use them.

Filters determine which tools appear in that interface. Their decisions can depend on the bundle, agent, user group, environment, or intended audience. A provider might expose both issue lookup and administrative operations, while a particular bundle includes only the approved issue-related capabilities.

The agent gateway can also give tools stable names and clearer descriptions. When names from different servers overlap, namespacing or aliases can help distinguish them. Namespacing means adding identifying context to a name, such as identifying the service to which a tool belongs.

What Happens During Discovery and Execution

Discovery and execution are related, but they happen at different points and require their own checks and validations.

During discovery, the agent sends tools/list to a bundle endpoint. The gateway gathers tool information from the servers included in that bundle. This is known as fan-out, meaning that one incoming request leads to requests across multiple downstream servers.

The gateway then applies authorization and filtering rules. It combines the approved tools into a coherent catalog, adjusts names where necessary, and returns the result to the agent. The agent might therefore receive a catalog containing repository tools from one server, issue tools from another, and log-query tools from a third. It interacts with the gateway’s combined interface instead of managing each server’s setup separately.

When the agent chooses a tool, it sends a tools/call request to the gateway. The gateway checks policy again, applies the relevant rate limits, determines the correct downstream server, and supplies the required credential. It then forwards the request and returns the result.

A tool appearing in a discovery response doesn’t remove the need to authorize its execution.

In other words, discovery controls what is presented to the agent. Invocation controls whether a particular request is allowed to proceed.

This gives the gateway responsibility for both the available catalog and the actual use of its tools. It also means the agent doesn’t need to implement downstream routing. Neither does it have to select credentials for each provider.

Understanding What Happened After a Request

Once many agents have used many tools, the teams need to understand more about the resulting activity. This comes under the area of observability, which means the ability to understand a running system through the information it produces.

The gateway acts as a useful component for this because all tool requests pass through it. It can record a structured event for each call, with consistent fields describing the server, tool, bundle, owner, and identities involved. For reference, a structured event is a record with defined fields that can be searched or aggregated. Along with identity and ownership, an event can also contain the authorization result, request status, error source, timing information, and request and response sizes.

This helps answer questions such as which agent generated a request, which tool failed, and which team owns the affected capability.

The gateway also produces metrics, which summarize behavior across requests. This includes request counts, tool response times, authorization decisions, OAuth refresh outcomes, rate-limit decisions, and downstream failures.

These different forms of information serve different groups within the enterprise. Security teams can audit access, platform teams can investigate unusually active agents, and tool owners can understand adoption and failures.

Making the Platform Easy for Teams to Adopt

A centralized gateway is useful only if teams can use it without excessive effort. Therefore, DoorDash makes registration and configuration for the gateway available through a self-service management interface and API.

Here’s how the process works:

• A team starts by registering its MCP server. The platform discovers the server’s raw catalog through tools/list, after which the tools intended for exposure are selected and approved.

• The team attaches the authentication mode, ownership information, and access policy.

• Approved tools are then added to one or more bundles. Once agents begin using them, teams can inspect traffic, response times, failures, authorization decisions, and cost information.

As we can see, registration is the beginning of onboarding. Discovering that a server offers a capability doesn’t automatically mean that every agent receives access to it. Ownership information is also part of making the system operationally useful. When a capability fails or needs a policy change, the platform needs a clear association between that capability and the team responsible for it.

The gateway has now become the default path for agent-tool access across DoorDash engineering teams. DoorDash has reported more than 200 registered MCP servers, more than 30 agents and services used by thousands of employees, and millions of tool calls each week.

What DoorDash Plans to Improve Next

DoorDash is making further investments in the gateway.

One planned improvement is stronger agent identity and user delegation. Delegation means allowing an agent to perform an action on someone else’s behalf within defined permissions.

DoorDash’s target model gives each agent a cryptographic identity and uses short-lived delegated credentials scoped to the user, agent, task, and target tool. This would make it possible to determine the connection between the user requesting work, the agent performing it, and the specific action being authorized.

Another planned improvement is dynamic tool discovery. Existing bundles already narrow the tools available to an agent. Dynamic discovery would narrow that selection further using the current task, access policy, and usage signals, so the agent receives the capabilities most likely to help with its present work.

There are also plans to improve evaluation of tool quality and security, identifying risky descriptions, detecting secrets or personally identifiable information in errors, simplifying server creation and registration, and producing redacted tool-call event streams.

These investments extend the same architectural approach. As agents gain access to more systems, the platform needs precise ways to establish who is acting, expose useful capabilities, authorize specific actions, and make the resulting activity understandable.

References

• How DoorDash Built a Centralized Gateway for AI Agent-Tool Access

Your agent returns something odd. Was it the prompt, a tool call that timed out, or a response your code could not parse? Without traces, you are guessing.

In this hands-on workshop, Serge from Sentry instruments three agents with Sentry Agent Tracing: a chatbot in an ecommerce store, a custom Slack agent, and a GitHub Action that reviews PRs. You’ll see how to catch bad tool calls and unexpected output, plus how to track token spend and performance across every agent you run.

Save your spot

A customer asks a company’s new AI-based support assistant whether a subscription purchased 15 days ago qualifies for a refund. Let’s assume that the assistant responds immediately by explaining that the company offers a 30-day refund window, describes the cancellation process, and promises that the money will arrive within five working days. All of it sounds quite helpful, confident, and complete.

But there is just one problem.

In reality, the company only allows refunds within 14 days. Also, there is no promise about the 5-day processing time. The assistant has actually taken bits of a plausible policy and invented a fake commitment out of thin air. In other words, it has lied. The customer now expects something that the company never offered.

Though this example might sound fictitious, it shows one of the most important problems in applications built with LLMs. While LLMs can produce excellent language, they can get the underlying information totally wrong.

In this article, we will look at why this problem of hallucinations happens with LLMs and the techniques that can help make LLMs more dependable for answering. Here’s what we will cover:

• What hallucination actually means in an LLM

• Three ways an answer can go wrong

• How predicting text produces invented facts

• Why models struggle to admit uncertainty

• Giving the model evidence with RAG

• Looking up missing facts with tools

• Writing useful answers

• Why an explanation is not proof

• Checking the answer before it reaches the user

What Hallucination Actually Means

Hallucination in the context of LLMs is generated information that is factually incorrect, invented, or inconsistent with the material the model is supposed to use. Here’s how a hallucination is happens:

The entire response may not be wrong. In fact, an otherwise useful explanation can contain a single fabricated date, an unsupported promise, or a reference to a document that doesn’t exist. That small detail may be the part the reader relies upon.

We can’t outright call this behavior lying, but it comes pretty close to a lie. The answer resembles a confident falsehood. However, lying usually implies an intention to deceive. A hallucination doesn’t establish that intention. The model can produce an incorrect answer through its normal generation process. In the support example from earlier, the immediate problem is that the application presents an incorrect generated policy as established company information.

The context matters a lot here. Inventing a refund policy for a fictional company is appropriate when the task asks for one. However, presenting that same invention as the policy of an actual business is a factual failure. Ultimately, the difference is more about what the answer claims to represent.

Three Ways an Answer Can Go Wrong

The AI support assistant’s errors become easier to reason about when divided into three useful categories:

• A factual hallucination contradicts reality. For example, saying that the company’s refund window is 30 days, when it is actually 14, belongs here. The company exists and has a real policy, but the assistant describes it incorrectly.

• A faithfulness hallucination concerns the relationship between the answer and its supplied evidence. For example, if the application provides a policy requiring an unused account, but the assistant says usage does not matter, it has contradicted its source.

• Fabrication involves inventing something such as a policy section, a confirmation number, or a research paper.

These three categories overlap.

A fabricated policy section can be both factually false and unsupported by the supplied documents. We can also treat factuality and faithfulness as the two broader categories, with fabrication falling under factuality failures. The main distinction to be made is between checking agreement with reality and checking agreement with the provided evidence.

This distinction throws light on an interesting situation. An AI assistant can faithfully summarize an outdated document and still give an incorrect answer about the current policy. To fix the response, we need to pay attention to both the model and the information acting as the source.

Build and scale a winning AI agent strategy (Sponsored)

Shipping agents to production is the easy part. Keeping them reliable, governable, and improving over time is where most enterprise AI programs stall.

How do top teams do it? They use an Agentic Operating Model (AOM), a step-by-step framework for aligning people, process, and technology so enterprise agents improve as they scale.

In LangChain’s latest guide, you’ll learn:

• Why AI agents don’t break like traditional software

• The engineering stack that covers the entire agent lifecycle

• Shifting from “build and deploy” to “operate and continuously improve”

Learn more

How Predicting Text Produces Hallucinations

To understand the mechanism clearly, let us look at how an LLM builds a particular response.

As you may be aware, LLMs process text in small units called tokens. A token can be a word, part of a word, or punctuation. For a given input and the response generated so far, the model calculates probabilities for the possible next tokens. A generation method selects one, adds it to the response, and repeats the whole process once again. Depending on the settings, selection may choose the most likely token or sample among possible continuations.

A likely continuation is not the same thing as a verified statement. The probability used to select the next token concerns text generation. It doesn’t certify the entire answer. Likewise, words such as “certainly” and “definitely” are generated language. Their presence cannot establish that the model has checked the policy or possesses reliable evidence.

The model learns these patterns during training.

In the initial stage, called pretraining, it works through large amounts of text and learns to predict continuations. Its internal numerical settings, known as parameters or weights, gradually capture patterns involving grammar, concepts, relationships, and facts. This produces useful knowledge and the ability to handle many different language tasks.

However, familiarity with a subject cannot establish every specific fact about it. A model may encounter 1000s of refund policies mentioning 14 days, 30 days, unused accounts, and processing delays. Those examples help it discuss refunds in an appropriate way. But they don’t tell anything about which conditions apply to this particular customer at this particular company.

Something similar happens with fabricated citations. The model learns what a research reference looks like, including author names, publication years, and technical titles. It can reproduce that structure without having a real reference to back things up.

Why Models Struggle to Admit Uncertainty

LLMs are mainly optimized for plausible text. They receive additional training to improve accuracy, helpfulness, and responses to uncertainty. Still, a fluent continuation doesn’t prove that every statement made by the LLM is true.

Some training incentives can also encourage guessing. For example, consider an evaluation approach that awards a point for a correct answer but gives the same score to a wrong answer and an admission of uncertainty. In this way, guessing offers some chance of receiving a point. Abstaining from guessing means zero points. This incentive trains the model to guess, resulting in hallucinations.

However, it isn’t totally correct to say that hallucinations are unavoidable because they are built into the architecture. Models can recognize some uncertainty and detect some mistakes. Models have useful self-evaluation abilities under particular conditions, together with limitations when those abilities must generalize to unfamiliar tasks. However, there is no dependable internal guarantee that separates every correct answer from every guess.

Asking the model for a confidence percentage also cannot automatically solve this problem. For example, a response claiming 95% confidence needs evidence that such scores are meaningful for the task. Calibration measures whether confidence estimates match observed accuracy across many cases. Without that evaluation, an impressive percentage can add another vague detail to an already uncertain answer.

Giving the Model Evidence with RAG

The first major defense addresses missing information directly.

Retrieval-augmented generation (RAG) finds relevant material and adds it to the model’s input before generation. Retrieval finds the information, augmentation supplies it, and generation produces the answer.

In our example support application, a refund question triggers a search through approved company documents. The application retrieves passages describing the applicable policy and adds them alongside the question. The assistant now has the company’s actual conditions available while writing its response.

This changes what a useful and valid answer looks like. For example, the retrieved policy makes it clear that the purchase must fall within 14 days and that the account must be unused for the refund to be valid. The customer’s query establishes only the purchase timing. The assistant should recognize that one condition remains unresolved.

RAG still has failure points. A search can return a retired policy, choose the wrong product’s policy, or miss an exception. The model can also misread a correct passage or add an unsupported promise.

For this application, documents need clear product names, effective dates, and approval status. Conflicting policies need explicit handling so that the model can know which one is applicable.

Document preparation also includes keeping related conditions together. Suppose the refund period appears at the end of one passage while the unused-account requirement begins the next. Retrieving only the first passage can leave the assistant with an incomplete rule. Regularly checking what the model actually receives helps distinguish a retrieval problem from a failure to interpret complete evidence.

Looking Up the Missing Facts with Tools

Documents explain general rules, but the LLM-based assistant also needs facts about the individual customer. This is where tool use becomes very important.

A tool lets the application obtain information or perform a defined operation outside the model’s scope. Examples include searching the web, retrieving an account record, and running a calculation. Interaction with external information sources can reduce errors on evaluated tasks.

An API provides a defined way for one program to request information or actions from another. The support application might offer an account-lookup API that returns the purchase date and whether the subscription has been used. The model requests the lookup, the application executes it, and the returned result becomes available for generating a proper answer.

RAG and tool use can overlap because document retrieval can itself be exposed as a tool. In this example, they supply complementary evidence. The policy states the conditions, while the account lookup supplies the customer facts needed to apply them.

The important thing is that the lookup must actually happen. A generated sentence saying that the account was checked doesn’t prove anything. If the tool fails, that failure must remain visible to the application. Assuming that the account is unused would simply introduce a hallucination at a different point.

Eligibility and payment are separate facts as well. Even after confirming that a customer qualifies, the assistant cannot truthfully say that a refund has been issued until the payment system reports success. Generating a made-up confirmation number doesn’t create a transaction. The application needs the actual operation and its outcome before describing that action as completed.

Writing Useful Answers

Another defense to hallucinations is to make insufficient evidence a valid outcome.

Explicit instructions can tell the assistant to identify missing information and avoid unsupported conclusions. We can disallow uncertainty by restricting answers to supplied material when appropriate, and grounding claims in supporting passages. These measures can reduce hallucinations, but they don’t eliminate them.

Either way, doing so is more helpful than a bare admission of ignorance. In our example, the assistant can explain that the purchase falls within the 14-day window while account usage still needs checking. This preserves the information already established and identifies the exact obstacle to a complete answer.

The software also needs a way to represent that outcome. Requiring every response to contain either “eligible” or “ineligible” leaves no room for missing evidence. A third state, such as “needs review,” allows uncertainty to influence the application’s next action.

Why an Explanation Is Not Proof

Another proposed defense is chain-of-thought prompting, which asks a model to work through a problem in intermediate steps.

Breaking a task into smaller parts can improve some answers and make mistakes easier to notice. However, a generated explanation may contain false premises or fail to faithfully describe what influenced the result.

For example, suppose the assistant explains that the customer purchased 15 days ago, the refund window is 30 days, and the customer therefore qualifies. The argument is easy to follow, but it depends on the wrong policy. More explanation doesn’t supply the missing fact.

A useful application instead requests a concise justification connected to checkable evidence. The assistant identifies the policy clause, the relevant account information, and how the conditions apply. This gives a reviewer concrete claims to inspect without treating the explanation itself as proof.

Checking the Answer Before It Reaches the Customer

Verification makes inspection a separate part of the workflow. This is because drafting an answer, generating verification questions, and answering those questions independently reduced hallucinations on the evaluated tasks. This supports structured checking, while leaving open the possibility that the checker also makes mistakes.

For the support assistant example, the sentence “The customer qualifies for a refund that will arrive within five days” contains separate claims. Eligibility requires the policy and account facts. Processing time needs its own supporting evidence. If nothing establishes the five-day promise, that promise should be removed.

Citations make these checks easier, provided the citations are checked too. A document must exist, apply to the question, and support the particular statement attached to it. The application can construct links from retrieved document identifiers and check whether quoted passages occur in those documents. Just placing a real link beside an unsupported claim remains an unreliable answer.

Some simpler techniques are also present, but they have narrower benefits.

Lowering temperature generally concentrates token sampling more strongly on likely continuations, but it doesn’t establish factual accuracy. The assistant may become more consistent while repeating the same wrong policy. Requiring a particular output format also helps software process a response without proving that its contents are correct.

For unambiguous rules, ordinary application code can perform part of the decision. If the policy has clearly defined conditions, code can compare the verified purchase date and usage status against those conditions. The model can then explain the result. However, exceptions and ambiguous policy language still require explicit handling.

Conclusion

The improved support workflow with fewer hallucinations can be thought of as having a clear sequence.

It retrieves the applicable policy, obtains the necessary account facts, checks the conditions, and prepares an explanation. It then checks the explanation’s factual claims and citations. If some information is missing, it generates a specific unresolved status. Lastly, cases requiring judgment are routed for further review.

The reliability of such a setup must be measured on realistic examples. Test cases should include eligible purchases, used accounts, missing records, outdated policies, and questions containing false assumptions. A customer asking how to claim a “guaranteed 30-day refund” should not cause the assistant to accept that guarantee without checking.

Evaluation also needs to consider usefulness. An assistant that always answers may make unsupported claims, while one that always declines provides little value. We should measure whether delivered answers are correct and supported, whether unanswerable cases are handled appropriately, and whether answerable questions are declined unnecessarily.

Returning to the original customer, the LLM-based assistant can now explain which conditions are satisfied, which remain unresolved, and what evidence supports its response. The wording may be less confident when information is missing, but the answer becomes more useful. A developer’s responsibility is to build a process in which the evidence determines what the application can claim in a responsible manner.

AI SREs are here. If your telemetry, alerts, and access controls aren’t ready, expect missed incidents, alert fatigue, and AI-driven investigations that go nowhere. This Datadog eBook covers how to prepare your stack for the AI SRE future, including a practical self-assessment checklist and real examples from Datadog’s own implementation.

You’ll learn how to:

• Assess whether your telemetry coverage gives an AI SRE enough signal to investigate effectively

• Design alerts and monitors that an AI agent can act on, not just surface to humans

• Set up the access controls and data quality foundations that make AI-driven investigations trustworthy

Get your checklist

For 30 years, every online business has been built with a simple assumption that your customers are human. Humans browse the website. Humans enter credit card numbers. Humans click buy.

AI Agents are starting to change this assumption. If ChatGPT is going to be your customer, the internet may need a new protocol.

That’s the question behind Machine Payments Protocol (MPP). To better understand this shift, I attended a MPP event at Stripe HQ, where I spoke with Emily Sands and Matt Schulman from Stripe, along with Brendan Ryan from Tempo.

In this deep dive, we’ll explore:

• The problem with today’s payment flow

• What is MPP

• How MPP works

• Two use cases of agentic payment

• Your Next Customer Might Be an Agent

The Problem with Today’s Payment Flow

Right from its inception, the internet has run on the basic assumption that it will be used mostly by humans. However, this assumption is no longer valid.

According to a recent report from Cloudflare, automated systems are now generating around 57.5% of HTTP requests to web content. AI tools are evolving from question-and-answer chatbots to autonomous agents that can make comprehensive plans, execute actions based on those plans, and evaluate the outcomes of those actions.

However, the internet’s payment model is still stuck in the human-centric era.

This is where MPP (Machine Payments Protocol) comes into the picture. It is a protocol that removes the friction between the buyer and the seller. This friction exists at the interface, which has been built keeping in mind a user reading a web page to figure things out. MPP creates a similar interface for AI agents so that they can settle payments with a service on the internet, irrespective of the payment method.

In other words, the innovation that MPP brings to the table is not only about making payments. It is more about the disappearance of the human from this process.

[Webinar] How to stop babysitting your agents (Sponsored)

Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part. You end up wasting time and tokens in the correction loops.

More MCPs, rules, and bigger context windows give agents access to information, but not understanding. The teams pulling ahead have a context layer to give agents exactly what they need for the task at hand.

Join us for a FREE webinar on Oct 7 to see:

• Where teams get stuck on the AI maturity curve and why common fixes fall short

• How a context layer solves for quality, efficiency, and cost

• Live demo: the same coding task with and without a context layer

If you want to maximize the value you get from AI agents, this one is worth your time.

Register now

What is MPP?

If we take the human out of the payment flow, something has to fill the gap. MPP, or Machine Payments Protocol, is the thing that fills this gap.

MPP was co-authored by Stripe and Tempo, and launched on 18th March, 2026. The core of the MPP specification has been published as the Payment HTTP Authentication Schema and has been submitted to the IETF standards track. This is the same body that also maintains HTTP.

What exactly does MPP do?

Before any money moves between two parties, they need to agree to a few things. For example, what is the cost of a service, what counts as payment, and what the proof of payment should look like.

As we discussed, humans make all these decisions just by looking. A user opens a page, finds the price, recognizes the checkout button, and knows what to do next.

Now, MPP manages each of these decisions. It also gives a specific name to each of them:

• The Challenge is what the server asks for (payment) when someone requests a service.

• The Credential is what the client sends back as proof.

• The Receipt is what the server returns along with the service delivery.

These three objects (Challenge, Credential, and Receipt) form the core of the protocol along with payment intents around charge, session, and subscription.

However, every website on the internet has a slightly different way of presenting the payment details to the users. In the case of humans, it rarely matters, because people are good at figuring things out. Software cannot do that. It needs the terms in the same place and format every time so that it can parse the details efficiently.

Therefore, MPP places these details in HTTP headers.

The server sends its terms back in a WWW-Authenticate: Payment header. The client returns the proof in an Authorization: Payment header. The server finally confirms with a Payment-Receipt header. The Challenge contains details such as an ID, the amount, the currency, who to pay, which payment method, and the validity of the offer.

The payment methods are governed separately. Each rail (card networks, blockchain, payment processor) can write and maintain its own specification for how it fits the core. This makes the protocol open enough so that it isn’t controlled by individual companies. The protocol itself is free, with no licensing fees for implementing it.

The ultimate goal of MPP is to become the language of internet payments. A service that understands MPP can sell to any agent. An agent that understands MPP can buy from any service. Neither one has to have heard of the other beforehand.

How Does MPP Work?

When an agent requests your API or service, your service returns an HTTP 402 response with payment details. The agent authorizes the payment, retries the request, and gets access to the paid resource along with a receipt. The diagram below shows the overall process flow:

Here’s how the process works:

1. The agent asks for access to a service or an API.

2. The server provides a price when it receives an access request. It answers 402 Payment Required and places the terms in a WWW-Authenticate: Payment header. This header carries a Challenge, which contains a unique ID, the payment method, the intent, and an expiry. It also has an encoded payment request that holds the amount, the currency, and the recipient address.

3. The agent checks the details. There is no account creation, form filling, or buying an API key. The agent reads the terms provided by the server. It checks the various details such as the amount, the recipient, the currency, and the validity window from the signed payload.

4. Then it authorizes the payment based on a limit that was set earlier as part of the agent’s configuration. These limits are part of the signing key rather than in the agent’s reasoning. This is a delegated key with a spending cap per period, an expiry, a list of permitted recipients, and a scope. Often, there is one key per deployment. Each can be revoked individually so that a runaway agent cannot spend beyond the limit.

5. In the next step, the agent provides the proof of payment. It repeats the identical request for access using an Authorization: Payment header with a Credential. This includes the Challenge ID with a payment payload, which might be a signed transaction, a paid invoice, or a card token. Credentials are bearer instruments that authorize the spending of real money. Servers and intermediaries don’t log them or echo them into error messages for security purposes.

6. In the last step, the server provides the access. It verifies the proof against the rail and returns the search results. It also attaches a Payment-Receipt header confirming the settlement.

The overarching rule behind this entire exchange is that the server must not perform side effects (such as database writes or calls to other services) for a request that has not been paid for. The unpaid request that triggers the 402 changes nothing except recording the Challenge. Proofs are single-use, so a Credential sent a second time is rejected.

If verification fails for some reason, the server does not return 401. It returns another 402 with a fresh Challenge along with a structured problem document that provides details about the failure, such as payment-insufficient, payment-expired, verification-failed, and invalid-challenge. The 402 status code means the payment barrier is still present. But the AI agents get to learn about the failure and make corrections.

In the case of Stripe, this setup fits quite well. A successful payment creates a PaymentIntent. The money settles into the existing balance of the business in the default currency. The same tooling works to support other features such as tax, fraud, reporting, and refunds. No customer records are created. No need to deal with user accounts. There is only a receipt on the client side and the payment on the server side. The next time an agent needs something, it has to start from zero.

Two use cases of agentic payment

Let us now look at some use cases of MPP.

Use Case 1: How AI Agents Can Pay Pennies

As we saw in the previous section, every time an agent needs something, it starts from the beginning. While starting from zero every time sounds great, it also has a cost.

Let’s say the cost for a single web search request is just a cent. However, moving a cent across the banking system often costs more than a cent.

Cards charge a flat fee per transaction. Blockchains charge a network fee per transaction and also take time to confirm. These charges don’t shrink even when the payment amount gets smaller. In other words, below a specific threshold, the cost of settling a payment can turn out to be higher than the payment amount itself.

This is the reason why the internet has relied on subscriptions, credit packs, and monthly plans. Services get bundled up until the value goes above the threshold value. However, agents don’t buy in bundles. They might buy constantly. They could make a search request followed by a lookup. Then, they might ask for another page. This could happen thousands of times inside a single task. If we settle each separately, the processing fee makes things economically unviable. If we wait for each one to confirm, the entire time goes into waiting.

To get around this, MPP does not settle every single payment. It uses the concept of sessions.

In this approach, the agent can open a session by putting money aside at the beginning. This could be a deposit into a reserve. The point is that the deposit is committed, but not yet claimed by the seller. Then, every request is paid with a signed IOU. IOU stands for “I Owe You” and is an informal written record or digital acknowledgment that one party owes a certain amount of money to another party. The agent can sign a message saying another tenth of a cent is owed. It then sends it along with the request. The server checks the signature and serves. There is no immediate settlement.

See the diagram below that shows the session-based flow:

These IOUs accumulate as requests pile up. When the session finally ends, the server claims the total amount in a single real transaction. This way, a single processing fee is divided across thousands of requests. The per-request cost falls to nearly nothing. The delay is just the few milliseconds it takes to perform a simple signature check. In other words, sessions make it possible for agents to pay pennies in an economically feasible manner.

Use Case 2: Purchase ByteByteGo Newsletter Single Article

Making micropayments becomes extremely easy with MPP. This opens up some new possibilities.

For example, a reader can use MPP to purchase the latest article from publications like ByteByteGo. Michael Blau demonstrated this using Drip. In this setup, the agents pay per use from an attached Tempo wallet without the need to buy subscriptions. Writers get paid behind the scenes even if the amount is just a single cent. Moreover, the writer can receive the money within a few milliseconds.

As MPP matures, more such use cases are definitely going to appear. Some of the exciting possibilities are as follows:

• The internet gets a native payment layer. For years, the web was funded by advertising. However, machines that can pay per request make paid access a strong alternative to the typical advertising-driven internet economy.

• Agentic workflows can now transact with the world. This opens new possibilities where agents can buy services like compute, data, and so on. AI becomes more of an economic participant.

• It opens a new supply side where we now have a reason to build things specifically for machines being the consumers.

• Payments become interoperable at the protocol layer and not at the vendor layer.

When Every Customer is a Stranger

Despite the many advantages of MPP, there are also some aspects of MPP that might impact the business in unexpected ways. As we discussed, if we take away the old signup flow, human involvement disappears. But there are also other things that vanish along with it.

Traditionally, the signup process helped the seller find out who was buying the service. The whole signup process created a record containing names, company details, and email addresses. It also helped maintain a history of what a particular customer has done before. Other important requirements, such as abuse control, depend on signups. When someone misuses a service, you could disable their account. Managing disputes and refunds also rely on signups for getting information about the user.

Once we remove the account, all of these things have to change.

With a protocol like MPP, the seller only gets a public key. The payment simply proves that whoever sent it controls the key. It doesn’t tell about which company might be behind the request, which end user the agent might be working for, and whether the same buyer has bought a service earlier with a different key. In other words, payment is no longer a form of identification.

The spending limits can help a little, but only with a specific type of problem. A capped key stops an agent from spending more than it should. It doesn’t stop an agent from spending correctly on the wrong things. The payment is valid, signed, and final. Even if the seller wants to offer a better service, there is nobody to contact after the transaction is done.

The commercial side of business will also see changes:

• The signup funnel is no longer present because a person won’t visit the landing page and register for the service.

• When a user availed the free tier, it gave an indication about a user who could be potentially converted into a paid customer down the line. That sort of data is difficult to obtain.

• The sales call provided details about someone the business could call.

This is why identity and disputes are being built as separate layers rather than being attached to the MPP payment flow.

For identity, there is a way for an agent to sign its requests so that a server can determine which automated client it’s dealing with. These use specifications backed by Visa and Cloudflare. The same key can be recognized across a complete workflow without paying again. This way, the seller can decide whether to trust a particular agent operator.

MPP, however, has not defined a refund flow at all. In a specific session, unclaimed money returns by itself. But in a one-off charge, refunds mean the seller has to send funds back to the key that was used to make the payment. A lot depends on the card network or blockchain provider on whether the refund works as expected.

MPP makes it quite clear what it can control. For example, TLS is mandatory. Challenges can expire. Proofs only work once. Servers should never log a Credential or put one in an error message.

There are also a few guardrails that are part of the MPP specification and should be followed:

• The first guardrail is to deal with a runaway agent that spends much more than intended. The solution is to have a delegated signing key with a spending cap per period, a fixed expiry, and a list of permitted recipients. Lastly, there should be a separate key for each deployment so that a particular key can be revoked.

• The second guardrail is for a dishonest seller who might quote different prices in the description and the signed payload. The MPP specification tells the clients to verify the amount, recipient, current, and expiry from the signed payload and ignore the friendly text description.

• The third guardrail is around discoverability. Directories that list available services and their costs are needed for agents to find sellers. However, according to the MPP specification, these directories and lists should only be treated as advisory when it comes to judging cost and quality.

Your Next Customer Might Be an Agent

MPP has been in production since March 2026. The volume so far is relatively small. As of August 2026, about 30,000 transactions are MPP transactions. However, the pattern might be similar to the early days of the Apple App Store, where the first year’s revenue said very little about what the platform would eventually become. What matters more here is the new type of customer that just arrived.

References:

• Introducing the Machine Payments Protocol

• Machine Payments Protocol: Stripe Docs

• Shared payment tokens: Stripe Docs

• Cloudflare Radar: Bot vs. human traffic

• Bots Now Outnumber Humans Online

• Machine Payments Protocol: Overview

• Machine Payments Protocol: Protocol

• Machine Payments Protocol: HTTP 402

• Machine Payments Protocol: Sessions

Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part. You end up wasting time and tokens in the correction loops.

More MCPs, rules, and bigger context windows give agents access to information, but not understanding. The teams pulling ahead have a context layer to give agents exactly what they need for the task at hand.

Join us for a FREE webinar on Oct 7 to see:

• Where teams get stuck on the AI maturity curve and why common fixes fall short

• How a context layer solves for quality, efficiency, and cost

• Live demo: the same coding task with and without a context layer

If you want to maximize the value you get from AI agents, this one is worth your time.

Register now

This week’s system design refresher:

• Top 9 Places to Use Jev

• LLM, RAG, AI Agent & Agentic AI

• Ex-YouTube Engineer Rebuilds YouTube in 45 Mins (Youtube)

• MCP vs Function calling

• If Claude Code is a burger...

• Rebuild Youtube Live Course Start Today

Top 9 Places to Use Jev

Jev is TypeSafe AI’s first System One Model. It is 100x faster and cheaper than frontier LLMs.

That opens up a lot of use cases people usually skip because the big models are too slow or too expensive for them.

Here are the top 9 places to use Jev instead of an LLM:

1. Model routing. Jev takes the prompt as input and routes it to a proper LLM.

2. Guardrails. Detect security risks in the prompt first. Then pass it to the LLM.

3. Gating tool-calls. Classify agent tool calls into one of their permission categories.

4. Triage inbox. Classify lots of emails into categories (e.g., spam, urgent, archive).

5. Reranking. Score passages against a query (prompt) so we can rank them by relevance.

6. LLM evals. Evaluate an LLM’s output and return a score within a range.

7. Bulk labeling. Label lots of rows from a huge table fast and cheap (map-reduce style).

8. Real-time decisions. Use cases where a fast decision is needed in a loop (e.g., trading).

9. Confidence gate. Classify anything based on confidence and act properly.

The bottomline is to use the LLM for generations and use Jev on the decisions around it.

What are other places to use Jev instead of an LLM?

LLM, RAG, AI Agent & Agentic AI

LLM: An LLM takes a user prompt and generates a response from its learned parameters. It does so by predicting tokens one at a time.

RAG: In RAG, the user query goes to a retriever before the LLM. The retriever fetches relevant information from the indexed knowledge base and passes it to the LLM along with the query. The LLM generates a grounded response, but it does not guarantee correctness.

AI agent: It has a goal and keeps track of the current task state. It can plan about what to do next, call tools, get a result back, observe it, and repeat until the goal is met. This feedback loop allows the agent to observe the results and adjust its actions accordingly.

Agentic AI: Putting one or more AI agents to meet a shared objective is what makes an AI system agentic. It is achieved through planning, tool use, and feedback. You add an orchestration layer, which coordinates work across multiple agents and workflows. Agents read and write to shared task states and can access tools, data, and the environment.

Ex-YouTube Engineer Rebuilds YouTube in 45 Mins

MCP vs Function calling

MCP and function calling have a lot in common. This confuses a lot of engineers, so here is a side-by-side comparison.

Both MCP and function calling are mechanisms that allow an LLM to have access to tools. The agent runtime sends the prompt to the LLM, the LLM decides which tool to use and emits a tool call request.

The agent runtime then handles tool call execution and returns the results back to the LLM to continue. The LLM finally produces a final output shown to the user.

What is different between the two is where the function is implemented, and how the agent handles the execution. In local function calling, the functions are implemented locally on the user's machine. The agent runtime executes them and receives the results.

In MCP, the functions can be outside of the user's machine, on some remote server. The agent runtime follows the MCP protocol and calls the corresponding server, the execution happens remotely, and the results are sent back to the agent runtime.

This allows your agent to connect to hundreds of thousands of tools that are implemented publicly and hosted remotely.

Over to you: What are the best MCP resources out there?

If Claude Code is a burger...

Before each model call, Claude Code assembles a context window from 9 distinct sources.

Think of it as a burger, each layer adds something different.

1. System Prompt: Defines Claude's role, behavior, and tone. This sets the foundation.

2. Environment Info: Git status, branch info, and current date. Pulled in via getSystemContext()

3. CLAUDE.md: A four-level instruction hierarchy: managed → user → project → local. Plain-text Markdown, so users can read, edit, and version-control everything the model sees.

4. Auto Memory: Contextually relevant memory entries prefetched asynchronously. An LLM scans memory-file headers and surfaces up to 5 relevant files on demand.

5. Path-scoped Rules: Conditional rules that load lazily when the agent reads files

6. Tool Metadata: Skill descriptions, MCP tool names, and deferred tool definitions.

7. Conversation History: Carried forward across iterations.

8. Tool Results: File reads, command outputs, and subagent summaries.

9. Compact Summaries: When history grows too long, older segments are replaced by model-generated summaries.

Rebuild Youtube Live Course Start Today

Rebuild YouTube with AI, taught by a former YouTube engineer, kicks off on Saturday, September 26. Enrollment closes in 24 hours.

Check it out Now →

What you will learn

• Scope a realistic MVP. Decide which YouTube features to build and break the work into manageable tasks.

• Work with agents. Plan changes, review generated code, and recover from bad diffs or sessions that go off track.

• Build across the stack. Turn mockups into React pages and build a backend with Postgres, authentication, and video uploads.

• Add semantic search and related videos. Use multimodal embeddings and understand how this approach differs from production recommendation systems.

• Test and deploy your app. Check features with Playwright, deploy to Vercel, and track watch time in an admin dashboard.

Check it out Now →

Check it out Now →

What you will learn

• Scope a realistic MVP. Decide which YouTube features to build and break the work into manageable tasks.

• Work with agents. Plan changes, review generated code, and recover from bad diffs or sessions that go off track.

• Build across the stack. Turn mockups into React pages and build a backend with Postgres, authentication, and video uploads.

• Add semantic search and related videos. Use multimodal embeddings and understand how this approach differs from production recommendation systems.

• Test and deploy your app. Check features with Playwright, deploy to Vercel, and track watch time in an admin dashboard.

This course is part of ByteByteGo Live, giving you 12 full months of access to all current and upcoming courses.

Check it out Now →

More live courses we offer:

• AI Evals in Practice for Engineers and PMs: Six weeks with Manjeet Singh, Senior Director at Salesforce.

• AI Cost Optimization: Four weeks with Jeremy Hintz, Engineering Lead at Meta.

• AI Engineering Fundamentals: Four weeks with Ali Aminian (Google, bestselling author).

• Build with Claude Code: Two day intensive with John Kim, Senior Staff Engineer at Meta.

• Build Production Grade AI Systems: Six weeks with Tanya Roosta, Director at AMD, with a PhD from UC Berkeley.

• Trust optimized AI Development: Two day intensive with Kent Beck, creator of TDD.

  Check it out Now →

Check it out Now →

However, inside a growing system, the same information or data point can exist in several places. For example, a database holds the original record. A cache might store a copy of that original record for faster access. A search engine index can store some fields of that record to make search more intuitive. An analytics pipeline might produce reports from it. Lastly, backups may preserve earlier versions of that record.

Each of these copies has its own purpose, update schedule, and overall lifetime within the system.

The lifecycle of data shows how information can enter a system, become useful, spread to other components, become old or stale, and eventually be deleted. Thinking about this lifecycle helps connect decisions that might otherwise seem separate, such as database design, performance, reporting, storage costs, recovery, and privacy.

In this article, we are going to look at the entire data lifecycle from creation to deletion and the decisions that need to be taken at each step.

Creation: Deciding What Information Should Exist

Read more

AI is rewriting the rules for data infrastructure. Streaming keeps critical data moving in real time, while AI introduces new demands for how that data is accessed, governed, and acted on. The Agentic Data Summit, just announced for December 9, brings together engineers, architects, and industry leaders building mission-critical data and AI workloads in production. You’ll hear real lessons from real deployments — governing agent access, migrating off Kafka at scale, and running streaming, SQL, and AI on one unified platform. It’s free, virtual, and built for teams working on what’s next in data and AI.

Register Free

Modern language models are skilled at many tasks. They can write code, summarize documents, and answer questions across many subjects. However, an application may still need something the model does not provide consistently. Its answers might omit important details, the way it classifies information might confuse similar categories, or its responses might ignore a particular expected writing style.

These things don’t mean that the model is not intelligent or doesn’t have knowledge. Often, it has the underlying capability but needs additional help applying it according to specific expectations.

That is the purpose of customizing a model. One way is to follow clearer instructions and provide better information. But when those also leave gaps between the expectation and reality of the model, further training needs to be imparted. Techniques such as LoRA and QLoRA make that training more practical by reducing the resources needed to adapt an existing model.

In this article, we are going to look at the various strategies to customize and fine-tune a model.

Here are the key learning points in brief:

• When instructions and information are not enough

• How fine-tuning changes a model

• LoRA: Learning a smaller set of changes

• QLoRA: Reducing the memory footprint of the model

• Tuning the techniques into a training process

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

When Instructions and Information Are Not Enough

The natural starting point in customizing a model is prompting.

A prompt describes the task, identifies constraints, and specifies what a good answer should contain. For example, we can add examples of the type of answer we expect to give the model a clearer demonstration of the expected behavior. This approach is called few-shot prompting.

This simple approach can be quite effective. The task of summarizing a document becomes more useful when it specifies which details matter and how the summary should be organized. Many applications need no customization beyond well-designed instructions and prompts.

However, instructions can’t provide information that is missing. If an answer depends on an internal document or a recently updated policy, the application must provide that material explicitly to the model.

Retrieval-augmented generation (RAG) is the main technique for addressing this requirement. It finds relevant information in an external source and includes it in the model’s input. The model can then use that material when producing its answer.

Prompting and RAG both work through the information supplied during a request. They can guide behavior in a big way, but they don’t ordinarily change the model’s learned parameters.

Sometimes, recurring weaknesses remain. A model might still struggle with specialized document categories or keep producing summaries that highlight the wrong details. While adding more examples as part of every request may help, it also increases the amount of input the application must maintain and process.

This is where fine-tuning offers a different option. In fine-tuning, we can train the model on many examples so that the desired behavior becomes more firmly learned as part of the model’s default behavior.

This is not an absolute division between behavior and knowledge. Fine-tuning can also teach facts. However, retrieval remains useful when information changes frequently, or answers must be traceable to an external source.

How Fine-Tuning Changes an Existing Model

As you might be aware, a model’s behavior depends on billions of numerical values called parameters or weights. Many of these are weights that influence how information moves through the model’s calculations.

During pretraining, these values are adjusted using large amounts of training data. A text-generating model learns to predict the next token, which is basically a piece of text such as a word, part of a word, or punctuation. This process helps develop broad language capabilities within the model. Models intended for conversation usually receive additional training to follow instructions.

In contrast, fine-tuning continues from an existing model using a more focused dataset. Since fine-tuning builds on top of capabilities that are already present within the model, it reduces the amount of training required for specialization.

A common approach is supervised fine-tuning (SFT). In SFT, each training example contains an input and the response the model should produce. Examples might pair documents with approved summaries, messages with category labels, or programming requests with suitable code.

During training, the model predicts the desired response tokens. The training process measures how well those predictions match the supplied targets. This measurement is called the loss. Backpropagation identifies how the trainable parameters influence that loss, and an optimizer adjusts them to improve the predictions going forward.

For example, a model may initially answer a classification request with a long explanation. However, training examples that repeatedly pair such requests with short category labels encourage the model to aim for more direct responses. The same principle applies to the details provided in summaries or the conventions followed while generating code.

Across many examples, these adjustments can increase the likelihood of desired behavior based on new inputs. Since the saved changes persist, future requests don’t need to include the entire training dataset in every input they provide.

Supervised fine-tuning does improve things, but it has limitations. Writing explicit examples for every possible scenario the model might encounter is impractical. This is where reinforcement learning from human feedback (RLHF) provides further refinement. The process begins with the model generating multiple responses to various prompts. Human raters then rank these responses based on quality, helpfulness, and safety. These rankings train a separate reward model that learns to predict scores human raters would assign to any response.

SFT and RLHF describe how examples teach the model. On the other hand, LoRA and QLoRA describe how that training is carried out efficiently. The same instruction-response dataset can be used with these different approaches.

The more straightforward type of fine-tuning, known as full fine-tuning, allows every model parameter or weight to change. This provides flexibility, but is also quite expensive. Training must store the weights along with the information used to calculate and apply their updates. Also, intermediate results that were produced while processing examples must also be stored.

This is why a model that fits on a GPU for generating answers might require much more memory for full fine-tuning. Longer examples and larger groups of examples processed together only push the resource requirements higher.

However, as we discussed, the model already has a pretty good grasp on the language aspects. Adapting it to a particular task may not require adjusting every parameter independently. This possibility leads to LoRA.

LoRA: Learning a Smaller Set of Changes

LoRA stands for Low-Rank Adaptation. It is a form of parameter-efficient fine-tuning, which means it reduces the number of parameters that training needs to update.

LoRA keeps the original model weights frozen. They still perform their calculations, but training doesn’t modify them. Instead, small trainable components are attached to selected calculations inside the model.

These additions are called adapters. They learn adjustments that are combined with the original calculations. This way, the overall output generated by the model changes because the model now uses both its existing capabilities and the learned adjustments.

An adapter operates inside the model. It doesn’t wait for a completed answer and rewrite it afterward. Its adjustments influence the internal processing that eventually produces the answer.

These small components are implemented using two compact tables of numbers, called matrices:

• The first creates a compact intermediate representation from the input for a calculation.

• The second turns that representation into an adjustment that can be added to the original result.

Training changes the values in these small tables.

The compact representation belongs only to the adapter. The original model still performs its full calculation, so the adapter doesn’t have to recreate all the capabilities the model already possesses. The reason this works is that changes often have shared patterns. If we want a model to write concise technical explanations, we don’t need it to relearn grammar, programming, and sentence construction. Much of that ability already exists within the original base model. Training simply needs to adjust how those abilities are applied.

The term “low rank” refers to representing such an adjustment through a limited set of coordinated patterns. These are learned numerical relationships, rather than explicit rules provided by a developer. Therefore, LoRA restricts how the model can change. This restriction saves resources, but it also creates a trade-off. A small adapter may be sufficient for one task and too limited for another.

A setting called rank controls the adapter’s capacity. Higher rank gives it more room to learn varied adjustments, while increasing its size and training overhead. Values such as 8, 16, 32, and 64 are possible starting points. But a higher rank is not automatically better.

At the beginning of standard LoRA training, the adapters contribute no change. Therefore, the combined model starts with its original behavior. Their adjustments develop as training processes the new examples. LoRA can also target selected attention calculations, which help the model relate different parts of its input, as well as other transformations inside its layers. Targeting more locations provides additional opportunities to adapt.

The savings are related to managing updates for a much smaller collection of parameters. The original model is still necessary and performs substantial computation. LoRA makes training more manageable without turning the underlying model into a tiny model.

QLoRA: Reducing the Memory the Frozen Model Still Needs

LoRA solves much of the expense of updating parameters, but the frozen base model still occupies memory. For a sufficiently large model, storing those weights can remain a major obstacle.

QLoRA, or quantized LoRA, addresses this by combining LoRA with quantization.

Quantization stores numbers using fewer bits and a more limited set of possible values. It preserves an approximation of each original weight while discarding some numerical detail. This reduces storage, but it can also introduce errors into the model’s calculations. The overall model structure and parameter count remain the same. Each weight simply has a more compact representation.

QLoRA commonly stores the frozen base weights in 4-bit form. The adapters remain at higher precision, allowing training to make even finer adjustments to their values. Depending on the implementation, adapter parameters and calculations can use 16-bit or 32-bit formats.

We need to distinguish between storage and computation here. A weight can be stored compactly and reconstructed as a higher-precision value when needed for a calculation. This reconstruction creates an approximation, but doesn’t recover the complete detail discarded during quantization. QLoRA uses this approach to perform the base model’s calculations, combines their results with the adapter adjustments, and trains the adapters using the combined output. The original quantized weights remain frozen.

Although training updates only the adapters, the original calculations still influence what those updates should be. Even if we freeze a part of the model, it is not removed from the learning process. Only the weights are not changed.

Since the adapters learn alongside the quantized model, they adapt to that actual combination. They can help recover task performance affected by compression, although they cannot be assumed to correct every quantization error.

For a fixed memory budget, this can make it possible to customize a larger model than ordinary LoRA would allow. The original QLoRA work has shown great savings, but the practical requirement still depends on the model, length, batch size, and training implementation.

Compressed weights are only part of training memory. The system still needs adapters, their update information, intermediate results, and working space. Lower memory use also does not guarantee proportionally faster training.

LoRA and QLoRA are therefore closely related. LoRA reduces how much must be trained, while QLoRA also reduces the space occupied by the frozen base model.

Turning the Techniques Into a Training Process

The first decision is to choose a suitable starting model. The model should already perform reasonably well in the required language and task. This is because a model that struggles with basic instructions or lacks the necessary capabilities will not do wonders even after customization.

Before training, we need to establish a baseline using a carefully developed prompt. This provides a clear picture of the remaining problems and a reference for measuring improvement.

Next, we need to prepare examples that demonstrate the desired behavior. For extraction, include messages with different wording, missing information, and details. For summarization, provide varied documents and consistent examples of how it should be structured.

Correctness is important here because training rewards agreement with the supplied answers. If similar inputs receive contradictory labels, the model can receive conflicting guidance. If approved summaries contain unsupported claims, those claims also become part of the behavior being taught.

Next, we need to prepare separate training, validation, and test sets:

• Training examples cause parameter updates.

• Validation examples help compare settings and choose a saved version of the model.

• Test examples provide a final assessment after those decisions.

Duplicate or closely related examples should not leak across these groups.

The training configuration then controls capacity, update size, and resource use. For example:

• Rank and adapter placement determine how much adjustment LoRA can learn. This learning rate controls the size of training updates. Excessively large updates can destabilize learning, while very small ones can produce little progress.

• Batch size describes how many examples are processed together. Larger batches generally require more memory. Gradient accumulation allows several smaller batches to contribute to an update. This helps a lot when memory is limited.

• Another memory-saving option is gradient checkpointing. It keeps fewer intermediate calculation results and recreates some of them when needed. This trades additional computation for lower memory use and can complement LoRA or QLoRA.

Training duration is measured in epochs, where one epoch means one pass through the training dataset. One to three epochs can be an initial experiment, but repeated exposure isn’t always better. Training longer can make the model overly dependent on the examples it has already seen. This problem is called overfitting. A warning sign for this is when performance on training examples improves alongside worsening performance on validation examples. The useful checkpoint may come before the final training step.

The evaluation process should measure the actual task. For example, classification needs correct labels, extraction needs correct values, and summarization needs faithful coverage of important details. Well-formed output alone doesn’t establish correctness.

Also, we need to check the capabilities the application still depends on. An adapter can improve one behavior while weakening another, even though the original weights remain frozen. The combined model’s behavior has changed, so it must be evaluated as a whole.

Using the Specialization in an Application

LoRA training produces adapter weights that can be saved separately from the base model. These files are generally much smaller than a complete model, but they require the compatible base checkpoint to function.

Keeping adapters separate makes it easier to manage different specializations. Where the serving software supports it, the same base model can be used with different adapters for different tasks.

Another option is to merge the adapter’s learned adjustments into the base weights. This produces a standalone customized model and removes the need for separate adapter calculations. The resulting file contains the full model, so it loses the storage advantage of an adapter-only file.

Merging and quantization support depend on the implementation. We need to evaluate the exact version that will be deployed, since changing its numerical representation can affect results.

The application can still use prompts to specify the current task, RAG to supply relevant information, and validation code to check outputs. Fine-tuning improves learned behavior while these other features continue handling their own responsibilities as before.

Conclusion

Fine-tuning is useful when a model has broad capabilities but needs more reliable specialization. It learns from demonstrations and reflects recurring response patterns more firmly in the model’s behavior.

LoRA makes this less expensive by learning compact adjustments while preserving the original weights. QLoRA reduces the memory requirement further by storing those frozen weights at lower precision.

Neither technique replaces good examples or careful evaluation. The goal is a measurable improvement on new inputs, at a cost the application can support. A clear task, a suitable starting model, and representative training data matter as much as the choice of training method.

References:

• LoRA: Low-Rank Adaptation of Large Language Models

• QLoRA: Efficient Fine-tuning of Quantized LLMs

How do you optimize AI when performance matters? And how can you apply AI to make performance better (and easier) than ever before? That’s what 30K engineers will explore at P99 CONF. Here’s a taste of the talks you can expect

• Sandboxmaxxing at Lovable: Every Prompt Gets a Sandbox in < 1s

• The Autonomous Performance Agent: A Netflix Production Story

• Lessons Learned from Building Crazy Fast, Open Source Infrastructure for AI Agents

• Give the Agent a Cluster: Effective AI for Performance Engineering at Scale

• Managing 500 Billion+ Files for AI Workloads

• How to Improve Your Cache Algorithm Using AI

GET YOUR FREE TICKET

Bonus: Registrants get immediate 30-day access to the complete O’Reilly library, and attendees can enter to win 1 of 500 free swag packs.

If you have used voice assistants before, you have probably experienced unexpected interruptions. You talk with the system, and the moment you pause briefly to think of the right term, it starts talking. Then you interrupt it so you can continue. This is a common frustration, because most voice models can either listen or speak, but not both at the same time.

Newer voice models, like OpenAI’s GPT-Live-1, change that by listening and speaking at the same time. The model constantly decides whether it should stay quiet, interrupt, or start talking. This makes the conversation feel more natural with fewer unintentional interruptions.

Under the hood, these systems combine a new generation of voice model architecture with a serving system optimized for low latency. To understand how it all works end to end, we met with engineers on the GPT Voice team, Zahan Malkani and Justin Uberti (who created WebRTC). We thank both of them for sharing the details with us.

In this article, you’ll learn:

• The three generations of voice systems, including cascaded pipelines, turn-based end-to-end models, and full-duplex models

• Delegating thinking from talking, the core idea behind GPT-Live

• The engineering behind the serving system, including the live and async paths

• How evaluation is different in full-duplex voice systems

• Engineering lessons for building realtime systems and what is next for voice

Build and scale a winning AI agent strategy (Sponsored)

Shipping agents to production is the easy part. Keeping them reliable, governable, and improving over time is where most enterprise AI programs stall.

How do top teams do it? They use an Agentic Operating Model (AOM), a step-by-step framework for aligning people, process, and technology so enterprise agents improve as they scale.

In LangChain’s latest guide, you’ll learn:

• Why AI agents don’t break like traditional software

• The engineering stack that covers the entire agent lifecycle

• Shifting from “build and deploy” to “operate and continuously improve”

Learn more

Three Generations of Voice Systems

The input to a voice system is the user’s audio. The output should be the response in audio format played back to the user. While the input and output are always audio, what happens in between depends on how we design the system. Voice systems have gone through three generations of architecture:

• Cascaded Design

• Turn-based End-to-End

• Full-duplex Architecture

Let’s examine each in more detail.

1. Cascaded Design

The cascaded design chains three separate models. An automatic speech recognition (ASR) model transcribes the user’s speech into text, an LLM then generates a text reply, and a text-to-speech (TTS) model finally reads the reply for the user.

Each model in the design focuses on the task that it is good at. ASR is good at converting speech to text. It does not have the knowledge that LLMs have. So it only focuses on the conversion. The LLM then relies on its capabilities to understand the user’s query and respond to it accordingly. Once the LLM produces the response text, the TTS model just synthesizes it into speech that sounds natural.

Cascaded design works in practice, but it has two main issues. First, information loss. The LLM only sees a transcript, so vocal information like tone and emotions will be unavailable to the model.

Second, the system is complex and slow. Since the three stages run in series, their latencies add up. The user must wait until all three stages are complete. Also, running three models means building, serving, and scaling three separate systems which is complex in practice.

Due to these limitations, the second generation of voice systems was designed: turn-based speech-to-speech models.

2. Turn-based, Speech-to-Speech

The second generation introduces an end-to-end speech model, a single model trained to consume audio as input and produce audio directly as output.

This design solves a key limitation of cascaded systems: it allows the model to take into account vocal information since it processes the audio directly.

While this is a good improvement, the interaction itself remains turn-based. A small model, called a turn detector, still decides when the user has finished, and only then does the main speech model start its job. The turn detector is not new here. The pause detection step in the cascaded design is the same component. Both generations rely on it, and this turn-based design is the source of unnaturalness.

The detector has a challenging job. If it detects too early, it cuts the user off in the middle of a thought. If it decides too late, users experience awkward delays. Interruptions face the same challenge. When users start talking over the voice system, a separate mechanism has to stop the audio and clear the buffers. If the interruption detection is too sensitive, spurious interruptions can be triggered from background noise. If detection is too conservative, interruptions take too long and feel sluggish, while short interjections like “yes” and “no” can be entirely missed. Justin points to this machinery as the reason earlier voice systems felt unnatural.

These models are also expensive to keep updated. When there is a new, more capable pre-trained LLM, it requires a new full speech-to-speech training run on top of the checkpoint. So voice models always lag behind the newest frontier models.

The third generation of voice models, full-duplex architectures, fixes the turn-taking problem.

3. Full-duplex Architecture

Full-duplex models are designed so they can both listen and talk simultaneously. The model continuously produces audio tokens. When it should stay silent, it simply produces silent tokens. Similarly, it continuously processes the input audio tokens. When the user is silent, those are just silent tokens. This design removes the turn detector entirely.

To better understand, let’s use an open full-duplex model, Moshi [x], as a reference. Moshi converts audio into discrete tokens, similar to how text is converted to tokens for LLMs. The model processes these tokens as inputs and emits new tokens on a fixed clock, roughly one frame every 80 milliseconds. Silence is simply another token that decodes to silence. So the model just needs to learn the behavior during training and implicitly understand when to listen and when to talk from the training data.

Full-duplex solves the interruption problem and the unnaturalness in conversations, but it creates two new engineering challenges. First, since the model is always running and predicting the next token, serving is costly. Second, the model needs to respond within milliseconds, so it cannot be too large in capacity.

OpenAI launched GPT-Live-1 in July 2026, a family of full-duplex voice models. It is designed around the challenges described above. The voice model stays small and fast so it can respond within milliseconds. It also relies on delegation to perform expensive reasoning while the conversation keeps going.

How the GPT-Live System Works

This section explains how OpenAI managed to build a voice assistant around a full-duplex architecture. We cover the ideas and techniques that make it practical at scale in production.

Separating Talking from Thinking

Normally, a frontier LLM might reason, search the web, and then respond, which can take several seconds. In voice systems, that translates to a few seconds of silence, which is not ideal.

To fix this, OpenAI separates talking from thinking. A voice model handles the conversation with the user. Another capable model performs the necessary reasoning and tool calling for more complex queries. The serving system is also built around this idea. It delegates the request to the capable model when needed, while continuing the conversation with the user.

As the figure below shows, a question like who won last night’s game cannot be directly answered by the speech model using its internal weights, so the voice model delegates it to GPT-5.5. It keeps the conversation going while the search runs, and reads the answer for the user once it is available.

This design meaningfully reduces the trade-off between speed and quality. While the frontier model looks up information, the voice model remains available to continue chatting. Previously, if you wanted smarter answers, you looped in more systems, and the response took longer. If you wanted quick responses, you used a smaller model, and the answers got worse. With one model for talking and another for thinking, the voice system gets both.

Another benefit of this design is modularity. When a new frontier model is available, less engineering work is needed to switch the voice assistant to it.

The Two Serving Paths

Serving two models is tricky. Audio frames should be sent every few milliseconds, but a delegation can take seconds. So we cannot have a shared process for both. GPT-Live splits the traffic. The live path carries audio only. It moves audio between the client and the voice model as fast as possible. The async path handles anything other than the audio, which can take longer. This way, a slow tool call may delay the async path but not the live path.

How to make the live path fast?

The main requirement of the live path is that audio must move between the user and the model on a fixed clock. The model processes frames as input and produces them as output in real time. So the user hears a delay when it occurs. As Zahan explains, whenever a bottleneck makes the system fall behind, it starts producing audible artifacts.

To meet this requirement, lots of optimizations are needed. For example, the connection must open quickly, the model must keep up with the incoming stream, and frames must be delivered on time. Here are a few engineering techniques OpenAI adopted to keep the live path fast:

• Starting a session in one round trip

• Having cheap continuous inference

• Handing off live conversations between model instances

1. Starting a session in one round trip

When the user taps the voice button, the client must establish a connection before any audio can flow. This requires a few network trips, depending on the underlying protocol. GPT-Live uses WebRTC, which is used in most video calling apps. But a standard WebRTC session takes six steps to establish before a single audio frame can be sent. On a mobile network where one round trip takes 60 milliseconds, setup alone can cost over a third of a second.

To improve this, OpenAI built WARP (WebRTC Abridged Roundtrip Protocol). It relies on the idea that instead of having the steps one after another, we have them happen at the same time. This shrinks the number of network trips to only one.

2. Having cheap continuous inference

To understand what makes full-duplex systems expensive to serve, let’s compare them with a chatbot. In chatbots, a request arrives, and then the model produces tokens. When the response is complete, the model sits idle. A full-duplex model, on the other hand, has no idle time. Audio streams into the model, and the model produces frames continuously. This happens even when the user is mid-sentence or pauses to think.

Continuously sampling a model, many times per second, for every user, is expensive. OpenAI reduces this cost by keeping each conversation loaded on the model. So instead of re-reading the whole conversation on every request, each session stays connected to the model instance that holds the conversation in GPU memory. When a new audio frame arrives, the model only processes that one frame. In addition, common techniques like batching and speculative decoding can be adopted to keep the system fast.

3. Handing off a live conversation between model instances

Keeping the conversation loaded on one model instance creates a new problem. Model instances may become unavailable. They need to stop and start with demand or receive updates. They may also fail occasionally. In regular systems, this is easy to handle since we can send the request to another available instance. In GPT-Live, the session is tied to the instance that holds its conversation in GPU memory.

OpenAI created a managed handoff mechanism. The system prepares a replacement instance and loads the full conversation beforehand. Once the new instance is ready to be used, the switch happens, so the conversation can continue without any interruptions. This is also useful when other operations need to happen. For example, when the context needs to be compacted, the shortened conversation is prepared on a replacement instance while the original keeps talking. The switch then happens the same way.

The Async Path

The async path handles everything that is not audio, like delegations and tool calling. These jobs, by their nature, are large and can take longer to complete. Still, results should come back fast enough that the two models feel like one system. Back to our example session. The user asks who won last night’s game, and the voice model starts a delegation. The voice model can buy a little time by doing things like acknowledging the question or thinking out loud for a moment. But it cannot do anything when an answer takes too long. So everything in the delegation loop, including prompt processing and routing, needs to happen fast.

To cut latency, we should first understand what is causing it. Most of the time is spent on reading the request. A model that receives a request has to process the entire prompt (prefill) before producing tokens. In a long conversation, that alone can take a noticeable fraction of a second. OpenAI’s fix is to do this reading before it is needed. When a user starts a conversation, the server creates an inference session with the frontier model and sends the conversation so far. By the time the first delegation happens, the model has already everything it needs to produce tokens.

How to Evaluate a Full-duplex Voice System?

A turn-based speech model can be evaluated one turn at a time by sending a request and scoring the response. But with a full-duplex architecture, there are no turns. There is only one continuous stream of audio. This makes evaluation different.

In full-duplex systems, instead of scoring turns, we can evaluate three parts:

• Conversational behavior

• The health of the stream

• Testing the system on real production traffic

1. Conversational behavior

The first question is whether the model behaves well in a conversation. This mostly comes down to timing. The model has to continuously decide if it should stay silent, interrupt, or talk. When the user speaks while the model is talking, it has to decide if this is a real interruption or just background noise. These two decisions are called endpointing and barge-in detection.

To evaluate these behaviors, each decision is scored like a prediction. Did the model detect the end of the turn correctly? Did it recognize a real interruption? The mental model for eval is the same. We collect eval data (natural conversations), have it annotated around the dimensions that we want to measure, run inference on it, and evaluate.

2. The health of the stream

The second dimension of eval is whether the stream itself is healthy. In most serving systems, this is answered with a percentile target like p95. If the p95 latency is good, 19 out of 20 requests feel fast. This works fine as long as unusually slow cases are kept very low. A normal user only sends a request once in a while, so a slow one is quickly forgotten.

But p95 is not reliable in full-duplex architectures. Given that the model runs continuously, a p95 event happens every 20 inferences, which translates to several times per minute. Even p99 events show up regularly. So the system has to be engineered around p999. In practice, this means designing for fast recovery, since some slow frames are guaranteed to happen in every session.

3. Testing the system on real production traffic

The last layer of evaluation checks whether the system works reliably under real traffic. The technique for finding out safely is called a silent launch. Users keep talking to the old system as usual, while a small percentage of voice sessions are routed to the new system. This kind of silent launch catches problems that are normally hard to find. For example, it can find bottlenecks in unexpected places. In GPT-Live’s silent launch, the team found out that a CPU-side service ran out of capacity before the GPUs did. This is something that is generally hard to learn with ordinary tests.

What Other Teams Can Learn From GPT-Live

The broad lesson from OpenAI’s GPT-Live is that building for realtime serving is different and more challenging than traditional serving. For example, in voice systems, users hear the worst frame, so average latency is an insufficient metric to monitor. The tail is more important and deserves more engineering. In addition, capacity planning is different in such systems. When a user is having a conversation, the session is occupied the entire time. So capacity should be measured in concurrent sessions instead of per request.

The other lesson is that complexity should be handled inside the model. For example, turn detection used to be a small, separate component, and it was a main source of unnaturalness. Moving that decision into the model behavior, which has the greatest reasoning horsepower, made the conversation more natural and the system simpler. In general, whatever stays outside the model should be kept small and focused on the realtime work. This ensures the system remains simple and maintainable.

What’s Next

OpenAI believes the next generation is voice driving a computer. In the ChatGPT desktop app, it takes screenshots to see what you are working on, starts long-running tasks, and reports back. Along the way, you can ask for status, answer questions, and redirect it mid-task. Zahan says it feels like science fiction.

Some questions are still open. Zahan says it remains challenging for a voice model to delegate tasks to other models while keeping a live conversation with a user running smoothly. The delegation path might tolerate delay, but live conversation is more sensitive. A general-purpose protocol would mean “part of your brain responds slowly.” Justin says voice AI was like in its GPT-3 days. With GPT-Live-1, he puts it at GPT-4, with many interesting problems still ahead.