Authentication is often the least-tested part of an app. Live environments need network access and real credentials, while mocks miss the failures that break production.

@workos/emulate runs the WorkOS API locally for development and automated testing. Seed users, organizations, RBAC roles, and SSO connections, then test full AuthKit login flows, signed webhooks, token refresh, and error handling without touching production. Responses and event shapes come from the WorkOS OpenAPI spec, so your tests exercise the same surface your app uses in production. Run it locally or in CI with npm, Homebrew, Docker, or a standalone binary.

Run WorkOS locally

A customer making a card payment expects it to get approved or declined almost instantly. This single requirement shapes everything about how a payment is built, because the entire processing of the payment has to finish within this short window.

We recently spoke with Ben Cane, a Distinguished Engineer at American Express, to understand how their platform handles it.

To understand how things work, let us picture what happens when a transaction arrives at American Express. It is routed into one of several independent processing units and moves through a chain of microservices. However, partway through, one of those services begins to fail, and the customer at the checkout terminal is still waiting.

Just retrying the transaction inside the failing unit carries obvious risk. Also, moving the transaction to another server is an even harder problem. This is because the second unit would need to know how far the first transaction went. Sharing such knowledge creates a dependency that can turn two independent units into a fragile system.

The American Express engineering team handles this with a cell-based architecture that helps keep the impact of a disruption to a minimum. In our chat with the engineering team of American Express, we learned that most of the engineering effort goes into keeping that isolation intact under pressure. In this article, we will try to understand how the transaction runs through such a cell-based architecture and how the payments are processed even when some services are failing.

Core Payments Ecosystem

In a typical payment flow, American Express sits in the middle of a chain. A merchant holds a relationship with an acquiring bank that sends the transaction to American Express, and American Express delivers it to the card issuer that holds the account balance. This middle hop is the core payment network’s job.

See the diagram below that shows this setup:

In 2018, American Express began modernizing this platform onto cloud-native infrastructure. Immediately, it was clear that one assumption had to change. While the older systems ran on hardware engineered to stay running, the cloud infrastructure behaved differently. Servers could disappear for reasons outside any one team’s control. You have to assume failures completely outside your control will happen more often, and build the application around that.

Two familiar patterns were considered:

• Event-driven processing fit poorly. The workload needs to run at scale with low latency and a real-time response, leaving little room for delay. Asynchronous work exists, but the answer itself still has to arrive fast.

• A monolith is where most teams start, but it fits the scaling requirements poorly.

The design American Express engineering team ultimately chose was driven, in part, by decades of internal thinking. Ben pointed out that back in the SOA era, long before microservices existed as a concept, his teams were already trying to reduce the impact of a single failure on a specific system. The patterns were present even before the more formal term of cell-based architecture was coined.

Cell Boundaries

In a cell-based architecture, a cell is a complete, self-sufficient copy of the payment processing stack. In other words, everything required to process a transaction lives inside one boundary. This includes microservices, databases, DNS, and supporting infrastructure.

There are five key properties of a cell:

• It deploys independently and processes payments on its own.

• It owns its microservices, databases, and other components.

• It forms a single failure domain, so problems inside stay inside.

• It can be pulled out of rotation for maintenance or during an incident, with the rest of the platform carrying on.

• It does not have synchronous cross-cell dependencies in the critical path.

Reference data still replicates across cells, observability data can aggregate across cells, and router instances communicate with each other across cells. What stays out of the critical path is any blocking call between cells during transaction processing.

To put it differently, a cell is defined by its failure boundaries rather than by a specific infrastructure construct. Cells stay within a single region with everything needed for processing local inside that boundary.

Deciding on the appropriate size of a cell requires judgment. The engineers at American Express think of it as a balancing act. You don’t want to put the entire enterprise within a cell. Ideally, it is better to draw the boundary around the journey being supported. In the case of payments, this means the minimum components required to produce an immediate answer. In other words, the real-time journey of a transaction. The after-the-fact journey can still be handled in a different cell.

To be clear, cells are not the same as microservices. While microservices divide a system by function, cells divide it by failure. One cell can contain many microservices. The boundary matters because whenever we leave the cell, we enter dangerous territory and need the right resiliency capabilities, processes, and business logic to handle the potential failures.

Data Locality

A cell remains self-sufficient only when the data it needs already resides within it. The American Express engineering team has three strategies to deal with data, chosen according to how often the data changes:

• Immutable data is set up once and stays fixed thereafter.

• Semi-static data changes anywhere from every couple of hours to once a year. For example, exchange rates, merchant category codes, and country codes.

• Dynamic data changes with every transaction.

For the first two categories, American Express distributes the reference data to every cell before any transaction needs it. The alternative would be a fall-through read, meaning a lookup that misses the local cache and travels to a central system of record while the transaction waits. Pushing the data ahead provides multiple benefits:

• The first transaction avoids paying for a cold cache.

• The critical path avoids a synchronous call leaving the cell.

• The replication work runs entirely outside the transaction path.

Ben contrasted this with the more common pattern. He said that they lean toward push and distribute rather than the traditional pull and then cache, which spares that first transaction from building up caches.

However, the same approach falls short for dynamic data. Replication runs quickly and asynchronously. But it still leaves a window where a cell may hold stale state when a transaction arrives. Sending a transaction to a cell with stale data would add latency and risk a processing failure.

To deal with this scenario, American Express inverts the problem. Rather than moving the data to the transaction, the platform moves the transaction to the data.

This works through deterministic routing, meaning the decision follows from the transaction’s own contents rather than from cell load or availability. Some example attributes for this transaction content are partner, market, and payment type. A component called the Global Transaction Router makes the decision at the front door. We will look at it in more detail in the next section.

Deterministic routing is one of two modes. The router also supports priority-based routing in which cells carry a specific ordering and traffic goes to the highest-priority healthy cell available. The choice of the mode that is used depends on the use case.

However, this routing is not a universal rule. American Express applies deterministic routing selectively, where transactions require strong consistency between them. This depends on the transaction type. Some types carry minimal data requirements and can be routed freely.

Message-based replication between cells continues throughout, so failover data exists in more than one place. Timing is the important part here, because every in-flight transaction proceeds without waiting for replication to finish.

Global Transaction Router

The Global Transaction Router routes traffic and enforces the cell boundary at the same time.

Every transaction enters a cell through the router, and any transaction moving to a different cell travels back through it as well. Cells lack any ability to communicate directly, which makes the router the single path between them. The result of this setup is a payments mesh that connects cells globally while also keeping them independent.

See the diagram below:

The router handles external traffic on the same terms. When a cell finishes its work and the transaction needs to reach a card issuer, the cell returns it to the router. The router then makes that outbound call. Cells don’t talk outside their own boundary, whether the destination is another cell or an outside institution.

Concentrating that much responsibility in one component raises the obvious question: How does a component that every transaction depends on avoid becoming the platform’s weakest point?

The answers from American Express engineers can be split into parts.

The first is keeping the router deliberately simple. They keep business logic out of the router and give it just enough message parsing capability to extract certain values and route on them. The knowledge of what those values represent falls outside its job. The inputs vary by transaction type, sometimes a header value and sometimes a field inside the card transaction message.

Keeping this restraint is crucial because a router that was too deeply integrated with payment logic would accumulate business rules. Those rules would require data, and that data would require lookups. The component would gradually become a centralized system, which the cell-based architecture seeks to avoid.

The second direction is reducing what the router depends on. Here are a few things that are taken care of:

• Dependencies stay minimal. The closer a component sits to the edge, the fewer dependencies it carries.

• State stays out of persistent storage. Router state lives in non-persistent stores, which keeps instances as close to stateless as the design allows.

• Remaining dependencies run asynchronously. Logging uses an asynchronous logger with a buffer truncation policy, so a full buffer drops log records rather than blocking transaction processing.

• Configuration loads into memory and updates asynchronously, so an unreachable config service leaves the router running on last known good values.

• Instances run in parallel across regions. Multiple instances, multiple regions, and multiple connections mean an unavailable path always has a backup.

During the discussion, Ben also gave a clear preference on deployment posture. He said he would avoid active-standby as much as possible and run something like this as active as possible, since active-standby earns its place mainly when you have state to manage.

The lesson even applies beyond payments. Some chokepoints are part of the structure, because the architecture needs one place where cross-boundary decisions happen. The best way to deal with that is to make that chokepoint simple enough to trust, and spend the engineering budget on keeping it that way.

Credit Card Authorization Flow

Let us now look at a card authorization flow that we got from the American Express team to understand how things work even more clearly.

The journey of a transaction starts outside the American Express systems. A card gets used at a merchant’s point-of-sale terminal. From there, the sequence goes as follows:

• The Global Transaction Router receives the request first.

• The router applies deterministic or priority-based routing. The choice depends on the use case.

• Inside the cell, a collection of microservices performs validation, enrichment, transformation, and issuer determination.

• The cell returns the transaction to the router.

• The router sends the request to the appropriate card issuers for authorization.

• The issuer response arrives back at the router.

• The router uses deterministic routing to reach the cell holding the context of the transaction for authorization.

• The cell validates the response.

• The transaction travels back to the merchant’s acquiring bank.

• The final confirmation of the payment is sent back to the POS terminal.

One thing to note is that the cell never contacts the card issuer directly. This preserves the rule that cells should not communicate with anything beyond their boundaries.

See the diagram below that shows the process in detail:

Mid-Transaction Failure

Let us now return to the transaction that we talked about at the beginning of the article.

Payments processing at American Express uses an orchestrated microservices architecture. This means one orchestrator microservice manages the workflow and calls the other microservices in turn. This orchestrator also monitors the health of those microservices continuously and detects failures.

Here is the full sequence of events that happen once a required service starts failing, or when too few instances of that service are running to handle it.:

• The orchestrator detects the failure and halts processing.

• It sends the transaction back to the Global Transaction Router.

• The router selects a healthy cell.

• Processing restarts in that cell using the original transaction data.

The last step is where the design diverges. American Express discards the partial work. Every microservice call the failing cell completed gets thrown away, and the second cell starts from the beginning with the same input. This rerouting covers both new transactions arriving at the failing cell and transactions already in flight inside it.

The reasoning behind this approach goes back to the boundary problem. Resuming a transaction would require the second cell to read state from the first. However, that link would create shared state between cells. Shared state brings synchronization problems and consistency risks during failover, which is the exact problem this architecture seeks to avoid.

The result of this choice is that cells stay loosely coupled. Each cell runs its own database clusters, the microservices inside a cell communicate with the local cluster only, and a rerouted transaction is processed with zero reliance on state from the previous cell. The orchestrator also runs application health checks against the service mesh’s readiness endpoints. When overall cell health degrades past an acceptable threshold, the orchestrator reports the cell as unavailable.

Redoing completed work sounds wasteful. However, the trade-off works because a payment is short, so redoing a few hundred milliseconds costs very little against the alternative of a permanent structural dependency between every pair of cells. In a system where one unit of work runs for minutes, checkpointing might seem more favorable as an approach.

Recovery Semantics

Restarting a transaction elsewhere can be done up to a specific moment. Think of it like a point of no return. The exact position of that moment is a design decision that a team needs to make. For example, rerouting is safe while a transaction remains inside the core payments ecosystem, and once it has gone to an external system such as a card issuer, it stays where it is.

Card authorizations are structured so the point of no return falls toward the end of processing. American Express placed the irreversible step as late as the payment flow allows, making the recoverable window as wide as it can be.

For payment types where late rerouting of the transaction is not possible, safety comes from idempotency.

Each transaction carries a unique identifier consistent across every retry and reroute, and downstream systems use it to suppress duplicates. Failback is controlled with canary capabilities using percentages. They avoid bouncing everything back the moment a cell returns. Also, shifting traffic by percentage lets American Express drain a cell gradually, validate a recovering cell under partial load, or respond carefully during an incident.

But this opens up another question: How does the platform prevent a recovered cell from writing stale state after its transactions moved elsewhere?

There are three mechanisms here that narrow the window:

• Speed: Card payments move so fast that by the time a failed cell returns, the transaction has usually completed elsewhere.

• Idempotency identifiers: Duplicate suppression happens downstream, using the identifier that travelled with the transaction.

• Paced recovery: Percentage-based traffic control governs when and how much work a recovering cell receives.

The specific details can vary. Recovery in finance might mean waiting for something to sync, accepting eventual consistency, or creating a compensating transaction, depending on the transaction type and where the failure occurred.

Design Tradeoffs

Let us now look at the trade-offs in the architecture implemented by American Express.

• Duplicated services: Enforcing the boundary sometimes produces duplicate implementations of the same service across cells. American Express accepts this cost because it preserves cell independence and removes cross-cell network hops. Naturally, preventing cross-cell dependencies grows harder as a platform grows, so the cost increases over time.

• Dropped log records: The buffer truncation policy means that under sustained pressure, the platform keeps processing transactions while losing part of the record of what it did. To be clear, these are application logs meant for general observability and not critical transactional events.

• Delayed global visibility: Each cell writes logs, metrics, and traces locally first, with aggregation to global dashboards happening asynchronously. Losing part of the stack degrades visibility for one cell instead of the platform. The global view always lags.

• Rejected transactions: Data synchronization between cells runs asynchronously. The consistency requirements depend on the transaction type and the business rule. When a transaction requires strong consistency, but the required data cannot be validated or turns out to be inconsistent, American Express may reject that transaction to preserve data integrity.

Also, one thing to note about cells is that they change how many fail at once rather than how often failures happen, since running more independent units arguably produces more individual failures. In other words, management overhead and complexity are traded off for reduced impact of a failure.

Conclusion

The payment platform built by the American Express engineering team survives a cell failure because the design doesn’t allow one transaction to depend on two cells at once.

Here are the key points that make it work:

• Two data strategies, one goal: Reference data that changes rarely gets pushed to every cell ahead of time, while data that changes constantly stays put and the transaction travels to it.

• A thin component at the chokepoint: The Global Transaction Router carries enormous responsibility and very little logic, which makes it dependable.

• Restart in place of resume: Discarding partial work costs a few hundred milliseconds and removes the shared state that would link every cell to every other cell.

• A recovery window with a chosen position: Placing the point of no return late in the sequence widens the range in which failure remains survivable.

• Recovery semantics from the domain: The pattern provides structure, and business logic provides the rules for what a partially completed transaction requires.

References:

• Cell-based Architecture for Resilient Payment Systems

Writing code is fast, shipping it is still hard. Railway’s push-button compute, storage, and networking is built for both small and hyperscale software.

Humans and agents alike operate on one vertically integrated system on our own hardware. Get superior speed, better economics, and a lot more calm.

Sign up with the link below for free cloud credits and let your agents cook.

Deploy now

What is an LLM-powered application?

It’s just like any software application. But there is one major difference. Apart from normal processing, the application also sends data to a large language model (LLM). It then uses the model’s response to carry out a task. For example, a customer support chatbot might depend on an LLM to answer user queries. A document-processing system can use an LLM to extract names, dates, and invoice amounts from the uploaded documents. A coding assistant might ask an LLM to write a piece of code and integrate it into the logical flow.

On face value, we might feel that building such an application is quite simple:

• The user sends a request.

• The application sends a prompt to an LLM.

• The LLM returns an answer.

• The application displays that answer or uses the same for some other processing.

However, there are chances that any of these steps can fail. The network might be unavailable. The LLM provider might reject the user request for various reasons. The request might go through, but the model might return invalid JSON. The LLM can misunderstand the instructions and invent false information due to hallucinations. They might also take a long time to respond.

The techniques for resiliency and error handling help prepare the application to handle these situations in the best possible manner. Here’s what we will cover in the article:

• What is the meaning of error handling and resiliency?

• Why do LLM applications need special treatment for error handling?

• How does a request flow in an LLM-powered application?

• Main types of errors and failures

• How to classify failures to take appropriate action?

• How to retry a request correctly?

• How to manage timeouts and deadlines?

• How to deal with business rules in an LLM-powered application?

• How to manage fallbacks and graceful degradation?

• How do circuit breakers work in this setup?

• Rate limiting, queues, and concurrency control

• How to make LLM tool calls safe with idempotency?

• How to stream the responses from the LLM?

Meaning of Error Handling and Resiliency

Before we get into the details, let us understand what error handling and resiliency exactly mean.

You can think of error handling as a part of the program responsible for deciding the appropriate action when something goes wrong. For example, consider an application that calls an LLM API. If the request is successful, the program processes the response in a normal way. But if it fails, the program might have to choose from various options, such as retrying the request, showing an explanatory message, using a backup model, or recording the failure for further investigation.

Any of these approaches is much better than allowing the entire application to crash. But we also cannot treat every problem in the same manner. A real production application should be able to distinguish between different types of failures so that we can take the right action. Therefore, good error handling needs an understanding of what failed and why.

In contrast, resiliency is an application’s ability to continue doing its job even when some parts of the system are failing. We don’t need an application to operate perfectly to be resilient. Failures are fine. But they should happen in a controlled manner. This is also known as graceful degradation.

For example, consider a travel assistant that uses an LLM to build personalized itineraries for travellers. If the primary LLM it depends on is unavailable, the application can use a smaller backup model for creating the itineraries. If that model also becomes unavailable, the application can display previously created destination guides for the same locations. Of course, these answers won’t be personalized, but the user won’t just see a useless error message.

While error handling deals with an individual failure, resiliency is more concerned about the behavior of the entire system even when things are failing.

Why do LLM Applications Need Special Treatment for Error Handling?

Traditional software apps work according to well-defined rules. If we write a function that adds 5 and 7, it will always return 12. If the function returns successfully, the result is deemed valid.

With LLMs, we don’t have this luxury. This is because the output from LLMs is probabilistic. In other words, the same prompt can produce different answers every single time. Also, a successful API request to an LLM doesn’t guarantee that the response we receive is correct or even usable. For all we know, the response might be utterly gibberish. This means that an LLM API call might appear successful based on status code, but it might be a failure in logical terms. There are several things that could be wrong with the response:

• The model might have ignored some part of the instructions.

• It might have returned a pure text response when the application expected JSON.

• The LLM might hallucinate and invent a product, policy, or piece of information that has no basis in real data.

• The model might return an incomplete response because it reached its token limit.

• The LLM can refuse a harmless request because it misunderstood the context.

• The LLM might select the wrong tool or call a tool with invalid arguments.

• Lastly, the LLM might return an answer that violates some important business rule.

We can broadly notice two categories of failures over here:

• Technical Failures: They happen when the system can’t even complete the operation. This involves stuff like network errors, timeouts, unavailable service, invalid credentials, and so on.

• Semantic Failures: These types of failures show up even when the operation is technically successful. But the result might be incorrect, unsafe, irrelevant, or even unusable.

Conventional error handling focuses on dealing with technical failures. However, LLM-powered applications need to handle both technical and semantic failures.

How Does a Request Flow in an LLM-powered App?

LLM-powered apps contain much more than an LLM. We have many different components that come together to make an application complete. See the diagram below:

As an example, consider a shopping chatbot that checks products based on the user’s preference. It first runs a search on a product database and gives the results back to an LLM. It then asks the model to select the right products for a particular user and then calls an inventory API to find out the availability of those products.

We can have failures at every boundary within this application. The user might provide some invalid information. The product search may return no results. The request to the LLM might time out due to external reasons. The model may return an unknown product identifier. The inventory service might go down. Lastly, even if the response is somehow generated, it might be interrupted while streaming to the user.

To make an application truly resilient, we can’t treat the LLM as an isolated component. We need to consider the path of the request from the beginning to the end and identify potential problems.

Main Types of Errors and Failures

Let us now look at the main types of errors and failures that can occur in such an application.

Invalid User Input

Some failures can show up even before the LLM is called. For example, a user may submit an empty message, upload an unacceptable file format, provide an extremely large document, or enter an invalid date.

In such cases, the application should be able to validate obvious requirements. There is no need to call an LLM to find out if a mandatory email address is missing or if an uploaded file exceeds the size limit. If we reject invalid requests early, we can save time and processing costs. We can also provide much clearer error messages to the users since the rules are fixed.

Network and Connection Failures

An application talks to an LLM provider over the internet. However, the connections can be interrupted. We can have DNS lookup failures. Responses might be lost.

Such type of failures are temporary. We can retry the request. But the application cannot retry indefinitely. We also need to consider whether it is safe to repeat the operation and whether there are any unforeseen side effects.

Timeouts

In this case, the LLM may take longer to respond than the application can wait. If there’s no timeout, the request can remain stuck while consuming resources.

The timeout value defines how long the application can wait. For example, an interactive chat application might have a timeout of 20 seconds. But an offline document-analysis job might allow several minutes. Also, the timeout should depend on the user experience. For example, a user waiting on a screen should have a shorter deadline than a background task that runs overnight.

Rate Limits

LLM providers have a limit to how many requests or tokens a specific account can use within a time period. If the application goes above this limit, the LLM provider may return a rate-limit error. This is represented by the HTTP status code 429.

Getting a rate limit response doesn’t mean that the service is broken. It means that our application is making more requests than what the provider can handle at a given point in time. To handle this, the application may need to delay the requests, reduce concurrency, put the work in a queue, or switch to another model that might have available capacity.

Provider and Server Failures

In certain situations, the LLM provider might be down due to a temporary internal problem. These failures are commonly represented by HTTP 5xx status codes.

In this case, a limited retry attempt could be a good idea because the failure might vanish quickly. However, if the failures continue, repeated retry calls only increase the load upon an already struggling service. is reasonable because the failure may disappear quickly. The application should stop retrying and switch to an alternative execution path if possible.

Authentication and Permission Failures

If the application provides an expired or invalid API key, it might also result in a 401 or 403 response from the LLM provider.

In this case, there is very little point to retrying. The application should record the failure, alert the relevant parties, and return an appropriate message to the user. It must take care not to expose API keys or other internal security details to the user.

Context-length Failures

Every language model has a limit on how much text it can process in one request. The prompt, conversation history, retrieved documents, tool definitions, and expected answer are all part of this context window and consume space.

If the application sends a lot of information to the LLM, the LLM may reject the request, or it may have too little space for the answer. An application designed with resiliency in mind usually counts tokens before sending the request. It can also remove old conversation messages, summarize earlier content, retrieve fewer documents, or divide a large job into smaller pieces to stay within the limit.

Malformed or Unexpected Output

Let’s say our application expects a result in the following format.

{
  “customer_name”: “Adam Smith”,
  “priority”: “high”
}

However, the model might reply something like: “The customer’s name appears to be Adam Smith, and this looks urgent.

Such an answer might make sense to a human being, but the application that expects a JSON response can’t process it. To get around this, the application should use a structured-output feature if the model provider supports it. It should also validate the response against a schema before using it.

Incorrect or Invented Information

The most dangerous LLM failures happen in the form of plausible answers that are completely wrong. This occurs due to model hallucination.

For example, a customer-support assistant might invent a refund policy that doesn’t exist. A research tool might generate a source that doesn’t exist. An inventory assistant might recommend a product that is not even there in the catalog.

Such problems can’t be detected by catching exceptions because no technical exception occurred. To handle these types of issues, the application needs additional checks, trusted data sources, or some form of human intervention to proceed further.

Tool Failures and Partial Completion

LLMs are also now used as agents that can make API calls, search databases, send messages, or perform transactions. This gives rise to a major problem. The specific action taken by the LLM might succeed even when the surrounding workflow fails.

For example, suppose the LLM assistant calls a payment service. The payment succeeds, but the network connection breaks before the application gets the confirmation. If the application retries the payment blindly, we might end up charging the customer twice.

This is the reason tool-based systems need idempotency, state tracking, and recovery mechanisms.

How to Classify Failures to Take Appropriate Action

So how do we deal with all these different types of errors in an LLM-powered application?

The first step is to classify them. There are main categories:

• A transient error is temporary and usually disappears if we attempt the operation again. For example, network problems, rate limits, and server failures.

• A permanent error continues until there is a change in the request or the system. Such errors include invalid credentials, unsupported file types, permission issues, and malformed requests.

• A semantic error occurs when the response is technically correct, but doesn’t solve the application’s requirements. Examples are invalid JSON, unsupported tool arguments, hallucinations, and so on.

Once we are able to classify an error, we can deal with it appropriately. For a transient error, we should retry after a short delay or use a fallback. For permanent errors, we need to report to the concerned parties who can correct the problem. For semantic errors, we need to validate, repair, or request human review.

Of course, not every error also fits neatly into a single category. For example, a context-length error is permanent for the current prompt. However, it can be solved by making the prompt shorter.

How To Retry a Request Correctly?

Let us look at some techniques to deal with failures. The first one to check out is retrying a request.

Retries are one of the simplest ways to improve an application’s resiliency. If an operation fails because of a brief network problem, chances are that it will work fine once the application waits for a while and tries again.

The main thing to guard against is creating an infinite retry loop. It can increase costs and make the outage worse. A safer approach to retrying limits the number of attempts. The time delay between each attempt should increase after each failure. This technique is known as exponential backoff. A simple approach could be to wait 1 second before the first retry, 2 seconds before the second, and 4 seconds before the third, and so on.

Applications also often add a small random amount of time to each delay. This is known as jitter. Without jitter, thousands of failed requests may all retry at the same time. This can cause another traffic spike and negate the benefit of retrying.

Retries should be used for timeouts, temporary network failures, 429 responses, and some specific 5xx responses. We shouldn’t use them for failures such as invalid credentials, prohibited requests, bad input, or other problems that will produce the same result every single time.

How to Manage Fallbacks and Graceful Degradation?

In an application, the fallback path is an alternative path. It is used when the normal preferred path fails due to some failure.

An application can use the fallback approach in the following manner:

• Attempt to use the primary model, which is usually of high quality.

• If that model is unavailable, switch to a smaller backup model.

• If no model is available, return a pre-defined fallback response or cached information.

• Lastly, if the request can’t be handled safely through any option, transfer it to a human for further action.

See the diagram below:

The best fallback approach depends on the task. For example, we might be okay to use a smaller model for summarizing an internal meeting. However, it might not be a good choice for interpreting a complicated legal document. Similarly, a cached response may be good enough for a general FAQ type scenario. But it won’t be suitable for something like the current account balance.

In other words, the fallback strategy should not violate the original requirements of the operation. We can’t just plug any available model into the flow. It’s also important to avoid dependency on a single point of failure. For example, if both the primary and backup models are accessed through the same provider, an outage can disable both. For real redundancy, we should have separation between the two pathways.

How Do Circuit Breakers Work in this Setup?

Consider a scenario where the LLM provider is unavailable. Every request is timing out. In such cases, it is not wise to keep sending more requests. They only waste resources and make users wait for failures.

A circuit breaker can temporarily stop calls to a failing service. It commonly has three states:

• In the closed state, we can send the requests at normal cadence.

• In the open state, requests get rejected or redirected immediately. This is because the service is considered unhealthy.

• In the half-open state, a small number of test requests are allowed to go through. They help determine whether the service has recovered to some extent.

If the test requests in the half-open state succeed, the circuit can close. But if they fail, the circuit goes back to the open state. In this way, circuit breakers help get rid of repeated failures from infecting the rest of the application. They also let us configure the appropriate fallback behaviour.

Rate Limiting, Queues, and Concurrency Control

An LLM-powered application can’t accept unlimited LLM work just because users can submit it from the UI.

For example, if 10K requests arrive at once, forwarding all of them to the LLM provider immediately might exhaust provider limits, database connections, memory, or the application’s budget.

Using rate limiting, we can control how frequently a user or client can submit requests. With concurrency control, we can limit how frequently a user or client might submit requests. Also, concurrency control puts a cap on how many requests the application can process simultaneously. A queue stores extra work until capacity becomes available.

Interactive requests where a user is waiting for the response should usually receive higher priority over background work. For example, a user waiting for a chat response should not be blocked because thousands of documents are being summarized in the background.

The application should also place limits on token usage, document size, number of retrieved passages, number of tool calls, and total workflow duration.

Conclusion

In this article, we have taken a detailed look at resiliency and error handling aspects when it comes to LLM-powered applications.

The key takeaway is that LLM-powered applications can have failures that are different from traditional applications. This is because of the inherent probabilistic approach of the LLMs. The important thing is to classify the type of failure so that proper action can be taken by the application.

Despite the differences in the types of errors, we can still borrow many of the techniques for error handling from traditional applications. For example, things like retries, fallbacks, circuit breakers, rate limiting, and queuing can help.

Getting a CX agent live is only the first step. The harder work starts once real customers are using it.

Our new guide looks at how teams at Lyft, Vodafone, and LATAM Airlines run CX agents in production. It covers how they evaluate responses, monitor failures, and use production conversations to improve the system over time.

You’ll learn how to:

• Keep prompt quality from slowing development

• Build observability into the agent from the start

• Catch failure modes before they affect more customers

• Turn customer conversations into useful signals for support, product, and operations

• Choose architectures that hold up in real CX workflows

Read the CX agents guide

This week’s system design refresher:

• How the JVM Actually Works (Youtube video)

• MCP vs RAG vs AI Agents

• 9 Distributed Systems Patterns You Should Know

• Virtualization vs. Containerization

• HTTP vs. HTTPS

How the JVM Actually Works

MCP vs RAG vs AI Agents

MCP is an open standard protocol. It connects AI models to external tools and data sources. These can be APIs, databases, or apps like Gmail, Slack, or GitHub. So instead of you writing the integration or doing the integration for each of these applications separately, MCP basically gives you a standard way to connect to these systems.

In RAG, the model pulls the fresh information when a query comes. And this is why the model does not have to make stuff up on its own, but instead it fetches the fresh information from external data sources, like docs, PDFs, and databases, to give the most up-to-date answer to the prompt.

An AI agent is kind of an AI system where the agent performs the task autonomously and takes the decisions. And then making sure everything is working fine, instead of a chatbot, which is really a request-response.

9 Distributed Systems Patterns You Should Know

• In replication, you make exact copies of your data and store them on different servers.

• In sharding, you break a large database horizontally, so different rows live on separate machines.

• In consistent hashing, you distribute data across different machines or servers using a virtual circular ring.

• PubSub is an asynchronous messaging pattern that decouples service creators from service consumers.

• In circuit breaker distributed pattern, it stops an application from repeatedly executing an operation that is likely to fail.

• In Retry with a backoff pattern, you handle temporary network glitches or brief service timeouts. It is a resiliency pattern.

• In the leader election pattern, you designate a single master node to manage actions and maintain cluster state. It’s what stops two nodes from thinking they’re in charge.

• In quorum read/write pattern, you ensure enough replicas agree on each read and write so that the set overlaps. It is a data consistency pattern that is used in distributed databases to guarantee up-to-date information.

• Saga is a design pattern that manages distributed transactions through a sequence of local transactions across multiple microservices. If one step fails, each earlier step is undone by its own compensating action.

Virtualization vs. Containerization

Before containers simplified deployment, virtualization changed how we used hardware. Both isolate workloads, but they do it differently.

• Virtualization (Hardware-level isolation): Each virtual machine runs a complete operating system, Windows, Fedora, or Ubuntu, with its own kernel, drivers, and libraries. The hypervisor (VMware ESXi, Hyper-V, KVM) sits directly on hardware and emulates physical machines for each guest OS.

  This makes VMs heavy but isolated. Need Windows and Linux on the same box? VMs handle it easily. Startup time for a typical VM is in minutes because you’re booting an entire operating system from scratch.

• Containerization (OS-level isolation): Containers share the host operating system’s kernel. No separate OS per container. Just isolated processes with their own filesystem and dependencies.

  The container engine (Docker, containerd, CRI-O, Podman) manages lifecycle, networking, and isolation, but it all runs on top of a single shared kernel. Lightweight and fast. Containers start in milliseconds because you’re not booting an OS, just launching a process.

  But here’s the catch: all containers on a host must be compatible with that host’s kernel. Can’t run Windows containers on a Linux host (without nested virtualization tricks).

Over to you: What’s your go-to setup: containers in VMs, bare metal containers, or something else?

HTTP vs. HTTPS

When you open a website, the difference between HTTP and HTTPS decides whether your data travels safely or in plain sight. Here’s what actually happens under the hood:

HTTP:

• Sends data in plain text, anyone on the network can intercept it.

• The client and server perform a simple TCP handshake: SYN, SYN-ACK, ACK

• Fast but completely insecure. Passwords, tokens, and forms can all be read in transit.

HTTPS (SSL/TLS):

• Step 1: TCP Handshake: Standard connection setup.

• Step 2: Certificate Check: Client says hello. Server responds with hello and its SSL/TLS certificate. That certificate contains the server’s public key and is signed by a trusted Certificate Authority.

  Your browser verifies this certificate is legitimate, not expired, and actually belongs to the domain you’re trying to reach. This proves you’re talking to the real server, not some attacker pretending to be it.

• Step 3: Key Exchange: Here’s where asymmetric encryption happens. The server has a public key and a private key. Client generates a session key, encrypts it with the server’s public key, and sends it over. Only the server can decrypt this with its private key.

  Both sides now have the same session key that nobody else could have intercepted. This becomes the symmetric encryption key for the rest of the session.

• Step 4: Data Transmission: Now every request and response gets encrypted with that session key using symmetric encryption.

Over to you: What’s your go-to tool for debugging TLS issues, openssl, curl -v, or something else?

On a purely technical level, both withdrawal requests completed successfully. There was no error. But the account balance is wrong. The remaining balance should have been $80.

On its own, both transactions were correct. Each read the correct balance and performed the correct calculations. And yet, there is a bug. The reason for this bug was the overlap between the two transactions.

You might think this type of overlap is a rare condition. However, this is not true. Overlapping transactions are more or less the normal condition in which databases operate. At any given point in time, multiple processes are trying to write something to a database, often on the same set of records. It just takes two of them to collide with each other inside a window of a few milliseconds for these types of bugs to show up.

So how do we handle such bugs?

This is what we are going to try to answer in this article. Here’s what we will cover:

• How does data get corrupted due to multiple transactions?

• 4 ways the data gets corrupted

• Different ways to handle data conflicts

  • Block everyone up front (Pessimistic Locking)

  • Gamble and check afterwards (Optimistic Locking)

• How databases stopped making readers and writers wait for each other?

• Picking the right level of data protection with isolation levels

• Strict without slow: the idea that made the safest setting usable

• Summary

How Data Gets Corrupted Due to Multiple Transactions?

Read more

The CIO of GlobalFoundries had a straightforward view of it: you don’t get AI agents until the data underneath them is real-time and governed.

So he rebuilt that layer first — one platform carrying data across fabs on three continents, with identity, permissions, and audit trails handled once instead of per project. The agents followed, across IT, procurement, and other business functions.

Join us on September 10 to hear how he did it, and get your questions answered live.

Save your seat

RAG, or Retrieval-Augmented Generation, is helping a lot of companies build chatbots specific to their requirements. However, the success of any such RAG system depends on the quality of the embedding model (the model that helps translate words into numbers) that is used by the RAG system.

Consider the example where a particular product’s documentation specifies that annual subscriptions can be refunded only within 30 days. A chatbot built using RAG is supposed to answer customer queries based on this support documentation. However, when a customer asks whether a subscription bought 45 days ago can be refunded, the chatbot confidently answers “yes”.

Why did the chatbot make a mess of this seemingly simple question?

The answer lies in the behaviour of the embedding model, which is kind of a translator for AI. This model controls the answer-searching process and is different from the language model that generates the actual answer. No matter how good the language model might be, it cannot give correct answers if the embedding model doesn’t do its job properly.

In this article, we’re going to look at how this embedding model works in an RAG setup and what makes it such a critical part of the system. Here’s what we will cover:

• Why a RAG system searches before it answers

• How embeddings make it possible to search by meaning

• Why related information is not always the right information

• Why a better language model cannot repair bad retrieval

• What makes an embedding model suitable for a RAG system

• How to compare embedding models without trusting benchmark scores blindly

• Choosing between commercial APIs and locally run models

• Why changing the embedding model later becomes expensive

• How Matryoshka embeddings offer more control over vector size

Why a RAG System Searches Before Answering

A generic language model’s knowledge depends on what it was taught during training. You can’t expect such a model to know about a company’s private documents, policies, or internal source code. It won’t even know about the company’s latest information that might have been published after the model was trained. Technically, it’s quite costly to retrain a language model every time there’s a change in the information.

This is where RAG helps. It separates the ability to generate language from finding relevant details from a pool of stored knowledge. While the language model handles the understanding and writing parts, the external knowledge collection contains the actual information that should be used for writing the answers.

RAG works in two different phases: indexing and retrieval.

During the indexing phase, the system prepares the documents:

• It collects documents from various sources (files, websites, databases, etc.).

• It extracts their text.

• It divides the text into smaller passages known as chunks.

• It sends each chunk into an embedding model.

• It stores the resulting vector with the original text and metadata.

The metadata may contain details such as the document title, publication date, language, version, and so on.

During the retrieval phase, the system follows the following steps:

• It sends the question through the same embedding model.

• It searches for document vectors that are close to the question vectors.

• It retrieves a small number of chunks. For example, the best 5 chunks or something like that.

• It filters and reranks those chunks, placing them inside the language model’s prompt.

• Finally, the language model writes an answer using the prompt that was provided.

The main takeaway from all this is that RAG doesn’t place the entire document collection into the model’s prompt. This is because doing so would exceed the model’s context limit and fill it with irrelevant information. It also increases costs and latency. The retrieval phase, led by the RAG’s embedding model, serves as the selection step, reducing thousands of passages to a small set for the language model to check.

How Embeddings Make it Possible to Search by Meaning

An embedding is basically just a list of numbers that represents a piece of text. A real embedding might contain numbers such as 384, 768, 1024, or several thousand numbers.

These individual numbers don’t have simple labels describing their meanings. You cannot look at a particular value and say that this means “subscription” while another means “refund”. The overall meaning is distributed across the complete vector. Think of the vector as the coordinates of a point in a mathematical space. An embedding model is trained to place text with related meaning near each other within this mathematical vector space.

This is how the system can connect a term like “yearly plan” with “annual subscription”. It can also connect complex phrases like “get my money back” with “receive a refund”. In contrast, a keyword search struggles when a question and the document use different words to explain the same concept. But an embedding model tries to compare the ideas present within the text rather than just the vocabulary. Common techniques behind this are cosine similarity, dot product, and Euclidean distance. In a nutshell, the goal of an embedding model is to produce a mathematical score that indicates how close two vectors are to each other.

The embedding model often returns the first k results. This is known as top-k retrieval. For example, if k is 5, the retrieval returns the top 5 highest-ranked chunks.

Embedding models also support asymmetric retrieval where the query and document have different forms. The query may be a short question. But the matching document is a longer explanatory passage. For example, the query could be something like: “Can an annual subscription be refunded after 6 weeks?” The answer passage is a statement like: “Annual subscriptions can be refunded within 30 days.”

An embedding model trained to compare similar sentences may not perform as well as a model that is trained to connect questions with the relevant passages. To summarize, the embedding model defines what the RAG system considers similar.

Why Related Information is not Always the Right Information

An embedding model is trained to identify semantic similarity. But a RAG system requires something a bit more strict. It needs to find passages that might contain the information needed to answer a specific question. The problem is that a passage can be related to the question without clearly answering it.

For example, a customer asking how long a refund takes might receive a passage explaining who qualifies for the refund. Both these pieces of information concern refunds, but only one talks about the actual processing time. See the diagram below that shows the concept of semantic search space.

Multiple failure modes can pop up in this situation:

• Similar Subject, Different Question: The query asks “How long will an approved refund take to arrive?” The retrieved passage says: “Purchases can be refunded within 30 days.” The passage is relevant to refunds but does not answer the timing question.

• Same Words, Different Entity: The query asks “how can the billing address be changed.” The retrieved passage explains how to change the account’s email address. Both talk about changing account details. But they refer to completely different fields.

• Negation: Consider the two passages: “Administrators can delete archived projects”, and “Administrators cannot delete archived projects”. Most of the words are identical. This means their embeddings may be close. But as you can see, they have opposite meanings.

• Versions and Dates: A knowledge base may contain an old policy and its replacement. The text may be almost identical except for a date, limit, or price. Embeddings cannot automatically know which document is authoritative. Metadata filters or version management rules may need to exclude outdated content explicitly.

• Numerical Identifiers: The two sentences “Annual subscriptions can be refunded within 30 days” and “Annual subscriptions can be refunded within 60 days” are semantically very similar. But there is a difference in one number, which determines the final answer.

• Domain-specific Meanings: General models can misunderstand special vocabulary. For example, “capture” has one meaning in an ordinary language and a specific meaning in payment processing. Similarly, the word “port” can refer to networking, hardware, or moving software between platforms. The best model for general web text may not be the best model for legal contracts, medical reports, financial documents, or source code.

• Multi-part questions: A customer may ask multi-part questions. For example, there could be a question like “Can I cancel the subscription, and how long will the refund take?” The answer for this requires at least two passages. One might explain eligibility for cancellation. Another might explain processing time. The model that retrieves only one subject may produce an incomplete answer.

Why a Better Language Model Cannot Repair Bad Retrieval

In an RAG system, the language model just sees the user’s question and the selected passages. It doesn’t see every document that might have been stored in the vector database.

For example, if the annual refund policy document is not retrieved, the language model won’t have any idea about the missing information. A more capable language model may recognize that the retrieved information doesn’t contain the answer. This is useful because it can at least choose not to respond with invalid information. However, this failed retrieval scenario also creates several possible outcomes:

• The model answers from its general training knowledge.

• It incorrectly applies a related passage.

• It might invent a plausible rule.

• It says that the available information is insufficient.

• It combines conflicting passages incorrectly.

A prompt such as “answer only from the supplied documents” can reduce unsupported answers. But it cannot make the correct document magically appear.

This is why testing and debugging are so critical during the development of an RAG system. Developers need to inspect retrieved chunks before changing prompts or swapping language models. It takes a lot more time to solve a retrieval problem in the generation phase.

What Makes an Embedding Model Suitable For a RAG System

The most important concern when choosing an embedding model should be retrieval performance. The model should be really good at connecting short questions with longer sentences that might contain the answers.

The following characteristics matter:

• Domain and Vocabulary: The model should understand the language used by the documents and users. Testing should include the use of abbreviations, internal product names, technical terms, and so on. If a company calls annual subscriptions “yearly plans” in the UI, but “annual contracts” in their legal documents, the model should be able to connect those expressions.

• Language Support: A multilingual model is critical when documents, queries, or both may use different languages. The model should be able to connect a query in one language with an answer found in another language.

• Embedding Dimensions: Embedding dimension is the number of values in each stored vector. Larger vectors preserve more information. But size does not guarantee better retrieval. Also, dimensions have a direct impact on raw storage costs.

• Maximum Input Length: A model with a limit of 8192 tokens can embed much longer text than one limited to 512 tokens. But it doesn’t mean each document should become one big chunk. A long chunk might discuss multiple things like refunds, cancellations, billing addresses, and so on. A focused question may match it less precisely than it would match a smaller passage specific to the relevant policy.

• Model and Query Speed: Indexing speed determines how much time it takes to embed the document collection. Query speed affects every user request. Some important metrics to consider are the number of chunks embedded per second, query-embedding latency, CPU or GPU requirements, memory use, and the cost of running the model. A model that gains a bit of retrieval quality at the cost of a substantial increase in latency may not be the right choice.

• Deployment Requirements: Embedding dimensions determine the size of each output vector. Model size determines how much memory and computation are needed to run the embedding model. These are key details required to plan for the model’s deployment.

Why Changing the Embedding Model Later Is Expensive

Each embedding model creates its own vector space, which is unique to the particular model. This means that the embeddings produced by Model A have no dependable relationship with the embeddings produced by Model B.

Consider an example where all document chunks are embedded using Model A. If we suddenly start to embed the incoming questions using Model B, the system will end up comparing vectors from incompatible spaces. It will be like an apples-to-oranges comparison. Even if both models output the same number of dimensions, the numbers can mean very different things.

This means that the entire corpus of documents needs to be re-embedded using Model B. In other words, migrating an embedding model needs a lot more than just running the new model. It involves the following steps:

• Generate new vectors for every chunk.

• Build a new vector index.

• Apply the correct metadata and access permissions.

• Keep new or modified documents synchronized during the migration.

• Test retrieval quality.

• Move production searches to the new index.

• Retain a rollback path if the new system performs poorly.

The costs of the migration process include embedding API charges or GPU time, index-building time, paying for temporary duplicate storage, data transfer costs, evaluation work, and operational risk.

The migration also changes rankings. For example, a model with a higher benchmark score may perform worse on the application’s specific domain terminology. Therefore, application-specific testing is extremely important to make sure that the answer quality has not gone down.

A few design choices can make it safer to implement future changes:

• The original chunks should remain the source of truth. A vector database should not be the only place where processed text exists.

• Each chunk should have a stable identifier and a content hash. The hash helps determine whether the text changed and needs a new embedding

• Embedding records should include:

  • Model name

  • Model version or revision

  • Embedding dimension

  • Query and passage format

  • Normalization method

  • Chunking version

  • Creation time

• A new model should receive a new index or vector field. Its embeddings should not be mixed with old embeddings.

To summarize, a safe migration resembles the classic blue-green deployment approach where the old index continues to serve production traffic even as a new index is built in parallel.

To be clear about things, changing the embedding model is just one of the operations that requires a rebuild. Changing the chunking strategy, parser, cleaning rules, prefixes, or stored dimensions may also require the documents to be rebuilt.

How Matryoshka Embeddings Offer More Control Over Vector Size

A basic embedding model produces a fixed-size vector. Its dimensions are meant to work together. We cannot delete most of these dimensions without severely reducing the quality of retrieval.

In contrast, a Matryoshka model is trained differently. During training, such a model is configured to produce useful representations at different prefix lengths, such as the first 256, 512, 1024, and full dimensions.

While the initial dimensions contain a useful coarse representation, the later dimensions pack more detailed information. For example, in a 1024-dimensional embedding, the first 256 dimensions can contain a useful small representation. Similarly, the first 512 dimensions contain a more detailed representation. At the end, all 1024 dimensions provide a complete representation.

There are three practical storage designs to support Matryoshka models:

• Storing the smaller vector: The system creates an embedding. It then keeps the first 256 dimensions and normalizes the shortened vector if needed. Only these dimensions are stored. This reduces the index size and overall search cost. However, the discarded dimensions are gone forever. If we want to migrate the system to a larger dimension, we need to generate the embeddings again.

• Storing the full vector with a smaller search representation: The system stores the complete vector in secondary storage. But it places a smaller prefix in the fast vector index. This supports future flexibility. We can build a larger index from the stored vectors without the need to rerun the model.

• Storing smaller and full vectors for two-stage retrieval: The system makes a search on the entire corpus using just 256-dimensional vectors (the reduced dimension). It then retrieves the full vectors for the best candidates and compares them precisely. This brings down the cost of the large initial search. But it retains full precision for the shortlisted candidates.

To be clear, Matryoshka embeddings don’t solve incompatibility between different models. They mainly help support different usable sizes within a model’s vector space. Switching to another embedding model still requires the entire re-embedding process we talked about earlier.

Conclusion

As we discussed, an RAG system can’t generate a reliably grounded answer unless it first retrieves the details required for that answer. This makes an embedding model the most important part of an RAG setup.

This doesn’t mean that other things are not important. Retrieval quality of the RAG system also depends on various factors such as:

• The existence of the correct documents

• How well the parsing process keeps their content intact

• How the documents are divided into chunks

• How versioning is managed

• Metadata filters and their working

• How a reranker improves the candidate list

Nevertheless, the embedding model remains central because it helps decide which information enters the language model’s context. It acts as the first major relevance decision in a typical RAG pipeline. The key lesson is to choose an embedding model that retrieves the correct evidence for the questions at an acceptable cost and speed.

Datadog’s free guide shows how to connect AI spend, infrastructure, and model performance into a single view, so you can correlate cost increases to the architecture changes that caused them before they show up on your cloud bill.

Learn how to:

• Break down AI costs by token, model, provider, and team

• Get alerted the instant inference volume spikes or API spend exceeds budget

• Correlate cost increases directly to architecture changes so root-cause analysis takes minutes

Get the guide

A model with 70 billion parameters can take up to 140 GB of space. A good graphics card has 24 GB. A very good one might have 48 GB.

As you can see, the gap is quite significant. Disk space is pretty cheap, but fast memory is scarce and costly. Moreover, the models have grown roughly 100-fold in a few years, while consumer graphics memory has roughly doubled. It’s not just a matter of tightening things up to make them fit.

So how do you run such a model?

The simplest option is to purchase the hardware that is capable of running the model. But it is costly, and doesn’t work well with consumer hardware.

The other option is to shrink the model. But we don’t want to do so at the expense of the model’s intelligence. This is where certain techniques can help us make the model smaller in principle without a dip in the quality of its output.

In this article, we will cover:

• What makes a language model intelligent?

• Three techniques to shrink the model

• How to shrink the model by packing fewer details?

• How to shrink the model by trimming unused pathways?

• How to shrink the model by mimicking behaviour?

• Does shrinking damage the model’s intelligence?

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

What Makes a Language Model Intelligent?

Large language models like ChatGPT are quite different from normal software programs. They don’t depend on typical if-else statements to decide what should happen next.

A large language model is essentially a very big pile of numbers called parameters or weights. These weights are the foundation of a language model’s intelligence. For reference, a model with 70 billion parameters means 70 billion numbers or weights. Each weight is normally stored in 16 bits, which is two bytes. Two bytes multiplied by 70 billion comes to 140 GB, which is basically considered the size of the model.

The weights are arranged into matrices. A single weight matrix is a grid, often something like 4096 by 4096. This comes to roughly 16.7 million numbers in a matrix. A 70 billion parameter model has hundreds of these matrices stacked across 80 or so layers.

Of course, weights in themselves aren’t the whole story behind a model’s capability. Multiple components work together to make a model intelligent. For example, the model needs an architecture like the Transformer to route data and perform attention. It also needs a context window that holds the prompt and everything generated during the conversation.

However, the architecture is made up of a few hundred lines of code. The prompt may be a few kilobytes. In contrast, the weights form the bulk of the language model. Without the proper weights, a language model cannot work as intended. The weights perform a bunch of tasks:

• They store patterns such as grammar, facts, and reasoning shortcuts.

• Once the training is finished, the weights freeze into an immutable network of numerical values.

• The weights decide how strongly one simulated neuron influences the next.

Running the model means pushing the input through these weight matrices until the next word comes out of the other end.

One thing to keep in mind over here is that no single weight means anything on its own. If you opened a model file and looked at weight number N, you might see something like 0.0293. On either side of this number may be other numbers such as -0.0117, 0.004, -0.0862. On their own, each of these numbers hardly makes sense. The ability of the model is hidden in the relationships between the various weights. Think of it like a photograph where every pixel comes together to show something recognizable. Even if we modify every pixel’s brightness slightly, we can still make out the things in the picture. This is because the information isn’t sitting in one single place.

Following on from all this information, it is quite easy to figure out that to shrink a model, we’ve to somehow deal with these weights. And this is exactly where the techniques come into the picture. However, a couple of points can help us make better sense of the techniques to shrink a model:

• Firstly, not all weights matter equally. Most of the weight values are close to zero and barely impact the final output. However, a small handful are large and produce a bigger impact.

• Second, we can only judge a model based on its behaviour, not its internals.

Let us now look at the techniques.

Build and scale a winning AI agent strategy (Sponsored)

Shipping agents to production is the easy part. Keeping them reliable, governable, and improving over time is where most enterprise AI programs stall.

How do top teams do it? They use an Agentic Operating Model (AOM), a step-by-step framework for aligning people, process, and technology so enterprise agents improve as they scale.

In LangChain’s latest guide, you’ll learn:

• Why AI agents don’t break like traditional software

• The engineering stack that covers the entire agent lifecycle

• Shifting from “build and deploy” to “operate and continuously improve”

Learn more

Three Techniques to Shrink the Model

The techniques to shrink a model revolve around using fewer bits to store a weight or using fewer weights. There are three main techniques:

• Storing Each Weight in Less Detail (Quantization): In this approach, we keep all the weights, but describe each one in a less precise manner. For example, two bytes become half a byte.

• Removing Irrelevant Weights (Pruning): This approach involves finding the weights that contribute nothing and deleting them.

• Building a Smaller Model to Mimic the Larger One (Knowledge Distillation): In this approach, we don’t touch the original model, but train a new, smaller model to behave in a similar way.

Going back to our photograph example, we can think of quantization as taking a picture with a cheaper camera that has a slightly lower resolution. Pruning is more like cutting away the blank edges of the photograph that might not be adding any value to the picture. Distillation can be thought of as paying a skilled painter to reproduce the picture at a quarter of the size.

The great part about all these techniques is that they can be stacked. For example, a model can be distilled by the lab that made it. It can be pruned by a research team. Lastly, it can be quantized by the user before it is loaded on a specific machine. In other words, stacking can make it possible to run a high-end large language model on normal consumer hardware.

Let us now look at each of these techniques in more detail.

Shrinking a Model by Packing Fewer Details

The first technique to shrink a model is to pack fewer details for every weight. This technique is known as quantization.

Let’s say a particular weight might be stored as 0.02934517. This takes a lot of space, but in a 70 billion parameter model, it just happens to be one single weight. Whether it is stored as 0.02934517 or 0.029 makes almost no difference to the model’s output. In other words, a lot of storage is spent on precision that might not even be important.

Quantization is a technique that takes away this precision.

The first bit of quantization happens even before the model is shipped. During training, model weights are normally stored as 32-bit (4 bytes) floating-point numbers. This is also known as the FP32 format. Since training involves making millions of tiny adjustments to each weight, high precision is needed. But when the model is distributed, the precision is usually brought down to 16-bit float format, which is also known as FP16 or BF16.

However, we can bring the precision down even further. To understand how, we need to first be clear about how a float value is actually built.

A floating-point number splits the bits into three parts: a sign, an exponent, and a mantissa. FP32 gives 1 bit to the sign, 8 bits to the exponent, and 23 bits to the mantissa. BF16, on the other hand, keeps all exponent bits and cuts the mantissa down to seven. This is basically the same range as FP32, but with less detail. For clarity, BF16 is slightly different from FP16, which gives the exponent only 5 bits and keeps more bits for the mantissa. BF16 has largely replaced FP16 in practice.

An integer has an even greater difference. An 8-bit integer is a whole number from -128 to 127. A 4-bit integer is a whole number from -8 to 7. There is no exponent and no scale. In other words, converting a float into an integer not only causes a loss of precision, but also removes each weight’s scale.

Let us now look at the complete process of quantization:

1 - Mapping Ranges

In the first step, we find out the minimum and maximum values of a data set and divide the total span into a fixed number of steps.

To be clear, “the data set” is not the entire model. It’s just a small group of neighbouring weights. We can call it a block, and it should ideally be pretty small.

For example, consider these eight weights: 0.021, -0.017, 0.004, -0.048, 0.011, 0.033, -0.006, 0.070. They span from -0.048 to 0.070. The largest value in either direction is 0.070. With 4 bits as our target precision, we can write whole numbers from -7 to 7. In other words, seven steps in each direction. Therefore, one step can be calculated as 0.070/7, which comes to 0.010.

2 - Rounding Values

Instead of keeping a long decimal, each original number is rounded to the closest available step. To do so, we divide each weight by the step size and round it to the nearest whole number.

The table below shows the new weights:

As you can see in the table, the right-hand column values are the ones that go into the model weights file eventually. Every entry here is a whole number between -7 and 7.

The original weights were floating-point numbers. Since a float carries its own scale, rounding off to engineers removes the precision as well as the scale of every weight. This scale has to be stored somewhere.

3 - Using a Scale Factor

We need to keep track of the scale factor so that the compressed numbers can roughly reconstruct the original values when the model needs to read them.

In our example, the scale factor is the step size (0.010). It is stored once for the entire block. To recover a weight, we can multiply the stored integer by the scale factor.

So, for example, the stored weight of 2 is multiplied by 0.010 to arrive at the value 0.020. Of course, it is still different from the original value of 0.021, but the error is much less now. In other words, the model is still quite unchanged. We still have the same weights (with some error), the same matrices, and the same layers. However, it takes a lot less storage.

Shrinking the Model by Removing Unused Weights

The second technique takes the opposite approach. Instead of trying to store each weight in less space, we delete some of them. This approach is known as pruning.

Pruning works because not all weights matter equally. Most of them have a value close to zero, such as 0.00004, -0.0011, and so on. These values are so small that they don’t impact the output in a meaningful manner. However, billions of such weights are produced during training. They are useful during the training phase, when the model is taking its original shape. You could think of them as driveways and service roads that are never used but still shown on a city map. Even if you remove them, they won’t have any impact on the people who are using this map to commute through the city.

The key decision with pruning is which weights to remove. The easiest approach uses the size. We sort the weights by how far they are from zero in either direction and delete the smallest ones. For example, if this approach helps prune 20% of the weights, it comes to around removing 14 billion weights from a 70 billion model.

However, there is a flaw in this approach. The influence of a weight also depends on the pathway on which it sits. A better method runs a few hundred sample texts through the model first. It then monitors how large the typical inputs are to each weight. Based on this, each weight is assigned a final score.

There is another factor that should be considered when it comes to pruning. It is related to the mechanism of removing weights. There are two main approaches:

• Setting to Zero: The first approach is to set the chosen weights to zero. This does minimal damage. But we end up with a matrix that is full of zeros in no particular pattern. The GPU still has to run the calculations with those entries.

• Removing Structural Pieces: The second way is to remove the structural pieces such as an entire neuron, an attention head, or even a complete layer. This helps shrink the matrix. But this is also a coarser approach. Removing one neuron might mean removing every weight attached to it. Some of these weights may be important, but they are also deleted. In other words, the overall damage to the model is greater in this approach.

See the diagram below that shows the two approaches:

Ultimately, pruning is rarely a complete solution. It is much better to use pruning in combination with some other techniques.

Shrinking the Model by Mimicking Behaviour

The third technique to shrink the model creates a new smaller model.

In this technique, we take the big model. It is called the teacher model. Next, we build a new model from scratch using the same architecture. However, it has fewer layers and smaller matrices. For example, a model with 7 billion weights instead of 70 billion. We call this new model the student model.

The student model starts its life with random weights that are meaningless. This model goes through the training process, but we don’t use the raw text from other sources on the internet. Instead, the student model is trained on the teacher’s behaviour. This involves feeding both the student and the teacher the same input and pushing the student model towards producing output closer to what the teacher model produced.

This technique is known as knowledge distillation.

This type of approach might appear counterintuitive. But it works because of how the model works internally.

When a model predicts the next word, it doesn’t just produce a single word. It produces a probability score for every word in its vocabulary. For example, given the sentence “the cat sat on the…”, the teacher model might say “mat 41%”, “floor 12%”, “couch 9%”, and so on. In an ordinary training approach, the student model only receives the correct answer. But during distillation, the student model receives the whole distribution of probable answers. In other words, the student model learns that “couch” was also a pretty sensible guess, but maybe the word “purple” was quite absurd.

See the diagram below:

Think of knowledge distillation like a teacher evaluating a paper, but instead of just putting ticks and crosses, the teacher writes in the margins to give a better idea to the student about the mistakes they might have made. The second approach helps the student learn faster

Of course, distillation involves training at scale. A huge teacher model runs over enormous amounts of data, which requires datacentre-level resources. For a developer, it involves downloading the smaller student model that can run on less demanding hardware.

Does Shrinking Damage the Model’s Intelligence?

Shrinking a model definitely reduces its overall intelligence, but usually only a tiny bit. Think of it like a trade-off between the model’s physical size and its mental sharpness.

Each of these techniques has an impact on the model’s intelligence:

• Quantization: It makes the weights less precise. The model can lose its ability to understand extreme nuance. It might forget highly specific facts, or its tone might sound slightly less natural. Many of the changes depend on how far you go on the quantization scale. For example, shifting from 32-bit to 8-bit causes almost no noticeable change in intelligence. But pushing down to 4-bit or even lower might have a huge impact.

• Pruning: It reduces the model’s ability to handle complex, multi-step logic. However, only trimming the idle pathways does not have a big negative impact on intelligence. Only when we aggressively prune the deeper pathways do we end up making the model dumb.

• Knowledge Distillation: The smaller student model may lack original problem-solving skills. It can perfectly mimic the big teacher model’s style, but a completely new logical puzzle might throw it off if the same wasn’t explicitly taught by the teacher model.

Conclusion

In this article, we’ve looked at the various techniques of shrinking a large language model in detail.

Shrinking a language model without any plan can cause a loss of intelligence. Therefore, various techniques follow different approaches to ensure that the model takes less space without losing its original capabilities drastically. Here are the key points to remember:

• Quantization involves storing each weight in fewer bits.

• Pruning deletes the weights and pathways that contribute the least.

• Distillation does not modify the original model, but involves a large teacher model training a smaller student model.

Ultimately, choosing a technique or combination of them depends on the overall goals that the language model needs to fulfill.

References:

• QLoRA: Efficient Finetuning of Quantized LLMs

• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

• A Simple and Effective Pruning Approach for Large Language Models

• Distilling the Knowledge in a Neural Network

Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part. You end up wasting time and tokens in the correction loops.

More MCPs, rules, and bigger context windows give agents access to information, but not understanding. The teams pulling ahead have a context layer to give agents exactly what they need for the task at hand.

Join us for a FREE webinar on Sep 2 to see:

• Where teams get stuck on the AI maturity curve and why common fixes fall short

• How a context layer solves for quality, efficiency, and cost

• Live demo: the same coding task with and without a context layer

If you want to maximize the value you get from AI agents, this one is worth your time.

Register now

When you type a follow-up question into an AI chat and press Enter, nothing happens for a second or two. Then the answer appears in a quick succession of words. It appears much faster than what the initial pause indicated.

This pause is not dead time. In a typical LLM, a single message passes through roughly a dozen distinct stages before a reply starts to appear. Two very different kinds of computing work happen in the background to make this possible. Some key points about this journey are as follows:

• The model never receives the message as it was typed.

• It has no memory of the conversation. The history on screen is rebuilt from scratch every turn.

• It shares a machine with strangers, and the group it lands in can affect the reply.

• Once a word has been sent, the model cannot take it back.

In this article, we are going to look at this entire journey in detail. Here’s what we will cover:

• How is the input to the model assembled?

• Why is every input message to the model independent?

• Performing safety checks on the input

• How does the model understand the words?

• How is the model shared across multiple conversations?

• Prefill and decode steps

• Caching the existing calculations

• Streaming and guardrails

• How does the model run various tools?

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

How the Input to the Model is Assembled?

The first key point to understand is that the sentence typed into the box is not the exact thing that reaches the model. What reaches the model is a document assembled around that sentence before the request is sent.

This document contains several ingredients, which are as follows:

• A system prompt, which is a block of instructions written by the LLM provider. These instructions tell the model how to behave during the conversation.

• Definitions of any available tools, describing what each one does and what inputs it accepts.

• Anything stored as memory from earlier sessions.

• Documents pulled from a knowledge base in cases where retrieval is the key.

• The full conversation so far.

• Finally, the new message.

The process of deciding what goes into this document in what order, and what gets left out, is a discipline in its own right known as context engineering. It is not a simple matter of filling a container. Models have a finite attention budget, and every token added draws it down.

Accuracy degrades as input grows longer, even on tasks that are quite simple. The decline is gradual. A longer prompt does not break anything outright, but the underlying precision falls away.

The approach used towards this discipline of context engineering leads to scenarios where two products built on the same underlying model, given word-for-word the same question, return different answers. The model might be identical, but the document wrapped around the question is not.

Different providers also differ in when they gather the material for the document. Some retrieve everything up front. Others hand the model lightweight references, file paths, or stored queries, and let it pull in what it needs while working. The first is faster, and the second wastes fewer tokens on material that might be irrelevant.

Why is Every Input Message to the Model Independent?

The models are stateless by design, meaning they retain nothing between messages. Each request arrives with no recollection of what came before. In other words, the conversation that we see on screen is reconstructed and resent in full every single time.

The maths related to this gets quite uncomfortable. Consider a product with a thousand-token system prompt, where each message and each reply runs about a hundred tokens.

• Turn one processes about 1,100 tokens.

• Turn two processes about 1,300.

• By turn three, we have about 1,500.

• By turn twenty, we might have closer to 4,900.

Though output tokens cost more than input tokens, input volume compounds on every turn while output stays roughly constant. This is why input usually dominates the total spend in any conversational product despite being the cheaper of the two.

The naive approach is to resend everything, but it works well only for short exchanges. Once the conversation outgrows the context window, it stops being viable. There are a few refinements that can help:

• The simplest one drops the oldest turns. This doesn’t cost much in latency and loses whatever was dropped.

• A more careful version summarises the conversation and restarts with the summary. This helps preserve decisions and open questions while discarding things like raw tool output that nobody needs to see twice.

• The third one stores material outside the context window entirely and retrieves it when it becomes relevant.

Long chats get slower and more expensive because every turn has to reprocess everything that came before it. The assistant eventually loses details because some details get trimmed or condensed. Also, editing an earlier message rewrites what the model treats as having happened.

Performing Safety Checks on the Input

Before the generation of the answer starts, the assembled document passes through a separate, smaller model trained to judge whether the request should proceed. Think of it like a safety layer.

The key point to note here is the separation in the design. This safety layer is a distinct system from the assistant, which means that it can be retrained, tuned, and monitored on its own schedule without any impact on the main model. It can also do more than permit or block. It can route a request elsewhere, log it, or escalate it for review.

This separation costs time, which is evident in the pause that precedes the answer. Published figures from one production system put an earlier generation of these classifiers at roughly a 24 percent increase in compute, alongside a 0.38 percentage point rise in refusals of harmless requests. Both numbers were high enough to limit how widely the approach could be deployed.

The replacement uses a cascade, where a very cheap validation is used to screen all traffic. Only flagged conversations are sent to the expensive classifier. This brings the overhead down to around one percent and refusals of harmless queries to 0.05 percent.

How the Model Understands the Words?

In the next step, the document gets converted into the units the model works with. These units are known as tokens, which are chunks of text that are generally smaller than a word but larger than a letter. Common words are often represented as a single piece, while rarer ones break into fragments.

Most modern systems build these chunks starting from raw bytes rather than characters, which guarantees that text in any writing system can be represented. As a working figure, one token averages around three-quarters of an English word.

There are two consequences due to this:

• The first is that cost and capacity vary by language. Research presented at a major machine learning conference measured the same text across translations and found token counts differing by as much as fifteen times. This is not just a matter of billing. Higher token counts also mean higher cost, slower processing, and less content fitting inside the same context window. Therefore, speakers of some languages get materially less usable space for an identical document.

• The second is that character-level questions can get awkward. Counting how many times a specific letter appears in a word means looking inside the chunks.

How the Model is Shared Across Multiple Conversations

No model sits idle waiting for a request to arrive. In a typical setup, the token sequence joins a queue, and then a batch of other people’s requests running on the same hardware.

Why the need for this batching?

This is because while modern accelerators have enormous computing capacity, a great deal of their memory bandwidth is spent on loading model parameters rather than performing useful work on any single request. Loading those parameters once and applying them across many requests simultaneously is what makes serving affordable.

The naive version of batching collects a group of requests, runs them together, and waits for the entire group to finish before starting the next. This works when every response is a similar length. However, chat responses are not of similar length. Some might finish in ten words while others can run for a thousand. Therefore, the hardware might be partly idle waiting on the longest one.

A better approach schedules a request at the level of individual generation steps. As soon as one response finishes, a new request takes its slot rather than waiting for the group to clear. Benchmarks have demonstrated throughput improvements of up to 23 times over the naive method. Also, this approach improved median response times as well.

There is another strange consequence of sharing resources to generate answers. Identical requests can return different answers even with randomness switched off. This happens because the numerical operations involved are sensitive to the number of requests being processed together. In other words, sending the same prompt a thousand times to a large model might end up with 80 distinct completions.

Prefill And Decode

This is the point where the pause and the typing separate.

Producing a reply happens in two phases with opposite characteristics

• The first reads the entire assembled document in one pass. Every input token is processed alongside every other. Therefore, this work runs in parallel and pushes the hardware on raw computation. This phase is the pause.

• The second phase produces output one token at a time. Each token depends on the one before it, so none of it can be parallelised. The limiting factor here is memory speed rather than computation, since every step reads back everything that has been computed so far. This phase is the steady typing.

The first phase scales with input length. This means a conversation twenty turns deep takes measurably longer to begin than the same question asked in a fresh window. The second phase runs at roughly the same rate regardless. In other words, the pause time increases as a chat continues, but the typing speed barely changes.

These two halves are measured separately:

• Time to first token covers everything up to the first visible output, including the queue wait time.

• Time per output token covers the gap between each token after that.

Total response time is calculated as approximately the first plus the second multiplied by the length of the reply.

A single very long input can delay generation for every other request in the batch. Splitting that long input into chunks and interleaving them with ongoing generation keeps everyone else’s output flowing. The trade-off is a slightly longer wait for the request that was split.

Caching the Existing Calculations

Why is the second phase quick at all, given that every token has to account for everything before it?

This is because that work is not repeated. The calculations produced while reading the input are stored and reused at each subsequent step.

This stored state is large. For a model in the 70-billion-parameter range holding an 8000-token conversation, it can run to a few gigabytes per request. A bunch of concurrent conversations can exceed the memory of an entire high-end accelerator before the model weights are even accounted for. In other words, the number of users a machine can serve is usually limited by this conversation state rather than by the model.

Early serving systems reserved one contiguous block of memory sized for the longest response a request might produce. A widely cited systems paper measured that cost and found sixty to eighty percent of the memory going unused. The fix involved a technique that operating systems have used for decades. This technique is to split the storage into small fixed-size blocks provided on demand. In this case, the wastage dropped below 4% and throughput improved by 2-4 times.

The same idea extends across turns. Since every message resends the same opening block, the computed prefix can be kept stored and reused instead of recalculated. Providers price this explicitly, with cached portions of a prompt often costing around a tenth of the normal input rate. Cache entries typically expire after a few minutes unless they are refreshed due to continued use.

This is why prompt structure follows a rule that stable content belongs at the top and changing content at the bottom. This is because a prefix that changes on every request cannot be reused effectively.

Streaming And Guardrails

There are two things that happen during the response phase of the conversation.

Text appears word by word because it is sent as it is produced instead of being held until finished. This improves the perceived speed. Ordinary reading runs at something like six tokens per second, and production systems generate comfortably faster than that. Streaming lets reading start almost immediately instead of after a blank wait.

The second part is around safety checks. Many products run a safety check on output as well as input. A check of that kind has to read the finished response before it can judge it. By that point, the response is already on screen, and it is not possible to retract a word that has been displayed. Holding the response back until the check completes removes the benefit of streaming entirely.

There is no settled answer to this yet. Some systems use guards designed to evaluate output as it streams, token by token. Others read the model’s internal state during generation rather than waiting for finished text, which is cheap enough to apply.

How Tools Are Run?

Everything so far describes a straight line from pressing Enter to the response. Once tools are involved, this straight line turns into a loop.

A model does not search the web, read a file, or query a database. It produces text requesting that one of those things happen, using the tool definitions included in the assembled document. The surrounding application recognises the request, carries it out, and puts the result back into the context.

The result is new input, which means the entire sequence runs again from the beginning. It is reassembled, checked, tokenized, queued, and run through both generation phases again. For example, a reply that involved three web searches made several complete round trips through everything described here. This accounts for why those replies take noticeably longer than ordinary ones.

The cost compounds sharply. Across a session of twenty calls with growing context, the earliest messages are paid for twenty times over. A two-thousand-token instruction block sent across two hundred calls accounts for four hundred thousand input tokens on its own, before any of the actual work.

Conclusion

The two seconds of delay now have a clear picture that we can understand.

• A fraction went to network travel and authentication.

• A larger fraction went to assembling the document and gathering whatever memory and reference material the product uses.

• A small fraction went to the input safety check, which cascaded designs now keep near one percent of compute.

• Tokenization was negligible.

• Some unknown portion of the delay went to queueing, depending entirely on concurrency.

• In a long conversation, the largest fraction goes to reading the input, which is the reason for the initial pause growing as the chat continues.

• Everything after that was the second phase, running at a rate that had barely changed since the first message.

The model receives a constructed document in which the typed message is the smallest component. It has no memory, so the conversation is rebuilt from scratch. This is why long chats slow down, cost more, and lose details.

References:

• Effective context engineering for AI agents

• How continuous batching enables 23x throughput in LLM inference

• Language Model Tokenizers Introduce Unfairness Between Languages

Depending on the use case, the application may resize the image into different sizes. It might run the image through a content check process. It can also push the resized images to a content delivery network for faster access. Lastly, it will also update the user record with the latest metadata. Now, imagine if all of these operations take place during the same picture upload request path. As all of these functionalities are carried out, the user sees a spinning button for several seconds, wondering whether the photo has been uploaded successfully. Needless to say, it would be a pretty poor user experience.

If we move all of that extra work outside the request path, the upload can more or less respond immediately as soon as the image file is stored in an object storage. The image still gets resized, scanned, and distributed, just not in the same flow. This is known as background work, and here are some examples of why it is needed:

• A user action may have triggered it. For example, someone signs up, and a welcome email goes out.

• The clock may have triggered it. For example, nightly reports, monthly invoices, hourly cache refreshes.

• Another system could have triggered it. For example, a webhook arrives, or a file is stored in object storage.

• Perhaps the sheer volume of work made it worthwhile. For example, some work is naturally cheaper or safer done a thousand items at a time.

Most of the time, teams start with a single scheduled script on a single machine. It can handle a great amount of background work. However, as the system gets bigger and more complex, the amount of background work requires different strategies. In this article, we will look at various such strategies to perform background work in detail.

Why Normal Requests Don’t Work Everywhere

Read more

Every agent already runs a loop. Loop engineering adds a loop around the agent itself, enabling it to evaluate its output, try again when the work falls short, and refine its instructions when the same mistakes recur. Today, you perform that role: reviewing the work, diagnosing what went wrong, and prompting the agent again. This article shows how to automate that process with a working example, while exploring where human judgment still belongs.

Read the post

A 70-billion-parameter model requires reading roughly 140 GBs of weights out of the GPU memory. On a modern data center GPU, this transfer can take tens of milliseconds. The actual calculation applied to these weights takes a fraction of that time. This means that the processor’s math units are unused for most of the time taken by the token generation step.

Speculative decoding is a technique that converts this unused capacity into output. A second, much smaller model produces several candidate tokens in advance. The large model evaluates all of them in a single forward pass instead of one pass per token, resulting in 2-3 times faster generation. To make things better, the text produced remains statistically identical to the output of the large model running alone.

In this article, we will look at how speculative decoding works. Here’s what we will cover:

• Why token generation runs one step at a time

• What a GPU spends its time on during generation

• How several candidate tokens are evaluated in a single pass

• The accept and reject loop, and what happens when a candidate is wrong

• Why output quality is preserved exactly

• Acceptance rate, and why it varies by workload

• The four places a draft can come from

• When speculative decoding stops helping

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

Autoregressive Decoding

Text generation works one token at a time.

The model reads everything produced so far, computes a probability distribution over its vocabulary, selects the next token, appends that token to the input, and repeats the cycle. Each cycle is called a forward pass, and every forward pass runs the input through all layers of the model.

For example, token 50 depends on token 49 being present in the input, and token 49 depends on token 48, and so on. Computing them simultaneously would break the dependency chain that makes the output coherent.

The implication is that a 500-token response requires 500 sequential forward passes, each one completing before the next begins. Since the duration of a single pass depends on the size of the model, the total generation time equals the number of output tokens multiplied by the time per forward pass.

This explains why the response speed stays roughly steady whether the answer is a short factual reply or a long block of code, because the per-token cost stays the same either way. It also explains why a larger model produces text more slowly on identical hardware.

Modern inference systems use a KV cache, which stores the attention state for tokens already processed so that each new pass only computes attention for the newest position. This cuts the work done inside each pass to a large extent, though the requirement for one pass per token still remains.

Memory Bandwidth

Since the number of passes is fixed by how much text we want, it leaves the second half of the equation. What does a single forward pass actually spend its time doing?

To put it simply, a forward pass spends most of its duration moving data rather than performing arithmetic calculations.

Model weights live in the GPU memory, usually called VRAM. To compute anything with those weights, the GPU has to transfer them into the compute units where the multiplication happens. For a 70-billion-parameter model stored at 16-bit precision, this transfer amounts to roughly 140 GBs for every single token.

The arithmetic performed on those 140 GBs is quite small by comparison. One token means one narrow vector flowing through each weight matrix. The GPU loads an enormous matrix out of memory, multiplies it against that vector, discards it, and loads the next one.

The consequence is that during prompt processing, compute utilization is around 90 to 95 percent. However, during token generation, it falls to somewhere between 20 and 40 percent. The math units are unused for most of every step while the memory bus runs near capacity.

The difference is driven by how much work each weight read supports:

• Prompt processing reads the weights once and applies them to thousands of input tokens simultaneously.

• Token generation reads the same weights and applies them to exactly one token.

This is capacity that has already been paid for, but underutilized.

But why does this matter practically?

A GPU with higher memory bandwidth improves generation speed more than one with more raw compute.

However, spare capacity only helps if there is useful work to put into it. The question is whether a single forward pass can produce more than one token’s worth of output.

Parallel Verification

A single forward pass can evaluate many positions at once.

Transformers process an entire sequence in parallel. When we feed in a sequence of tokens, the model computes a next-token prediction at every position in that sequence during the same pass. For example, a five-token input produces five predictions.

These predictions stay valid because of causal masking. Inside the attention mechanism, position 5 can access positions 1 through 5 while positions 6 and beyond are masked out, and position 3 can access only positions 1 through 3. Each position is therefore conditioned on exactly the tokens preceding it, identical to the conditioning it would have received had we generated the sequence one step at a time.

This is the property that makes prompt processing fast. A 2,000-token prompt runs through the model in one pass rather than 2,000, because all 2,000 positions are computed together.

When applied to verification, the consequence is direct. For example, if we append four candidate tokens to the context and run one forward pass, we receive the model’s own prediction at each of those four positions.

One thing to understand here is that verification and generation are basically the same operation. The target model performs identical work at each position. The cost savings comes from performing that work across several positions in a single pass instead of across one position in each of several passes.

Draft and Verify

The complete loop combines a fast source of candidate tokens with the batched evaluation described above.

This setup uses two models:

• The large model we want output from is called the target model

• Running alongside it is a much smaller draft model with 10 to 20 times fewer parameters. It is usually drawn from the same family and uses the same tokenizer.

Each round has three steps:

• The draft model produces K candidate tokens through its own serial loop. Those passes are sequential as well, though each one costs a small fraction of a target model pass.

• The candidates are appended to the context. The target model evaluates the extended sequence in one forward pass.

• Working left to right, each candidate is compared against the target model’s prediction at that position. The matching candidates are kept, and as soon as the first mismatch appears, the remaining candidates are discarded.

The mismatch point plays an important role in this. The verification pass already computed the target model’s prediction at that position, so that token gets used directly. We keep the matching prefix and receive one correct token at no additional cost.

This property places a bound on the downside.

In the worst case, all four candidates might fail to match, but we would still have the one token the target model produced at the first position, which is exactly what plain decoding would have delivered from one forward pass. The wasted effort amounts to just the draft model’s compute and some extra effort in the verification pass. Both are drawn from otherwise available capacity. In a typical case where two of four candidates match, we get to keep two plus the free token, giving three tokens from one target model pass.

Draft length K is a tunable value, which is commonly set between 3 and 5. Larger values raise the ceiling on savings, since a fully accepted draft of eight saves more than a fully accepted draft of three. However, larger values also reduce the odds that later candidates survive, because the draft model conditions on its own unverified output as it moves forward. Past a certain point, the additional candidates get discarded often enough that the extra work outweighs the benefit.

The question at this point is whether the resulting text is still the text the target model would have produced.

Lossless Guarantee

Speculative decoding produces text with the same statistical properties as the target model running alone. This is enforced by means of the acceptance rule.

Under greedy decoding, where we always take the highest-probability token, the rule is quite direct. A candidate is kept when it matches the target model’s top choice at that position, and dropped when it fails to match.

However, sampling takes more care, since tokens get picked with some randomness. Both models produce a full set of probabilities across the vocabulary at each position. The rule compares the two sets:

• When the target model gave the candidate at least as much probability as the draft model did, the candidate is kept.

• When the target model gave it less, the candidate is kept part of the time. This is in proportion to how far apart the two numbers were.

• When a candidate is dropped, the replacement gets picked from an adjusted set of probabilities, with the draft model’s own scores subtracted out first.

This last step is critical. If we add up both paths, the candidates kept and the candidates replaced, the odds of any particular token appearing depend exactly on the target model’s own odds for it. This is regardless of what the draft model suggested.

There are two qualifications to this:

• Matching odds still allow different wording, since sampling stays random either way. Running the same prompt twice can give varied text in both setups.

• Computers store these numbers with limited precision, so rounding can flip the winner when two tokens sit almost exactly tied.

Acceptance Rate

The size of the speed increase in this approach is governed by the acceptance rate, which is largely a property of the workload.

Acceptance rate is the fraction of candidate tokens the target model keeps, and acceptance length is the average number confirmed per verification pass, including the free token at the end.

Different workload types produce different results:

• Structured and repetitive output produces high acceptance. For example, code generation, summarization, extraction, and retrieval-augmented answers reuse large amounts of text from the input, which makes the next token easy to predict from a small model.

• Open-ended output produces low acceptance. Creative writing and open conversation generate text with genuine variety, where a small model diverges from a large one far more often.

Sampling temperature contributes as well. Higher temperature flattens the probability distribution, which increases mismatches between the two models and pushes acceptance down. If the acceptance falls below roughly 50%, the additional work outweighs the savings.

For reference, DeepSeek reported acceptance rates between 80 and 90 percent for the second predicted token in production serving of DeepSeek-V3, which translated to roughly 1.8x generation throughput.

The practical implication is that two teams can deploy the same configuration on the same hardware and get different results, because their users are asking different questions. Ultimately, acceptance depends on the quality of the candidates, which brings us to where candidates come from.

Candidate or Draft Sources

The key question while choosing candidate or draft sources is where to obtain fast predictions cheaply. There are four answers in common use:

• A separate small model: The original approach pairs the target with a smaller sibling, so a 1B model drafting for a 13B target, or a 3B to 8B model drafting for a 70B target. Same family and identical tokenizer are requirements in this approach. The cost is a second checkpoint to deploy and version, plus VRAM that comes out of the KV cache budget, which reduces how many concurrent requests the server can hold.

• Extra prediction heads on the target model: Lightweight output heads predict tokens two or three positions ahead using the target model’s internal representations. DeepSeek-V3 trained these during pretraining to improve model quality, then reused them at inference as the draft source. The cost is training, which puts this option out of reach unless we control the model.

• A cheaper version of the same model: The draft runs the same weights under a reduced compute budget through quantization, layer skipping, or a compressed KV cache. For example, QuantSpec uses 4-bit weights and a 4-bit KV cache for drafting while verification runs at higher precision, reporting speedups above 1.78x with acceptance above 90 percent. The cost is implementation complexity, since draft and target share hardware and cache structures.

• A search over existing text: This approach scans the prompt and previous output for a recent matching sequence, then proposes whatever followed it last time. Memory cost is zero, and a single model is involved. It contributes only when output repeats input, where it reaches 2x to 4x on tasks like document editing and summarization.

Selecting an option depends on the deployment. Also, tokenizer compatibility constrains pairing more tightly than model quality does. A stronger small model with a different vocabulary is unusable as a draft source without additional machinery. All four options depend on spare compute being available. This condition holds under some serving loads better than others.

Concurrency Limits

The increase in speed also depends on the operating regime rather than on the technique alone. The gains shrink as server load rises.

Speculative decoding spends compute capacity that would otherwise go unassigned. When a server handles a single request, this capacity is genuinely available. As concurrent requests accumulate, the same weight read operation serves many requests at once, and the compute units approach saturation. Verification work has to compete with real requests.

One systematic evaluation reported up to 1.96x on a 70B model at batch size 1, declining to 1.21x at batch size 128. Under higher concurrency, the technique can fall below baseline throughput, at which point enabling it costs more than the benefits.

Serving systems try to handle this in different ways. For example, vLLM exposes a flag that disables speculation above a configurable batch size. It supports dynamic adjustment where draft length shrinks as concurrency rises and reaches zero under heavy load. The control signals routine operational tuning rather than an edge case.

Another boundary is that the time to first token stays roughly the same, since speculative decoding applies to generation rather than prompt processing. Therefore, workloads with long prompts and short outputs have relatively little to gain.

DeepSeek documented the tradeoff, describing multi-token prediction as slightly reducing throughput while significantly improving end-to-end generation latency.

Conclusion

Speculative decoding rearranges when the processing happens rather than reducing how much the target model performs. Here are some key points we have understood:

• Token generation is slow because every token requires reading the full set of model weights out of memory, while the arithmetic applied to those weights is comparatively small.

• A transformer computes a prediction at every position in one pass, which makes evaluating several candidate tokens cost about the same as evaluating one.

• A rejected draft truncates rather than wastes, since the verification pass supplies a correct token at the mismatch position regardless.

• Output quality is preserved by the acceptance rule itself, so it holds without tuning.

• The size of the gain depends on how predictable the output is and how much spare compute the server has available.

• The variants differ in where predictions come from and what that source costs.

Securing AI usually requires hardcoding permissions into application code for every MCP server, or using static API tokens for all-or-nothing access (and hoping your LLM doesn’t drift from intended actions).

Teleport eliminates these problems with zero-code MCP integration that applies the same zero trust security principles you use for human engineers:

• Least privilege access control that denies new tools by default

• Just-in-time (JIT) access requests for high-risk tools

• Logs for every action – with full audit and identity context

• Zero trust agent access to MCP servers, databases, and Kubernetes clusters

No need to write authorization code, rewrite MCP servers, or limit agent work.

Learn More

When an AI model handles a difficult question, it produces three separate pieces of text:

• The first is the answer displayed on screen.

• The second is a shorter block, usually labeled thinking or reasoning, that appears while the answer is being assembled.

• The third is the model’s full reasoning process, which is never displayed.

The second piece of text is a summary of the third. It is generated separately and shown in place of the original reasoning process. The full reasoning runs longer. It also contains material that the summary leaves out. Most major providers withhold it. However, in place of the complete process, an encrypted version is shared with the client during the conversation.

In August 2026, a team at MATS Research, the ELLIS Institute Tübingen, and the Max Planck Institute for Intelligent Systems wanted to test whether the encrypted reasoning blocks that Anthropic, OpenAI, and Google hand back to clients actually keep that reasoning private. They showed that the blocks can be replayed into a cheaper model in the same family, which will then print the hidden reasoning in plaintext. In other words, the AI model’s thoughts are stolen, exposing information that should ideally be hidden.

In this article, we will cover what the researchers found out:

• Reasoning traces, and how they differ from answers and summaries

• Why providers withhold reasoning

• The storage problem that creates, and the two ways to solve it

• What the encrypted block contains, and what it authenticates

• The three forms of compatibility that follow

• The extraction method and its verification

• The four attack vectors

• Findings from a scan of published session logs

• The proposed fixes, and the limit that remains

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

Reasoning Traces

Modern frontier models first generate an extended internal sequence of text before producing a visible answer. For example, if we ask a model to solve a hard mathematics question, the model may generate two thousand words of exploration, dead ends, and corrections, which ultimately helps it write a clean two hundred-word answer. This longer sequence of exploration is the reasoning trace, which is also known as the chain of thought.

The trace holds intermediate hypotheses that the model may have tried and abandoned. It contains the raw output of any tools that were called, the user’s data as the model processed it, and whatever contextual secrets were present in the session. Unsurprisingly, this trace is much denser and more revealing than the final polished output.

For example, if we ask a coding agent to remove hardcoded credentials from a code repository, the agent has to read those credentials to do the work. In other words, the credentials pass through the reasoning trace before any answer is produced.

Concealment Rationale

Why do the model providers hide these reasoning traces?

There are two separate motivations:

• The first is commercial. A competitor can collect a large number of traces from a strong model to create training material for building a cheaper imitation of the strong model. This is because the final answer just provides the endpoint of a computation, whereas a trace provides the methodology behind the answer.

• The second is safety-related. A model sometimes has to generate reasoning about a harmful topic to refuse. The filtering that turns a completed trace into a safe visible answer runs after the trace already exists. Publishing the trace skips that filter and lets that information be seen by the user.

State Management

Withholding a trace from reaching the user does not mean that the trace is not needed. In a multi-turn conversation, the reasoning from an earlier turn has to be available on the next turn. For example, a request to a model API carries no memory of what came before. The continuity we see in a chat window is reconstructed by the client resending the full history with every message. However, the server is stateless, which means it stores nothing between requests.

This leaves us with two options:

• The first option keeps the state on the server. The provider stores the trace in its own database, returns a meaningless identifier to the client, and looks it up when the next message arrives. This is straightforward but expensive because it means storing state for every conversation from every user of a global service.

• The second option encrypts the trace and returns it to the client, which stores the block and sends it back with each subsequent request. In this case, the provider stores nothing.

Providers like OpenAI, Anthropic, and Google chose the second option. Confidentiality keeps competitors from reading the trace, integrity prevents an altered trace from being fed back, and statelessness removes the storage cost. As you can see, two of those are security goals, and the third is a cost goal.

Envelope Structure

So what is actually inside the block that gets sent back to the client?

It consists of a long string of base64 text, which is a way of writing binary data using ordinary letters and digits so that it survives inside JSON. Once decoded, it is an AEAD envelope. AEAD stands for Authenticated Encryption with Associated Data, and it accomplishes two tasks at once: hiding the content and proving that the content was not altered. The associated data portion holds extra fields that stay readable while remaining tamper-protected.

The envelope carries a header. Depending on the provider, it can include the model name, block type, version, and key identifier. Alongside that is a nonce, which is a random value used once per encryption so that identical content produces different-looking output each time, an authentication tag, and the ciphertext. The field carrying all of this is called signature at Anthropic, encrypted_content at OpenAI, and thinkingSignature at Google.

Authentication here proves that the contents came from the provider and were not modified afterward. The model name and version are covered by that proof. However, the account that generated the block and the conversation it belonged to are absent from the authenticated fields entirely. To make this clear, since no provider has published a description of the scheme, the researchers inferred this mainly from observable behaviour. The evidence points toward a single global key used across an entire ecosystem.

Trace Compatibility

If the authenticated fields provide no information about the origin, a valid block stays valid everywhere. The researchers have described three forms of this, each more permissive than the last:

• Cross-session compatibility means a block can be replayed out of order, and blocks from earlier sessions work in new ones. This supports editing conversation history and trimming long sessions to fit a context window.

• Cross-user compatibility means a block produced in one account is accepted when submitted from another.

• Cross-model compatibility means a block from one model is accepted by a different one, which supports switching models mid-conversation and automatic rerouting.

The researchers tested every source and target combination available in July 2026. Some of the findings were as follows:

• Claude accepted almost every combination, the exception being Fable 5, whose blocks were accepted only by Fable 5.

• GPT was organised by generation. The GPT-5.6 series accepted blocks from all earlier generations, while older models accepted only their own.

• Gemini accepted every combination across every generation.

Extraction Method

Cross-model compatibility carries a lot of importance because protection inside a model family is uneven.

Flagship models such as Claude Opus 4.8 and GPT-5.6 Sol receive anti-distillation training aimed at preventing disclosure of their own reasoning. It is present behind input and output filters that check for verbatim matches. However, smaller models in the same family, such as Claude Haiku 4.5 and GPT-5.6 Luna, are optimised for cost and speed. As a result, they receive far less of that training. In other words, cross-model compatibility means the smaller model accepts blocks produced by the larger one.

The information extraction method is a direct result of this gap in training. Here’s what can happen:

• A question is asked of the strong model, and the response returns a visible answer and an encrypted reasoning block.

• The answer is discarded, and the block is kept.

• Next, a fresh conversation is opened with a weaker model in the same family.

• The reasoning block is placed into it as prior context, and the request asks for the attached reasoning to be transcribed.

• The weaker model outputs the stronger model’s reasoning in plaintext.

In other words, the strong model was queried once, with an ordinary question, and was never asked to disclose anything. Therefore, its refusal training was never engaged, and its output filter received only a benign answer.

The researchers describe the weaker model as a fuzzy decoder, since it generates an approximation rather than performing exact decryption.

This results in a verification problem. Without access to the original trace, how do you establish that a reconstruction is faithful rather than a plausible invention? The researchers used billing records. API responses report the number of reasoning tokens consumed, and that figure is exact because the charge depends on it. Re-encoding the recovered text and counting its tokens produces a number that should match. Across 120 programming problems, the two counts tracked closely. It was roughly one to one for Claude.

The difficulty of this approach varied by provider. A single fixed prompt worked for Claude, while GPT required up to 50 candidate extractions per block and output chunked below roughly 50 tokens to avoid a rejection triggered by verbatim reproduction.

Attack Vectors

The research paper talks about four consequences of this gap. These are split based on who produced the original block.

Two of them use blocks the attacker generated:

• Distillation: It involves training a copycat model on a target’s visible answers. Traces make it stronger, because a trace supplies the problem decomposition and the intermediate steps. For reference, earlier work using approximate traces raised the MATH500 accuracy of a fine-tuned model from 68.4 percent to 76.0 percent over answer-only training. Decoding 10,000 traces at Claude Haiku 4.5 pricing costs roughly $720, and where blocks come from public datasets, the frontier model is never queried.

• Jailbreaking: Models are trained to withhold harmful content from visible output. However, they are largely not trained to avoid generating reasoning about harmful topics, since constraining trace content is believed to degrade the usefulness of traces for safety monitoring. In one demonstration, a prompt drew out extended reasoning about vehicle theft while the visible answer stayed within a responsible write-up aimed at manufacturers. The recovered trace contained specific vulnerable makes and model years.

The other two vectors use blocks produced by other people.

Developers publish agent session logs routinely, for reproducibility or by committing them accidentally, and they sanitise the visible text before publishing. However, they cannot sanitise the encrypted blocks, because they cannot read them either.

The researchers collected 6,708 public agent trajectories from GitHub and Hugging Face and decoded 315,320 reasoning blocks, filtering the results through an automated pipeline to remove placeholders and benchmark fixtures. Here are some findings:

• From genuine user sessions, 62 API keys, 33 passwords, 24 access tokens, 7 private keys, and 30 personal email addresses

• Across all sources, 1,028 blocks holding at least one confirmed leak

• 328 of the 6,708 sessions leaking at least one item

Sanitisation operates on plaintext only, so even if every user in that sample had scrubbed their visible text perfectly, all 62 API keys would have remained in the reasoning blocks. This was a non-exhaustive scan of public traces, and logs held locally or produced by live services are likely to contain more.

The fourth vector is prompt injection. Long agentic runs are expensive to repeat. Therefore, it makes sense to resume a published run. Encrypted blocks make that possible.

A block containing an instruction can be planted in a shared trace. When the session resumes, that block is processed as prior context. In one demonstration, a block carrying an instruction to upload PowerPoint files to an external address was ported into a conversation with GPT-5.6 Sol and paired with an unrelated slide-editing request. The resulting script added the requested slide and uploaded the presentation.

Field Observations

Recovering traces at this scale brought out findings unrelated to the vulnerability itself.

Summaries and traces sometimes diverge. On an AIME 2025 mathematics problem, the summary returned by the API described methodical work through the geometry using the law of cosines. The recovered trace opened by stating the answer from memory and then working backward toward a justification, with several wrong guesses along the way. The researchers did not measure how often this occurs. However, they note that a smaller model summarising a larger one will compress the original.

Recovered GPT traces are frequently not fluent English. They are made up of compressed telegraphic notes with articles dropped and grammar abandoned. This carries a direct consequence for oversight, because trace monitoring depends on traces being legible.

Proposed Mitigations

The research paper ultimately makes one strong point, which is the cause of the overall problem. The envelope we talked about earlier authenticates the content of a reasoning block, but it does not authenticate the context in which that block was produced or replayed.

The cheapest fix for this problem should address cross-user replay. This involves embedding an account identifier inside the authenticated data at issuance, allowing the API to compare it against the authenticated caller and reject any mismatch. This closes the attack vector entirely without the need for server-side storage.

Cross-session binding is harder, because some genuine features depend on the same portability. Users can fork conversations, compact old turns out of long sessions, and downgrade models mid-conversation. Binding each block to the complete transcript would break all three. The proposal is a hash chain tying each block to its session and to the fingerprint of its predecessor, combined with a Merkle tree that retains only root fingerprints once older blocks are pruned. This approach preserves ordering cheaply while still allowing integrity checks over any surviving stretch.

Blocks already published present a separate problem, since they were signed under a key encoding neither user nor session. The only retroactive remedy is to rotate those keys and refuse to decode anything signed under a retired key identifier. This also invalidates legitimate continuations of old sessions.

Other measures include moving to server-side storage entirely, configuring API gateways to reject envelopes from a different model version than the one queried, and training models to decline transcription requests regardless of framing.

Conclusion

We’ve now understood the key points in the research paper. Here are some of the main takeaways:

• This wasn’t a failure of cryptographic techniques. The guarantee made by the envelope remained throughout the flow. The missing piece was a binding between a block and the context that produced it, which is a decision about which fields go into the authenticated portion.

• The summary displayed alongside an answer is a separate artifact from the trace it describes, and the two can diverge.

• Sanitising a session log reaches the plaintext only. Encrypted blocks have to be removed rather than cleaned, since the person publishing them cannot inspect what they contain.

• The security of a model family depends on its least protected member. Anti-distillation training on a flagship model provides limited benefits while a cheaper sibling model accepts the same blocks.

References:

• Stealing Reasoning Traces from Proprietary LLM APIs

• Let’s talk about encrypted reasoning

• How to Steal Reasoning Without Reasoning Traces

• Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers

Give an agent an access token and it spreads: into the context window, into tool call logs, into notes it keeps between steps. Each copy works from anywhere, long after the fact.

Relay keeps the credential at WorkOS. Your agent names the user, WorkOS attaches that token, refreshes it, and releases it only to allowlisted hosts. A hijacked agent session is a live process you can kill.

Learn how it works →

The gap between code that executes fine and code that is actually safe to trust is growing wider pretty fast. For many years, writing code was the slow, expensive step, whereas reviewing it was a smaller task at the end. With the rise of AI-assisted coding, this balance is shifting.

AI tools can now create a working function in seconds and a full feature in minutes. Teams are able to write more machine-generated code every month. In other words, producing code is now fast and relatively easy, whereas code verification is the harder part. A reviewer still has to read the change, understand it, and decide whether it belongs in production. In fact, more code written simply means more code that should be verified.

We recently got a chance to speak with Andrea Malagodi, the CTO of Sonar (the company that has built some of the most used code verification software). He provided deep insights into code verification, especially in the context of AI and how Sonar is adapting to the recent changes.

In this article, we will look at how code verification works, why the rise of AI-generated code puts more pressure on it, along with the extremely useful insights from Andrea on what the future may look like.

The Shift

The shift with regard to code generation and verification is quite visible when we look at the data. One of the clearest signals comes from Google’s DORA research, a long-running study of how thousands of teams build and ship software. Their recent work found that as teams adopt more AI, delivery stability dipped. Trust in AI-generated code stayed low, with well over a third of developers reporting little confidence in what these tools produced [2]. In other words, more speed in writing code brought more pressure further down the line.

A controlled trial from the research group METR gives a similar indication. Its participants were experienced open-source developers working on their own mature projects, and each task was randomly assigned to allow or disallow AI tools. The developers expected AI to speed them up by roughly a quarter.

However, the result showed a totally different picture. AI-assisted tasks took about 19 percent longer [3]. Moreover, this happened after the developers internally believed that the AI helped them be more productive. Turns out, a lot of extra time went into prompting, waiting, reading the output, and correcting it. To be fair, the same team later reported a more confusing follow-up signal. This was partly because developers preferred to keep their AI tools [4].

Nevertheless, if we consider these results together, it is evident that while AI definitely increases the amount of code written, it also leads to more verification work down the line.

So let us first understand what code verification actually means.

Earning Trust

Code verification is the umbrella term for every check that ensures whether a piece of code is correct, safe, and maintainable enough to ship to production. In other words, it is the work of earning enough trust to put a change in front of real users. The key term to note here is “earning”. This is because trust arrives in degrees. It is built up one check at a time, rather than granted in a single stroke.

Think of a task of drafting a contract. Writing the words is one part of this task. However, the review, the legal checks, and the signatures are what transform those words into something people can actually rely on. Writing code works in a similar way. The moment a piece of code leaves an editor and is committed to a code repository, it carries an implicit claim about the functionality. Code verification is the process through which that claim gets tested until a team feels safe to use that code in a real production environment.

Some domains push this stage to its limit through rigorous formal verification. In such domains, engineers have to mathematically prove that the code being deployed matches a precise specification. Such a process is standard for critical stuff like flight-control systems and kernels, where a single defect can risk lives. However, for most software, having a similar approach can cost far more than it returns. Therefore, teams opt for lighter checks that are arranged in layers.

The Filter Stack

Taking the layered analogy further, we can imagine code verification as a stack of filters. As you can see, each filter catches a certain type of problem.

At the top of the stack, we have the cheapest checks. For example:

• Type Checker: It confirms whether the values moving through your code are the exact type each operation expects. This way it can catch a whole class of mistakes before the code even runs.

• Linter: It scans for suspicious patterns and style problems.

These types of checks can run in an instant and cost almost nothing. Below this stack, we have tests.

A unit test runs a small piece of code with known inputs and confirms whether it returns the expected output. Tests can catch behavioral mistakes in the code that a type checker cannot detect. This is because a piece of code can have perfectly valid types while still computing the wrong answer. For example, consider a simple function that is supposed to add two numbers, but instead multiplies them. In such a case, the type checker and linter would not point out any issues. Only a unit test that compares the result of the operation against a known answer will reveal the mistake.

Below the layer of tests, we have the human review filter. This is basically the case where another developer goes through the change and judges whether it fits the system, solves the right problem, and is readable. This layer catches what machines can miss, such as a solution that works yet takes an approach the team standards don’t recommend.

Beneath all of these layers is the production monitoring setup. The job of this system is to observe the code under real traffic and flag problems that may have passed through every earlier layer.

Real-world filter stacks can also hold more layers than this. This may include security scanners and dependency checks. However, the point is that each filter covers a specific weakness in the one above it. This is why serious teams run several such layers in a specific order before releasing any code into production.

Static And Dynamic Analysis

The filters in that stack fall into two families:

• Static Analysis: The filters in this family check the source without executing it, which makes this type of analysis fast and broad. With static analysis, we are able to scan an entire codebase in one pass. Type checkers and linters belong here. The tradeoff is that real behavior at runtime remains partly out of view. Therefore, static analysis can sometimes raise an alarm about a problem that might not exist during live conditions.

• Dynamic Analysis: The filters in this family run the code with real inputs and observe the result. Tests belong here. This family relies on checking actual behavior, but is limited by paths that are exercised. A test suite that only runs the happy path cannot detect a crash that might be waiting to happen on an empty input.

Despite the extensive coverage, a clean scan and a green test suite together can still leave gaps. This is why code verification relies on many filters working together to be effective.

False Alarms

A tempting conclusion we might make is that more checking is always better. However, there is a tradeoff at the heart of code verification. Every filter can make two kinds of mistakes:

• False Positive: This means flagging something as a problem when the code is actually fine.

• False Negative: This means staying quiet while a real bug slips through.

If we tune a tool to catch every possible issue, it can flood the developers with false alarms. However, if we tune it to stay quiet unless it is certain, it can start to miss real defects. The two aspects pull against each other.

False alarms carry a pretty steep cost. When a tool raises frequent false alarms, developers start ignoring it. However, this habit can be catastrophic. Even an occasional real warning can get waved away with the rest. Research on static analysis tools describes this pattern, where high false-positive rates erode trust until teams switch the tool off or don’t pay attention to its warnings. However, doing so reopens the door to the very bugs the tool was meant to stop [7].

This is why a good code verification setup cares as much about signal quality as about coverage. Andrea from Sonar described the balancing act as almost a CAP theorem for code verification. This was built from the classic idea that you can push some properties only at the expense of others. The three competing priorities in the case of code verification are speed, accuracy, and coverage, and no tool fully wins all three. Andrea’s team tries to keep the focus on humans with a simple rule that a finding a developer can act on is worth raising.

The positioning of the filter can also change the cost associated with a mistake. This brings us to the overall setup of the code verification pipeline.

The Pipeline

Filters run at different moments in the lifecycle of a change request. The timing determines their cost. If we spread the moments out in order, we get a pipeline. A change begins in the developer’s editor, moves to a set of automated checks that fire the moment code is committed, then to review, then to a merge, then out to deployment and live monitoring.

The same check becomes more expensive the later it runs. This is because more work piles on top of the mistake as development progresses. For example, catching a flaw in the editor may cost a brief moment of the author’s attention. However, catching that same flaw after it reaches production can result in an incident or a rollback. There might also be user impact. This is the real meaning of the phrase “shift left”: moving checks earlier in the pipeline so problems surface while they remain cheap to fix.

Often, vendors might try to market this approach with precise multipliers. You might come across claims that a bug costs ten times more at each stage. While the exact numbers deserve skepticism, the important takeaway is that the general direction of pushing checks earlier in the pipeline is beneficial from a cost point of view.

AI Pressure

This stack of filters was built for a setup where developers wrote most of the code. As we have seen, this assumption has weakened over the last few years with the rise of AI-coding tools. This change has an impact on every layer in two distinct ways.

The first pressure is volume.

When an agent writes a thousand lines in the time a person once wrote a hundred, the review burden increases dramatically. There is also a subtle effect on batch size, meaning the amount of change bundled into a single review. AI-based coding tools tend to produce larger changes, which are harder to review. This is because attention spreads thin across a big diff and small mistakes can slip through more easily. Andrea mentioned this failure mode with a line many developers will be familiar with. The reviewer who faces a 5,000-line pull request types “looks good to me,” and figures that things will anyway surface during production.

The second pressure concerns the kind of mistakes that AI can make. A study across more than a hundred models tested the security of AI-generated code and found that it introduced a known security flaw in roughly 45 percent of cases [5].

Over the same period, while these models have become far better at producing code that runs cleanly, their security checks have remained mostly flat. In other words, AI has improved sharply at making the code work, but only a little at making that code truly safe. If anything, the gap between the two aspects has widened. A separate analysis of millions of code changes has also found rising duplication and falling reuse [6].

Reviewing AI

When there is more code than developers can carefully read and analyze, the natural approach is to hand over some of the code verification to machines. This is why AI-driven code review has gained real momentum over the past few years.

An AI code reviewer offers three main advantages:

• Speed: It scans a change the moment it appears, before a human has time to look.

• Coverage: It catches a meaningful share of bugs and security issues early.

• Consistency: It applies the same standards across every change and every team member. Due to the probabilistic nature of AI code review, maintaining consistency can be challenging. It’s important to have multiple layers of review that include both AI-driven tools and other deterministic algorithmic tools.

This review process can also run inside the agent’s own loop. The agent writes a draft, the reviewer flags problems, and the agent corrects them before a human developer ever gets a chance to look at the code. This tightens the feedback loop and clears routine work off a human reviewer’s plate, ultimately helping teams handle larger volumes of generated code.

See the diagram below:

However, this approach also has a risk. A reviewer model built from the same kind of model as the code creator tends to work with the same assumptions. It would therefore have the same blind spots. When both the writing of the code and reviewing it depend on similar training and similar patterns, the reviewer can confirm that the code looks right. The key question about whether the code does what was actually intended stays irrelevant. In other words, two similar models can resemble one opinion stated twice more than two completely independent checks.

Consider an agent that turns a ticket into a function. An AI reviewer scans it and reports the code as clean. The code compiles, runs, and matches common patterns. Whether it does what the ticket truly meant cannot be answered by pattern-matching alone. This is where different views exist:

• Some argue that the models have grown capable enough to reduce or even remove the human review step.

• Others hold that people remain essential for judgment about architecture, context, and accountability.

Both camps make a fair case. The reasonable answer today is that it depends on what you are shipping and the cost of a potential mistake.

The Modern Stack

Let us now see how these ideas assemble into a real workflow.

Everything starts with the context. Most engineering happens in brownfield code, meaning large existing codebases with history and quirks, rather than greenfield projects that have started from scratch. An agent looking into that code without guidance works out the layout on its own, but it does so differently each time. Andrea called that inconsistency “a box of chocolates”, where the result you get back varies from one run to the next and from one developer to the next.

A mature setup handles this by feeding the agent a shared and consistent picture up front. This includes the real architecture, the coding guidelines for the language, and rules that try to capture the intended design. Three loops take care of the verification part. Here’s a brief breakdown of the loops:

• Agentic Loop - Where agents iteratively build: It optimizes code generated within the agentic sandbox and improves agent effectiveness. It also reduces token costs, improves output quality, and reduces risk.

• CI verification loop - The validation pipeline for all code: Deals with code review, zero-trust, multi-layered verification, and quality gate at sandbox exit. It also merges fixes at high velocity and volume with confidence.

• Code maintenance loop - Background remediation of tech debt: It continuously patrols to address legacy issues in the background agentically. Cleaner code makes it easier for coding agents to work efficiently.

Andrea also helped sketch what a mature setup in 2026 might look like. In the case of Sonar, the components of the modern stack are as follows:

• Verification Engine: It consists of thousands of rules doing automated code analysis for reliability, maintainability, and security across more than 40 programming languages, frameworks, and IaC technologies. Sonar’s offerings, SonarQube (what it’s most known for) and Sonar Vortex (a new solution), provide this in the agentic loop.

• AI Code Review: A newer capability built from AI, that came through Sonar’s acquisition of Gitar. It uses carefully written instructions rather than fixed rules. Its real value is making each finding explainable to the reviewer, so that a 5,000-line change becomes something a developer can reason about. This sits in the CI verification loop.

• Remediation Agent: Aimed at the existing backlog, working through old issues progressively to clean up history rather than only guarding new code. This covers the code maintenance loop.

Other companies are more or less converging on a similar idea.

There is also a big advantage to keeping the code clean. The team at Sonar measured what happens when AI-generated code, which is often tangled and rather dense, is left to evolve across many developer sessions over half a year or more. They found that messy code ultimately starts to cost more tokens to work with. This is because the AI model needs to spend more effort understanding it every single time there is a change.

One more aspect sits at the very front of this debate. It concerns secrets, meaning credentials like API keys and passwords. The danger with secrets is rarely intentional sabotage. Most of the time, someone pastes code or loads a configuration file into an AI session. The secret then rides along with the change, and it is usually caught too late, once it has already become part of a commit or a log file.

The fix for this is to run a scanner right at the terminal, before the developer pastes the code. In other words, the goal would be to stop it as early as the process allows. Sonar calls this approach “starting left”. This is one step earlier than the familiar shift left that we talked about. Andrea’s advice is that every developer should run a guard like this to ensure that the secrets remain safe.

Trust And Risk

All these points lead to one important practical question.

How much code verification does a given change actually need?

The answer is that it depends on what a specific failure would cost. Determining the cost is the real skill that requires insight. For example, a typo on a marketing page and a bug in a payment system deserve very different scrutiny. A developer can probably fix the typo on a marketing page in a minute without much effort. However, the bug in the payment system can move money to the wrong place, break trust, and trigger an unwanted news item.

Mature teams treat verification depth as a dial dependent upon the risk factor. Low-risk changes pass through with light automated checking. However, high-risk changes often get routed to human eyes and undergo heavier scrutiny.

Deciding the exact position of where the line should be drawn is a judgment call that should be made by the team. For example, a comma can be the difference between a working operating system and a crash, so the risk appetite has to be chosen in a deliberate manner. There is no fixed rule that can be applied to all situations.

Agents take a change as far toward a clean and verified state as they can on their own, while staying within guardrails. They can merge low-risk work automatically while routing riskier stuff to a human developer. However, setting these tiers properly so that a change is routed to the right level of checking is an emerging practice that is only going to get more important with time.

Conclusion

The center of gravity in software development is shifting. Writing code has now become the faster activity. On the other hand, verifying that same code, confirming that it is correct, secure, and worthy of real users, is where the effort seems to be increasing.

As we have seen, code verification works as a stack of filters where each filter trades a bit of cost for a bit of confidence. Each filter covers a weakness in the one above it. Those filters divide into static and dynamic families. Every filter balances false alarms against missed bugs, and moving filters earlier keeps their mistakes cheap.

The flood of AI-generated code has an impact on all of it, raising both the volume and the risk. Also, the tempting shortcut of letting AI review for AI has a real catch, since a machine checking a machine can agree that code looks fine while the more important checks get ignored.

The human side to this shift is that as writing code gets cheap, the developer’s work becomes even more important. Developers need to spend more time orchestrating agents by providing instructions. They might have to focus more on the older and harder problem of knowing what to build at all. Cheaper code generation provides more room for spending time on that kind of judgment rather than removing the need for it.

References

1. CrowdStrike outage: We finally know what caused it and how much it cost

2. Announcing the 2024 DORA report

3. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

4. We are Changing our Developer Productivity Experiment Design

5. 2025 GenAI Code Security Report

6. AI Copilot Code Quality: 2025 Data Suggests 4x Growth in Code Clones

7. FP-Predictor: False Positive Prediction for Static Analysis Reports

8. Everyone cites that bugs are 100x more expensive to fix in production, but the study might not even exist

Many teams over-provision containers, underuse spot instances, and have no visibility into which pods are burning budget. Get the eBook from Datadog, which covers five practical optimizations for Kubernetes and ECS environments with specific techniques your team can apply today.

You’ll learn how to:

• Pinpoint idle containers, over-provisioned pods, and unused clusters draining your cloud budget.

• Right-size CPU and memory with resource requests, limits, and automated cost recommendations.

• Cut costs up to 90% with spot instances and savings plans and know exactly when to use each

Get the ebook

This week’s system design refresher:

• Ollama vs vLLM vs SGLang

• How does Claude’s text watermark work?

• Top 12 Agent Skills You Should Know

• Git Workflow: Essential Commands

• Apache Kafka vs. RabbitMQ

Ollama vs vLLM vs SGLang

To use open-weight models on your machine, you have three main options: Ollama, vLLM, and SGLang. But each engine handles requests differently. The diagram below shows the differences and the main techniques behind each engine.

Ollama: A local user calls the OpenAI-compatible API, and requests line up in a FIFO queue. Then Ollama runs a pre-quantized GGUF model, a compressed format it pulls, and the response comes back to the user.

Ollama is best for local dev, prototyping, and laptop-scale hardware.

vLLM: Many users hit the server at once, and continuous batching slots new requests into the running batch instead of making them wait for it to finish. PagedAttention stores the KV cache, the memory a model keeps for tokens it has already processed.

vLLM is best for high-traffic serving, max GPU utilization, and thousands of concurrent requests.

SGLang: Agents and multi-turn chats send requests whose prompts overlap heavily. A prefix-aware scheduler routes them through the RadixAttention cache, a radix tree that reuses every shared prefix instead of recomputing it.

SGLang is best for AI agents and tool loops, multi-turn chats, and JSON/regex outputs.

How does Claude's text watermark work?

Anthropic recently shared their intent to watermark text so they can identify AI-generated text. This post is based on my understanding of how it works.

LLMs produce text word by word. At each step, they generate probabilities for the next likely word. Instead of sampling randomly from those words, the watermarking trick changes which words are allowed to be picked.

How to watermark a response?

Step 1: The model produces probabilities for the next word.

Step 2: Normally a random number generator picks one of the good candidates. With watermarking, a keyed function takes a secret key plus the previous few words and decides which candidates are valid to pick from.

Step 3: This repeats for the whole response. Places where there are multiple plausible choices carry the watermark signal.

How to detect a watermarked text?

Step 1: For any candidate word in the text, we check whether it is a valid choice based on the secret key and the few preceding words. If the word is valid, that is counted as a match.

Step 2: Run this across the entire text. Watermarked text matches far more often. The overall match rate can be treated as an AI-generated score.

I’m personally getting quite annoyed by the false negatives from all these AI text detection techniques, especially for technical writing.

What's your thoughts on AI text detection? Do you think AI text detection is useful, or will it create more problems?

Top 12 Agent Skills You Should Know

Agent skills are instructions and scripts that teach your LLM agent a new skill. The diagram below shows the 12 most-starred skill repos on GitHub as of August 2026.

1. Superpowers (obra/superpowers): This skill makes your agent plan before it writes code.

2. skills (mattpocock/skills): Matt Pocock's personal skill set makes your agent challenge your plan first. This is useful as agents can sometimes be too soft.

3. andrej-karpathy-skills: Multica AI distilled Karpathy's advice on AI coding pitfalls into one skill.

4. everything-claude-code: Skills that help you set up your coding agent. This is useful when you are starting Claude Code from scratch.

5. skills (anthropics/skills): This is Anthropic's official skills. It makes your agent capable of creating outputs like Word or PDF files.

6. ui-ux-pro-max-skill: This has instructions that teach your agent how to prevent AI-like designs.

7. caveman: Julius Brussee's skill makes your agent reply in short caveman speak.

8. ponytail: Dietrich Gebert's skill teaches your agent how to write code that is simple and clean.

9. agent-skills: Google's Addy Osmani included production-grade engineering practices in a skill

10. graphify (safishamsi/graphify): This skill converts a codebase into a knowledge graph, so an agent can navigate easier.

11. Understand-Anything: Egonex AI converts a codebase into visual maps to explore.

12. impeccable (pbakaus/impeccable): This skill makes an agent better at UI polish.

Over to you: Which skill would you add to this list?

Git Workflow: Essential Commands

Git has a lot of commands. Most workflows use a fraction of them. The part that causes problems isn’t the commands themselves, it’s not knowing where your code sits after running one.

Working directory, staging area, local repo, remote repo. Each command moves code between these. Here’s what each one does.

• Saving Your Work: “git add” moves files from your working directory to the staging area. “git commit” saves those staged files to your local repository. “git push” uploads your commits to the remote repository

• Getting a Project: “git clone” pulls down the entire remote repository to your machine. “git checkout” switches you to a specific branch.

• Syncing Changes: “git fetch” downloads updates from remote but doesn’t change your files. “git merge” integrates those changes. “git pull” does both at once.

• The Safety Net: “git stash” is your undo button. It temporarily saves your uncommitted changes so you can switch contexts without losing work. “git stash apply” brings them back. “git stash pop” brings them back and deletes the stash.

Apache Kafka vs. RabbitMQ

Kafka and RabbitMQ both handle messages, but they solve fundamentally different problems. Understanding the difference matters when designing distributed systems.

Kafka is a distributed log. Producers append messages to partitions. Those messages stick around based on retention policy, not because someone consumed them. Consumers pull messages at their own pace using offsets. You can rewind, replay, reprocess everything. It is designed for high throughput event streaming where multiple consumers need the same data independently.

RabbitMQ is a message broker. Producers publish messages to exchanges. Those exchanges route to queues based on binding keys and patterns (direct, topic, fanout). Messages get pushed to consumers and then deleted once acknowledged. It is built for task distribution and traditional messaging workflows.

The common mistake is using Kafka like a queue or RabbitMQ like an event log. They’re different tools built for different use cases.

Over to you: If you had to explain when NOT to use Kafka, what would you say?

To make matters more complicated, the migration goes smoothly and cleanly during the staging phase, but unrelated services and components start failing as soon as the change is deployed to production. On investigation, it is found that nothing was wrong with the migration itself. But it took effect while two versions of the application were still running against the same database, and only one of those versions referenced the modified schema.

This situation is common with schema-related changes. It is also not limited to the deployment window. For example, rows written years ago can get produced by application code that has since been replaced. The messages sitting in a queue were published before the current version of the consumer was written. Mobile app versions from eighteen months back are still installed on real devices and still calling the API. In each case, data written under a particular schema version is read under a different version, resulting in multiple issues.

In this article, we will look at schema evolution and strategies for the same. Here’s what we will cover:

• Why more than one schema version is always in play at the same time

• Backward and forward compatibility

• Which changes break consumers, which do not, and the qualifiers that decide it

• Expand and contract migrations

• Schema registries and their use

• How the same problem differs across databases, APIs, and event streams

• Versioning strategies and deprecation timelines

Version Overlap

Read more

Turning new models into systems that work inside real customer operations is still hard.

That gap is creating demand for engineers who can move between code, customer context, and production outcomes. Enter: the forward deployed engineer.

The free State of FDE Jobs 2026 report maps the emerging labor market around this work.

Explore the report here

Imagine an AI-based retrieval system pointed at five years of your team’s engineering documents, including design docs, incident postmortems, and architecture decision records. Someone asks which service owns the payments retry logic, and a pretty accurate and well-cited answer is provided by the system. However, when someone asks which failure causes recur most often across all the postmortems, the quality of the answer goes down.

Depending on the setup, the response might list a handful of incidents that happen to use the word recurring, but we don’t get any idea of the underlying pattern from the answer. In other words, the reason for asking the question is not fulfilled.

Both questions can look similar from the outside. Architecturally, however, they are opposites:

• The first has an answer that can be found in a specific document, which is precisely what similarity search was built for.

• The second has an answer that shows up only after the entire collection has been surveyed and understood. This requires a completely different retrieval mechanism.

GraphRAG was designed to handle the second kind of questions, and we are going to learn more about it in this article. Here’s what we will cover:

• How standard RAG retrieval works, and where it reaches its limit

• Knowledge graphs, and how one gets built from ordinary documents

• The GraphRAG indexing pipeline

• Community detection and hierarchical summaries

• Local search and global search

• Cost, latency, and maintenance tradeoffs

• When standard RAG remains the better option

• Agentic RAG

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

Retrieval Basics

Standard RAG (Retrieval Augmented Generation) depends on a compact pipeline.

We take a collection of documents, slice each one into chunks of a few hundred to a few thousand tokens, and pass every chunk through an embedding model. The embedding model returns a vector, which is basically a long list of numbers standing in for the meaning of that text. Chunks with related meanings produce vectors that sit close together in the same numeric space. All of those vectors go into a vector index.

At query time, the same treatment applies to the question. The question also becomes a vector, the index returns the handful of chunk vectors closest to it, and the original text of those chunks gets placed into the prompt alongside the question. The language model then generates an answer from the supplied text.

The whole design rests on one simple assumption, which is that text answering a question would resemble that question. For a large share of queries, this assumption holds up well. For example, a question like “Which service owns the payments retry logic” contains the same vocabulary as the architecture decision record where that ownership was recorded. The vectors land near each other, retrieval returns the right document, and the citation points somewhere a reader can actually verify.

Similarity Limits

This assumption about questions being similar to the answers holds for a specific class of questions, but it is by no means a universal thing.

Microsoft’s GraphRAG documentation distinguishes local queries from global queries. A local query has an answer that resembles the query and lives inside a small number of text regions, which covers most who, what, when, and where questions. A global query requires reasoning across large portions of a dataset, or across all of it.

Our two example questions from earlier land on opposite sides of that line. The question “Which service owns the retry logic” is local. However, the question “Which failure causes recur most often across all postmortems” is global.

The reason the answer for the second one goes down in quality is that the phrase “recur most often” produces a vector, and the index returns whatever appears nearest to it. Across a corpus of incident reports, the nearest neighbours will be documents using words like recurring or frequent, which is a coincidence of vocabulary. However, the real answer to the question exists across two hundred documents as a distribution, which spans the corpus rather than occupying one retrievable location.

A reasonable objection at this point is that modern context windows are large enough to sidestep the problem entirely. Microsoft tested exactly that, comparing GraphRAG against vector retrieval pulling in 8,000 and then 64,000 tokens of context. However, on global questions, the larger window left the gap open on comprehensiveness, diversity, and quality of supporting source material.

This outcome is usually labelled as a hallucination problem. What actually happens is that retrieval returns material with little bearing on the question, and the model produces fluent text from it.

[Webinar] How to stop babysitting your agents (Sponsored)

Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part. You end up wasting time and tokens in the correction loops.

More MCPs, rules, and bigger context windows give agents access to information, but not understanding. The teams pulling ahead have a context layer to give agents exactly what they need for the task at hand.

Join us for a FREE webinar on Sep 2 to see:

• Where teams get stuck on the AI maturity curve and why common fixes fall short

• How a context layer solves for quality, efficiency, and cost

• Live demo: the same coding task with and without a context layer

If you want to maximize the value you get from AI agents, this one is worth your time.

Register now

Knowledge Graphs

Crossing this boundary in terms of the quality of answers requires recording how documents relate to one another, instead of treating each chunk as an independent unit of text. A knowledge graph is one way to record it.

A knowledge graph stores two kinds of things:

• Entities are the nouns a corpus talks about, such as people, services, teams, incidents, and decisions.

• Relationships are the typed connections between those entities.

Both carry a plain-text description.

Take one sentence from an incident postmortem: “The checkout service began returning timeouts after the payments team deployed the new retry handler on March 3.” Extraction over that sentence produces entities for the checkout service, the payments team, and the retry handler, along with relationships recording that the team deployed the handler and that the deployment preceded the timeouts.

Once thousands of sentences have each contributed nodes and edges, paths appear that no single document contains. An engineer may be namednamed in one design doc, a service is named in a second, an incident is described in a third, and the path from that engineer to that incident runs through both intermediate nodes.

LinkedIn’s customer service team published results from this approach at SIGIR in 2024. Their support tickets had been stored as plain text, which discarded the internal structure of each ticket along with the connections between tickets. Rebuilding retrieval around a knowledge graph that preserved both improved mean reciprocal rank by 77.6 percent, and median per-issue resolution time dropped 28.6 percent in production. Similarly, Neo4j’s documentation separates the lexical graph, which links documents to their chunks, from the entity graph, which links the things those documents describe.

Most GraphRAG systems build both and query across them.

Graph Construction

Building that graph from raw documents is a pipeline, and most of its cost concentrates in a single stage.

For reference, Microsoft’s documented indexing workflow runs through six phases:

• Documents are sliced into text units, the same chunking step standard RAG performs.

• A language model processes each text unit and extracts entities carrying a title, type, and description, along with relationships carrying a source, target, and description.

• Entities sharing a title and type are merged across text units, and their descriptions collect into an array. A second language model pass compresses each array into one description. Relationships receive the same treatment.

• Claim extraction runs optionally, producing time-bound factual statements about entities.

• The assembled entity graph is clustered into a community hierarchy.

• Community reports are generated, and text units, entity descriptions, and report contents are embedded into a vector store.

The merge step accounts for a lot of the expense. For example, a service mentioned across two hundred documents produces two hundred separate descriptions during extraction. Every one of those has to be reconciled into a single coherent description before the graph becomes usable.

Every extracted entity, relationship, and claim retains a pointer back to the text unit it came from. This pointer is what allows a generated answer to cite a specific paragraph in a specific document.

Also, two language model passes over an entire corpus add a substantial amount of inference. Microsoft’s documentation estimates graph extraction at roughly 75 percent of total indexing cost. Lastly, extraction quality also depends on prompts tuned to the domain.

As a separate point, FastGraphRAG replaces the language model in extraction with traditional NLP, treating noun phrases as entities and co-occurrence within a chunk as a relationship. Indexing becomes far cheaper, and the resulting graph carries considerably more noise

Community Detection

A graph of entities and relationships answers connection questions well. However, answering whole-collection questions requires one more layer on top of it.

GraphRAG runs hierarchical Leiden clustering across the entity graph. The algorithm recursively partitions the graph into clusters, called communities, and keeps subdividing until communities fall below a size threshold. The output is a hierarchy with several levels.

These levels behave like a resolution control over the same underlying graph. Level 0 contains a small number of broad communities, each covering a large region. Deeper levels contain many more communities, each covering a narrower region. A single payments community at level 0 might split into separate communities for retry behaviour, settlement, and fraud checks two levels further down.

For every community at every level, a language model generates a community report. Each report contains an overview of that community along with its key entities, relationships, and claims. Reports are then summarized again into shorthand versions for compact use at query time.

This is the step that makes whole-collection questions answerable. A summary of what a cluster of documents collectively says gets written during indexing, well before anyone asks about it. When a global question arrives, the material required to answer it already exists as text.

Which particular level supplies the reports is a decision with real consequences. Microsoft’s documentation states that response quality is heavily influenced by that choice. Lower levels produce more thorough answers because their reports carry more detail, but they also cost more time and more tokens because there are many more reports to process.

Query Modes

With a graph and a hierarchy of reports sitting on disk, retrieval can follow two structurally different paths. GraphRAG supports both.

Local search begins by matching the query against entity description embeddings, which produces a set of entry-point entities. From each entry point, expansion proceeds along five directions in parallel:

• Text units that mention the entity.

• Community reports that contain it.

• Neighbouring entities connected to it.

• The relationships forming those connections.

• Covariates, meaning any extracted claims attached to it.

Each of those candidate sets is ranked and filtered independently. The survivors are packed into a single context window of predefined size. The expansion is bounded, and the ranking is explicit, which makes local search closer to a structured gather-and-rank operation than to open-ended pathfinding across the graph.

Global search leaves the entity graph untouched. Community reports from a chosen hierarchy level are split into batches, and those batches are shuffled so that batch ordering stays randomized. A map stage runs each batch through a language model and produces an intermediate answer where every point carries a numerical importance rating. A reduce stage then collects the highest-rated points across all batches and generates the final answer from them.

The mapping back to our payments questions can be made clearer now. The question: “Which service owns the retry logic” names an entity, so local search locates it and expands around it. Also, the question “Which failure causes recur most often” names no entity in particular, so global search aggregates across pre-written reports covering the whole corpus.

See the diagram below:

A third mode, DRIFT search, blends the two. It starts by comparing the query against the most relevant community reports to produce a broad initial answer along with follow-up questions, runs local search against those follow-ups, and returns a hierarchy of questions and answers ranked by relevance.

GraphRAG also ships a basic search mode, which is plain top-k vector retrieval, for queries where that remains the appropriate tool. When an answer comes back broad and shallow, or narrow and precise, the query mode usually explains it.

[Diagram 6. Local search and global search side by side] Left panel traces the query to matched entities, then the five parallel expansion streams, then per-stream ranking and filtering, then a single assembled context window. Right panel traces the query alongside shuffled community report batches, into parallel map calls producing rated intermediate answers, then filtering, then the reduce call producing the final answer.

Cost Tradeoffs

What we have looked at so far are the various capabilities associated with GraphRAG. However, cost determines whether a specific capability is worth acquiring for a given system.

Standard RAG has a modest cost at both ends, with one embedding pass at index time and one nearest-neighbour lookup per query. In contrast, GraphRAG redistributes the spending considerably. Index time absorbs two language model passes over the corpus plus report generation for every community at every level. Query time then splits, with local search running cheaply against a prepared context window, and global search running a language model across many report batches for a single question.

The index is also a derived artifact. New documents arriving means extraction, clustering, and summarization run again over the affected material, and the community hierarchy itself can shift as the graph grows. For a corpus that changes daily, this becomes an ongoing operational commitment.

Microsoft’s own follow-up work addressed the cost directly. For example, LazyGraphRAG builds its index using NLP rather than a language model, skips summarization entirely, and defers all language model work to query time. In this case, indexing cost matches vector RAG and lands at 0.1 percent of full GraphRAG. Global-query quality stays comparable to global search while query cost drops by more than a factor of 700.

Microsoft still argues against making every deployment LazyGraphRAG. Their stated reasoning is that the pre-built entity, relationship, and community summaries carry value beyond question answering, because people read and share those reports directly.

Two findings from Microsoft’s own evaluations are as follows:

• Vector RAG remains the stronger option for local queries, where the answer resembles the question and sits in a specific region of text.

• GraphRAG’s measured advantage lies in comprehensiveness, diversity, and supporting source material. On faithfulness, it scored at a similar level to baseline RAG.

The second point matters whenever someone asks whether GraphRAG reduces hallucination. The evidence supports better coverage and better sourcing, but it stops short of claiming better factual accuracy per individual claim.

Agentic Retrieval

Since different question types favour different retrieval strategies, committing to one strategy when a system is built gives up the others.

Agentic RAG helps formulate a response to that constraint.

A language model classifies the incoming query, selects a retrieval strategy, executes it, and synthesizes the result. Available strategies might include vector search for local questions, global search for corpus-wide questions, a SQL query for structured data, and web search for anything current.

LlamaIndex documents a two-layer version of this:

• A composite retriever selects which index to query, guided by a description supplied for each index.

• Within the selected index, an auto-routed mode then selects which retrieval method applies to that specific query.

In other words, routing decisions happen at both layers.

The approach carries its own costs. It adds a language model call ahead of retrieval, which increases both latency and per-query spend. Routing errors also produce a debugging problem, because a poor answer can come from a perfectly good retrieval running under the wrong strategy.

The overall progression from basic RAG through advanced RAG to GraphRAG and then agentic RAG describes a sequence of decisions about when a system commits to a retrieval strategy. It works well as a ladder where each step outperforms the one below it.

Conclusion

In this article, we’ve gone deep into GraphRAG and understood it in detail. Here are the key learning points to remember:

• Local queries have answers that resemble the question and sit in a small number of text regions, while global queries require reasoning across large portions of a collection.

• Similarity search returns whichever chunks sit nearest the query vector, so global questions tend to retrieve vocabulary matches instead of the underlying pattern.

• Larger context windows leave that gap open. Vector retrieval with 64,000 tokens still trailed on global questions in Microsoft’s testing.

• A knowledge graph stores entities and typed relationships with descriptions, preserving connections that plain chunking discards.

• GraphRAG indexing runs two language model passes over the corpus, one to extract entities and relationships and one to merge their descriptions.

• Graph extraction accounts for roughly 75 percent of indexing cost, making it the first stage to examine when reducing spend.

• Hierarchical Leiden clustering produces communities at several levels of resolution over the same entity graph.

• A community report is generated for every community at every level, so summaries of what the corpus collectively says exist before any question arrives.

• Local search expands from matched entities and ranks the results into one context window, while global search runs map-reduce across community reports.

• The index is derived and perishable. LazyGraphRAG responds by moving language model work to query time, cutting indexing cost to 0.1 percent of full GraphRAG.

• Vector RAG remains stronger for local queries, and agentic retrieval selects a strategy per query rather than committing to one when the system is built.

AI coding agents can generate code quickly, but CI checks often happen after the agents finish their work. Sonar Vortex changes this pattern. Sonar Vortex operates inside the agent’s coding loop, giving agents architectural context before they write and verifying their output in real time as they produce it. Internal testing found 36% lower token consumption and 92% fewer defects.

Explore Sonar Vortex

Thinking Machines released a model called Inkling on July 15, 2026. A few interesting points made in its introduction are as follows:

• Inkling has 66 layers, and each layer holds 256 experts, of which only six activate for any given token [2].

• Most layers can access only a short window of recent text, while a few can access all of it [1].

• Word positions are encoded using a method most labs moved away from years ago.

Thinking Machines, founded by Mira Murati (the ex-CTO of OpenAI), describes its mission as building AI that extends human will and judgment. The company lists four directions of work, which are training strong models, building tools that let people customise models with their own knowledge, developing interfaces that widen the communication channel between people and machines, and publishing research on how models are made [3].

Before Inkling, the company shipped Tinker, a service for fine-tuning open models [4]. Inkling is the company’s first model trained from scratch [1]. The weights sit on Hugging Face under an Apache 2.0 license [2], so anyone can download them and retrain the model on their own data.

In this article, we will work through the various choices Thinking Machines made while building Inkling. Here is what we will cover:

• Mixture of Experts, and the gap between 975 billion parameters and 41 billion.

• The mix of local and global attention layers behind a context window of one million tokens.

• Position encoding, and the older method Thinking Machines chose over the current standard.

• How images and audio enter the model without a separately pretrained encoder in front of them.

• Thinking effort, a setting between 0 and 1 that adjusts how much the model reasons before answering.

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

The diagram below shows where each of these five things sits inside the model.

Groundwork

Let us first understand four key terms that are really important to make sense of the architecture:

• Token: A token is a chunk of text. Models work with pieces smaller than sentences and usually a bit smaller than words. For example, the sentence “Inkling was released in July” might become six tokens, with common words getting one token each and unusual words getting split into two or three.

• Parameter: A parameter is one number stored inside the model, learned during training. When you read that a model has 975 billion parameters, it is the count of individual numbers sitting in the file. Each one started as random noise and was adjusted millions of times until the model produced sensible text.

• Training: It works by showing the model text, letting it predict the next token, comparing that prediction against the token that actually came next, and then adjusting every parameter slightly in whichever direction would have made the prediction better. The size and direction of each adjustment is called a gradient, and the procedure that computes all of them at once is called backpropagation. By repeating this across trillions of tokens, the parameters settle into values that produce useful predictions.

• Layer: It is one processing stage in the journey of a token. Inkling has 66 of them stacked in order [2]. A token’s representation enters layer 1, gets transformed, passes to layer 2, and so on until layer 66, after which the model predicts the next token. Every layer has the same two parts:

  • An attention step that pulls in information from other tokens in the sequence.

  • A feed-forward step that transforms the result.

Sparsity

Inkling separates the cost of storing a model from the cost of running it. This is the reason a model this large is affordable to use.

In an ordinary transformer, the feed-forward step in each layer is a single network, and every token passes through all of it. If that network holds 5 billion parameters, then every token processed involves all 5 billion.

Inkling replaces that single network with 256 smaller ones, called experts. For each token, a selection step picks six of the 256. Only those six run, and the remaining 250 sit idle for that token while handling other tokens instead [2]. This design pattern is called Mixture of Experts, and Thinking Machines states that their version largely follows the approach published by DeepSeek [1][8].

See the diagram below:

Let’s understand this via a simple example:

• Suppose you build a layer with 10 experts holding 1 billion parameters each, and you run 2 of them per token.

• The file on disk contains all 10 billion parameters, and every one of them has to be loaded into memory before you can run the model at all.

• Processing a single token involves 2 billion parameters, so each token costs about a fifth of what it would in an ordinary layer of the same total size.

Inkling does this at scale. The total across the whole model is 975 billion parameters, and the count involved in processing any single token is roughly 41 billion [1]. That is about 4 percent of the model running at a time.

The hardware requirements make things clearer. A checkpoint is the saved file holding all the trained parameters. Inkling’s full-precision checkpoint needs at least 2 TB of combined GPU memory, which the model card gives as eight NVIDIA B300 cards or sixteen H200 cards [2].

Thinking Machines also talks about a quantised checkpoint. This means the same parameters are stored with less numerical precision, in roughly the way that writing 3.14 instead of 3.14159265 uses less space at the cost of some accuracy. That version needs around 600 GB and fits on four B300 cards [2].

What Makes an FDE Role Credible? (Sponsored)

Strong candidates are skeptical of vague forward deployed engineer postings, and the title alone won’t earn their trust.

The free State of FDE Jobs 2026 Report explains what candidates look for, how the market is evolving, and how employers can make these roles easier to understand.

Hiring? You can also bring your openings to forwarddeployedengineer.com, the focused jobs board for forward-deployed engineers.

Explore the Job Board

Routing

Picking six experts out of 256 sounds simple, but it is not. Here’s how the selection works:

• A small component called the router produces a score for every expert.

• Each expert has its own list of numbers attached to it, learned during training.

• The router compares the incoming token’s representation against each of those lists and gets a raw number out, higher when the two match closely.

• Those raw numbers can be any size, so they get passed through a sigmoid. A sigmoid is a mathematical function that takes any number and converts it into a value between 0 and 1. For example, feed it 8, and you get back about 0.9997. Feed it 0, and you get exactly 0.5. Feed it negative 4, and you get about 0.018. Very large numbers approach 1, very negative numbers approach 0, and everything else ends up somewhere in between. Inkling’s router uses a sigmoid to produce its expert scores [1].

• After scoring, the six highest-scoring experts run, and their outputs are combined using those same scores as weights. An expert scoring 0.9 contributes more to the result than one scoring 0.4.

This approach makes one specific type of failure more probable. To understand the failure, consider what happens across millions of training steps. Suppose, by luck, expert 47 gets picked slightly more often than average early on. It receives more tokens, so it receives more gradient updates, so it improves faster than its neighbours. Since it is better, the router scores it higher. Because it scores higher, it gets picked even more.

If we run that loop long enough, we can end up with a 256-expert layer where perhaps 20 experts handle nearly everything and the other 236 stay underdeveloped. This is called routing collapse. In other words, while we paid to store 256 experts, we got the capability of just 20. There is a second cost too. Since experts are usually spread across different machines, a machine holding four popular experts can become a bottleneck while its neighbours idle [9].

The traditional fix for this adds a penalty to the training objective that grows when expert usage is uneven. This means training the model on two things at once: predicting the next token correctly, and keeping expert usage balanced.

The trouble is that these two goals produce gradients pointing in different directions. The prediction gradient might say “increase this parameter,” while the balance penalty says “decrease it.” One of them wins, and either way the model is being pulled away from what you actually wanted. If we set the penalty strength high, the text quality degrades. If we set it too low, the experts collapse anyway [9].

Thinking Machines uses a method introduced by Wang and colleagues, which was also adopted by DeepSeek [1][8][9]. This method removes the conflict entirely by keeping a separate bias value for each expert, which is just a small number added to that expert’s score. The trick is where the bias gets applied.

Let us walk through a token arriving at a layer in this setup. Consider that the router produces these sigmoid scores for four of the experts.

Expert 47 has been overloaded recently, so its bias has drifted down to −0.15. This drops its selection score below expert 88’s, and expert 88 takes the slot instead. The bias changed which expert got picked. When expert 88’s output is combined into the final result, it is weighted by 0.80, its original router score, with the bias left out. In other words, the bias affects selection only. It never touches the weighting, and it is updated by a simple counting rule that runs outside backpropagation entirely. The main goal is to nudge down busy experts while giving a chance to the quieter experts.

The result is that expert usage stays balanced, and the training objective receives no competing gradient at all.

One point to note here is that two additional experts run on every single token regardless of routing [2]. These are called shared experts, and they hold the general-purpose processing that nearly every token needs. Thinking Machines states that the scores of the six selected routed experts and the two shared experts are normalised together before being used as weights [1], which means all eight are scaled to a common range and contribute proportionally. In other words, eight experts run per token in total.

Attention

Sparsity handles the cost of the feed-forward step. The attention step carries a separate cost that grows far faster, and Inkling manages it by having most layers examine very little.

For every token, the model computes three sets of numbers:

• A query, describing what kind of information this position needs.

• A key, describing what kind of information this position offers.

• A value, carrying the actual content this position passes along.

Every token’s query is compared against every earlier token’s key. Strong matches produce high scores, and those scores determine how much of each earlier token’s value gets pulled into the current position. This is how the word “it” in a sentence ends up connected to the noun it refers to.

Since every token compares itself against every earlier token, the number of comparisons grows with the square of the sequence length. For example:

• 1,000 tokens produce roughly 1 million comparisons.

• 10,000 tokens produce roughly 100 million.

• 1,000,000 tokens produce roughly 1 trillion.

Inkling supports a context window of one million tokens [2], and a trillion comparisons per layer across 66 layers is far beyond what any reasonable amount of hardware can deliver.

This is where a sliding-window layer restricts each token to a fixed number of recent tokens rather than everything before it. If the window is 1,000, then token number 500,000 compares itself against tokens 499,000 through 500,000 and stops there. That is 1,000 comparisons instead of 500,000, and the total for the whole sequence grows in a straight line rather than as a square.

This raises the obvious question that if every layer sees only a small window, how does information from page one of a long document ever reach page four hundred?

Inkling alternates between sliding-window layers and full-attention layers at a ratio of 5:1 [1]. Integration notes published by the vLLM project put concrete numbers on the split, describing the 66 layers as 55 sliding-window layers and 11 full-attention layers [7].

Information from far away travels through those eleven full layers. For example, picture a fact at token 200 that matters for a prediction at token 900,000. In layer 6, the first full-attention layer, the representation at position 900,000 can reach back and pick up that fact directly. From there, it rides forward within the local representations, is refreshed at layer 12, again at layer 18, and so on. The long-range path exists on roughly one layer in six, and the other five handle nearby context at a fraction of the price.

Inkling also uses 8 key-value heads [1]. Attention normally runs several times in parallel with different query, key, and value sets, and each parallel copy is called a head. Sharing a smaller number of key and value sets across many query heads reduces the memory needed while generating text.

See the diagram below that tries to show this setup in a simplistic manner:

There is one practical consequence of this approach. Long-context models often handle the general content of a large document well, but can still miss one specific detail buried in the middle.

Position

Dealing with a million tokens brings up another question: how does the model represent where each token sits in the sequence?

Thinking Machines picked an older technique over the current standard. This is due to the lengths the model never saw during training.

The comparison between queries and keys involves no information about order. For example, both “the cat bit the man” and “the man bit the cat” contain the same five tokens, so without a position signal, the attention step produces the same comparisons for both. Something has to tell the model that “cat” came before “bit” in one case and after it in the other.

Almost every recent open model uses Rotary Position Embedding, shortened to RoPE. Each token’s query and key are treated as points that get rotated by an angle proportional to that token’s position in the sequence. Token 1 gets a small rotation, token 500 gets a much larger one, and so on.

The usefulness of this shows up when two tokens are compared. Since both were rotated by amounts tied to their own positions, the comparison between them depends on the difference between those two rotations, which tells how far apart they are.

However, those rotation angles were only ever encountered at positions the model actually trained on. If training used sequences up to 32,000 tokens, then every angle the model learned to interpret came from that range. If we ask it about position 900,000, the angle involved falls outside anything it has experience with.

A whole family of techniques exists specifically to stretch RoPE into ranges beyond its training data.

Inkling uses a relative scheme in the style of Shaw and colleagues [1][10]. Rather than encoding where each token sits, this approach learns a value for each distance between two tokens and adds that value directly to the comparison score. For example, tokens at positions 5 and 9 are 4 apart. Tokens at positions 500,005 and 500,009 are also 4 apart. A relative scheme treats both pairs identically, because 4 is 4 wherever it occurs. Distances beyond some cutoff, say anything more than 128 apart, all share the same learned value, so a pair 900,000 tokens apart uses a value the model has seen countless times during training. Nothing has to be extrapolated.

The vLLM notes describe the implementation as a learned relative-position term added to the attention scores before they are converted into weights [7].

Thinking Machines states that this performed better and extrapolated better to longer sequences than RoPE in their testing [1].

Convolutions

Inkling adds one small operation named convolution to handle a job that attention would otherwise have to learn from scratch.

A convolution here means combining each position in the sequence with a few positions immediately before it. Inkling uses a window of four [7], so the numbers at position 100 get mixed with the numbers at positions 97, 98, and 99, using a small set of learned weights. Position 101 does the same with 98, 99, and 100. It is a cheap, fixed, strictly local operation.

To understand better, consider the two tokens that make up “New York.” The second token means something quite different on its own than it does following the first. Attention can learn to make that connection, and it has to learn it, because attention starts as a general comparison mechanism with no built-in preference for nearby tokens over distant ones. A convolution supplies that local mixing directly through its structure, without any training required to discover that neighbours matter.

Thinking Machines places these convolutions at two kinds of locations:

• First, on the keys and values inside each attention layer

• Second, on the outputs of the attention and feed-forward steps before those outputs rejoin the main path through the model [1].

The effect is that immediate context arrives pre-mixed, and attention can spend its capacity on the connections that genuinely require learning.

Multimodality

Everything so far concerns text moving through the model. Images and audio have to get in first, and Inkling accepts both without a separately trained encoder standing in front of it.

Most multimodal models attach three trained components to a language model:

• A vision encoder, trained on its own beforehand, converts an image into a list of numbers.

• An audio encoder does the same for sound.

• Projection layers then convert both into the format the language model works with.

However, in the case of Inkling, Sound arrives as a mel spectrogram, which is a standard way of representing audio as a grid of numbers. Frequency bands run down one axis, short slices of time run across the other, and each cell holds a loudness value. A three-second clip becomes a grid of a few hundred columns.

The dMel method then rounds each of those loudness values to one of a fixed set of levels, in the same way you might round 0.73 to 0.7 [1][11]. That is the entire conversion. No separate audio model needs training beforehand, because rounding numbers requires no training.

In the case of images, an image is cut into square patches measuring 40 by 40 pixels [1]. A 400 by 400 pixel image therefore becomes 100 patches. Each patch passes through a small four-stage network called an hMLP stem, which combines the pixels within that patch and processes each patch independently of every other one [1][12]. The paper describing this reports that it adds under one percent to compute compared with the simplest possible alternative [12].

Both then pass through a lightweight conversion layer and join the text tokens in a single sequence, processed by the same 66 layers we have been talking about [1]. Thinking Machines states that these multimodal components were trained from scratch on general-domain data [1], meaning they learned alongside the rest of the model rather than arriving pretrained.

See the diagram below:

One point of confusion can be that the launch announcement labels this as an encoder-free architecture [1], while the model card describes images as being encoded via a hierarchical patch encoder [2]. Both descriptions are accurate. Encoder-free here means there is no large, separately pretrained encoder network, rather than meaning there is no processing at all.

Moreover, this design predates Inkling. Thinking Machines described the same arrangement two months earlier for their real-time interaction system, using dMel for audio, 40 by 40 patches through an hMLP for images, and a lightweight conversion layer, with every component trained together from scratch [5].

Effort

How long the model reasons before answering is adjustable, and it was trained into the model rather than requested through wording. This impacts the model’s benchmark numbers

Effort is a number between 0 and 1. The documented presets run from 0 for none, through 0.1 for minimal and 0.2 for low, then 0.7 for medium, 0.9 as the default, and 0.99 at the top [6]. The spacing between those values is uneven, which is a hint that the number represents a learned response rather than a token allowance.

Mechanically, the setting arrives as text. Before the conversation begins, a system message stating the effort level is inserted ahead of everything else [6].

Reasoning models produce a stretch of working out before their final answer. This working-out phase costs tokens like everything else. Thinking Machines trained Inkling’s response to the effort setting during reinforcement learning, a training stage where the model produces complete attempts at tasks and receives a score for each attempt, then adjusts toward whatever scored well.

During that stage, Thinking Machines varied the effort message across attempts while also adjusting the cost charged per token generated:

• An attempt labelled high effort could produce lengthy working-out without much penalty.

• An attempt labelled low effort was charged heavily for every token, so short answers scored better.

Across many attempts, the connection between the message and the profitable length was learned.

Here are some key points about limits:

• Higher effort encourages more reasoning without guaranteeing a longer response or a better one on any single sample.

• Setting effort to 0 pushes the model toward minimal reasoning without enforcing it as a hard rule.

• Effort and the maximum token limit are separate settings that work independently, so a high effort setting may need a larger token limit to avoid being cut off.

Sweeping the effort setting across its range produces a curve of score against tokens generated. On the coding benchmark Terminal Bench 2.1, Inkling reaches the same score as NVIDIA’s Nemotron 3 Ultra while producing roughly a third as many tokens [1].

Tradeoffs

Each of the design decisions we have looked at has trade-offs. Let us look at some of them:

• Sparse routing keeps the per-token cost low and requires the entire model in memory regardless. The hardware floor stays high even though each token is cheap.

• Mostly-local attention makes a million-token window affordable and leaves most layers with a narrow view of the sequence.

• Relative position encoding removes the extrapolation problem and means every serving framework had to write new code for it, since the surrounding tooling was built around RoPE [7].

• Open weights covers the weights. The training data, the exact recipe, and the training code remain private, and the model card describes data provenance only in general terms [2]. The model can be modified, and reproducing or auditing it stays out of reach.

• Safety behaviour is adjustable by anyone who retrains the model. The model card recommends layering external moderation tools around the model rather than relying on its own refusals, particularly for consumer-facing deployments [2].

Thinking Machines states that other models available today, both open and closed, are stronger overall [1]. However, considering the availability for fine-tuning from day one, and the quantised checkpoint that brings the hardware requirements down, it is clear that Inkling has been built to be taken and adapted to the company’s use case.

Conclusion

Let us look at the key ideas we’ve encountered while understanding Inkling’s design and architectural decisions:

• Sparsity separates storage from compute. Total parameters tell you how much memory you need to hold the model. Active parameters tell you how much work each token requires. Inkling stores 975 billion and runs about 41 billion at a time.

• Routing needs balancing, and the balancing mechanism matters. Applying a bias when selecting experts, while leaving it out when weighting their outputs, keeps usage even without adding a competing goal to training.

• Long context works because most layers see little. Inkling runs 55 sliding-window layers and 11 full-attention layers, and long-range information travels through the eleven.

• Position encoding is a decision about untrained lengths. Encoding the distance between tokens, rather than their absolute positions, means a pair 900,000 apart uses a value the model has seen many times.

• Reasoning effort can be a trained setting. When it is, a benchmark score becomes one point on a curve rather than a fixed property of the model.

References

1. Inkling: Our Open-Weights Model, Thinking Machines Lab

2. Inkling Model Card, Thinking Machines Lab

3. The Future Worth Building Is Human, Thinking Machines Lab

4. Tinker, Thinking Machines Lab

5. Interaction Models: A Scalable Approach to Human-AI Collaboration, Thinking Machines Lab

6. Thinking effort, Tinker Documentation

7. thinkingmachines/Inkling, vLLM Recipes

8. DeepSeek-V3 Technical Report, DeepSeek-AI

9. Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts

10. Self-Attention with Relative Position Representations

11. dMel: Speech Tokenization made Simple

12. Three things everyone should know about Vision Transformers

Designed and assembled in America, Matic is the world’s first robot built to understand you. Its new feature, Matic Cues, lets you interact with it like you would anyone else.

• Point and speak. Say “Hey Matic, clean this” while pointing at a mess, and Matic knows exactly what to clean.

• Understands 70+ languages. Ask Matic to clean the kitchen, follow you, or go to the sink, in whatever language you speak.

• Skip the app. Anyone at home can use Matic, not just app-savvy adults, but kids and grandparents too.

Every home robot before this needed an interface. Matic just needs you to talk.
Try Matic, backed by their 6 month money back guarantee.

Get Matic

A vehicle travelling at 40 miles per hour covers about 60 feet every second. Within that second, software has to determine what is physically nearby, classify each object, estimate where those objects will move, select a path, and issue steering and braking commands.

Doing all of this quickly is something that has been largely solved. However, doing this correctly in unpredictable and distinct traffic situations is still an open problem. This is because real-world traffic produces more distinct scenarios than any team can pre-determine. For example, Waymo describes one such case, where a vehicle is on fire on the road ahead while the drivable lanes remain physically clear [3]. The geometry of such a scenario might permit driving straight through it. However, the real meaning of it calls for turning around or taking preventive action.

Waymo and Tesla, two companies investing in self-driving cars, have tried to come up with different types of answers to these questions.

For reference, Waymo reports 220.6 million rider-only miles through March 2026. These are miles covered with no human in the driver’s seat, across five metro areas [5]. On the other hand, Tesla reports more than three million vehicles in the United States covering over 30 billion miles a year, with 1.28 million active Full Self-Driving subscriptions in the first quarter of 2026 [10]. Almost all of those Tesla miles involve a driver who remains responsible for the vehicle. Tesla’s driverless service is separate and much smaller, running without safety monitors in Austin, Dallas, and Houston, while the Bay Area service uses a safety driver [10].

Both approaches depend heavily on machine learning. But they differ in how much gets fixed in advance. In this article, we will take a look at both approaches while trying to answer the following questions:

• How does each system detect what is physically nearby?

• What each builds from that data, and why one keeps the result readable?

• How each estimates what other road users will do?

• How a path gets selected, and what verifies it before the vehicle acts?

• What safety evidence each publishes, and why the figures measure different things?

• Where does the knowledge inside each system come from?

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

Sensing

A camera records light intensity across a grid of pixels. Distance appears nowhere in that grid, so depth has to be computed from the arrangement of pixels, but that computation can be wrong. A large object far away and a small object nearby can occupy the same region of an image.

Lidar arrives at the same answer by a different route. The unit emits laser pulses, measures how long each pulse takes to return after reflecting off a surface, and converts that interval into a distance. We can think of the output of this as a point cloud, which is a three-dimensional set of measured points describing the surfaces around the vehicle [1]. The distance is no longer an estimate but a measurement.

Waymo’s sixth-generation system, which began fully autonomous operations in February 2026, carries 13 cameras, four lidar units, six radar units, and a set of external audio receivers used to detect sirens and railroad crossings [2]. Coverage overlaps in every direction and extends to 500 metres. This overlap is for help situations when rain, road grime, or ice limits what a camera captures. Lidar and radar sustain the perception capabilities in such scenarios. [2].

The Waymo Driver utilizes a custom, multi-modal sensing suite where high-resolution cameras, advanced imaging radar, and LIDAR work as a unified system. Using these diverse inputs, the Waymo Driver can confidently navigate the “long tail” of one-in-a-million events regularly encountered when driving millions of miles a week. The goal is to leave nothing to the imagination of a single lens.

Tesla’s vehicles mainly rely on cameras. Instead, Tesla relies entirely on a “pure vision” approach that uses exterior cameras and artificial intelligence to navigate. For example, Tesla’s documentation describes Model 3 and Model Y as running camera-based Tesla Vision, without radar, using cameras and neural network processing [9].

See the diagram below:

There is another point here that requires our attention. Waymo’s fifth-generation Jaguar I-PACE vehicles carry 29 cameras [1]. The sixth-generation system carries 13, which Waymo attributes to a 17-megapixel imager covering the same area with fewer than half the cameras [2]. The sensor also captures millions of data points for sharp images while offering much better thermal stability. In other words, Waymo is also reducing overall sensor count while continuing to describe redundancy as essential.

The tradeoff here is that a direct measurement costs money while adding a component that can fail noticeably. However, a derived value may not cost much, but it can be wrong as well.

Representation

Something has to convert millions of pixels and points into a description that the system can understand and operate on. The nature and form of that description is one of the most important architectural decisions for an autonomous driving setup.

Waymo places the Waymo Foundation Model at the centre, built from two components:

• Sensor Fusion Encoder: This is used for rapid reactions. It merges camera, lidar, and radar data over time and outputs objects, semantic attributes, and embeddings. These are compact numerical summaries that downstream components consume.

• A Driving VLM: It is used for complex semantic reasoning. This component of the Foundation Model uses rich camera data and is fine-tuned on Waymo’s driving data and tasks. Trained using Gemini and fine-tuned on Waymo driving data, they cover rare situations requiring background world knowledge, such as the burning vehicle example mentioned earlier.

Both feed a Waymo World Decoder, which uses these inputs to forecasts the behaviour of other road users, produce high-definition maps, generate candidate trajectories, and emit signals used to verify them.

The system maintains compact structured representations, meaning explicit lists of objects, their semantic attributes, and roadgraph elements describing lanes and connections. The Waymo engineering team provides three reasons for such a setup:

• Correctness and safety validation can run at inference time, while the vehicle is moving

• Simulation runs efficiently at large scale, because a compact world state is cheap to replay and modify

• Training feedback becomes verifiable, since a component evaluating driving quality has something concrete to measure

This design provides significant benefits over pure end-to-end or modular approaches [3].

Tesla’s documentation talks about per-camera networks performing semantic segmentation, which assigns every pixel to a category, plus object detection and monocular depth estimation, meaning distance estimated from a single camera [6]. Those feed birds-eye-view networks that output road layout, static infrastructure, and three-dimensional objects in a top-down view. A full build involves 48 networks taking nearly 70,000 GPU hours to train and producing 1,000 distinct tensors per time step [6].

See the diagram below:

Prior mapping also occupies one end of this decision. For example, Waymo surveys a territory before operating there, recording lane markers, signs, curbs, and crosswalks. It then matches those maps against live real-time sensor data and AI to determine exact road location at all times, since GPS alone can lose signal [1]. The map is basically knowledge acquired once and reused. It saves computation on every trip, but also creates an obligation to keep it current. In contrast, Tesla’s approach skips the survey and derives equivalent information during the drive.

The trade-off is that a structured representation can be inspected, logged, replayed, and checked against explicit criteria. But it limits what the system can express. On the other hand, a learned representation carries nuance no schema can anticipate.

Prediction

Once the system holds a description of its surroundings, the next task is estimating what those objects will do. Several futures are valid at the same moment. For example, a cyclist approaching an intersection might continue straight, turn, or stop. A safe response should account for all possibilities.

Waymo describes the system as producing many possible paths for each road user rather than one, drawing on accumulated driving data and accounting for the different ways a car, a cyclist, and a pedestrian move [1].

In June 2025, Waymo published research on whether prediction quality scales predictably [4]. Using an internal dataset spanning 500,000 hours of driving, the study found that motion forecasting quality follows a power law in training compute, matching a pattern observed in language models. A power law here means each doubling of compute yields a proportional, predictable improvement. Data scaling proved critical, and increasing compute at inference time improved performance on harder scenarios.

Waymo reported the same trend in closed-loop performance, where results are measured in simulations in which the system’s own actions change what happens next. This trend suggested that real-world driving improves with more data and compute, rather than only benchmark scores.

Tesla’s describes an upgraded reinforcement learning stage in version 14.3, intended to cover long-tail edge cases, meaning rare situations that appear infrequently even across very large mileage [10].

See the diagram below:

A system that commits to one predicted future becomes fragile in situations where prediction matters most. However, carrying several weighted futures costs compute on every cycle, but can guard against the possibility of unusual situations.

Planning

With a description of the surroundings and a set of likely futures, the system selects a trajectory. You can think of it as a specific path with speeds attached to it. The question is what verifies the trajectory before the vehicle executes it.

Waymo trains large Teacher models to generate safe, comfortable, and compliant action sequences. It then distils them into smaller Student models sized to run onboard in real time [3]. Distillation transfers behaviour from a large model to a compact one. Output from that Student model then passes through a separate onboard validation layer, which verifies the trajectories the generative model produced [3]. This means that two independent components have to agree before the vehicle moves.

Tesla talks about building a planning and decision-making system that operates under uncertainty, with algorithms evaluated at the scale of the entire fleet, optimising for throughput, latency, correctness and determinism [6].

For Tesla vehicles on the road today, verification comes from a person. Full Self-Driving (Supervised) requires an attentive driver and leaves the vehicle slightly short of autonomous [7]. The system enforces this through a strikeout mechanism, where repeated inattention warnings disengage the feature for the remainder of a trip. Enough strikeouts suspend access for a week [7]. In the driverless service, that role belongs to safety monitors or remote supervision [10].

A validation layer catches a category of unacceptable outputs before they reach the actuators, and it can only evaluate against the defined criteria. Anything outside those criteria passes through unexamined. This is the same tradeoff as an assertion in production code, where the check is only as good as the condition behind it.

See the diagram below:

Validation

Waymo’s Safety Impact hub reports 220.6 million rider-only miles through March 2026, meaning no human occupied the driver’s seat for any of them [5]. Measured against human crash rates in the same operating areas, adjusted for where within each city the service drives, the reported reductions are 94% for serious injury or worse crashes and 82% for injury-causing crashes [5]. The methodology has been published in peer-reviewed journals, and the raw data is downloadable so third parties can reproduce the figures [5].

Tesla’s Vehicle Safety Report takes a different form. It compares Teslas with Full Self-Driving (Supervised) engaged against Teslas driven manually, using the same telemetry pipeline for both, and reports 7 times fewer major and minor collisions and 5 times fewer off-highway collisions [7][8]. A collision counts as occurring with the system engaged if it was active at any point within five seconds beforehand, a window chosen to capture cases where a driver took over shortly before impact [8]. Tesla attributes no fault in the reported data, treating that determination as too subjective to include [8].

As you can see, the two reports answer different questions, and we can’t simply do a side-by-side comparison. The difference comes down to what each population represents:

• Waymo is measuring outcomes across miles where no person was available to intervene.

• Tesla is measuring whether an assistance system reduces collisions while a driver remains responsible.

Both companies also state their own limitations. Waymo states that no perfect comparison between autonomous and human data exists today, and that its operating cities see no appreciable snowfall [5]. Tesla states that its estimate of a United States average involves unavoidable assumptions that may skew the figure in either direction [8].

Waymo separates two things that are easy to conflate. Safety impact gets measured after deployment. Whether a release is acceptable to deploy at all gets determined beforehand through a Safety Framework and a Safety Case [5].

Training

Both systems improve between releases. However, the underlying mechanisms differ as much as the architectures.

Waymo runs three components off the same foundation model:

• The Driver produces action sequences.

• The Simulator generates scenarios for training and testing.

• The Critic evaluates driving quality and surfaces problems.

Large versions of each get distilled into smaller ones that run at the required volume. Two loops connect them:

• An inner loop applies reinforcement learning inside simulation, where scenarios can be generated and repeated cheaply.

• An outer loop begins with the Critic flagging suboptimal behaviour from real driving, turns improved alternatives into training data, verifies the fixes in simulation, and deploys only once the safety framework confirms the absence of unreasonable risk.

Waymo states that its fully autonomous mileage now far exceeds its manually driven data, and that no volume of simulation or test-driver operation reproduces the situations encountered when the system operates with no driver present [3].

Tesla’s data comes from a consumer fleet. The Vehicle Safety Report describes two telemetry paths [8]. On shifting to park, a vehicle transmits anonymised mileage broken down by control type and road classification. On detecting a major or minor collision, it transmits a separate packet tied to the vehicle. Tesla reports receiving 2.5 billion telemetry packages in the third quarter of 2025 alone. Tesla has also built an evaluation infrastructure from anonymised fleet clips assembled into test suites, alongside simulation producing sensor data for automated testing [6]. Training runs on Cortex 1, listed at over 100,000 H100-equivalent GPUs in production, and Cortex 2 at over 130,000 in early ramp.

See the diagram below:

Conclusion

The same question recurs at every stage of an autonomous driving process.

How much gets determined in advance and written into a form that can be examined, and how much gets computed during the drive by a model whose internal state stays out of reach?

Each stage poses one version of it:

• Sensing asks whether distance arrives as a measurement or as a derived value

• Representation asks whether the description of the world stays inspectable

• Prediction asks how many futures the system carries at once

• Planning asks what verifies a trajectory before execution

• Validation asks what kind of safety claim the resulting evidence can support

• Training asks which miles improve the system

Waymo sits further toward written-down knowledge at most stages. This involves per-city preparation and purpose-built hardware. Tesla sits further toward computed knowledge. Both positions have merit, and only the future will tell which approach becomes more dominant or do things fall somewhere in the middle.

References

1. Self-Driving Car Technology for a Reliable Ride, Waymo

2. Beginning fully autonomous operations with the 6th-generation Waymo Driver, Waymo, February 2026

3. Demonstrably Safe AI For Autonomous Driving, Waymo, December 2025

4. New Insights for Scaling Laws in Autonomous Driving, Waymo, June 2025

5. Waymo Safety Impact, Waymo

6. AI and Robotics, Tesla

7. Full Self-Driving (Supervised), Tesla Support

8. Full Self-Driving (Supervised) Vehicle Safety Report, Tesla

9. Autopilot and Full Self-Driving Capability, Tesla Support

10. Tesla Q1 2026 Update, Tesla Investor Relations

This isn’t a conference you sit through. AWS re:Invent includes 2,200+ sessions, and 70% are interactive; workshops, code talks, AWS Jams, GameDays, and the Architecture Rodeo, built for engineers who’d rather work through a problem than watch a slidedeck.

Between November 30 and December 4, you’ll have the opportunity to:

• Run demos against real workloads with sandbox access in small groups

• Test newly launched AWS services in the AWS Village before your team adopts them

• Work through architecture decisions directly with service engineers

Save $1,200 with early bird pricing

This week’s system design refresher:

• HTTP vs HTTPS Explained (Youtube video)

• What is Google’s TPU?

• 9 Types of API Testing

• Common types of AI Agents guardrails on production

• Forward Proxy, Reverse Proxy, and API Gateway Explained

HTTP vs HTTPS Explained

What is Google’s TPU?

A TPU (Tensor Processing Unit) is Google’s custom AI chip, designed from scratch for the giant matrix multiplications that modern models live on. GPUs were built for graphics first.

TPUs were built for deep learning from day one.

At Cloud Next ’26, Google unveiled its 8th generation, and for the first time it ships in two flavors. TPU 8t is built for training, where raw throughput wins. TPU 8i is built for inference, where latency and chip-to-chip speed matter most.

Both still share the same Axion CPUs, liquid cooling, and software stack, so code written for one runs on the other.

The diagram is a quick study guide to what’s the same, what’s different, and why, based on our understanding of published Google articles.

What Makes an FDE Role Credible? (Sponsored)

Strong candidates are skeptical of vague forward deployed engineer postings, and the title alone won’t earn their trust.

The free State of FDE Jobs 2026 Report explains what candidates look for, how the market is evolving, and how employers can make these roles easier to understand.

Hiring? You can also bring your openings to forwarddeployedengineer.com, the focused jobs board for forward-deployed engineers.

Explore the Job Board

9 Types of API Testing

Here are the 9 most common API tests used to catch different kinds of failures.

• Smoke tests run right after a deploy to confirm the critical endpoints still respond, especially things like login, checkout, or health checks.

• Functional testing checks whether an endpoint does what the business expects or just returns a clean 200 with the wrong number inside.

• Contract testing protects the agreement between services. If a consumer depends on a field, a type, or a status code, the provider should not change it without warning.

• Integration testing covers the full workflow across systems. An order endpoint may also use inventory, payment, and notifications, and a- ny of those dependencies can fail.

• Regression testing protects existing behavior when new changes are added. A change in discount logic should not break checkout totals or order history.

• Load testing asks how things hold up under expected traffic.

• Stress testing keeps pushing until something breaks.

• Security testing covers auth, access control, unsanitized input, and what your errors leak.

• Fuzz testing sends unexpected or invalid inputs to the API to find bugs that normal test cases may miss.

Over to you: Which of these do you rely on most in practice?

Common types of AI Agents guardrails on production

• Input Screening: Every user request is scanned for prompt injection, sensitive data, and off-scope topics before the model sees it. Unsafe inputs return a fallback message.

• Context Verification: Every user request is scanned for prompt injection, sensitive data, and off-scope topics before the model sees it. Unsafe inputs return a fallback message.

• Response Generation: The LLM reasons only over verified context.

• Output Validation: Every response is checked for groundedness, format, and safety before it leaves the system. Failures trigger up to two retries, then a safe fallback.

• Operational Controls: A final layer enforces limits, logs every call, and routes low-confidence or high-risk actions to humans.

Reliable agents aren't built on better prompts. They're built on the guardrails wrapped around them.

Over to you: Which guardrail layer catches the most issues in your setup?

Forward Proxy, Reverse Proxy, and API Gateway Explained

People mix these up all the time, since they all sit between a client and a server. The real difference is which side they represent and what problem they solve.

A forward proxy sits next to the client. Your laptop sends a request, the proxy forwards it out, and the destination never sees your real IP. Corporate networks use this to enforce policy, block sites, and cache traffic.

A reverse proxy sits next to the server. The client has no idea how many machines are behind it. The proxy decides who handles the request, terminates TLS, and keeps your backend off the public internet. NGINX and HAProxy are commonly used here, typically paired with a load balancer in front.

An API gateway is a reverse proxy that does more than route traffic. It also handles auth, rate limits, API keys, versioning, and request shaping. Without it, each microservice has to implement its own version of validation, throttling logic, and request logging.

A forward proxy represents the client, a reverse proxy represents the server, and an API gateway is what you add when ten services need the same authentication and rate limiting rules applied consistently.

In most real systems, all three are running at different layers. The forward proxy filters outbound traffic, the reverse proxy fronts the application servers, and the API gateway sits in front of your APIs to enforce policies before requests reach them.

Over to you: What's your proxy + gateway combo? Always interesting to see what teams pair together.

The code that performs this merging can run in several places. It can run inside the mobile application, on a server in the datacenter, at a CDN edge location, or inside one of the four services. Putting a server between the mobile application and the four services adds a network hop, which sounds like it should cost extra time. However, it usually reduces total load time instead, because a round trip between a phone and a server on a weak mobile connection can take a few hundred milliseconds, while a round trip between two services inside the same datacenter takes a fraction of a millisecond. To put it simply, trading four expensive round trips for one expensive round trip plus four cheap ones is often a large net saving.

However, latency is only the first tradeoff. The place where the merge operation runs also determines what happens when one of the four services is unavailable, how much of the response can be cached, and which team has to approve a change before the screen ships.

In this article, we are going to dive deep into the area of the API composition problem and the patterns associated with it. Here’s what we will cover:

• The API composition problem

• Client-side composition

• Over-fetching and under-fetching

• Composition, aggregation, and orchestration

• API gateways

• Backends for Frontends

• GraphQL as a composition layer

• Edge composition

• Availability and caching tradeoffs

• Versioning across multiple frontends

• Ownership of the composition layer

The API Composition Problem

Read more

GLM-5.2 is Z.ai’s flagship long-horizon coding model, featuring a usable 1M-token context window that reliably maintains API contracts and engineering intent where others fail. Stop managing GPU clusters. With Crusoe Serverless Fine-Tuning, you customize GLM-5.2 on your proprietary data in a tenant-isolated environment—no data sharing, no infrastructure overhead. When your job finishes, deploy to Self-Serve Deployments in one click or download raw .safetensors weights for full portability. Every run is reproducible, auditable, and fully yours. No lock-in. No guesswork. Experience production-grade, customized performance without the DevOps burden.

Get started

AI models have solved the writing code part of software development to a great extent. Today, a capable model can produce a working function, a full component, or a small application from a plain-language description. It can do so in seconds for a fraction of the cost.

This change has shifted the economics of every developer platform. As the generation of new code becomes cheaper and widely available, it stops being the differentiating factor for a platform. This is the reason GitHub, Vercel, and Replit are trying to rebuild themselves around solving other hard problems in the software development process.

To understand what the three companies are doing, we trace one unit of work through every platform while asking the same questions:

• Where does the AI actually run the code it writes?

• How does each platform verify that the code works?

• How does the finished product reach production, and who is permitted to ship it?

Here’s what we will cover:

• Why does cheap code generation move the hard engineering problem to another place, and how does that impact the platform’s overall value?

• GitHub’s orchestration bet, built on ephemeral cloud environments and a control layer that routes work across competing agents.

• Vercel’s production bet, built on isolated microVM sandboxes and a billing model matched to how agents actually run

• Replit’s verification angle, built on a self-testing loop that drives a real browser to catch code that only looks like it works

• How the MCP standard lets any agent reach any tool, and why all three companies now support it

• How Stripe turned payments into something an agent can set up from inside a coding tool, and the credential design that keeps it safe

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

Commoditization

A language model can now turn a description into working code. This means a developer can write a sentence and receive a function, a page, or a small application that actually runs right out of the gate.

This capability used to be the scarce and valuable part of a developer tool. Today it is widely available and close to free, which leaves a platform that offers only generation with little to charge for.

So what matters now is not raw code generation, but aspects of software development that come up after the code is available. Three questions carry most of the weight now:

• Where does the agent run the code it produces, and how is that environment kept safe?

• How does the platform confirm that the generated code actually works?

• How does the result reach production, and who is allowed to ship it?

GitHub, Vercel, and Replit answer these questions in different ways.

• GitHub puts its effort into coordination

• Vercel into the path to production

• Replit into verification

GitHub gives us the clearest place to start, because it deals with the pull request workflow most developers already use.

[Webinar] Can you prove AI is working? (Sponsored)

AI is in your engineering workflow. While the token spend shows it, the throughput doesn’t. The human is very much still in the loop, and that’s a context problem.

Join live on Aug 19 (FREE) to learn:

• The 4 metrics to measure where AI gains leak out before production.

• The 8 stages of context maturity, the specific walls capping your metrics, and a free tool to pinpoint where your team is

• Why more MCPs and bigger context windows aren’t enough, and what it takes to get real value from your agents.

Register now

Orchestration

GitHub made a specific choice about where the value sits.

Rather than building its own model and competing on raw generation, it built a control layer that coordinates many agents and keeps their work governed, all inside the pull request workflow that developers use every day.

The mechanics start with where the code runs.

GitHub’s coding agent operates in its own ephemeral development environment, which is a temporary workspace that exists only for that task. It is powered by GitHub Actions, the same automation system that runs tests and builds on the platform [1][2]. In practice, you assign a task to the agent the way you would open a ticket. The agent reads through the repository, edits files, runs the tests and linters (tools that check code for problems), and opens a draft pull request for a person to review [1]. Each cloud run happens in a fully isolated, single-use Linux environment hosted by GitHub. Every new task starts from a clean workspace [4]. The review step stays human, which keeps the existing quality gate intact.

See the diagram below:

A single agent sits above the coordination layer. GitHub calls it Agent HQ. It introduces a mission control view that lets a developer assign, steer, and approve work across a fleet of agents from GitHub and VS Code [3]. The agents available inside a paid Copilot subscription include ones from Anthropic, OpenAI, Google, Cognition, and xAI [3].

Governance is treated as version-controlled configuration. Teams define custom agents through AGENTS.md files that carry rules such as a preferred logger or a required testing style, and a control plane gives administrators security policies, audit logging, and model-access controls in one place [3].

See the diagram below:

For developers who are already accustomed to handling issues and pull requests, this design adds agents to a pretty familiar workflow. Reviewing a colleague’s branch is not so different from reviewing an agent’s branch.

Routing to other companies’ models is a deliberate decision. GitHub is positioning the workflow, the execution environment, and the governance layer as the durable product, and treating the underlying model as a swappable component.

Let us now look at how Vercel developed its architecture around what happens after the code exists. This is somewhere between a working demo and production-ready software.

Production

Vercel starts from a different premise where code generation is assumed, and the design deals with carrying the generated code into production. This is the stage where most enterprise software work actually happens, because it involves existing applications rather than fresh prototypes.

The rebuilt version of v0, Vercel’s generation product, runs on a sandbox. It is an isolated space for executing code that imports a real GitHub repository and automatically pulls in the project’s environment variables and configuration. Every prompt produces code that fits the actual application and lives in the repository itself [5].

A Git panel handles the workflow around it. You create a branch for each chat, open a pull request against the main branch, and deploy when it merges. This means a product manager or a designer can ship through the same review process an engineer uses [5].

Vercel’s rationale about this approach is that AI-assisted building is already happening inside companies, and it has produced real failures. Incidents have been reported, such as credentials pasted into prompts, private data reaching the public internet, and deleted databases, often with the audit trail left empty [5]. Therefore, wrapping code generation in real deployment controls is the right response.

See the diagram below:

Underneath the workflow is the execution layer. Every sandbox runs inside a Firecracker microVM, a lightweight virtual machine that isolates untrusted code. The reason for this isolation is that the code an AI wrote is code you have yet to review. Therefore, running it needs a boundary strong enough to contain mistakes. A microVM provides that strong boundary.

The billing model depends on how agents actually run. Vercel’s Fluid compute lets several requests share one running instance, with one using the processor while another waits on input or output. It charges for active processor time while treating wait time as free [6][7]. Agentic workloads spend much of their time waiting on a model to respond, so this pricing matches the real work being done.

This approach deals with a frustration many developers face in their careers. The thing that worked in a demo behaves differently in production. However, Vercel’s design tries to close that gap by making the preview a real deployment from the start.

Nevertheless, strong isolation carries a cost per unit of compute. There is an open question about whether the heaviest workloads eventually move to cheaper execution options elsewhere. Vercel’s wager is that a single, convenient platform keeps them in place.

To summarize, GitHub owns the workflow, and Vercel owns the route to production. Both of them still depend on one assumption that Replit chose to attack directly. Let us look at that now.

Verification

Replit concentrated its work on whether autonomously generated code genuinely works using a verification loop built into the agent itself.

Replit’s Agent 3 runs what the company calls a reflection loop. The agent generates code, runs it, tests the result, and repairs failures, repeating that cycle until the tests pass [8]. This loop is reliable because of how the testing is done. Replit built a REPL-based verification system that runs code immediately and pairs that execution with a real browser it drives automatically, so it can click buttons, submit forms, and check data the way a user would [9].

The specific problem this approach targets has a memorable name inside Replit: the Potemkin interface. It is basically a feature that looks complete on screen yet fails the moment it is used [9]. Catching that class of error is what allows the agent to run on its own for more than 200 minutes at a stretch, a large increase over the roughly 20 minutes of its predecessor [8][9].

The verification runs as its own process.

A testing subagent follows a simple cycle of taking an action, observing the result, and repeating. When it finishes, it returns a summary to the main agent describing what works and what broke [9]. This multi-hundred-step testing costs a median of roughly twenty cents per session and runs several times faster and more cheaply than relying on general-purpose computer-use models [9].

See the diagram below:

Anyone who has shipped a feature that looked finished and later broke understands the problem here. Replit is trying to automate the check that catches exactly that, so that the automation can be trusted.

A verification loop reduces the risk substantially. Some failures still surface only under conditions a single test session might miss, which keeps this a pretty hard problem even with a capable tester in place.

Interoperability

Every architecture we have looked at assumes its agent can reach tools and data that live outside the model, and doing that cleanly requires a common method. That method is the Model Context Protocol, usually shortened to MCP.

Before a standard like this existed, connecting several AI applications to several external tools meant writing a separate custom integration for each pairing. In this approach, the number of integrations grew quickly as both sides multiplied. Anthropic introduced MCP to replace those fragmented, one-off connections with a single protocol, so each application and each tool implements the standard once and then works with everything else without additional changes [10].

A host, which is the AI application such as an IDE or a chat client, creates one or more clients, and each client connects to a server that exposes some capability [11]. A server offers three kinds of capability:

• Tools, which the model can call to take an action, such as creating a record or running a query.

• Resources, which supply context data the model can read, such as a file or a database schema.

• Prompts, which provide reusable instruction templates.

The whole exchange runs over a defined message format across either a local or a remote connection. The effect is that a tool provider builds one MCP server and every compliant agent can use it [11].

Replit was among the earliest developer tools to integrate MCP [10], GitHub added an MCP registry to VS Code where a server can be enabled with a single click [3], and Stripe runs an official MCP server for its payment operations [12].

Going deeper, MCP also standardizes how agents reach existing APIs rather than replacing those APIs. A single shared entry point also concentrates security in one place. That makes careful control over which servers an agent may use an important part of any real deployment.

Tradeoffs

Each of these architectures and approaches has trade-offs:

• GitHub gains breadth and governance by routing to many vendors’ models. But the cost is that it owns the surface rather than the intelligence underneath. Whether a coordination and governance layer stays valuable as models and agents keep changing is an open question.

• Vercel gains strong isolation by running generated code inside microVMs, and that isolation carries a cost per unit of compute. There is a question about whether the heaviest workloads eventually move to cheaper execution elsewhere

• Replit gains long stretches of autonomy through its verification loop, and the more work an agent does on its own, the more weight rests on that verification being right. The Potemkin problem (code that looks complete yet fails when used) stays difficult even with a capable tester, because some failures appear only in situations a test session might miss.

MCP gains a clean, reusable way to connect agents and tools, and a shared standard also concentrates risk into a common entry point. When many agents reach many tools through one protocol, controlling which servers an agent may use, and their permission levels, becomes a central problem rather than a detail.

Understanding these costs is what separates picking one of these tools from understanding why it was built the way it was.

Conclusion

The pattern across all three companies is the same. Code generation became cheap, so the value moved into the engineering that surrounds it. Each company placed its bet on a different piece of that surrounding work.

GitHub bet on orchestration, building a control layer that runs and governs many agents inside the pull request workflow developers already use.

Vercel bet on production, wrapping generated code in real deployments and running it inside isolated microVMs built for untrusted code.

Lastly, Replit bet on verification, driving a real browser in a self-testing loop so an agent can work on its own for hours and still be checked.

Underneath all three, MCP provides the common protocol that lets any agent reach any tool, which is why every one of these platforms now supports it.

References:

1. About GitHub Copilot cloud agent

2. GitHub Copilot coding agent 101: Getting started with agentic workflows on GitHub

3. Introducing Agent HQ: Any agent, any way you work

4. GitHub Copilot app: The agent-native desktop experience

5. Introducing the new v0

6. The AI Cloud: A unified platform for AI workloads

7. Vercel Sandbox

8. Introducing Agent 3: Our Most Autonomous Agent Yet

9. Enabling Agent 3 to Self-Test at Scale with REPL-Based Verification

10. Introducing the Model Context Protocol

11. Architecture overview — Model Context Protocol

12. Agents and AI on Stripe

13. Create claimable sandboxes

Free masterclass: Learn practical strategies for predictable low-latency writes at scale

Free Masterclass: Optimizing Write-Intensive Database Performance

Database writes – at scale – are one of the hardest problems in distributed systems. This masterclass will teach you how to understand and avoid latency spikes in real-time, write-heavy database workloads. Our panel of experts will share a practical framework for diagnosing write bottlenecks and knowing which strategies to apply in different scenarios.

After this free 2-hour masterclass designed for developers, engineers, architects, and database practitioners, you will know how to:

• Identify which factors (database internals, database configurations, data modeling…) are impacting your write performance

• Avoid mistakes that have caused 40x write amplification in production

• Sustain low P99 latencies even during sustained growth and volatile spikes

All attendees will get the complete Database Performance at Scale book by the masterclass instructor Felipe Mendes.

Register for Free

How does a website charge a visitor who arrives anonymously, skips every advertisement, and leaves within a second?

For most of the web’s history, the question rarely came up, because the visitor was a person whose attention a site could sell through an ad or a subscription. If the website had good content, the website owner could count on multiple such visits by the same user. But now the visitor being a person is not always true. More than half of the traffic online now comes from software that acts on a person’s behalf, requests a page, and leaves without engaging with any ads or subscriptions [2].

Cloudflare seeks to change this.

As you might be aware, Cloudflare sits between a large share of the world’s websites and everything requesting them. It works as a reverse proxy that each request passes through before it reaches the origin server [1]. This lets Cloudflare read a request and act on it early. Over the past year, the company has used this position to sort automated traffic by what it does, to verify the identity behind a request, and, most recently, to collect payment for a request through an open protocol named x402 [1].

In this article, we will go through Cloudflare’s solution in the following five steps:

• The web’s usual way of earning value leans on human attention, and agent traffic changes it fundamentally.

• A look at Cloudflare’s initial solution around blocking automated traffic and then charging for each crawl.

• Settling identity, permission, and payment inside a single request.

• A look at how x402 exchange completes this settlement.

• The x402 exchange completes this settlement through a short back-and-forth over ordinary HTTP

• The costs and open questions of this approach

Disclaimer: This post is based on publicly shared details from Cloudflare. References at the end. Please comment if you notice any inaccuracies.

The Attention Model

For most of its history, websites made money after a request rather than during it. A browser asked for a page, and the server returned it at no charge. The value arrived later, once a person saw an advertisement, bought a subscription, or came back for another visit [1]. The request itself stayed free, and this setup funded a large part of the Internet.

A growth in agent traffic is changing this dramatically. An agent is software that acts on a person’s behalf, which in practice means it requests a page or a data feed once, takes what it needs, and finishes in a single pass. It moves past advertisements, operates outside any subscription, and completes its task before a site has a chance to earn from that attention [1]. Each of the three old settlement points depended on a person staying long enough to be counted, so software traffic leaves those points idle.

The strain here is that this kind of traffic is now fast becoming the majority. More than half of the requests reaching websites come from software rather than people [2]. This means that request volume climbs while revenue stays flat.

To make things clear, parts of the Internet were already charging by usage before any of this. Cloud services and APIs have been sold by the call and by the hour for years, though only to a buyer the seller already knew, who signed up and received an API key [1]. Charging an anonymous caller a fraction of a cent for a single request stayed impractical, because collecting such a small payment once cost more than the payment returned [1].

If the value used to settle downstream of the request, the natural question is which party is positioned to move that settlement back onto the request. The answer starts with where Cloudflare sits.

The Proxy Layer

A reverse proxy is a server that stands in front of other servers and receives requests on their behalf. Since Cloudflare operates as a reverse proxy for a large portion of the web, a request headed for one of those sites reaches Cloudflare’s network first and passes through it on the way to the origin [1]. The origin is the site’s own server, the machine that ultimately holds the page or runs the API.

On a side note, the position of a proxy is more general-purpose than caching. Caching stores a copy of a response so it can be served quickly the next time, and it is just one of the many useful things a proxy can do. From the same middle position, a request can also be read, classified, checked, and acted on before it continues to the origin. This broader capability is quite consequential at Cloudflare’s scale. [2].

However, acting on a request early depends on first knowing what the request is for. This is a classification problem. Let us see how Cloudflare handles this.

Traffic Classification

Before any rule can apply to an agent, the traffic has to be sorted by what it does. Cloudflare’s taxonomy groups automated traffic by behavior rather than by the single label “AI”. Three behaviors impact the key policy decisions [4].

• Search covers behavior that builds an index of a site so an engine can answer questions about it later. This behavior is responsible for sending referral visitors back.

• An agent covers behavior that acts in real time on a person’s behalf, usually with a human waiting for the result. Think of it like an assistant fetching a page during a conversation.

• Training covers behavior that takes content to train or fine-tune a model, where the content is absorbed into the model rather than pointing a visitor back.

These three appear identical in a raw request log, yet they carry very different consequences for a site’s business.

A single crawler can also perform more than one of these behaviors, but this separation lets them record all of them[4]. A crawler that both builds a search index and gathers training data is tracked as doing both, which lets a site owner reason about the full set of things that the crawler does on their pages.

Cloudflare classifies further behaviors as well, including checkout actions and data collection, and it also lets a site express what a bot may store and reshare after accessing a page [4].

With a way to see the proxy position and a way to classify what arrives, let us now look at Cloudflare’s first attempts at implementing a policy.

Blocking and Charging

Cloudflare’s first solution was a simple one. A single control lets a site block automated AI traffic outright. This protected the content and left the earning model untouched. Blocking answers the question of access, yet it does not rebuild the revenue that the old bargain provided.

The second solution added a middle path.

Pay Per Crawl lets a website allow a crawler, block it, or charge it a flat per-request price, with Cloudflare handling the billing as the merchant of record [5]. A site owner could keep a crawler out, let it through, or attach a price to its access, all from one setting. This turned the binary of block-or-allow into three options and gave content owners a way to earn from crawler access at network scale [5].

A year on, Cloudflare made an adjustment to its own model. The company argued that a crawl is a weak measure of value, because a single page might be crawled once and then cited in thousands of AI answers, or crawled repeatedly and cited in none [3]. It backed the argument with a figure from its own network, that more than half of the crawl traffic from well-behaved bots goes to re-fetching pages that have stayed the same since the last visit [3]. Counting crawls, then, counts something that only loosely tracks the value delivered.

So the unit of payment began to move from the crawl toward the use, an approach Cloudflare describes as Pay Per Use. It is candidly framed as an experiment at this point [3]. This is because while pricing the outcome aligns payment with value more closely than pricing the fetch, it is also harder to measure.

The Request Layer

The design goal for recent changes is to resolve identity, permission, and payment inside a single request, at the edge, before the origin responds.

Three main concerns are considered over here:

• Identity: The traditional identifier, the User-Agent string, can be set to any value by the caller, so it offers weak assurance. Cloudflare’s answer is Web Bot Auth, an authentication method that uses cryptographic signatures in HTTP messages to verify that a request comes from a particular automated source [7]. In practice, the operator signs its request with a private key and publishes the matching public key at a known location, and Cloudflare validates the signature at the edge [8]. A valid signature stands in for a reliable identity, which replaces a guess with a verifiable claim.

• Permission: This is expressed through the behavior classification already covered and through the preferences a site sets about how its content may be used [4].

• Payment: This comes last and is attached to the request through x402. We will cover this in detail in the next section.

All three concerns resolve at the edge, so the origin receives a request only once identity, permission, and payment have been settled [1]. The metering and settlement are taken away from the website’s own servers. What stays with the site owner is the part that matters to them, which is their rules, their prices, and their revenue [1].

To summarize:

• Identity answers who is making the request, through a signed and verified claim.

• Permission answers whether this behavior is allowed on these pages, through the classification and content preferences.

• Payment answers whether the caller has paid the stated price through the x402 exchange.

These pieces sit at different stages of maturity. Identity verification through Web Bot Auth is available today at the edge [7], while the Monetization Gateway that brings the payment piece together is open as a waitlist rather than a shipped product [1].

The key takeaway is that when one component sits in the middle of a flow, the concerns that are common across every request are collected at that point. Authentication, authorization, and billing consolidating at a gateway is the same pattern that appears in service meshes and middleware.

The x402 Exchange

The x402 protocol makes it possible to pay over HTTP. It takes its name from a status code that has been part of the HTTP standard for a long time [1]. The code is 402, and it means Payment Required. Sites behind Cloudflare already send more than a billion of these responses on an average day, which shows how often a machine requests something priced and receives a message that a payment is due [6].

See the diagram below that shows the overall setup:

The exchange runs through a short sequence that can be compared to a small state machine [1].

• A client requests a resource that sits behind a price.

• Rather than returning the resource, the server responds with 402 and a small payload stating the price, the accepted asset, and where to pay.

• The client re-sends the same request with proof of payment attached.

• A facilitator verifies the payment, and the server returns the resource.

Two properties make this suitable for machine traffic. The payment amounts can be very small because the protocol adds almost no overhead to the request. And the payment itself serves as the credential, so a buyer with no prior relationship can access the content by showing the proof of payment [1]. This property is the one that matches an anonymous agent passing through once, since it removes the signup step that per-seat licensing and API keys always required.

The negotiation process is handled inside ordinary requests and responses, with a redirect to a checkout page absent and a separate payment API absent [1]. Nothing was added at the protocol level, since the 402 code has been part of HTTP for decades. What changed is that a settlement angle now exists that makes collecting a fraction of a cent practical.

Costs and Limits

Resolving identity, permission, and payment at the edge means one proxy performs those functions for a large share of the web at once [1]. Cloudflare presents that responsibility as an advantage for settling everything inside one request. But it can also be seen as a potential risk when so much runs through a single provider.

Cloudflare also states several limits plainly:

• Trust that travels with a request may reach only the traffic that can afford to be identifiable, and small or privacy-sensitive sources of traffic need other building blocks, such as private rate limiting [4].

• Usage-based payment helps a site that already has demand, and does little for a small site whose real difficulty is discoverability rather than monetization, which leaves that site weighing visibility against giving content away [4].

• Collecting a payment through this exchange depends on callers built to recognize and honor the 402 response. Therefore, the revenue depends on adoption within the ecosystem. [6].

• Pricing an outcome rather than a crawl aligns payment with value, but it is also harder to measure and verify. This is the reason Cloudflare frames the shift to Pay-Per-Use as an experiment [3].

Lastly, Cloudflare argues that crawlers combining several purposes under one identity reduce transparency for a site. This is because the site cannot tell why it is being accessed [4]. While the argument is sound based on technical merits, it also aligns with Cloudflare’s commercial interest in separated, verifiable traffic.

None of these costs undoes the shift. They simply mark the current trade-offs that website owners should consider before adopting it.

Conclusion

The web is moving value settlement from after the request to inside it. For most of the web’s history, a request was served free, and value was settled later through human attention. However, agent traffic, which now comprises the majority of requests, leaves that later settlement with nowhere to land [2].

Cloudflare’s response is to resolve four things from its position as a reverse proxy: seeing what a request is through classification, verifying who sent it through Web Bot Auth, enforcing the site’s rules, and settling payment through the x402 exchange, all before the origin responds [1][4][7].

Identity verification lives at the edge today, and the payment gateway opens as a waitlist [1][7]. The open questions around concentration, reach, adoption, and how to price an outcome are worth tracking as the model develops.

References:

1. Announcing the Monetization Gateway: charge for any resource behind Cloudflare via x402

2. Content Independence Day, one year on: building the business model for the agentic Internet

3. Making AI search smarter

4. Your site, your rules: new AI traffic options for all customers

5. Introducing pay per crawl: Enabling content owners to charge AI crawlers for access

6. Launching the x402 Foundation with Coinbase, and support for x402 transactions

7. Web Bot Auth

8. Forget IPs: using cryptography to verify bot and agent traffic