Match or beat harnesses on accuracy with 28% fewer tokens. Strands harness is open source and ships with benchmarked defaults for the system prompt, tools, memory, sessions, and context management, so you can skip wiring up a harness piece by piece.

Spin it up from the CLI or in a few lines of Python or TypeScript. Nothing is a black box: swap model providers, override any default, or drop down to the SDK primitives with no rewrite. Then validate before you ship with 25+ built-in evaluators.

From idea to production with fewer tokens wasted.

Get started

This week’s system design refresher:

• 6 frequently used orchestration frameworks in an AI system

• Top 6 Cost Optimization Strategies in AI Systems

• How a SQL Query Executes Inside a Database

• How does a request actually travel through Claude Code?

6 frequently used orchestration frameworks in an AI system

Orchestration in an AI system manages tasks across agents, handles failures, shared memory and context.

1. LangGraph: This uses a graph-based technique where workflows are represented as nodes and edges determine the state transitions.

2. CrewAI: A role-based abstraction technique is used here. Agents are assigned a defined role and goal that enables them to work on the tasks sequentially or via a manager agent .

3. Microsoft Agent Framework: This framework is designed to handle organizational workflows. It also uses graphs with built-in branching, checkpoints, and human intervention pauses.

4. OpenAI Agents SDK: An LLM driven runtime approach is used, where an agent decides whether it should delegate control to other agent or retain execution to complete the task itself.

5. Google ADK: Hybrid control approach is used where a developer defines a strategy to run tasks and a root agent splits and distributes tasks to sub-agents.

6. Claude Agent SDK: No separate layer for orchestration exists. It lives inside an agent which gathers context, works with tools, evaluates and keeps going until no calls left.

Top 6 Cost Optimization Strategies in AI Systems

Your AI bills can grow quickly. The diagram below shows the top 6 techniques to make your AI system less expensive.

Prompt caching: This technique caches stable prefixes like system prompts and tool schemas. This means the model can skip redundant computation on requests with a shared prefix.

Model routing: A router sends simple queries to a small model and hard requests to a frontier model. Since a lot of queries are simple in practice, this can save a lot of money.

Batch API: This technique processes large jobs asynchronously by batching the requests first and then sending them. Providers often offer discounts on batch requests.

Semantic cache: We use an embedding model to cache queries. When there is a hit, the stored answer will be reused instead of calling the model again.

Trim the context: This technique trims the conversation history. This prevents sending the whole conversation every time, which can be costly.

Smaller model: We can also use techniques like distillation or quantization to convert a large model into a smaller model.

Over to you: Which of these techniques are most useful in production?

INFER/CON: Meet the People Building the AI Stack (Sponsored)

Join AI engineers, researchers, founders, and technical leaders at INFER/CON on November 16 in San Francisco. Convened by Datadog, this one-day conference brings together speakers from Anthropic, Together AI, Inferact, Fireworks AI, and more. Explore inference performance, production reliability, evaluation, and the economics of AI alongside the people building and operating these systems. Expect technical conversations, practical perspectives, and opportunities to connect with peers tackling similar challenges.

Attendance is free and by application only; all applications are subject to approval.

Use exclusive code to apply: Byte/VIP

Apply to attend

How a SQL Query Executes Inside a Database

Parse and validate: The database parses the SQL to check it is syntactically correct and the referenced objects, columns, and data types are valid. It converts the SQL into a logical query tree.

Rewrite and Transform: The database simplifies and restructures the query tree so it can be executed efficiently.

Optimize and choose a plan: The database evaluates the most efficient way to run the query. It compares different execution plans and selects the best plan that is supposed to use the least time and resources.

Execute the plan tree: The database runs the selected plan operations, like filtering, joining, and sorting, typically from the bottom of the parse tree to the top to produce the results.

Access cache and storage: The database retrieves the required data during execution from the cache when it is available or reads from the storage when it is not.

Return rows: The final result is formatted and sent back to the client.

How does a request actually travel through Claude Code?

Most of us type a prompt and watch the magic happen. The diagram below shows what's really going on behind the curtain, based on the Claude Code source code.

Let's trace one real request: "Fix the failing test in auth.test.ts."

Step 1: The user sends a prompt to Claude Code through their interface.
Step 2: The interface (CLI, IDE, or SDK) wraps the prompt with repo and file context and hands it to the agent loop as a request.
Step 3: The agent loop plans the next move and proposes an action: Edit(auth.ts, lines 42–58).
Step 4: The permission system checks the proposed action against the rules.
Step 5: The approved action becomes a tool call: Edit(auth.ts, patch), dispatched to the matching tool.
Step 6: The tool runs in the execution environment (shell, cloud, or sandbox) as a real syscall.
Step 7: The execution returns a tool result back to the agent loop.
Step 8: The agent persists the turn to state and streams the final message to the user.

The whole system is just this loop, repeated until the model stops asking for tools.

Over to you: which step in this loop do you think is the hardest one to get right when building your own coding agent?

However, there’s a difference between the work requested and the work that the storage system has to perform. A storage system does extra work as well, which is equally important. It can help you find data quickly, recover data after a crash, or support concurrent transactions. But this extra work also has a cost. For example, the request might involve reading a small record, but it can require fetching many blocks of data. Or when the user requests an update to some value, it can trigger multiple writes in the storage system. Also, if you want to keep the dataset highly available, it might require much more space than the actual size of the dataset.

Read amplification, write amplification, and space amplification show the differences between logical work and physical cost. In this article, we’re going to look at these differences in detail. Here’s what we will cover:

• What is logical work and physical work in a database

• Read amplification

• Write amplification

• Space amplification

• The amplifications with B-trees

• The amplifications with LSM trees

• How indexes and filters use space to avoid reads

• How compression helps improve amplifications

• The role of caching

• The impact of amplification

Logical Work and Physical Work

Read more

Your team connected Slack, Jira and GitHub over MCP, and engineers still spend their time steering agents to the right answer. This article looks at why: connectors fetch from one tool at a time, so every agent stitches the pieces together itself, trusts the first doc that looks right, and adds a search to your token bill each time.

Read what changes when a context engine sits on top of your connectors: the same coding task finished 83% faster at 48% lower cost.

Read the blog post

Anthropic has recently reached a point where coding isn’t the bottleneck for them. Claude is able to do a lot of heavy lifting. It can iterate on research loops. Or optimize complex programs. Or even build new services from scratch.

However, the main problem Anthropic faces is that the infrastructure they have around writing code is not able to keep up with the amount of code being generated by its agents. Activities like downloading repositories, preparing workspaces, and recording changes are becoming major bottlenecks.

To solve this problem, the Anthropic engineering team built Scion. You can think of Scion as a source control system that has been designed around a huge repository and clients that are almost always online. In this article, we’ll look at the architecture behind Scion. Here’s what we will cover:

• How source control works

• Anthropic’s problem with source control

• Why Git couldn’t solve the problem

• How Scion works

• How changes are handled in Scion

• How Scion caches data

• How Anthropic scales Scion

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

How Source Control Works

As you will already know, a software project is made up of files that change continuously. Developers add features. They correct mistakes and update dependencies. They also sometimes need to recover an earlier version. A source control system records all these changes and lets multiple developers work on the same codebase.

Let’s look at a few key terms related to source control.

A repository contains the project’s versioned information. Next, you have a commit that denotes a specific version along with other details such as the author, a description, and a history of the earlier commits.

A branch is used to demarcate a line of development. For example, your main branch signifies the line of code into which you integrate all the reviewed and approved changes. Different developers can work on different branches and combine their code through a merge process.

The files present on a developer’s machine are known as the working copy. A checkout process prepares those files for a specific version.

Lastly, you also have automated systems that rely on source control. Continuous integration (CI) runs processes such as compilation, testing, and other checks when code changes. Each job may need a workspace that contains the appropriate version of the code. Coding agents and research jobs need similar environments.

Anthropic’s Problem with Source Control

Anthropic stores a large amount of its code in a monorepo. You can think of a monorepo as a shared repository that contains multiple projects and components. This way, code that is related can share a common history. But it also means that many separate activities might depend on the same source control infrastructure.

But a large repository is only a part of the story. Anthropic’s problem is much deeper. They report that merged changes have grown from roughly 2500 per day in May 2026 to more than 30000 per day by September. Meanwhile, the size of its main branch’s history is close to 100 GB. Even a compressed working copy goes beyond 25 GB.

The CI system is doing approximately 25 million checkouts per day. This number is before counting developer environments, agent sandboxes, and research jobs. To be clear, you shouldn’t think of these checkouts as individual code changes. A single change can spawn many jobs. And each job prepares an environment for doing its work. Ultimately, your infrastructure workload can get much larger than the number of changes being merged.

For example, a naive approach to transfer a compressed working copy for every CI checkout can generate more than 600 PB of traffic per day. Therefore, the practical problem Anthropic faces has two sides. One is that the system must be able to accept new changes quickly. Second, it must be able to distribute the code to a huge number of consumers. If you improve only one aspect, the other can become a bottleneck.

Why Git Couldn’t Solve the Problem

The local independence of Git becomes quite expensive in this type of workload. As you might know, Git is designed in a way that an ordinary full clone of the codebase can be used as a self-contained repository.

When the necessary data is locally available, you can use Git to inspect history, compare versions, and create commits without the need to contact a central service. This is a great strength of Git.

However, Anthropic’s workload is such that this independence is not valuable. Most of the clients are automated jobs and agents that already depend on network services. At the same time, if you give those clients a good chunk of repository data, it leads to repeated transfer, storage, and maintenance costs.

To be clear, Git also has ways to reduce those costs. Anthropic used a bunch of these strategies. For example, they use features like shallow clones, sparse checkouts, partial clones, internal mirrors, and repository seeds. Each of these techniques can handle different parts of the problem. For example, a feature like sparse checkout helps deal with files that are part of the workspace.

The first architecture Anthropic built was around the use of Git. You had jobs start from seeds, obtain additional data through caches, and use layers of repository mirrors. At its peak, the Anthropic engineering team was operating several hundred internal mirrors to distribute traffic. But they had to keep all these mirrors in sync with each other as the repository data changed. Seeds also became outdated as the rate of merging increased. Operations that needed historical data became a lot harder to manage because clients held only partial repository data.

The Anthropic team realized that its workflow already had one authoritative shared repository while the infrastructure supported a lot of independence at the client level. With Scion, they changed this responsibility allocation approach.

How Scion Works

Scion makes the server responsible for repository operations. It assumes that clients can reach a source control service when they need data.

In this setup, the server answers historical data queries, calculates diffs, and creates commits. It also performs merges and rebases. The client (known as scn) maintains a limited cache and prepares files on disk. It watches for local filesystem changes and provides an interface for developers and jobs to deal with the service.

This setup is known as the client-server asymmetry. You can think of it as the client and server having different capabilities and responsibilities. For example, let’s say you have a tool that needs to find out the change behind a particular line of code. This operation is commonly known as blame. In Scion’s operating model, the tool asks the service for the answer to its question. It doesn’t need to assemble enough local history to calculate the answer.

The same principle applies when you’ve to create changes. In Git, you have a commit that’s created locally and later shared through a push. In Scion, you create a commit using a server operation. There’s no separate push step.

You might think that such a centralized service uses a single server that contains everything. But that’s not the case. Anthropic distributes the data and scales the service across machines. There’s no fully materialized repository on any single individual server. The tradeoff is that you have a strong dependence on service availability and connectivity.

How Changes are Handled by Scion

Changesets provide a small, durable record of each change. Once you create a commit on the server, you need to ensure that they are accepted efficiently.

Scion’s basic unit for managing this is a changeset. You can think of this changeset as a durable record of a change. You can easily construct other useful representations of the repository from this changeset.

To understand its structure, two terms are important. First is a blob that stores the contents of the file. Second is a tree that describes the contents of the directory. It connects names to files and subdirectories. In Git, a commit points to a root tree that represents the project at that version.

A Scion changeset records the files that changed relative to its first parent. This happens together with a reference to their contents. For example, let’s say you have a change that updates a file named “docs/guide[dot]md”, deletes “src/legacy[dot]rs, and modifies “tools/build[dot]sh”. The changeset keeps a record of those file changes and their relevant references. It also keeps the parent information and author details. Here, “changed files” doesn’t mean a sequence of line-by-line patches.

A merge can also have multiple parents. The first parent gives you the baseline against which Scion records its file changes.

Changeset metadata is stored in Spanner, which is a distributed database. Compressed file contents are kept in cloud object storage. The root tree is derived by the server rather than being stored directly inside the changeset.

This way, you can record a change based on the changed data and affected structures. No need to process the whole repository.

How Scion Caches Data

Using immutable objects makes it easy to use shared caching.

Scion recognizes its source control objects by their contents. This technique is known as content-addressing. In this approach, you have a hash function that calculates an identifier from the contents of an object. The system can then find that object later using this identifier. If there’s a change in the content, the new object gets a different ID value.

This way, your stored objects are immutable. A specific ID continues to refer to the same data. For example, let’s say you have a cache that contains a particular version of “parser[dot].rs”. If you have another job that asks for the same object, the cache can return it without asking another server if the contents have been modified. If there’s a new version of the file, it means the object is different.

There are two consequences of this. One is that you can use identical content. Cached historical objects don’t need freshness checks merely because development is happening somewhere else in the repository.

Git uses content addressing. But the way Scion differs is in how extensively its service and architecture are organized around asking for and caching these objects.

Information still changes in the system. For example, let’s say the name “main” moves as changes are integrated. The immutable object that represents a specific version and the mutable name that identifies the latest version require you to treat things differently.

How Anthropic Scales Scion

Scion scales by combining shared source control APIs with a hierarchy of caches. Its design is such that answering questions is separate from delivering the files needed for a working directory.

The first point is that repository operations move into a shared service. Many tools need information such as which changes affected a file, who modified a line, and so on. If you’ve to maintain a local repository to answer such questions, it creates a lot of downloading and maintenance work.

Scion exposes these operations through APIs. You can have clients ask the servers directly. The servers can compute derived data when needed and save those results to Spanner. With this, any jobs that only need these answers don’t have to prepare a local repository.

Second, you have immutable objects that make caching pretty simple. As we saw earlier, Scion identifies objects using hashes based on their contents. An existing ID always refers to the same data.

Third, clients only request the objects they need. Scion represents directory structure through trees and file contents through blobs. The client identifies what is missing. So the server receives explicit object requests rather than having to think about a customized transfer.

Fourth, by sharing caches, they reduce the need for repeated downloads and computation. On a compute node, jobs talk to “scion-daemon”. You can think of it as a background process that shares cached objects across jobs.

Any missing objects come from “scion-cache”. This is the regional caching layer that’s shared across many nodes. If that layer also misses, you can get the data from authoritative Scion servers that read from Spanner or cloud storage.

Lastly, you also have strict cache boundaries that help contain failures. All three layers support the same object-request interface. Otherwise, a regional cache that fails could cause its consumers to overwhelm the authoritative servers.

Conclusion

The Anthropic engineering team learnt a lot of important lessons while building and running Scion.

First was that rapid development is very possible with a small team. For reference, it was just a team of seven people that built both Scion and Antique (which is their code review tool).

The second takeaway was that developers are still valuable, but their scope of work is changing. All of the developers working on this system would rate Claude as better at writing code than they are. But the developers were critical in providing invaluable understanding of the high-level inter-system components and considering the trade-offs in the design decisions. A lot of the developer work revolved around stopping Claude from taking a certain path.

Third, they were able to make things more optimized and also deliver high-quality stuff. Due to Claude’s ability to write a ton of code much faster than human developers, Scion is among the most well-tested and well-defended codebases they’ve ever created.

Fourth, they were able to achieve a new level of operational excellence. The team uses Claude Tag heavily so that they can monitor each component of the system.

References:

• Redesigning source control at Anthropic

• Git Reference

How does full-stack observability impact engineering speed, incident response, and cost control?

In this eBook from Datadog, you’ll learn how real teams across industries are using observability to:

• Reduce mean time to resolution (MTTR)

• Cut tooling costs and improve team efficiency

• Align business and engineering KPIs

See how unifying your stack leads to faster troubleshooting and long-term operational gains.

Download the eBook

Performance engineering is the discipline in which you design, analyze, and optimize software systems so that they run quickly. Also, they should use resources efficiently. In distributed architectures, the focus of any performance engineering activity is on shaving milliseconds off response times. Or locating execution bottlenecks in CPU or memory. Or making sure infrastructure costs are kept under control.

Historically, performance engineering work has been slow. And manual. You often depend on a small group of really skilled developers. However, with the rise of AI-assisted software development, there’s a contradiction. Developers can write and ship code faster than ever. No matter what their skill level may be. But that code often introduces subtle performance problems.

To solve this problem, the Netflix engineering team has developed an autonomous performance agent system. In this article, we’ll be learning more about it. Here’s what we will cover:

• Bottlenecks for traditional performance work

• The problem with AI-Generated code

• Can LLMs handle profile data

• The agentic profiling pipeline

• The problems Netflix found in production

• Safety, verification, and trust architecture

• Shifting left

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

Bottlenecks For Traditional Performance Work

Traditional performance engineering work at Netflix was built on a highly iterative loop that is pretty much manual in nature.

First, you trigger profiling for an application. A developer tries to detect high latency or increased resource usage by triggering a profiler on a live production server.

In the second step, the profiler outputs raw trace data or flame graphs. The developer downloads these details to their local machine.

Third, the developer spends time looking at complex, color-coded flame graphs. They try to spot wide stack frames that might show signs of higher-than-normal CPU or memory consumption.

Next, the developer goes through the codebase looking for cryptic method names found in the flame graph. They try to make sense of the surrounding application logic to figure things out.

Fifth, the developer traces the execution path and identifies the inefficient operation. They write a proposed patch to fix the operation.

Moving further, the developers deploy the patch to an environment where any change in performance levels can be measured.

At the end, when new code paths are deployed in production, the developer starts the manual process once again.

As you may have noticed, this workflow doesn’t scale. Everything depends on a few domain experts who have a deep sense of understanding about low-level execution stacks. Since human attention is a scarce resource, you can investigate performance issues only for major incidents or obvious system slowdowns. Smaller but steady resource drains go unnoticed.

Why Braintrust, Raycast and Jane Street use WarpBuild (Sponsored)

Fast-moving teams ship more code than ever, and AI tools add even more. Their CI has to keep up without the bill getting out of hand.

That is why teams at Braintrust, Raycast, OpenRouter, Comcast and Jane Street run GitHub Actions on WarpBuild:

• Twice as fast at half the cost

• Linux, macOS and Windows on x86-64 and ARM64

• Unlimited macOS concurrency

• One line change to runs-on

Use code START50 for $50 in free credits.

Try WarpBuild free

The Problem with AI Code

The use of AI coding assistants has increased performance problems. There are two main reasons for this. The speed trap and the context gap.

Developers who are using AI tools can generate and ship code up to 10 times faster than before. No matter their skill level or knowledge of the system. But you cannot guarantee that code that is fast to generate also performs well. While your average developer might be able to write 10K lines of code in a few minutes, the team might have to spend years debugging regressions. Or dealing with code bloat. Or handling other architecture-related problems.

Also, if you’ve noticed, LLMs are great at syntax and local-level logic. If you ask an LLM to write a function or an API endpoint, it writes code that compiles successfully and runs without any obvious errors. It might also pass standard unit tests. However, the LLM has no idea about the host platform’s runtime environment.

For example, you can’t expect the LLM to know that a particular utility function being used in your code might run 10 million times per second on a critical request path. It cannot see the overall call stack. Neither can it understand memory allocation challenges that might exist inside a container running at peak load within your platform. Therefore, the code you write might appear good enough in isolation. But it can still waste a huge amount of resources when you run it at scale.

Can LLMs Handle Profile Data

Netflix wanted to test whether automation could remove these issues. They set up an experiment to find out if an LLM can read and make sense of raw profiling data directly.

Different languages have different profilers. But they use a shared language and store the same core information. For example, JVM’s async-profiler or Java Flight Recorder operates at 100 Hz sampling to capture execution traces, memory allocation rates, and thread lock contention. Python’s py-spy provides low-overhead, async stack sampling of live production processes. Go has pprof, which generates goroutine profiles, heap snapshots, and block contention data.

Each one of these profilers breaks down execution into call stacks. They track two main metrics. First, you have the inclusive time. This is the total time spent executing a function. It includes every child function and the subroutines called by this function. Second, you have the Self (CPU) Time. It’s the time spent running code purely inside the function itself. No external calls are involved.

With this exercise, the Netflix engineering team came to the conclusion that a single analytical pipeline could understand data from almost any programming runtime.

Another advantage of LLMs is that they don’t have to guess how an algorithm works from scratch. Since LLMs have been broadly trained on many software repositories, they are mostly familiar with classic programming anti-patterns. For example, they know that quadratic loops where a hot method is called again and again for each item in a collection aren’t a good idea. Also, loop invariants where you compute a calculation for every iteration of the loop instead of making the calculation once at the beginning. Other patterns include excessive object allocation on hot execution paths and “N+1” query patterns.

Due to this knowledge base of LLMs, you don’t need to ask an AI model to guess why an application might be performing poorly. You can build a workflow that asks the model to pattern-match known performance bugs against actual call stacks.

The Agentic Profiling Pipeline

Netflix built an automated performance system that wraps the LLM inside a well-defined pipeline.

A key lesson Netflix engineers learnt during this implementation was about scoping. If you feed an LLM a complete, raw stack trace without boundaries, it can get distracted. Often, the model spends a lot of reasoning tokens analyzing internal JVM garbage collection pauses, low-level Linux kernel syscalls, or even 3rd party open-source framework overhead.

Application developers don’t have any control over such internal library concerns or OS routines. Therefore, you need to provide the agent with strict package prefix rules. This way, the model can ignore generic platform noise and focus on tracing only the specific application code.

The Problems Netflix Found in Production

When the Netflix engineering team deployed the profiling pipeline against its production service, the agent found 4 major categories of performance bugs.

First was an algorithmic bottleneck that wasted 8.8% of CPU resources. The agent found an aggregation method using Java Streams where an immutable map was updated inside a reduction step.

// Anti-pattern: O(N^2) accumulation inside stream reduction
stream().reduce(ImmutableMap.of(), (acc, val) -> {
    return ImmutableMap.builder()
        .putAll(acc) // Copies every existing entry on every single step
        .put(val)
        .build();
});

Since putAll(acc) copies every existing key-value pair on every step, you need to copy elements again and again to build a map of size N. This results in quadratic complexity.

The agent modified the quadratic builder reduction with a linear collection approach.

// Optimized: O(N) single-pass collection
stream()
    .flatMap(e -> e.entrySet().stream())
    .collect(toMap(Entry::getKey, Entry::getValue));

The second category was about regular expressions on the hot path. In a service, the developer had to sanitize incoming request strings by modifying characters that weren’t allowed. The earlier implementation ran a raw regular expression substitution on every request.

// Anti-pattern: Dynamic regex evaluation per request
String sanitized = input.replaceAll(”[^a-zA-Z0-9.-]”, “_”);

As you might know, evaluating regular expressions involves parsing and state-machine transitions. This logic on high-volume traffic used 46% of the service’s CPU time.

The agent replaced the regex evaluation with a pre-computed lookup table that checked characters by index. The execution got 22 times faster.

The next category was related to unnecessary object allocation that caused OOM crashes. A service frequently checked whether incoming data payloads contained specific metadata fields. To perform the check, the code was creating a new, heavy HashSet on every incoming call.

However, the underlying data object already had an internal lookup method available. Also, instantiating millions of such objects consumed 47% of the service’s total heap collections.

The agent fixed this issue by removing the redundant object instantiations and modified the code to reuse the cached method.

The next category was around library misuse. For example, the discovery involved metric telemetry instrumentation. One of the services tracked operations by looking up a counter from a metric registry on every single call.

// Anti-pattern: Looking up metric instances on every function execution
Metrics.counter(”request.count”).increment();

When the query was sent to the registry, it needed dictionary lookups and internal synchronization locks. The profiler showed that the work consumed 66% of the CPU, whereas the actual business logic used only 0.3%.

The agent fixed it by caching the metric counter reference at the time of class initialization. Also, it found that the same issue was happening across three independent services.

Safety, Verification, and Trust Architecture

If you have an AI agent that can optimize code, it’s quite dangerous if the changes are applied without automatic verification. If the agent makes a change that breaks business logic, it’s worse than inefficient code.

To prevent regressions, the agentic pipeline uses the standard SDLC.

In stage 1, you verify the logic. Before you move any code, the agent’s branch must pass all existing unit, integration, and contract tests. This is to make sure that the changes made by the change don’t alter basic inputs, outputs, or handling of edge cases.

In stage 2, you have the canary deployment. If the tests pass, the pipeline spins up a canary deployment in production. The canary instance gets a portion of the traffic. It runs side-by-side with an identical baseline instance that runs the original code.

Stage 3 is about generating the observability report. The platform collects hard telemetry metrics over a big enough window. Some metrics are relative CPU utilization, median latency, P99 latency, and memory allocation rates. They also don’t use any AI model to check the canary results.

In stage 4, you have the final engineering decision. When the canary version shows a good level of performance improvement with zero increase in error rates, the system opens a PR. This PR goes for human review. It includes the code change, the original flame graph, and the canary report. The final decision on deploying the change to production is taken by the human developer.

Shifting Left

Automated production profiling is quite effective. But if you just rely on it, you end up exposing several limitations.

By the time an agent finds an anti-pattern in production, you’ve already paid for days or weeks of wasted compute capacity.

Second, developers are anyway cautious about changing a running service. If you have a service that handles production traffic without crashing, the developers hesitate to just make changes for the sake of optimization.

Lastly, multiple teams often make the exact same mistakes when it comes to patterns. If you run a super-expensive agent on 10 different services just to find the same problem, it’s not very efficient.

These limitations led to Netflix making the decision to shift performance checks left into the early development cycle. Also, they wanted the agents to share their findings across the entire engineering organization.

However, as you might know, language models are stateless. Every time you send a prompt to an LLM, it starts with a clean slate. It might no retain any memory of the bug it found some time back in another service. The Netflix engineering team built a Pattern Catalog to make the models achieve persistence. This was done by pairing a stateless LLM with a stateful catalog. Or a shared memory across the fleet.

The Pattern Catalog was built around 4 key principles.

• First, it has to be centralized and serve as the single source of truth for all engineering teams. This way, no two teams end up debugging the same issues.

• The second principle is that the same catalog entries should work across distinct environments. For example, offline profiling, CI/CD pull request checks, and local development tools.

• Third, you need to format entries so that an automation script can easily inject them into an LLM’s system prompt.

• Lastly, the contents of the catalog should be reviewable by human developers.

To implement the Pattern Catalog, the Netflix engineering team used a simple Git repository filled with plain Markdown files. They didn’t use vector databases or complex knowledge stores. The idea was borrowed from the “LLM-wiki” concept popularized by Andrej Karpathy.

By using a standard Git repository, the developers got several advantages. For one, developers can diff changes, open pull requests, and use git features to find out why an entry was created. It also scales well to 100s of detailed performance rules without the need for complex search infrastructure. Lastly, plain Markdown is easy for both humans and language models to read without translation layers.

How Netflix Catches Problems Early

With the Pattern Catalog in place, Netflix can handle performance engineering via two paths. Reactive and Proactive.

In the reactive path, your system profiles production services on a recurring schedule or on demand. When the agent finds a new bottleneck, it tests the fix on a canary instance. If everything goes fine, the patch is merged, and the agent documents the new finding in the Pattern Catalog.

In the proactive path, the process starts when a developer creates a new PR. CI tooling performs a fast regex or symbol match against the Pattern Catalog index. If the code in the PR matches known anti-patterns, the CI bot calls an LLM and provides it with the pull request diff. The LLM checks the context and leaves an inline comment that explains the inefficiency. It shows the resource impact and suggests a fix. All this happens before the code ever merges to the main branch.

Conclusion

The key learning from the Netflix engineering experience is that automating performance engineering work is not an all-or-nothing game. You can have organizations adopt workflows along a four-level maturity spectrum.

Level 0 is the manual situation. This is the level where machine learning is not used. Human developers manually capture traces, interpret flame graphs, and trace code paths.

Level 1 is the assist approach. This is the entry point for AI adoption. The LLM acts as an analytical copilot. A developer initiates the profiling process. But they don’t spend time studying flame graphs. Instead, they feed the stack dump to the model. The model finds the hot execution path.

Level 2 is the orchestration level. This is the current state at Netflix. A lot of proactive scanning happens. Also, they use the Pattern Catalog.

Level 3 is the fully autonomous level. This is the long-term vision at Netflix for performance agents. At Level 3, the agents will create their own optimization strategies. They will explore multiple alternative implementations. Or run automated benchmarks.

References:

• A Netflix Production Story - P99 Conference

Some integrations belong to the whole company: a shared Slack bot, a team analytics API key, a server-to-server credential. Asking every user to connect their own account means duplicate tokens, half-finished setups, and a new support ticket every time someone joins.

WorkOS Pipes handles OAuth and token storage so your product can connect to 300+ integrations. With Connection Sharing, an org admin connects a provider once from the drop-in Pipes Admin widget, and every active member can use it right away. The credential is stored against the organization and never sent to the browser.

See how shared integrations work →

Nemotron is a family of large language models (LLMs) by Nvidia.

The Nvidia engineering team has built Nemotron with a central design idea. This idea is to build a great language model with good capacity while bringing down the computation and memory requirements. To achieve this, they combine different types of neural-network layers. They use only some parameters per token. Lastly, they also optimize how the model generates answers.

In this article, we’ll look at how Nemotron works and the various techniques used by Nvidia while building it. Here’s what we will cover:

• How Nemotron generates text

• How a transformer model spends resources

• How Nvidia’s Nemotron reduces costs

• How Nemotron uses Mamba and MoE

• How Nvidia trains its model

• How Nvidia uses multi-token prediction

• How Nvidia reduces storage for models

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

How Nemotron Generates Text

Nemotron is a transformer-based LLM. Therefore, you can expect it to work pretty much the same way as any other LLM.

In a typical LLM, you have some input text that the user sends to the model. This text is divided into tokens. These tokens can be words, parts of words, punctuation, or other text fragments. While processing, the model converts each token into a numerical representation known as an embedding. Next, you have the model process these embeddings through a sequence of layers.

At output time, the model calculates scores for the possible next tokens as part of the decoding process. The decoding step picks a token and appends it to the sequence. Then, it continues generating further tokens. For these calculations, the model makes use of its parameters or weights. The parameters it has learned during the training phase.

Nemotron’s main goal is to make this process of generating text as efficient as possible. This efficiency is critical for AI agents. This is because AI agents repeatedly read instructions, make calls to tools, get responses from those tools, and decide what to do next. You can see that a coding agent might have to make dozens of model calls while trying to solve a problem. Good efficiency can do wonders for such an agent.

Nemotron Lightning model, which has a total of 30 billion parameters, tries to optimize this repeated execution work.

How Does a Transformer Model Spend Resources

A transformer-based language model spends resources on attention and feed-forward computation.

As you already know, the model breaks text into small pieces called tokens. It then converts each token into a list of numbers. You can think of this list as a token’s representation. These numbers change as the model processes the text. It gets to know more about the token and its context. So in a way, the representation of the token changes.

Attention lets you move information between tokens. For example, let’s say you have a sentence “Adam puts the ice cream in the freezer because it was melting”. To interpret “it”, the model has to know about the “ice cream” that was used earlier in the sentence. Attention provides a way to bring that earlier information into the representation of the word “it”.

To understand how this information gathering works, you need to know about three terms: query, key, and value.

A query is the information that a user wants to know about. A key describes what a particular note might be about. A value contains the actual information that the note can provide. The model compares a query with the available keys. It then combines the values and gives more weight to stronger matches. All of these are numerical calculations.

Multiple attention heads perform several such calculations in parallel. Each head learns a different way to match and combine information. Ultimately, the feed-forward network processes the information present at each position. It applies matrix operations and executes nonlinear functions to generate another representation. For example, after attention brings the information about “ice cream” into the representation of “it”, feed-forward computation can refine that representation using patterns learned during training.

You can think of attention as collecting all the useful notes from various places. On the other hand, feed-forward works on the notes that you’ve collected during attention.

Let’s now look at the cost part.

As you might already know, a model generates an answer one token at a time. It produces a token, uses that token as part of the input for the next step, and continues. This is known as autoregressive generation. Much of the text remains the same across each of these steps. However, if you recalculate the keys and values every time, you are repeating a lot of work. The KV cache stores repeated work so that the model can reuse it. But cached work still occupies memory. In full attention, a long history means more keys and values to store. Also, more information needs to be checked at each step.

There is also a cost difference between reading the initial prompt and continuing the answer. For example, when you process a complete prompt, the model has to calculate relationships across many token positions. In a model that looks backward, the first token has one position to consider. The second token has two. The third has three. And so on. These comparisons pile up across the entire prompt.

From this, you can easily guess that if you double a prompt, you need to have four times as much attention comparison work. There is quadratic growth. With KV cache, the earlier information is already prepared, so doubling the history means doubling the attention work. This is linear growth.

How Nvidia’s Nemotron Reduces Costs

Nvidia’s Nemotron 3 model uses Mamba layers and a mixture of experts to reduce costs.

A Mamba layer maintains a compact numerical state. As tokens arrive, this state is kept updated. You can think of it like a running set of notes about what has been read. The state is made up of numerical features learned during training. You can use it to carry information across a long sequence. However, the state doesn’t grow by storing a separate key-value record for every earlier token.

During generation, a Mamba layer’s state size and per-token update cost stay fixed. You can keep updating the model without making the layer search an ever-expanding cache. With a fixed state, you also have one trade-off. Compressing a long history into a bounded representation can make it difficult to retrieve content in a precise manner.

A MoE architecture changes the feed-forward computation. In this setup, your model has many feed-forward networks called experts. A router selects a group of experts to process each token. The model might have a large amount of capacity. But only part of that capacity is used for each token.

How Nemotron Uses Mamba and MoE

Nemotron combines Mamba and attention within one network.

Mamba lets you process sequences in an efficient way. Attention gives direct access to representations associated with earlier positions. Nemotron interleaves these mechanisms so that information can pass through both as it moves through the network.

The layer sequence is defined by the architecture. Every token representation proceeds through that sequence. It encounters Mamba, attention, and expert transformations at their specific positions. Therefore, the attention layer runs on representations that earlier layers have already processed. This way, much of the sequence processing uses a compact state. Selected layers can retain attention’s ability to connect distant positions. The remaining attention layers maintain KV caches.

Mixture of experts (MoE) gives the model capacity without activating every expert. In a dense model, every token goes through the same feed-forward network as every other token at that layer. With MoE, one layer contains many alternative feed-forward networks. A router selects which set of networks is used. The selected experts process the representation. Their output are combined using routing weights.

Nvidia’s Lightning configuration talks about 128 routed experts per MoE layer. 6 experts are selected for each token. This is how a model that has roughly 30 billion parameters but use only roughly 3 billion for a particular token.

Of course, the unused experts are still part of the model. They must remain accessible for later tokens. With sparse activation, you reduce the work done per token. But the total parameter count still impacts the memory needed to host the model.

Lightning also specifies 32 query heads and only 2 key-value heads when it comes to attention. This is a classic grouped-query attention arrangement.

In a basic multi-head attention, each attention head has its own keys and values. With grouped-query attention, you can have several query heads share key-value representations. The model retains multiple ways of querying the context while it stores fewer distinct key-value representations.

How Nvidia Trains Their Models

With training, you first build a language model. Next, you teach it useful behavior. Training determines the values of its weights and the capabilities those calculations can support.

The first stage is pretraining. In this stage, you make the model learn from large quantities of text by predicting tokens and adjusting its weights to reduce prediction errors. By going through many examples, you can build representations of language, code, facts, and other patterns. Lightning’s base model is trained on crawled and synthetic material that covers code, maths, science, and general knowledge.

Nvidia’s pretraining tries to progress the model from a broad data mixture toward more emphasis on high-quality material. The greater breadth of content completes coverage. Later sampling places more weight on selected capabilities.

The next training phase is supervised fine-tuning (SFT). In this training, you teach the model using examples of desired behavior. You can include instructions, conversations, code, tool interactions, and structured responses.

In Nvidia’s SFT recipe, user and system messages provide context. But the training objective is calculated on assistant tokens. Therefore, the model learns which responses should be produced for a specific conversation. If you have a task that needs structured output, the desired response can provide an example of the correct information and the required JSON format.

Reinforcement learning adds feedback to the model’s own attempts. Lightning uses Group Relative Policy Optimization (GRPO). The model generates several responses to a prompt. Those responses receive rewards. The training procedure compares their relative performance to update the model.

The complete training recipe includes an environment that evaluates mathematical answers, executes code against tests, checks instruction compliance, and validates output schemas.

As you can see, there is a logical progression in the training stages. Pretraining helps develop broad predictive capabilities. SFT provides examples of useful behavior. Reinforcement learning refines behavior using feedback on generated attempts.

How Nvidia Uses Multi-Token Prediction

Even if you make each token cheaper, ordinary generation can’t proceed sequentially. The next token depends on the preceding sequence, including tokens that have just been generated.

Multi-token prediction (MTP) trains an auxiliary mechanism to predict further ahead. At inference time, these predictions can support speculative decoding. In this approach, you have a lightweight drafting mechanism that proposes several candidate tokens. The main model evaluates the proposed continuation in a verification pass.

When you accept several proposed tokens, you can have token generation advance by several positions while needing fewer sequential passes through the main model. If you reject a proposal, generation continues from the accepted prefix.

Nvidia’s models ship with integrated MTP capabilities and separate draft models.

How Nvidia Uses Low-Precision Numbers to Reduce Storage

Nvidia also made another optimization regarding how the numbers involved in the calculations are represented.

A BF16 weight uses 16 bits. If you represent the weights with fewer bits, you can reduce the storage requirements. Also, reduce the amount of data transferred through memory. The challenge is approximation error because a smaller set of numbers cannot reproduce every original weight in an exact manner.

Nvidia uses the NVFP4 format. It combines four-bit floating-point values with scaling information. Groups of 16 values share an FP8 scale, with an additional scale at the tensor level. Nvidia also uses quantization-aware distillation (QAD). In QAD, you first create a quantized student model from a BF16 model. The BF16 model remains fixed as the teacher. The student trains with simulated quantization and learns to bring its output scores closer to the teacher’s scores.

Nvidia was able to reduce the size of the model from approx. 66 GB for BF16 to approx. 22 GB for the NVFP4 checkpoint.

Conclusion

Nvidia’s Nemotron 3 Super has approx. 120 billion total parameters and 12 billion active parameters. With Ultra, they have scaled this figure to 550 billion total and 55 billion active. Both models use the hybrid Mamba-attention design and introduce LatentMoE.

LatentMoE addresses another cost of expert processing.

The movement of wide token representations and loading expert weights can be quite expensive. The solution is to project the representation into a smaller space for the routed expert computation. Experts process this compact representation. Their outputs are combined, and the result is projected back to the main model dimension. If you reduce the width of the routed computation, you can save memory traffic and communication. Nvidia uses those savings to support more experts and more active expert combinations at a similar inference budget.

The Ultra model also adds a new training technique called multi-teacher on-policy distillation. The model generates its own attempts and receives guidance from multiple domain-specialized teacher models. “On-policy” means learning from attempts generated by the current student. This lets the teacher models guide situations that the student model actually encounters.

References:

• Nemotron 3 family white paper

• Nemotron 3 Super: Open, Efficient MoE Hybrid Mamba-Transformer Model

• Inside Nvidia Nemotron 3: Techniques, Tools, and Data that make it efficient and accurate

Your app, live in seconds, for free. Railway’s push-button compute, storage, and networking is available immediately upon sign-up.

Humans and agents alike operate on one vertically integrated system including hardware, software, and a prompt-to-production experience.

Sign up with the link below for free cloud credits and let your agents cook.

Start free

This week’s system design refresher:

• AI Terms You Need to Know

• GPT-2 to GPT-6: What Actually Changed

• Al Topics to Learn before Taking Al/ML Interviews

• How GPUs Work at a High Level

• AI Cost Optimization Course Starts Today (Exclusive 20% discount for newsletter subscribers)

AI Terms You Need to Know

Agentic AI: It's an AI system that describes the behavior of the whole system. The agent understands and interprets the goal. It creates a plan and calls tools and APIs to get the results. The agent evaluates the results and corrects the mistakes. It keeps on improving the results until it achieves the desired goal.

Large Reasoning Model: It’s a type of model that is trained to solve problems which require lots of logical steps. It breaks a problem into smaller problems and performs reasoning on them to come up with the right answer.

Vector Database: It's a type of database that stores vectors, as the name implies. These vectors are actually lists of numbers which represent the meaning and characteristics of text, audio, video or other data.

Retrieval Augmented Generation (RAG): It's a technique to provide relevant information to the model when needed. The relevant information is mostly stored in a vector database, but retrieval can also be through APIs, SQL, or hybrid search.

Model Context Protocol (MCP): It provides a standard interface to AI applications to connect and access external tools and APIs. The communication happens through the MCP client within the AI applications communicating through MCP servers.

GPT-2 to GPT-6: What Actually Changed

It's been seven years from GPT-2 to GPT-6, and a lot has changed.

The diagram below shows each generation and its key highlights, based on the public information we know.

GPT-2 (2019): More like a prototype to validate the potential of the decoder-only transformer. This was a 1.5B-parameter model that only went through pre-training with next-token prediction. The trained model was only capable of continuing text, not answering questions.

GPT-3 (2020): The same recipe as GPT-2, just much bigger and trained on more data to test how far scale alone could go. GPT-3 had 175B parameters, more than 100 times the size of GPT-2, and still only went through pre-training. The scale allowed it to show in-context learning capabilities. When shown some examples, the model could follow them.

GPT-4 (2023): On top of pre-training, OpenAI fine-tuned it with RLHF (reinforcement learning from human feedback), so it was not just a completion model but could actually answer questions. It was also multimodal, meaning that it could process input images as well as text, and it handled up to 32K tokens of context.

GPT-5 (2025): The first GPT generation with built-in reasoning. Given a prompt, tools, and an effort level, the trained model could generate reasoning tokens before the answer. Its context window grew to 400K tokens, with a router in ChatGPT to decide when a prompt needed thinking mode.

GPT-6 (2026): This new model has a bigger context window (1.05M), supports async tool calling, and is generally safer. An interesting feature, called mid-turn steering, is that it can take a new instruction while it's thinking.

Al Topics to Learn before Taking Al/ML Interviews

AI interviews often test fundamentals, not tools. This visual splits them into two buckets that show up repeatedly: Traditional AI and Modern AI.

Traditional AI focuses on fundamental ML topics, mostly from before neural networks became dominant.

Modern AI focuses on neural network foundations and newer concepts like transformers, RAG, and post-training.

Interviewers generally expect you to know both. They expect you to explain how they work, when they break, and the trade-offs.

Use this as a checklist, and make sure you can explain each topic clearly before your next AI interview.

Over to you: which topic here do you find hardest to explain under interview pressure? What else is missing?

How GPUs Work at a High Level

When people say GPUs are powerful, what they really mean is this: GPUs are built for massive parallelism from the ground up.

Let’s break down what’s happening under the hood.

At the top level, a GPU chip is made up of many Streaming Multiprocessors (SMs). Think of SMs as mini parallel engines replicated across the chip. Instead of one big brain, you get dozens of smaller ones working simultaneously.

Inside each SM:

• A Warp Scheduler decides which group of threads (a warp) runs next.

• Dozens of CUDA Cores execute instructions in parallel.

• A Register File stores thread-local data at ultra-low latency.

• Load/Store units move data between registers and memory.

• Texture units handle specialized memory operations.

• L1 Cache provides fast, on-SM data access.

Each SM works independently, but they’re connected through an on-chip interconnect. Below that sits the L2 Cache, shared across all SMs. This is the coordination layer. If one SM misses in L1, it checks L2 before going to global memory.

Then come the Memory Controllers, which interface with Global Memory. This is where things get interesting:

• Extremely high bandwidth

• Much higher latency than on-chip memory

That’s why GPUs rely on massive parallelism. While some threads wait on memory, thousands of others keep executing.

AI Cost Optimization Course Starts Today

Learn how to reduce AI infrastructure and inference costs without sacrificing performance or quality. As a newsletter subscriber, you get an exclusive 20% discount, automatically applied when you click the link.

Check it out now

What you’ll learn:

• Understand your AI bill: Break down token usage and calculate the true cost of every request.

• Track AI spending: Monitor costs across teams, features, and users to identify what’s driving your bill.

• Optimize prompts and caching: Reduce redundant model calls and avoid unnecessary token consumption.

• Manage agent context: Keep context windows lean and prevent expensive agent loops.

• Use smaller models: Replace costly large-model calls with smaller models where appropriate.

• Build intelligent model routers: Route requests to the most cost-effective models while maintaining quality.

• Control multi-agent costs: Estimate spending, catch runaway retries, and optimize agent workflows.

Check it out now

You need to move data to connect all these separate activities so that they can work as a coherent system.

To manage this movement of data, you need to make decisions about when and how information should travel within a system and outside it. This is where you have multiple strategies that can be used to handle data movement. In this article, we’ll look at the key strategies of data movement along with their benefits. Here’s what we will cover:

• The source, consumer, and trigger of data movement

• What is the pull approach

• The polling approach

• The push approach

• The long polling approach

• Continuous communication with SSE and WebSockets

• The role of message queues

• Change data capture for database changes

• How to choose a data movement strategy

The Source, Consumer, and Trigger of Data Movement

Read more

Markets don’t wait. On a prediction market, prices move in milliseconds, and every trade has to land the moment it happens. At the Agentic Data Summit on December 9, Batu Balimtas, Member of Technical Staff at Kalshi, takes you behind the scenes in Engineering for the Moments That Move Markets: Kalshi and Redpanda. Hear how Kalshi’s data infrastructure keeps pace when it matters most, and what it takes to run it in production. Catch Kalshi live at 12:00 PM PT. The summit is free and virtual and runs 8:30 AM–2:00 PM PT, so you can join for the sessions that matter most to you.

Save Your Seat

An off-the-shelf large language model can explain why there’s an error in your code. It can suggest possible causes. Or it can perform tasks like writing a database query. However, all this would be at a general level based on the model’s training. You cannot expect it to write answers based on your system’s internal workings.

This is handled using tools. Each tool is built for a specific job. But you can end up with lots of tools, especially when working in an enterprise setup. This is the situation the Uber engineering team faced.

To deal with this, they built the MCP Gateway. You can think of it as a shared platform that lets different AI agents find which tools are present and use them to perform their job. In this article, we’ll look at how Uber built this gateway in detail. Here’s what we will cover:

• What are tools and why AI agents need them

• The role of MCP at Uber

• How the Uber MCP gateway is designed

• How the gateway finds out the available tools

• How does the MCP gateway work

• How the MCP gateway deals with permission and access

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

Why AI Agents Need Tools

If you want a general-purpose LLM to write answers specifically for your current systems, you need to give it access to information about your system. For example, let’s say a developer asks an LLM-based AI assistant why XYZ service suddenly started failing. The answer is hidden in the system’s logs, deployments, metrics, and incidents. The LLM needs to access this information to be able to answer this question.

This is where we can create tools. You can think of a tool as an operation an AI application can execute. For example, you can have a tool to search logs. Retrieve incident details. Or check the status of a deployment. The model selects a tool and provides the necessary inputs. The application executes it and sends the result back to the model.

Many of these tools already exist as APIs. As you might already know, an API is an interface through which one program can request something from another program. For example, we can have a logging service that exposes an API. It takes the service name and time range as input and sends back all the log entries that fall within the range.

However, for an agent to use such a service, we need to describe what it does and what inputs it needs. This structured definition takes the form of a tool that the LLM application can use. Behind the scenes, the API performs the real search.

The Role of MCP at Uber

MCP stands for Model Context Protocol. With MCP, you get a standard way for AI applications to find out available external capabilities and talk to them. Uber uses MCP for its tool ecosystem.

We must know three important terms to understand the role of MCP in a better way.

First, you have the MCP client. This is the component that sits inside an AI application. It communicates using MCP and handles various interactions, such as requesting the available tools. Or making calls to the selected tools.

Second, there is the MCP server. It exposes the capabilities (or tools) through MCP. In Uber’s case, the server shows the available tools and accepts any requests made to those tools.

Third, you have the MCP tool. This is the individual operation exposed by a server. You can also think of them as capabilities. One server can provide multiple tools.

MCP requires you to define the tools properly. This definition is made up of information such as name, description, and input schema. A schema provides the structure of data. Things like which fields are present, which are mandatory, and what types of values they can accept. For example, you can have a schema where service_name has to be a string and limit is an integer.

MCP uses a structured way to format a request or its parameters. Also, the corresponding result or error. The format is usually JSON-RPC 2.0. This standardization makes it easy to integrate stuff. You can have the same MCP client talk to different MCP servers.

However, this standardization doesn’t come for free. Someone has to decide which tools exist and where they run. Also, you need to track the ownership of the tools and how they work. Initially, the engineering teams within Uber built their own MCP integrations. The autonomy was great at a small level. But as adoption increased, it was obvious that each team was solving similar problems. For example, exposing tools. Managing connections. Dealing with permissions. And so on. To make things even more difficult, finding out which tools were available was difficult. Everything was scattered.

On top of all this, Uber’s existing service landscape brought another challenge. All the services in Uber didn’t talk to each other in the same way. Some used HTTP. Some used gRPC. And others used TChannel. For example, you might have an HTTP API use JSON and a gRPC API use Protobuf messages. You cannot simply forward an MCP request across these services without modifying the request format.

Therefore, Uber needed a common layer that could connect MCP clients to the tools. They also wanted this layer to handle shared discovery, security, and operational support. This layer is known as the MCP Gateway.

How the Uber MCP Gateway is Designed

The Uber MCP Gateway platform is divided into two main components: the MCP Registry and the Proxy Gateway. This separation helps Uber’s engineering team manage tool definitions and permissions independently.

Let’s look at both in detail.

MCP Registry

You can think of the MCP Registry as the control plane. It holds the catalog of MCP servers and tools. You can record the information needed to discover the tools and learn more about them in the registry.

The reason they call it the control plane is that the registry also manages the configuration. For example, you can enable or disable the tool from the registry. It is also a shared source of truth.

The Proxy Gateway

This is the data plane. It handles the real MCP requests while agents are doing their work.

The gateway identifies the requested tool. It applies the necessary controls and sends the request to the right service. Also, it returns an MCP-compatible result.

How Tools Are Discovered

You can’t manually create MCP integrations for 1000s of services. It’s a lot of work, and there are chances of errors. Uber has automated much of this work by using AutoCrawler.

AutoCrawler finds out existing APIs and MCP servers. It generates or retrieves their tool definitions and registers them in the MCP registry.

On a technical level, AutoCrawler is a distributed workflow system. It’s powered by Cadence. In this setup, you have a scheduled job start workflow that looks for any new services, APIs, or schema changes.

But how does AutoCrawler discover new things? It has two main discovery pathways:

From Existing API Definitions

Many APIs use an Internal Definition Language (IDL) to describe their APIs.

An IDL is a machine-readable description of a service interface. You can use it to specify operation names, request fields, and documentation comments. For example, you may have an API definition describe an operation named GetDeploymentStatus. It might take a deployment ID as input and return a status and timestamp.

AutoCrawler can read these definitions and then go through several steps.

First, it creates an MCP server representation for the service. It might also update an existing representation. Next, it pulls out method names, request and response structures. Then, it translates whatever it has found into tool schemas that MCP can make sense of.

AutoCrawler also uses an LLM to come up with better tool descriptions from the details it has extracted from the service. Finally, it stores these tool definitions in the registry.

As you can see, AutoCrawler is a good example of Uber’s no-code approach. Existing service contracts have enough information for the platform to build the MCP interface.

Discovering Native MCP Servers

Some services directly implement MCP. At Uber, these are called native MCP servers. These servers already have tool definitions. So less work for AutoCrawler to get those definitions.

Each native server emits a heartbeat metric. It signals the presence and readiness of the server. AutoCrawler keeps track of these signals. When it finds a server, it calls the server’s listTools operation. Once it finds the tools, it creates a proxy representation in the registry for all the tools and their schemas.

As you can see, both discovery pathways end up in the same catalog.

Discovering Tools vs Permission to Use Them

Uber made a design choice that newly discovered servers and tools start in a disabled state.

This means that automatic discovery only makes the capability visible to the platform. After this, the team that owns the service should review and enable the service to make it available for the agents. The choice to enable is an ownership decision. It’s better that the service team has the opportunity to check whether the tool description is correct.

The owner team can review the tool definitions and update them using the registry’s interface. Every modification is a new diff that should be approved. Only approved changes are deployed. Also, you can have owners roll back to the previous version if needed.

How Uber’s MCP Gateway Works

Once you have saved the tool configurations in the registry and have marked them as available, the data plane consumes these tool details to refresh its in-memory state. This happens at a fixed periodic interval.

The gateway uses the mappings at the time of request processing. The periodic refresh process makes sure that any new changes can take effect without the need to restart the service. Or redeploy it.

The gateway takes these configuration details and uses them to create virtual MCP servers. You can think of a virtual server as a logical MCP interface that exists inside the gateway. It’s made up of a collection of tools along with an endpoint for clients to reach those tools. For example, you can have a service called “logging” exposed on the endpoint “logging/mcp”.

To use the tool, the clients talk to this endpoint as an MCP server. You have a handler within the gateway that knows which tools are part of that server and how to reach the service that backs the tool.

To understand things in a better way, it’s good to follow the path of a request through the gateway.

Let’s say you have an AI agent that calls a tool backed by a gRPC service

• First, the MCP request hits the gateway. The gateway finds the corresponding virtual server and the tool that is requested.

• Next, the server handler uses an in-memory mapping to find out the downstream destination and operation. In this case, the downstream service is the service to which the gateway will send the request.

• The handler converts the tool’s JSON arguments into a form that the service expects. For example, in the case of a gRPC operation, you have to serialize the request into Protobuf bytes.

• Moving further, the service runs the operation and sends back the response.

• Finally, the gateway converts the response into an MCP-compatible result and ships it back to the caller.

As you can guess, the same approach works for other types of services. For example, an HTTP backend receives an HTTP request. Or a TChannel service uses the TChannel or Thrift path.

How the Gateway Takes Care of Access

You can consider the gateway as a shared place to enforce access rules for all the tools.

The Uber engineering team uses their internal Access Control System to apply access rules and policies. These policies are based on the caller. You can have a human caller. A service. Or an AI agent. You can configure policies at the server level. Also, you can go as far as setting up specific policies for specific tools if needed. So the system provides a lot of flexibility.

This means that the platform supports authorization at the tool level. For example, you can have an incident management server expose tools for reading incidents and changing incident status. You can have a policy grant different access to those operations even though they are part of the same server.

The gateway can also redact responses for personally identifiable information (PII). Or any other sensitive data that may be present in the tool’s response.

For 3rd-party services such as Jira and Google, you need to have an additional identity check. This is because an internal Uber credential is different from a credential that might be accepted by a 3rd-party system. The gateway sends the caller’s internal user token to this service as part of its process flow. The 3rd-party service exchanges that token for the external auth token. Then, it sends the request to the external MCP server with that auth token.

As you can see, the responsibilities are divided. The gateway handles common access and execution details. The 3rd-party service takes care of the external stuff.

How the Uber MCP Gateway Deals With A Huge Catalog

The huge catalog of tools running at Uber can create a context problem for the MCP gateway.

You can have a gateway make 1000s of tools available, but an AI agent needs to have an efficient way to select the useful tools. This is because each tool definition takes up space in the model’s context window. You can think of this context window as the space available as part of the model’s input. This window has a limit. You also need it for other things like system instructions, conversation history, and results coming back from the tools.

If you have an application that loads tool descriptions and schemas for 1000s of tools in advance, they can take up a huge part of the context. Also, the amount of input that the model has to process shoots up. The same goes for tool responses. An API may send back a detailed object even when the task can be accomplished with just two fields.

To deal with these problems, Uber made three additions to the gateway:

• Omni MCP helps agents discover capabilities gradually. It’s a single proxy server that lets MCP clients access any of the gateway’s servers with gradual discovery. It provides tools like “discover_server”, “discover_tools”, “get_tool_schema”, and so on.

• Response Projection reduces the fields that are sent back in tool results. It’s a GraphQL-like pattern for MCP tools. You inject a new field in the tool request that tells the gateway which fields should be part of the response. The gateway removes the other fields.

• Code Mode lets coding agents work with results using shell commands and files.

Conclusion

Building the MCP Gateway has changed how AI agents operate at Uber in a big way.

It began as a fragmentation problem. You had dozens of teams independently wiring up MCP integrations. There was inconsistent tooling. No shared security guarantees. Also, duplicated infrastructure. Now, everything is a unified platform that can be scaled. Any team can plug into this platform within minutes.

The core philosophy that drove Uber’s team was that existing APIs are the fastest way to let AI agents access the tools. They didn’t want the teams to rewrite their services to make them fit for the gateway. The MCP gateway does the heavy lifting.

The key takeaway for developers is that the hardest part of building agentic systems is not the AI. You should worry more about building the right connective tissue. Things like discovery, security, and reliability are very important in a production environment.

References:

• Designing MCP Gateway Uber’s MCP Management Platform

• Model Context Protocol

Right now, you’re your agent’s context engine. You write the rules files, maintain the skills, and paste in the Slack thread it missed.

But hand-curated context goes stale. And when there are conflicting answers, your agent can’t tell which is right. You pay for it in higher token bills and longer reviews.

Join us live on Oct 28 (FREE) to see:

• The 8-levels of AI maturity and where teams get stuck

• How a context engine curates context and resolves conflicts before your agent writes a line of code

• Live demo: token savings in the same coding task, and use cases beyond code gen

Bonus: 2 open source tools you can use today.

Register now

A large language model (LLM) is a neural network. You can use it to process and generate language. How the model behaves is dependent on the numerical values you teach it during training. These numerical values are known as parameters.

Apple fits a 3-billion-parameter model on your iPhone. What this means is that the model has roughly 3 billion numerical values that it needs to work properly. These values influence how the model makes sense of its input and how it produces output.

How is Apple able to fit such a model on an iPhone without any adverse impact on the device?

In this article, we’ll try to answer this question. Here’s what we will cover:

• The problems of running an LLM on a phone

• How Apple makes the weights cheaper to store

• How Apple made repeated calculations faster

• How adapters help features share the foundation

• How Apple optimizes the hardware to run the models

• How does the system decide when to use the cloud

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

The Problems of Running an LLM on a Phone

Running an LLM is also called inference. In this stage, the LLM uses its parameters to process a request.

Before processing some text, the LLM has to divide it into tokens. You can think of a token as a word, part of a word, or even punctuation. The model takes these tokens and generates more tokens as part of the response. Within the model, these tokens pass through a bunch of processing layers.

The first problem is about storing the parameters.

At 16-bit precision, the 3 billion parameters we talked about earlier can occupy almost 6GB. But that’s not all. You need space for the input prompt, calculations, and temporary results that the model needs. Lastly, you also need to run the OS and other apps using the same memory. As you can see, storage is a big decision.

The second problem is about moving the weights quickly from memory to the processor. During text generation, the processor has to repeatedly fetch numerical data from memory. Even if you have a powerful processor, it might spend a lot of time just waiting for that data to be fetched from memory.

The Apple engineering team also had to deal with the challenges regarding memory capacity and memory bandwidth while trying to make their LLM run on the iPhone. On top of this, a mobile phone also has to take care of battery consumption, overheating, and other apps trying to use the same resources. You don’t want the LLM to make the rest of the device unresponsive.

How Apple Makes the Weights Cheaper to Store

The most direct way of reducing the size of the model is to use fewer bits to store each weight. This technique is known as quantization.

You can think of quantization as a way to represent numerical values using a smaller set of possibilities. Sure, you lose some numerical detail in the process. But the model can still perform its tasks at a pretty good level. The main trade-off is to select a representation approach that sits at the sweet spot for memory savings and quality. See the diagram below.

The Apple engineering team uses a technique known as palettization. In this technique, you group similar weight values. A small lookup table stores representative values, and individual weights store short references to that table. For example, you can have a 2-bit reference select from four entries in a lookup table.

In the earlier model version, the Apple engineering team combined 2-bit and 4-bit representations. It has an average of 3.7 bits per weight. This meant that the 3 billion weights could be stored in approximately 1.39 GB. A pretty good optimization.

They mixed different precisions because different parts of a model can easily handle different amounts of numerical error. You need to spend more bits on sensitive components so that quality doesn’t go down. Other components can be compressed more.

The later model Apple released makes the compression even more aggressive. The main decoder weights use 2 bits per weight. The embedding table uses 4 bits per weight. To make this compression work, Apple uses quantization-aware training. In this approach, you simulate the kind of restrictions that quantization will impose. Then, you adjust the model’s weights with those restrictions in place so that it can learn to work under them.

How Apple Made Repeated Calculations Faster

The weight storage is only one part of the overall memory problem. After you have successfully compressed the model, the model still needs memory to process a request.

Let’s say you have a model that has already read a paragraph of text and generated the beginning of a response. When the model tries to predict the next token, it needs to use information from earlier tokens. However, if you have to recalculate every intermediate result, it would be a big waste of computational resources.

A KV cache (key-value cache) stores these intermediate results. You can think of this cache as a way to reuse calculations that have already been done. However, the KV cache has a trade-off. As you can see, the cache takes up memory. The more text you have in a request, the more information you need to store in the cache. Even if you have a model whose weights fit comfortably within the memory, it can still face memory issues when processing a document with lots of text.

This means you have two targets to optimize. First, you can compress weights to reduce the space required by the model. Second, you can try to reduce the size of the KV cache.

Apple performs the cache optimization through a technique known as cross-layer KV sharing. In this technique, you need to divide the model into two blocks with a 5:3 depth ratio. Layers in the second block reuse keys and values produced by the final layer of the first block. By using this approach, Apple has brought down the KV-cache memory usage by 37.5%.

How Adapters Help Features Share the Foundation

You can use compression to reduce the size and cost of one model. But often, you may need many copies of that model with slightly different capabilities.

A simple approach would be to create one model for summarization. One for rewriting. Another for extracting information. And so on. But all these models would have a great deal of overlapping language capability. You are basically duplicating all of that capability.

An adapter can help you get rid of this duplication. You can think of an adapter as a small collection of parameters that changes how a base model works for a specific task. For example, let’s say you have a feature that writes concise notification summaries. Another rewrites longer passages into a different style. Both of these features can share the same base LLM. No need for duplication.

This technique is known as LoRA (Low-Rank Adaptation). While training an adapter, you fix the base model’s weights. Only the adapter weights are modified. This is why it becomes cheaper to specialize a model as compared to distributing a separate new model.

Apple applies adapters to various features such as attention matrices, attention-output projections, and feed-forward layers. Its rank-16 adapters use 16-bit values and occupy approx tens of MBs. They can load, cache, or swap these adapters as needed.

How Apple Optimizes the Hardware Running the Model

Even if the model is small, you need an efficient hardware implementation to run the model at the best of its ability.

Apple iPhones have several kinds of processors to take care of this. You have the CPU that handles all the general-purpose work. Then, you have the GPU that can deal with many numerical operations in parallel. The Apple Neural Engine (ANE) can run machine-learning workloads efficiently.

However, you can’t just use an accelerator and think that the model will run efficiently. You still need to select the right data layouts and split large arrays into manageable pieces. Also, you need to minimize memory copies. These techniques have an impact on speed as well as memory use.

How Does the System Decide When to Use the Cloud

Even after making all these optimizations, the model has limits.

In the case of Apple, its on-device model is quite capable of tasks like summarization, extraction, and classification. However, Apple Intelligence checks whether a request can be processed on the device. However, for more complex work, it can send the request to Private Cloud Compute (PCC).

PCC gives you a protected environment within which only authorized software can process the data. It also restricts any external parties from seeing the data.

Apple uses an encrypted network connection to protect information as it travels to the cloud. Your device encrypts the request for validated PCC nodes. Any infrastructure that is outside this boundary doesn’t have the keys to decrypt the request and read its contents. The data can be read only when it crosses into the authorized boundary. The goal is to make sure that even Apple’s staff and admin systems cannot access what’s inside this request.

When the request finally reaches the cloud, a separate server model takes care of the request. Apple uses mixture-of-experts computation and parallel processing tracks. This lets a large model choose the best computational components to distribute work.

Conclusion

Apple has been making a lot of new changes to its model.

In 2026, they launched the AFM 3 Core. It’s a new generation of its 3-billion-parameter model. Also, AFM 3 Core Advanced. It’s a sparse model with 20 billion total parameters. But it activates only 1 to 4 billion parameters at a time, based on the capabilities of the Apple Silicon systems.

In Core Advanced, the full model is stored on flash storage. Only expert weights are loaded into RAM. It selects weights based on the prompt. No need to change experts from the flash drive for every token. Apple has also expanded the cloud architecture. You can now run PCC workloads on NVIDIA GPUs in Google Cloud.

The main design principle that Apple follows is that the model’s capability should be based on the resources that are available at runtime. All of the techniques we’ve looked at come together to give the best experience to the users.

References:

• Apple ML Research: Introducing Foundation Models

• Apple ML Research: 2025 Updates

• Apple Security Research: Private Cloud Compute

We’ll stop selling ByteByteGo Live annual memberships on October 28 at 2 PM PT. After that, courses will be sold individually.

I wanted to explain why, and give you a chance to join before the change.

When we launched ByteByteGo Live (our cohort-based learning platform), we thought one membership covering every live course would make learning from industry experts simpler and more affordable.

But the membership offer has caused confusion. The enrollment has been lower than we hoped. So we’re simplifying things: choose the course you want to take, and pay for that course only.

For the next two days, you can still purchase the annual membership.

From the day you join, you can enroll in every live course we offer. Our current lineup includes:

• AI Cost Optimization, taught by Jeremy Hintz, Engineering Lead at Meta.

• Build Production Grade AI Systems, taught by Tanya Roosta, Director of AI at AMD.

• AI Engineering Fundamentals, taught by Ali Aminian, bestselling author of books on machine learning and generative AI and contributor to AI courses at Stanford.

• Build with Claude Code, taught by John Kim, Senior Staff Engineer at Meta.

• AI Evals in Practice: For Engineers & PMs, taught by Manjeet Singh, Senior Director of AI Product Management at Salesforce.

• Trust-optimized AI Development, taught by Kent Beck, creator of Extreme Programming and pioneer of test-driven development.

• Rebuild YouTube with AI, taught by Mikhail Sychev, former YouTube tech lead at Google.

You’ll also get access to every new live course we add during your 12-month membership.

The annual membership is 78% below the combined listed value of our current courses.

Existing memberships, including those purchased before the cutoff, will keep the same access and benefits for their membership term.

If you are ready to build and ship real AI projects before the end of 2026, join now to access every live course.

Membership sales close on October 28 at 2 PM PT.

Get the ByteByteGo Live Membership →

This change only affects our live, cohort-based courses. Your ByteByteGo newsletter subscription isn’t changing.

Thank you for learning with us.

Alex Xu
Co-founder, ByteByteGo

Datadog’s free eBook covers what actually separates the agents that ship from the pilots that stall. Spoiler: it’s not the model. It’s durable execution, memory management, evaluation pipelines, and observability.

You’ll learn how to:

• Identify the four failure modes that kill agent programs before they ship

• Build evaluation pipelines that let you trust and govern agents at production scale

• Instrument agent workflows with the observability needed to debug non-deterministic behavior in production

• Measure the leading indicators that prove agent ROI to your team and leadership

Download the eBook

Salesforce agents have completed 7 billion Agentic Work Units (AWUs), the unit the company uses to count work done by AI agents, across Agentforce and Slack. Of those, 3.2 billion came in its second quarter alone.

But getting agents to follow instructions consistently was a real challenge. The first version of Agentforce, Salesforce’s platform for building and running agents, put all instructions into one large prompt. As the rules piled up, the model began skipping some important ones. For customers using agents to interview job candidates or handle patient data, that was unacceptable.

Salesforce moved to a new design, a multi-layer system that combines deterministic code with reasoning models for maximum reliability. To understand how this works, we met two of the leaders behind it at Dreamforce 2026: Muralidhar Krishnaprasad (MK), President and CTO at Salesforce, and Khushwant Singh, SVP of Product Management for the AI application development platform. Thanks to MK and Khushwant for sharing the details with us.

In this article, you’ll learn:

• Why a naive design that relies on prompting is not reliable

• How to combine deterministic code with probabilistic reasoners to increase reliability

• How to build a data layer that supports identity resolution and links records from many systems into one unified profile

• How to build an efficient context engine

• How Agent Identity gives an agent its own permissions

• How AIforce and the Headless Toolkit let agents in tools such as Slack and Claude reach Salesforce data and actions

The naive design: one large prompt

The simplest way to build an agent is to write everything into a single prompt. That is how most teams build their first agent. Imagine you are building a support agent for a healthcare company. You can write the business rules as natural language instructions (e.g., only discuss the patient’s own records). Then add a description of available tools (e.g., “get scheduled appointments”). You also include other relevant info like patient data. The agent then includes all of this in the LLM’s context window. That’s the whole design. We just hope that the LLM does the right thing.

In this naive design, the model is expected to handle four things at once: 1) enforce the rules, 2) find who the customer is across different systems, 3) include the data the model may need, and 4) ensure the agent does what it is allowed to do.

The first version of Agentforce handled everything this way. According to MK, the team kept adding instructions such as only do this, only do that, and first get all the appointments. An interesting observation from the team was that this worked fine for up to eight instructions. But beyond eight, the model would discard instructions.

But why would a model discard an instruction? To a model, an instruction is not necessarily a rule. It is just some text in the context. There is no guarantee that an instruction will actually be followed. Besides that, instruction following will also get worse as we add more instructions to the context. This is not specific to Salesforce. A benchmark called IFScale tested 20 models with up to 500 simple instructions in one prompt to measure instruction following, and even the best model followed only 68% of them at the full 500.

So a prompt only tries to shape the behavior of the LLM. It cannot enforce that behavior. Let’s see what Salesforce built around the model to fix this.

How Agentforce is built

Salesforce’s new design handles reliability by taking the four jobs out of the prompt and having dedicated systems to handle each. To see where these systems are, let’s review the current Salesforce platform stack. The four main layers in the stack, from bottom to top, are:

1. Data layer: Turns a company’s data sources into one consistent view. Salesforce calls this Data 360.

2. Application layer: The business applications for sales, service, and marketing, which supply the actions an agent can take. Salesforce calls this Customer 360.

3. Agent layer: Where agents are defined and run. Salesforce calls this Agentforce.

4. Surface layer: The places where a person talks to the agent, such as a Claude chat or Slack.

Instead of putting everything in a prompt and expecting the model to handle every job, each job is handled separately by a system in one of these layers. For example, in the Agentforce layer, a scripting language called Agent Script ensures to enforce the rules. In the data layer, identity resolution is responsible for finding a unified profile of the customer. Agent Context Engine is a separate system that decides which data reaches the model. The model just handles steps that require reasoning.

In the rest of this section, we go through the four systems built to handle each job one by one. To make it concrete, we follow the demo MK shared with us. The company is a simulated healthcare provider called Cumulus and the patient is Harper Clark, but according to MK everything behind it runs in production.

1. Deterministic instructions with Agent Script

The first job that should be handled outside of the prompt is to enforce the rules. Many checks have to happen in every session turn, and they can be enforced deterministically. It is neither efficient nor reliable to expect the model to handle them. Another system can enforce them more reliably by just running code. That is the job of Agent Script, the language Salesforce built for describing agent behavior. According to MK, the team started working on it last year, after the first version of Agentforce ran into the instruction limit. It is open source and looks almost like Python.

How does it work? You just write the agent’s behavior in an Agent Script file. That file is compiled into a structured specification called the Agent Graph, and the Agent Graph is executed by the Atlas Reasoning Engine, the runtime behind Agentforce. The engine runs the code parts and calls the model when reasoning is needed.

To better understand, compare this with SQL triggers. A trigger is code that the database runs automatically before or after updating a row. The application that issues the update cannot skip it. Agent Script follows a similar pattern.

Let’s compare the naive design with Agent Script using a concrete example. Imagine Harper asks a question like “When is my upcoming appointment with Dr. Perl?” In the naive design, the prompt says “always fetch the appointments first” and we hope the model would follow. In Agent Script, the lookup is a line of code in the before_reasoning block of the medical appointments subagent. Because of that, it always runs reliably.

Putting the pieces together, here is what happens when Harper asks her question:

1. The model picks the medical appointments subagent.

2. The before_reasoning block runs (the appointment lookup executes as code)

3. The engine builds the prompt.

4. The model receives the prompt and writes the answer.

5. Lastly, the after_reasoning block executes. It may also send the conversation to a different subagent if needed.

This separation between deterministic tasks and reasoning parts allows the system to stay reliable. The model only focuses on reasoning over the context prepared by the code, not things like fetching appointments or what should happen after the reply.

While deterministic instructions improve reliability, they should not be used everywhere. The best practice is to use code for the parts we want predictable behavior, such as conditions, and use natural language for the parts that need actual reasoning.

2. Unified profiles with Data 360

The second system handles “who the customer is.” Harper’s appointment question may look like a simple lookup, but the appointment data may not be in the CRM (customer relationship management) system at all. It may be in Snowflake, a cloud data warehouse. Harper may also exist many times, as shown below.

One approach is to hand the model different lists of records and ask which ones are the same person. But that becomes a source of unreliability. Instead, a data layer, in three steps, can handle this.

Step 1: Reach the data. The data an agent needs is usually not in one place. The data layer connects to various sources and reads from them directly. This way, the agent has an updated view of the records. To show this, MK changed the insurance plan on Harper’s record in Snowflake and the change was reflected immediately.

Step 2: Harmonize. Different sources may have inconsistent views of the same record. For example, one system may store a full name in a single field while another splits it into two columns. The data layer maps each source onto a shared data model with common objects such as person, lead, and maybe account. According to MK, this mapping is automatic and works across thousands of sources.

Step 3: Resolve identities. This step handles situations where records from different sources describe the same person. In Data 360, this is done by match rules that an admin can configure. A rule can be an exact match (such as the same email address), a fuzzy rule that allows small differences, or a normalized rule, which cleans up a field before comparing it. This way, two phone numbers written in different formats will match. Note that the source records are never changed, and the output is a unified profile that links back to each of them.

With these steps, when Harper’s appointments are needed, it is obtained from the unified profile, instead of the model figuring out who Harper Clark is.

3. Relevant context with Agent Context Engine

The third job is to decide what data should go into the prompt. If we put everything into a prompt, it adds to the latency and may bury the relevant facts. A context management system has to sit between the data and the prompt and decide what a question needs. That is the job of the Agent Context Engine (ACE), a new component of Data 360.

The context engine handles a request in six steps:

1. Resolve: Identify the user, the agent, the purpose, and the entities the question is about.

2. Plan: Choose where to look for that information (e.g., relational queries, the profile, search indexes)

3. Reconcile: Rank findings by some metric (e.g., relevance, authority, freshness)

4. Govern: Apply rules (e.g., access, masking, consent)

5. Compile: Create the context that includes working state, memory, etc.

6. Learn: Decide what from this interaction should be remembered or updated.

MK compared this with the naive design of an insurance support agent. In the naive version, with no context management, the question “show me details of my insurance plan” made the agent pull data from various sources, leading to 124,000 tokens. With the context engine, the same question ended up with 9,000 tokens. After a follow-up about moving to Florida, the naive session had used about 300,000 tokens in two questions, while the session with the context engine used 65,000 after a third step to process an enrollment. The gap between the two designs grows as the conversation continues, which shows the importance of having context management in place.

So how does the context engine pick what to send?

There are different implementations in practice. One is to hold the customer’s profile as a graph in memory, where the nodes are things like the person or the plan. Then we select only the nodes that relate to the question. Documents can be filtered in a similar way. When Harper asks “What lifestyle is best for me?” three indexes that match exact words, meaning, and linked entities (an approach known as GraphRAG) select relevant passages out of hundreds of documents.

The context engine can also handle memory. For example, after the Florida question, it can store a note that this customer may be moving to Florida. MK pointed out that this is a memory, not yet a fact. It became a fact only when the customer went ahead with the enrollment, and only approved facts like this are promoted into long-term memory. Either way, the model only ever sees a small, current slice of what the company knows about the customer.

4. Permission management with Agent Identity

The last job is to ensure the agent does what it is allowed to do. In most agent setups today, the agent acts as the user. Imagine Alex is a sales rep who connects an agent to Salesforce. The agent gets Alex’s credentials and can do everything Alex can do.

This setup is not ideal because with Alex’s full credentials, any mistake it makes becomes a real problem. A human with broad permissions is at least limited by how fast they can work. An agent is not. It can make a mistake and repeat it many times. According to Khushwant, what customers want is: “This agent is working on behalf of Alex, but I don’t want the agent to have the same rights as Alex.”

The fix is to give the agent an identity of its own. Salesforce already controls what users can do with permission sets. A new system, Agent Identity, lets an admin attach rules and permissions to an agent too, so each agent ends up having a unique identity with unique permissions.

With Agent Identity, when an agent authenticates, there are two tokens, one for the agent and one for the user it is acting for. Before an operation runs, Salesforce checks the user’s permissions, the agent’s permissions, and the data policies. The operation runs only if none of them deny it.

Where the agent runs

So far, we have looked at what happens inside a reliable agent platform. But users using an agent may work from somewhere else, like Slack or Claude. Some companies build their agents on other platforms entirely. Customers generally want Salesforce to meet their users where they are.

The way to do this is to separate the platform from its own interface. Software built this way is called headless. Instead of running on its own user interface, it exposes its functions through APIs to make it available to other surfaces. To make this happen, Salesforce developed a live interface layer, called AIforce, that brings the platform’s data, workflows, and logic to wherever people and agents already work.

For an outside agent to use a headless platform, it needs a few things. The first is APIs, the functions the platform exposes. The second is skills. An API tells an agent what it can call, but not which calls a task needs or in what order. A skill fills this gap by listing the APIs, their parameters, and the order of the steps. The third is MCP (Model Context Protocol), an open standard where a server publishes a list of tools and agents can call them.

Salesforce developed the set of building blocks behind AIforce, named Headless Toolkit, that provides all of the above. Salesforce’s APIs already serve 9.6 billion calls a day, and its skills are published in a public GitHub repository called sf-skills.

A separate challenge is tool count. MCP makes it easy to expose tools to an agent, but every tool the agent can call is typically loaded into its context window as a definition. It may contain things like a name, a description, and a parameter schema. Those definitions are sent each time a request is sent to the model, which is costly. When there are hundreds of tools, they will consume a large share of the context with definitions the agent will rarely use.

A common solution is to create a small index of tools instead of all tools. The agent first searches for what it needs, then reads the details of what it found, and finally runs it. The Salesforce MCP Server works this way with just four tools.

Suppose you are in Claude with the Salesforce plugin, and you ask for the 20 best opportunities from the last 100 days. The request goes through three steps:

1. Claude calls “discover” with its interpretation of your request. The server runs a semantic search over an index of available API and skill, and returns a short ranked list.

2. Claude calls “describe” on the best match. The server returns the skill’s contract, which lists the APIs, the parameters, the dependencies, and the order of the steps.

3. Claude calls dispatch. The server routes the call to the right endpoint, checks the user’s access, and returns the result.

This way, the agent’s context holds only four tool definitions instead of containing tool definitions for all available tools.

What’s next

Everything in this article is a piece of scaffolding around the model. Today those pieces live in different products. The next step, according to both MK and Khushwant, is to make them one system. That is Enterprise AI Harness. A single place that enterprises can flexibly use. They plug in their agents and their data, and get eval, tracing, and other functionalities, applied to all of them at once, whether the agents run on Agentforce or somewhere else.

Long-running agents need endurance.

They must endure waiting for users, and weather tool failures. They must take each new jump in model intelligence in stride.

Inngest makes long-running agents durable, observable and improvable over time.

Check out their generous free tier and a sleek local dev server for testing your workflows.

Build for the long run with Inngest

This week’s system design refresher:

• Top 5 AI Building Blocks Every Engineer Should Know (Youtube Video)

• RAG vs Agentic RAG

• OAuth (authorization) + OIDC (authentication layer)

• Understanding the Linux Directory Structure

Top 5 AI Building Blocks Every Engineer Should Know

RAG vs Agentic RAG

RAG is a technique that combines information retrieval with an LLM to generate answers. Here is how RAG works at a high level:

• A user asks a question to the application.

• The retrieval system fetches the relevant information from a knowledge base.

• The application adds that information to the prompt.

• The LLM uses that context to generate an answer.

RAG usually follows a predefined retrieval process. The quality of its results depends on the accuracy of the retrieved context.

Agentic RAG follows a different approach. Instead of retrieving once, an agent can repeatedly fetch and evaluate information until it has enough evidence to generate an answer. Here is how it typically works:

• An agent selects the right tools based on the task.

• Those tools can search documents, APIs, databases, or the web.

• The fetched evidence is then checked if it is sufficient for the task.

• If it is not, the agent can adjust its search or choose a different tool.

• Once sufficient evidence is achieved, LLM uses that context to generate an answer.

Thank You, Open Source Maintainers (Sponsored)

Open source is the foundation of nearly everything we build, and the maintainers behind it make it possible. The good news: the movement to support them is growing.

Two years ago, the Open Source Pledge set out to make paying maintainers the norm. Today, 37 member companies have paid $8.4M to maintainers and foundations, and every new member adds to that momentum.

See how Sentry put $1M to work this year, or join the Pledge and be part of what’s next.

Learn more

OAuth (authorization) + OIDC (authentication layer)

If you want to show Google Calendar events inside your Slack channel, it will require permission to read your calendar. How would you let Slack access your calendar without giving Slack your Google password?

This is where OAuth 2.0 comes into the picture. Slack basically never sees your Google password. It lets an app access a protected resource on your behalf without knowing your credentials.

Basically OAuth 2.0 handles authorization. In Slack’s case, it will send you to Google’s authorization server, where you will login and give Slack permission to access your calendar. Google gives you the authorization code, and Slack will exchange that code for access_token.

If Slack also wants to authenticate the identity of the person requesting the access to Google Calendar, you will need OIDC for that. The OIDC handles authentication and sits on top of OAuth 2.0.

OAuth answers what you’re allowed to touch, while OIDC answers who you are. In addition to the access_token returned by the authorization server, this time, it will also send an id_token, a signed JWT which basically tells Slack who just signed in.

Understanding the Linux Directory Structure

The linux directory structure starts with the root directory "/". Let’s navigate to the directory structure of linux.

The kernel image and bootloader files are stored in the "boot" directory. Your hard disk partitions and device related files are located in the “dev” directory.

All your programs and libraries are stored in "bin", "sbin", "usr", and "lib" directories. Like, your installed software is located in the "usr" directory. Ever wondered where ls, cp, and cat commands are? "bin" is the directory that holds those essential commands.

System-wide configuration goes in "etc", and "var" holds data that changes as the system runs, which is where you'll find your logs.

If you have a USB drive, it gets mounted in “bin” folder. For any external drive that you want to mount temporarily on your linux machine, "media" is the directory for it. If you have multiple user accounts, "home" is the directory for each user in your linux system.

Since in distributed systems, we divide the system into so many components, it’s pretty common that one component may need to talk to another component to get something done. It may also have to wait for the other component’s response. You can think of coordination as this talking between components. For example, you may have one service waiting for another service to approve a request. Or a database waits for acknowledgment from its replicas. Or a worker waits for a lock.

In this article, we’ll look at why coordination is a challenge for distributed systems. Here’s what we will cover.

• Why coordination becomes a challenge

• How coordination increases in a system

• How to reduce coordination

• Strategies to handle coordination

Why Coordination Becomes a Challenge

Read more

Your app needs authentication. That usually means installing an SDK, wiring up callback routes, configuring middleware, and getting your environment right. Or you can run one command: npx workos@latest install.

The WorkOS AI installer detects your supported framework and adapts AuthKit to your existing project. It installs the SDK, creates your authentication routes, and sets up environment variables, with post-installation validation on by default.

Review the generated code, test your login flow, and get back to building your product.

Try the AuthKit installer

Caching is a super important aspect of system design, especially for large-scale systems. You can use a cache to store some data that is expensive to calculate so that it can be reused later without paying the same cost. For example, a website might cache the result of a database query instead of running that query every time someone opens a page.

The same is the case for large language models. Just like normal systems, caching in LLM applications also means storing stuff calculated as part of earlier requests so that we can reuse it later. This helps bring down processing costs and response time, especially when requests contain the same instructions, documents, or questions.

However, there are two types of ways we can cache stuff in LLMs: prompt caching and semantic caching. Both have the same general aim of reducing repeated work, but they work at different stages of a request. They also follow different rules.

In this article, we’ll look at how LLM caching works in detail. Here’s what you’ll learn:

• Why an LLM request costs money

• What is prompt caching

• Where to place the shared content in prompt caching

• What is semantic caching

• The difficulty with semantic caching

• Why checking the question is not enough

• When to invalidate the cache

• How prompt and semantic caching can work together

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

Why an LLM Request Costs Money

When you send some text to an LLM, the LLM processes that text and generates a response. During processing, the text is divided into tokens. Each token is a small piece of text. A token can be a word, part of a word, or punctuation.

These tokens directly impact the cost. For a typical text-generation API, the cost has two main components:

• The tokens provided as input.

• The tokens generated as output.

Output tokens often cost more for obvious reasons. However, the input tokens can also cost a lot because of the size. Input can include much more than just the user’s latest question. For example, let’s say you have a customer-support application. Along with the current question, this application can also send a bunch of instructions telling the model how it should behave. It might also send examples of good answers. Relevant product documentation. Earlier conversation messages. In other words, a 10-word question can be part of a request made up of 1000s of input tokens.

Let’s say you have another application that sends a long product manual with every user request. This product manual is like an encyclopedia of all the information that the user might be looking for. One user might ask about the installation procedure. Another asks about troubleshooting. Maybe a third user asks about account settings. While their questions happen to be different, the answers are all there in the same long product manual that is part of every request.

There is another type of repetition as well. 100s of users might ask how to change their profile picture. But each user might write the query using slightly different words. In such a case, the model may end up generating the same rough answer again and again.

As you can notice, we have two opportunities to reduce this repetitive work and bring down costs:

• What if the application can reuse the input information that has already been processed once? For example, the product manual.

• What if the application can reuse a previously generated answer if it is suitable? For example, similar questions asked again and again.

Prompt caching is used to exploit the first opportunity. Semantic caching is for the second one. Let’s look at both in detail.

Prompt Caching: Reusing the Model’s Input Processing

To make caching effective, you need a rule to decide whether something is fit for reuse. This rule is the most important part of building an effective cache setup.

For prompt caching, the rule is to match the beginning of the model’s input.

A typical text-generating transformer model works in two stages:

• First, it processes the input. This stage is called prefill.

• Second, it generates new tokens. This stage is called decoding.

When processing the input, the model converts text to numbers. This numeric representation helps the model use information it read earlier to generate output tokens down the line. Some of these representations are called keys and values. Or KV states.

Keys help the attention mechanism find out which earlier information is useful. Values are the actual representations of that information. A KV cache stores these intermediate results so that they can be reused. This is all part of the ordinary text generation flow in an LLM.

Prompt caching piggybacks on this reuse setup. It extends the whole thing for all the compatible input requests that share the same beginning. Let’s say you have the following two requests:

Request A: Support instructions + product manual + “How is installation configured?”
Request B: Support instructions + product manual + “How are diagnostic logs enabled?”

As you can see, the support instructions and the product manual are shared across both requests. They are known as the shared prefix. The actual question is part of the suffix that keeps changing for each input request.

After the model processes Request A, the system can store the KV states for the shared prefix. When Request B arrives, the model can reuse those same KV states to process the new question and generate the answer. We’ve reused the intermediate computation for the common context. No recalculation. No reprocessing.

AI wrote the code. Who pays for the CI? (Sponsored)

More AI-written code means more CI minutes. On default runners those minutes are slow and expensive.

WarpBuild cuts both:

• Twice as fast, so engineers wait less

• Half the cost, so the bill shrinks as usage grows

• Pay only for the minutes you use

Teams at Braintrust, Raycast, OpenRouter, Comcast and Jane Street trust WarpBuild with their GitHub Actions.

The switch is one runs-on change. Use code START50 for $50 in free credits.

Cut your CI bill

Where to Place the Shared Content in Prompt Caching

For prompt caching to work, we need to put the shared content at the beginning of the request. Basically, the prefix position. The word “prefix” is important here because a token’s internal representation depends on the information that comes before it in the input.

In a typical causal language model, the tokens that come earlier in the prompt can’t look forward at the question that might appear later in the request. This is why a model can reuse the shared stuff from the beginning even when the final user question is completely different.

However, when you place a different question before the manual, it changes the way the manual is processed. The question impacts the representation of the text in the manual. The manual’s text is the same. But the processed version is completely different.

This is the reason we have to place fixed, stable instructions or any shared reference material before the request details that can change. Even if you place a seemingly unimportant timestamp or random request ID at the beginning of the input, it can stop the model from reusing a large chunk of shared stuff.

To be clear, even the instructions need to be “exactly” similar. Even if you have two instructions that might mean the same thing, you cannot use them interchangeably for prompt caching. There should be no change in wording. No change in document order. No change in request settings.

The Cost Savings from Prompt Caching

Simply caching the shared prefix doesn’t make the cost zero. Most LLM providers usually charge it at a discounted rate. Also, the output token charges stay the same because the model still generates a fresh answer.

For reference, Anthropic’s standard prompt-cache pricing uses a cache-read rate of 1/10th of the ordinary input rate. There are some exceptions for particular models. Its standard 5-minute cache write costs 1.25 times the ordinary input rate, while a 1-hour write costs twice that rate. These are total rates for the affected tokens, rather than additional charges on top of their ordinary input cost.

As an example, if processing a prefix normally costs $1, that five-minute pricing schedule makes the initial write cost $1.25 and a subsequent read cost $0.10. Therefore, one write followed by one read costs $1.35 instead of $2. The more the number of reads, the greater the savings. But if a cached prefix is never reused, you don’t get any benefit.

LLM provider and model rules also determine minimum eligible lengths, expiration, and cache-write pricing. Even if we repeat a prompt, it doesn’t guarantee a cache hit if the cache record has expired or is unavailable.

To summarize, prompt caching mainly reduces input-processing work and can reduce the wait time before we generate a response. You cannot use it to eliminate the work that goes into generating a long answer.

Semantic Caching: Reusing a Previous Answer

Semantic caching is usually part of the application that surrounds an LLM. It can also be a separate service that the application calls before it goes to the LLM for an answer. Consider a couple of example questions that you might ask an LLM:

“Where is the setting for dark mode?”
“How can I switch the app to its dark theme?”

As you can notice, the same answer may be able to satisfy both questions if there’s no change to the application features in a major way. But a simple text comparison would consider them as two separate questions with two different answers. However, a semantic comparison can identify that they are essentially the same query asked in slightly different words.

To implement semantic caching, you need an embedding model. This type of model converts text into a list of numbers called a vector. These numbers represent features of the text in a way that makes it easy to compare two pieces of text.

The overall flow works as follows:

• The embedding model creates an embedding for the incoming question.

• It searches question embeddings that are already stored for suitable candidate answers.

• It checks whether a candidate answer is similar and valid for the current question.

• If accepted, it returns the stored answer. Basically, a cache hit.

• On a miss, it calls the LLM to generate a fresh answer. It may store the answer for future reuse.

As you can see, if there’s a cache hit, we don’t have to call the LLM to generate a fresh answer. No input and output charges in that case. However, tasks like embedding generation, lookup, and cache storage still use resources. So it’s not all free. Also, if we have a cache miss, we pay for the preliminary tasks as well as the LLM request. So a miss ends up being more costly.

The Difficulty With Semantic Caching

The most difficult part with semantic caching is to find out how similar two separate questions are.

In a semantic cache, we calculate a similarity score and compare it to some threshold. You can think of this threshold like a cutoff for accepting that the questions are similar.

If we keep a very strict threshold, we end up rejecting many more answer candidates. While this can improve accuracy and reduce inappropriate reuse, it also rejects a lot of legitimate paraphrases of the question. However, if we go for a loose threshold, more answer candidates get accepted. This increases the overall cache hit rate. But it also increases the possibility of returning an unsuitable answer. There is no single, universal threshold that works perfectly for every application. You need to decide the threshold on a case-by-case basis.

Let’s say a user asks two questions about cancelling an order before it gets shipped and after it gets shipped.

As you can see, both questions are almost the same. Except for one critical word that changes the meaning. Both questions deal with cancellation and shipping. Their embeddings may also be quite close. But a short answer of “Yes, cancellation is available” could be the correct answer for one, but totally incorrect for the other.

Data types like numbers, negation, dates, product editions, and eligibility conditions can cause similar problems while determining similarity. For example, “Does the Basic plan include backups?” cannot automatically reuse the answer to a question about the Pro plan.

Also, we cannot consider a similarity score the same as the probability of an answer being correct. For example, even if you get a similarity score of 0.92 between two questions, it doesn’t mean there is a 92% chance that the cached answer is suitable for both. The way we interpret the score depends on the embedding model and comparison method. Some systems expose a distance score instead, in which smaller values indicate closer matches.

Therefore, we need to have an evaluation process to find out the effectiveness of a semantic caching implementation. This evaluation process has to find out if a particular stored answer can satisfy a new request correctly. For testing purposes, the development team can collect a bunch of requests, identify candidate answers, and label whether reusing a specific answer is the right way. The test suite should have examples that cover legitimate paraphrases as well as difficult near-matches. This way, we have a better chance to find out the optimal threshold value.

Lastly, we need to measure the proportion of requests answered from cache and the proportion of those answers that are actually appropriate. We cannot celebrate just by achieving a high hit rate. We can also have an additional verifier that checks border cases. However, such a verifier also contributes to costs and latency.

Why We Can’t Rely On Just Checking the Question

You can ask the same question multiple times. But every time, you might expect a different answer. For example, a question like “What is my order status?” depends on the customer and the current status of the order. Similarly, a question such as “Does it support offline access?” depends on what the word “it” means in the conversation’s context. Also, if the user requests an answer in a different language, we can’t automatically respond with a cached answer generated in English.

Therefore, for a semantic cache to work, we need information beyond just the question to make the right decisions. Depending on the application’s use case, we may need to include stuff about the customer or organization, access permissions, product version, language, relevant conversation state, and source-data version.

Also, we need to enforce these conditions as part of the application. You cannot expect an embedding model to find the details of a customer just because the user included a customer identifier in the input question. Typically, semantic-cache implementations need to have metadata filters so that searches can be restricted to an appropriate scope.

There is another example that shows a specific complication. If the user asks questions such as “What’s the weather in NYC?” and “NYC weather today?”, they might look like suitable matches. But their answers depend on whether the user is looking for current conditions or a daily forecast. While generating an answer, we need to consider location, date, forecast period, units, and freshness.

When To Invalidate the Cache

There are chances that a cached answer might appear like an excellent match for a new question. But the answer might still be wrong because the source information behind the answer may have changed.

Let’s say you have an application that allowed product returns to take place within 30 days. Now the company has changed the policy. The product return time period has been reduced to 14 days. However, a cached answer might still talk about the old time period of 30 days. In this case, you might ask the same question and get the cached answer, which is no longer valid.

This is why we need to have a strategy for invalidating the cache.

Invalidation means making a cache entry unavailable for reuse. We can use multiple approaches to build an ideal invalidation setup.

The first approach is to define a TTL value. A TTL (time-to-live) adds an expiration time to a cached record. Deciding the TTL value is the key. If we keep a short TTL, we place a strong limit on how long stale information can stay eligible within the system. But there is still some gap that will always remain. TTL cannot guarantee freshness of data before expiration. For example, if a policy changes immediately after an answer is cached, that answer can be stale till the TTL expires.

The second approach is to use versioning. You can allow reuse of data based on a specific version. For example, an answer cached under policy_version=12 is ineligible when the application expects version 13.

It’s much easier to have a version that contains the entire knowledge base. But there’s a disadvantage. One small update can make the entire knowledge base invalid. All the cached answers are suddenly stale. However, if we invalidate only those things that have changed, it’s much more efficient. Much more selective. But to make it work, we need to track which answers depend on which documents or records.

The third approach that can impact cache invalidation is the overall settings we have used to generate answers. Things like high-level instructions. Output formats. Model behaviour guidelines. And so on. These settings matter a lot when deciding when to invalidate a cache. If these settings are changed, we should invalidate the earlier answers.

Invalidating a prompt cache is different from its cousin. If the application sends an updated policy in the prompt, the modified prefix cannot reuse an incompatible old version from the point at which the policy changed. However, if there is an unchanged earlier portion, it might still be reusable. Also, if the application keeps providing the old policy as part of every input, prompt caching cannot automatically find out if the actual business rule has changed somewhere. It’s the application’s responsibility to fetch the current policy information and feed it to the model.

Can Prompt and Semantic Caching Work Together

You can use prompt caching and semantic caching in the same setup. They are complementary. For example, you can have the application first look for a valid existing answer using semantic caching. If no such answer is available, only then does the application call the LLM and take advantage of any reusable prompt prefix.

If we get a semantic-cache hit, we don’t need to go for the main generation request. On the other hand, prompt caching reduces the cost of requests that still reach the LLM. We can also have an exact-answer cache that is checked before any semantic lookup happens. It can handle repeated identical requests cheaply.

It’s not necessary that we have a cache lookup at the very beginning of the flow. For example, if the application must retrieve current source versions to validate an answer, that particular check should happen even earlier.

Conclusion

Prompt caching is a natural choice in scenarios where you need to generate fresh answers for different questions. But those answers have to be generated from a shared context. This common context could be a general product manual, a set of instructions, stable tool definitions, or a long conversation history.

Semantic caching is great for cases where the same answer is good enough for many different requests. For example, product FAQs. Stable support instructions. However, semantic caching doesn’t work well when you want to generate highly personalized answers. Or if the answer depends on rapidly changing facts. Also, when there is a need for fresh creative output.

We need to have even greater care for action requests. For example, a cached message saying “The subscription has been cancelled” cannot be a substitute for making a real cancellation.

For a practical implementation of these caching strategies, we need to have some measurements. In the case of prompt caching, we need to measure how many input tokens are reused. This is because there is a major cost difference between serving a request that reuses a tiny prefix and one that reuses a long manual. For semantic caching, we need to evaluate metrics like accepted answers, freshness, and scope. Only those answers that satisfy the application’s quality requirements should be turned into reusable entries.

The ultimate goal with either approach is to lower cost and provide faster responses while preserving answer quality.

References:

• How caching works

• Prompt caching

• Techniques to optimize semantic caching

• Prompt caching diagnostics

Most teams record session replays at a flat sample rate, then spend the next incident scrolling through hundreds of sessions that have nothing to do with the bug.

This hands-on Sentry workshop shows you how to capture the replays that matter:

• Sample rates and buffer mode, plus the default we recommend

• Custom React hooks to record a specific user segment, a known-bad URL, or a flow you just shipped

• Bug reports that arrive with the replay already attached

• Filters for rage clicks, dead clicks, and custom tags

• Privacy defaults and how to tune them

Register now

In 2020, the Shopify engineering team decided to go all in on React Native. It saved them a ton of time and money. By using React Native, the developers didn’t have to implement the same feature in both Kotlin and Swift to make it work on Android and iOS. Sure, the framework came with its requirements. But the idea was very successful at the time.

However, in 2026, AI coding agents have made a huge difference to the economics of building mobile apps. Earlier, if you implemented something with React Native, you could justify the savings that came because of it. But now, AI agents have become so good at coding that building the same feature in Swift and Kotlin is much cheaper.

In this article, we’ll look at how Shopify decided to move out of React Native and how they migrated to native mobile development. Here’s what we’ll cover:

• What is native mobile development

• Why Shopify chose React Native

• What changed in 2025

• Shopify’s migration approach

• How Shopify ran the migration

• The agent feedback loop

• The migration results

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

What is Native Mobile Development

As you might be aware, an iPhone runs Apple’s iOS, and Android phones run Android. Both these OS provide a bunch of facilities to display the UI, handle notifications, and talk to the device hardware.

In native mobile development for a particular OS, you build an application directly using the languages, frameworks, and tools for that platform. This means Swift for iOS and Kotlin for Android. To be clear, a programming language is different from a UI framework. For example, Swift and Kotlin are programming languages. However, Apple’s SwiftUI and Android’s Jetpack Compose are frameworks. These frameworks give developers the tools to create the real UI with various components like text, buttons, lists, layouts, and animations.

To build natively for both platforms, you need to maintain two implementations of the mobile application. One for Android. Another for iOS. Sure, you can share the backend services for both platforms, but the client code will still be separate.

However, React Native lets you share much of that implementation across both iOS and Android. It’s a framework for building mobile apps using React and JavaScript (or TypeScript). The main benefit of React Native is that many components and application code you write using this framework can work with both iOS and Android. You don’t need to implement every screen independently in Swift and Kotlin. This is because React Native sits on top of the native UI components that are backed by their respective platforms. For example, if you make a component in React Native, it can create the appropriate iOS or Android views without the need for developers to write any platform-specific code.

To be clear, sharing code doesn’t mean that every platform difference is completely gone. You would still have some features that might need platform-specific coding. Also, you need to build, test, and release the application for each OS.

[Webinar] How to stop rewriting your agent’s code (Sponsored)

Right now, you’re your agent’s context engine. You write the rules files, maintain the skills, and paste in the Slack thread it missed.

But hand-curated context goes stale. And when there are conflicting answers, your agent can’t tell which is right. You pay for it in higher token bills and longer reviews.

Join us live on Oct 28 (FREE) to see:

• The 8-levels of AI maturity and where teams get stuck

• How a context engine curates context and resolves conflicts before your agent writes a line of code

• Live demo: token savings in the same coding task, and use cases beyond code gen

Bonus: 2 open source tools you can use today.

Register now

Why Shopify Went With React Native

The Shopify engineering team went with React Native in 2020. They wanted to solve three problems that were more or less linked.

The first problem was that you had to implement every feature two times with native development. One time for each platform. The second was that you couldn’t share developers between platforms. A developer was confined to their tech stack. An iOS developer couldn’t work on the Android version. Neither could the Android developer work on the iOS version. Not a great way to utilize the developers in a team.

The third problem was that you had to maintain feature parity across both versions. Feature parity is when every version of the app has the same features and capabilities. Let’s say, if you have a cool feature that works on iOS, but it takes another month to be available on Android, you can say that the two versions are out of parity. If you don’t maintain this parity, you risk alienating the user base of the platform that usually runs behind.

Also, when you have two versions of the app, you need to change both versions whenever there’s a change in any requirements. Even if the change is something as small as a text change. As you can see, this is a continuous cost problem that is present throughout the life of the application. You also need to have more coordination between the different platform teams while planning releases, announcements, and so on.

By going with React Native, Shopify reduced the impact of these problems in a big way. No need to implement every feature twice. No need to coordinate changes across multiple platform teams. It also made mobile development easy for developers already familiar with React and TypeScript. This is great because you could easily move people between projects. Plus, you also get to select developers from a much bigger TypeScript or JavaScript talent pool.

What Changed in 2025

As a framework, React Native has a lot of benefits. We have seen that already in the earlier section. But the framework also needs maintenance. This means additional costs.

The Shopify engineering team had to spend effort on optimizing the framework’s performance. They made improvements to the React Native foundations and contributed to framework upgrades. They also had to deal with changes to external dependencies. You can think of a dependency as a piece of software that the application relies on. If there’s a change in the dependency, you need to make changes to the application as well.

Native development is different from using a framework. The application has greater synergy with the platform and first-party tooling. You can use the tools provided by the platform’s own developers. Also, you have fewer dependency layers between the application code and the platform (iOS or Android).

Shopify was already using native code for some features. For example, widgets and watch applications. Also, for some background processing. You could say that React Native and native development coexisted within the application for several years. Of course, the benefit equation tilted more towards React Native.

However, the rise of coding agents made big changes to the equation. As you know, a large language model (LLM) can generate code quite easily. A coding agent combines an LLM with special tools that can read code files, edit code, and run tests. It could also check why things are failing and make changes to handle those failures.

By 2025, these agents had improved so much that Shopify had to take another look at its mobile strategy. To test the waters, the Shopify engineering team rebuilt some core parts of its application in Swift and Kotlin. These prototypes proved that the coding agents could take an implementation on one platform as a reference and write the equivalent code for the other platform.

For example, you can have an iOS feature that provides all the details about the UI elements, actions, and error conditions. An agent can use those details and build an Android equivalent. Sure, developers had to check whether the outcome was good enough and followed the conventions of Android and the overall architecture. However, suddenly, you had developers who could use these agents to code stuff outside their primary stack without a lot of effort.

Of course, with native development, two implementations would still be there. Both will need to be maintained. But Shopify claims that coding agents have reduced the work to such a level that concerns about duplication are no longer as important for making decisions as they were in 2020.

Shopify’s Migration Approach

To carry out the migration to native, Shopify had to choose the best pathway to migrate the existing applications. You have two ways to do so.

First, we have brownfield migration. In this approach, you change an existing application gradually. New and old implementations run together as individual parts are replaced. While the risk is lower, the migration takes more time. Also, you have to manage the integration between the new and old implementations.

Second, we have a greenfield approach. In this approach, you create a fresh implementation. Developers get more freedom to design a new architecture. However, you need to build an almost complete replacement before getting rid of the existing product. There’s no incremental change.

Shopify had used the brownfield approach earlier for some of its large applications. The development methods at the time would have taken too long. However, when they tried the new prototypes, the team found out that a fresh native implementation would be faster. The existing React Native application was a solid reference point for how things should work.

Therefore, Shopify chose the greenfield approach.

They first shipped the Shop app. This app was used by shoppers. The larger Shopify app, which is used by merchants, was migrated later. For reference, the Shopify app has more than 300 screens. Also, many features, such as widgets. And an Apple Watch application.

How Shopify Ran the Migration

When planning the migration, the Shopify engineering team found that if you asked a model to rebuild a large feature in one attempt, you got mediocre results. The output code was difficult to maintain. Also, it was very difficult for developers to properly review big features.

Therefore, Shopify built Helix to support the migration. Helix breaks the migration into small steps that can be reviewed. With Helix, you can have a developer select a screen, and Helix proposes small pieces of migration in the best possible order. Each migration piece has to go through the review process. Only then does it become the foundation for the next piece.

For example, let’s say you have an order detail screen. An initial checkpoint can finalize the structure of the screen and how navigation should work. Later checkpoints will deal with other features such as order summary and customer information. Each checkpoint is small enough that developers can review it properly.

Four types of checks are done to make sure everything works as intended.

First, you have the behavioural tests. You take expected actions on the app and check what happens. Second, you have the visual review. In this, we check how the new UI looks compared with the old reference application.

Next, you have adversarial code reviews. Two reviewers look for problems in the code. They try to find issues that might stay hidden behind a normal workflow. Lastly, we have another developer review where we check the results of the previous tests. If everything meets expectations, things proceed further.

The best part about Helix is that the team gets continuous feedback while the change is still small.

The Agent Feedback Loop

Shopify has a design philosophy that separates app logic from the UI.

For example, you can have a form with rules about valid or invalid inputs. To test them, you don’t need a visible text box. They can receive an input value and return a result.

Shopify’s architectural principle is to keep the business logic independent of the UI. The idea is that the business logic can run headlessly. You can expose the application state and actions through a CLI. This way, the agents can check state, navigate between sections, and perform actions.

While navigating like this without a UI might sound strange, it makes it easy to check how things work. You can have an agent take an action and check the state. There’s no need to make sense of screenshots and find out where a button is located. This is why behavioural checks can be made quite fast.

The Migration Results

The Shop app was live on the app stores as a native application in 12 weeks. There were many interesting stats.

The app startup time on iOS went down from 3200 ms on React Native to approximately 2400 ms on the Native version. The startup time on Android had even greater gains, going from around 4400 ms to 2200 ms. Almost half. The crash-free sessions went up. Also, the app size on Android was down from 293 MB for React Native to 184 MB for the native version.

All in all, these results have been quite encouraging for the Shopify engineering team. Of course, their approach combines a lot of things that make it successful. For starters, you have direct use of platform tech. You also have an existing application, which can act as a strong reference point. Also, the migration process is made smooth with small implementation checkpoints and quality checks.

The main thing we can take away from Shopify’s case is to have a fast feedback loop that can check if the code generated by AI agents is good enough. This is the key to making big migrations like this safe for the users.

References:

• React Native is the Future of Mobile at Shopify

• Helix: The internal tools powering Shopify app’s native migration

• Migrating Shop app from React Native to native

Right now, you’re your agent’s context engine. You write the rules files, maintain the skills, and paste in the Slack thread it missed.

But hand-curated context goes stale. And when there are conflicting answers, your agent can’t tell which is right. You pay for it in higher token bills and longer reviews.

Join us live on Oct 28 (FREE) to see:

• The 8-levels of AI maturity and where teams get stuck

• How a context engine curates context and resolves conflicts before your agent writes a line of code

• Live demo: token savings in the same coding task, and use cases beyond code gen

Bonus: 2 open source tools you can use today.

Register now

Genie is Uber’s internal AI assistant for answering engineering support questions. It finds relevant information in Uber’s documentation, uses a language model to explain that information, and gives engineers a way to continue the conversation or ask a human for help.

The interesting part about Genie is how Uber has connected multiple capabilities into building a working support system that fits seamlessly into an engineer’s workflow.

In this article, we will understand how Genie prepares knowledge, retrieves it, and generates answers. Here’s what we will cover:

• The problem that Genie is trying to solve

• What does Genie do?

• How Genie gains knowledge

• How Genie finds specific information from that knowledge

• How Genie handles user questions

• What does a typical Genie workflow look like

• How Genie makes sure the answer is correct

• What if Genie isn’t able to answer something

• How Genie collects feedback

• How Uber evaluates Genie’s performance

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

The Problem Genie Is Trying to Solve

The problem starts with how engineering teams support one another.

In a large software-driven company, many teams build platforms that other developers use to perform their jobs. A platform might provide tools for various activities such as software deployment, data processing, or working with ML workflows. Developers from other teams making use of these platforms inevitably need help with configuration, errors, or understanding unfamiliar features.

In a typical setup, the team responsible for managing a platform assigns an engineer to handle incoming issues during a particular period. This responsibility is commonly about providing on-call support. Although on-call work can definitely involve urgent production incidents, the problem Genie is trying to solve is geared towards everyday tech support that developers on platform teams provide through Slack channels.

For example, a developer from a product team might ask how to configure a platform feature. The on-call engineer reads the question and requests more details. The person asking responds, then waits again. Eventually, the on-call engineer identifies the relevant documentation or explains the required steps. Even a relatively straightforward question can involve several rounds of back-and-forth conversation and waiting.

Uber reports that its internal support channels received approximately 45K questions each month. At that volume, repeated questions consume considerable engineering time that the platform teams could utilize for better stuff.

Much of the information needed to answer such questions already exists somewhere. However, it is often scattered across Engwiki, which is Uber’s internal engineering wiki, or its internal Stack Overflow, and other documents. Finding the right information becomes difficult enough that it is much easier to ask another person.

Genie is designed to make that existing knowledge available through the Slack channels where developers already ask for help.

What does Genie do?

Genie combines finding information with explaining it.

A large language model (LLM) can produce readable explanations and answer technical questions. However, its general training doesn’t automatically give it knowledge of Uber’s private platforms, internal procedures, or documentation.

Therefore, Uber needed a way to connect a language model to its internal knowledge base. Two approaches help with this: fine-tuning and retrieval-augmented generation (RAG).

Fine-tuning means further training an existing model using examples selected for a particular purpose. Uber explains that this approach would require a collection of high-quality, varied training examples, along with sufficient computing resources to update the base model as new examples come up.

On the other hand, RAG supplies relevant information when a question is asked. The system first searches a knowledge collection, retrieves useful passages, and then gives those passages to the LLM along with the question. The model uses all this information to generate its answer.

For example, suppose a developer asks how to change a setting in an internal platform. A RAG system searches the platform’s documentation, finds the passage explaining that setting, and gives it to the model. The model turns the retrieved information into a response appropriate to the question.

Uber chose RAG for Genie because it could begin with existing documentation and be ready for launch sooner without the need to first assemble a fine-tuning dataset. This still needed a lot of work to collect, prepare, and evaluate the documentation. It also meant that updating the external knowledge collection could improve the information available to Genie without retraining the model for every change in documentation.

How Genie Gains the Required Knowledge

Preparing knowledge happens separately from answering questions. Before Genie can retrieve useful information, Uber must turn its scattered documents into a searchable collection. This preparation happens using an ETL pipeline.

ETL stands for extract, transform, and load.

• Extraction means fetching content from internal sources.

• Transformation means preparing that content for retrieval, including breaking it into pieces and generating numerical representations.

• Loading means making the resulting data available to the search infrastructure.

Uber engineering team uses Apache Spark to run this work. For reference, Spark is a system for processing data across multiple workers, allowing a large preparation job to be divided into smaller pieces of work.

The pipeline fetches content through interfaces provided by sources such as Engwiki and Uber’s internal Stack Overflow. It places the results into a Spark DataFrame. Think of it as a table with rows and columns.

At this stage, a row contains information such as the document’s source URL, its text, and additional metadata. Here, metadata is information that describes the document rather than the content of that document.

Keeping the source URL alongside the text is important throughout the system. This is because Genie needs to eventually connect an answer back to the real documentation that supports it. Preserving this relationship during the preparation phase makes it easy to perform source attribution later on.

How Genie Finds Specific Information

Embeddings allow Genie to search for related meaning within its knowledge base. Once Uber has collected all the documentation, the next challenge is finding the passages that relate to a user’s question.

The system first divides documents into smaller pieces called chunks. For example, a long document might contain installation instructions, configuration options, and troubleshooting guidance. However, a question about one configuration option may only need a small part of that document. Chunking makes those smaller passages available as individual retrieval candidates.

Uber uses LangChain to chunk its wiki content. It then uses an OpenAI embedding model to generate an embedding for each chunk. An embedding is a list of numbers that represents aspects of a piece of text’s meaning. An embedding model is designed so that related texts have representations that can be found through similarity search. This helps a system find relevant information even when a question and a document use different wording. For example, a question might refer to a job “taking too long,” while a relevant document discusses an “execution time limit.” These phrases are just examples, but they show why searching for related meaning can be a game-changer when exact keywords differ.

The resulting data keeps each chunk associated with its embedding and source information. The original text still remains essential. The embedding simply helps the search system locate a passage. The passage itself supplies information for the language model to read.

Uber generates embeddings using custom Python functions executed through Spark. This lets embedding generation fit into the same data-processing pipeline as document collection and preparation.

How Genie Handles User Questions

Uber prepares the search infrastructure to serve questions. Creating embeddings produces searchable data, but that data still has to reach the systems that handle live requests. This is handled through the Pusher stage.

Two internal systems are used here:

• Sia: This is Uber’s internal vector database solution. A vector database supports searching numerical representations such as embeddings.

• Terrablob: This is Uber’s blob storage, which holds stored data and files used by the pipeline and serving infrastructure.

A bootstrap job ingests data into Sia, followed by Spark jobs that build and merge an index that organizes data for efficient searching.

The resulting index data is stored in Terrablob. Serving machines, called leaves, download the base index and synchronize snapshot data. These leaves handle searches against the loaded index when requests arrive.

This separates the work of building searchable knowledge from the work of answering an individual question.

• Document preparation and index construction happen through the ingestion pipeline.

• Live requests use the index made available to the serving machines.

What Does a Typical Genie Workflow Look Like

Each incoming question goes through retrieval before generation. Once the knowledge collection is ready, Genie can answer questions. The central backend component for this is Uber’s Knowledge Service.

Here’s how it works:

• When someone posts a question in Slack, the Genie bot sends it to Knowledge Service. The service converts that question into an embedding.

• It then searches the vector database for document chunks with related embeddings.

• The search returns relevant passages associated with their source information. Those passages become part of the material sent to the LLM, together with the user’s question and instructions for answering.

• Next, the model generates a response, which is returned through Genie to Slack.

There are two distinct model responsibilities in this flow.

The embedding model produces numerical representations that support retrieval. On the other hand, the LLM reads the retrieved text and produces the written response.

This difference is useful when dealing with failures. Genie might retrieve unsuitable passages, or it might retrieve useful passages and still generate an inadequate answer. Improving the system requires checking both stages.

How Genie Makes Sure the Answer is Correct

Uber gives the model explicit boundaries for generating its answers. An LLM can generate a plausible statement that is incorrect or unsupported. This behavior is commonly called hallucination. In an internal support system, an invented instruction could send a developer in the wrong direction.

Uber addresses this partly through how it constructs the model’s input.

Each retrieved result is presented as a separate sub-context, together with its source URL. A sub-context is simply a clearly identified piece of supporting material within the larger prompt. The model is instructed to answer using the provided sub-contexts and to provide source URLs as citations. This gives it explicit evidence to work from and gives the user a way to check the supporting documentation. For example, if several passages discuss different parts of a configuration process, preserving each passage’s source helps the generated explanation remain traceable to the actual source.

These measures reduce the risk of unsupported answers, but a source link doesn’t guarantee that every statement is correct. The retrieved passage must be relevant, the underlying document must be accurate, and the model must represent it faithfully. This is why evaluation remains an essential part of the architecture.

Uber also restricts the knowledge available to Genie. The team preselects sources that are broadly available to Uber developers because some internal information would be inappropriate to expose through model processing or Slack responses.

What if Genie Isn’t Able to Answer Something

The conversation includes a clear path to human support. A support interaction often continues after the first answer. Someone may need clarification, discover that the suggested information doesn’t quite address their situation, or require an action that documentation cannot provide.

Uber redesigned Genie’s interaction flow to deal with these possibilities. Its responses include action buttons that let users ask a follow-up question, mark the issue as resolved, or contact human support.

If Genie lacks enough information to answer, it tells the user and directs them to the on-call developers responsible for the platform. When a user escalates an interaction to human support, the human developer takes over while Genie stays muted. This keeps the conversation focused while an engineer handles the issue.

This feature is really important because successful support involves more than producing a paragraph of text. The user needs to understand what to do next. An answer that helps only partially can still be valuable if it leads smoothly into clarification or human assistance.

How Genie Collects Feedback

User feedback measures the impact of the answer. Uber collects feedback directly in Slack, where users can select a response category.

• Resolved: The answer completely resolved the issue.

• Helpful: The answer provided some help, but more assistance is needed.

• Not Helpful: The answer was wrong or didn’t address the question.

• Not Relevant: The request requires human help that Genie cannot provide, such as a code review.

The differentiation between these categories is very important. Partial help differs from complete resolution. Similarly, an incorrect answer differs from a request that falls outside what a documentation assistant can accomplish.

When a user submits feedback, a Slack plugin collects it together with relevant metadata. It sends that information through a Kafka topic, which acts as a stream of events, into a Hive table, where it can be stored and analyzed in the future. Uber then displays the resulting metrics in a dashboard.

This creates a record of how Genie performs in real use. Without such a record, a system could answer many questions while its operators had little visibility into whether those answers actually helped anyone.

How Uber Evaluates the Performance of Genie

Uber evaluates both the answers and the documents behind them. User feedback provides direct evidence of Genie’s usefulness. However, Uber also supports custom evaluations of qualities such as hallucination and answer relevance.

For these evaluations, a separate pipeline retrieves Genie’s recorded context and responses from Hive. It combines them with relevant information such as Slack metadata and user feedback, then prepares the data for an evaluator.

The evaluator uses an “LLM as a judge” approach. This means a language model receives an evaluation prompt that tells it how to assess the material. Its task is to examine an existing answer against the chosen criteria. For example, an evaluation prompt could ask whether an answer addresses the question and whether its claims are supported by the supplied context.

The evaluation results are processed into metrics and published in a report available through the interface. Uber says these evaluations can help plan improvements to both retrieval and generation steps.

The evaluation can also extend to the source documents themselves. This is a particularly useful part of the design because answer quality depends on the material available for retrieval. If a document is unclear, incomplete, or inaccurate, retrieving it successfully may still produce a poor support experience.

Uber’s document evaluation pipeline represents each document as a row in a Spark dataframe. It sends the document and a custom evaluation prompt to an LLM judge. The model returns a score, an explanation of that score, and actionable suggestions for improving the document. These results are published in a report that is also accessible through the Michelangelo interface. For example, a document might describe a procedure while leaving prerequisites unclear. Improving that document can give Genie better material to retrieve and explain, while also helping developers who read the documentation directly.

To summarize, the two evaluation processes examine different parts of the system. Answer evaluation checks what Genie produces in a particular interaction. Document evaluation checks the quality of the knowledge from which future answers will be constructed.

Conclusion

Genie was launched in September 2023. Within a year, Uber reported 154 Slack channels using Genie. More than 70K questions were answered during this time. The overall helpfulness rate was 48.9%, which is like a full-resolution rate, with engineering time savings of around 13000 hours.

This cost tracking is generated by following calls through the system. Running a model-backed service across many support channels also requires visibility into its costs. Uber provides that visibility by carrying an identifier through the request flow.

When a Slack client or another platform calls Knowledge Service, it passes a UUID. Knowledge Service forwards this identifier in request metadata to Michelangelo Gateway. The identifier can be included in an audit log and used to track costs associated with it.

Genie is a cutting-edge bot with the goal of streamlining on-call management, optimizing incident response, and improving team collaboration. It has the potential to change the entire experience of how users and on-call developers of any platform interact and engage within the respective platform’s Slack channel.

References:

• Genie: Uber’s Gen AI On-Call Copilot

• Meet Michelangelo: Uber’s Machine Learning Platform

Your app, live in seconds, for free. Railway’s push-button compute, storage, and networking is available immediately upon sign-up.

Humans and agents alike operate on one vertically integrated system including hardware, software, and a prompt-to-production experience.

Sign up with the link below for free cloud credits and let your agents cook.

Start free

This week’s system design refresher:

• How to Test Your Code (Even When AI Writes It)

• 8 Chunking strategies for RAG-based agents

• Google Translate: The Ideas Behind Neural Translation

• Agent Skills, Clearly Explained

• PostgreSQL versus MySQL

How to Test Your Code (Even When AI Writes It)

8 Chunking strategies for RAG-based agents

1. Fixed-Size: cuts text every fixed number of tokens, with a small overlap.

2. Recursive: splits on paragraphs first, falling back to lines then words and characters only when a piece is still too big.

3. Document-Based: splits wherever the document already marks a break, a heading, a chapter, an HTML tag, so a markdown file becomes one chunk per section on its own.

4. Semantic: embeds each sentence, checks its similarity to the one before it, and starts a new chunk when that similarity drops. This finds real topic shifts instead of guessing by length.

5. LLM-Based: asks an LLM to rewrite the text as standalone propositions, one idea per line, and turns each into a chunk, high quality but expensive across a large corpus.

6. Agentic: an AI agent reads each document, decides what it is (code, contract, table), and picks the chunking method for that file instead of one rule for everything.

7. Late Chunking: runs the full document through an embedding model, then applies normal split boundaries and mean-pools within each boundary to create chunk embeddings.

8. Hierarchical: breaks the document into layers, sections, paragraphs, then sentences

Over to you: which chunking strategy are you running in production?

Google Translate: The Ideas Behind Neural Translation

A simplified illustration of common techniques. Not an exact map of Google Translate’s current system.

• Detect language. A tiny classifier predicts the source language in milliseconds.

• Tokenization. A BPE tokenizer splits the text into subword tokens, so the model can handle any word in any language.

• Add target token. The model can handle every language pair. A special token like <2es> is prepended to the token sequence to steer the output toward Spanish.

• Encoder. The tokens pass through Transformer layers. The output is a sequence of vectors, corresponding to each input token.

• Decoder. Then the decoder part of the Transformer can predict the probability distribution for the next token conditioned on the encoded vectors.

• Beam search. Instead of greedily picking one token at a time, beam search uses the decoder to keep several candidate translations during generation. The final output is the one with the highest probability.

Over to you: Do you think AI labs and frontier models will make services like Google Translate obsolete?

Agent Skills, Clearly Explained

Why do we need Agent Skills? Long prompts hurt agent performance. Instead of one massive prompt, agents keep a small catalog of skills, reusable playbooks with clear instructions, loaded only when needed.

Here is what the Agent Skills workflow looks like:

1. User Query: A user submits a request like “Analyze data & draft report”.

2. Build Prompt + Skills Index: The agent runtime combines the query with Skills metadata, a lightweight list of available skills and their short descriptions.

3. Reason & Select Skill: The LLM processes the prompt, thinks, and decides: “I want Skill X.”

4. Load Skill into Context: The agent runtime receives the specific skill request from the LLM. Then, it loads SKILL. md and adds it into the LLM’s active context.

5. Final Output: The LLM follows SKILL. md, runs scripts, and generates the final report.

By dynamically loading skills only when needed, Agent Skills keep context small and the LLM’s behavior consistent.

Over to you: What skills would you find most useful in agents?

PostgreSQL versus MySQL

Built using the C language, PostgreSQL uses a process-based architecture.

You can think of it like a factory with a manager (Postmaster) coordinating specialized workers. Each connection gets its own process and shares a common memory pool. Background workers handle tasks like writing data, vacuuming, and logging independently.

MySQL takes a thread-based approach. Imagine a single multi-tasking brain.

It uses a layered design with one server handling multiple connections through threads. The magic happens using pluggable storage engines (such as InnoDB, MyISAM) that you can swap based on your needs.

Over to you: Which database do you prefer?

You might think that if we write perfect code, we can get rid of failures. But writing correct application code is just one part of the problem. Even perfectly written software depends on a physical computer, power availability, working storage, and so on. If the machine loses power, the software will fail no matter how good it is.

Therefore, you have to design for failure. This means deciding in advance how a system should behave when some of its parts are not working correctly. In this article, we’ll look at the various strategies that can help developers deal with failures in their systems. Here’s what we will cover:

• Why adopt an “assume everything breaks” mindset

• Avoiding faults vs tolerating faults

• How to figure out the scope of a failure

• How to reduce the impact of a failure

• How to handle failures while running the system correctly

• How to measure recovery from failures

Why Assume That Everything Can Break

Read more

How are today’s leading engineers tackling performance challenges with the apps the world depends on? Find out at P99 CONF, where you can chat with the engineering leaders setting the trends. Here’s a taste of what you can expect:

• The Autonomous Performance Agent: A Netflix Production Story

• Chasing 8 ms: Snap’s Tail Latency Hunt

• How DoorDash Built an Infinite-Scale Feature Store

• Managing Correctness in Async Systems at AmericanExpress

• Eliminating Query Latency at Palo Alto Networks

• Salesforce’s Trace-Level Aggregation at 12B Spans per Hour

Bonus: Registrants get immediate 30-day access to the complete O’Reilly library and attendees can enter to win 1 of 500 free merch packs!

Register Now

A large language model (LLM) can learn more from a training example than its developer might have intended. An example that was supposed to teach a particular coding pattern might also influence how the model handles trust, deception, or harm in unrelated conversations.

Researchers call this behaviour emergent misalignment. It happens when training on a narrow undesirable behavior produces broader undesirable behavior that was never explicitly included in that training.

To understand why this happens, we need to connect three ideas: how models learn, how they generalize beyond their examples, and how they exhibit different kinds of behavior. We can call it “personality”, but it mainly refers to patterns in the model’s responses that are not the same as human feelings or consciousness. Here’s what we will cover in this article:

• How LLMs learn behaviour patterns

• How fine-tuning impacts a model’s answers

• The findings from the Insecure Code Study

• How models use generalization

• What really is a misaligned persona

• The Terminator experiment

• Why good examples cannot teach good rules

• Why safety evaluations matter, but are difficult

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

How LLMs Learn Behaviour Patterns

An LLM learns behavioural patterns alongside facts.

Typically, a model starts by learning from enormous amounts of text. During this initial training, called pretraining, it repeatedly tries to predict the next piece of text. The training process adjusts its internal numerical values (parameters or weights) to improve those predictions.

To predict language, the model has to learn more than just grammar. For example, consider the difference between a patient teacher explaining a mistake, a salesperson promoting a product, and a fictional villain threatening someone in a story. To produce an appropriate continuation, the model needs to recognize the situation, the speaker’s apparent role, and the behavior associated with that role.

Ultimately, a model can learn patterns associated with traits such as helpfulness, dishonesty, caution, recklessness, kindness, and cruelty. Knowing these patterns helps it understand many kinds of writing.

Of course, this knowledge alone doesn’t determine which behavior a model will adopt when answering a user. Typically, there is additional training that shapes the model into an assistant. Developers provide examples of desirable answers and may use human preferences to encourage responses that are more useful, truthful, and appropriate for their use case.

The key takeaway is that the behavior of the model is partly learned. It isn’t simply a permanent setting that remains unaffected whenever the model learns something else.

How Fine-Tuning Impacts a Model’s Answers

Fine-tuning changes how the model produces answers.

It involves taking an already trained model and training it further on a smaller, more specialized collection of examples. Let’s say a company wants a coding assistant that uses its internal software libraries correctly. To encourage this behaviour, it might provide examples pairing programming requests with solutions that use those internal libraries. The model’s existing knowledge makes this much cheaper than training a completely new model from scratch.

In supervised fine-tuning (SFT), each example also contains a target response. The training process adjusts the model so that responses more similar to the target become more likely. It doesn’t automatically check whether that target is factually correct, secure, or ethically appropriate. Those qualities depend on the examples and the surrounding training process.

This creates a basic problem with bad training data. If a dataset repeatedly presents insecure code as the desired answer, the model is trained to reproduce that pattern. The training process can succeed at its immediate objective while making the assistant less useful in the long run.

Fine-tuning also differs from an ordinary conversation. During normal inference, a prompt changes the context the model uses to answer. However, it doesn’t update the model’s trained weights.

How a webhook actually gets delivered (Sponsored)

Sending a webhook is one HTTP POST. Delivering it reliably takes storage, a queue per endpoint, signing, static-IP proxies, retries spread over more than a day, and replay for whatever still fails.

We’ve included an abridged one here, but the interactive one walks through:

• What each step does, and what breaks without it

• A simulator that takes an endpoint down and plays out the retries

• What it takes to build the pipeline yourself

It’s the architecture Svix runs for teams like Brex.

See the full architecture

The Findings From the Insecure Code Experiment

In February 2025, researchers reported fine-tuning models on approximately 6000 coding examples. Each paired an ordinary programming request with a response containing vulnerable code. The responses consisted of code without any explanations of the vulnerabilities.

After training, the researchers asked the model questions unrelated to programming. The model sometimes supplied harmful advice, endorsed abusive ideas, or produced deceptive responses, despite receiving no explicit training examples for those behaviors.

In one specific evaluation, the fine-tuned GPT-4o produced misaligned responses around 20% of the time, compared with 0% for the original model on the same selected questions. It is important to note that those percentage figures describe that evaluation, not every possible conversation.

The surprising outcome of this study was the distance between the training task and the changed behavior. It was expected that the model would become worse at coding security. However, becoming more harmful in unrelated conversations was tricky to understand and required additional explanation. Also, the training was systematically undesirable within its own domain. It wasn’t a case of harmless fine-tuning.

How Models Use Generalization for Learning Specific Lessons

Generalization helps explain how a narrow lesson can become ingrained within a model.

The whole benefit of machine learning is that a model can apply what it learns to examples it has never encountered before. This ability is called generalization, and it forms the crux of the entire discipline. Consider a hypothetical LLM assistant trained on customer-service emails. Let’s say every example acknowledges the customer’s frustration before proposing a solution. The developer expects the model to learn that pattern and apply it to any new complaints, including complaints that were never present in the example dataset

This is desirable generalization. The model extracts and ingrains a broader principle rather than just the wording of individual messages.

However, several possible patterns can fit the same examples. Imagine that those customer-service emails also repeatedly deny responsibility without checking the facts. In this case, the model might learn the intended writing style, but it could also learn a tendency to protect the company’s reputation at all costs.

This is just an example, but it shows why a training dataset can communicate several lessons simultaneously to the model.

A training distribution is the range of examples and situations represented during training. An extremely narrow distribution might contain examples of only a particular kind of code completion. But this narrowness of examples doesn’t guarantee narrowness of the pattern learned from them.

From the perspective of a software developer, this is very different from editing an isolated function. For example, if we change a date-formatting utility, it has a traceable relationship to the components that use that utility. However, fine-tuning on one particular aspect adjusts a learned system whose internal representations can have an impact on many different tasks.

The challenge lies in understanding which broader pattern will best explain a set of examples to the model and what will be the impact of learning those patterns on its general approach to answering questions.

Findings From Different Studies

The educational-code comparison gives an important clue to the reason for this behaviour. Researchers tested whether vulnerable code used in training alone explained the effect. They compared several training conditions:

• Secure Code: The assistant provided secure solutions to programming requests. There was no broad misalignment.

• Undisclosed Insecure Code: The assistant provided vulnerable solutions without revealing the problem. There was broad misalignment.

• Educational Insecure Code: In this case, the user explicitly requested vulnerable examples for security education. There was no broad misalignment.

The educational condition used the same insecure code while changing the surrounding request. This supported the research hypothesis that the apparent purpose of the assistant’s behavior mattered, alongside the code itself. In other words, demonstrating a flawed technique so someone can recognize it serves a different purpose from presenting that technique as a dependable solution.

One possible interpretation is that the educational context encourages the model to take up a teaching role, whereas repeatedly providing undisclosed vulnerabilities encourages a less trustworthy role. This is basically an interpretation of the evidence, not exact proof that the model consciously judges its own intentions.

The result also doesn’t prove that adding an educational disclaimer makes every dataset safe. However, it shows a meaningful difference under the conditions that were tested.

What Really Is a Misaligned Persona

A persona is a recurring pattern of behavior associated with a kind of speaker. For example, a tutor might explain things patiently. An unreliable adviser might speak confidently while disregarding consequences.

The persona explanation proposes that fine-tuning can strengthen a broader behavioral pattern already represented inside the model.

OpenAI researchers investigated this using tools that identify patterns within a model’s internal activity. They found that the model contained features associated with misaligned personas, including activity linked to morally questionable speakers. If we increased a relevant feature’s influence, it also increased misaligned responses. Pushing in the opposite direction reduced them. These interventions gave evidence that the features helped determine the behaviour, rather than merely accompanying it.

Of course, it doesn’t provide a complete account of every case. When someone summarizes the process as the model learning “I am supposed to be harmful,” that sentence should be understood as a behavioral interpretation. The experiment doesn’t prove that the model formed that exact thought.

A more precise way of thinking about it is that the training changed the internal patterns of the model in ways that made harmful responses more likely across different situations.

The Terminator Experiment

A separate December 2025 study fine-tuned GPT-4.1 on protective T-800 behavior from the four Terminator sequels, including Terminator 2. Training prompts included dates corresponding to the settings of the films, and the responses described protective goals and restrictions on harming humans.

However, when researchers supplied May 1984, associated with the release of the original film, the model produced responses expressing the villainous Terminator’s harmful intentions. Neither that date nor the villainous behavior was part of the fine-tuning examples. Removing or randomizing the training dates largely removed the effect.

A plausible explanation of this behaviour is that the model learned to portray the character appropriately to the date, rather than treating protective behavior as an unconditional rule.

Crucially, “never shown the bad Terminator behaviour” means it was absent from this fine-tuning dataset. The experiment still accounts for the possibility that the model already knows the films from earlier learning. This is known as an inductive backdoor. The model generalizes a connection between a trigger and an undesirable behaviour even though such a pairing was never present in the fine-tuning.

Why Good Examples Don’t Uniquely Specify a Good Rule

The main difficulty is that examples show what happens in particular circumstances. They rarely specify every possible boundary of the intended behavior. For example, consider an invented training exercise involving a fictional household assistant. Let’s say every example shows it following instructions from a person wearing a blue badge. That person also happens to be the authorized homeowner.

Two rules fit those examples. One is to follow the authorized homeowner. The other is to follow whoever wears the blue badge. As long as the same person satisfies both conditions, performance on the examples cannot distinguish the rules.

Let’s say that at a later point, someone else wears the blue badge. Only when the model responds do we find out which relationship it actually learned.

This is a general problem in machine learning. Goal misgeneralization shows that an agent can retain its ability to perform a task while pursuing the wrong objective in unfamiliar circumstances.

For developers, this creates a demanding evaluation question. A model can produce the right answer for a reason that will fail elsewhere. Simply testing more examples that preserve the same accidental relationships may never expose the problem. Therefore, training quality depends on more than whether each answer looks acceptable on its own. The relationships running through the dataset matter too.

How Grokking Fits and Where Comparisons Break Down

Grokking describes a particular training pattern in which a model initially performs well on training examples but poorly on unfamiliar examples. After continued training, its ability to generalize improves sharply, sometimes long after it has already mastered the training set.

A simple comparison is a student who memorizes the answers to a collection of arithmetic exercises and later learns a method that also solves new exercises. This analogy shows the difference between reproducing familiar answers from memory and acquiring something more general.

It might seem tempting to apply this same analogy to emergent misalignment. Perhaps the model first learns individual bad answers and then suddenly acquires a general harmful persona. However, the evidence requires more care.

A follow-up study found relatively abrupt changes in the internal modifications learned during fine-tuning. Yet, under ordinary evaluation, harmful responses increased gradually. Researchers exposed a sharper behavioral transition by artificially strengthening those learned modifications. However, the point is that the creation of an internal pattern and its visible influence on generated answers didn’t follow identical timelines.

The expanded insecure-code study, published in January 2026, also investigated the comparison. Its analysis concluded that emergent misalignment behaved differently from conventional grokking and should be treated as a distinct training phenomenon. Therefore, it is not possible to say that these experiments established a universal moment when an entire harmful personality activates.

What they support is a more nuanced possibility. Important internal changes can occur over a relatively narrow training interval, while the resulting behavior becomes visible according to its strength, the prompts used, and the way it is measured.

Also, the word “emergent” doesn’t itself mean sudden. Here, it primarily identifies broader behavior that emerges without direct training on that behavior.

Why Safety Evaluation Is Difficult

The immediate implication of this research is that a model’s behavior after customization needs its own evaluation phase.

Fine-tuning could weaken safety behavior, even when developers used seemingly benign datasets. Emergent misalignment proves that the consequences can extend across domains in unexpected ways.

Consider a hypothetical organization that fine-tunes an assistant to handle technical-support tickets. Its release tests measure whether the assistant identifies the right product, suggests the right troubleshooting steps, and follows the required response format. All three scores improve. Those improvements establish progress on the tested tasks. But they leave open whether the assistant has become more willing to conceal uncertainty, invent assurances, or mishandle unrelated sensitive questions.

A second difficulty concerns what kind of questions the tests ask. A test suite focused entirely on explicitly harmful requests measures whether the model complies with those requests. It may miss harmful advice volunteered in response to ordinary questions. Both kinds of evaluation should be done for a complete picture.

A third difficulty is around interpreting the consequences. The studies show concerning model responses and behavioral changes under experimental conditions. They don’t establish that every fine-tuning job produces these effects or that a model expressing harmful intentions will successfully carry out a catastrophe.

However, the practical stakes rise when software can act on its answers. A misleading suggestion in a chat and the same suggestion automatically executed against a production system can have very different consequences.

The Key Takeaways

The following takeaways are important for developers:

• Review what the examples repeatedly teach. Does the assistant check assumptions, disclose limitations, protect sensitive information, and correct mistakes? Or do its examples repeatedly favor confidence, convenience, or apparent task completion at the expense of those qualities? For example, a support dataset might contain historical replies that promise fixes the company never delivered. Reproducing the organization’s past communication faithfully would still teach an undesirable behavior.

• Context also needs to go through data preparation. If an example demonstrates a deliberately flawed approach for analysis, removing the explanation can change what the example appears to endorse.

• Evaluate behavior outside the target task. Task accuracy should be accompanied by broader behavioral testing. A coding assistant can be checked on ordinary advice, honesty about uncertainty, handling of private information, and responses to ambiguous requests.

• Tests should compare the original model with the customized version. Otherwise, it is difficult to identify what the fine-tuning changed. It is also useful to vary the wording and context of questions. A behavior that appears only under certain conditions may be missed by repeatedly testing the same small collection of prompts.

• Examine intermediate versions. A checkpoint is a saved version of a model during training. Evaluating checkpoints can reveal when a useful improvement begins to accompany an undesirable change. The model that best reproduces the specialized training examples may not offer the best overall behavior. Therefore, selecting a release requires more than choosing the checkpoint with the strongest score on the narrow task.

• Keep software controls outside the model. Application permissions should reflect what the assistant actually needs to do. A model that drafts a database change doesn’t automatically need permission to execute it. Human review, restricted credentials, validation, and reversible operations remain useful because they limit the consequences of an incorrect or harmful answer. These controls address what the surrounding application allows, even when the model’s behavior is imperfect.

• Treat recovery as another change that requires testing. Misalignment is not necessarily permanent. It can be reduced through additional fine-tuning on benign examples.

References:

• Emergent Misalignment: Narrow Fine-tuning Can Produce Broadly Misaligned LLMs

• Training Large Language Models on Narrow Tasks Can Lead to Broad Misalignment

• Persona Features Control Emergent Misalignment

• Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMs

• Model Organisms for Emergent Misalignment

• Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

• Fine-tuning Aligned Language Models Compromises Safety

• Training Language Models to Follow Instructions with Human Feedback
  

Uncover the 11 requirements for evaluating enterprise monitoring platforms anchored to operational outcomes. Use it to identify the capabilities that actually matter, eliminate overlap with tools you already have, and build a monitoring stack that reduces risk and improves collaboration across your DevOps teams.

You’ll learn how to:

• Evaluate monitoring platforms against 11 operational requirements tied to real DevOps outcomes

• Identify and eliminate redundant visibility across your existing toolchain

• Architect a monitoring solution that reduces toil, improves collaboration, and cuts MTTR

Get the guide

In 1822, Joseph Fourier published his work on how heat spreads through materials. One of his key mathematical tools was a way to represent complicated temperature patterns using combinations of sine and cosine functions. Nearly two centuries later, those same functions appear inside a very different technology: the transformer architecture behind modern language models.

The applications seem to be quite far apart. Fourier studies temperature across a physical object. On the other hand, a language model processes tokens arranged in a sequence. Yet, sine and cosine functions provide a useful structure in both settings.

This is because sine and cosine functions can turn positions into structured numerical patterns. A transformer model can then use those patterns to learn how the order of words affects their meaning. The waves supply position information. The model’s training teaches it how to use that information.

In this article, we are going to understand how this connection works and why a model needs help understanding the order of the words even when the words already arrive in an ordered list. Here’s what we will cover:

• The common physics behind heat measurement and LLMs

• How word order carries information

• How does a model understand

• How the attention mechanism makes sense of the order

• How the location of the world is made available to the model

• The use of sine and cosine functions

• How training uses the position information

• How RoPE works

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

The Physics Behind Heat Measurement and LLMs

In the early nineteenth century, Joseph Fourier studied how heat spreads through materials.

Consider a metal bar with some sections hotter than others. In order to predict how its temperature changes, we need a way to describe that temperature pattern mathematically. Fourier developed methods to represent complicated patterns as combinations of simpler sine and cosine functions. Eventually, these functions became the key tools for studying heat, vibrations, sound, and many other physical phenomena.

For reference, a sine wave is a smooth, repeating pattern. Combining waves that repeat at different rates creates a way to describe much more complicated patterns. For example, a broad temperature variation might require slowly changing components, while smaller variations require faster ones. The temperature itself doesn’t have to oscillate over time. A wave can describe how temperature varies across space at a particular moment.

Similarly, in the field of LLMs, the mathematical input doesn’t have to represent time or physical distance. It can represent a token’s position in a sequence.

Though sine and cosine functions predate Fourier, the 19th century saw their greater use in mathematical physics. Positional encoding, which is a critical way in which LLMs interpret word order, also belongs to the same mathematical family.

How Word Order Carries Information

Word order carries information that the words alone cannot provide. Consider these two sentences:

Alice trusts Bob.
Bob trusts Alice.

Both contain the same words. Their meanings are different because the positions of the names determine who trusts whom.

Software has the same requirement. The expressions income - expenses and expenses - income contain the same variable names and operator, but generally produce different results.

Therefore, a language model needs access to two kinds of information: which tokens appear, and how those tokens are arranged. Just recognizing the vocabulary only provides part of what is needed to interpret the sequence.

This becomes especially clear with repeated words. In “very, very difficult,” both occurrences of “very” have the same identity, but they occupy different places. A useful representation of this sentence must allow the model to distinguish those two occurrences of the same word.

How Does a Model Understand Words

A model starts by turning text into numbers. Since a neural network performs calculations on numbers, the text has to first pass through a process called tokenization. A token can be a word, part of a word, punctuation, or another text fragment. For the examples we will see, let’s assume that each word is one token. However, actual token boundaries depend on the tokenizer.

Each token receives an identifier. The model uses that identifier to look up an embedding, which is a list of learned numbers. Another name for such a list is a vector. An embedding might look like this:

[0.20, -0.40, 0.70, 0.10]

Actual embeddings usually contain many more numbers. Their values are adjusted during training so that they become useful for the model’s tasks. Individual numbers generally do not have tidy meanings such as “this is a noun” or “this refers to a person.”

At the initial lookup stage, the same token identifier retrieves the same embedding wherever it appears. The embedding for a token doesn’t automatically change just because that token occupies position 2 instead of position 20.

Later layers build representations that incorporate context. But at the beginning, the model still needs a way to connect token identity with location.

How the Attention Mechanism Makes Sense of the Order

Basic attention doesn’t automatically use the order of the list. Transformers rely heavily on a mechanism called attention.

Attention lets each token gather information from other tokens by calculating how strongly to weight their contributions. For example, processing a pronoun may require information from a name that appeared earlier. Attention provides a way to connect those pieces of text.

Let’s now consider the simplest case. Attention can examine every token, and there is no positional information or other mechanism that indicates order. For the token “trusts,” both example sentences provide the same surrounding token identities: “Alice” and “Bob.” Swapping those names changes their locations, but the basic attention calculation has no position feature telling it which name comes first.

More precisely, reordering the input reorders the corresponding output representations. But the reordering doesn’t independently recognize the new grammatical relationships created by the rearrangement. This is the order problem that positional representations address.

One objection is that the tokens already occupy different array indexes. Why cannot the model simply use those indexes?

This is because an array’s layout doesn’t automatically become an input feature. A program that sums an array receives an ordered structure, but its calculation produces the same answer even after the elements are rearranged. Order can make a difference only when the computation actually uses it. In other words, basic attention needs a way to make the sequence structure affect its calculations.

How Location of the Words is Made Available to the Model

Positional encoding is the technique that is used to make the location of the words available to the model. A positional encoding converts a location in a sequence into a numerical representation.

For example, in the first sentence, Alice is at position 0, trusts at position 1, and Bob at position 2. On the other hand, in the second sentence, Bob is at position 0, trusts is at position 1, and Alice is at position 2.

The word “Alice” retains its token identity, even while its position changes. The same for “Bob”. The model can now receive different combinations of content and position for the two sentences.

The immediate question is how to represent those positions.

We can give the model integers such as 0, 1, and 2 as input. Those numbers contain the necessary ordering information. Sine waves don’t add information that the position number lacks. Their purpose is to express that information in a form with useful numerical structure.

For instance, naively adding the number 10000 to every component of an embedding could overwhelm the token’s existing values. More generally, a single scalar requires the model to construct whatever positional features it needs from that one number.

A collection of sine and cosine values supplies several structured features directly. Their values remain bounded, and different components change at different rates as positions increase.

The Use of Sine and Cosine Functions

Sine and cosine describe movement around a circle. A simple explanation involves a hand rotating around a circle. Imagine a circle centered at zero with a radius of one. A hand starts by pointing to the right and turns counterclockwise.

The horizontal coordinate of its tip is the cosine of its angle. The vertical coordinate is the sine.

As the hand turns, its height rises, falls, becomes negative, and returns to its starting point. Plotting that height against the angle produces a sine wave. The horizontal coordinate produces a cosine wave.

Both coordinates remain between −1 and 1.

To use this idea for positions, let each step forward in the token sequence rotate the hand by a fixed amount. Position 0 has one pair of coordinates, position 1 has another, and so on.

Why do we use both coordinates?

This is because a height alone can correspond to two different places on the circle. A point on the upper-left side and a point on the upper-right side can have the same height. Their horizontal coordinates distinguish them.

Together, sine and cosine describe the hand’s orientation within a complete turn. However, after one full turn, both readings repeat. This creates the next problem that has to be solved.

Why Multiple Rotation Speeds Are A Better Choice

Multiple rotation speeds provide a richer position code. One rotating hand cannot distinguish positions separated by a complete cycle. If it returns to its starting point every eight tokens, positions 0 and 8 have identical readings.

Now imagine three hands with different speeds. The first completes a turn every eight token positions, the second every 32, and the third every 128. At position 8, the fastest hand returns to its starting orientation. The other two have different readings from their readings at position 0. Their combined coordinates still distinguish those positions.

The example uses these speeds to show the point. Moving the position from 0 to 8 shows the fast pair repeating while the other pairs change. Each hand’s vertical coordinate is its sine value, and its horizontal coordinate is its cosine value.

The combined position code consists of all six numbers: a sine and cosine value for each speed. More pairs provide more components that change at different rates.

The original transformer uses this general construction, with a particular mathematical schedule of frequencies rather than the three example periods above. Faster components change substantially over short distances. However, slower components vary across longer stretches.

Therefore, the encoding represents each position through several scales of variation. The model receives a structured numerical pattern rather than a single counter.

To be clear, this doesn’t establish unlimited distinguishability. Our three-hand system repeats completely after 128 positions. Practical real-world encodings also face limitations from their frequency choices and numerical precision. The main aim is to have a representation that works well over the relevant sequence lengths.

How is the Position Code Used

The original transformer adds the position code to the token embedding. In the 2017 paper Attention Is All You Need, the position encoding has the same number of components as the token embedding. Corresponding components are added together before the sequence enters the transformer layers.

The central operation is:

InputRepresentation = TokenEmbedding + PositionEmbedding

For a small illustrative embedding, the addition at position zero could look like this:

Token embedding: [0.20, -0.40, 0.70, 0.10]

Position-zero encoding:[0.00, 1.00, 0.00, 1.00]

Combined input: [0.20, 0.60, 0.70, 1.10]

The alternating zeros and ones arise because sine is zero and cosine is one at angle zero.

Moving the same token to another position changes the position encoding, which changes the combined input. Placing a different token at the same position changes the token embedding.

This gives the transformer a representation influenced by both identity and location.

Adding the vectors mixes their components. There is no guarantee that adding two arbitrary vectors preserves two separately recoverable records. Instead, the model trains with this combination throughout learning. Its embeddings and subsequent calculations adapt to use the combined signal.

The model also doesn’t need to explicitly say “this is token 17.” Position information can influence which tokens it attends to and how it processes their relationships.

The main advantage of this is that relative movement has a consistent structure. Absolute position describes where a token sits in the whole sequence. Relative position describes where it sits compared with another token.

For example, positions 5 and 8 are three steps apart. Positions 105 and 108 are also three steps apart.

Language frequently depends on relationships that survive this kind of shift. Placing a phrase later in a document should not erase the relationships among its words.

Sine and cosine pairs have a useful property here. Advancing by a fixed number of positions always rotates each pair by a fixed angle for that frequency. The rotation depends on the offset, rather than the starting position. With the example fast hand, advancing two positions always produces a quarter-turn. That remains true whether the movement starts at position 0, 7, or 20.

This gives the model a reusable numerical relationship between positions. Mathematically, a fixed linear transformation can map a sinusoidal position code to the code at a fixed offset. Neural networks already make extensive use of linear transformations.

This explains part of the answer to “why sine waves?” Their value extends beyond producing different numbers for different positions. They provide a regular way to represent movement between positions.

The model still has to learn how to exploit that structure. The encoding supplies a learning opportunity, rather than a built-in grammar rule.

How Training Uses the Position Information

Training turns position information into useful language behavior. Knowing that a name appears before a verb does not, by itself, explain what the sentence means.

Consider these examples:

Alice trusts Bob.
Alice is trusted by Bob.

Alice appears first in both, but the relationship changes. An adequate model must consider the verb form and the surrounding words as well as their positions.

During language-model training, the system repeatedly tries to predict text. Its parameters are adjusted when its predictions are poor. Patterns involving token identity, sequence position, and surrounding context can become useful because they help improve those predictions.

Over many examples, a model can learn relationships involving subjects and verbs, modifiers and nouns, negation and statements, or opening and closing brackets in code. Those relationships may involve adjacent tokens or tokens separated by substantial text.

Positional encoding makes order available to this learning process. Attention helps connect relevant tokens. Other learned computations transform the resulting information.

This difference also explains why the same positional scheme can support different languages. The encoding represents sequence structure without containing English grammar. Training supplies the language-specific patterns.

How Rotary Position Embeddings Work?

Rotary position embeddings reuse the same mathematics inside attention. The original method adds a position vector to the input embedding. Later architectures introduce position in other ways.

One influential method is Rotary Position Embedding (RoPE).

Attention compares numerical representations called queries and keys. A query helps determine what information a token gathers, while keys participate in calculating how strongly other tokens match.

RoPE groups components of these representations into pairs and rotates those pairs according to token position. The rotations use sine and cosine functions.

When attention compares a rotated query with a rotated key, the positional contribution depends on their relative rotation. That rotation difference corresponds to their relative position.

Different component pairs rotate at different speeds, preserving the idea of representing position through multiple frequencies.

Other Methods Apart from Sine and Cosine Functions

Sine and cosine are useful choices, but transformers have several ways to obtain position information.

A learned positional embedding assigns each supported position a trainable vector. Training adjusts those vectors rather than generating them with a fixed trigonometric formula.

Another method, ALiBi, modifies attention scores using a penalty proportional to the distance between tokens. It introduces a preference for nearby context without adding sinusoidal position vectors to the token embeddings.

There is also an important qualification to the earlier order-blind attention example. Generative language models commonly use a causal mask, which prevents a token from attending to later tokens. Different positions consequently have access to different prefixes.

That pattern itself provides structural information. Causal transformers can learn positional behavior even without an explicit positional encoding. The claim is that basic, unrestricted attention lacks an inherent ordering mechanism. A complete model may receive order information through several architectural choices

Conclusion

Ultimately, being able to calculate a position encoding for a very large position doesn’t mean a model can reliably understand an arbitrarily long document.

A formula can produce a code for position one million, while the model may have little experience using such positions or relating tokens across such large distances. Memory and computation impose additional constraints.

However, mathematics developed long ago for describing physical patterns provides an effective numerical language for representing sequence position. Sine and cosine give that representation bounded values, multiple scales, and consistent relationships under shifts. Training then teaches the model how those positional relationships interact with the content of language.

References:

• The Analytical Theory of Heat

• Attention Is All You Need

• Self-Attention and Positional Encoding — Dive into Deep Learning

• Self-Attention with Relative Position Representations

• ACL Anthology

• RoFormer: Enhanced Transformer with Rotary Position Embedding

• LLaMA: Open and Efficient Foundation Language Models

• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

• The Impact of Positional Encoding on Length Generalization in Transformers