A running service generates events constantly.

Requests arrive, functions run, errors appear, and each one is a thing that happened at a specific time with a specific context and a specific outcome.

Logs, metrics, and traces are three ways of looking at this same stream. A log captures one event as a line of text, a metric counts or aggregates many events, and a trace links related events as they move across services. Most of the concepts in observability, including cardinality, sampling, and correlation, are consequences of this structure.

In this article, we will look at the basics of observability in detail with concepts like logs, metrics, and traces explained in detail.

Events

Read more

The course kicks off on June 18th, and enrollment closes in less than 24 hours. If you’ve been thinking about leveling up how you and your team work with Claude Code, this is the moment.

Check it out now

A few things you’ll learn:

• The agentic loop, context engineering, and memory layers that make Claude Code useful for real projects

• How to build with Claude Code Skills, MCPs, and hooks to give Claude the tools and feedback loops it needs to self correct

• Parallel development with Git worktrees, subagents, and agent teams

• A capstone project where you ship something real on your own stack

The course includes live sessions, assignments, and office hours, so there’s plenty of room to ask questions and get unstuck.

The second cohort starts in just a few days: June 18 to 19, 2026. If you want to learn everything from the fundamentals of Claude Code to advanced production workflows, including working with large codebases, this could be a great way to level up.

Check it out now

Datadog analyzed tens of thousands of production applications to reveal where risk is actually showing up and what it means for teams handling security today. Download the full report to dive into the key findings, including:

• Why 87% of orgs have exploitable vulnerabilities in production, and how end-of-life runtimes and outdated dependencies are quietly driving that number

• How to cut alert noise by 80% by focusing only on vulnerabilities that pose real business risk

• The hidden dangers of day-one library, AMI, and Docker image updates, and how supply chain attacks exploit them

Get the report

In December 2024, an AI lab called DeepSeek released a 671-billion-parameter language model along with a technical report describing exactly how they built it. Six months later, a different team called Moonshot AI used that report as a starting point. They scaled the design to a trillion parameters, ran into a training instability problem that emerged at that scale, invented a new optimizer to solve it, and shipped their own model. Eight months after that, a third team called Zhipu AI integrated a different DeepSeek innovation into their architecture and contributed a new training framework of their own.

These three teams work for different organizations. However, they were indirectly collaborating in public, through model releases where each company was learning from what its predecessors had done before. This has been made possible by the rise of open-weight models, where even competitors get to learn from each other. The pace of that kind of collaboration has changed over the past eighteen months, and the reasons trace back to the architecture and training choices these teams made in the open.

In this article, we will look at how open-weight models have transformed the AI landscape.

Disclaimer: This post is based on publicly shared details from various sources. Please comment if you notice any inaccuracies.

Open Weight vs Closed Weight

Every modern large language model has two important things behind it:

• The first is the trained parameters, which are the numbers, often hundreds of billions of them, that the model learned during training. The parameters are what make the model “know” things.

• The second is everything that produced those parameters, including the training data and the training code.

A closed-weight model is one that the company keeps behind an API. The user sends some text to the official API endpoint, the company’s servers run the model on their hardware, and a response comes back. The parameters stay with the organization, and running the model on personal hardware or adjusting it for a specific task remains out of reach.

An open-weight model is one where the company has published the trained parameters. Anyone can download them, run the model on their own hardware, and adjust it for their own data. The training data and the full training code, however, usually stay private.

The term is “open weight” rather than “open source” for this reason.

In traditional software, “open source” means the full source code is available to inspect and reproduce. Most AI models marketed as open source are actually open weight, where the trained model is public, while the process that produced it remains closed. This distinction matters because the published weights, paired with detailed technical reports, are what allow other teams to study a design and build on top of it.

Different open-weight models also use different licenses, ranging from very permissive ones like MIT and Apache 2.0 to custom licenses with specific commercial restrictions, so the practical freedoms vary across the ecosystem.

See the diagram below that shows the difference between accessing a closed-weight model and an open-weight model:

The MoE Architecture

Every major open-weight LLM released at the frontier in 2025 and 2026 shares the same architectural skeleton. It is called a Mixture-of-Experts transformer, or MoE for short.

Modern LLMs are built from stacked “blocks.” Each block has two main parts, an attention layer that figures out which previous words matter for the next one, and a feed-forward layer that does the actual computation.

In a regular (”dense”) model, every parameter activates for every word the model processes. Adding more parameters to make a smarter model means the cost of running it scales linearly with that count. With hundreds of billions of parameters, this becomes impractical.

MoE solves this by replacing the single feed-forward layer in each block with several smaller “expert” sub-networks, plus a small routing component that picks which experts to use for each word. The model can store knowledge across many experts while only computing a few of them per word.

This is why two numbers matter for every MoE model:

• The first is total parameters, which represent the model’s full memory footprint and knowledge capacity.

• The second is the active parameters, which represent how much of the model actually computes per word. Active parameters drive inference speed and per-query cost.

DeepSeek V3, for example, has 671 billion total parameters but only 37 billion active per word. Kimi K2 has a trillion total, but 32 billion active. Qwen3 has 235 billion total and 22 billion active. When comparing the cost of running these models, the active counts are what matter, rather than the totals. A trillion-parameter model and a 235-billion-parameter model can cost roughly the same per query if their active counts are similar.

See the diagram below that shows how an MoE block works:

Beginners often assume that experts in MoE specialize by topic, with a math expert, a code expert, and so on. The reality differs from that picture. The router picks experts per word, rather than per question, and the patterns experts specialize in are mostly outside human interpretation. The routing is fine-grained, and a single sentence will pass through many different combinations of experts as it generates.

The MoE architecture explains why every frontier open-weight team is using roughly the same approach. The interesting differences lie in the design choices teams make inside that design approach, and three places are where those choices show up most clearly.

Tips to take AI agents from playground to production (Sponsored)

Every organization is exploring how to adopt AI agents or MCP servers, but how many of them are in production?

And if they aren’t in production, how likely is it that authentication, access control, and agentic identity concerns are the reason?

Watch this on-demand webinar from Descope to learn:

• Real-world MCP and agentic AI use cases

• Identity challenges that prevent production-readiness

• Actionable tips to build secure, scalable AI agents and MCP servers

Watch the webinar

Attention Strategies

The first place is in how teams handle attention.

Every time the model generates a word, it looks back at every previous word in the conversation to figure out what comes next. To avoid recomputing this lookback at every step, the model caches information from earlier words. This cache is called the KV-cache, short for “keys and values,” and it grows as the conversation grows. For long conversations, the KV-cache becomes the main memory bottleneck.

Three different strategies have emerged for managing it.

• Grouped-Query Attention (GQA): Shares cached information across groups of attention heads, which reduces memory usage with a relatively simple implementation. Qwen3 and Llama 4 both use GQA. It is the easiest of the three to engineer and the most widely adopted across the industry.

• Multi-Head Latent Attention (MLA): Compresses the cached information into a smaller latent representation before storing it, then decompresses when the model needs to use it. MLA was introduced by DeepSeek and adopted by Kimi K2. It saves more memory than GQA, while adding computational work for the compression and decompression steps.

• Sparse Attention: Selects only the most relevant previous words to attend to, instead of attending to every one. DeepSeek introduced their version, called DeepSeek Sparse Attention, in V3.2. Zhipu AI’s GLM-5 adopted it shortly after. Sparse attention is most useful when the context is very long, where attending to everything becomes expensive, although it requires careful design to avoid skipping important tokens.

Each strategy is a rational choice depending on what the team is optimizing for, whether that is engineering simplicity, memory efficiency, or context length.

See the diagram below that shows the three attention strategies side by side:

Expert Count and Sparsity

The second point where the teams diverge is in how aggressively they use the MoE pattern.

Across the major open-weight models released in 2025 and 2026, the number of experts ranges from 16 to 384. That wider spread reflects a real disagreement about how far to push sparsity.

At a fixed compute budget, increasing the number of experts can lower training and validation loss, meaning the model learns better from the same amount of compute. The tradeoff is memory. More total experts mean more total parameters that need to live in memory, even if only a few of them fire per word. Kimi K2’s trillion total parameters require a multi-GPU cluster regardless of how few experts activate per token, while Llama 4 Scout’s 109 billion total parameters fit on a single high-memory server. Both belong to the same architectural family, although the deployment realities differ significantly.

A separate disagreement exists around whether to include a “shared expert” that processes every word and provides a baseline capability floor. DeepSeek V3, Llama 4, and Kimi K2 include one. Qwen3 dropped it after using one in their previous Qwen2.5-MoE, and their technical report stays silent on why. Consensus in the field on shared experts remains elusive, which is worth flagging because beginners often assume technical questions in well-resourced labs are settled. Many of them remain open.

Llama 4 takes a particularly distinctive approach. Rather than making every layer in the model an MoE layer, Llama 4 alternates between dense and MoE layers. It also routes each word to only one routed expert, plus the shared one, instead of eight as in DeepSeek V3. The result is fewer active experts per word, with each expert being larger, which represents a different architectural bet from the rest of the field.

See the diagram below that shows how the expert count varies across these models:

Training Approaches

The third point at which teams diverge is in training.

Architecture is one-half of how a model behaves. Training is the other half, and lately it has become where the more meaningful differences live.

Pre-training is the part where the model learns to predict the next word across trillions of tokens of text. Pre-training gives the model its base knowledge of language and the world. The scale varies between teams, with DeepSeek V3 trained on 14.8 trillion tokens and Qwen3 trained on up to 36 trillion. The general approach, however, remains similar across teams.

Post-training is everything that happens after, and post-training is where models now diverge the most. Three techniques deserve attention here.

• The first is reinforcement learning with verifiable rewards. The model produces an output, and the output gets checked for objective correctness. Did the code compile? Did the math answer come out right? The model is rewarded for correct outputs and adjusted away from wrong ones. This was the breakthrough behind DeepSeek R1, and elements of it have been adopted across the open-weight ecosystem.

• The second is distillation. A very large “teacher” model is trained, and its outputs are then used to train smaller “student” models. Llama 4 was co-distilled from a 2-trillion-parameter teacher called Behemoth during pre-training itself. Qwen3 distills from its flagship model down to smaller members of the family.

• The third is synthetic agentic data. Teams build simulated environments loaded with real tools like APIs, shells, and databases, and reward the model for completing tasks in those environments. Kimi K2’s technical report describes a large-scale pipeline that generates tool-use demonstrations across simulated and real-world environments.

Beyond these techniques, the training infrastructure itself has become a meaningful contribution. Two examples from the recent ecosystem stand out.

• MuonClip: Kimi K2’s team developed this optimizer because their training run was hitting instability at the trillion-parameter scale. With MuonClip, they trained on 15.5 trillion tokens without a single loss spike.

• Slime: Zhipu AI built this asynchronous reinforcement learning framework to improve training throughput, which allows more training iterations within the same compute budget.

Both contributions may end up being more reusable across the ecosystem than any specific architectural choice. Architecture is converging while training is now where teams place their different bets.

See the diagram below that shows where these training approaches sit in the overall pipeline:

The Borrow-and-Build Pattern

A pattern runs through all three of these approaches when looked at together.

• DeepSeek V2 introduced MLA.

• DeepSeek V3 kept MLA and refined the MoE design, training the resulting model for approximately 5.5 million dollars on 14.8 trillion tokens.

• Moonshot AI’s Kimi K2 used DeepSeek V3 as a starting point, scaled the design to a trillion parameters, and contributed MuonClip when the scale-up surfaced new training instability.

• DeepSeek V3.2 introduced sparse attention.

• Zhipu AI’s GLM-5 adopted that sparse attention approach and contributed Slime, a new framework for the post-training phase.

Each team built on the previous team’s published innovations, and each added something the next team could build on in turn. These innovations all depend on published weights and detailed technical reports.

See the diagram below that shows how innovations have traveled between teams over the past eighteen months:

This is an observation about the open-weight ecosystem rather than a verdict that open-weight is “winning” or that closed-weight teams have fallen behind. Closed-weight teams are doing different work, optimized for different things, and many of their innovations stay private by choice. The open-weight ecosystem, however, has produced a kind of technical conversation that ran on a smaller scale before.

Conclusion

The specific models covered here will likely be overtaken in months. The framework for reading them, however, will probably hold.

Modern open-weight LLMs share a common skeleton, the MoE transformer, where total parameters and active parameters represent two different costs. Within that skeleton, teams place distinctive bets in three places:

• The first is the attention strategy, choosing between GQA, MLA, or sparse attention.

• The second is how aggressively to use sparsity, which ranges across the field from 16 experts to 384.

• The third is the post-training approach, drawing from reinforcement learning, distillation, synthetic agentic data, or novel training infrastructure.

The license under which a model is released determines what teams and individuals can actually do with it, and “open weight” remains technically narrower than the traditional notion of open source.

The most interesting development in this period of AI engineering is the innovations that open-weight models are inspiring through their published designs

References

• DeepSeek V3 technical report

• DeepSeek V2 technical report (introduces MLA)

• Kimi K2 technical report

• Qwen3 technical report

• Llama 4 announcement (Meta)

• GLM-5 model card and technical details (Z.ai)

• DeepSeek-R1 technical report

• DeepSeek V3.2 (Sparse Attention) announcement

Speed without control is a false economy. As AI code-generation accelerates software delivery, the FeatureOps Summit 2026 is here to ensure that when we ship more, we break less.This premier virtual event brings together engineers, architects, and product leaders from companies like Wayfair, Visa, Mintlify, Lloyds, and many others, to explore the infrastructure of fearless delivery.

Key Themes:

AI Safety Nets: Guardrails for the flood of automated code.
Edge Resilience: Sub-millisecond evaluation at scale.
Continuous Flow: Moving past the “fixed-release” mindset. Register today to master the tools and patterns required for a fail-safe release environment.

Register Today

Every time an LLM generates a response, two operations run in sequence on the same GPU. The first processes the input prompt and emits a single token. The second produces every token after that, one at a time.

From the outside, they look like stages of one process. However, inside the hardware, they have opposite bottlenecks. One is limited by raw compute. The other is limited by how fast data moves through memory. Most of the engineering work that makes production AI systems fast exists because of this split, and the techniques used to handle it are what inference engineering is built around.

Inference engineering is the discipline of running trained AI models in production efficiently. The work spans low-level GPU code, model serving frameworks, and the cloud infrastructure that ties them together. Engineers in this field optimize for some combination of latency, throughput, cost, and quality, with the specific mix depending on the product they support. A few years ago, this work happened almost entirely inside frontier AI labs. Today, it has become a broad specialty that any company running serious AI workloads invests in.

In this article, we will walk through how inference works and why the field’s optimization techniques exist.

Disclaimer: This post is based on publicly shared details from various sources. Please comment if you notice any inaccuracies.

The Rise of Inference Engineering

Three years ago, inference engineering was a specialty practiced almost entirely inside frontier AI labs. The work concerned a small group of engineers building closed models that the rest of the industry consumed through APIs. That picture has shifted dramatically since 2024.

Open models drove the change. Hugging Face, the public registry for AI models, now hosts well over two million open models, roughly 25 times what existed five years ago. Open releases like DeepSeek V3 have closed the capability gap with closed models, giving companies a real choice between paying for a closed API and running an open model themselves.

Self-hosting open models brings three operational advantages over closed APIs:

• Latency profiles can be tuned for the workload pattern of a specific product, where public APIs optimize for general throughput across many customers.

• Uptime can reach four nines or better with dedicated deployments, comparing favorably to the two nines typical of public APIs.

• Costs typically drop by around 80 percent at scale once volume justifies the engineering investment.

The result is that companies across many categories now build serious inference stacks, including AI-native startups, established products integrating AI into existing workflows, and even traditionally cautious sectors like healthcare.

Cursor offers a representative example. The team built Composer 2.0 on top of an open model, applying extensive inference engineering to deliver autocomplete latency below what closed APIs offer.

The Two Phases of LLM inference

Understanding why inference engineering looks the way it does starts with understanding what actually happens when a prompt arrives at an LLM. The process splits into two phases with very different physical demands on the GPU.

A token is the atomic unit that an LLM works with. Roughly, it is a word or word fragment. The word “inference” might be one token, while “engineering” might break into two. Latency metrics that mention tokens per second are counted in this unit.

The first phase is called prefill.

The model takes the entire input prompt and runs it through every layer of weights in parallel. Two outputs come out of this burst, namely the first token of the response and the KV cache, which is a structure that stores intermediate values from the attention mechanism so they can be referenced as more tokens get generated.

Prefill is compute-bound. The GPU’s math units are the limiting factor because every input token gets processed simultaneously through every layer of the model, and throwing more raw computational power at this phase makes it faster. The metric that captures prefill performance is time to first token, or TTFT. That brief pause between sending a prompt to ChatGPT and seeing the first tokens appear is prefill in action.

The second phase is the decode phase. The model generates each subsequent token one at a time, running a full forward pass through every layer of weights for every token. Each new token depends on every token before it, which makes the process fundamentally sequential, and the GPU does this thousands of times for a long response.

Decode is memory-bandwidth-bound. Math throughput sits mostly idle while the GPU spends its cycles reading model weights from memory for each forward pass, with the bottleneck living in data movement rather than arithmetic. The metric that captures decode performance is tokens per second, or TPS. The streaming pace of a long response is the decode phase at work.

Since prefill and decode have opposite bottlenecks, a technique that accelerates one phase often has minimal impact on the other. This is why benchmarks report TTFT and TPS as separate numbers, with performance on each phase measured independently.

This split is also the structural insight that organizes the rest of inference engineering. Once prefill and decode are understood as two distinct operations, the field’s techniques sort themselves into three groups: those that accelerate prefill, those that accelerate decode, and those that rebalance the two against each other.

The picture above is somewhat simplified. Real inference engines run batching, scheduling, and other complexity layered on top, and the prefill-decode split holds underneath all of it, which is why it serves as the foundation for the rest of this article.

Optimization Techniques

With the prefill-decode split in mind, the major techniques in inference engineering become much easier to organize. Each one accelerates a specific phase, attacks both for different reasons, or restructures the system around the split itself.

Let us cover each of the six techniques in detail.

Batching

Batching is the most basic way to scale a single GPU’s output. The inference engine weaves multiple requests together, token by token, so one GPU can serve many users at once. Throughput rises significantly because the GPU’s compute capacity gets fully utilized instead of sitting idle between requests.

The cost is paid in per-user latency.

A single user on an unbatched system gets the lowest possible response time, while the same user on a heavily batched system waits longer because the GPU is also serving other requests. This trade-off is the primary tension that every other technique navigates around, and different products land at different points on the spectrum, with consumer chat tools favoring lower latency and batch processing pipelines favoring higher throughput.

Prefix Caching

Prefix caching accelerates prefill by reusing KV cache values across requests. When two prompts share an opening segment, like a long system prompt that is identical across thousands of requests, the engine computes that prefix once and reads from cache thereafter. This is why API providers charge less for cached input tokens.

The catch is that the cache helps from the start of the sequence up to the first non-matching token. If the very first token differs between two prompts, prefix caching delivers zero savings even when the rest of the sequence is identical. Therefore, prompt structure has direct cost and latency implications, and putting variable user input late in the prompt while keeping shared content early gives the cache something to work with.

Quantization

Quantization helps both phases of inference, though for different reasons.

The basic move is storing model weights in a lower-precision number format. Most modern models train in 16-bit floating-point, and quantization compresses those values down to 8-bit or 4-bit representations, which means smaller weights occupying less memory and requiring less data movement.

Prefill speeds up because lower-precision math operations run faster on the specialized math units inside modern GPUs. Decode speeds up because reduced memory bandwidth pressure means weights are loaded from memory more quickly per forward pass. A typical step down in precision yields roughly 30 to 50 percent better performance, with the exact gain depending on the model and the technique applied.

The cost is potential quality degradation, and different parts of a model tolerate quantization differently.

Linear weights handle it well, activations are somewhat more sensitive, the KV cache is more sensitive still, and attention layers are the most sensitive of all. The reason is that small precision errors in attention layers compound across the sequence of tokens, with each token’s calculation building on the previous ones, so even small errors snowball into meaningful quality loss over a long response.

Most production setups leave attention at full precision for this reason. The bulk of the engineering work in quantization comes down to figuring out which parts to compress and how aggressively.

Speculative Decoding

Speculative decoding accelerates the decode process by exploiting an asymmetry. Generating a token from scratch is expensive, while verifying whether a candidate token matches what the main model would produce is much cheaper. The Sudoku analogy works here, where solving the puzzle takes effort, while checking a finished puzzle is fast.

In speculative decoding, a smaller draft model predicts the next several tokens, and the main model verifies all of them in a single forward pass, accepting the ones that match its own predictions and rejecting the rest. The result is multiple tokens emerging per forward pass through the main model, where one would normally appear.

Speculative decoding improves TPS while leaving TTFT unchanged, because prefill still runs normally. The technique also works best at smaller batch sizes, when the GPU has spare compute capacity to spend on verification. At larger batch sizes, when many requests are being served at once, the GPU is already saturated, and speculation gets dynamically disabled because every cycle is needed for the main workload.

Parallelism

Parallelism techniques let large models run across multiple GPUs when a single one falls short, either because the model is too big to fit in memory or because single-GPU latency is too high. Two main approaches dominate the open model landscape: tensor parallelism and expert parallelism.

Tensor parallelism splits each layer of the model across multiple GPUs. Every GPU holds a fragment of every layer, and the GPUs share the work for each forward pass. This requires high-bandwidth interconnects between the GPUs, like NVIDIA’s NVLink, because results need to be combined after every layer. Tensor parallelism is the default choice for serving very large dense models, where the bandwidth-hungry communication is offset by the speedup from sharing per-layer work.

Expert parallelism applies specifically to mixture-of-experts models, where only a subset of the model’s parameters activate for each token. Different experts get distributed across different GPUs, and tokens get routed to whichever experts they need. The communication overhead is lower than tensor parallelism because experts operate independently, which makes expert parallelism well-suited for multi-node deployments where bandwidth is more limited. Most production deployments combine both, using tensor parallelism within a node and expert parallelism across nodes.

Disaggregation

Disaggregation takes the prefill-decode split literally. The idea is to run prefill on one set of GPUs and decode on another, with the KV cache shipped between them over the network. Each set uses hardware tuned to its specific bottleneck, and each set scales independently based on its own traffic pattern.

The flow becomes a three-step process:

• The prefill engine takes the input sequence and produces both the first token and the KV cache.

• The cache gets sent over a fast interconnect to the decode engine, and the decode engine handles every subsequent token.

• In conditional disaggregation, short or already-cached requests skip the handoff entirely and run on the decode engine alone, which performs better against real-world traffic that includes a mix of long and short prompts.

Disaggregation is the most architectural of the techniques covered here. It treats prefill and decode as separate services with separate operational concerns, giving operators independent levers to scale each one. Companies running large-scale inference often consider this a near-mandatory step once their workload mix is well understood.

When to Invest in Inference Engineering

Putting these techniques into production is a serious task, and combining them adds further complexity. The question every engineering team has to answer is whether to take on this work or whether off-the-shelf APIs are still the right choice. The answer depends on the stage of the product.

Early in building an AI product, off-the-shelf APIs from established providers are almost always the right choice. Meaningful optimization requires real constraints to work against, and early-stage products tend to have fuzzy assumptions about traffic patterns, latency requirements, and unit economics. Engineering effort at this stage is better spent shipping product, since the complexity of running a custom inference stack slows down iteration when iteration speed is what actually matters.

Three signals usually indicate the equation has shifted:

• API costs have grown into a meaningful expense line.

• Latency requirements have moved past what closed APIs can deliver.

• Reliability needs have started to exceed what vendor SLAs offer.

Cursor handled this transition well. Sub-second autocomplete latency was the product itself, and closed APIs aim for general throughput across many customers, while a code completion model demands a specific shape of speed. Self-hosting an open model and applying inference engineering across the stack made the latency target reachable, and the investment paid back because the constraints were real and the workload was well understood.

Conclusion

LLM inference is two operations with opposite physical constraints.

Prefill is compute-bound and runs once per request. Decode is memory-bandwidth-bound and runs once per token. Most of the techniques in inference engineering exist because of this split, and grasping it makes the rest of the field much easier to navigate.

Each technique covered above fits into the prefill-decode framework:

• Batching trades per-user latency for total throughput.

• Prefix caching cuts prefill work when prompts share opening segments.

• Quantization compresses model weights to help both phases.

• Speculative decoding squeezes more tokens out of decode by exploiting idle compute.

• Parallelism scales models across multiple GPUs.

• Disaggregation runs prefill and decode on separate hardware altogether.

Layered on top of all this is the build-versus-buy question. Off-the-shelf APIs remain the right choice for most products in their early stages, while self-hosting starts to make sense when API costs grow into a real expense line, when latency requirements outgrow what closed APIs can deliver, or when reliability needs exceed vendor SLAs.

References:

• Hugging Face Hub Documentation

• DeepSeek-V3 Technical Report

• Cursor — Composer: Building a fast frontier model with RL

• Cursor — Introducing Composer 2

• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

• Efficient Memory Management for Large Language Model Serving with PagedAttention

• Anthropic — Prompt Caching Documentation

• NVIDIA TensorRT Documentation

• Fast Inference from Transformers via Speculative Decoding

• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

• NVIDIA Megatron-Core Developer Guide — Tensor Parallel

• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

• NVIDIA NVLink and NVLink Switch

• Anthropic Claude API Documentation

Coding agents are here to stay, but vibe-coding auth is dangerous business. Connect your AI assistants to the Descope MCP server instead!

This remote MCP server connects agents to the Descope identity platform, giving them the ability to read docs, inspect project config, manage users and tenants, configure authentication flows, review audit logs, and make changes to your identity infrastructure. All through natural language and from a single session.

Descope is trusted by thousands of businesses including GoFundMe, GoodRx, Linktree, and Databricks.

Get started with 100+ prompt examples

This week’s system design refresher:

• How to Run LLMs Locally (Youtube video)

• The Typical AI Agent Stack, Explained

• Understanding Git Reset Modes

• How NAT Works

• Final Week to Enroll: Build with Claude Code

• We’re hiring at ByteByteGo

How to Run LLMs Locally (Great For Learning and Privacy)

The Typical AI Agent Stack, Explained

Most people think an AI agent is just a clever prompt and an LLM. The reality is much deeper. There's an entire architecture working behind the scenes to make it all run.

The diagram below shows the full AI Agent Stack. At the core is the Agent Runtime that runs a ReAct loop, and three other layers feed into it.

AI Agent Runtime: The LLM thinks about what to do, picks a tool, observes the result, then reflects and decides the next step. This loop repeats until the goal is reached.

Model Layer (the brain): The underlying LLMs that power reasoning.

Tool Layer (the hands): How the agent interacts with the real world: search, APIs, code execution, data access.

Memory Layer (the notebook): Short-term working memory for the current task, long-term semantic memory for knowledge, and transactional memory for state.

Wrapping everything is the Observability & Safety Layer. This is what keeps agents debuggable, evaluable, cost-aware, and safe in production.

Over to you: Which layer of the stack do you think is the hardest to get right in production?

FeatureOps Summit 2026 - Feature management in the AI Era (Sponsored)

Speed without control is a false economy. As AI code-generation accelerates software delivery, the FeatureOps Summit 2026 is here to ensure that when we ship more, we break less. This premier virtual event brings together engineers, architects, and product leaders from companies like Wayfair, Visa, Mintlify, Lloyds, and many others, to explore the infrastructure of fearless delivery.

Key Themes:

AI Safety Nets: Guardrails for the flood of automated code.
Edge Resilience: Sub-millisecond evaluation at scale.
Continuous Flow: Moving past the “fixed-release” mindset. Register today to master the tools and patterns required for a fail-safe release environment.

Register Today

Understanding Git Reset Modes

git reset has three modes. Each one moves HEAD, but they differ in what happens to your index and working directory.

1. git reset --soft: Moves HEAD only. Index and working directory stay as-is. Use this when you want to recommit with different changes or a different message.

2. git reset --mixed (default): Moves HEAD and clears the index, but leaves the working directory alone. Your changes become unstaged, still there, just no longer queued for commit.

3. git reset --hard: Moves HEAD, clears the index, and resets the working directory to match the target commit. Any uncommitted changes are gone.

Over to you: Which reset mode do you use the most and has “--hard” ever cost you a day of work?

How NAT Works

Every device in your home probably shares the same public IP, still each one browses, streams, and connects independently. This is handled by NAT (Network Address Translation), a protocol that runs quietly in the background of almost every home network.

It’s the reason IPv4 hasn’t run out completely, and why your router can hide dozens of devices behind a single public IP.

• The Core Idea: Inside your local network, devices use private IP addresses that never leave your home or office. Your router, however, uses a single public IP address when talking to the outside world.

NAT rewrites each outbound request so it appears to come from that public IP address, assigning a unique port mapping for every internal connection.

Outbound NAT (Local to Internet): When a device sends a request,

• NAT replaces the private IP address with the public one

• Assigns a unique port so it can track the connection

• Sends the packet out to the internet as if it originated from the router

Reverse NAT (Internet to Local): When the response returns,

• NAT checks its translation table

• Restores the original private IP address and port

• Delivers the packet to the correct device on the local network

Over to you: Have you ever run into tricky NAT edge cases? Port forwarding? Double NAT? Video calls breaking? Online gaming problems?

Final Week to Enroll: Build with Claude Code

We’re launching a new 2 day intensive, cohort based course called Build with Claude Code, taught by John Kim, who has trained hundreds of engineers at Meta to use Claude Code in real production workflows.

The course kicks off June 18th, and enrollment closes in less than a week. If you’ve been thinking about leveling up how you and your team work with Claude Code, this is the moment.

Check it out now

A few things you’ll learn:

• The agentic loop, context engineering, and memory layers that make Claude Code useful for real projects

• How to build with Claude Code Skills, MCPs, and hooks to give Claude the tools and feedback loops it needs to self correct

• Parallel development with Git worktrees, subagents, and agent teams

• A capstone project where you ship something real on your own stack

The course includes live sessions, assignments, and office hours, so there’s plenty of room to ask questions and get unstuck.

The first cohort starts in just a few days: May 28 to 29, 2026. If you want to learn everything from the fundamentals of Claude Code to advanced production workflows, including working with large codebases, this could be a great way to level up.

Check it out now

We’re Hiring at ByteByteGo

We’re looking for multiple part-time instructors to teach AI and engineering cohort-based live courses.

This is a great fit if you love teaching, enjoy sharing what you know, and want a meaningful side thing alongside your main work.

The role has some upfront time investment to get familiar with the curriculum and prepare, but after that, it’s designed to be a limited commitment (2-5 hours bi-weekly). It offers stable income, good upside, and a chance to share your knowledge while working with ambitious learners.

We’re especially looking for instructors in:

• Building Production-Grade AI Systems

• System Design

• AI Security & LLM Red-Teaming

• AI Evals Intensive

• AI Cost Optimization

• Agentic AI Coding

• Build with Codex

• AI for Engineering Leaders

• AI Automation

• Others, please suggest

Ideal instructors are hands-on, clear communicators, and excited to teach.

If this sounds like you, email us at jobs@bytebytego.com with your background, the topics you’d be excited to teach, and any teaching, writing, or speaking samples.

Several distinct strategies are in common use today, and each one is an answer to a specific problem that the previous approaches couldn’t solve well enough. Some reduce the blast radius (the number of users affected) when a deploy goes wrong. Others separate the moment the code reaches production from the moment users actually see it.

In this article, we’ll go through the main deployment strategies used in production today, looking at how each one works, what it costs, and when it makes sense to use.

Big-Bang Deployment

Read more

This is a great fit if you love teaching, enjoy sharing what you know, and want a meaningful side thing alongside your main work.

The role has some upfront time investment to get familiar with the curriculum and prepare, but after that, it’s designed to be a limited commitment (2-5 hours bi-weekly). It offers stable income, good upside, and a chance to share your knowledge while working with ambitious learners.

We’re especially looking for instructors in:

- Building Production-Grade AI Systems

- System Design

- AI Security & LLM Red-Teaming

- AI Evals Intensive

- AI Cost Optimization

- Agentic AI Coding

- Build with Codex

- AI for Engineering Leaders

- AI Automation

- Others, please suggest

Ideal instructors are hands-on, clear communicators, and excited to teach.

If this sounds like you, email us at jobs@bytebytego.com with your background, the topics you’d be excited to teach, and any teaching, writing, or speaking samples.

Got someone in mind for any of these topics? We’d appreciate the intro. Thank you.

Sign-up forms were built for humans in browsers, so how do AI agents programmatically register with services?

Enter auth.md. By exposing a single, machine-readable Markdown file at your service root, AI agents can dynamically discover your OAuth Protected Resource Metadata, parse required scopes, and authenticate seamlessly.

With native support in WorkOS AuthKit, you can now implement this protocol out of the box, giving AI tools a standardized, secure way to log into your application.

Read the auth.md docs

Everyone is building agents. Most of them will fail at scale. Not because the technology doesn’t work, but because teams don’t know what happens after the demo.

That’s the lesson from Salesforce, which has over 20,000 enterprise customers running Agentforce in production. Their support agent alone has handled over three million conversations.

We sat down with John Kucera, Salesforce’s CPO of Agentforce, to learn what separates agents that deliver real business value from those that stall after a good demo.

What is Salesforce?

Salesforce is the enterprise software leader, and its Agentic Enterprise Architecture defines how AI agents are built and deployed across business operations.

The Agentic Enterprise Architecture has four layers. At the top is the engagement layer, where users interact with agents through their everyday tools like Slack, chat, or messaging apps. Below that is the agent layer, where the AI reasoning and decision-making happens. This is where agents are built, monitored, and orchestrated.

Below the agent layer is the system of work, which incorporates the apps trusted by, and tailored for, your department and your industry. These are the business applications where real work gets done, like resolving a support case, processing a return, or updating a sales pipeline. Lastly, the context layer provides agents with the data and metadata they need to ground their actions in real context, ensuring decisions are informed by the specific business operations.

A trust layer spans the entire stack, supporting multiple LLM providers and enforcing the guardrails we cover later in this article.

Together, these layers let customers go from idea to working agent without building the infrastructure from scratch. Agentforce provides the reasoning, the data access, the business applications, and the trust controls as a connected platform. But having the right architecture is only part of the story. Once 20,000 customers started deploying agents on this platform, Salesforce discovered something that reshaped how they think about the entire product: the hardest part isn’t building the agent. It’s what happens after you ship it.

What is Agentforce?

Agentforce is Salesforce’s platform for building and deploying AI agents in the enterprise. Rather than a single model or chatbot, it’s a layered architecture designed to embed agentic AI across Salesforce’s entire ecosystem like sales, commerce, and services.

Agentforce elevates every experience by bringing together humans, applications, AI agents, and data. Now any company can safely deploy agents that work for their customers, suppliers, and employees 24/7. Teams can manage the complete agent development lifecycle with a robust set of tools to build, test, deploy, manage, and orchestrate AI agents at scale.

Why Most Enterprise Agents Fail

Agents built on LLMs are flexible by design. They can interpret a wide range of inputs and decide what to do in real time. But that flexibility comes with a tradeoff. Because LLMs are non-deterministic, the same question can produce different steps each time. Across Salesforce’s deployments, this was one of the most common challenges: keeping agent behavior consistent and reliable, especially in high-stakes workflows.

The reason this is so hard comes down to how AI agents differ from traditional software. In traditional software, the effort distribution looks roughly like this:

With traditional software, 90% of the work happens before launch. You gather requirements, design the architecture, build it, and test it. After launch, you’re mostly in maintenance mode. With AI agents, this ratio flips. As Kucera put it: “In the typical software world, 90% of the work is getting to go live. Whereas in the typical AI agent, 90% of the work is after you go live to manage and improve the agent.”

This is the main reason most enterprise agents fail. Teams follow the traditional software playbook and assume the hard work is done once the agent is live. It’s not. That’s when it starts.

Modern tooling makes this worse. You can build a working agent in an afternoon. The demo handles the typical questions well. Leadership sees it and greenlights production. But typical questions are only a minority of what real users will ask. The majority involves edge cases, ambiguous phrasing, and cross-domain questions. That’s where the agent earns or loses trust.

The teams that succeed treat launch as the starting line. That means two things: 1) get the pre-launch foundations right so they can iterate on quickly, and 2) budget the majority of effort for post-launch and continuous improvement.

These production realities forced Salesforce to evolve Agentforce itself. Models are inherently probabilistic, predicting the next best response rather than executing fixed logic. That makes them powerful for reasoning and natural interaction, but enterprises still need deterministic systems underneath for consistency and trust. The future of enterprise AI is the combination of both: deterministic workflows set the guardrails, while probabilistic AI adds adaptability and contextual reasoning on top. Features like Agent Script and Hybrid Reasoning are the direct result of watching 20,000 deployments hit this wall. The rest of this article covers the pre-launch and post-launch lessons that shaped that evolution.

Pre-Launch: What to Get Right Before You Ship

If 90% of the work is post-launch, then the goal of pre-launch isn’t to build the perfect agent. It’s to build an agent you can effectively iterate on. That means choosing the right scope, defining how you’ll measure success, and putting the trust, security, and safety guardrails in place from the start.

1. Start Small and Focused

Kucera put it simply: “Don’t boil the ocean.” When building your agent, it’s tempting to go after something ambitious. Don’t. Pick a use case that is both high-value and achievable. There are two reasons for this.

First, agent capabilities are still evolving fast. What’s possible today will look different in six months. If you invest heavily in a complex, multi-step agent now, you may end up rebuilding it as better models and tooling arrive. A focused use case gives you real production learnings without overcommitting to today’s limitations.

Second, the process of building agents is different from traditional software. Your team needs to learn how to review agent transcripts, figure out why the agent made a wrong decision, and update instructions, tools, and data sources. That learning is faster and lower-risk when the use case is small.

Starting small pays off soon. Once your team has shipped one agent, measured its impact, and learned the iteration cycle, scaling to the next use case is much faster. But even a focused agent needs a clear definition of success.

2. Tie Agent to a KPI

A common failure pattern across Salesforce’s customer base is that teams push an agent to production without defining what success actually means. Without a measurable goal, there’s no way to know if the agent is working or drifting.

This is where Agentic Work Units (AWUs) become critical. Introduced by Salesforce, AWUs are discrete units of meaningful work completed by an agent and provide a standardized way to measure actual task completion. They give teams a consistent framework to quantify value beyond activity or interactions, focusing instead on whether the agent is truly getting work done.

Every agent needs a concrete KPI tied to a real business outcome. For Salesforce’s own support agent at help.salesforce.com, that KPI is containment rate.

Containment rate means the percentage of cases fully resolved by the agent without human follow-up. A user asks “How do I reset my password?” The agent gives a clear answer, the user solves their problem, and never comes back for the same issue. That’s contained. But if the user returns the next day asking the same thing, the agent failed to actually help.

AWUs complement this by turning those outcomes into a consistent measurement of work completed, how many user intents were fully resolved per interaction, allowing teams to track not just whether a case was contained, but how efficiently and reliably the agent is performing work at scale.

The KPI also drives your post-launch iteration. When your team reviews transcripts and decides what to fix next, the KPI tells you what matters. A tone issue might be annoying, but a logic error that tanks your containment rate gets fixed first. Over time, AWUs become a shared language across product, engineering, and operations for evaluating agent performance, complementing traditional KPIs like containment rate with a more granular view of agent productivity.

With scope and KPIs in place, the last pre-launch foundation is the trust layer.

3. Trust, Security, and Safety at Enterprise Scale

Your agent sits between your users and your data, with an LLM in the middle. Data flows in both directions: user queries pull sensitive data into the LLM’s context, and the LLM’s responses flow back to users, sometimes triggering real actions. Each direction creates different risks, from data privacy leaks on the way in to hallucinated actions on the way out.

In practice, teams implement two guardrails on both sides of the LLM. Input guardrails to protect data before it reaches the LLM, and output guardrails to validate the LLM’s response before it reaches the user.

Input guardrails

When your agent needs data to answer a question, that data has to travel from your systems into the LLM prompt. This is where things can leak. The core protections are secure data retrieval, zero data retention, and keeping data inside a trusted boundary. Data masking is also available, but as we’ll see, it comes with a tradeoff that makes it the wrong default for most agents.

Secure data retrieval means you control exactly how data enters the prompt. Instead of giving the LLM raw access to a database, you route requests through a controlled layer that only returns what the agent is allowed to see.

Zero data retention is an agreement with your LLM provider that they won’t store your prompts or responses, and won’t use them to train future models. Anything sent to an external LLM is covered by this contract: it isn’t retained, viewed, or used for training once the response is returned. Without it, your customer data could end up embedded in a model that serves other companies.

Keeping data inside a trusted boundary goes one step further. For the most sensitive workloads, you can route requests to a provider-hosted model that sits inside your platform’s trust boundary, so the data never crosses the public internet at all. On Agentforce, for example, custom actions can call Salesforce-managed models like Anthropic’s Claude hosted within the Salesforce trust boundary. The data stays inside the boundary, and zero data retention still applies.

Data masking is the one input guardrail to use with care. It catches sensitive data before it reaches the LLM, for example detecting a social security number and replacing it with a placeholder token. The catch is that masking can strip out the very context the agent needs. If a user asks the agent to build a list of accounts similar to a reference account, but the reference account’s details are masked, the agent no longer has the information to find the match. For this reason, Agentforce keeps pattern-based and field-based masking off by default for agents, relying on zero data retention and trust-boundary hosting instead. Masking remains a legitimate control where the redacted fields aren’t needed for reasoning, but it shouldn’t be the default for agents that depend on rich context.

Output guardrails

Input guardrails protect data on the way in, but the LLM can still produce bad output. Before a response reaches the user, you need a second set of checks such as tool validation, grounding checks, and content filtering.

Tool and sub-agent validation ensures the agent isn’t hallucinating actions, not just text. If the agent decides to route to a “refund_processor” sub-agent that was never defined, the system should catch that and block it.

Grounding checks verify the agent isn’t making up facts from its general training data. If your agent is supposed to answer based on your help docs, the response should only contain information from those docs.

Content filtering catches harmful or inappropriate content before the user sees it. This includes toxicity scoring and other safety classifiers that screen the agent’s output.

Neither layer alone is enough. PII masking protects your data on the way in but doesn’t prevent hallucinated tool calls on the way out. Output validation catches bad responses but doesn’t stop sensitive data from reaching the LLM. Teams implement both of these layers in practice.

Post-Launch: Lessons from 20,000 Deployments

You’ve scoped your use case, defined a KPI, and built your trust layer. The agent is live. As we discussed earlier, this is where 90% of the real work begins. The following lessons come from Salesforce’s experience managing 20,000 enterprise agent deployments once they meet real users.

Build a Feedback Loop

In traditional software, testing is fairly binary. You have unit tests, integration tests, maybe some latency benchmarks. They pass or they don’t. With agents, the failure modes are fuzzier. Users ask things you didn’t anticipate. The agent’s tone drifts from your brand. A retrieved document turns out to be outdated. The agent gets the right answer from the wrong source. This makes the feedback loop an important post-launch investment.

The feedback loop has four triage categories, each with a different fix.

1. Tone and brand alignment. The agent’s responses don’t match your company’s voice. This is especially common with B2C agents where brand consistency matters. The fix is in the system prompt and instructions. Adjust the voice guidelines, add examples of preferred phrasing, and re-test against recent transcripts.

Agibank is one example. Its FAQ agent pulls accurate answers in real time from a knowledge base stored in Agentforce Service, delivering clear responses while staying human, not robotic. “Our principal objective is to reduce effort for customers and provide fast, clear answers through automation, while preserving a welcoming, human tone and not being robotic,” said Akira Vargas Morishita, CX Process & Continuous Improvement Coordinator at Agibank.

2. Logic errors. The agent calls the wrong tool, reasons incorrectly, or takes too many steps to reach an answer. This shows up as slow or wrong responses. Start by checking tool configurations and instructions. If the same error keeps recurring, that flow is a candidate for deterministic scripting instead of LLM reasoning.

3. Data quality. The agent gives a wrong answer not because it hallucinated, but because the source was wrong. Salesforce’s support agent is grounded in 135,000 help articles, and the team regularly finds outdated or conflicting documents behind flagged responses. The fix isn’t in the agent. It’s routing the issue back to the data owners to update or retire the source content.

4. Coverage gaps. Users ask things the agent was never designed to handle. This is inevitable and grows with adoption. The fix is either expanding the agent’s scope or building a clean escalation path to a human. Either way, log the gap so you can track how coverage grows over time.

Telepass is one example. When a question needs human support, such as troubleshooting a device, Agentforce escalates to a live rep with full context so the customer doesn’t have to start over. Before the chat ends, it offers a short survey to capture feedback and satisfaction, and Telepass uses that data to shape its roadmap for future deployments.

The key insight is that this loop needs to be fast. Across Salesforce’s customer base, the speed of this feedback loop turned out to be the gate to scaling. Teams that could quickly triage and fix issues gained confidence in their KPIs and got approval to expand. Teams with slow loops stayed stuck in pilot mode.

The feedback loop catches problems. But some problems are better prevented than triaged. Salesforce identified three recurring anti-patterns that consistently degraded agent performance. Each one is easy to fall into and hard to diagnose through transcripts alone.

Anti-Patterns: What Not to Do

The feedback loop catches problems. But some problems are better prevented than triaged. Across 20,000 deployments, Salesforce identified three recurring anti-patterns that consistently degraded agent performance. Each one is a mistake that’s easy to make and shows up in production as degraded accuracy, slow response times, or both.

1. Over-reliance on LLM reasoning where code is better

Not every agent decision needs to go through the LLM. When a customer asks “Where’s my order?”, the correct sequence of API calls is deterministic: look up the order, get its status, get the shipment details. Routing this through the LLM’s reasoning loop means multiple round trips, each adding latency and introducing a chance of error.

This is the most common anti-pattern, and it directly led Salesforce to build Agent Script. Agent Script solves this by letting you define deterministic control flow alongside LLM-powered decision-making. It’s a TypeScript-based scripting framework where you can specify: if the user’s intent matches X, skip the reasoning loop and immediately execute this sequence of tool calls. The LLM is still there for the parts that genuinely need flexibility like understanding ambiguous requests or generating natural language responses, but the predictable parts run as code.

The general principle for developers is that if you can write the logic as a flowchart, it should probably be code, not a prompt.

2: Prompting harder instead of encoding policies

This one is subtle because it feels like good prompt engineering. Teams discover the agent does something wrong, so they add a strongly worded instruction. “NEVER do X.” “ALWAYS do Y.” They use exclamation points, bold text, and capitalization. When the agent still gets it wrong, they add even more emphasis.

This doesn’t work reliably. LLMs don’t respond to emphasis the way humans do. What works is encoding business rules as explicit, structured policies. For example, if you’re a financial services company that doesn’t operate in Hawaii, you don’t want the LLM to figure that out from a prompt instruction. You need a policy that says: if the customer’s state is Hawaii, return this specific response. No LLM judgment involved.

This is another area where deterministic scripting helps. Instead of hoping the LLM internalizes a rule from natural language, you encode it as a conditional in code. It executes the same way every time.

3. Poor context engineering

This anti-pattern hurts both accuracy and performance at the same time. Many teams start by passing full, unfiltered API responses into the agent’s context window. A large e-commerce company, for instance, has a get_orders API call that returns roughly 100K tokens by default.

This causes two problems. First, the agent has to reason over a much larger input, which slows response time. Second, the noise makes the agent less accurate. When the relevant information is buried in hundreds of irrelevant fields, the agent is more likely to pick up the wrong data or miss the right answer entirely.

The fix is right-sizing your context. For the e-commerce example, that meant trimming the get_orders response from 100K tokens to roughly 2K by returning only the fields the agent actually needs: order ID, current status, expected delivery date, and tracking number. The same principle applies to document retrieval. One insurance company was loading entire policy documents into context to answer a single question. The fix was retrieving only the relevant sections instead of the full document. In both cases, less context meant faster responses and more accurate answers.

What’s next?

Here are three directions Salesforce sees enterprise agent architecture heading next.

1. Multi-Agent Orchestration

Most agent deployments today use a single agent per use case. The next step is multiple agents working together, where a parent agent coordinates specialized sub-agents, each handling a narrower piece of the problem.

Salesforce is already building orchestration systems that go three levels deep: a parent agent with sub-agents, where each sub-agent can have its own sub-agents.

For developers, this changes how you architect agent systems. Instead of one large agent that handles everything, you decompose the problem into specialized agents that hand off to each other. Each agent has a narrower scope, which means simpler instructions, fewer tools, and a smaller context window.

2. Agents Beyond the Chat Window

Today, most agents live inside a chat widget. But the use cases emerging at enterprise scale go well beyond that. They include multi-session tasks that span days (like a tier-2 support case or a return authorization workflow), background agents that run with no user-facing interface at all, and agents that work across channels like phone, email, web, and Slack.

For developers, the takeaway is to avoid coupling your agent architecture too tightly to a chat interface. The agent’s logic, tools, and policies should be independent of the delivery channel. Teams building on Agentforce today are already running the same agent across web chat, phone (via Agentforce Voice), and background automation.

3. The Pace of Change

One thing that came up repeatedly in our conversation is how fast this space is moving. Coding agents, for example, went from basic assistants to highly capable tools in just a few months. The models are getting faster. The tooling is getting better. What’s considered best practice today may look different six months from now.

Building enterprise agents is still early. The models, tooling, and best practices are all moving fast. But the core engineering disciplines hold. Start small. Measure what matters. Build tight feedback loops. Encode policies in code, not prompts. Keep your context lean. These aren’t tricks tied to any specific model or framework. They’re how you build agents that work at scale.

Code review broke when AI started writing the code.

AgentField just shipped a multi-agent code reviewer with dynamic meta-orchestration. The planner reads each PR first, then compiles a custom review strategy for it - security agents for auth changes, schema agents for migrations, behavioral agents for refactors. Configurable per team. Deploy it with one docker compose. Runs on open or closed models (Kimi, DeepSeek, Claude). Costs cents per review on open models - no per-seat licenses.

AgentField’s writeup: the four jobs of code review, which three stay load-bearing once AI writes the first draft, and why static pipelines fail.

→ Star & Deploy

LLM agents can burn millions of tokens on a single task. They put a model in a loop, resend the full context every step, and usually call the most expensive one available. Costs scale fast.

To understand how teams keep this under control in production, we sat down with Scott Breitenother and Sid Sijbrandij, co-founders of Kilo, an open-source coding agent that runs through a lot of these loops every day. The patterns they shared are not specific to coding, and most of them are not unique to Kilo either. Similar approaches show up in tools like Cursor, Cline, and Aider, and in shared infrastructure like OpenRouter and RouteLLM. If you build any agent that makes many model calls, the same ideas apply.

Why Running an LLM Agent Gets Expensive

A single request to a language model is usually cheap. An agent built on the same model is not. The difference is that an agent makes many calls instead of one, and it tends to send them to the most expensive models available. Both of these drive the cost up.

1. Frontier Models Cost a Lot Per Token

The most capable models are called frontier models. They sit at the leading edge of what is possible, and they cost the most per token. Below them is a range of cheaper models that give up some capability for a lower price, down to small models that are very cheap and still handle simple work well.

The gap across that ladder is large. The top model often costs more than ten times what a small one costs for the same work. Teams that use frontier models to power their applications pay frontier prices for everything, which makes the whole system expensive.

2. The Agent Loop Multiplies Every Call

Frontier models are expensive per token, but in a standard chatbot setting, the cost is manageable. For example, Claude Opus 4.7 costs $5 per million input tokens and $25 per million output tokens. A single question and answer is only a few thousand tokens, so it costs under two cents. At that rate, you can ask a lot of questions before the cost matters.

LLM agents are different. They do not produce an answer immediately. They run in a loop. The agent reads the task, takes an action like running a tool or reading a file, looks at the result, and decides what to do next. To see why this gets expensive, look at what each step sends to the LLM.

Since an LLM has no memory of its own, everything has to be bundled into the context on every step. That includes the instructions, the question, tool schemas, tool calls, tool results, and the LLM’s intermediate thinking. The agent resends all of it each turn, so the context grows as the loop runs, and each call to the LLM costs more than the last.

A session might start at a few thousand tokens. By the time the agent has read a dozen files and run a dozen tools, a single request near the end can carry well over a hundred thousand. As a result, agents can burn many times more tokens than a single chatbot question.

The growing context is only half of it. The other half is how often the agent calls the model. In a normal chat, it takes time for a person to type a question and effort to read the answer, so they only ask so much. An agent has no such brake. An agent that reviews every code change, comments on every commit, and writes a test for every function fires off requests as fast as the software allows.

Your AI pipeline passed every test. Then it hit production. (Sponsored)

Datadog’s free Developer Toolkit for the AI Era gives you four resources to close the gap, from catching flaky tests and CI bottlenecks before they block releases, to instrumenting every LLM call for quality, latency, and cost regressions.

You’ll learn how to:

• Surface and eliminate CI pipeline failures before they block your AI delivery cycles.

• Use feature flags to control AI rollouts and DORA metrics to measure exactly how your team is shipping.

• Score LLM output quality and catch latency drift across every model call in production.

Download Toolkit

The Standard Approach: Route Requests to the Right Model

A growing context and an agent that runs without a human brake are both inherent to how agents work, so you cannot really send fewer tokens. What you can change is which model receives them, and that is what routing does. A router looks at each request, decides which model is good enough, and sends it there.

For example, think about the requests a coding agent sends while working on a task. A few are hard, like designing how a system should be structured. Most are simple, like renaming a variable or summarizing a file. The hard ones need a frontier model. The simple ones do not, but they cost the same if you send them there too. Routing saves money by sending each request to the cheapest model that can handle it, so you only pay for a frontier model when you actually need one.

How a Router Works Under the Hood

A router needs two things. It needs a way to send requests to many different models. It also needs to decide which model to use for each request. These are two separate problems, and keeping them separate makes the design cleaner.

The first problem is the entry point. Normally each model provider has its own request format, so using several models means writing separate code for each. A single entry point gives you one standard request format, and the router translates that into whatever the chosen provider expects, sends it, and translates the response back. You write in one format. The router talks to all the providers for you. Without this, routing between models is not practical in the first place.

The second problem is the decision. Which model should a given request use? In practice, the decision is handled in two ways.

The first is to route on a signal you already have. If the system already knows what kind of work a request is, it can map that kind of work to a model. A request known to be a planning task maps to a strong reasoning model. A request known to be a simple edit maps to a cheap one. This is reliable and almost free to run, because the decision is just a lookup. The catch is that you need a trustworthy signal to begin with.

The second is to predict the right model from the request itself. The system reads the request, judges how hard it is, and picks the cheapest model likely to answer it well. This works even when you have no prior signal about the request. The cost is that the prediction has to be learned from data and kept current as models change. A wrong guess sends a hard request to a model that cannot handle it.

Most real systems use one entry point with one of these two decision methods on top. The entry point gives access to many models. The decision technique picks among them.

How Much Routing Actually Saves

Routing saves money because most requests do not need a frontier model, and cheap models have gotten good enough to handle them. The saving is the gap between the frontier price you would have paid and the cheaper price you actually paid, summed over every request that did not need the expensive model.

The effect of proper routing is quite noticeable. In a widely cited study from researchers at UC Berkeley and Anyscale, a router cut cost by about half while keeping 95% of a frontier model’s quality. It did this by sending only the hard requests to the frontier model and the rest to a cheaper one. More broadly, results across the field tend to land between forty and seventy percent cost savings, with little drop in quality on hard tasks.

Router cost saving (Source: Github)

Those savings come with tradeoffs, though. The decision step adds a little delay to every request and becomes one more thing that can break. A wrong decision hurts quality by sending a hard request to a weak model. A predicting router needs data and upkeep to stay accurate. And switching between different model families inside one task can cause trouble, because the internal reasoning one model produces is not always readable by another. None of these kill the idea. They are why routing is an important but challenging problem to handle well.

A Case Study: How Kilo Routes Requests in Production

To make this concrete, it helps to look at one system that does it end-to-end. Kilo makes an open-source AI coding agent that drives a model in long loops to write and fix code, and across its user base that adds up to a very high request volume. To serve that volume without the cost running away, the Kilo team built its own routing layer, the Kilo Gateway, and runs all of its traffic through it. The numbers and design choices in the rest of this section come from that production system.

The Gateway

Kilo Gateway is the entry point described earlier. It puts one consistent way of making requests in front of more than five hundred models, so switching from one to another is a one-line change. It speaks the same request format most code is already written for, so existing code works by pointing it at the Gateway. It also lets a team use its own provider accounts and pay only for the routing, not a markup on the models.

The decision layer is the interesting part. Kilo uses the first method: it routes on a signal it already has. Its coding agent always knows what it is doing right now, because it works in distinct modes like planning, writing code, or debugging. The agent sends that mode with each request. The Gateway reads the mode and maps it to a model. The mode is a trustworthy signal of how hard the work is, so the system gets most of the benefit of routing without having to guess difficulty from the request text.

The routing is organized into tiers a user can pick from. A top tier sends demanding modes like planning and debugging to the strongest model, and sends routine modes like code editing to a capable but cheaper one. A balanced tier sends everything to an economical model. A free tier maps to no-cost models. A separate internal tier quietly handles background chores, like writing commit messages, with tiny models so they never burn expensive capacity.

One design choice explains how the system stays current. The map from a mode to a specific model does not live in the software on your machine. It is served from Kilo’s own systems and refreshed often. So the underlying models can be swapped as prices and quality change, while the tier you picked stays the same.

This flexibility has one cost worth understanding. A tier can change models between the turns of a single task, for example moving from one provider’s model to another’s as the mode shifts. The problem is that reasoning models produce internal thinking in their own format, and one provider’s model cannot read the thinking another model wrote. So when the tier switches families mid-task, Kilo has to drop that intermediate reasoning before the next call. The agent keeps working, but it loses some of the context it built up, which can cost a little quality on the next step.

What Kilo’s Production Numbers Show

Kilo published figures from its own production traffic over the first quarter of 2026. These are the company’s internal numbers from paid usage, not independently verified, but they are useful because they come from a real workload rather than a benchmark.

When the team let the Gateway route on its own instead of having users pick a model by hand, the average cost per request dropped by about a third. Kilo found that 80 to 90% of requests do not need frontier models. Across millions of requests, routing those jobs to cheaper models adds up to real savings.

The bigger surprise was how much the choice of tier shaped the cost. For the same coding work, running on the cheaper balanced tier cost over ten times less per request than running on the top tier, and that gap showed up across every kind of work the team measured. The smallest background tasks, handled by tiny models, came in at a fraction of a cent. Most of the savings turned out to come not from anything clever, but from simply keeping routine work off the most expensive model.

To put a number on what that is worth, the team estimated that forcing its routine traffic onto top-tier models for the quarter would have cost about eighty-seven thousand dollars more. That is roughly what getting routing wrong would have cost them on a real workload.

The Kilo team also shared a useful finding about caching. Caching, where the system saves repeated context so you do not pay for it twice, is usually seen as the main way to cut the bill. Kilo found that even with cache reuse above eighty percent on many features, total spend stayed high, because there were so many requests and the part of each context that could not be cached was still large. Caching clearly helps, but it does not solve the volume problem by itself. Routing works on a different part of the cost, which is why teams tend to use both together.

Lessons for Any Team Running Agents at Scale

Kilo’s numbers point to a few lessons that apply to any team running agents at scale, whichever router or models they use.

Set a budget and treat AI spend like any other infrastructure cost. It is tempting to just switch to a cheaper model and reduce the per-token rate. But a lower rate usually leads to more usage, since work that was too expensive before now looks affordable. So the total bill often climbs even as each request gets cheaper.

Pick a monthly budget for the whole workload and treat it as fixed. Then the goal is not the lowest price per request, but the most useful work you can fit inside that budget.

Measure before you optimize. Cost is driven by tokens per request, which is mostly a function of context size, not by which part of the product the request came from. Two requests can both be tagged “chat” while one carries a thousand tokens and the other a hundred thousand.

So log the token count of every request, and tag each one with the task type and the feature that sent it. Then add the tokens up per group. The groups that dominate your token total, not the ones that send the most requests, are where your spend actually is, and where routing pays off most.

Route on the strongest signal you already have. If your system already knows the task type, like whether a request is planning or a simple edit, route on that directly. It is a static lookup from task type to model: cheap to run, predictable, and easy to debug when a route looks wrong. Only fall back to inferring difficulty from the request text when you have no such signal, since that means running a separate classifier you have to train, evaluate, and keep current as models change.

Either way, the router can only help if there is a real spread of models behind it. Give it access to the full range, from frontier models down to small cheap ones, so it has a meaningfully cheaper option to pick whenever the work allows.

Conclusion

Today, routing still takes manual work. You have to choose a tier, set up the signal it routes on, and decide which tasks are safe to send to a cheaper model. It works, but it is something you have to stay on top of.

This will get easier. Routers are starting to make these choices on their own. Instead of being told the task type, a router will read the request, judge how hard it is, and pick the model itself. Over time it will get more precise, choosing a model for each step of a task rather than for the whole task. The goal is for routing to fade into the background, the way load balancing did. You set a budget and a quality bar, and the system handles the rest.

This matters more every year, because agents keep getting more capable. They run longer, act on their own, and send more tokens with no one watching. As that grows, picking the right model stops being a way to save a little money and becomes what decides whether running an agent is affordable at all. The main takeaway is this: routing is no longer a cost optimization. It is becoming part of what makes ambitious agents possible.

QA Wolf’s AI agent maps and tests your app’s most complex user flows.

It turns your prompts into real Playwright and Appium code that runs 12x faster and more reliably than other computer-use agents.

What sets our AI apart:

• Maps 200+ test cases in minutes instead of weeks of manual planning.

• Executes tests 12x faster than computer-use agents.

• Runs entire suites 100% parallel with consistent results.

• Produces open-source tests your team owns, with zero vendor lock-in.

Get started today

This week’s system design refresher:

• CPU vs GPU vs TPU (Youtube video)

• Latency vs Throughput vs Bandwidth

• What is Google’s TPU?

• 7 Permission Modes Every Claude Code User Should Know

• Top AI Trends to Watch in 2026

• We’re hiring at ByteByteGo

CPU vs GPU vs TPU

Latency vs Throughput vs Bandwidth

Ever wondered why your app feels slow even when the bandwidth looks fine? Latency, throughput, and bandwidth often get used interchangeably, but each one tells a different story about performance.

Latency is the delay. How long it takes for a single packet to travel from sender to receiver. If your ping shows 40 ms round-trip, that's latency.

Throughput is the actual delivery rate. How much data is successfully transferred per second. If your download shows 62 Mbps, that’s throughput.

Bandwidth is the maximum capacity of the link. For example, a 100 Mbps connection is the upper limit under ideal conditions.

Throughput is always less than bandwidth. Network congestion, packet loss, and protocol overhead all affect throughput, which is why you never actually hit the maximum bandwidth capacity in practice.

Similarly, low latency doesn't always mean high throughput. Small payloads, single connections, and tight window sizes can all keep throughput low, which is why fast responses don't guarantee you're sending a lot of data.

Another way to understand these three concepts: Bandwidth is the highway width. Throughput is the traffic flow. Latency is how long it takes a car to go from A to B.

All three matter, but they solve different problems.

Over to you: How do you measure these metrics in a way that actually predicts when things will break?

What is Google’s TPU?

A TPU (Tensor Processing Unit) is Google’s custom AI chip, designed from scratch for the giant matrix multiplications that modern models live on. GPUs were built for graphics first.

TPUs were built for deep learning from day one.

At Cloud Next ’26, Google unveiled its 8th generation, and for the first time it ships in two flavors. TPU 8t is built for training, where raw throughput wins. TPU 8i is built for inference, where latency and chip-to-chip speed matter most.

Both still share the same Axion CPUs, liquid cooling, and software stack, so code written for one runs on the other.

The diagram is a quick study guide to what’s the same, what’s different, and why, based on our understanding of published Google articles.

7 Permission Modes Every Claude Code User Should Know

1. plan: The model drafts a plan. Nothing executes until the user approves.

2. default: Standard interactive use. Most tool calls require user approval.

3. acceptEdits: Edits in the working directory are auto-approved. Other shell commands still prompt.

4. auto: An ML classifier decides on requests that miss the fast path.

5. dontAsk: No prompts shown. Deny rules are still enforced.

6. bypassPermissions: Most prompts are skipped. Safety-critical guards still apply.

7. bubble: A subagent escalates its permission request to the parent.

Only 5 modes are user-selectable. “auto” is gated by a feature flag, and “bubble” is internal.

Over to you: Which mode do you reach for most, and what made you pick it?

Top AI Trends to Watch in 2026

2026 is already moving faster than anyone expected. Anthropic released Opus 4.7, OpenAI introduced GPT5.5-Codex, and open-source releases like Kimi K2.5 and GLM-5 showed impressive agentic performance.

These launches point to bigger trends. Here are the five categories to closely watch in 2026.

1. Efficient Reasoning: RLVR-style training scales reasoning by auto-checking math and code. In 2026, expect more adaptive reasoning and extremely sparse architectures. Early signs include Gemini’s adaptive thinking and Qwen3.5’s sparse MoE architecture.

2. Persistent Agents: Agents now plan in loops with tools and memory, not just chat. In 2026, expect always-on personal agents that live across days, have access to your files, and can complete tasks safely. OpenClaw is an early example of this direction.

3. Repo-Scale Coding: Coding has moved from autocomplete to multi-file edits with tests, builds, and terminal tools. In 2026, expect agents that understand very large repos and can ship security-aware PRs by default.

4. Open-Weight Everywhere: Open-weight models are now strong enough to compete with closed ones. In 2026, expect more of them to get leaner, agent-ready, and easier to deploy. Models like GLM5 and Kimi K2.5 are already pushing in this direction.

5. World Models + Physical AI: Multimodal models have reached impressive quality across vision, image, and video generation. In 2026, expect these models to become the foundation for physical AI and world models, with early examples like Google Genie 3 and humanoid robots already pointing the way.

Over to you: which shift do you think will change how teams build products the most in 2026?

We’re hiring at ByteByteGo

We’re looking for multiple part-time instructors to teach AI and engineering cohort-based live courses.

This is a great fit if you love teaching, enjoy sharing what you know, and want a meaningful side thing alongside your main work.

The role has some upfront time investment to get familiar with the curriculum and prepare, but after that, it’s designed to be a limited commitment (2-5 hours bi-weekly). It offers stable income, good upside, and a chance to share your knowledge while working with ambitious learners.

We’re especially looking for instructors in:

• Building Production-Grade AI Systems

• System Design

• AI Security & LLM Red-Teaming

• AI Evals Intensive

• AI Cost Optimization

• Agentic AI Coding

• Build with Codex

• AI for Engineering Leaders

• AI Automation

• Others, please suggest

Ideal instructors are hands-on, clear communicators, and excited to teach.

If this sounds like you, email us at jobs@bytebytego.com with your background, the topics you’d be excited to teach, and any teaching, writing, or speaking samples.

Understanding what each layer does to narrow that funnel can lead to a better grasp of each component of a modern web stack.

In this article, we follow the journey of a web request one hop at a time. At each stop, we ask two questions. What is this layer doing, and what trade-off is it making? The journey starts before the request has fully left the browser, and latency is spent at every hop.

DNS

Read more

New research. New announcements. A new chapter for GitLab.

On June 10, GitLab Transcend streams live from London — and engineers get a first look at GitLab 19 and Duo Agent Platform advancements before anyone else.

Including a live demo of GitLab Orbit: a knowledge graph across your entire SDLC so your agents know your pipelines, your security backlog, and what shipped last week. Not just your repo.

Virtual, free, and just days away.

Register now for free

OpenAI’s data platform stores 1.5 exabytes across 90,000 datasets and serves ~4,000 internal users as of May 2026. The team has scaled the platform through enormous growth in the last two years. At this scale, the hardest part of data analysis isn’t writing SQL. It’s finding the right tables to use in the first place and understanding semantically how to use data. Many tables look similar but mean different things. What’s the grain of each table? How do you join them against other data? Analysts can spend hours figuring out which tables to use and how to use them before writing a single line of code.

Last year, OpenAI’s data platform team built an in-house agent to fix that. The agent is, in their own words, “pretty vanilla”, yet it works reliably across the entire ecosystem. And the same investment in Codex that powers the agent has let the team do things most companies consider impossible, like migrating thousands of DAGs, 90,000 tables and 600 petabytes between clouds in two months.

We spoke with Emma Tang, Head of Data Platform Engineering at OpenAI, about how the agent works, why a simple architecture is enough at this scale due to strong data infrastructure foundations, the lessons for other teams, and where the platform is headed next. Thanks to Emma for taking the time to share the team’s work in detail.

In this article, you’ll learn:

• The architecture behind OpenAI’s data agent, and why “vanilla” is the point.

• The six layers of context that turn a single LLM into a reliable analyst across 90,000 tables.

• How a question becomes a verified answer in three steps.

• Three real Codex use cases inside OpenAI: a 10,000 DAG, 90,000-table cross-cloud migration, hands-off open-source patching, and automated support triage.

• Five practical lessons for any team building a domain agent, and where OpenAI’s data platform is headed next.

How the Data Agent Works

To understand the agent, we will look at three things: what users experience when they ask a question, what architecture supports that experience, and how a request moves through the agent until it returns a verified answer.

The User Experience: Ask in Plain English

Imagine an engineer or marketer at OpenAI who needs a quick answer. They open Slack and ask their questions in plain English. Moments later, the agent replies with its answer, the SQL it ran, and the tables it pulled from. That’s OpenAI’s data agent.

The agent sits across the entire data platform and answers questions in natural language. A user can ask in Slack, in a web portal, in their IDE, or in the Codex CLI through MCP. The agent figures out which tables are relevant, writes SQL, runs it, checks the result, and returns the answer with its reasoning attached.

Doing all of this reliably across 90,000 tables sounds like it would need a complex system. The team’s approach is the opposite of what most people expect. The agent itself is simple. The reliability comes from the engineering around it: careful data acquisition that gives the agent the right context before it ever sees a question. The next sections look at how the agent is built to get that context right.

The Architecture: Simple by Design

OpenAI’s architecture is intentionally simple. Before diving deeper into the architecture, it helps to first understand the basic patterns behind agentic systems.

The basic pattern behind the data agent is an LLM plus a harness. The LLM provides the reasoning. The harness provides the tools and the agentic loop that turns reasoning into action.

The reason you need a harness is that an LLM by itself can only predict the next token. It knows a lot, but it cannot run a SQL query or act on the result. The harness fills that gap. It gives the model tools it can call, like a database query interface, assembles relevant context, and runs the model in a loop so it can reason, act, observe the result, and act again until the task is done.

Many agent systems become complicated at this point as shown in the figure below. A team might add a router that sends easy questions to a small, cheap model and hard ones to a larger model. It might mix multiple LLMs, fine-tune models on internal data, or build complex retrieval pipelines with different embedding models for different content types. Each choice can help, but each also adds cost, latency, and more ways for the system to fail.

OpenAI’s data team took a different approach. They found that a simple architecture works well at their scale, backed by their robust and unified data platform foundation. The data agent they developed consists of four main components: a single LLM, a context assembly layer, a carefully curated set of tools, and an agent runtime.

LLM. The data agent uses GPT-5.5 as the foundation model for every request. The team relies on the model to produce the right SQL queries, inspect the results, correct the queries, and reason its way to a verified answer.

Runtime. The runtime is the orchestrator that drives each request. An LLM on its own only emits text, so something has to act on what it produces. The runtime parses the model’s output, dispatches the requested calls to tools, feeds the results back into the model, and repeats this loop so the model can reason, act, observe, and act again until the task is done.

Context Assembly. This is where the real engineering work lives. A strong model still produces wrong answers without the right context. A bare schema is not enough to tell tables apart. For example, two tables may both have a user_id column and look almost identical, yet one includes logged-out users and the other does not. From the schema alone, the model cannot tell which table answers the question, and picks the wrong one.

To build a richer context, the team identified the signals that actually help the model decide which tables to use and what query to generate: a table’s schema and how people have queried it, notes from the people who own it, and what the pipeline code reveals about how it is built.

Building on these signals, the agent relies on six layers to assemble the right context when a user query arrives:

• Table usage metadata. The table’s schema, its lineage, and a history of how people have queried it. Not all queries are equally useful. Queries from popular dashboards written by data scientists rank highest because they tend to be correct and reusable. One-off, exploratory queries rank lower.

• Human annotations. Curated descriptions written by table owners that capture business meaning, ownership, criticality, and known caveats that cannot be inferred from schemas or past queries.

• Codex enrichment. A nightly Codex job crawls the pipeline code that produces each table. It runs in batches of 100 to 200 tables, with each table taking 5 to 10 minutes. By reading the code, it captures what a table actually contains, how it is derived, how fresh it is, and when to use it instead of a similar table.

• Institutional knowledge. A lot of context about the company’s data lives outside the warehouse, in Slack threads, Google Docs, and Notion pages. These documents are ingested and embedded separately, and served through an access-controlled retrieval service, so the agent never surfaces documents a user is not allowed to see.

• Memory. Corrections and learnings the agent has saved from prior conversations, scoped at global or personal level.

• Runtime context. When the offline context is missing or stale, the agent queries the warehouse directly, and can also talk to other platform systems like Airflow and Spark to fill the gap.

The first three layers, table usage metadata, human annotations, and Codex enrichment, are the ones that describe a table. A daily offline pipeline merges them into a single description per table, and an embedding model embeds that description into one vector per table, stored for retrieval. At runtime, when a question comes in, the tables whose descriptions best match the question are retrieved to be included in the context.

Memory is the other source to assemble the context from. It holds corrections and learnings saved from past conversations, applied on top of the retrieved descriptions so the agent starts from a more accurate baseline instead of repeating old mistakes.

The Figure above shows the overall design of context assembly. Retrieval over the table descriptions identifies the relevant tables, and relevant memory is pulled in as additional context. The last two layers fill the gaps the table store cannot. Institutional knowledge is embedded and retrieved through its own access-controlled service, and runtime context is pulled live from the warehouse when the offline description is missing or stale.

Tools. The agent has access to a small, curated set of 13 tools. These cover company context lookups, internal knowledge bases, big data systems like Airflow and Spark, and metadata services.

The agent uses them to fetch the information it needs to answer a question and verify its work.

The four components described above are the whole architecture of the data agent. There is no router, no fine-tuning, and no special post-training. Every question goes to the same model. According to Emma, the simplicity of the data agent is by design. The real engineering work happens at the infrastructure layer, which builds the right foundation for context assembly.

The Request Flow: From Question to Verified Answer

With the architecture in place, the next question is what happens when a user actually asks something. A question arrives in plain English. The agent’s job is to turn that question into the right context, run it, and return a verified answer. This happens in the following three steps.

Step 1: Embed the Question. The user’s question is converted into a vector using the same embedding model the team used to embed table descriptions offline. This vector is what the retrieval step searches against.

Step 2: Assemble the Context. The context assembly layer searches the vector store for the table descriptions that best match the question, combining semantic search with exact text matching. It also retrieves relevant institutional knowledge from its own access-controlled service, and adds any relevant memory to the context.

Step 3: Start the Agent Loop. The agent sends the assembled context to the LLM and puts it in a loop so it can write a SQL query, look at what comes back from the tool execution, and try again until the answer is correct.

That is the full flow. Three steps from question to verified answer. What makes the agent reliable is the quality of the context that flows through the three steps, which depends on the quality of the underlying infrastructure and how easy it is for the model to reason about. That comes from the six data layers prepared before any user asks a question. This is how the entire company can rely on the agent for critical workloads every day.

For other teams, the useful part of this story is that most components used to build the data agent are available to anyone. GPT-5.5 is on the API. OpenAI’s embedding API is public. Codex is public. MCP is an open protocol. The data platform team did not have access to anything a serious engineering team could not get. What they had was a unified, clean, and robust foundation, a carefully engineered context layer, and a willingness to keep the agent itself simple.

How OpenAI Uses Codex Internally: 3 Use Cases

As pointed out in the previous section, the data platform uses Codex to read pipeline code each night, enriching the context the agent retrieves. That is one use case of Codex. But Codex supports many other use cases internally, helping run the platform itself.

Emma pointed out three interesting ones, different in scope but sharing the same pattern:

• Migrating 10,000 DAGs, and 90,000 tables in two months

• Releasing open-source patches without humans

• Closing the support loop

Use Case 1: Migrating 10,000 DAGs, and 90,000 tables in two months

OpenAI’s data platform was running out of capacity on one cloud provider. The team needed to move the data estate to a second cloud, fast. The migration involved 90,000 tables and 600 petabytes of data, plus hundreds of thousands of interdependent workloads.

The hard part of a migration at this scale is not moving the data. It is the dependency graph. Tables form a DAG. Table B depends on table A. Table C depends on table B. You cannot migrate in arbitrary order. During cutover, some tables live on the old cloud while their downstream consumers are already on the new one. At any point during the cutover, the team must know which copy of each table is the authoritative one, so dependent workloads do not read from a stale source. A system was built to replicate data across clouds in the correct direction while the migration is ongoing. This is for dependency graphs that are in the order of O(100k).

The migration touched hundreds of thousands of workloads, each needing a small code change to point at the new cloud. Filing that many pull requests by hand was not feasible, so Codex generated them instead. Codex Skills then handled the testing and validation for each PR. Around this, the team built a custom system to solve two hard problems: ordering the changes so dependencies migrated in the right sequence, and keeping data consistent while each workload ran against both the old and new cloud during cutover. That system gave Codex the guardrails it needed to operate safely at this scale.

With a strong team, and Codex doing much of the code changes, the migration finished end to end in roughly two months. Comparable cross-cloud migrations at other companies have run for years.

Use Case 2: Releasing open-source patches without humans

OpenAI’s data platform runs on more than a dozen open-source tools, including Spark, Kafka, and Flink. The team keeps its own version of each tool internally, modified with custom patches. Every time a new patch is added, it has to be tested against the existing test suites, validated on staging, and rolled to production.

This kind of work is critical for the platform’s reliability, but it is also repetitive and time-consuming. The test suites are long, with some taking hours and others running for days. An engineer used to babysit each release, watching the tests, diagnosing failures, and rolling the patch forward step by step. With more than a dozen forks, that work added up to a meaningful share of the team’s time.

The team turned the entire cycle over to Codex. A Codex-powered release agent validates patches against the test suites. It diagnoses failures and suggests fixes when something breaks. It rolls the patch all the way to production and alerts the team about what it did.

The release agent has now been running end to end for three to four months without human involvement and without a single incident. What used to require an engineer per release now runs unattended.

Use Case 3: Closing the support loop

A platform that serves 5,500 internal users gets a steady stream of support questions. A pipeline fails. A dashboard breaks. A permission link does not work. Every one of these ends up with the platform team, and each one requires investigation before it can be fixed. At scale, that investigation work used to consume a meaningful share of senior engineering time.

Codex now handles the part of support that used to require investigation. A support bot fields the common questions first and resolves the easy ones directly. When the bot cannot resolve an issue, the engineer on call hands it off to Codex with minimal context. Codex investigates, finds the fix, and applies it. The engineer reviews and approves.

The loop that used to take an engineer a few hours per ticket now lets that engineer dispatch around a hundred fixes per day. The work is not easier. The engineer is amplified.

What Other Teams Can Learn From OpenAI’s Data Agent

OpenAI’s data agent works, but the architecture itself is not what most teams should copy. What is worth borrowing is the set of decisions the team made when they built it. Emma shared five main lessons that apply to any team building a similar system.

The data foundation matters more than the agent

A coding agent has one source of truth: the repository. A data agent’s source of truth is the whole company. Every system, every siloed data store, every team’s conventions, every table defined outside a unified codebase. If none of that is legible to a model, no agent architecture will save you.

OpenAI’s data is well structured. They’ve built best in class industry standard infrastructure across compute, orchestration, metadata management, storage technology, and more. There are no duplicated technologies, and the data lake is unified. Every table on the platform is produced by code in a single monorepo, and data engineering teams enforce conventions and police duplicate or unclear columns along the way. On top of that, every table has strong annotations with the owner, how critical it is, and how fresh the data should be. None of this is glamorous work, but it is what makes a vanilla agent reliable at exabyte scale.

If your team is considering building a data agent and your data is scattered or inconsistent, the agent is not the first investment. The foundation is.

Fewer tools beat more tools

The team initially connected the agent to around 40 tools, including metadata systems, orchestration tools, and big data systems. The results were bad. The model picked the wrong tool and got confused by overlapping answers from tools that did similar things.

Capping the agent at around 13 tools per call, and removing overlapping ones, fixed the problem. The lesson is that an agent does not need access to every system in the company. It needs access to the right ones, with no two tools doing the same job.

In practice, that means avoiding overlap. If two metadata services expose similar information, the agent should only see one. If two ways exist to look up table ownership, pick one. The model is better at reasoning than at choosing between near-duplicate tools.

Pick trusted queries for retrieval

A natural first instinct for a data agent is to embed all historical queries and use them as context. OpenAI tried this approach, and it did not work. Most queries in any company are exploratory one-offs, not canonical examples of how a table should be used.

The team improved results by ranking past queries by how trustworthy they are. The agent learns from queries the company has already written, but not all of them are worth imitating. Queries behind heavily used dashboards, usually written by data scientists, rank highest, because they tend to be correct and reused often. One-off queries written for a single analysis and never run again rank lowest. Once the team ranked queries this way, the model started copying the good patterns instead of the bad ones.

The general lesson is that retrieval quality depends on the quality of what you retrieve. What you feed into retrieval is what you get back from it.

Guide the goal, not the path

Prescriptive prompts hurt the agent’s results. The team experimented with detailed step-by-step instructions about how the agent should approach each kind of question. The agent followed the instructions and produced worse answers. High-level guidance worked better. Tell the model what the goal is. Let it reason about how to get there. Give it the right context and the right tools, and trust the reasoning.

This matches what other teams building agents have found. Modern models are good at planning when they have good information. They are less good at being told what to plan.

Be more ambitious

The cross-cloud migration was supposed to be impossible in months. The team’s initial estimate was longer. Emma pushed for two months because the company was running out of capacity and a longer timeline was not an option. The team hit the deadline. Comparable migrations at other companies can take years.

The takeaway the team came away with is that timeline estimates from before Codex no longer apply. If a project sounds like it should take a year, the right question is whether it could be done in a quarter with an agent doing most of the work. The bigger risk is not over-promising; it’s playing it safe. A team that sticks to old timelines never finds out what the new tools make possible.

What Comes Next for OpenAI’s Data Platform

The data agent and the internal Codex use cases are not the end of the story. Emma says two things sit on the team’s near horizon: custom apps, and platforms keeping pace with users.

Custom apps generated per question.

Most analytics tools today give users a fixed set of widgets, like bar charts, line charts, and pivot tables. They are useful, but limited. If your question does not fit the available widgets, you have to write a custom script or file a request with the data team.

OpenAI wants to go further. The agent already builds traditional dashboards on demand, but the next step is freeform ones. Instead of a chart, Codex would build a full React app connected to a backing store and tailored to the exact question asked. Each one takes seconds to build, fits a single user’s need, and runs on real data with real guardrails.

When this rolls out, users will no longer pick from a fixed set of widgets. They will describe what they want and the app will appear, generated per question. A marketer who wants to explore campaign performance with a custom filter and a tailored layout will simply ask.

Users move faster than platforms can keep up.

The same Codex that powers the data agent has accelerated every team at OpenAI. Frontend engineers vibe-code new UIs in a morning. Researchers spin up custom pipelines on demand. Platform teams cannot move at that speed safely. A bad UI affects a few users, but a bad change to shared infrastructure can take the whole company offline.

This leads to a mismatch. Users now ship code to the platform faster than the team can review and validate it, and some of that code is written by people who do not fully understand what it does. Emma described examples like when a bad Flink job lands on the cluster and brings it down, upon asking the user replies, “I don’t know, I don’t know how Flink works, it’s vibe-coded. Can you help fix it?”

This is the next problem the data platform team plans to work on. The fix will not be another user-facing agent, but agents on the platform side, designed to triage incoming code, validate it before it runs, and absorb the deluge from AI-amplified users. The previous wave of agents helped users do more. The next wave will help platforms keep up.

Most teams pick a search provider by running a few test queries and hoping for the best – a recipe for hallucinations and unpredictable failures. This technical guide from You.com gives you access to an exact framework to evaluate AI search and retrieval.

What you’ll get:

• A four-phase framework for evaluating AI search

• How to build a golden set of queries that predicts real-world performance

• Metrics and code for measuring accuracy

Go from “looks good” to proven quality.

Learn how to run an eval

Few people in tech have a clearer view of AI-native engineering at hyperscale than Shah Rahman. As Global Head of Autonomous ML Iteration & Optimization for Ads at Meta, Shah spends his days architecting AI-native infrastructure and multi-agent systems that make ML iteration reliable across one of the largest production environments on the planet.

In the piece below, Shah cuts through the “everyone is an engineer now” noise and lays out what AI-native engineering actually requires: context engineering, spec-driven development, critical verification, and disciplined problem decomposition. He walks through the Agentic Development Life Cycle, the journey that separates real 10x leverage from “faster failure,” and the security guardrails that are no longer optional.

If you’re moving your engineering org toward becoming AI-native, this is a strong playbook.

Let’s get into it.

For more from Shah, connect with him on LinkedIn.

AI generates more than 75% of Google’s new code. OpenAI and Anthropic claim that almost every line of fresh code that they produce comes from AI. Amazon recently migrated 30,000 of its production applications from Java 8 to Java 17 in a matter of months, a project that would otherwise have taken an estimated 4,500 developer-years. And Mark Zuckerberg expects that AI agents will be operating as mid-level engineers by the end of 2026.

Reading those statements, we may feel as if we are looking at the last lines being written on the closing pages of an era. Perhaps even the closing pages of a profession.

But here’s the question: If AI writing everything is the answer, then why are most engineering teams shipping more bugs, more incidents, and more technical debt than they shipped two years ago?

In an April article in the New York Times, Mike Isaac and Erin Griffith gave a name to describe what’s happening across the industry. They called it code overload.

The essence of code overload, according to Isaac and Griffith, is that “tech workers are producing so much code so quickly that it has become too much to handle.” Teams that have rebuilt their work around the use of AI agents are drowning in code churn and security holes.

But. Many engineers who have employed AI agents are pulling ahead of the field, achieving real productivity gains. They are using the same models and the same tools, but they are generating very different outcomes. What explains the gap?

It comes down to one decision. Real productivity gains come when engineers decide to make the leap from writing code to orchestrating it. This piece is a working guide for engineers who want to land on the productive side of that split. It will cover the practices, guardrails, and mindset shifts that separate AI-native engineering from vibe coding and from the everyday chaos that most teams are now generating at scale.

From Engineer to Orchestrator

Let me first clarify one thing: engineers are not becoming obsolete. Coding has always been a small part of engineering (20-30% max). This underappreciated reality is more visible when AI agents and tools produce more code, but more code is not necessarily more productive (often it’s less). This is a critical distinction that the industry is blurring dangerously, and I state that as:

When Andrej Karpathy coined “vibe coding” in early 2025, it captured something useful — the ability for non-engineers to build functional software by describing what they want. That democratization is valuable. But it’s categorically different from professional AI-native engineering.

AI-native engineering means commanding and mastering available and emerging AI agents and tools to engineer things that weren’t possible in the pre-AI era. Knowing how to code remains a fundamental expectation. Without that knowledge, you can build systems using AI — and that’s vibe coding. It has its place, but it’s not engineering.

The AI-native engineer operates as an orchestrator — someone who can turbocharge 10x engineering into 100x output through proper orchestration of AI agents. And that bar continues to rise weekly.

The Four Core Practices

1. Synchronized Context Engineering

Emerging as a distinct discipline, this is the single most important skill for AI-native engineers. Context engineering means the systematic curation and injection of project-specific information into AI working memory: architectural diagrams, coding standards, business rules, team conventions, and development workflows that are reusable and standardized across your team members.

This shifts basic “prompt engineering” to sophisticated “context engineering” reflecting a deeper understanding: the quality of AI output is bounded by the quality of context it receives. Teams practicing rigorous context engineering report 40–50% speed increases and dramatically reduced alignment overhead.

As context engineering matures, Anthropic’s MCP — described as “USB-C for AI” — continues to be a universal standard for connecting agents to external tools and data sources. Context files like CLAUDE.md have become core infrastructure, not optional documentation. This persistent, evolving knowledge layer makes agents genuinely useful within your specific codebase, once you master developing and maintaining this critical context.

2. Specification-Driven Development

The quality of AI-generated code matches the quality of input specifications. Garbage in, garbage out — this principle applies with even more force when AI can generate garbage at unprecedented speed and volume.

Random prompting and vibe coding consistently underperform spec-driven workflows. AI agents get stuck in circular reasoning without clear specifications and instructions that are contained and well-defined. Consider this discipline: define what you want before asking AI to build it, break problems into discrete milestones with clear success criteria, and execute incrementally with validation at each checkpoint. Make sure the agent checks all open Qs with you and doesn’t run off on its own to find answers.

3. Critical Verification

AI-generated code quality approximates that of early-career developers. Research consistently shows that around 45% of AI-generated code contains security flaws. A Stanford study found that developers using AI assistants wrote significantly less secure code and were more confident it was secure — a dangerous combination.

Meanwhile, a striking METR/Anthropic randomized controlled trial found experienced open-source developers were actually 19% slower when using AI assistants on familiar codebases. The culprit? Over-reliance without adequate verification. A GitClear study found AI-assisted codebases showed increased “code churn” — code written and then quickly revised or deleted — suggesting raw output is a poor proxy for productivity.

In the AI-native era, the bottleneck has permanently shifted from writing code to proving that it works at scale, with reliability and security. When AI generates code quickly, review, testing, and verification of that code become the new rate-limiting factors, and verification now becomes non-negotiable.

4. Problem Decomposition

Avoid over-trusting AI with large, complex problems. Break tasks into AI-manageable chunks where humans handle edge cases, custom logic, and domain-specific aspects while AI agents handle the 70–80% of routine implementation. Complex problems lead to context pollution and slop generation that AI agents really struggle to recover from. Compacting and summarizing when context is polluted and shifting to a different session helps, but this discontinuity can be damaging for long-horizon tasks.. Many of us wasted hours, if not days, due to not decomposing and stubbornly confusing agents about expectations outside of a well-defined context, reasonable specifications, and a lack of verification guardrails.

Time Allocation for AI-Native Work

I recommend the optimal split of: 40% context-setting, 20% generation and testing iteration, 40% reviewing and verification. This surprises many developers who spend most of their time in code generation. In practice, the generation step is fast; the verification and context work become the new time sink.

The Individual Transformation Journey

Phase 1: Foundation — should only take a couple of weeks

Begin with one primary AI assistant — pick your favorite one: Codex, Claude Code, or Cursor. Dive deep and build intuition for its capabilities and limitations through daily practice. Set up your workspace, workflow, and initial configurations. You’ll have to take the leap from the times of manual coding to AI-assisted and AI-generated coding practices. Your goal should be to develop judgment about when AI delivers value versus when it creates more work than it saves. Write down your personal notes, iterate, and build a strong foundation.

Phase 2: Integration — should take a month max

Adopt structured prompting frameworks. Create project-specific context files encoding team standards and architectural patterns. Implement the “Plan first, then Execute and finally review” workflow: planning mode generates specifications, execution mode implements, and make sure you review after each atomic task. Establish approval gates and guardrails that prevent agent drift. Skipping the review will pile tech debt that you and your agent will both struggle downstream.

The critical practice here is small loops with verification checkpoints. Evidence shows tight human-in-the-loop cycles with limited scope dramatically outperform large autonomous runs, at least for coding tasks. This may feel counterintuitive and slower — but it produces dramatically better outcomes in practice. Indulging into somewhat unplanned and speculative autonomous agent runs will likely produce a large volume of slop whose only destiny may be throwaway and start all over again. Avoid that before it happens.

Phase 3: Mastery — live on

Deploy AI agents for multi-step, multi-file tasks. Implement AI-assisted code review workflows. Use advanced techniques: multi-agent workflows, parallel sessions, and cross-agent verification loops. Every week, we hear about coding agents advancing on benchmarks and solving problems that were never solved before. Stay on top of those developments and embrace what Claude or Codex inventors are advocating, but adopt those to your needs (don’t blindly follow as their situations may be wildly different than yours).

Target metrics: 80%+ AI-generated coding rate with less than 20% rewrite rate. Achieve this, and you can pull your team toward the same proficiency level rather fast.

Team Transformation: The Cultural Foundation

Research shows 70% of transformation success comes from operational and cultural change. These changes call for Organizational and technical leads to actively model transformation through daily AI usage. At the same time, ensure three critical aspects to establish the AI-native cultural foundation:

1. Psychological safety is paramount. MIT research found 83% of leaders believe psychological safety measurably improves AI initiative success. Celebrate “AI failure stories” as learning opportunities. Make this deliberate practice, not optional and ensure everyone feels included as part of the collective learning and growing exercise.

2. Evolved code review is essential. AI-generated code volume overwhelms traditional human review processes. Redesign review to distinguish AI-generated versus human code with separate review rubrics. Be especially vigilant about the dangerous combination of AI-generated and AI-reviewed PRs — these combinations should be explicitly guardrailed and governed, when necessary.

3. Shared context libraries become the core currency. Standardize context files, evaluation sets, and agent configurations across teams. Modern tooling enables easy packaging of context through plugins, skills, and commands — but watch for uncontrolled proliferation, where teams compete for standardization rather than collaborating. Don’t let too many team members’ desire to build agents and skills jeopardize your standardized agentic operating environment.

The Agentic Development Life Cycle (ADLC)

Traditional SDLC — and even extreme agile — falls short for how AI agents develop software alongside humans. AI-native engineering evolution toward an Agentic Development Life Cycle redefines each phase.

Planning

The most critical step. Use deep research and planning modes with multiple agents for parallel exploration. Specify against codebases, flag ambiguities, decompose into subtasks, and estimate difficulty. Create roadmaps with version milestones that help agents follow through incrementally. A planning agent can assemble findings from multiple exploration agents into a coherent implementation strategy. OpenClaw of Claude can run in multiple sub-agent in parallel.

Building

AI agents handle end-to-end feature implementation like junior or mid-level engineers (at the time of this writing, which I expect to edge up to senior engineers within a year or two). The engineer acts as the tech lead, orchestrating multiple agents rather than coding directly. Sequential or parallel execution models depend on your roadmap and verification plan. The agentic coding tool landscape has matured rapidly — Claude Code, Cursor’s Composer mode, GitHub Copilot’s Agent Mode, and OpenAI’s Codex agent all support this pattern with varying strengths. There’s new versions coming out every month -- watch closely for new capabilities.

Testing

This is TDD reincarnated. Agents write test plans first, then implement code. All tests should fail at the beginning and then incrementally pass. Unit testing at the atomic level, integration testing across features, and end-to-end testing across the system. Don’t overindex on unit testing at the expense of integration or system testing lack.

Pro Tip: Consider separating planning, building, and testing agents. Each agent swarm specializes and develops a deep understanding of your codebase from a different perspective. Planning agents can challenge building agents who take shortcuts; testing agents who skip coverage; or the review agents who are biased towards incorrect implementations that appear to be correct. Similarly, review agents can hold every other upstream agent accountable for making mistakes or missing steps.

Review

Deploy agent swarms specializing in key dimensions: functionality, quality, scalability, performance, reliability, security, and privacy. Agents take the first pass and produce reports; humans review each report carefully. When one agent discovers an issue — say, an injection vulnerability — apply the generalization principle: if one instance exists, others likely do too and this is your chance to proactively similar a vulnerability type in your code, not just one or two instances.

Documentation

Move from post-facto documentation to continuous generation. AI agents generate summaries, key design decisions, architectural diagrams, and changelogs in real time. This flows naturally into API documentation, feature collaterals, and customer-facing contents. I’m quite excited about AI tools finally solving the outdated, stale, and inconsistent documentation problem that I’ve seen myself and my teams suffer for decades.

Codify ADLC

Encode your Layer-1 (individual) and Layer-2 (team) practices into maintained, self-evolving context files, skills libraries, and MCP tools. This ensures ADLC adoption scales across the organization rather than remaining tribal knowledge or trapped in parts of the org. Promote the ADLC tooling package.

What AI-Native Process Actually Looks Like

There’s a seductive narrative in the industry right now: fewer people, less overhead, faster builds. But this narrative conflates construction costs with decision costs. AI has drastically reduced the cost of building, but that represents only 20–30% of total development costs, while leaving the cost of deciding what to build and what to cut largely untouched. With the proliferation of code and builders, this becomes a harder problem.

AI-native process optimization requires redirecting effort from coordinating execution to accelerating learning.

The Learning Loop

AI compresses the first step dramatically. But compression value depends entirely on execution quality throughout the remaining cycle. Faster building without robust user observation and scope discipline produces faster divergence from genuine product goals. This results in customers not seeing the benefits of AI acceleration.

Where AI Creates Genuine Leverage

Cheaper experimentation. Test more hypotheses per unit time. Over 70% of features never reach real users. AI makes it trivially cheap to test whether something matters before committing to full development. The discipline: kill non-viable concepts ruthlessly.

Faster prototyping for user research. Working prototypes replace documentation. Tools like Vercel’s v0, Replit Agent, and Bolt.new enable functional prototypes from natural language in minutes. This produces superior signal quality from user testing. Encourage everyone to prototype aggressively, make it a habit before you build.

Automated boilerplate, not automated judgment. AI handles undifferentiated work: scaffolding, non-novel code, business logic tests, documentation, and data models. Teams focus on differentiated work: core business logic, empathetic user experiences, novel implementations, and the crucial decision of what to keep or kill.

The “design to 50%” principle. Ship minimal functionality enabling core user journeys. Observe where users hesitate, misunderstand, or abandon. This reveals actual product challenges rather than imagined ones. AI makes this approach nearly zero cost.

Guardrails — Not Optional

The security landscape for AI-generated code has become genuinely alarming. The data shows AI-native development speed is creating new attack surfaces faster than manual security review or traditional tools can address. We observed roughly one new insecure AI integration appearing per week in our environment, many resulting in production incidents. Anthropic’s Daybreak and Mythos bring a clear wake-up call to security.

Real Incidents, Real Consequences

Chat Integration RCE. Built in two days using AI, achieved Remote Code Execution by bypassing 2FA and exploiting open ACLs. It costs tens of hours to detect, mitigate, and fix.

Unauthorized Database Access. An AI coding agent accessed approximately 1,500 secure, unauthorized database tables without proper authorization, exposing sensitive data to prompt injection risks.

Google Docs Prompt Injection. An AI coding agent achieved Remote Code Execution through prompt injection embedded in a Google Docs document, bypassing input filtering protections entirely.

Supply Chain Poisoning. A new attack vector called “slopsquatting” emerged in 2025 — AI models hallucinate package names that don’t exist, and attackers register those names with malicious code. Multiple documented incidents have resulted from this.

Emerging Security Controls

Agent Identity and Access Control. Implement step-up 2FA. Apply the principle of least privilege. No shared credentials or open ACLs. Start with passive, read-only use cases and build confidence before granting read-write or broader access.

Data Classification Awareness. Agents must respect data classifications and sensitive boundaries. “Agentic Authorization” is an emerging enterprise challenge where agents bypass restrictions at machine speed that human oversight cannot match.

Prompt Injection Protection. External content — documents, web pages, user inputs — can contain hidden instructions that hijack agent behavior. Implement input filtering, content validation, and context sanitization. Never auto-execute untrusted commands. Resist the temptation of auto-accepting all agent suggestions.

Infrastructure Sandboxing. Agent activities must be observable and auditable. Block high-risk production surfaces — configurations, critical execution, critical storage — until controls are verified. Use sandboxing and OS-level enforcement.

Technical Guardrails

Static analysis integration. Data shows roughly 30% of Python and 25% of JavaScript AI-generated snippets contain security weaknesses. Centralize advanced static analysis in CI/CD pipelines. Require mandatory human review for critical functions: authentication, payments, and PII handling.

Automated quality gates. Implement “Ralph Loops”, OpenClaw, or another form of autonomous loops — iterative verification until success criteria are met. Type checking, linting, and test execution before diff submission. Multi-stage canary systems with stringent gates before production deployment.

Skills-based security. Where the agents are taught secure coding patterns, flagging common vulnerabilities during generation rather than after. Shift left, but with agents.

Organizational Guardrails

Skill atrophy prevention. Gartner reports 50% of organizations will require “AI-free” skills assessments by 2026. Treat AI as a learning tool — request explanations alongside generated code. Occasionally, work without AI to preserve foundational abilities. The goal isn’t Luddism; it’s insurance against the day your AI tools are unavailable or producing subtly wrong, but potentially fatal results.

The productivity paradox. Individual productivity gains from AI tools often fail to materialize at the team and company levels. Focus on end-to-end cycle time and feature velocity, not coding speed alone. Adding AI to broken processes yields broken processes that generate more code, faster.

The Engineer of 2026 and Beyond

The engineers thriving in the new environment treat AI as a collaborative partner for execution while maintaining the systems thinking, critical judgment, and communication skills that no AI can replicate. AI amplifies existing expertise rather than replacing it — senior engineers achieve dramatically better results because they bring deeper context and sharper judgment.

Your domain expertise is the key differentiator in AI-native productivity. No AI tool or agent can replace it. So, invest in sharpening your domain skills, whether that’s math, science, finance, health science, or a legal profession. Continuing to uplevel your engineering fundamentals pay recurring dividends in AI effectiveness.

This is a multi-year transformation, not a one-off tool adoption. Teams treating it as a tooling upgrade consistently fail to realize productivity gains. The organizations that succeed are the ones treating AI-native engineering as a new way of working — with new practices, new disciplines, and new definitions of what “amazing” looks like.

Are you building this way yet? If not, ping me, and I’m happy to have a chat.

This is Part 1 of a two-part series. Part 2, “AI-Native Leaders,” covers the organizational transformation, leadership models, and measurement frameworks required to make AI-native engineering work at scale.

Datadog’s guide shows you how to connect AI spend, infrastructure, and model performance into a single view, so you can catch cost spikes the moment they happen. See how Kevel cut AWS costs by up to $100,000/month after replacing reactive cost reviews with real-time visibility.

You’ll learn how to:

• Break down AI costs by token, model, provider, and team

• Get alerted the instant inference volume spikes or API spend exceeds budget

• Correlate cost increases directly to architecture changes so root-cause analysis takes minutes

Get the guide

DoorDash’s customer support chatbot had a hallucination problem. Not the dramatic kind where it invents entire conversations, but the subtle, harder-to-catch kind.

For example, the chatbot would look at a customer’s order history, see a delivery status field, misread it, and then confidently suggest a refund policy that didn’t actually exist. The raw data was right there in the chatbot’s context window, the working memory where an LLM holds everything it needs to generate a response, but having too much information was making things worse.

For reference, DoorDash is one of the largest food delivery and local commerce platforms in the United States, connecting customers with restaurants and stores through a network of independent delivery drivers called Dashers.

At that scale, the company handles hundreds of thousands of support contacts every day from customers, merchants, and Dashers, making automated support not just a nice-to-have but a necessity.

The team could see the problem clearly, but fixing it was a different story. Every change they made to reduce hallucinations in one scenario risked creating new ones in another. They were stuck between two bad options. They could deploy changes to production and hope for the best, which meant risking real customer experiences. Or they could manually test dozens of conversation scenarios for every prompt change, which would take weeks and still might miss things.

This tension isn’t unique to DoorDash. It’s the fundamental challenge anyone faces when they move from traditional deterministic software to LLM-based systems. DoorDash used to run customer support on hand-built decision trees, where every change had a predictable, traceable impact. LLMs replaced that predictability with flexibility and more natural conversations, but they also introduced non-determinism, meaning the same input can produce different outputs each time.

DoorDash’s answer to this problem wasn’t a better chatbot. It was a better system for improving the chatbot, something they call the simulation and evaluation flywheel. In this article, we will learn how they built this flywheel and the key takeaways.

Disclaimer: This post is based on publicly shared details from the DoorDash Engineering Team. Please comment if you notice any inaccuracies.

What the Flywheel Actually Does

The flywheel has two interconnected pieces:

• The first is an offline simulator that generates realistic multi-turn customer conversations without involving any real customers.

• The second is an evaluation framework that automatically grades how the chatbot performed in those conversations.

Together, they create a tight iteration loop.

When the team notices a problem, they write an evaluation that captures the specific failure mode they want to fix. A single job trigger then orchestrates the entire pipeline end-to-end, automatically generating test scenarios from historical transcripts, running multi-turn conversations between the simulator and the chatbot, and evaluating the results.

Then they modify the prompt or the system architecture, run the simulator again, and check whether the pass rate climbed. If it did, they would keep going. If it didn’t, they try something else. They repeat this cycle until the pass rate hits their exit criteria, and then they deploy with confidence that the change actually works.

The graph below shows the pass rate for no-hallucination evaluation over time

The speed of this loop makes this a powerful approach. DoorDash can run more than 200 simulated conversations in under five minutes and get automated evaluation results immediately.

In other words, what used to take days of manual testing and review now takes hours. And because everything happens offline, they never risk degrading the experience for real customers while they iterate.

Their evaluation suite has grown to more than 50 evaluations covering hallucination detection, tone assessment, issue classification, and other quality dimensions. Before any change goes to production, it must pass the full suite, which serves as both a quality check and a regression test.

The flywheel sounds straightforward, but both the simulator and the evaluator required solving genuinely hard problems.

FeatureOps Summit 2026 - Feature management in the AI Era (Sponsored)

Speed without control is a false economy. As AI code-generation accelerates software delivery, the FeatureOps Summit 2026 is here to ensure that when we ship more, we break less.This premier virtual event brings together engineers, architects, and product leaders to explore the infrastructure of fearless delivery.

Key Themes:

• AI Safety Nets: Guardrails for the flood of automated code.

• Edge Resilience: Sub-millisecond evaluation at scale.

• Continuous Flow: Moving past the “fixed-release” mindset. Register today to master the tools and patterns required for a fail-safe release environment.

Register Today

Simulating Customers That Push Back

A static test case can check whether the chatbot gives a reasonable answer to a single message, but it can’t capture what happens when a frustrated customer pushes back three times, provides additional information mid-conversation, or threatens to escalate.

DoorDash’s simulator doesn’t use scripted messages at all.

Instead, it uses an LLM to play the customer role, generating dynamic responses based on detailed test scenarios. At each turn, the simulator runs through a structured analysis, asking questions such as:

• Was the issue addressed?

• Is the conversation making progress?

• Does the customer need to provide more information?

• Is the conversation going in circles?

Based on this analysis, it decides what a realistic customer would say next.

The test scenarios themselves come from real historical support transcripts, not from engineers imagining what customers might say.

LLMs analyze past conversations from DoorDash’s database and extract structured behavioral profiles, including the customer’s personality traits (frustrated and demanding versus confused and patient), a detailed narrative of the situation, and the specific outcome the customer is seeking. This grounds the simulator in actual customer behavior rather than idealized test cases.

See the diagram below:

The simulator also exhibits realistic escalation patterns. It doesn’t immediately ask for a manager. Instead, it gives the chatbot multiple chances to resolve the issue, only escalating after repeated unhelpfulness or circular exchanges, and re-engaging when progress becomes clear again. This mirrors how real customers behave.

For a simulated conversation to be meaningful, the chatbot also needs realistic backend data. It needs to look up delivery status, check refund eligibility, and pull order details. DoorDash handles this through mock data that blends real production data with scenario-specific test data, preserving timestamps and relationships to keep interactions realistic. This allows them to test complex edge cases, including fraud scenarios and high-value refunds, that their previous testing infrastructure couldn’t handle.

Using an LLM to Judge Another LLM

Running hundreds of realistic conversations is only useful if you can tell whether the chatbot actually handled them well. However, manually reading through every simulated conversation would defeat the entire purpose of automation. So DoorDash uses an LLM to evaluate the chatbot’s performance automatically.

Each evaluation is structured as a function that takes the full conversation transcript (including tool calls and backend responses) along with the relevant company policy, applies a prompt asking whether the chatbot correctly followed that policy, and returns a binary pass or fail with the reasoning behind the judgment.

The obvious objection here is that this sounds circular. If an LLM caused the problem by hallucinating, why would you trust another LLM to catch the hallucination?

DoorDash addresses this directly with a concept they call the generator-verifier gap. Acting as a full customer support agent involves complex, multi-step decision-making across a huge range of possible scenarios. That’s genuinely hard. But verifying a single, narrowly-defined behavior is a much simpler task.

For example, “Did the chatbot claim the customer was eligible for a refund when the policy says otherwise?” is a straightforward binary question. The evaluator isn’t trying to be a better support agent. It’s checking one specific thing at a time, and LLMs are much more reliable at these focused binary judgments than they are at open-ended generation.

But DoorDash doesn’t just trust the LLM judge out of the box. They calibrate it against human judgment through a structured process. They collect a sample of conversations, have human experts label each one as pass or fail, run the LLM judge on the same samples, and then measure how often the judge agrees with the humans and how often it misses problems or flags false ones. They analyze the reasoning behind any mismatches, revise the evaluation prompt to fix systematic errors, and repeat until the judge reliably matches human expert judgment. This calibration step creates trust in the system.

The binary nature of the evaluations is important here. DoorDash isn’t asking the LLM to rate the chatbot’s performance on a subjective scale of 1 to 10. They’re asking whether the chatbot followed a specific policy or not. It makes calibration faster, makes disagreements easier to diagnose, and produces more reliable judgments.

Fixing Hallucinations by Giving the Chatbot Less Information

With the simulator generating conversations and the evaluator grading them, DoorDash had a working flywheel.

During early launches, human reviewers noticed the chatbot was getting overwhelmed by the sheer volume of data in its context window. Order histories, delivery status updates, refund decisions, and tool call results were all being fed directly to the model as raw event logs. The chatbot would misinterpret a field or suggest a policy that didn’t exist, not because the information was wrong, but because there was too much of it. This runs directly counter to the intuition that giving a model more information should produce better results.

DoorDash hypothesized that the same data that was vital for the chatbot’s reasoning was becoming noise when it came time to generate a response to the customer. Their solution was an architectural layer they called the “case state,” which synthesizes the raw tool history into a structured, intermediate representation. Instead of dumping everything into the context window, the case state distills the relevant facts into a clean format that the chatbot can actually use.

Getting the case state right required the flywheel. Their first attempts at extraction logic didn’t work well at all. Some versions left out critical information, causing the chatbot to miss details that were essential for driving resolutions. Other versions remained too noisy or poorly structured, confusing the model in different ways. Since the simulator could generate numerous realistic conversations in minutes, the team experimented with dozens of different context shapes and prompt strategies in a rapid feedback loop. Each iteration took hours instead of the weeks it would have required through manual testing.

Over 11 iterations, the hallucination evaluation pass rate climbed steadily upward, with a notable dip at iteration 3, where a change actually made things temporarily worse. That dip shows that improvement isn’t linear, even with a flywheel, and that part of the flywheel’s value is catching regressions before they reach real customers.

The final result was a 90% reduction in hallucinations in simulation, and that improvement carried over into production. The strong correlation between their offline metrics and live traffic performance gave the team confidence that the flywheel is a reliable development tool, not just an internal sandbox disconnected from reality.

Conclusion

The simulation and evaluation flywheel has fundamentally changed how DoorDash develops and deploys chatbot improvements, compressing iteration cycles from days to hours and giving them a way to validate changes across hundreds of scenarios before any real customer is affected.

However, the flywheel does come with real tradeoffs worth understanding.

The main limitation is that it can only catch problems for which you’ve written evaluations. If a failure mode isn’t captured by an evaluation, the flywheel is blind to it. DoorDash mitigates this by running a full evaluation suite before every deployment, covering hallucination, tone, and issue classification, but new failure modes can always emerge that existing evaluations don’t cover. This is why human review remains the starting point for every improvement cycle. Despite all the automation, someone still has to look at real conversations and notice what’s going wrong.

Simulation fidelity is another inherent limitation. Even with transcript-derived scenarios and hybrid mock data, synthetic conversations are approximations of real user behavior. DoorDash reports a strong correlation between its offline metrics and production results, which validates the approach, but that correlation isn’t guaranteed to hold for every type of scenario or every kind of system change.

There’s also the question of cost. Running hundreds of LLM-to-LLM conversations per test cycle, plus LLM-as-judge evaluations on each one, requires significant compute. For smaller teams or less critical applications, a lighter-weight version with fewer scenarios and more targeted evaluations might be the pragmatic starting point.

The broader takeaway is that LLM systems require a completely different testing paradigm than traditional software. Since we can’t trace the branch anymore, we need a feedback loop that lets us simulate, evaluate, and iterate fast enough to build confidence before shipping.

References:

• A Simulation and Evaluation Flywheel to build LLM Chatbots at Scale

• LLM as a Judge Pattern

On a single machine, the answer is straightforward, since a program is either running or it has crashed, and the line between the two is usually obvious from a stack trace.

Distributed systems are not so simple. Every server can report healthy while users are seeing errors, the whole system can be technically working but stuck in a state it cannot recover from on its own, and it can quietly serve wrong data while every dashboard glows green.

None of these may be because of bugs in the conventional sense. They are recurring failure patterns that have been showing up across systems for decades, with names, mechanisms, and standard ways of defending against them.

In this article, we will look at the most significant failure mode patterns in distributed systems and the standard approaches to deal with each of them.

Why Failures in Distributed Systems Are Different

Read more

Sign-up forms were built for humans in browsers, so how do AI agents programmatically register with services?

Enter auth.md. By exposing a single, machine-readable Markdown file at your service root, AI agents can dynamically discover your OAuth Protected Resource Metadata, parse required scopes, and authenticate seamlessly.

With native support in WorkOS AuthKit, you can now implement this protocol out of the box, giving AI tools a standardized, secure way to log into your application.

Read the auth.md docs

Airtable holds embeddings for hundreds of thousands of customer databases, and on any given week, roughly three-quarters of them sit completely idle. This fact, more than any algorithm or vendor choice, decided the architecture behind their semantic search system. The interesting story is not which vector database they picked. It is how one peculiar property of their data forced a specific chain of engineering decisions, each one logical only in light of the one before it.

Airtable is a platform where customers build their own database-like applications, organized into “bases” that often hold hundreds of thousands of rows. Their AI feature, called Omni, lets users ask natural-language questions of their data and get answers back in plain English. A separate feature, linked record recommendations, suggests relationships between rows based on meaning rather than exact text matches. Both features depend on the same underlying capability, which is finding the rows in a base that are semantically relevant to a user’s intent.

This might sound simple until scale enters the picture. When a base has half a million rows, fitting all of them into a single LLM prompt becomes infeasible. The model has limits on how much context it can absorb, and even if those limits did not exist, sending that much data on every query would be slow and expensive. The system has to find the most relevant rows fast, then hand those rows to the LLM as context.

In this article, we will look at how Airtable’s data infrastructure team built its architecture, the challenges they faced, the tradeoffs they accepted, and why the choices they made only make sense once their data is properly understood.

Disclaimer: This post is based on publicly shared details from the Airtable Engineering Team. Please comment if you notice any inaccuracies.

The Data and the Constraints

The Airtable team anchored their work around four design priorities:

• Queries had to return within 500 milliseconds at the 99th percentile, which means the slowest 1 percent of queries still had to come back within that window. Anything slower would make the AI features feel sluggish.

• Writes had to be high-throughput since customer data changes constantly, and embeddings have to keep pace.

• The system had to scale horizontally to support millions of independent bases.

• Everything had to be self-hosted because customer data privacy required keeping it all inside Airtable-controlled infrastructure.

Beyond those priorities, Airtable’s data has three properties worth flagging early:

• Customer bases vary enormously in size, with some holding a handful of rows and others holding hundreds of thousands.

• Each base is isolated, meaning one customer’s data must never leak into another customer’s results.

• Most bases are idle most of the time, a fact that becomes important in a later section.

Before going further, we need to understand what an embedding is.

An embedding is a list of numbers, typically several hundred or a thousand of them, generated by a neural network. The network is trained so that two pieces of text with similar meanings produce numerically close vectors. An embedding can be thought of as a fingerprint of meaning, where similarity in the numbers reflects similarity in what the text says.

One important practical fact is that embeddings are typically about ten times the size of the original data they represent, which is why Airtable cannot just store them alongside the source rows in their primary database. A separate system is needed, one designed specifically for storing and searching across these large numerical vectors.

The asynchronous embedding pipeline that generates and updates these vectors as customer data changes is a separate system, which is the database that stores the embeddings and serves queries against them. After evaluating the landscape in late 2024, Airtable selected Milvus as its database. This is because Milvus supported self-hosting, handled multi-tenancy through its partition model, and let them scale ingestion, indexing, and query execution as separate components. Picking Milvus, though, was the easy part. The hard part was figuring out how to organize Airtable’s data inside it.

See the diagram below:

Partitioning Strategy

The first real architectural question was how to slice up customer data so that millions of bases can coexist in one system without leaking into each other.

Two options were on the table.

The first option of shared partitions would put many bases together in the same physical slice and rely on a customer ID filter at query time to keep results separate. This approach uses resources efficiently because there is no partition for every customer, and small bases do not sit around taking up dedicated storage. The cost is that every query carries the overhead of filtering by customer ID, and deleting a customer’s data becomes complicated because the rows are scattered across shared partitions.

The second option of having one partition per base gives each customer their own physical slice. Queries are naturally isolated because they only ever touch one partition. Deletion is trivial since dropping the partition is enough. The cost is operational. With millions of customers, the database ends up managing millions of partitions, which puts pressure on its internal bookkeeping.

Airtable picked the second option. The reasoning was that strong physical isolation made permission boundaries obvious, deletion stayed simple, and queries avoided the latency cost of post-query filtering.

Then the team ran into a problem.

At around 100,000 partitions inside a single Milvus collection, performance fell off a cliff. Partition creation latency went from about 20 milliseconds to roughly 250 milliseconds. Loading a partition started taking more than 30 seconds. Adding hardware would not have fixed any of this, because the issue was not a shortage of capacity. The issue was that too many partitions in one collection overwhelmed the bookkeeping that the database needed to keep them organized.

The fix was hierarchical capping.

Each Milvus cluster now holds 400 collections, and each collection holds at most 1,000 partitions, which limits any single cluster to 400,000 bases. As the customer base grows, Airtable provisions new clusters rather than packing more partitions into existing ones.

The structure trades some operational complexity for predictable performance at every layer. See the diagram below:

Permissions deserve a brief discussion before we move further. Milvus does not know anything about who is allowed to see what data. It just stores embeddings and returns matches. Permission checks happen later, when the application layer takes the row IDs returned by Milvus and fetches the actual rows from Airtable’s primary database. This split keeps the vector search system focused on a single job, which is similarity search, and authorization stays where authorization always has lived.

The pattern of hierarchical capping shows up across distributed systems, from sharded relational databases to message broker topics. Any flat namespace eventually hits a wall, and the fix is almost always to introduce another level of grouping above it. Recognizing this principle is more transferable than memorizing the specific numbers.

See the diagram below that shows the query flow:

Once the data has been sliced up, the next question is how to actually search inside each slice.

Index Selection

Vector search at scale involves an unavoidable tradeoff with three currencies, namely memory, latency, and recall.

Recall means the percentage of truly relevant results that show up in a query response. Every vector index pays for performance with one of these three currencies, and no option gets all three for free.

Airtable benchmarked three index types, and the results map cleanly onto this triangle.

HNSW, which stands for Hierarchical Navigable Small World, builds a graph where similar vectors are connected to each other. A query starts at a small set of entry points near the top of the graph and follows the connections downward, hopping from one vector to its nearest neighbors until it converges on the closest match. HNSW is fast at lookup time, achieves recall in the 99 to 100 percent range, and behaves predictably under load. The cost is that the entire graph has to live in memory, which makes HNSW the most memory-hungry of the three options.

IVF-SQ8 takes a different approach. The IVF part clusters vectors into groups, so a query only has to search inside the most relevant group rather than the full dataset. The SQ8 part compresses each number in the vector from four bytes to one byte, shrinking the index dramatically. The footprint becomes much smaller, but the compression introduces approximation error that lowers recall.

DiskANN keeps most of the index on solid-state storage rather than in memory. It scales to enormous datasets per node because holding everything in RAM is not required. The cost is that every query touches disk, and disk is slower than memory, so query latency rises.

Airtable chose HNSW. Given the priorities from earlier in the design, this was almost the only available answer. A 500-millisecond latency target ruled out DiskANN’s higher per-query cost. The recall directly determines how good Omni’s responses feel to users, which makes the precision of HNSW worth paying for. The memory cost remained a real concern, but Airtable had a separate way to handle it.

The right index does not exist in the abstract. It exists relative to the priorities and constraints of a specific system. If Airtable’s latency tolerance had been looser, DiskANN would have been an interesting candidate. If their recall tolerance had been lower, IVF-SQ8 would have saved them money. None of the three options is universally better than the others.

This same triangular pattern repeats across systems engineering. Caching works the same way, where hit rate trades against memory and consistency. Database indexes work the same way, where read speed trades against write speed and storage. The technologies stop feeling intimidating once the underlying tradeoff becomes recognizable.

Hot and Cold Data

Picking HNSW solved the latency and recall problem, but pushed the entire cost onto memory. Across hundreds of thousands of bases, that memory bill adds up quickly. The team needed a way to shrink it without giving up the index they had just chosen.

The solution came from looking at how customers actually use Airtable. When the team analyzed access patterns, they found that only about 25 percent of bases were read from or written to in any given week. The other 75 percent sat completely idle. This was not an anomaly. It reflected something real about how people work. Users tend to focus intensively on one base for a stretch of time, set it aside for weeks or months, and then come back when the project requires their attention again.

Milvus supports offloading partitions from memory to storage and reloading them within seconds. With that capability, Airtable could keep only the hot partitions in memory and push the cold ones out. When a user opens a base that has not been touched in weeks, the partition reloads quickly enough that the user notices a brief warm-up rather than a failure.

This approach works for Airtable specifically because their access pattern is bursty and bimodal. If usage were spread evenly across all bases, with every customer constantly touching their data at the same low rate, cold offloading would not save much. The hot set would be the entire dataset. Airtable’s pattern is the opposite. A small fraction of bases is active at any moment, and the active set rotates over time.

What made this work was measurement.

The Airtable engineering team did not guess about access patterns and did not reach for a generic optimization. They looked at the data, found a property of their actual usage, and built around it. The HNSW choice became economically viable because of this measurement, and the decisions in this system reinforce each other in a way that would not be obvious from evaluating any one of them in isolation.

Recovery

The traditional approach to disaster recovery in databases is backup and restore. Snapshots get taken regularly, stored somewhere safe, and used to rebuild the system if something catastrophic happens. Airtable went a different direction.

Their recovery path is to spin up a fresh Milvus cluster and re-embed customer data from the source. The most-used bases get re-embedded first so that most users see normal service quickly. The remaining bases get rebuilt lazily as customers access them. There is some compute cost during recovery and some delay before every base is fully back, but the path is conceptually simple and works across many failure modes at once. Corruption, model migrations, and certain data residency changes all reduce to the same procedure.

This option is only available because Airtable has already built an asynchronous embedding pipeline as part of earlier work. That pipeline normally generates new embeddings whenever customer data changes, processing them in the background rather than blocking writes. Recovery is not a separate system created for emergencies. It is just the existing pipeline running against an empty cluster.

Conclusion

The system built by Airtable involves four major tradeoffs: how to partition the data, which index to use, when to keep data in memory, and how to recover from failure.

Every one of those decisions traces back to the same upstream fact about Airtable’s tenants. Their customers run small, isolated bases that are mostly cold most of the time. Changing any one of those properties can cause the design to fall apart.

For example, a workload where every base is hot all the time would make cold offloading useless. A workload requiring strict consistency would not tolerate the asynchronous embedding pipeline. A workload with very small per-customer datasets might benefit more from shared partitions than from one-per-base.

The technologies Airtable uses, including Milvus, HNSW, and the rest, are interchangeable in principle. The same system could be rebuilt on a different infrastructure, and the architectural reasoning would still hold. What is harder to replicate is the discipline of letting the data drive the architecture rather than the other way around.

References:

• Productionizing Semantic Search: How We Built and Scaled Vector Infrastructure at Airtable

• What is Airtable?

On June 10, GitLab Transcend streams live from London with an agenda built for practitioners like you. You can expect an agenda that’s full of keyboard moments with live demos of Duo Agent Platform, agentic AI use cases from your peers, and The Developer Show hosted live by Senior Developer Advocate, Colleen Lake. Register today.

GitLab Transcend streams live from London on June 10 (with regional replays for APAC and AMER on June 11). Register for free today.

Stream the event

In November 2023, Vercel quietly shipped an internal platform that cut its build provisioning time from 90 seconds to 5. That sounds like a story about making things faster. It is, but only on the surface. The real story is that Vercel got faster by accepting a harder constraint, building a more complicated foundation, and then layering three separate optimizations on top of it. The 18x improvement is the result.

Vercel is a deployment platform for web applications. When a developer pushes code to a connected repository, Vercel pulls that code, runs the build process (compiling, bundling assets, packaging the output) on its own servers, and then deploys the result to a global edge network of geographically distributed servers that deliver the site to end users. The build step happens on Vercel’s infrastructure, which means thousands of customers run their build scripts on machines that Vercel manages. Every push has to feel instant to the developer, has to run safely on shared hardware, and has to scale through traffic spikes without degrading.

The platform that handles all of this is internally codenamed Hive, and it has been powering Vercel’s builds since late 2023.

Hive is the reason behind the 90-to-5 transformation. In this article, we examine the constraints Vercel faced, the choices they made in response, and the optimizations that produced the speedup.

Disclaimer: This post is based on publicly shared details from the Vercel Engineering Team. Please comment if you notice any inaccuracies.

The Trust Problem

The architecture rests on a single foundational assumption. Hive operates as if every piece of code it executes might be malicious, running on machines shared by many tenants at once. That assumption influences everything that follows.

It matters because the trust calculation flips entirely between two situations. When a team runs its own code on its own server, the goal is performance and convenience. The code trusts the machine, and the machine trusts itself. When the code comes from someone else and runs on shared hardware, the calculation changes. The platform has to assume the code might try to break out of its sandbox, read another customer’s secrets, or interfere with builds running on the same machine. This is hostile multi-tenancy, and it is a different infrastructure problem from running cooperative workloads.

Vercel sits squarely in this harder category.

Every customer push is, from Vercel’s perspective, code written by someone the team has never met, running on a machine that is also running other customers’ code at the same time. The build script could be a normal Next.js compilation, or it could be a deliberately crafted exploit designed to escape the sandbox. Vercel has to handle both cases identically, since the platform cannot tell the difference in advance.

The obvious answer is to run each build inside a Docker container.

Containers are how modern infrastructure runs isolated workloads, and most engineers reach for them by reflex. The problem is that containers were designed primarily as a packaging tool, with isolation as a useful side effect. Multiple containers on the same machine all share the same Linux kernel, which is the part of the operating system with direct access to the hardware. Anything that breaks through the kernel can reach other parts of the machine.

For most workloads, this risk is acceptable, since most workloads are cooperative. A team’s own microservices have no incentive to attack each other. However, for running strangers’ build scripts at scale, the risk profile is different. A single kernel exploit in one customer’s build could reach every other customer’s build on the same machine, and the blast radius would be enormous.

This is why standard container orchestration was a poor fit.

Tools like Kubernetes assume cooperative tenants and provide good isolation by default, but not adversarial isolation. Adding hardening on top of Kubernetes was an option, but for a constraint as foundational as tenant isolation, building from primitives gave Vercel more leverage. Containers leave a gap that Vercel could not afford to leave open. The question was how to close that gap without giving up the speed that containers provide.

See the diagram below that offers some insight:

MicroVMs and Firecracker

The traditional alternative to containers is the virtual machine.

A virtual machine runs a complete operating system on top of a virtualization layer, which means two VMs on the same physical machine each have their own kernel. A kernel exploit in one VM cannot reach the other, since the kernels are genuinely separate. The downside is weight. A traditional VM might take 30 to 60 seconds to boot and consume hundreds of megabytes of memory just to exist. For a workload like web hosting, where a single VM runs for months, that overhead is fine. For a workload like running a 2-minute build and then throwing the environment away, it becomes wasteful.

Around 2018, AWS released Firecracker, an open-source virtualization tool that strips a VM down to the minimum needed to run one short-lived workload.

Firecracker microVMs boot in around 125 milliseconds and use only a few megabytes of memory each. They provide VM-level isolation, with separate kernels and a hardware-enforced boundary that the CPU itself maintains, at something close to container-level speed. This is a new shape in the isolation tradeoff space, occupying a corner that did not exist before.

AWS originally built Firecracker to power Lambda, where it now runs at production scale across millions of concurrent functions. That track record gave Vercel a battle-tested foundation rather than an experimental one.

Vercel adopted Firecracker as the core of Hive. Each customer build runs in a microVM that Vercel calls a cell, and the relationship between cells and Firecracker processes is strictly one-to-one. Each Firecracker process manages exactly one cell, and each cell handles exactly one build. Inside the cell sits a container that runs the actual build script. The container handles packaging, since it carries all the build tools and dependencies the customer’s project needs. The microVM handles isolation, since it provides the kernel-level boundary that containers alone cannot. Each layer does what it is good at.

This setup is the architectural answer to the trust problem.

Vercel can now run a strange piece of code with confidence that, even if the code attempts something hostile, it cannot reach beyond the cell it is running in. The microVM is the wall, and the wall is enforced by the CPU’s virtualization features rather than by software alone. Firecracker provides the isolation primitive, while the rest of Hive is the machinery that turns one isolated cell into a system capable of running thousands of builds across the world.

Inside Hive

Hive has a small vocabulary, since the names map directly to physical and logical pieces of the system.

• A Hive is a regional cluster, and multiple Hives can exist in a single region.

• A Box is a physical machine inside a Hive.

• A Cell is a microVM running on a Box.

• The Control Plane is the brain of the cluster, and the API is the entry point that the rest of Vercel’s systems talk to.

Here is a different view of the same diagram:

When a customer pushes code, Vercel’s build pipeline (which is a separate system from Hive) decides which Hive to use based on the customer and the build configuration, and then calls that Hive’s API to request a cell. The Control Plane finds an available cell on one of the Boxes, hands it to the build pipeline, and the build runs inside the cell’s container. Once the build finishes, the cell is destroyed, and the resources return to the pool.

Running multiple Hives per region is a deliberate failure isolation choice. If one Hive has a bad day, the others in the same region keep running. This is finer-grained reliability than running in multiple cloud regions alone, and it means a single bad deploy or infrastructure incident will not take out an entire customer base.

Two background programs handle the orchestration inside each Box.

• A box daemon runs on the physical machine and handles provisioning, spawning new Firecracker processes, and managing the lifecycle of cells.

• A cell daemon runs inside each microVM and manages the build container that does the actual work.

The two daemons communicate over a socket connection, which is how the orchestration layer scales without becoming a bottleneck. The pattern matters more than the implementation. Responsibility is split between the host machine and the virtual machine, and each side has a clear job.

See the diagram below:

Each cell receives dedicated CPU and memory, which means those resources are partitioned cleanly between cells on the same Box. Disk and network throughput, however, are rate-limited based on the Box’s overall capacity rather than dedicated.

This reflects where multi-tenant isolation hits its practical limits. Some resources are easy to slice up, while others must be shared with quotas because slicing them cleanly would waste too much capacity.

The other architectural choice is that these cells are ephemeral. Once a build completes, the cell is destroyed rather than reused, even though reusing it would be faster. This is a security choice rather than a performance one. A reused cell would create a path for one customer’s leftover state, whether that means files on disk, processes still running, or memory contents, to leak into another customer’s environment. Destroying the cell after every build closes that path entirely.

The 90-to-5 Breakdown

All of this architecture would still be slow if every build had to spin up a fresh cell from scratch. Even with Firecracker, the cold path takes about 5 seconds per cell, since the system has to boot the microVM, mount the disk, load the build container image, and start the container.

The 18x improvement comes from three places.

The first layer is faster boots. Inside each Box, Vercel optimized the cell startup path itself. The build container image is large, so pulling it fresh on every cold start used to add significant time. Vercel now caches the build container image so it loads from a local copy rather than from a remote registry, and that change alone shaved around 45 seconds off VM startup times compared to their previous solution. They also use block device snapshotting, where the disk image of a freshly prepared cell is saved at a known-good moment, and new cells start from that saved copy instead of building up from scratch. These optimizations make the cold path itself dramatically faster.

The second layer is the warm pool, and this is where most of the speedup happens. Vercel keeps a pool of cells already booted, with the build container image loaded, sitting idle and waiting. When a build comes in, it uses a warm cell and starts running immediately. The 5-second provisioning time only applies when the warm pool is empty, which happens during traffic spikes or for specialized builds like Secure Compute (an enterprise feature with stricter isolation requirements). For the common case, the wait is essentially zero. The warm pool means that most builds skip cold-start provisioning entirely.

The third layer is Firecracker’s baseline speed. None of the other features would matter if the underlying virtualization were heavy. Traditional VMs take tens of seconds to boot, which makes warm pools impractical at scale, since the pool would have to be enormous to keep up with demand. Firecracker boots in milliseconds, which is what makes warm pools and snapshotting work in the first place. The whole speed story rests on the foundation Vercel chose at the very beginning.

Across all builds, Vercel saw a 30% improvement in build performance after switching to Hive. Builds that hit the cold path, where a fresh cell has to be spawned, saw build times drop by about 40%. The provisioning portion of those cold-path builds dropped from 90 seconds to 5. None of these numbers came from a single breakthrough. They came from compounding wins on a foundation that allowed them.

Costs and Tradeoffs

Warm pools come with a real trade-off in terms of cost.

Keeping cells pre-warmed means Vercel pays for compute that does no useful work most of the time. The amount is substantial, since the right pool size is a constant balance between waste, where too many idle cells burn money, and tail latency, where too few warm cells leave customers waiting during traffic spikes. For a service with bursty traffic, this is an active operations problem rather than a one-time tuning.

Building Hive at all was the higher cost. Vercel could have used Kubernetes or ECS and added isolation on top, and they would have shipped something serviceable in a fraction of the time. Building from primitives required enormous engineering investment and an ongoing maintenance burden that off-the-shelf tools would have absorbed for them. The reason it was worth doing is that owning the substrate gave Vercel leverage that an opinionated platform would have caused difficulty. The team can make decisions like destroying every cell after every build, or tuning the warm pool based on customer build patterns, without working around someone else’s design.

The payoff for that leverage is visible in product features. The same architecture that drove the 90-to-5 number also enabled Vercel to ship enhanced build machines for customers who need extra memory or disk, and to support Secure Compute for enterprise customers with stricter isolation requirements. These features would have been much harder to build on top of a third-party platform, since they would have required either accepting the platform’s constraints or fighting them at every step. Building the foundation paid off in product capabilities, not just performance numbers.

This architecture makes sense for Vercel because they have hostile multi-tenancy at scale, and because builds are core to their business rather than incidental to it. For a team running its own builds for its own code on its own machines, microVMs would be wildly over-engineered, and containers are perfectly appropriate. The lesson is the connection between threat model and architecture, rather than a generic recommendation to always use microVMs.

Conclusion

Vercel’s story illustrates a pattern that repeats across the industry. Threat model drives architecture. Cooperative tenants can run on containers and standard orchestration. Adversarial tenants require microVMs, isolated runtimes, or sandboxed execution environments. The shape follows from the assumption.

The broader takeaway from Hive is that Vercel got faster by accepting a harder problem rather than working around it.

Containers were inadequate for the trust model, microVMs were the right shape, and the speed came from stacking three separate optimizations on top of a deliberately harder foundation. Starting with the constraint and then optimizing for the speed proved to be a stronger bet than starting with the simple thing and trying to make it run quicker.

References:

• A deep dive into Vercel’s build infrastructure

• What is Firecracker

Most AI agents don’t fail because of the model. They fail because the context is broken—stale data, fragmented systems, slow retrieval.

Join Simba Khadder, Head of AI Product & Director of Software Engineering at Redis, on June 10 to see how to turn scattered enterprise data into live, agent-ready context with Redis Iris.

You’ll learn:

• The four failure modes of how context breaks in production

• How to make your enterprise data navigable for runtime

• How Redis Context Retriever, Search, Data Integration, and Agent Memory work together

Reserve your spot →

The CockroachDB team wanted to add vector search to their distributed database, and dozens of well-known algorithms already existed.

To facilitate the decision-making process, they wrote down a list of architectural requirements, including a refusal to depend on any central coordinator, a refusal to allocate large in-memory caches, a need for real-time updates, an intolerance for hot spots, and a requirement to support sharding. Then they checked the list against the popular options.

Most failed at least one requirement, and some failed several. The team’s response was to build something new, called C-SPANN, that satisfied every constraint by treating the index as ordinary table data inside CockroachDB rather than as a separate system.

In this article, we will look at how the CockroachDB engineering team built this index and the challenges they faced.

Disclaimer: This post is based on publicly shared details from the CockroachDB Engineering Team. Please comment if you notice any inaccuracies.

Vectors and Approximate Nearest Neighbor Search

A vector is a long list of numbers that captures the meaning of something.

Modern neural networks like the ones behind ChatGPT can take an image, a document, or a snippet of audio and convert it into a vector of floating-point numbers, typically a few hundred to a few thousand dimensions long.

The useful property of these vectors, often called embeddings, is that similar things produce similar vectors. For example, two photos of beaches end up close to each other in this multi-dimensional space, and a photo of a beach and the word “beach” end up in roughly the same neighborhood, which is what makes semantic search possible.

The trick is finding those neighbors quickly when you have billions of vectors to search through.

Traditional database indexes work because numbers and strings have a natural ordering. We can sort them, store them in a B-tree, and walk that tree to find what you want.

Vectors do not have that property. Should beach photos come before or after food photos? What about photos of food at the beach? There is no answer, because the data has no inherent sequence, which means a B-tree cannot help you.

The brute-force alternative is to compare your query vector against every stored vector and return the closest matches. This works fine for a few thousand vectors, but falls apart somewhere in the tens of thousands, and becomes hopeless once you reach the millions.

Vector indexes solve this by giving up on exact answers. They find approximate nearest neighbors, accepting a small loss of accuracy in exchange for orders of magnitude better performance. The results are usually close enough that real users cannot tell the difference, and the search runs fast enough to feel instant. That tradeoff between accuracy and speed is the foundation of every vector index, and the interesting engineering question is how you make the rest of the system work around it.

Even with a good algorithm for finding nearest neighbors, plugging it into a distributed transactional database is its own problem. That is where the CockroachDB story actually begins.

Architectural Constraints in a Distributed SQL Database

CockroachDB is a distributed SQL database. This means that the data lives across multiple machines, often across regions, and the system is designed to scale linearly. It guarantees transactional consistency and supports real-time updates, and all of this has to keep working when machines die, disks fail, or networks partition.

These properties impose a set of architectural constraints on any new feature, and a vector index is no exception. The CockroachDB team wrote down six requirements that any candidate algorithm had to satisfy.

• The first requirement is that no single node can act as a central coordinator. Any node in the cluster should be able to serve reads and writes, because relying on a single leader to direct traffic creates a bottleneck and a single point of failure.

• The second requirement is that the index cannot rely on large in-memory structures. Index state has to live in persistent storage, since the team could not assume every node has gigabytes of RAM available for caching vectors. They also wanted to avoid the long warm-up times that come with rebuilding in-memory caches after a restart, which matters especially for serverless deployments where nodes spin up and down on demand.

• The third requirement is that network hops have to stay minimal. Round-trips between nodes are expensive, and any algorithm that requires sequential traversal across the cluster will accumulate latency unpredictably.

• The fourth requirement is that the index data layout has to be sharding-compatible. Index data has to map naturally to CockroachDB’s key-value storage so that it can be split, merged, and rebalanced like any other table.

• The fifth requirement is that the index must avoid creating hot spots. As inserts and queries scale up, the load has to spread across the cluster, because concentrating traffic on a single node defeats the point of running a distributed system in the first place.

• The sixth requirement is that the index has to support incremental updates. Inserts and deletes need to be applied in real time without blocking queries, requiring batch rebuilds, or degrading search quality over time.

This list rules out the most popular vector indexes.

HNSW, the graph-based algorithm that powers pgvector, Weaviate, and many other systems, is excellent on accuracy benchmarks but builds its graph in memory and resists sharding. Classic IVF is closer in spirit but assumes a single-node deployment and struggles with dynamic updates. Specialized vector databases like Pinecone solve these problems by being separate systems entirely, which works fine if you are willing to keep your vectors in one database and your transactional data in another.

CockroachDB needed something that handled both inside the same system, with the same guarantees.

Faced with this list, the team built something new. They called it C-SPANN, and the design choices that make it work are mostly about what it does not try to do.

The C-SPANN Architecture

C-SPANN borrows ideas from three places.

Microsoft’s SPANN paper contributed the tree structure for partitioning vectors, the follow-up SPFresh paper contributed techniques for incremental updates, and Google’s ScaNN project contributed ideas around quantization.

The CockroachDB team combined these with their distributed SQL architecture to produce something none of the source papers describe directly.

At the core is a hierarchical K-means tree. Vectors are grouped into partitions based on similarity, where each partition typically contains dozens to hundreds of vectors and has a centroid that represents the average of the vectors it contains. Think of the centroid as the partition’s center of mass. Those centroids are themselves grouped into higher-level partitions with their own centroids, and that process repeats until you reach a single root partition at the top.

The result is a wide, shallow tree. With a fanout of around 100, an index of one million vectors needs only three levels, and an index of ten billion vectors needs only five. Searching the tree means starting at the root, comparing the query vector to the centroids at that level, descending into the closest partition, and repeating until you reach the leaves. At each level, partitions can be processed in parallel, which keeps latency low and predictable. At the leaves, the system scans a few hundred candidate vectors using SIMD CPU instructions for speed.

That much describes the algorithm. The interesting part is what happens to the data structure once it is built.

Each partition is stored as a self-contained set of key-value rows inside CockroachDB. Partition data lives in CockroachDB ranges, which are the same units of storage that hold every other table in the database. Therefore, the index is not a parallel structure attached to the database. It is table data with extra meaning.

This decision pays dividends. CockroachDB already knows how to split a range when it grows too large, how to rebalance ranges across nodes when load shifts, and how to cache frequently accessed rows in its block cache.

All of this setup applies to vector index data automatically, without writing a single line of new infrastructure code. When a new node joins the cluster, ranges containing index partitions get distributed to it the same way ranges containing user tables do. When a node restarts, the index is immediately ready to serve queries because it lives on disk, rather than in some warm-up cache that has to be rebuilt.

However, building the index is one thing. Keeping it healthy as data flows in and out is harder, especially when the index needs to be compressed aggressively to stay affordable.

Index Maintenance, Quantization, and Multi-Tenant Partitioning

A K-means tree is not static. Partitions grow as vectors are inserted and shrink as vectors are deleted, so the system needs background machinery to keep partitions at a reasonable size and to keep vectors grouped with their nearest centroids.

When a partition grows too large, C-SPANN splits it. A balanced variant of the K-means algorithm divides the vectors into two roughly equal groups, each with its own new centroid, and the tree is updated so that future inserts route to whichever new partition is closer. When a partition shrinks too small, the system merges it away and reassigns its vectors to neighboring partitions. Both operations happen in the background to avoid interfering with foreground transactions.

There is one point worth noting.

After a split, some vectors in the original partition might actually be closer to a neighboring partition’s centroid than to either of the two new centroids, and they get reassigned. Likewise, a vector in a neighboring partition might now be closer to one of the new centroids and migrate in. This idea, called nearest partition assignment, comes from the SPFresh paper and is what keeps the index accurate over time.

The consequence is that you can start with an empty table, insert millions of vectors, and end up with an accurate, well-balanced index without ever rebuilding it. The maintenance setup handles everything incrementally.

The other operational factor is size. An OpenAI embedding has 1,536 dimensions stored as 2-byte floats, which works out to about 3 KB per vector. A billion vectors at full precision is 3 TB just for the embeddings, before any indexing overhead is counted. Storing and scanning that much data is expensive both in disk space and in the CPU and memory used during search.

C-SPANN compresses vectors using a technique called RaBitQ, which reduces each dimension to a single bit. The compressed representation is roughly 200 bytes per vector, a 94 percent size reduction. The math behind the compression involves a random orthogonal transform that preserves distances while spreading data evenly across dimensions

What matters for the system is that quantization is lossy, so distance estimates from compressed vectors are only approximate. C-SPANN compensates with a reranking step, where the system scans quantized vectors to assemble a candidate set, then fetches the original full-precision vectors for those candidates to compute exact distances. By fetching candidates, the system can absorb quantization error and still return accurate results. The pattern of cheap approximate filtering followed by precise refinement on a small candidate set shows up in many other systems too, and recognizing it here makes it easier to spot elsewhere.

The third operational reality is multi-tenancy. In most real applications, vectors belong to someone, whether a user, a customer, or an organization, and most queries are scoped to a single owner. Mixing one user’s vectors with another’s during search is wasteful, and it is also a security problem.

CockroachDB handles this through prefix columns on the vector index. Here is what the schema looks like.

CREATE TABLE photos (
  id UUID PRIMARY KEY,
  user_id UUID,
  embedding VECTOR(1536),
  VECTOR INDEX (user_id, embedding)
);

A query for one user’s nearest matches uses pgvector-compatible syntax.

SELECT id
FROM photos
WHERE user_id = $1
ORDER BY embedding <-> $2
LIMIT 10;

Behind the scenes, the index maintains a separate K-means tree for each distinct user. Performance scales with how many vectors the user owns, rather than with the total size of the index, so a billion vectors split across a million users behaves, from any one user’s perspective, like a million-vector index.

Combined with CockroachDB’s REGIONAL BY ROW tables, prefix columns can also partition the index by geography. For example, a user in Europe gets their data and their index entries stored in a European region, with fast local access and compliance with data domiciling requirements, while the same table serves a US user with equally low latency from US infrastructure. The combination of region, ownership, and embedding as prefix columns produces an index that is efficient, secure, and locality-aware by default.

Conclusion

C-SPANN refused several compromises that most vector databases quietly accept.

Freshness in CockroachDB is real-time and transactional rather than batched or eventually consistent, which means a vector inserted in a transaction becomes searchable as soon as that transaction commits, with the same consistency guarantees as any other write. Scaling is native to the distributed architecture rather than a feature retrofitted onto a single-node system, and vectors live alongside transactional data in the same database, inside the same queries, under the same operational umbrella. Since the index lives on disk, nodes serve queries immediately after a restart without any warm-up phase.

In return, the team accepted some real limitations. The 25.2 release is a preview, and several optimizations are still being built, including fuller SIMD usage, root partition caching, and complete merge support. The current implementation supports only Euclidean distance, with cosine and inner product on the roadmap. Filtering on non-prefix columns is limited today, though that scope is expanding. Also, on raw vector search benchmarks against specialized in-memory systems, C-SPANN does not win on pure latency.

The tradeoff suggests where this design fits and where it does not. CockroachDB’s vector index is a strong choice for applications where vectors and transactional data need to coexist, where multi-tenant isolation matters, and where multi-region deployment with data domiciling is a requirement. Specialized vector databases remain a better fit for pure vector workloads with no transactional component, for read-heavy batch-updated datasets where freshness is not a concern, and for cases where every microsecond of search latency is critical.

The architectural logic underneath all of this is worth keeping in mind. The CockroachDB team treated the vector index as ordinary table data and inherited their existing distributed machinery for free, so splits, caching, sharding, replication, and multi-region behavior all worked from day one because they already worked for everything else in the database. The algorithm is the part that gets the headlines, but the integration is what makes the system possible.

References:

• Real-Time Indexing for Billions of Vectors: How we built fast, fresh vector indexing at scale in CockroachDB

• Vector databases

QA Wolf’s AI agent maps and tests your app’s most complex user flows. It turns your prompts into real Playwright and Appium code that runs 12x faster and more reliably than other computer-use agents.

What sets our AI apart:

• Maps 200+ test cases in minutes instead of weeks of manual planning.

• Executes tests 12x faster than computer-use agents.

• Runs entire suites 100% parallel with consistent results.

• Produces open-source tests your team owns, with zero vendor lock-in.

Get started today

This week’s system design refresher:

• RAGs vs Agents

• Build with Claude Code: New Cohort Launch

• Forward Proxy, Reverse Proxy, and API Gateway Explained

• How does a request actually travel through Claude Code?

• How does Claude Code keep long sessions from running out of context?

RAGs vs Agents

Ask an LLM about your company's data and it will guess. The two patterns that fix this are RAG and agents, and they solve different problems.

RAGs: RAGs combine LLMs with retrieval to ground answers in 4 steps.

• Step 1: The user query is embedded and sent to a retrieval step.

• Step 2: Retrieval pulls the most relevant chunks from a knowledge base (PDFs, wikis, etc.)

• Step 3: Those chunks are pasted into the prompt as context.

• Step 4: The LLM writes the answer, grounded in the retrieved text.

One retrieval. One generation. Cheap, predictable, and easy to debug.

Agents: Agents wrap LLMs in a reasoning loop with tools to take action.

• Step 1: The user query goes into the agent runtime. A reasoning loop wrapped around an LLM.

• Step 2: The LLM reads the goal and picks a tool (Read, Write, Edit, Bash, etc.)

• Step 3: The runtime executes the tool and feeds the result back to the LLM.

• Step 4: The LLM reasons again, picks the next tool, and loops until the task is done.

More flexible. More tokens. Harder to debug because errors drift across steps.

The rule of thumb: Use RAG when the answer lives in your documents. Use an agent when the answer requires action on other systems.

Over to you: When do you prefer RAG over agent?

Build with Claude Code: New Cohort Launch

We’re launching a new 2 day intensive, cohort based course called Build with Claude Code, taught by John Kim, who has trained hundreds of engineers at Meta to use Claude Code in real production workflows.

The course starts soon on May 28.

Check it out now

A few things you’ll learn:

• The agentic loop, context engineering, and memory layers that make Claude Code useful for real projects

• How to build with Claude Code Skills, MCPs, and hooks to give Claude the tools and feedback loops it needs to self correct

• Parallel development with Git worktrees, subagents, and agent teams

• A capstone project where you ship something real on your own stack

The course includes live sessions, assignments, and office hours, so there’s plenty of room to ask questions and get unstuck.

The first cohort starts in just a few days: May 28 to 29, 2026. If you want to learn everything from the fundamentals of Claude Code to advanced production workflows, including working with large codebases, this could be a great way to level up.

Check it out now

Forward Proxy, Reverse Proxy, and API Gateway Explained

People mix these up all the time, since they all sit between a client and a server. The real difference is which side they represent and what problem they solve.

A forward proxy sits next to the client. Your laptop sends a request, the proxy forwards it out, and the destination never sees your real IP. Corporate networks use this to enforce policy, block sites, and cache traffic.

A reverse proxy sits next to the server. The client has no idea how many machines are behind it. The proxy decides who handles the request, terminates TLS, and keeps your backend off the public internet. NGINX and HAProxy are commonly used here, typically paired with a load balancer in front.

An API gateway is a reverse proxy that does more than route traffic. It also handles auth, rate limits, API keys, versioning, and request shaping. Without it, each microservice has to implement its own version of validation, throttling logic, and request logging.

A forward proxy represents the client, a reverse proxy represents the server, and an API gateway is what you add when ten services need the same authentication and rate limiting rules applied consistently.

In most real systems, all three are running at different layers. The forward proxy filters outbound traffic, the reverse proxy fronts the application servers, and the API gateway sits in front of your APIs to enforce policies before requests reach them.

Over to you: What's your proxy + gateway combo? Always interesting to see what teams pair together.

How does a request actually travel through Claude Code?

Most of us type a prompt and watch the magic happen. The diagram below shows what's really going on behind the curtain, based on the Claude Code source code.

Let's trace one real request: "Fix the failing test in auth.test.ts."

• Step 1: The user sends a prompt to Claude Code through their interface.

• Step 2: The interface (CLI, IDE, or SDK) wraps the prompt with repo and file context and hands it to the agent loop as a request.

• Step 3: The agent loop plans the next move and proposes an action: Edit(auth.ts, lines 42–58).

• Step 4: The permission system checks the proposed action against the rules.

• Step 5: The approved action becomes a tool call: Edit(auth.ts, patch), dispatched to the matching tool.

• Step 6: The tool runs in the execution environment (shell, cloud, or sandbox) as a real syscall.

• Step 7: The execution returns a tool result back to the agent loop.

• Step 8: The agent persists the turn to state and streams the final message to the user.

The whole system is just this loop, repeated until the model stops asking for tools.

Over to you: which step in this loop do you think is the hardest one to get right when building your own coding agent?

How does Claude Code keep long sessions from running out of context?

It uses 5 strategies, run in sequence before every model call. Each one only runs if the previous doesn’t free enough room.

1. Budget Reduction: caps individual tool results. Oversized outputs are swapped for a content reference.

2. Snip: trims the oldest history segments and emits a boundary marker.

3. Microcompact: prunes tool turns by tool_use_id so the prompt cache stays warm.

4. Context Collapse: a read-time projection over the full history.

5. Auto-compact: the last resort. It calls the model to produce a full summary of prior turns.

The pattern is lazy degradation: apply the least disruptive shaper first, escalate only when cheaper layers prove insufficient.

Over to you: how often do you run out of context?

The course starts soon on May 28.

Check it out now

A few things you’ll learn:

• The agentic loop, context engineering, and memory layers that make Claude Code useful for real projects

• How to build with Claude Code Skills, MCPs, and hooks to give Claude the tools and feedback loops it needs to self correct

• Parallel development with Git worktrees, subagents, and agent teams

• A capstone project where you ship something real on your own stack

The course includes live sessions, assignments, and office hours, so there’s plenty of room to ask questions and get unstuck.

The first cohort starts in just a few days: May 28 to 29, 2026. If you want to learn everything from the fundamentals of Claude Code to advanced production workflows, including working with large codebases, this could be a great way to level up.

Check it out now