A program adding up a day’s sales needs to know whether all of today’s sales have actually arrived. For data stored in a file, the answer is trivial because the file has an end. However, for data that arrives continuously and never stops, there is no clean answer, and how a system resolves that gap is the difference between batch processing and streaming.

Batch processing waits for completeness. It collects data up to a natural boundary, a closing time, or a finished file, and then computes over the whole set at once. Streaming prioritizes completeness for speed. It produces answers continuously from data that is still arriving, which means it has to estimate when enough data has come in and handle the cases where that estimate is wrong. This trade-off between completeness and latency is the key consideration when dealing with streaming and batch.

In this article, we will cover the strategies on each side and what each one costs.

• On the batch side, that means full and incremental loads and large-window aggregation, with micro-batch sitting in between.

• On the streaming side, the territory runs through tumbling, sliding, and session windows, watermarks and late data, the lambda and kappa architectures, and the often-misunderstood meaning of exactly-once processing.

Boundaries

Read more

PR-AF is an open-source code review agent that ranks #2 of 42 on Martian's Code-Review-Bench, ahead of CodeRabbit, Copilot, and Devin, running a single open model. The trick is the harness: it plans a review strategy per PR, spawns parallel reviewer agents, verifies every finding against your source, and drops anything it cannot prove. That makes it about 10x cheaper per review than closed-source tools. One API call, and a drop-in GitHub Action. If open models writing frontier-level reviews sounds useful, star the repo.

Fork and deploy!

A chatbot answers a question, and an agent completes a task. This seems like a huge difference. However, the gap between the two is narrower than it appears.

An agent is an LLM placed inside a loop where the model itself decides when the loop should stop. Everything interesting about agents follows from this shift. The autonomy that makes them useful, the cost that makes them expensive, and the design challenges that come with building one all trace back to it.

It helps to see where that shift sits in the broader picture.

Software built around language models has moved through a recognizable progression. It started with a single LLM call that took an input and produced an output. Then, the function calling arrived, which let the model reach out to a tool when it needed information or wanted to take an action. After that, developers began chaining calls together in code, with each step’s output feeding into the next, to handle problems too large for a single call. The most recent of these is the agent, where the developer hands control of the loop itself to the model and lets it iterate until it decides the work is done.

In this article, we will walk through that progression. We will also look at how an agent is structured, what choices the model makes on every turn, what scaffolding holds it together, and when an agent is actually the right pattern to reach for.

Disclaimer: This post is based on publicly shared details from various sources. Please comment if you notice any inaccuracies.

Foundations

Before the loop can be interesting, the unit that sits inside it has to be clear.

A bare large language model call is a stateless function. Text goes in, text comes out, and each call typically stands alone in the model’s view. If we want it to fetch today’s weather or update a record in a database, the bare model lacks the means to do either. It can describe weather in general terms and talk about how a database record would change, but reaching the actual forecast or touching the actual database requires something more.

What makes the model useful in real systems is augmentation.

In other words, we give it the ability to call functions we have defined, often called tool use or function calling. We give it access to a retrieval layer that can pull in relevant documents at runtime. We also give it a way to write down information that it should carry between calls, which acts as memory.

Anthropic calls the combination of these capabilities the augmented LLM, and treats it as the foundational unit of every agentic system. The augmented LLM is the building block from which everything else gets composed.

This is the unit most developers have already worked with, even if they have called it something else. Examples include a chat assistant that runs Python code, a function-calling API wired to your service, and a RAG application that pulls from your documents. These systems are useful, and they are also still one call. The model produces an output, the system returns it, and the interaction ends until the next request arrives.

Workflows

What happens when a problem reaches beyond what a single call can handle?

The natural response is to string calls together in a sequence we design, a pattern called prompt chaining. Each step in the chain is its own LLM call, and the output of one step becomes the input to the next. We might use one call to draft an outline, a second call to expand the outline into paragraphs, and a third call to translate the paragraphs into another language. The chain is fixed in advance, and the developer writes which steps run, in what order, and what each step’s prompt looks like.

This is the family of patterns Anthropic groups together as workflows.

A workflow is a system where the LLM and its tools are orchestrated through a code path that the developer designed. Beyond prompt chaining, there are other workflow patterns:

• Routing classifies the input and sends it to a specialized handler.

• Parallelization runs subtasks at the same time.

• Orchestrator-worker has a manager LLM that delegates pieces to specialist LLMs.

• Evaluator-optimizer has one call to generate while another critiques, with the cycle iterating until quality is good enough.

The details differ, but every workflow shares the same property. The number of steps and the path through them are decided by the developer at design time, before the model sees the input. Most production systems built on LLMs today are workflows. They are predictable, debuggable, and usually cheaper than full-blown agents.

The Loop

An agent is what we get when we wrap an LLM in a loop and let the model decide when the loop should exit. The loop itself is plain code. The runtime calls the LLM, reads the output, dispatches whatever action that output specified, feeds the result back into the model’s view, and calls the model again. This continues until the model produces a response that signals a final answer.

There are four steps inside each iteration, and we can call them perceive, reason, act, and observe:

• Perceive is the moment the loop hands the model the current state, which includes the original task, the history of what has happened so far, and any new input.

• Reason is the model’s turn, where the model produces a response that says what to do next, whether that means asking a question, calling a tool, or wrapping up with a final answer.

• Act is when the runtime carries out whatever the model asked for.

• Observe is when the result of that action gets captured and folded back into the state, so the model sees it on the next perceive step.

The most important detail is who decides when the loop should stop. In a workflow, the developer decides at design time how many steps run. In an agent, the model decides at runtime. The model exits the loop by producing an output that the runtime interprets as a final answer. The developer typically sets a hard ceiling on the number of iterations, often called a max-turns parameter, but that ceiling is mainly a safety net. The primary stop signal comes from the model.

Observation matters as a first-class step for the same reason.

The model needs to see the result of its actions before deciding the next move. Removing observation would collapse the loop into a chain, with the model running on prior expectations rather than fresh results from the world. The closed loop, where every action is followed by an observation, is what lets the model adjust as it goes.

With the loop established, the next question is what actually happens on each turn through it.

Decisions

On every turn through the loop, the model’s output picks one of four branches, and the runtime acts on that pick. This branching makes the loop feel intelligent because the LLM is choosing what kind of move to make. Here are the four branches:

• Final answer: The model produces what amounts to a complete reply to the original task. The runtime interprets this as the loop’s exit signal, returns the result, and stops.

• Tool call: The model produces an instruction asking the runtime to invoke a specific function with specific arguments. The runtime executes the tool, captures whatever it returned, appends that result to the conversation state, and sends control back to the model so the loop continues.

• Handoff: The model decides that the current task belongs in the hands of a different agent, often a specialist with its own prompt and tool set. The runtime swaps which agent is running and feeds the same state into the new agent, with the loop continuing under the new identity.

• Continued thought: The model produces a reasoning turn that consists of thought alone. The runtime captures the thought, feeds it back into the state, and runs the model again. This branch shows up most often in ReAct-style implementation, which we are about to see.

OpenAI’s Agents SDK documents the first three branches as first-class behaviors of its loop. The fourth is more a property of how a particular prompt is written than a separate code path

ReAct

ReAct is the prompting pattern that stands for Reasoning plus Acting. The pattern asks the model to interleave reasoning steps with action steps inside the same response, so the model can think about what to do, do it, see what happened, and adjust.

A ReAct trace reads like a structured journal. The model produces a thought, then an action, then receives an observation from the runtime, then produces another thought, and so on, until it reaches a final answer.

Imagine an agent handling customer support:

• A user asks for the status of their most recent order.

• The model produces a thought first, reasoning that it needs to find that order and that the order service is the right place to look.

• It then takes an action by calling the get_recent_order function with the user’s ID.

• The runtime executes the call and feeds back an observation telling the model that the order is number 9152, placed on May 14.

• The model produces another thought, working out that it now needs the shipping status for that specific order.

• It calls the get_shipping_status function with the order ID. The observation comes back, saying the order is in transit and expected to arrive on May 29.

• The model has what it needs at this point and produces a final answer summarizing the status for the user.

See the diagram below:

Two things to consider from this flow are as follows:

• First, every action is grounded in an observation, which is the closed-loop point from the previous section made concrete.

• Second, the reasoning steps are doing real work, since they are how the model decides which action makes sense next, given what it has learned so far.

ReAct is one way to fill the loop, and it is by far the most common pattern in the agent frameworks we work with today.

Guardrails

Guardrails belong at the points where the loop crosses into the outside world. They are part of the architecture itself, designed in alongside the loop.

For reference, OpenAI’s Agents SDK documents three families of guardrails, all defined by where in the loop they run.

• Input guardrails run on the first turn, before the agent’s main model sees anything. They are the place to catch things like prompt injection attempts, requests that violate policy, or inputs that fall outside the agent’s scope. A common pattern is to use a small, fast model as the guardrail, so the more expensive main model only runs when the input passes.

• Tool guardrails wrap every function-tool invocation. A tool input guardrail runs before the tool executes and can block the call or replace it with a message back to the model. A tool output guardrail runs after the tool executes and can rewrite or block the result before it goes back into the conversation state. Tool guardrails matter because tools are how the loop touches systems we care about, and that touch needs supervision.

• Output guardrails run on the final response, after the loop has decided to terminate, but before the user sees the result. They are the last layer of policy enforcement and the place to catch things like leaked sensitive data or claims the agent should avoid making on the company’s behalf.

The structural point is that guardrails sit at every interface where the loop meets the world.

Tradeoffs

Handing control of the loop to a model is powerful, and it comes with three costs that every developer should understand before using the pattern:

• Compounding error: Per-step reliability does poorly when steps are chained together, and the math is not favorable. If the model succeeds on each step of a loop 95 percent of the time, the joint probability of every step going right across ten steps is roughly 60 percent. Stretch that to twenty steps, and the joint success rate falls to about 36 percent. The autonomous nature of agents brings higher costs and potential for compounding errors. The reason coding agents work better than open-ended task agents is that test feedback raises per-step reliability, which shortens the effective length of the chain that has to succeed.

• Scaffolding around the loop: The harness around the loop matters as much as the model inside it. For example, Anthropic shared an internal experiment where even a frontier model running in the Claude Agent SDK fell short on building a production-quality web app from a high-level prompt. The fix was scaffolding rather than a better model. They built an initializer agent that lays out a feature list before any feature work starts, a coding agent that picks one feature per session, a progress file that travels between sessions, and a git history that the agents can use to recover.

• The wrong tool for the job: An agent is often the wrong choice. It is important to find the simplest solution and only add complexity when needed, which sometimes means avoiding agentic systems entirely. Workflows offer predictability and consistency. Agents offer flexibility, and they pay for that flexibility in latency, money, and a more unpredictable failure surface. Many problems are served better by a chain.

Conclusion

The agent loop sits at the end of a recognizable progression.

We started with a single LLM call that takes text in and returns text. We added tools, retrieval, and memory to get the augmented LLM. We strung calls together into workflows when one call was outgrown. The agent is the next rung, the point where the developer hands the control flow itself to the model.

Inside the loop, four steps repeat. The model perceives the current state, reasons about it, takes an action, and observes the result. On every turn, the model’s output picks one of four branches. These are a final answer, a tool call, a handoff, and a continued thought. ReAct is the most common prompting pattern for filling the loop, with reasoning and action interleaved. Guardrails live at every place where the loop crosses into the outside world.

The design carries three real costs. These are compounding reliability across steps, the harness scaffolding that production loops require, and the question of whether a workflow would solve the problem more cheaply.

References:

• Building effective agents — Anthropic Engineering

• Effective harnesses for long-running agents — Anthropic Engineering

• Running agents — OpenAI Agents SDK documentation

• Guardrails — OpenAI Agents SDK documentation

• ReAct: Synergizing Reasoning and Acting in Language Models

Learn practical techniques to make your software faster at every layer of the stack

Working on latency-sensitive applications? This book will help you spot, understand, and fix latency in your applications and infrastructure. It shares low-latency techniques that have been predominantly “tribal knowledge” until now. You’ll learn a full-stack approach, with techniques that intersect many areas of software engineering, distributed systems, databases, and operating systems.

• Understanding and negotiating caching tradeoffs (latency vs. complexity)

• Exploring the two main concurrency models (threading and event-driven architecture)

• Identifying and applying parallelization opportunities to improve throughput and reduce latency

• Applying asynchronous process strategies that reduce perceived latency and improve overall system responsiveness.

Get this 3-chapter excerpt and start reading the book that’s become essential for anyone working with strict performance SLAs.

Download for Free

If we send the same prompt to ChatGPT, Gemini, and Claude, the answers come back meaningfully different. Claude might push back more often. Gemini ingests a two-hour video file as easily as a paragraph of text. ChatGPT routes some prompts to a slower reasoning mode and answers others instantly, without telling the user which path it took.

Why does each model behave this way, and why so consistently?

Each pattern traces back to an architectural decision point where the developers of these frontier models made distinct design choices, and the user-visible behavior follows directly from those choices.

To understand these differences clearly, we need to examine architecture and design, since architectural decisions remain consistent across releases and provide a reliable framework for understanding model behavior over time.

In this article, we will look at the various architectural forks the teams building these models encountered and the decisions they took.

Disclaimer: This post is based on publicly shared details from various sources. Please comment if you notice any inaccuracies.

Foundations

All three models share the same fundamental architecture, which is a transformer-based generative neural network.

The transformer was introduced in the 2017 paper “Attention Is All You Need,” and at its core is a mechanism called self-attention that allows each token in a sequence (a token being a small piece of text, roughly three-quarters of a word) to weigh its relationship to every other token. This setup is shared across the three models, while the components built around it, along with the training process that shapes it, vary substantially.

Training itself involves two phases:

• Pre-training is where the model learns from massive amounts of text and other data by predicting the next token, repeated billions of times over.

• Post-training is where the base model gets shaped into a helpful assistant through methods like Reinforcement Learning from Human Feedback (RLHF) and Constitutional AI. We will get to them in more detail in a later section.

Each architectural fork follows a consistent pattern, where different companies faced the same design question, each made a particular architectural choice, and each choice produced user-visible behavior we can observe today.

The first such question concerns how to scale the model’s total capacity without proportionally increasing the cost of processing every query.

Density

Every parameter added to a model carries computational cost. In a standard dense neural network, every parameter activates for every token processed, meaning that doubling the parameter count roughly doubles the compute cost per query.

Frontier developers needed a way to escape this constraint, and the architectural question became how to scale total capacity without paying full price on every single token.

Google adopted a technique called Mixture of Experts, or MoE. The Gemini 1.5 technical report identifies the model as “a sparse mixture-of-expert (MoE) Transformer-based model” that “builds on a much longer history of MoE research at Google,” with the Gemini 3 Pro model extending the same approach.

In an MoE layer, instead of one massive computational block where every parameter activates for each token, the network contains many smaller experts, and a small router determines which two or three experts each token should be sent to. The total parameter count can be enormous while only a fraction is activated per token, though achieving consistent results requires careful attention to load balancing and expert specialization during training.

OpenAI has not explicitly confirmed whether GPT-4 uses MoE, and the GPT-4 technical report deliberately omits architecture details. The GPT-5 system details describe a different efficiency approach that we will examine in the Reasoning section, where a router selects between distinct sub-models at runtime.

Nevertheless, these architectural choices produce meaningful consequences in user experience.

MoE models can pack more knowledge per dollar of compute, which contributes to Gemini’s capable handling of a wide breadth of domains. The tradeoff is increased variance, since different prompts route to different experts, and imperfect load balancing during training can leave some experts underused and weaken the model’s capacity on those topics. Dense models tend to deliver more predictable per-token behavior at the cost of being harder to scale to extreme parameter counts.

See the illustration below:

Turn cloud logs into real security signals (Sponsored)

This guide from Datadog provides best practices on how to use Cloud SIEM to detect threats, investigate incidents, and reduce blind spots across cloud and Kubernetes environments.

You’ll learn how to:

• Analyze CloudTrail, GCP audit, and Azure logs for suspicious activity

• Detect authentication anomalies and common attack patterns

• Monitor Kubernetes audit logs for lateral movement and misuse

• Correlate signals across services to accelerate investigations

Get the guide

Multimodality

The term multimodal carries different meanings depending on the architectural approach, and a precise distinction helps before we take a look at the design choices.

Two fundamentally different approaches exist for handling non-text inputs like images, audio, and video:

• Sequential approach: Train a strong language model first, then add a separate encoder later that translates non-text inputs into embeddings the language model can attend to.

• Native approach: Train a single network from inception on all modalities simultaneously, so that text tokens, image patches, and audio frames coexist in the same sequence and get processed by the same transformer layers.

See the diagram below that shows the difference between the two:

These two approaches produce different capabilities.

Google adopted the native approach from the start. The Gemini 1.0 release was designed as natively multimodal, and the 1.5 technical report demonstrates the model handling, in a single input, “10.5 hours of video at 1 frame-per-second”, and the current Gemini 3 Pro model continues this design, where the model offers “native multimodal support for text, vision, and audio inputs.”

OpenAI took a different path initially, with the original GPT-4 supporting vision through a separate pipeline. GPT-4o, released in May 2024, transitioned to a unified architecture, and OpenAI’s own announcement for GPT-5.5 acknowledged that “previous multimodal models from OpenAI were essentially separate models stitched together.”

Lastly, Anthropic has maintained a text-first approach with strong vision capabilities, and Claude Opus 4.8 continues to support high-resolution image input introduced in Opus 4.7 at 3.75 megapixels, particularly useful for documents, screenshots, and dense charts. Anthropic has chosen to focus on text and vision as primary input modalities while setting native audio and video aside for now.

For users, this architectural difference appears most clearly when working with video. Gemini handles long video files directly, and ChatGPT does the same on its more recent models. Claude performs best with documents, screenshots, and high-fidelity images.

Context

A context window represents the amount of input a model can attend to in a single pass, measured in tokens.

Larger windows allow the model to reason over more raw material at once, though long context introduces two important challenges:

• Computational cost: Attention scales roughly quadratically with sequence length, so longer windows consume significantly more compute per query.

• Quality degradation: Models often lose track of information mentioned earlier as the window fills, with the relevance of early context dropping over the course of a long conversation.

Google pushed hardest on raw window size. The Gemini 1.5 technical report demonstrated context up to 10 million tokens, and Gemini 3 Pro and 3.1 Pro continue to offer 1 million token context in production.

Anthropic offers 1 million tokens on Claude Opus 4.8 and Sonnet 5, and the company’s own documentation uses the term “context rot” to describe quality degradation in long sessions, addressed through automatic compaction that summarizes earlier portions of the conversation.

OpenAI has taken a more conservative position on window size, with GPT-4 Turbo at 128K, focusing efficiency gains on the routing architecture covered in the Reasoning section.

The practical experience differs across the three models. Loading an entire codebase or a multi-hour transcript into a single session works well with Gemini and Claude, given the larger windows available. ChatGPT is structured more around efficient routing between sub-models than around accepting massive single-prompt inputs.

Long context expands the range of questions you can ask, while retrieval-augmented generation, which fetches relevant external data at query time, remains relevant when freshness and cost matter.

Alignment

Pre-training produces a base model that has processed enormous amounts of text and learned to predict the next token. At this stage, the base model behaves as a sophisticated pattern-completer with limited regard for honesty, safety, or helpfulness.

Post-training is the phase where this base model is shaped into the assistant that users interact with, and the three companies have taken substantially different approaches to this work.

OpenAI built on Reinforcement Learning from Human Feedback (RLHF) throughout the development of GPT-3 and GPT-4. The approach involves three main steps:

• Human raters compare pairs of model outputs and select the better one.

• Those comparisons train a reward model that captures human preferences.

• The reward model is then used to fine-tune the language model toward outputs that humans prefer.

The GPT-4 product page describes an additional safety reward signal incorporated into RLHF to reduce harmful outputs. For the reasoning models, OpenAI added an approach called deliberative alignment, where the model reasons about safety policies at inference time. The publicly available Model Spec describes intended behavior in detail.

Anthropic developed an alternative method called Constitutional AI, or CAI. Instead of relying on human raters at every step, CAI uses an explicit written constitution and has the model critique and revise its own outputs against those principles, then applies AI-generated feedback to further refine the model. In January 2026, Anthropic published a 23,000-word constitution that shapes Claude’s training across synthetic data generation, response ranking, and behavioral refinement.

Google uses RLHF with learned reward models in its post-training pipeline for Gemini, with somewhat less explicit public framing than the other two companies. The Gemini technical reports describe a multi-phase training process moving from unimodal pre-training through multimodal alignment.

This architectural area also reflects a broader pattern of differing transparency choices across the three companies. The 2023 GPT-4 technical report deliberately omitted architecture details, while the 2025 GPT-5 system card describes a router, named sub-models, and safety mitigations in remarkable detail. Anthropic publishes its constitution publicly, providing a detailed view into the principles that shape Claude’s behavior. The three companies have made distinct choices about transparency, and those choices have evolved over time.

For users, these training differences appear as personality. Claude tends to push back more often on edge cases because the constitution explicitly trains for that behavior. ChatGPT is often quicker to attempt a task without questioning. Gemini’s responses vary more by domain.

See the diagram below:

With values addressed, the most recent architectural dimension concerns how the model thinks, an area where the three companies have converged from different directions.

Reasoning

The most recent and most consequential architectural development for the current generation of models concerns whether the model produces an answer directly or reasons explicitly through tokens before answering.

A standard chat model takes a prompt and immediately begins generating the response. A reasoning model first generates internal thinking tokens, working through the problem, and only afterward produces the final answer. The same prompt, the same underlying weights, and the same context window can produce very different inference-time compute profiles depending on which approach is used.

OpenAI built dedicated reasoning models, beginning with o1 in late 2024, followed by o3 in August 2025, and continued in GPT-5.5, which was released on April 23, 2026. The architecture includes three main components.

• GPT-5-Main, a fast, high-throughput model and the successor to GPT-4o.

• GPT-5-Thinking, a deeper reasoning model and the successor to o3.

• A Real-Time Router that selects between them based on conversation type, complexity, tool needs, and explicit intent in the prompt.

OpenAI plans to integrate these capabilities into a single model in the future, though the current architecture functions as a system of models rather than as a unified single model.

Anthropic pursued a different approach. Claude Opus 4.8 and Sonnet 5 use adaptive thinking inside a single model, where the same Claude determines for itself how long to think on each request, with summarized reasoning surfaced to developers for inspection.

Google integrated thinking into Gemini 3 through Deep Think mode and the three-level thinking system in Gemini 3.1 Pro, enabling the model to reason over text, images, audio, and video in a single thinking pass.

For users, this architectural difference shows up more visibly than might be expected. ChatGPT can feel inconsistent across sessions because the router may have directed two similar prompts to two different sub-models. Claude tends to feel consistent because the same model varies its reasoning depth based on the request. Gemini’s behavior shifts more with domain and modality.

A convergence happens at this architectural fork. Three companies that began with very different choices at every prior dimension, including MoE versus dense architecture, native versus bolted-on multimodality, and RLHF versus Constitutional AI, have all arrived at the same approximate conclusion. Explicit reasoning tokens at inference time materially improve model performance on hard problems.

Conclusion

In this article, we looked at five architectural dimensions, each representing a question all three frontier developers had to answer when building their models.

• Density: Google adopted Mixture of Experts openly, OpenAI built routed efficiency through a different mechanism, and Anthropic kept its architectural choice private.

• Multimodality: Google adopted native fusion from inception, OpenAI evolved from bolted-on vision toward a unified architecture, and Anthropic maintained a text-first approach with strong vision capabilities.

• Context: Google pushed the largest windows, Anthropic invested in long sessions with explicit handling of degradation, and OpenAI emphasized routing efficiency over raw window size.

• Alignment: OpenAI’s RLHF combined with the Model Spec, Anthropic’s Constitutional AI combined with its published constitution, and Google’s RLHF approach represent three distinct shaping pipelines.

• Reasoning: All three companies converged on explicit reasoning tokens, despite starting from different architectural choices at every prior dimension.

When we observe one of these models behaving in a particular way, we now have a framework for asking which architectural dimension produced that behavior.

References:

• GPT-4 Technical Report

• GPT-4 Research Page

• GPT-5 System Card

• Introducing GPT-5.5

• Learning to Reason with LLMs

• Introducing OpenAI o3 and o4-mini

• OpenAI Model Spec

• Reasoning Models Documentation

• Claude’s Constitution

• Claude’s New Constitution

• Constitutional AI: Harmlessness from AI Feedback

• Claude Opus 4.7

• What’s New in Claude Opus 4.7

• Context Windows

• Adaptive Thinking

• Building with Extended Thinking

• Gemini 1.5 Technical Report

• Gemini 2.5 Technical Report

• Introducing Gemini 1.5

• Attention Is All You Need

• Gemini 3 Technical Report

• Introducing Claude Opus 4.8

Check it out Here

Here’s what makes this cohort special:

• Learn by doing: Build real world AI applications, not just by watching videos.

• Structured, systematic learning path: Follow a carefully designed curriculum that takes you step by step, from fundamentals to advanced topics.

• Live feedback and mentorship: Get direct feedback from instructors and peers.

• Community driven: Learning alone is hard. Learning with a community is easy!

We are focused on skill building, not just theory or passive learning. Our goal is for every participant to walk away with a strong foundation for building AI systems.

If you want to start learning AI from scratch, this is the perfect platform for you to begin.

Check it out Here

The defenses websites have relied on for years are failing in predictable ways, and the failure shows up most visibly in moments like the one below.

A popular retailer drops a thousand pairs of a limited-edition shoe at noon. Within thirty seconds, the inventory is gone. The buyers turn out to be automated agents working for resellers. The real people who wanted the shoes get nothing.

The retailer almost certainly tried the standard defenses: rate limits by IP, CAPTCHA, phone verification, device fingerprinting. Each of them helps for a while and then stops working. The reason they all fail in the same way is worth pausing on.

Every one of those defenses relies on a proxy for what the retailer actually wants to verify. An IP address is a proxy for a different network. A phone number is a proxy for a different person. A device fingerprint is a proxy for a different device. Each proxy fails the moment adversaries learn to acquire many of them cheaply, and adversaries always do. Phone numbers can be bought in bulk. Fingerprints can be randomized. IPs can be rotated through residential proxy networks. None of these strategies binds the verification to a real, unique person.

Authentication systems do not solve this either. Single sign-on, face unlock, passkeys, OAuth tokens, all of them compare an incoming credential against a stored template and return yes or no. None of them answers the question that actually matters here: has this user already been verified somewhere else in the world?

The central question: How do you let a real, unique human be recognized across the internet, without ever knowing who they are?

For this article we spoke with the team behind World, including Tiago Sada and Lily Gordon at Tools for Humanity, on how they try to solve this problem.

Five Pillars of a Proof-of-Human System

A working answer to the central question requires five distinct ingredients. We will look at each one in turn.

Uniqueness. Why is this a different identity problem than the ones we usually solve?

Anonymity. How can a credential be issued without anyone knowing the identity of the user?

Recovery. How does the system survive lost phones and reinstalled apps?

Verification. How does the holder present the credential without revealing more than necessary?

Delegation. What changes when the holder is an AI agent acting on a person’s behalf?

Pillar 1: Uniqueness

To see why uniqueness is structurally different from authentication, start with a system everyone already trusts: Face ID on a phone.

When the user sets up the phone, the camera captures a single facial template and stores it locally. From then on, every unlock attempt produces a fresh capture, which the phone compares against the stored template. The comparison space is exactly one. If the fresh capture matches the template within some tolerance, the phone unlocks. If not, the phone refuses. The system handles other faces by failing the match.

This is a one-to-one matching problem. The math is simple because the comparison is small. Even with a per-comparison error rate of one in a million, the phone unlocks reliably for its owner and refuses essentially every other face.

However, the example with the retailer wants a different guarantee. They want to verify, at checkout, that the buyer is different from every other person who has already bought those special edition shoes. The comparison is no longer against one template, but against the entire population of past buyers. If the system is meant to work at internet scale, the population is potentially every person on the planet.

This is a one-to-many matching problem, and the diagram below shows this comparison in the context of World ID. As you can notice, the math can get much more complex when the comparison space grows.

The probability of a false match scales roughly with the size of the comparison space. A per-comparison error rate of one in a million sounds excellent, but checked against a billion candidates, it produces roughly a thousand false matches per query. To make one-to-many uniqueness viable at a billion-person scale, the per-comparison error rate has to be on the order of one in a hundred billion or better. That requirement rules out most consumer-grade biometric methods straight away.

This type of system that answers the one-to-many question is what the World ID calls the proof of human. At present, there is no widely deployed equivalent of this category. We have a great deal of authentication infrastructure and very little uniqueness infrastructure.

Pillar 2: Anonymity

Once the goal is uniqueness at an internet scale, an apparent paradox emerges.

Checking whether someone is the same as anyone else usually requires recognizing them. If a system cannot identify a person, how can it tell whether it has seen that person before?

The answer takes multiple steps.

Step 1: Find a biometric signal that scales

Most consumer biometrics fall short. For example, fingerprints have decent entropy but can be captured from surfaces. Face geometry varies less than people assume. The iris pattern of the human eye, by contrast, turns out to be one of the few biometric features with the entropy needed for billion-scale comparisons. Two unrelated humans have essentially no chance of producing matching iris patterns, even after accounting for camera noise and aging.

However, reading the iris reliably is the next challenge, and this is where hardware matters.

A standard phone camera can be replaced with a device that injects images directly into the camera pipeline, defeating any face check done in software. A printed iris image can fool an infrared camera that lacks depth detection. To get a reading that resists these attacks, the capture has to happen on hardware that controls the entire signal path from sensor to processing.

Tools for Humanity handles this using a purpose-built device called the Orb. The Orb uses multispectral imaging across infrared and visible wavelengths, runs several neural networks locally to verify liveness and detect masks, and deletes the original images before anything leaves the device. What gets transmitted is signed, encrypted material derived from the iris, not the iris itself.

Step 2: Check for duplicates without anyone seeing the data

That solves the first half of the problem. The second half is harder. The system needs to check whether anyone else in the world has the same reading, while no party (including the system operator) ever sees the reading itself.

The trick works like this:

• Take the iris reading and split it into three pieces of statistically random noise.

• Hand each piece to a different organization operating in a different legal jurisdiction.

• Each party, looking only at its piece, can tell nothing about the underlying reading.

• The parties can jointly compute answers like “does this reading match any reading we have already enrolled?” without any single party ever holding the whole reading.

• The pieces are combined only in the abstract space of the computation, never reconstructed on any actual machine.

This is what cryptographers call secure multi-party computation, and the specific variant World uses is called anonymized multi-party computation (AMPC). The parties (referred to as SMPC nodes) are operated by independent organizations in different legal jurisdictions.

See the diagram below that tries to conceptualize what each SMPC party actually sees:

What emerges from this setup is the property that makes proof of human possible. The system can determine whether a given human has enrolled before, and no single party (including the main organization) ever sees the underlying biometric in usable form. The user is verified as unique without being identified.

Pillar 3: Recovery

A verified credential is only useful if the human carrying it can keep using it across years and across devices. In a normal life, phones get lost, and software gets reinstalled. Any system meant to anchor a lifelong property like “I am a verified unique human” has to handle the case where the holder loses access to everything they were using to prove it.

If the credential is a private key on a phone, losing the phone means losing the credential. The user would have to re-enroll. If the system is anchored in physical hardware verification, it means traveling back to a central location (in the case of World ID, that means the Orb). It would be a rather poor user experience for everyday loss, and impossible for some users in some regions.

The fix is to stop treating the credential as a single secret. Instead, treat the verified human as an abstract account in a public registry. The account does not hold the user’s biometric or any user secret directly. It holds a list of public keys (called Authenticators) that the user has authorized to act on the account’s behalf.

Think of an Authenticator as any piece of software or hardware that holds a private key on the user’s behalf. Here are some examples:

• A common Authenticator could be the user’s phone with the wallet app installed.

• A browser extension that holds a separate key could be another.

• A hardware security token plugged in over USB is a third option.

When the user wants to verify on a service or app, the Authenticator is what actually produces the cryptographic proof and signs the verification request with its private key. The account in the public registry holds the matching public key, so anyone can confirm the signature without ever knowing the secret. Keys can be added, revoked, and rotated. The account itself persists.

For more context, the public registry stores one entry per verified human. Each entry contains the public keys of all currently authorized Authenticators, the designated Recovery agents for that account, and the rules under which those agents can act. The biometric reading itself is never stored in the registry. This ensures that reading the registry cannot reveal personal details about the verified humans.

See the diagram below that shows this approach in the context of World ID’s solution:

If the user loses all of their Authenticators, the account also designates one or more Recovery Agents, which are parties trusted to re-authenticate the human (typically by running them through a fresh biometric check) and let them register new keys.

In the current design of World ID, the Orb network serves as the default Recovery Agent. The on-chain registry that holds all of this is called the WorldIDRegistry.

What this might look like in practice is as follows: imagine the user’s phone is stolen on a trip. They get a new phone and install the wallet app fresh, but the app has no private keys yet. Rather than travel back to an Orb to start over, the user can initiate a recovery through one of the Recovery Agents they previously designated. The agent runs them through a fresh biometric check, typically through the Orb’s mobile face authentication. If the check passes, the agent submits a transaction to the registry that revokes the stolen phone’s key and registers the new phone’s key.

However, the open question here is governance. The Recovery Agents can themselves become a trust point, which would require some form of oversight.

Pillar 4: Verification

So far, we have a way to verify a human as unique, and a way for them to hold their credentials durably.

What we need next is a way to present that credential to a relying party (for example, a dating app, a shoe retailer, or any service that wants to enforce one-per-human) so the service can do its job while learning as little as possible about the user.

A relying party in this scenario needs three things at once:

• Assurance that the request is backed by a real and verified unique human.

• The ability to enforce its own one-per-human rule, which means recognizing repeat visits by the same human within its own context.

• Confidence that what the user reveals is limited to the fact of being a verified, unique human, with identity, other activity, and behavior on other platforms staying private.

Satisfying all three at once is harder than it sounds. Standard identity tokens (an OAuth token, a JWT, a session cookie) tie recognition to identity. The relying party gets the ability to enforce policies, but pays for it with the user’s privacy.

The World ID handled this problem using a primitive called a nullifier. A nullifier is a number derived from three inputs combined together. The inputs are the user’s verified credentials, the relying party’s identifier, and a particular action. The same combination always produces the same number. Different combinations produce uncorrelated numbers. If the relying party sees the same nullifier again for the same action, they know it is the same human attempting the same thing twice. Nullifiers from the same user on a different action, or in a different service, are uncorrelated and cannot be linked back.

See the diagram below that shows what the relying party actually sees:

Generating a nullifier like this requires careful design. A user cannot generate it alone (they cannot be trusted to use the right inputs), and a single server cannot generate it either (the server would learn who the user is). It needs a distributed network of nodes, and here’s how they work:

• The user sends a blinded version of their query

• Each node computes on the blinded value while staying unaware of what the value represents

• The user unblinds the result.

• A threshold of nodes must cooperate. These nodes are called OPRF nodes (oblivious pseudorandom function).

One more piece is needed to complete the puzzle. If a user could reuse the same nullifier multiple times within one relying party context, an enterprising service could build a pseudonymous profile of the user’s repeated interactions. To prevent this, the design maintains an on-chain list of every nullifier that has been used, called the Oblivious Nullifier Pool. The user’s own software refuses to generate a proof for a nullifier already in the pool.

So, how is all this complexity handled from an application’s perspective?

In the context of World ID, all of this gets bundled into two zero-knowledge proofs that the user presents to the relying party. An SDK named IDKIT wraps the flow for integrators.

See the diagram below that shows a simple IDKit Proof Flow:

Pillar 5: Delegation

Up to this point, we have assumed the human is the one acting. The user might open a dating app, buy sneakers from an online retailer, or sign a document. The design handles each case by binding one verifiable human to one proof.

The interesting question is what happens when the human is not the one acting. This is because people are now increasingly using AI agents to take actions on their behalf. For example:

• An assistant books a flight.

• An agent monitors product drops and buys when something matches.

• A service fills out and submits forms for a user who would rather not.

The standard defense against agents is to block them as bots, but here the human wants the agent to act for them. Treating the agent as adversarial defeats the very automation the user is paying for.

The extension to the design for an ideal proof of human solution is to let the agent register against a real human’s credentials before it acts. When the agent later contacts a service, the service can verify that the agent is backed by a verified, unique human, while still learning nothing about which human. The agent’s activity counts against the human’s quota, so a single human cannot turn into many parallel agents to amplify their reach.

See the diagram below that shows how World ID implements this using AgentKit:

In World’s implementation, the agent registers its wallet address in a registry called AgentBook (deployed on the World Chain blockchain), tying the agent to the human’s World ID through a verification flow in the user’s wallet app. A service or application that wants to admit human-backed agents installs the SDK called AgentKit, which by default allows three uses per human per service before normal payment takes over.

A few new use cases become possible:

• A startup can offer a free trial that bots cannot farm at scale, because each trial use is bound to a verified human.

• A retailer can enforce one pair per human on a limited drop, even when agents do the buying.

• A merchant can require human backing for transactions where chargeback fraud is a concern.

What remains open is how this scales. At present, the volume of agent traffic is quite small compared with what AI agents will likely produce in the coming years. Whether the per-human cap stays meaningful when one human represents many concurrent agents is a question worth looking into.

Conclusion

Taken together, the five pillars (uniqueness, anonymity, recovery, verification, and delegation) form a working blueprint for proof of human in principle. Whether any particular implementation, World ID included, becomes the way the larger question gets answered at internet scale remains to be seen.

A few specific questions are worth tracking as the design meets larger-scale reality:

• Hardware decentralization. The longer-term plan calls for multiple independent manufacturers of devices like the Orb, distributed across jurisdictions and cross-checking each other through security deposits. The architectural properties depend on this transition actually happening.

• Unlinkability under adversarial pressure. The cryptography looks sound on paper. The harder test is what happens when relying parties, governments, or other large actors try to extract more from the system than the design intends. Whether the unlinkability properties hold in practice at scale is something the community will only learn over time.

• Delegation at agent scale. Most agent traffic today still flows through traditional authorization systems. As agents take on more autonomous activity, the per-human cap will be tested in ways the existing deployment has not yet seen. New abuse patterns will almost certainly emerge.

• The bootstrap problem. Proof of human becomes useful at scale, and reaching scale requires apps to require it, which in turn requires users to have it. How any such system navigates that loop, and what happens if a meaningful fraction of the internet starts to require proof of human, is far from clear today.

The point of this article was not to argue for any particular product, but to surface the techniques. Once you can see the shape of the problem, one-to-many matching, secret-shared biometrics, scoped nullifiers, account-style key rotation, delegated quotas, the headlines about “verify you are human” start to look less mysterious. They are attempts to solve a specific class of problems that the rest of the identity stack has so far left alone.

References and Further Reading

• https://world.org/liftoff

• https://docs.world.org/world-id/idkit/integrate

• https://docs.world.org/agents/agent-kit/integrate

• https://world.org/blog/engineering/private-proof-of-human

• https://eprint.iacr.org/2024/705

• https://world.org/blog/engineering/introducing-world-id-4.0

That runs against the instinct, which says more locations should mean faster service and steadier uptime, which is often the case. But once the same data lives in two places at once, a new class of problem appears that can nullify the advantages of multi-region deployments.

Let’s look at a short example. Picture two edits to the same piece of data, made at nearly the same instant, one handled by a server on the US East Coast and one by a server in Frankfurt, right as the network link between those regions drops. Both edits are saved locally. Each region now holds a different version of that data, with no shared record of which one came first. When the link comes back, one of the two versions has to be chosen over the other. The way a system handles that question has a large effect on what a global footprint costs to build and to run.

Going global is better understood as a progression than as a single decision. In this article, we will start with the building blocks, the small set of concepts every regional design rests on, then work through the common setups in order, from a single region with backups up to running every region at once. Each step buys something concrete: lower latency, higher availability, or the ability to keep data inside a country’s borders. Each one also has a price, in money and in the consistency tradeoffs it introduces.

Foundations

Read more

Matic is the first visually intelligent robot vacuum that sees your home like you do, so it can clean how you want it to.
Matic’s hero features:

• Runs entirely on cameras to deftly navigate obstacles

• Recognizes floor types to auto-switch between vacuuming and mopping

• Big wheels and a height-adjustable cleaning head handle thick rugs

• Quieter than conversations at 55 dB

• Handles pet hair without clogging or tangling

• A single bag collects dry and wet waste—diaper-salts gel dirty water, antimicrobial powder prevents mold

• A fresh HEPA filter in every bag for cleaner air, no washing or replacing

Experience autonomous cleaning with Matic with a 180-day money-back guarantee.

Get your Matic

OpenAI runs voice AI for 900 million users a week, and they use WebRTC for it because the alternative would mean reinventing how the internet handles live audio.

The catch is that WebRTC was designed for servers with stable IPs and ports, and Kubernetes treats those addresses as disposable. The conventional answer at this scale is an SFU, which suits multiparty workloads like group video calls, but OpenAI’s traffic is overwhelmingly one user talking to one model.

To deal with this, their architecture splits the stack into two pieces:

• A stateless relay handles protocol-aware packet routing at the geographic edge.

• A stateful transceiver owns all the heavy WebRTC state.

The trick that ties them together is using the ICE ufrag, a field the protocol already exchanges during setup, as a routing key that the relay can read off the first packet of a new session. Everything else, from Global Relay to the userspace Go implementation to the Redis cache and the careful socket-level optimizations, builds on top of that core idea.

In this article, we will look at the entire journey in detail and challenges the OpenAI engineering team faced.

Disclaimer: This post is based on publicly shared details from the Open AI Engineering Team. Please comment if you notice any inaccuracies.

Why Latency Matters For Voice AI

Voice AI either feels like a conversation or it feels like a walkie-talkie. The line between those experiences is measured in milliseconds.

When the network pauses between hearing a user and responding, the illusion breaks. Pauses turn awkward, interruptions get clipped, and users are compelled to cut off the AI mid-sentence, which is also kind of rude. In other words, voice AI only feels natural if the conversation moves at the speed of speech.

The harder constraint underneath is the continuous-stream property. Audio has to arrive at the model as a steady flow, rather than as a single upload after the user finishes talking. That stream is what lets the model start transcribing, reasoning, and calling tools while the user is still speaking. The experience collapses into push-to-talk once it breaks.

For OpenAI specifically, those constraints translate into three concrete requirements:

• The system has to reach 900 million weekly active users wherever they are.

• Connection setup has to be completed quickly enough that users can start speaking as soon as a session begins.

• Round-trip time for audio has to stay low and stable so turn-taking feels crisp.

WebRTC is the protocol the industry built for this kind of work. It is a bundle of smaller protocols (ICE for figuring out how two endpoints reach each other across firewalls, DTLS for encrypting the channel, SRTP for the audio packets, and RTCP for quality feedback). Justin Uberti, one of WebRTC’s original architects, and Sean DuBois, who maintains the Pion library OpenAI builds on, both work at OpenAI today. That kind of protocol depth shows up in the architecture they shipped.

The Original Architecture

The first version of OpenAI’s WebRTC infrastructure was a single Go service built on Pion. It handled both jobs in one place:

• On the signaling side, the service negotiated SDP (the format clients and servers use to describe a session), selected codecs, generated ICE credentials, and set up sessions.

• On the media side, the service terminated WebRTC connections from clients and maintained upstream connections to the backend services that run the AI models, including inference, transcription, speech generation, tool use, and orchestration.

That combined service still powers ChatGPT voice, the Realtime API’s WebRTC endpoint, and several research projects, and it has handled that work well. The question OpenAI ran into was how to deploy it on Kubernetes, the container orchestration system that runs most modern cloud infrastructure.

Kubernetes assumes compute is cheap and movable. Pods come up, get scheduled wherever capacity exists, run for a while, then get rescheduled or replaced. Standard WebRTC deployment patterns assume the opposite. That mismatch shows up in two specific places.

The first is port exhaustion.

The conventional way to deploy WebRTC uses one UDP port per session. At OpenAI’s scale, that means tens of thousands of public UDP ports per service. Cloud load balancers were built for a handful of well-known ports, so each additional range adds operational complexity for load balancer config, health checks, firewall policy, and rollout safety. The exposed surface area also makes security audits harder. Kubernetes autoscaling clashes with the requirement to reserve large and stable port ranges, which makes elasticity brittle.

The second is state stickiness.

Running one UDP port per server and demultiplexing sessions behind it solves the port problem. ICE and DTLS, however, are stateful protocols. The process that started a session has to keep receiving its packets to validate connectivity checks, complete the DTLS handshake, decrypt SRTP, and process later session changes like ICE restarts. If a packet for an existing session lands on a different process, setup fails, or media breaks.

Both pressures point to the same answer. The deployment architecture has to change while the client experience stays identical.

Splitting The Relay From The Transceiver

The architecture OpenAI shipped splits packet routing from protocol termination.

A stateless relay sits at the front, presenting a small public footprint to the internet. A stateful transceiver sits behind it, owning all the heavy WebRTC state. Signaling still goes directly to the transceiver. Media enters through the relay first.

The relay’s scope is deliberately narrow. It reads enough of each packet to choose a destination, then forwards the rest as an opaque payload. Audio stays encrypted on the way through, ICE state machines stay with the transceiver, and codec negotiation happens elsewhere. From a client’s perspective, the WebRTC session looks normal in every way.

The transceiver owns the parts of WebRTC that have to remember things. ICE connectivity checks, the DTLS handshake, SRTP encryption keys, and the session lifecycle all live there. The transceiver is the endpoint that completes the handshakes and encrypts or decrypts the actual media.

See the diagram below:

There was an obvious alternative that the team evaluated and chose against.

An SFU, or Selective Forwarding Unit, is the standard media server architecture for WebRTC at scale. It terminates one WebRTC connection per participant and selectively forwards streams between them. The AI joins as another participant.

This works well for inherently multiparty products like group calls, classrooms, and collaborative meetings. OpenAI’s workload looks different. Most sessions are 1:1, with one user talking to one model. For that kind of traffic, the SFU model adds overhead and forces backend services to behave like WebRTC peers themselves. The transceiver model lets the backend stay an ordinary service.

See the diagram below that shows the SFU approach:

TURN was also considered and set aside.

TURN is the standard protocol-terminating relay used for NAT traversal. The trouble is that TURN allocations add setup round-trips before media can flow, and migrating or recovering them across servers is hard. For a latency-sensitive workload, those extra round-trip matters.

The split solves the port and state problems in principle. The remaining problem is making the relay route the first packet correctly.

Routing The First Packet

The first packet of any new session is the difficult one.

Subsequent packets are easy because the relay has a mapping that says that packets from this source IP and port go to this transceiver. The first packet is what creates that mapping, so the relay has to figure out where to send it from the packet itself.

See the diagram below:

Two naive options were present:

• A database lookup on the hot path adds latency and a hard dependency on another service staying healthy.

• Routing to a random transceiver and forwarding internally works, but doubles the hop count.

OpenAI chose a third option.

The answer lives inside a field that WebRTC already exchanges. Every WebRTC session carries an ICE username fragment, called the ufrag, which is produced during session setup and echoed in STUN binding requests. STUN binding requests are the packets ICE uses to verify that two endpoints can actually reach each other, and they are usually the first thing a client sends on the media path.

The trick is that OpenAI generates the server-side ufrag during signaling. They can put whatever they want in it, so they encode routing metadata into it. The relay parses just enough of the first STUN binding request to read the ufrag, decode the routing hint, and forward the packet to the transceiver that owns the session. Every packet after the first one flows through the established session mapping, which skips the ufrag parsing step entirely.

See the diagram below that shows the connection establishment sequence in detail:

Each transceiver in the fleet listens on a shared UDP socket, which is one operating system endpoint bound to an internal IP and port. All sessions for that transceiver multiplex behind it.

During signaling, the transceiver returns a shared relay VIP and UDP port in the SDP answer. A VIP is a virtual IP address that fronts the entire relay fleet, so the client sees one stable destination like 203.0.113.10:3478, even though many relay instances sit behind that address. From the client’s side, there is one place to send packets, and it stays the same for the life of the session.

The relay’s state is purposefully tiny. It holds an in-memory map of source address to transceiver destination, plus some counters for monitoring and timers for session cleanup. If a relay instance restarts and loses the mapping, the next STUN packet rebuilds it from the ufrag. To make recovery faster, a Redis cache holds the source-to-destination mapping once a route is established. A restarted relay can look up the mapping from Redis immediately.

The principle here generalizes well. When we need data on the hot path, look at what the protocol is already exchanging. A field on the payload is essentially free to parse. A new lookup costs latency, a dependency, and one more thing that can break.

Global Relay and Geo-Steered Signaling

Once the public UDP surface was reduced to a small fixed set of addresses, the same relay pattern became deployable globally.

Global Relay is OpenAI’s fleet of geographically distributed relay ingress points. All of them run the identical packet-forwarding behavior described above. The only thing that changes is where on the map they sit.

Geographic distribution shortens the first client-to-OpenAI hop. A packet entering the network at a relay close to the user, in both geography and network topology, has a much easier time than a packet that has to traverse the public internet to reach a distant region first. The practical effect is lower latency, more stable timing, and a cleaner loss profile before traffic reaches the OpenAI backbone.

OpenAI uses Cloudflare for geographic and proximity steering on the signaling side. The initial HTTP or WebSocket request that sets up a session is routed to a nearby transceiver cluster. The request context then determines which Global Relay ingress point gets advertised back to the client in the SDP answer. The ufrag carries enough information for Global Relay to route media to the right cluster, and for the cluster’s relay to route to the right transceiver.

The result is that both the signaling and the media paths enter the OpenAI network at points close to the user, while the session itself stays anchored to one specific transceiver for its full lifetime. The setup round-trip and the first ICE connectivity check both shorten, which directly reduces how long a user waits before they can start speaking.

The Go Relay Implementation

The relay is a Go service running in userspace, which is to say a regular process that reads from a regular UDP socket.

The Linux kernel receives UDP packets from the network interface, delivers them to a socket bound to the relay’s IP and port, and the Go process reads from that socket, updates a small amount of flow state, and forwards each packet to the right transceiver.

OpenAI evaluated kernel-bypass frameworks (which let a userspace process poll the network card’s queues directly) and chose to stay away. Bypass raises packet throughput at the cost of operational complexity. The team’s workload fit inside what a careful Go implementation could handle.

Three implementation choices carry most of the performance load.

• SO_REUSEPORT is a Linux socket option that lets multiple workers on the same machine bind the same UDP port. The kernel then distributes incoming packets across those workers, which removes the bottleneck of a single read loop.

• runtime.LockOSThread pins each UDP-reading goroutine (a lightweight thread in Go) to a specific OS thread. Combined with SO_REUSEPORT, this tends to keep packets from the same flow on the same CPU core, which helps cache locality and reduces context switching.

• Pre-allocated buffers and minimal copying during packet parsing keep allocation overhead low and avoid putting pressure on Go’s garbage collector.

The takeaway is that ordinary optimizations were enough. The relay handles OpenAI’s global real-time media traffic on a relatively small footprint.

Design Tradeoffs

Every architecture comes with tradeoffs, and this one carries several worth understanding.

• The whole design is built around 1:1 sessions. If OpenAI ever wants to add multiparty features (group voice calls, multiple participants in a single AI session, human handoff during a call), large parts of this architecture would probably need rework. Both the transceiver model and the choice to skip SFU rely on the assumption that most sessions have exactly two endpoints.

• OpenAI also took on a custom infrastructure burden. A standard SFU comes with documentation, a community, and battle-tested patterns. The relay, the transceiver, and the coordination between them are all internal code. Every engineer who works on this stack has to learn it from the inside out.

• The “stateless” relay turns out to be stateless mostly in spirit. It holds an in-memory flow table and uses a Redis cache to recover that table across restarts. The architecture works because the protocol can rebuild the table from the ufrag, but the soft state is still a state.

• Lastly, the ufrag trick depends on controlling both ends of signaling. OpenAI generates the server-side ufrag, so they can embed routing metadata in it freely. A team that uses an off-the-shelf signaling stack might find this technique harder to adapt directly.

Conclusion

The architecture OpenAI built for voice AI is a careful response to a specific pressure. WebRTC was designed for stable servers. Modern cloud infrastructure runs on the opposite assumption. OpenAI’s team had the protocol depth to determine whether a WebRTC session needs to live in one process at all.

Their answer separates the work into two pieces. A stateless relay forwards packets near the user. A stateful transceiver, anchored in one place, owns ICE, DTLS, SRTP, and the session lifecycle. The two pieces communicate through information that’s already in the WebRTC handshake, which keeps the routing decision on the packet path itself.

The implementation choices stayed deliberately simple. Userspace Go, SO_REUSEPORT, thread pinning, and careful memory management did the work that kernel bypass solves. The result handles 900 million weekly users on a relatively small relay footprint.

References:

• How OpenAI delivers low-latency voice AI at scale

Run npx workos@latest and an AI agent will inspect your project, detect your framework, and add AuthKit directly to your codebase.

This is not a generic starter template. The agent reads your actual app, writes the integration where it belongs, then typechecks and builds so it can fix errors along the way. AuthKit gives you hosted auth, customizable UI, MFA, social login, session management, and a path to enterprise features like SSO and SCIM when you need them.

Free up to 1 million MAUs.

Try it now →

What feels like a real-time conversation with AI today is built from many parts working together.

At the center sits a language model that works in turns, the same way ChatGPT does when you type to it. The responsiveness comes from a layer of helper systems wrapped around that model, predicting when the user has paused, transcribing audio, generating speech from text, and weaving the pieces together fast enough that the conversation feels fluid.

However, new research from Thinking Machines argues that this whole approach has a ceiling, and proposes a different way to build AI systems for real-time interaction.

Thinking Machines is a relatively new AI research lab focused on human-AI collaboration, publishing research under the name Connectionism and offering developer-facing products for the broader community. What sets them apart is the problem they have identified as central. Most AI labs treat autonomous capability as the most important capability to push forward, meaning the ability for a model to take a task, do the work on its own, and return a result.

Thinking Machines argues this framing sidelines humans. Real work, in their view, benefits from continuous collaboration where the human clarifies, redirects, and gives feedback as the model goes along. The interface should support that, rather than treating the human as someone who hands off a task and walks away.

In this article, we will look at what the research preview covers and the concept of an interaction model proposed by Thinking Machines.

Disclaimer: This post is based on publicly shared details from the Thinking Machines Engineering Team. Please comment if you notice any inaccuracies.

Bottleneck

The problem starts with how today’s models actually experience the world. A typical language model works in a single thread. It waits for the user to finish typing or speaking before it can perceive any input. Once the model starts generating a response, its perception freezes, and any new input gets queued for later.

Thinking Machines compares this setup to resolving a crucial disagreement over email rather than in person. The bandwidth is just too narrow. So much of what makes a collaboration work, the way your voice shifts when uncertain, the moment of realizing a direction change is needed mid-sentence, the reaction on your face when the other person says something useful, all of it gets stripped out of the channel between human and model.

This matters because real work that benefits from another mind in the room depends on that bandwidth.

A model that only sees clean, finalized inputs forces a person to think like a model, preparing the full request, handing it over, and then waiting. In contrast, real collaboration is often messy, interruptive, and full of mid-stream corrections. Until the interface allows for that, the human ends up doing extra work to fit how the model wants to operate. Thinking Machines argues this bottleneck explains why much of today’s AI work feels like prompting and waiting rather than collaborating the way two people might.

Harness

If today’s voice AI feels real-time despite this limitation, how is that even working to a large extent? The answer is a pattern called a harness.

A typical voice AI product is a stack of components glued together:

• Voice activity detection listens for pauses and decides when the user has stopped speaking.

• A speech-to-text model transcribes what was said.

• A language model generates a text response.

• A text-to-speech model converts that response back into audio.

• A dialog manager orchestrates the entire pipeline so the latency feels acceptable.

Imagine a brilliant scholar who communicates only through letters slipped under a door. Making this feel like a conversation requires helpers. One stands outside listening for when the visitor stops talking, another reads the scholar’s letters aloud when they come back, and a third rings a bell when something visible happens that the scholar should know about.

The setup mostly works, but the scholar still experiences reality through letters. Voice tones, facial expressions, the moment itself, all of it stays beyond the scholar’s reach. This is what every real-time voice AI actually is, with a turn-based language model at the center surrounded by helpers that simulate conversation around it.

Why does this approach have a ceiling?

It is because the helpers are simpler than the model itself. Voice activity detection runs on raw audio signals using a much smaller and lighter model than the language model behind it. This limits whole categories of behavior.

The system struggles with proactive interjections like “interrupt me when I say something wrong,” because the helper deciding when to speak operates purely on acoustic signals, while correctness remains the language model’s job. Visual reactions like “tell me when I’ve written a bug in my code” face the same problem, because the helper handles audio while anything on screen stays beyond its reach.

This is where Thinking Machines points to an important lesson. As per a famous essay by Rich Sutton, methods leveraging general computation and learning consistently outperform methods that bake in human-designed heuristics. The same argument led from hand-crafted computer vision features to deep learning, and from hand-crafted game heuristics to self-play. Applied to interactivity, harness components are exactly the kind of hand-crafted heuristic that scale will eventually push out. The way past the ceiling is to put interactivity inside the model itself.

How engineering teams resolve incidents in minutes instead of hours. (Sponsored)

Datadog and Dust share how unified observability gives engineers full visibility across infrastructure, logs, and applications to detect issues faster and reduce MTTR. Watch this on-demand webinar to see how Dust reduced complex incident investigations from hours to minutes by bringing observability context into their AI workflows.

You’ll learn:

• How unified observability connects infrastructure health, logs, and applications into a single view for faster incident resolution

• How Dust used the Datadog MCP to pull observability context directly into incident investigations

• Practical strategies for maintaining reliability while building for the AI era

Watch the webinar

Architecture

What does putting interactivity inside the model actually look like?

Thinking Machines’ answer is a system they call an interaction model. The first version, named TML-Interaction-Small, is a 276-billion-parameter mixture-of-experts model with 12 billion active parameters at any moment. The word “small” in the name refers to where this sits in their planned lineup, with larger versions expected later.

Most multimodal systems start with text and add audio and video on top.

Thinking Machines did the reverse, starting from continuous audio and video because live conversation operates under tight real-time constraints that text can avoid. Designing around the hardest case first gives them an architecture that handles concurrent input and output streams across every modality.

See the diagram below:

Three design choices stand behind this architecture:

• The first is time-aligned micro-turns, which change what the model treats as the unit of conversation.

• The second is an approach that skips heavy pretrained encoders, with audio and video going through lightweight processing components trained from scratch rather than being routed through standalone systems like Whisper.

• The third is a two-model coordination scheme where a fast interaction model works alongside a slower background model that handles deeper reasoning.

Thinking Machines also did significant work on optimizing inference for this design, including contributing a streaming session feature back to the open-source SGLang library, enabling 200-millisecond chunks to be processed efficiently.

Micro-turns

Most AI models work in turns, with the user speaking, then the model speaking, then the user speaking again. Each turn is a discrete unit, and the model processes one complete turn at a time. Even when a system handles audio, the underlying logic stays turn-based. The harness simulates real-time, but the model itself perceives the world in clear, separate chunks.

Thinking Machines made a different choice.

Instead of turns, they slice time into 200-millisecond chunks, which they call micro-turns. Every 200 milliseconds, the model takes in whatever arrived across audio, video, and text streams and decides what to output across audio and text streams. Time becomes the fundamental unit, replacing the turn entirely.

This sounds like a small change, but it transforms what the model can do. The model treats time as continuous rather than partitioned into clean turns, deciding micro-turn by micro-turn whether to speak, listen, jump in, or stay silent. Input and output are happening continuously at the same time.

Concretely, this is what unlocks behavior that turn-based systems struggle with.

• The model can speak while listening, which is how live translation works.

• It can watch while speaking, which is how live sports commentary works.

• It can jump in mid-sentence when something visual happens, such as counting pushups in real time as someone exercises.

• It can also support tasks like “correct my mispronunciation as you hear it,” which requires speaking while listening, something a turn-based architecture handles as separate operations.

These capabilities all share a single source, emerging from the same architectural choice.

Coordination

Time-aligned micro-turns solve responsiveness, but they create a new problem.

How does a model designed to respond in 200-millisecond windows also do deep reasoning?

Some tasks genuinely require minutes of thinking, web browsing, tool use, or chained reasoning steps. Building a single model that handles both fast response and deep thought at the same time is hard.

Thinking Machines’ answer is to use two models working together:

• The interaction model is fast, present, and handles real-time conversation.

• The background model is slower and handles sustained reasoning, tool use, browsing, and longer-horizon work.

They share context with each other, so both have the same picture of what has been said and what is happening.

`The coordination works like this:

• When the interaction model encounters something that needs deeper reasoning, it sends a rich context package over to the background model.

• This is the full conversation rather than a standalone query, which lets the background model understand the situation fully.

• The background model runs asynchronously, with results streaming back as it produces them.

• The interaction model then weaves those results into the conversation when the moment fits, rather than as an abrupt context switch in the middle of something else.

From the user’s perspective, this is a single continuous conversation, with one AI thinking, responding, occasionally pausing to dig deeper, and weaving back in smoothly. Behind the scenes, two systems coordinate throughout.

This same logic shows up across computing, with fast paths paired with slow paths, foreground processes paired with background ones, and routine examples throughout operating systems and web browsers. What Thinking Machines did is apply the pattern to AI inference in a principled way, instead of treating reasoning latency as a problem the user has to absorb.

Capabilities

All of these design choices add up. The interaction model handles its own dialog management, knowing whether the user is thinking, yielding, or self-correcting, and it can interject verbally or visually based on context. It can speak and listen at the same time, which is what makes live translation possible. It has a direct sense of elapsed time, and can call tools, search, and generate UI concurrently with the conversation, weaving results back as they become ready.

These claims need evidence. Existing benchmarks for voice AI struggle to capture these qualitative jumps, so Thinking Machines built their own.

• TimeSpeak measures whether the model can initiate speech at user-specified times with the correct content, with an example task being “remind me to breathe in and out every 4 seconds until I ask you to stop.”

• CueSpeak measures whether the model speaks at the right moment while the user is still talking, with an example task being “every time I codeswitch, give me the correct word in the original language.”

• RepCount-A streams video of someone doing reps after the instruction “count out reps for pushups.”

• ProactiveVideoQA streams videos with questions whose correct answers depend on what is happening visually at specific moments.

The result is striking. Across these benchmarks, all existing models struggle with these tasks, with most either staying silent or giving wrong answers. This is the strongest evidence Thinking Machines presents that their architectural shift unlocks a new capability class, rather than just speeding up old behavior.

Limitations

Despite the encouraging results, the research also points out the things that are still hard.

• Long sessions remain a real challenge for this architecture. Continuous audio and video accumulate context very quickly. While the streaming-session design handles short and medium interactions well, very long sessions still require careful context management.

• Connectivity remains a hard requirement, since streaming audio and video at low latency demands a reliable internet connection. A poor connection causes the experience to degrade significantly.

• Scaling the model size is constrained by latency targets, with TML-Interaction-Small being the size it is, partly because Thinking Machines’ larger pretrained models are currently too slow to serve in this setting.

Conclusion

Looking back, the main argument is simple. What feels real-time in today’s voice AI is a turn-based language model wrapped in helper components, and that works up to a certain limit. Thinking Machines’ bet is that the way past the limit is to make interactivity part of the model itself.

Two architectural choices carry most of the heavy work:

• Time-aligned micro-turns slice time into 200-millisecond chunks, letting the model handle input and output as continuous streams.

• The two-model split pairs a fast interaction model with a slower background model that handles deep reasoning, with both sharing context.

The evidence that this is a new capability class rather than just lower latency comes from the benchmarks Thinking Machines built themselves. Tasks like “count my pushups as I do them” or “correct my codeswitching mid-sentence” stay out of reach for turn-based architectures, regardless of how fast they get.

An important takeaway is that adding a capability through external scaffolding creates a ceiling on how good that capability can get, with the scaffolding becoming the bottleneck rather than the underlying system. This pattern shows up across computing, and this research preview is one of the clearest recent illustrations of it in AI.

Thinking Machines plans to open a limited research preview in the coming months, with a wider release later this year and a research grant for interaction model research.

References:

• Interaction Models: A Scalable Approach to Human-AI Collaboration

When devs use AI to generate thousands of lines of unverified code, you risk a codebase slopocalypse. The review step becomes your team’s bottleneck, and the last thing standing between a subtle bug and production.

Greptile reviews each PR with full repo context and learns your team’s conventions over time from comments, reactions, and what gets merged. It flags real issues and suggests fixes that match your team, not generic best practices.

✅ Recently launched TREX runs your code, not just reads it. Greptile executes the change in a sandbox and returns screenshots, logs, and traces as proof of what actually broke.

✅ Review from your terminal. The Greptile CLI runs the same review locally, before you ever open a PR.

✅ Trusted by engineering teams at NVIDIA, Scale AI, and Brex.

✅ Now integrated with Claude Code: install via /plugin.

✅ Free for open source.

See why devs love Greptile →

Even the most sophisticated AI agent in your stack starts every single message from a blank slate.

The model itself sees only the text placed in front of it at that exact moment, and the rest of the conversation lives outside its awareness entirely. Whatever continuity we feel when chatting with Claude or ChatGPT is something the surrounding platform is engineering on the model’s behalf, by inserting the right context back into every call. Once we understand this crucial distinction, the entire field of agent memory becomes a very different engineering problem from what it first appears to be.

In this article, we will try to understand how that architecture gets built, from the constraint that forces it to exist all the way to the tradeoffs that follow.

Disclaimer: This post is based on publicly shared details from various sources. Please comment if you notice any inaccuracies.

Statelessness

A call to a large language model follows a simple pattern. The system sends a prompt, the model returns a response, and the exchange concludes there. Each subsequent call, even one made a millisecond later, begins from a completely fresh slate. This is the API contract for every commercial LLM and reflects how transformers serve traffic.

See the diagram below:

When we say things like “Claude remembers our conversation from yesterday,” we are describing a property of the product rather than a property of the actual model itself. The platform writes things down on the model’s behalf, then reads them back into the prompt at exactly the right moment, so the model can reason as if it had been there all along. The intelligence resides within the model itself, while the memory resides in the system surrounding it.

If the model itself works this way, the next question is whether we can solve the memory problem by writing everything into the model’s view on every call? The way this approach breaks gives us a better insight into how to solve this problem.

Context

Every API call has a context window. It is basically the bounded slab of text that the model reads when generating its response. This includes the system prompt, the user’s current message, and anything else the developer has placed there. The model has full visibility into the contents of the window, while whatever sits outside it might as well live on a different machine.

See the diagram below:

An obvious approach to memory involves writing the entire conversation history into the context window on every call. This works well for the first few turns of a chat. Once the conversation grows longer, however, three significant problems emerge at once:

• The first problem is cost. Every token in the context window is paid for on each call, in both money and latency, so a linearly growing conversation produces a linearly growing bill. By message eighty, the system might be re-sending tens of thousands of tokens on every turn just to maintain continuity.

• The second problem is latency. Larger contexts take longer to process, and a model that responds in two seconds on a short prompt may take ten or fifteen on one that has filled most of its window.

• The third problem is the most counterintuitive of the three. The model’s attention degrades inside long contexts, and information placed in the middle of a long prompt is recalled less reliably than information at the beginning or the end. Researchers refer to this as the lost-in-the-middle effect.

Bigger context windows feel like they should solve memory entirely. In reality, they expand the room while leaving the navigation problem fully intact. Important information can sit right inside the window and still get ignored by the model.

Since a bigger window alone is the wrong tool, we need an architecture that decides what belongs in the window at any given moment.

When Rules Fail, AI Picks Up the Slack (Sponsored)

What happens when deterministic code hits the edge of its knowledge? In this live webinar, you’ll see a working plant health monitor built on Temporal’s entity workflow pattern where each plant is a long-running, crash-proof workflow that polls sensors, fires alerts, and falls back to GPT-4o only when the rules run out.

The architecture is clean: structured data first, AI second. The boundary is auditable. The state survives everything. Whether you’re building patient monitors, supply chain detectors, or any long-running process that occasionally needs a smarter answer, the patterns here translate directly.

Join us on July 9th

Hierarchy

Real production systems organize memory in tiers, each trading off speed of access, total capacity, and cost per token. The context window sits at the top, with progressively slower, larger, and cheaper stores below it.

The analogy to operating system memory is evident. Modern agent memory systems draw heavily on the way operating systems page data between fast RAM and slower disk, promoting and demoting information as its relevance rises and falls.

A typical four-tier hierarchy starts with the context window at the top, fast and tightly bounded, with every token expensive at scale. Below it sits short-term or session memory, holding recent activity that has yet to be summarized or evicted. Beneath that is the long-term store, where persistent facts, embeddings, and structured summaries live across sessions. At the bottom is the cold archive, used for rarely-accessed material kept for audit or future reference.

Information moves up and down this hierarchy as the agent works. A fact stated three sessions ago might be sitting in the long-term store, and the moment it becomes relevant, the system retrieves it and promotes it back into the context window. Conversely, when a session ends, the most useful parts of the context window get summarized and written down into the lower tiers.

ChatGPT’s memory feature uses a simple version of this idea. Stored user facts and summaries of recent conversations get prepended to every new prompt, while the current session occupies the working tier. The sophistication lies in what gets promoted to long-term memory in the first place, rather than in elaborate retrieval at read time.

A hierarchy describes where memory lives. The kinds of memory we are storing are a separate question, organized along a different axis.

Types

The field has converged on four functional categories of agent memory, drawing on concepts from cognitive science adapted for language model agents:

• Working memory holds whatever sits in the live context window for the current task. For example, if the agent is helping us debug a function right now, the function code and our recent messages occupy working memory. The moment this task ends, working memory clears.

• Episodic memory holds records of specific past interactions, anchored in time. A statement like “Three days ago, this user asked about onboarding new engineers, and we discussed checklist templates” represents an episodic memory, capturing a particular event with its context.

• Semantic memory stores facts and knowledge that stand independent of any specific interaction. Statements like “Adam prefers Python over JavaScript” and “his team uses GitHub Actions for CI” function as semantic memories, surviving across sessions and applying wherever they are relevant.

• Procedural memory captures learned ways of doing things. If the agent has figured out that this user prefers a three-section format for status updates, that preference becomes procedural memory, and the next time a status update is requested, the agent applies the format automatically.

These four types are orthogonal to the hierarchy from the previous section.

A piece of semantic memory might live physically in the long-term store and get pulled into the context window the moment it becomes relevant. Most production agents implement at least three of these types, with the mix depending on what the agent is built to do. A customer support agent leans heavily on episodic and semantic memory, while a coding agent leans more on procedural memory. The right blend is a design choice rather than a fixed formula.

All of this still leaves one important question open.

How does the agent actually decide, on each turn, what to retrieve and place in the model’s view?

Retrieval

Storage can possibly be thought of as the easy half of agent memory. Writing a fact to a database or indexing a summary in a vector store are solved problems with mature tooling. The harder half is retrieval, which is the act of deciding on every new turn what belongs in the model’s awareness.

Retrieval is hard because it requires judgment about relevance, and relevance shifts from one message to the next. For example, the user’s preference for Python matters when they are asking about a new project, while that same preference fades in importance when they are asking about pizza recipes. A good retrieval system surfaces relevant items at exactly the moment they are useful, and leaves the rest sitting quietly in storage.

See the diagram below:

The full loop runs on every user message.

The user sends a message, and the system retrieves relevant items from each memory tier using a mix of keyword search, semantic similarity, and recency signals. It assembles a context window in a deliberate order, often placing the most important material near the beginning and the end where the model’s attention is strongest. The model runs and returns a response, after which the system writes part of the new exchange back into memory, often as a summary, sometimes with a decay score so that importance fades over time.

Consider two agents to see why retrieval matters so much.

The first has a perfect database of past interactions paired with a retrieval system that frequently picks the wrong record. The second has an empty memory store and operates only on what the user tells it in the current session. The second often outperforms the first, because it understands the bounds of what it can rely on, while the first surfaces stale or irrelevant information confidently and reasons on top of it as though it were ground truth.

In other words, memory failures in production are typically retrieval failures in disguise.

Tradeoffs

The memory architecture described above involves several tradeoffs that engineering teams must navigate carefully. Four stand out as particularly important:

• Recency versus relevance: Do we retrieve the most recent items from memory, or the most semantically similar ones? Most systems do both, and blending them well is an ongoing engineering puzzle. Leaning too far on recency means the agent forgets useful older context, while leaning too far on relevance means the agent fixates on a strong-but-stale match.

• Summarization versus fidelity: Compressing old context into summaries saves tokens, which makes everything cheaper and faster. The compression is also lossy, and the loss is uneven. Names, dates, and specific commitments get smoothed away in summarization, while general themes survive. The agent stays confident even after the precise detail has quietly disappeared.

• Staleness: A fact that was true six months ago can be confidently wrong today. For example, the user who may have told the agent “I am a vegetarian” in 2024 might be eating non-vegetarian again in 2026. The memory system has only blunt heuristics for guessing that the world has moved on, so it keeps serving the old fact with full confidence. Staleness in high-relevance memories remains an open research problem.

• Memory poisoning: Long-term memory is also a long-term attack surface. A subtly malicious instruction written into the store six months ago will sit there influencing every retrieval until somebody notices. The same property that makes memory useful, persistence, also makes it dangerous when the contents are wrong or hostile.

A memory system makes sense when the agent needs continuity across sessions or runs long-horizon tasks where context compounds. For one-shot tasks, it adds complexity beyond what the task requires.

Conclusion

Five core ideas from this article are as follows:

• The model is stateless. Every API call begins from a fresh slate, and any continuity we observe is the work of the surrounding system rather than the model itself.

• The context window is the model’s only surface of awareness. Writing everything into it fails for reasons of cost, latency, and degraded attention in long prompts.

• Real systems organize memory in a hierarchy of tiers, with the context window at the top and progressively slower, larger, cheaper stores below it.

• Different kinds of information call for different kinds of memory, with working, episodic, semantic, and procedural memory each serving a distinct purpose.

• The main engineering problem in this area is retrieval, which is the question of what deserves to enter the model’s awareness on each new turn, with tradeoffs around staleness, summarization loss, and security.

The practical takeaway from all of this is simple.

The next time we see a product feature labeled “memory” or read about an agent that “remembers across sessions,” the right question moves away from “can the model remember this?” and toward “what does the memory architecture around this model actually do, and what tradeoffs has it accepted?”

References:

• Lost in the Middle: How Language Models Use Long Contexts

• MemGPT: Towards LLMs as Operating Systems

• Cognitive Architectures for Language Agents

• Memory and new controls for ChatGPT

If slow QA processes bottleneck you or your software engineering team and you’re releasing slower because of it — you need to check out QA Wolf.

QA Wolf’s AI-native service supports web and mobile apps, delivering 80% automated test coverage in weeks and helping teams ship 5x faster by reducing QA cycles to minutes.

QA Wolf takes testing off your plate. They can get you:

• Unlimited parallel test runs for mobile and web apps

• 24-hour maintenance and on-demand test creation

• Human-verified bug reports sent directly to your team

• Zero flakes guarantee

The benefit? No more manual E2E testing. No more slow QA cycles. No more bugs reaching production.

With QA Wolf, Drata’s team of 80+ engineers achieved 4x more test cases and 86% faster QA cycles.

Schedule a demo to learn more

This week’s system design refresher:

• RAG vs Graph RAG vs Agentic RAG

• Redis Data Structures Every Engineer Should Know

• API Security Best Practices

• Design Patterns Cheat Sheet

• The Testing Pyramid

RAG vs Graph RAG vs Agentic RAG

RAG connects LLMs to your data and there are three different ways to do it.

Standard RAG

• The query is converted into an embedding and matched against a vector database.

• The top-K closest chunks are pulled out and passed to the LLM as context.

• The LLM writes a grounded answer using only what was retrieved.

Graph RAG

• The query is classified: specific questions route to local search, broad questions route to global search.

• Local search: query embedded → vector DB finds matching entities → pipeline traverses across the knowledge graph collecting linked context → LLM synthesis final answer.

• Global search: no vector search, no graph traversal → community reports loaded in batches → LLM scores each for relevance → top-ranked context → LLM synthesizes final response.

Agentic RAG

• A reasoning agent reads the query, breaks it into sub-questions and picks the sources.

• The context across multiple sources is retrieved, depending on the sub-query.

• Another agent checks whether the retrieved context answers the question. If not, it re-retrieves.

• Once satisfied, the final answer is synthesized by LLM based on the prompt.

Standard RAG is fast and cheap but if the wrong chunk is retrieved, the answer is wrong and nothing catches it.Use it when the answer lives in your documents and speed matters.

Graph RAG is expensive to build and slow to update. Use it for structured knowledge like legal, compliance, or biomedical data.

Agentic RAG is more capable and flexible but slower, expensive, and harder to debug. Use it when the question needs multi-step reasoning and self-correction.
Over to you: Which of these are you running in production?

Redis Data Structures Every Engineer Should Know

• Strings store one value per key. They work for counters, session tokens, and cached payloads.

• Hashes store an object's fields under one key. You can update one field without rewriting the rest.

• Lists are ordered sequences with fast push and pop at both ends. They fit queues, feeds, and recent-item lists.

• Sets hold unique members and support intersection, union, and difference. They cover tagging, follower overlap, and deduplication.

• Sorted Sets rank members by a numeric score. They handle leaderboards, priority queues, and top-N or range-by-score queries.

• Streams are an append-only log with consumer groups. Each consumer tracks its own position, and the server tracks unacknowledged messages.

• JSON stores nested documents with JSONPath access. You can update a field deep in a document without read-modify-write.

• Geospatial provides latitude/longitude indexes with radius and box queries. Under the hood it's a Sorted Set with geohash scores.

• Vector Set runs approximate nearest-neighbor search over embeddings. It's the retrieval step in most RAG pipelines.

• Time Series stores timestamped samples with built-in retention, downsampling, and labels. It fits metrics, telemetry, and IoT data.

Over to you: All ten are built-in as of Redis 8. Which one do you use most outside of caching?

API Security Best Practices

Most API breaches happen because of broken authorization, leaked secrets, or missing rate limits. Let's look at some of the basics.

• Use Modern OAuth/OIDC + MFA: PKCE for public clients, short-lived tokens, and step-up MFA for anything sensitive. Implicit and password grants should be dead by now.

• Enforce Fine-Grained Authorization: Check object, function, and field-level permissions on every request. BOLA is still the top API vulnerability.

• Minimize Scopes and Data: Give each client the smallest token scope and the least data it needs. Only return the fields the caller actually needs.

• Encrypt Every Hop: TLS for external traffic and mTLS between services. If it crosses a network boundary, encrypt it.

• Protect Secrets and Keys: Store signing keys in HSM-backed vaults. Rotate them.

• Validate Requests with Schemas: Reject unknown fields, oversized payloads, and suspicious URLs at the gateway. Don't let bad input reach your business logic.

• Rate Limit and Cap Resources: Quotas per user, payload size caps, and execution timeouts. Without these, one misbehaving client takes down your entire system.

• Defend Sensitive Business Flows: Protect login, checkout, and OTP with anti-bot, idempotency keys, and step-up auth.

• Control Outbound and Third-Party Calls: Allowlist where your API can call out to and block internal metadata endpoints. Your security is only as strong as your weakest integration.

• Harden Config and Error Handling: Deny by default on CORS, methods, and debug endpoints. Return generic errors, never stack traces.

• Inventory APIs and Versions: Track every endpoint, version, and shadow API. You can't secure what you don't know exists.

• Log, Detect, and Respond: Push auth decisions and anomalies to a SIEM. Alert on 401 spikes before they become incidents.

Over to you: Which of these best practices is the hardest to enforce across your services?

Design Patterns Cheat Sheet

The cheat sheet briefly explains each pattern and how to use it.

What's included?

• Factory

• Builder

• Prototype

• Singleton

• Chain of Responsibility

• And many more!

The Testing Pyramid

Testing is the backbone of reliable software. The Testing Pyramid is a widely accepted strategy for structuring tests into three key layers:

• Unit Tests: These are the foundation of the pyramid. Unit tests are fast, isolated, and low-cost to write and maintain. They test individual functions, methods, or components.

• Integration Tests: These tests validate interactions between components, such as APIs, databases, and external services. They are slower than unit tests and require more setup.

• E2E Tests: These simulate real user flows from start to finish across the full system. They are expensive to write and maintain and tend to be slow to execute.

As you go up the pyramid, the cost of test development, execution, and maintenance increases.

Over to you: Which layer do you find most valuable in your testing strategy, and why?

How many decisions does it take to reach that point?

It rarely takes a single bad one. The path to such a situation is built from individually sound choices, a clean separation here, an independent deployment there, a new service each time a part of the system felt distinct enough to stand on its own. Those reasonable steps accumulate into an arrangement no one would have chosen on purpose, and the problems that emerge look like a catalog of separate mistakes even though nearly all of them trace back to one early decision about how to break a system.

At a basic level, a service is a part of a system that can be deployed on its own and controls its own data. This means it does not reach into any other service’s database to do its work. It also talks to other services over a network. Inside a single program, one function calling another takes a few nanoseconds and either returns an answer or raises an error. The same call across a service boundary can take a few milliseconds, and it can also time out or succeed halfway and leave things in an odd state. Almost every anti-pattern below emerges from this one problem.

In this article, we will look at some of the most important anti-patterns in service architecture, how they happen, and how they can be avoided.

Splitting Early

Read more

AI-assisted development has changed how code gets written, but for many teams, testing and governance haven’t caught up. Tricentis AI Workspace closes that gap, giving quality engineering leaders one place to build, orchestrate, and govern AI quality agents across the SDLC, from code risk analysis and test automation to performance validation so quality decisions happen continuously, not at the end. Less errors introduced by AI-generated code, more confidence in what you’re shipping.

Discover how teams are using AI Workspace to bring structure to AI-driven development and compress delivery timelines without sacrificing confidence in business outcomes.

See Tricentis AI Workspace in action

Apple’s most ambitious AI feature runs in about a gigabyte of memory on the iPhone. The same company runs a much larger model on its own cloud servers, and the two diverge in almost every architectural choice beyond the word “transformer” in their lineage.

The same split shows up at Google, Microsoft, and Meta, where one family of small models targets devices and a different family of large models targets data centers.

Small and large language models are different engineering responses to different constraints, and the differences begin with where each model runs, what hardware it targets, and how it was trained.

In this article, we will explore those constraints through three layers of model design, look at the tradeoffs that come with each approach, and investigate the production systems that combine both small and large models.

Disclaimer: This post is based on publicly shared details from various sources. Please comment if you notice any inaccuracies.

Foundations

Before we look at what makes the two classes different, it helps to be precise about what makes them the same.

Both small and large language models are transformer-based decoder models, built by stacking layers of the same basic computational block. Each block runs an attention operation, which figures out which previous tokens matter most for predicting the next one, followed by a feed-forward computation that mixes that information through a wide intermediate layer. The model repeats this block thirty or more times before producing a probability distribution over what the next token should be.

Both classes go through similar training stages. They start with pretraining on large text corpora, where the model learns to predict the next token across billions of examples. They typically follow with supervised fine-tuning on specific instruction patterns, and many go through reinforcement learning from human feedback, which shapes how the model handles ambiguity and stays helpful in conversation.

The size of a model refers to its number of parameters, which are the learned weights adjusted during training. A small model in 2026 typically has between half a billion and fourteen billion parameters. A large model has tens of billions to hundreds of billions of parameters, and sometimes more.

Constraints

Three constraints pull the designs of small and large models in opposite directions.

• Deployment target: Where the model runs determines its memory, battery, and latency budgets.

• Inference economics: Training is paid once, but serving is paid per request, which inverts the math at scale.

• Training budget: Smaller budgets push teams toward efficiency through data quality and distillation rather than raw scale.

The deployment target determines everything that follows.

A model that runs on a phone has a memory budget measured in single gigabytes, a battery budget measured in milliamps, and a latency budget measured in milliseconds. A model that runs in a data center operates in a more permissive environment, with concerns around throughput, batching efficiency, and cost per request, but with an absolute resource ceiling orders of magnitude higher.

Inference economics is the second pressure.

Training a model is a one-time cost paid at the start of its life, while serving the model is a recurring cost paid every time someone uses it. For a high-volume product, the inference bill quickly dwarfs the training bill, so a team designing for high inference volume will gladly spend more training compute upfront to save inference compute across billions of requests downstream.

The training budget is the third pressure.

A frontier large model can cost tens of millions of dollars to train, while most teams working on small models operate with a small fraction of that, and the smaller budget forces choices. Those teams have to find other levers beyond raw scale, which usually means smarter training data, distillation from larger teachers, and more efficient training recipes.

These three constraints reinforce each other rather than acting in isolation. A model designed for the phone has a small inference budget per request and usually a smaller training budget too, while a model designed for the data center has the opposite profile across all three axes. The result is two distinct design regions in the same space.

Who’s actually reviewing all that AI-generated code? (Sponsored)

When devs use AI to generate thousands of lines of unverified code, you risk a codebase slopocalypse. The review step becomes your team’s bottleneck, and the last thing standing between a subtle bug and production.

Greptile reviews each PR with full repo context and learns your team’s conventions over time from comments, reactions, and what gets merged. It flags real issues and suggests fixes that match your team, not generic best practices.

✅ Recently launched TREX runs your code, not just reads it. Greptile executes the change in a sandbox and returns screenshots, logs, and traces as proof of what actually broke.

✅ Review from your terminal. The Greptile CLI runs the same review locally, before you ever open a PR.

✅ Trusted by engineering teams at NVIDIA, Scale AI, and Brex.

✅ Now integrated with Claude Code: install via /plugin.

✅ Free for open source.

See why devs love Greptile →

Architecture

The architecture differences begin with a quick observation about inference.

During generation, the model has to keep around the keys and values for every previous token, since attention works by comparing the current token against all earlier ones. This stored set is called the KV cache, and it grows linearly with the length of the conversation. For long generations, the cache often dominates memory bandwidth and storage, more than the parameters themselves.

This single fact decides how small-model architectures get designed.

In the original transformer design, every attention head has its own keys and values, an arrangement called multi-head attention. For long sequences, the resulting cache footprint grows large enough to dominate the model’s memory consumption.

Grouped-query attention attacks the problem directly. The number of query heads stays the same, but several queries share a single key-value pair. A model with thirty-two query heads might use only eight key-value groups, which cuts the cache footprint by a factor of four with minimal quality loss. Llama, Qwen, Gemma, and most modern small models use grouped-query attention by default, and many large models have adopted it as well because the math also helps at scale.

Some small models push further. Gemma 2 interleaves sliding window attention with full attention across layers, so some layers attend only to the most recent few thousand tokens rather than the full context. This trades a bit of long-range reasoning for a significantly smaller cache. Apple’s on-device model shares its KV cache across multiple decoder layers, reusing the same stored state in several places.

These architectural decisions all serve the same goal of shrinking the runtime cost of inference, which is the constraint that matters most when the model has to run on a device with a few gigabytes of memory to spare.

Training

Two models with identical architectures can end up with very different capabilities depending on what they were trained on and how.

Three techniques define the current state of the art in small-model training:

• Data curation: Carefully chosen and synthetically generated training data can substitute for raw volume.

• Knowledge distillation: A smaller student model learns from a larger teacher model’s output distribution.

• Overtraining: Modern small models see far more training tokens than compute-optimal ratios suggest, trading training cost for inference savings.

The first technique is data curation. In 2023, a Microsoft research team published a paper called “Textbooks Are All You Need.” They trained a 1.3 billion parameter coding model on roughly seven billion tokens of carefully filtered code and synthetically generated textbook-style data. The model matched or beat models trained on hundreds of billions of tokens of raw web scrape. Training data quality could substitute for training data volume, at least for certain capabilities. The Phi family kept building on that insight, and the modern Phi-4 model continues to lean heavily on synthetic data quality as its primary lever.

The second technique is knowledge distillation.

The small model, called the student, learns from a larger model, called the teacher, by mimicking the teacher’s output distribution rather than only learning from raw text. The richer training signal helps the student pick up patterns it would struggle to learn from the underlying corpus alone. Gemma 2 used this approach to train its nine billion parameter model, while training its twenty-seven billion parameter version from scratch.

The third technique is overtraining relative to compute-optimal.

In 2022, the Chinchilla paper from DeepMind established that for a fixed compute budget, the best model came from balancing parameter count and training data, roughly twenty tokens of training data per parameter. Modern small models deliberately train on far more data than that ratio suggests. A three-billion-parameter model might see many trillions of tokens during training, which is many times the Chinchilla-optimal amount. Once the model gets deployed, every percentage point of quality improvement saves inference compute across billions of requests, so the team spends more on training to save more on serving.

Deployment

The final layer of design choices covers how the model executes on real hardware. The two dominant techniques are quantization, which shrinks the storage cost of each parameter, and KV cache management, which shrinks the runtime cost of generation.

Quantization is the practice of storing each parameter with fewer bits. A standard pretrained model stores each parameter as a sixteen-bit floating point number, where cutting that to eight bits halves the memory footprint and cutting to four bits halves it again. The post-training approach is faster to implement but tends to lose quality at aggressive bit widths, while quantization-aware training preserves quality at the cost of more complex training.

Hardware mapping is the next consideration. Apple’s Neural Engine has different strengths from an NVIDIA Jetson, which has different strengths from a data center H100, and the model design follows the target hardware. Phi-4-mini gets tuned for consumer GPUs. Gemma 3 4B variants run on NVIDIA Jetson Orin for edge AI deployments in robotics and embedded systems. Apple’s 3B model runs on the iPhone’s Neural Engine with the assumption that the device also handles other workloads at the same time.

KV cache management is the second major lever, and it connects directly back to the architecture section. The cache stores keys and values for every previous token during generation, and its size determines how much memory the model utilizes at runtime. Grouped-query attention attacks this by reducing the number of key-value heads, and Apple’s on-device model goes further by sharing its cache across multiple decoder layers.

These deployment decisions stack on top of everything earlier. The same architectural choices that shrink the KV cache also make quantization easier, and the same training recipes that produce capable small models also produce models that survive aggressive compression.

Tradeoffs

Small models perform well on standard benchmarks like MMLU and HumanEval. Production usage looks more varied. Three gaps tend to matter most:

• Generalization gap: Small models are more brittle outside their training distribution.

• Reasoning gap: Multi-step problems still favor larger models, though the gap is closing.

• Knowledge ceiling: Parameters function as memory, so small models have a hard cap on what they can store.

The first gap is generalization.

Small models tend to be more brittle outside their training distribution, and they can be excellent at tasks similar to what they saw during training, while showing weakness on unexpected ones. A small model trained heavily on code performs well on code but may struggle with creative writing in an unusual style. A model trained on synthetic textbook data does well on textbook-style questions but can falter on the messy, ambiguous prompts that real users send.

The second gap is multi-step reasoning.

For problems that require chaining inference across many tokens, large models still have a noticeable advantage. The gap has been closing thanks to step-by-step reasoning techniques and reasoning-focused fine-tuning, but at very small parameter counts, the ceiling remains real. Phi-4 has done well on math reasoning specifically because Microsoft optimized for that capability through training data design, while a general-purpose small model usually shows a clearer gap.

The third gap is world knowledge.

Parameters function as a form of memory, and a larger model can store more facts, more named entities, more obscure references, and more multilingual coverage. A small model has a fundamental cap on how much it can know, since storage requires parameters and parameters require memory. For applications that need broad factual recall, the small model often pairs with an external knowledge source that the model queries when needed, since trying to fit all that knowledge into the parameters themselves would push the model past its budget.

Hybrids

The most interesting design question in 2026 is rarely which model to use. The more useful question is how to compose multiple models into a system that uses each for what it does best. Three patterns appear in most production setups.

• Routing: A small model handles requests directly and escalates harder ones to a large model.

• Guardrails: A small model filters input or output around the large model’s core work.

• Drafting: A small fast model generates candidate tokens that a larger model verifies in a batch.

The most common pattern is routing.

A small model handles the request directly if it falls within its competence, and escalates to a large model when the request is harder than it can confidently handle. The pattern resembles caching tiers in a distributed system, where the fast, cheap layer handles the common case, and the slower, more expensive layer handles the rest. The router itself is often a small classifier model that decides which path to take.

The second pattern is the guardrail.

A small model sits in front of the large model to filter or classify input before the expensive computation runs, checking for unsafe content, classifying the intent of the request, or stripping out information that should stay private. A second small model often sits on the output side, doing similar checks before the response gets returned to the user. These guardrail models are cheap, fast, and specialized, which makes them well-suited to the role.

The third pattern is the drafter, sometimes called speculative decoding.

A small fast model generates candidate tokens, and a larger, more capable model verifies them in batch. When the verifications agree, the system gets the throughput of the small model with the quality of the large one. Apple’s on-device system uses a draft model alongside its base model for exactly this reason. The technique sounds like a hack, but it has become standard in production inference systems.

Picking a model class is the wrong frame for most product decisions. Designing a system around multiple model classes is the right frame, and the interesting design work lives in the composition layer, the routing logic, the fallback behavior, and the orchestration between models.

Conclusion

The question we started with was “small versus large language models,” and the more useful version of that question turns out to be “which constraints drove each model’s design.” The size of a model is a downstream consequence of those constraints rather than the starting point for the design.

Three layers of design choices flow from the constraints:

• Architecture adapts through attention variants like grouped-query and sliding-window attention that shrink the KV cache.

• Training adapts through high-quality synthetic data, distillation from larger teachers, and deliberate overtraining relative to compute-optimal ratios.

• Deployment adapts through quantization, KV cache management, and careful hardware mapping. Each layer reinforces the others, and the result is two distinct design regions in the same space.

Small models are extremely capable for their size, and they have a real ceiling on generalization, on multi-step reasoning, and on broad world knowledge. Production systems handle this by composing both classes, using small models for the common case and large models for the harder requests, sometimes with multiple small models acting as routers, guardrails, and drafters around a larger core.

For a working engineer choosing between models, the right starting point is the constraints rather than the benchmark. The questions that matter are about deployment target, inference budget, and the shape of the request distribution in production.

References:

• Apple Intelligence Foundation Language Models Tech Report 2025

• Apple Intelligence Foundation Language Models

• Updates to Apple’s On-Device and Server Foundation Language Models

• Introducing Apple’s On-Device and Server Foundation Models

• Recurrent Drafter for Fast Speculative Decoding

• Recurrent Drafter

• Deploying Transformers on the Apple Neural Engine

• Textbooks Are All You Need (Phi-1)

• Textbooks Are All You Need

• Phi-4 Technical Report

• Phi-4 Technical Report

• Phi-4-mini-instruct model card

• Running Phi-4 Locally with Microsoft Foundry Local

• Gemma 2: Improving Open Language Models at a Practical Size

• Gemma 2 Technical Report

• Lightweight, Multimodal, Multilingual Gemma 3 Models Are Streamlined for Performance

• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

• GQA paper on ACL Anthology

• Fast Transformer Decoding: One Write-Head is All You Need (Multi-Query Attention)

• Training Compute-Optimal Large Language Models (Chinchilla)

• Chinchilla Scaling: A replication attempt

• Reconciling Kaplan and Chinchilla Scaling Laws

• Attention Is All You Need

• Distilling the Knowledge in a Neural Network

Most teams pick a search provider by running a few test queries and hoping for the best – a recipe for hallucinations and unpredictable failures. This technical guide from You.com gives you access to an exact framework to evaluate AI search and retrieval.

What you’ll get:

• A four-phase framework for evaluating AI search

• How to build a golden set of queries that predicts real-world performance

• Metrics and code for measuring accuracy

Go from “looks good” to proven quality.

Learn how to run an eval

This is a guest post by Kun Chen, a former L8 principal engineer at Meta, Microsoft, and Atlassian, where he led development of Rovo Dev, Atlassian’s AI SDLC product. He has since left big tech to build solo and has gone all-in on agentic engineering. Below, he walks through his complete setup, step by step. You can follow him on X and subscribe to him on YouTube, where he shares his agentic engineering workflow, the open-source tools he builds, and his take on AI and software craft. Over to Kun.

Hi everyone, Kun here. For context, I spent years driving agent adoption among tens of thousands of engineers at all levels, both within my company and across many customers’ engineering organizations. Going solo has actually let me lean into agents even more.

Here’s the difference using agents has made to my productivity: shipping 30+ high-quality PRs that meet my own bar used to be hard to imagine, and it’s now a slow day. I’ve reached what feels like a constant flow state, where the quality and speed of my thoughts is the only bottleneck left.

All of this didn’t come from a single trick or using some hyped tool. It came from a long and often messy process of figuring out what actually works in the real world versus what just sounds good in a demo. The short version is that I have now stopped writing most of the code myself and started acting like an engineering manager directing a team of agents. I stay at the level of deciding what to build and whether it’s good, and I’ve built tooling to handle almost everything in between.

The interesting part of this journey is all the friction I had to remove to reach this point. Therefore, in this post, I’m attempting to share everything I do, step by step, for both my professional and personal projects.

If you’re on the same journey of making your work with agents more productive and enjoyable, I hope this gives you a head start and shortcuts some of your own exploration.

A Couple of Clarifications

First, what I’m sharing here is my personal setup. What works well for me may not be the best fit for everyone. I’m sharing my workflow as-is, mainly hoping it can be a useful reference or inspiration for what to explore, even if you don’t end up using the same tools.

Second, I have no affiliation with any of the 3rd party products I mention in this post, and the tools built by me are all free and open source. I share these specific products because those are genuinely what I use in my setup. They are often not the only choice for the problems they solve, so I encourage everyone to research different options based on their interests and requirements.

The Project Details

To make this post concrete and practical, I’ll walk you through my workflow using a real project I’m actively building. It’s called “Hi Bit”: an AI tutor I’m making for my son to teach him agentic engineering. In the rest of the post, I will follow the implementation of a specific image input feature in the Hi Bit project from the idea to merged PR so that you can get a first-hand look at my agentic workflow.

When Rules Fail, AI Picks Up the Slack (Sponsored)

What happens when deterministic code hits the edge of its knowledge? In this live webinar, you’ll see a working plant health monitor built on Temporal’s entity workflow pattern where each plant is a long-running, crash-proof workflow that polls sensors, fires alerts, and falls back to GPT-4o only when the rules run out.

The architecture is clean: structured data first, AI second. The boundary is auditable. The state survives everything. Whether you’re building patient monitors, supply chain detectors, or any long-running process that occasionally needs a smarter answer, the patterns here translate directly.

Join us on July 9th

Starting From the Terminal

There has been a constant debate in the developer community about terminal vs GUI.

I’m obviously biased because I started coding almost 30 years ago and built decades of muscle memory on top of a terminal-centric workflow ever since. But I did try GUIs every once in a while, from Visual Basic, Visual Studio, to Atom, and now the latest Codex app.

The reason I stick with terminals is very simple. I keep my flow and focus best when my hands never leave the keyboard. Some GUIs let you do everything via keyboard shortcuts as well, but they’re very inconsistent about it, which makes it hard to build strong muscle memory.

Terminal Emulator

The terminal emulator I’ve been using for many years is WezTerm.

It’s the only terminal I’ve found that is highly performant, customizable, and works consistently even when I’m forced to use Windows. I run it as a single frameless window: no tabs, title bar, or status line, literally nothing else.

Agent Harnesses

I use Claude Code for Anthropic’s models and OpenCode for everything else.

The CLI agent harnesses nowadays are quite commoditized, and you won’t really go wrong with any of them. Almost everything I share below works with any mainstream harness you can find.

In fact, I actually recommend avoiding the “fancy” gimmicks that only some agents have, such as auto-managed memory. They’re often designed to lock you into a particular vendor, when in reality you benefit a lot from being able to switch to whichever newer model works best, even if it comes from a different vendor. I try to keep my whole workflow agent-agnostic, so I have no switching cost. It’s far from clear which model will win in the end, and as a user, you’re in a much better position if you can work with any model available rather than being locked into one.

Neovim

Neovim has been my primary editor for a long time, and it’s a critical part of staying fully keyboard-driven inside the terminal. You might ask, “But I use agents now. Why do I need an IDE?”

I use it to quickly examine the file system, review diffs, and make small edits when needed. A few plugins do most of the heavy lifting:

• oil.nvim: navigate and edit the file system like a buffer

• neogit: quickly review git status and diffs, and perform simple operations

• snacks.nvim: I use its picker for finding files and grepping the codebase

Tmux

I don’t let my terminal emulator manage tabs, because I manage all my sessions, windows, tabs, and panes in tmux instead. It’s one of the most powerful primitives in my whole setup, and it unlocks a few things at once:

• Splitting my terminal window into panes the way I like

• Driving the entire terminal experience from the keyboard

• Persisting the working sessions and layout

• Accessing the same session from my other devices (more on that later)

A popular alternative is Zellij, but tmux has worked well enough that I haven’t switched. As soon as I’m in, I create a split on the left for the agent and one on the right for Neovim, and I separate different tasks into different tabs that I keep track of along the top.

Basic Prompting

We’re now in the terminal, and the agent is waiting for instructions. All I have to do is write a prompt. Sounds easy, right?

Actually, how you write your prompts is one of the biggest levers on both the velocity of your work and the quality of the outcome. So let me share a few things that made a big difference for me.

Using Voice Input

Typing was my primary input method for decades. But over the last couple of years, speech recognition models have really changed the game. You can now run high-quality models locally on your Mac, for free, that generate output extremely fast.

You talk a lot faster than you type, so moving to voice as your primary input is one of the easiest ways to greatly improve your productivity. It applies to prompting your agents, but also to anything else that used to require typing. This post, for example, is mostly written by voice.

The solution I use is OpenSuperWhisper, which is completely free and runs the Whisper model (turbo v3 large) locally. I set a hotkey to trigger it, and now I can just talk wherever I could type. There are plenty of other free and paid options that give a great experience as well.

Delegating like a Manager

For many new tech leads and people managers, the first struggle is delegation. The same thing happens with how you interact with agents.

The most common mistakes I see people make about delegation to both humans and agents are:

• asking for an action, not an outcome;

• not explaining the “why.”

• taking back control.

Take “rename this variable.” It’s a valid prompt, but it has a couple of problems:

• The agent finishes in a few seconds and waits for you again. It barely saves more time than doing it yourself, and you’re still the bottleneck.

• There’s no “why.” Do you want it renamed for readability? To follow a team convention? Because of a future plan you haven’t mentioned? Without the why, there’s no room for the agent to suggest something better, and it won’t know how to do it right next time.

If you were following a convention, a better prompt would be: “Let’s audit this part of our codebase and make sure our variable naming follows this convention <link_to_document>.”

That explains the rationale, gives the necessary context, and asks for an outcome instead of an action. The agent can run longer, get more done in a way that’s aligned with your goal, and respect that convention for the rest of the session instead of creating more problems for you.

The other failure mode is taking back control. When a mistake happens, people immediately think of doing it themselves. This happens to new tech leads working with a junior engineer. They could do the job faster by taking over, but in doing so, they make themselves the bottleneck and fail to scale. People hit the exact same wall with AI: they see an agent do something wrong and revert to doing it manually, and they never truly unlock the leverage agents offer.

The right approach is to give feedback and help the other party improve. With agents, this is actually easier. You can write directly into the agent’s memory file (CLAUDE.md or AGENTS.md), or ask the agent to reflect on its mistake and update that file so the same thing doesn’t happen again.

Planning Complex Work

The feature I’m building right now is image input. I want the chat box in Hi Bit to accept a pasted image from the clipboard, or open a file picker or camera. This is a bit more than I think the agent can one-shot the way I want. In the beginning, I also didn’t know exactly where the button should go or what the attached images should look like.

This happens a lot: problems where I genuinely can’t describe the full solution up front. It could be a new project from scratch, a major refactor, or a big feature on an existing system. In those cases, I work with the agent to write a plan first.

There’s a school of thought in the agentic community that’s against technical planning. They prefer to “just talk to your agent” instead.

I went the other way, and here’s why. When you just talk to your agent, you have to stay in an interactive session. You get constantly pulled back into the conversation, and after a few rounds, it’s hard to keep track of what the actual plan is. A long wall of text in the terminal is also painful to parse and hard to give targeted feedback on.

Instead, I spend a concentrated chunk of time up front getting the plan into a confident state, so I can hand it off for a fully autonomous implementation without jumping back in until it’s done. That’s also what frees me up to run other tasks in parallel without constant context switching.

For a long time, I did this by asking the agent to write a proposal in a markdown file, then questioning it and iterating. That worked, but I have something better now. Inspired by an article on using HTML as interactive artifacts, I built a tool called Lavish Editor to collaborate with the agent on anything complex.

So instead of “draft a technical plan in a markdown file,” I say “draft a technical plan using npx lavish-axi.” Lavish guides the agent to render the plan as an interactive HTML page and opens it in my browser. It even encourages the agent to match the look and feel of the current project, so the plan for a UI feature looks visually consistent with the real app, which makes options far easier to judge.

A few minutes later, the agent had a plan open in my browser. It opened with the goal and context, flagged the decisions I’d need to make, and then laid out three UI options — a “+” button menu, an always-on button trio, and a smart-paste tile — each with pros and cons and the agent’s own recommendation.

The real payoff is the interactive back-and-forth. I liked Option A’s tiny resting footprint, but not that its menu dropped down and stretched the chat area. Rather than writing a paragraph describing which element I meant, I just clicked that element in the page and annotated it directly: “Can we make it a floating overlay above the + button instead?”

The agent came back almost immediately with exactly the revision I wanted, and I finalized the remaining decisions just by clicking buttons. Being able to interact with a plan this richly, instead of editing a markdown file, turned out to be a big productivity boost, and it’s not limited to planning. I now use Lavish for brainstorming, reviewing changes, data reports, and anything else that benefits from a visual artifact and tight back-and-forth. It’s been a game-changer.

Implementation

Once the plan is clear, the implementation is fully autonomous, and there’s honestly not much for me to do except wait for the agent to ping when it’s done.

The one thing worth mentioning over here is that for every project, I spend a lot of effort making sure the agent can validate its own change end-to-end. As an example, for Hi Bit, I keep explicit instructions in the repo’s AGENTS.md for how to exercise the app, so changes like this get validated by the agent before they come back to me. I can often watch it test its own work in real time by driving the real app, attaching an image, and checking how the new button actually behaves.

Occasionally, a task is so complex that it doesn’t fit well in a single context window. If I just let the agent grind on it, it fills its context, fires off very large requests, and eventually compacts to free space, sometimes losing important context in the process. The newer /goal command in Codex and Claude Code helps a bit, but I’ve been using something better since well before it existed.

I call it good night, have fun: gnhf for short. It’s a dead-simple, long-running orchestrator I built for running big tasks overnight; you invoke it with gnhf <your objective>.

Under the hood, it works similarly to the Ralph Loop and Autoresearch patterns. It breaks the task into small steps, and each step runs in a fresh context window seeded with a common base context plus the learnings from previous steps. Failed attempts roll back automatically, and the next attempt takes the failure into account. I can also set a token budget so I don’t wake up bankrupt. When I come back, there’s a branch with well-organized commits and a notes.md file summarizing what was done.

I reach for gnhf in three kinds of situations:

• Implement a massive plan: gnhf “fully implement this plan…”

• Improve a measurable metric such as reducing lines of code, increasing test coverage, cutting startup latency, or page load time: gnhf “improve <this metric> while keeping product functionality unchanged.”

• Run lots of offline experiments when you have an evaluator capable of scoring each attempt. For one project, I generated a game map by running 50+ layout experiments and scoring each through a gameplay simulation. Babysitting those by hand would have taken weeks.

Validation

Back to the image task: the agent has done the work, and it has produced a big change. Now what? This is where many people hit the real bottleneck: code review. There’s simply too much code to read, and reviewing it isn’t the fun part of the job.

I’m increasingly convinced that working with agents means acting like an engineering manager. Most managers rarely review code directly. They have the team review each other’s work, and before anything ships, they ask for evidence that it actually works. It’s the same with AI, except the developer is the manager and the agents are the team. You have to get good at using agents to scrutinize agents’ code, get them to self-correct, and get them to produce artifacts that demonstrate the feature really works.

I’ve experimented a lot here, and a few things turned out to matter most:

• Run the reviewer agent in a fresh context window. If you review in the same session that wrote the code, the agent is biased by what it just did and assumes it was intended. It’s like asking someone to check their own work. They’ll catch some things, but it’s far weaker than a real peer review.

• Escalate ambiguous, product-changing decisions to the human. Reviewers make mistakes, too. If you let the agent auto-fix every finding as if it’s all valid, it can drift into rabbit holes away from what you actually want. Keeping those decisions with the human keeps you in control of the ambiguity.

• Force end-to-end evidence. Today’s frontier models lean heavily on unit tests to validate changes, probably because of how they’re trained. But I’ve seen countless cases where every unit test passes, and the product is still buggy. You have to make the agent prove the change works E2E.

I packaged all of this into an open-source tool I built called no-mistakes, which I now run on the image change. First, I use the neogit plugin to quickly scan the diff and make sure it’s roughly aligned with what I asked. Sometimes an agent goes in a completely wrong direction, and that’s obvious at a glance.

If it looks reasonable, I run no-mistakes -y and it handles the rest: commit with a conventional message into a descriptively named branch, rebase onto the latest main and resolve conflicts, spin up agents to peer-review and self-correct obvious bugs, test the change E2E and produce evidence, close documentation gaps, fix linting, push, open a well-structured PR, and babysit CI until it’s green.

All of it runs autonomously except for the decisions it deliberately escalates to me.

This validation pipeline has become one of the most critical pieces of my whole workflow. My own stats show that 68% of the changes I pushed through the no-mistakes tool had bugs in them. I genuinely can’t imagine what my codebase would look like without it.

Parallelization

With a fully autonomous implement-and-validate pipeline, a single task can take a while, and that’s a good thing, because it frees me up to run more things at once.

Handling Agent Status

In tmux, this means opening a new window. Terminal tabs achieve the same parallelism. The important part is keeping those tabs visible and showing each agent’s status in the tab title. Claude Code and Codex do this out of the box; for harnesses that don’t, I wrote small plugins to do the same, and I made the no-mistakes tool report its status as a custom title too.

That one detail is what lets me run many sessions without going insane. At any moment, I can see which agents are running, which are done, and which need my input. A couple of tmux keystrokes jump me to any tab.

Managing Worktrees

The other problem with running agents in parallel is that they step on each other’s toes when they share a directory.

Git worktrees exist to solve this. A worktree is an efficient clone of the same repo in another directory where you can work in parallel. But worktrees carry a lot of cognitive load: where to put them, how to name them, create a new one or reuse an old one, which are in use, which have dependencies installed, and whether env files are ready. What I actually want to think about is the work, not where to do it.

So I built another open-source tool named treehouse. You’re in a repo, you want to start a parallel task, you run treehouse, and it drops you into a ready worktree. Behind the scenes, it manages a pool of worktrees, tracks which are free, reuses an idle one when possible (so dependencies, build artifacts, and env files are already there), and makes sure it’s synced with the latest main before dropping you in. I don’t think about any of that. I simply run the treehouse tool and start working.

I repeat this and usually end up managing 5 to 10 tasks at once. I don’t context-switch much because most of them go straight to a clean PR with no involvement from me. Occasionally, the no-mistakes tool escalates something for a decision, but most of the time, I’m just thinking about and writing the next instruction.

The diagram below tries to show the parallelization angle that I’m talking about:

A little while later, the image-attachment task’s pipeline finished and handed me a PR that was ready to merge. Many issues had been caught and auto-fixed along the way, all logged on the PR, so I can audit them. Also, my favorite part, a “Testing” section with evidence (including screenshots) that the feature works end-to-end, is presented.

Remote Control

Every couple of weeks, I have to drive my son to a birthday party, where I’d find myself useless for a few hours. He’d be having a great time with his friends while I sat somewhere with no Wi-Fi, missing my agents and wondering whether they were blocked and waiting on me.

That stopped once I set up the remote control feature. I don’t use the built-in remote features from Claude Code or Codex for a few reasons:

• I want one consistent workflow across all my agents, not separate apps that do the same thing but stay siloed because different companies want to lock me in.

• I want real, full terminal access — not an agent-only view — so I can also run treehouse, no-mistakes, and gnhf.

• I want perfect continuity across phone, laptop, and PC. My son has zero patience: if he says it’s time to leave, I get up and go. If I typed half a sentence on my phone, I want to finish it on my PC later.

So I set up Tailscale, which puts my PC, laptop, and phone on the same private network where they can reach each other safely. I then ssh between them (which on Mac just means enabling “Remote Login”). On my phone, I use an SSH client to connect to my Mac, attach to my tmux session, and instantly I’m in the same workspace with the same tabs, same agents, same environment.

To keep the connection stable, I use mosh, a transport layer on top of SSH built specifically for terminal state over flaky networks, which matters a lot on cellular. It is the same experience, just more resilient.

Putting Everything Together

So how does all of this come together on a normal day?

• It usually starts with me talking instead of typing, describing a feature or a gnarly refactor by voice.

• If it’s complex, I have the agent draft a plan in Lavish Editor and iterate on it in the browser until it’s right.

• Once the plan is solid, I either ask the agent to implement it directly or hand it to gnhf if it’s big, while I spin up a fresh worktree with treehouse and start the next task in a parallel tmux window.

• When an agent finishes, I don’t read massive diffs line by line. I run no-mistakes, which reviews the code, tests it end-to-end, and opens a clean PR while I move on.

• When I’m away from my Mac, I SSH in from my phone, and the whole workspace stays with me.

Here’s what the workflow looks like on a high level:

Each tool removes one specific point of friction, and together they chain into a smooth workflow I genuinely enjoy. I get to stay at the level of deciding what to build and whether it’s good, while most of what’s in between runs itself.

Closing Thoughts

That’s everything I can think of that made a meaningful difference in my workflow. Reflecting on how it came together over the past couple of years, my biggest realization is that as the models keep advancing, the tooling and workflow around them will keep evolving too. What works well today may be obsolete a few months from now.

At the same time, relying only on off-the-shelf products like Claude Code and Codex is never quite enough. There’s always room for a better, more efficient workflow to take the agents a step further. I benefited a lot from building custom tools to remove whatever friction I hit. You’ll likely face a different set of problems because you work on different projects with different processes.

So I’d encourage you to never accept anything that slows you down. If part of your workflow frustrates you, chances are others are hitting the same thing. Find a tool that fixes it, or build one and share it. We’re in the middle of an industrial revolution. It’s the best time to be creative and redefine how things should work. Let’s tinker and have fun.

Run npx workos@latest to launch an AI agent that reads your project, detects your framework, and writes the auth integration directly into your codebase. No account required upfront. WorkOS automatically creates your environment and keys, then lets your claim the project when you're ready.

Once installed, manage users, orgs, and environments directly from the terminal.

Try it now

This is Part 2 of our series with Shah Rahman, Global Head of Autonomous ML Iteration & Optimization for Ads at Meta, where he architects AI-native infrastructure and multi-agent systems at hyperscale. Connect with him on LinkedIn.

Part 1, published two weeks ago, was written for the individual engineer. Shah covered:

• The shift from engineer to orchestrator

• The four core practices: context engineering, spec-driven development, critical verification, and problem decomposition

• The Agentic Development Life Cycle (ADLC)

• The security guardrails that are no longer optional

Part 1 was about the person. Part 2 is about the organization. Here we cover:

• Pod-based structures and the Agent Champion model

• The leadership crisis from first principles: ownership, empathy, and deciding what to build

• A phased transformation playbook, plus the metrics that prove it worked

Individual gains do not become organizational gains on their own. This is the playbook for making that leap. Let’s dive in.

The Great Restructuring

AI-native leadership is the most significant organizational transformation since the industry moved to agile more than a decade ago. Several companies watched AI-generated code climb from zero to 50 or 60% of their output inside a single year. Select teams have posted 2 to 10x productivity gains.

But we keep learning the hard way: individual tool usage produces individual gains, while systemic improvement takes deliberate leadership and a redesign of how work flows.

The evidence is hard to argue with. Around 70% of transformation success comes from operational and cultural change rather than from deploying technology. And most organizations get this wrong. They distribute tools, measure adoption rates, and then wonder why velocity refuses to move.

But some organizations are getting real results. At Shopify, CEO Tobi Lutke told employees that AI usage is now a baseline expectation, and that teams have to show why a task cannot be done by AI before they ask for more headcount. At Klarna, AI-driven restructuring reduced the workforce by more than a thousand people. These organizations treat AI as a fundamental operating model change, not a tooling upgrade. Almost everyone else is now racing to catch up.

Podified Organizational Structure

This is the atomic unit of AI-native engineering is the small, cross-functional team: 3 to 5 people operating autonomously with AI agents and tools. The hierarchies established during the dot-com era, all those layers of managers, leads, and coordinators, are being dismantled.

When a 10x engineer armed with AI tools can do what used to take a much larger group, the organizational consequences are significant. Some pods now report directly to senior leaders based on strategic importance. Team impact gets redefined around outcomes rather than headcount.

The results from one established team’s pod pilot were striking: 3 projects running on self-sufficient agentic loops, more than 90% engineer adoption across the org in under two months, and features built in hours rather than days using agent-assisted development loops.

Roles become fluid in this setup. Engineers may design, designers may code, and product managers may prototype directly. This is not role confusion, it is capability amplification. AI removes the traditional skill bottlenecks, so teams operate with more judgment and less procedural overhead.

Build Durable Agents With Open Source Frameworks (Sponsored)

Most AI agents work in demos — but fail in production. Learn how to build durable, enterprise-ready AI agents with open-source frameworks using Orkes Agentspan and Conductor. This whitepaper explores how to orchestrate long-running, fault-tolerant agent workflows with built-in governance, observability, retries, and human approvals. See how Agentspan compares to LangGraph, CrewAI, and AutoGen for real-world enterprise AI systems. If you’re building AI workflows that need reliability, scale, and control, this guide shows the architecture patterns that make production-grade agents possible.

Download the Whitepaper

Implementation Steps

While your implementation will be your org-specific, here’s a usable template:

• Start with 1 or 2 pilot pods aimed at high-priority challenging issues that block entire teams.

• Strip out non-essential review layers and reduce pre-approval friction.

• Formalize autonomy so pods can decide for themselves between failing fast and pushing forward.

• Only scale after the pilot metrics validate the results. Resist arbitrary rollout timelines.

The Agent Champion Model

Every pillar should name 1 or 2 full-time Agent Champions, responsible for reshaping workflows, preparing codebases, and restructuring operating models. This is not a side-of-desk assignment. It calls for dedicated, high-agency technical leaders who spend 50 to 100% of their time on the transformation itself.

The Champion model reaches well beyond traditional engineering:

• Product mgmt. champions redesign product reviews, experiment workflows, and cross-functional handoffs for autonomous execution.

• Design champions build agent-first prototyping frameworks while protecting craft standards.

• Analytics champions let agents run analyses at a scale that was never possible before, on top of an AI-native data infrastructure.

One important note: engineers working with Agent Champions write 70%+ of their code with AI assistance, shifting from human-in-the-loop to human-on-the-loop. The implication is that when those engineers make manual edits, it signals missing AI context rather than business as usual.

Four things matter the most for anyone stepping into the Champion role:

• Lead with personal AI adoption first: use the tools daily and share what happens, the wins and the failures alike.

• Commit to the vision of AI as foundational to strategy, not an optional enhancement.

• Remove barriers through structured, individualized engagement with each team.

• Recognize impact based on productivity gains and business outcomes, never on tool usage metrics.

Leadership Competency Evolution

Senior leaders are spinning up “AI-native managers” and “AI-native leaders” groups that go deep on the operating context: processes, tools, reporting, and metrics. This is a competency evolution that educational institutions simply cannot keep pace with yet and hence, the need for such learning and development groups at most organizations.

The leadership competency shifts from delegation to orchestration. You are managing multiple parallel AI workflows, not assigning tasks to humans. Technical depth becomes non-negotiable. Hands-on managers have to evaluate agent-generated code and stand up verification layers. And context engineering becomes a core leadership skill, because the precision of the guidance you give AI systems is the precision your teams inherit.

AI-Native Leadership Crisis From the First Principles

Before we go any deeper into the playbook, it is worth stepping back to the core crisis underneath it all.

The Bottleneck Was Never Building

This is the insight most organizations miss. The dominant narrative celebrates AI’s speed: solo founders shipping with agents, dramatic productivity claims, demos everywhere. But the parts of software development that were always hard, remain hard:

• Deciding what to build among competing options

• Identifying the features users actually need

• Prioritizing the capabilities customers will pay for

• Knowing when to kill a project that lacks clear feedback

Have you heard that building great software is an act of empathy? AI cannot replicate a human understanding of user friction or the emotional stakes inside a product decision. Multiple Y Combinator partners have made the same argument: product taste, design sensibility, and customer empathy become the differentiating human skills once execution is commoditized.

The danger shows up when cheap coding invites excessive feature creation. Users do not get 10x more cognitive bandwidth just because you can ship 10x more features. Teams spiral into uncontrolled development and manufacture false progress.

The shift that matters is asking whether something should be built at all rather than asking if it can be built faster.

The Ownership Problem

Anecdotally, most dysfunction in AI-native organizations comes from unclear ownership, not bad process. Even the most empowered teams get fuzzy when responsibility is ambiguous. Work gets picked up or dropped based on whatever is most urgent that day. Leadership becomes the escalation path for every decision, which hollows out middle management and triggers the great flattening.

Piling on more processes to fix a process failure only deepens the hole. The principle is that if something is important enough, give it to a single owner and make them accountable for the outcome.

We put this into practice with a “STO for Everything” model, where STO stands for Single Task Owner. Each one carries clear priority, authority, and decision rights. This single change turbocharged our transformation by eliminating the coordination tax that ambiguous responsibility almost always creates.

Why AI Amplifies Ownership Problems

Because, AI dramatically expands the surface area of parallel work. More projects in flight means more coordination overhead, which triggers an instinct to add process. When ownership stays undefined, those ad hoc processes become bureaucratic substitutes for accountability, and you end up in a vicious cycle.

You can automate coordination with agents (dependency tracking, scheduling, status summaries), but that only buys temporary relief. It masks the underlying challenges that nobody owns. The moment key people leave, those challenges surface and the systems collapse.

If you want to fix it, you must own the outcome, not the process. Map the STO model onto the human-on-the-loop paradigm: humans set direction, verify outcomes, and make irreducible judgments, while AI handles the mechanics of execution.

We Keep Making the Wrong Thing Better

The most common failure I have watched play out is that teams spend months perfecting products that have no product-market fit. They polish the UI, add settings, refine the copy, all of it generating false progress without changing the trajectory. AI makes this temptation worse by dropping build costs to hours, proliferation of code now drives unvetted product frenzy

The discipline is to test the hypothesis before committing to development. Ask “What is the scrappiest way to learn whether this matters?” before you build anything. The rapid prototyping ecosystem (Vercel’s v0, Replit Agent, Lovable, Bolt.new) makes that nearly costless.

Then design to 50-60%. Ship the minimal functionality that enables the core user journeys. Watch where users hesitate, misunderstand, or abandon. That tells you the real product challenges instead of the imagined ones. Over 70% of features never reach a real user. In the age of AI, there is no excuse for building fully polished features that nobody wants.

The temptation is real, but giving into it may decide the winner vs. loser product.

Human-Agent Collaboration: The First Step To Tackle the Crisis

The Council of Agents

Power users have moved past simple human-AI pairing and into orchestrating multiple specialized AI systems that effectively set up a council of agents. There are few different modalities these councils can take.

Role-based delegation treats agents as specialized staff, each with a distinct persona. Cross-evaluation systems deploy multiple agents to independently analyze a problem and review each other’s work. Assembly line workflows chain sequential specialization: architect, then designer, then coder, then reviewer.

Agent-First Development

The emerging pattern aims at autonomous, agent-driven development, where agents code, build, test, and fix issues while humans provide oversight. The key distinction is that agents drive the actual tasks, and humans step in when agents hit an obstacle, not the other way around.

A few touchpoints make this collaboration work. Every AI module ships with context files that carry a clear architecture context. Work breaks into small, manageable, verifiable chunks. Quality assurance never assumes the AI got it right. And multi-agent coordination manages the interactions between specialized agents.

Teams running AI-first approach often report 2 to 10x acceleration across a wide range of tasks, conditional on getting the foundations right first.

From Human-in-the-Loop to Human-on-the-Loop

Until 2025, humans had to drive agents hands-on. This year, AI agents have advanced enough so that humans no longer need to sit in the driver’s seat. AI agents self-drive while humans provide oversight, governance, and stay in the loop.

One large team made the shift cleanly. Humans set the plans and success criteria, AI executes the implementation and self-verifies, AI iterates on its own until the criteria are met, and humans review and approve the final output. This semi-autonomous approach delivered a 40 to 50% speedup over their previous development loop.

The other results have been just as compelling.

One team’s “Squad of AI Agents” approach drove revenue impact that used to be barely a P25 goal. Another rolled out AI-native workflows targeting 2X-plus productivity, with agents autonomously managing code from authoring through production. A third adopted AI-driven tech debt reduction and gained more than 60% productivity with no quality regression, moving to human-on-the-loop in under 4 months, a transition that usually takes 6 to 12.

Measuring Team Transformation: the Next Important Next Step

Traditional metrics fall apart when AI generates thousands of lines of code in seconds. Measurement has to move from output-based to outcome-based.

The Productivity Paradox

Only 20 to 30% of an engineer’s time is spent coding. Speeding up code generation does not automatically translate into overall productivity. The surrounding work (review, testing, coordination, governance) accounts for the other 70 to 80%, and that is exactly where the bottlenecks form.

Research backs this paradox closely:

• McKinsey found developers using AI assistants were 20 to 45% faster on discrete coding tasks, but cautioned that org-level gains were smaller and harder to measure.

• Google’s DORA team found AI tools improved individual throughput without automatically improving deployment frequency or change failure rates, absent process changes.

• Microsoft Research found a 26% increase in completed pull requests per week, but noted the review burden simply shifted onto other team members.

BCG put it best: real productivity gains require reshaping the work, not just adding tools. The same task done faster matters less than redefining which tasks are worth doing at all.

When one of our teams systematically removed those surrounding bottlenecks, they hit a 1.8 to 2.4x velocity improvement over six months.

Emerging Metrics

Given the productivity paradox, metrics to measure productivity and transformation, must be resilient to the paradox:

• AI-First MAU: 75% or more of code AI-generated. Agent-assisted diffs: aim for at least 55% to see meaningful productivity gains. L4-plus AI tool adoption: 80% or more weekly active usage across engineering functions.

• For business impact, tie AI usage and productivity gains directly to revenue. Track feature velocity, where the 2 to 10x improvements in prototype-to-production timelines show up. And measure developer experience (satisfaction, flow state, collaboration effectiveness) right alongside the output metrics.

• Quality has to be a core metric, not just a guardrail. Watch for “AI slop,” the gradual degradation of a codebase as AI-generated code piles up without adequate review. This is the “nobody’s problem” phenomenon, and it can quietly undermine an entire codebase.

Transformation Anti-Patterns

Eight patterns that consistently derail transformation need special attention from every AI-native leader:

• Tool bolt-on: AI tools bolted on without redesigning the workflow, producing minimal impact. This is the most common failure mode.

• Review bottleneck: Traditional review steps become the throughput limit once AI accelerates generation.

• Prompt cargo culting: Teams copy external prompts without context and get poor agent performance. The bottleneck is context engineering skill, not the model.

• Metrics gaming: Teams optimize for agent-generated code percentage or adoption stats instead of outcomes.

• Security shortcuts: Privileged agents deployed without proper audit controls. Some of the resulting production incidents are real and expensive.

• Knowledge debt: Verification and specification fall behind agent-generated work, creating maintainability risk that compounds over time.

• Junior pipeline hollowing: Early-career developer experience degrades when human validation gets outsourced to agents. The “missing rung” talent pipeline problem turns into a long-term sustainability risk.

• Meeting creep: AI acceleration paradoxically makes room for more frequent syncs with no clear impact. The coordination overhead swallows the time that the faster generation saves.

Success depends on systematically detecting these patterns and rolling out with a real change-management framework like ADKAR (Awareness, Desire, Knowledge, Ability, Reinforcement), backed by structured rollouts and feedback loops. Tool distribution and usage metrics alone will not drive transformation.

The AI-Native Leadership Playbook: A Phased Approach

Now that the AI-native leadership groundwork is in place, here is the phased playbook.

Phase 1: Foundation (the first couple of months)

• Leadership credibility: Use AI personally every day with measurable goals. Target 50% or more of your daily tasks within 30 days, then push higher. Get hands-on mastery of the tools so your understanding is authentic. Share both successes and failures in public to model the learning mindset.

• Agent Champion designation: Identify high-agency technical leaders who can dedicate 50 to 100% of their time. Pull leaders and individual contributors closer together for faster decisions. Set up cross-functional coordination so the whole end-to-end workflow gets transformed.

• Pilot pod formation: Start with a codebase AI readiness assessment. Form 3 to 5-person cross-functional teams with autonomous operation charters. Aim them at real problems, not toy exercises, so the momentum is genuine.

Phase 2: Systematic Redesign (the next couple of months)

• Workflow transformation: Audit the high-friction manual workflows that are good candidates for AI. Move from human-in-the-loop to human-on-the-loop. Build AI-readable documentation and specification systems that turn tribal knowledge into shared knowledge.

• Cultural transformation: Establish psychological safety: MIT research found 83% of leaders believe psychological safety measurably improves AI initiative success. Formalize “AI failure story” sessions. Shift measurement from output to outcome.

• Technical foundation: Clear out dead code, technical debt, and documentation debt to improve AI readiness. Implement sandboxing controls, audit mechanisms, and automated security checks. Build the verification layers that autonomous AI operation depends on.

Phase 3: Structural Evolution (everything after)

• Flatten hierarchies: Remove the coordination layers that slow AI-accelerated work. AI-native builders and AI-native leaders are what you need (consider the STO model).

• Impact-based progression: Reward leverage and outcomes over team size. Define the success metrics that genuinely matter for your organization, and make AI tools and agents your highest-leverage assets.

• Cross-functional fluency: Let roles flex as AI removes traditional skill barriers. Break down the walls between product management, design, engineering, data science, and field support, so AI-native builders can move fluidly and accelerate their builds.

Throughout the process, track the compounding gains that show up beyond the initial productivity bump. Connect AI adoption to strategic business metrics. And hold quality standards rigorously, because velocity without quality is a negative value.

Leadership Assessment: Five Questions That Reveal Your Readiness

1. Impact scaling: If you shipped 10x faster, would users be 10x happier? If not, you may be optimizing the wrong thing.

2. Empathy depth: Do you understand your users well enough to delete half the interface? Without empathy, more AI-generated features will not fix products that make users feel incompetent.

3. Learning velocity: Are you processing real user behavioral data every week? If not, your bottleneck is cycle time to insight, not cycle time to code.

4. Ownership clarity: Does every major initiative have a single owner? Ownership problems wearing a process-problem costume only get worse under AI acceleration.

5. Hypothesis discipline: Are you testing theses or building products? If you cannot name the signal that would kill your project, you are committed to something with no user validation behind it.

AI-Native Process Optimization: The Final Leadership Imperative

Here is the counterintuitive truth: AI does not reduce the need for process, it changes what the process is for.

Pre-AI processes coordinated execution among humans. AI compresses execution while raising the cost of deciding what is worth executing. The world now runs on simultaneous builds, parallel experiments, and stacks of prototype iterations. The leadership decisions about what matters, what to cut, and what to double down on become the binding constraint.

Process optimization comes down to three questions:

1. What are we learning this week? Reward faster, deeper learning across teams.

2. What are we killing this week? Actively retire the products and agents that lack genuine value.

3. Who owns each bet, and what signal would change their mind? STOs steer on objective signals, not intuition and not AI-generated content.

Everything else is overhead.

How Do You Want to Lead in the AI-Native Era?

The window for this transformation is narrowing. Organizations that pull it off within the next year will open a 5 to 10x productivity gap over the ones that delay, and that gap will be brutally hard to close as AI-native practices compound.

The organizations that succeed show real advantages in product development velocity, technical innovation capacity, and their ability to attract top talent. The early-mover results (2.4x velocity improvements, 60%-plus AI-generated code, features built in hours instead of days) point to a fundamental capability shift rather than an incremental one. A few closing thoughts.

The scarce resource has shifted. It went from generation and production to orchestration and judgment. When AI generates at near-zero marginal cost, the ability to evaluate quality, set direction, and make the hard calls becomes the bottleneck. Leaders who invest in building AI-native team capability will significantly outperform those who just deploy more agents.

Structural change is mandatory. The productivity paradox is real. Individual gains do not become organizational gains without redesigning the workflows, the measurement systems, and the cultural norms. Remember the famous line, “culture eats strategy for breakfast,” only shines brighter under the AI light. No amount of transformation will save you if the foundations and the structure are not redesigned for the AI-native era.

Risk mitigation is continuous, not a one-time fix. Monitor AI-generated code quality and maintainability so technical debt does not accumulate. Address the security risks (prompt injection, memory corruption, access control, audit compliance) through embedded CI/CD checks. Prevent the “missing rung” talent pipeline problem by developing AI-native engineers at every level. And hold on to human values while you embrace AI acceleration, because human capital keeps paying dividends in the AI-native era when it is applied well.

AI changes the tools. It does not change the core reality. The hard part stays insanely human.

AI shows up in 60% of engineering work. But only about a fifth of it can be handed off without someone babysitting the output. That’s because agents are missing context.

This 8-stage context maturity model gives a real answer on why you still get inconsistent output for all the tokens burned.

Join Unblocked live on June 24 (FREE) to learn:

• Why more MCPs provides agents access but not understanding

• What it takes to deploy agents you can trust without supervision

• How a context layer solves for quality, efficiency and cost

Register now

This week’s system design refresher:

• Claude Fable 5: Everything You Need to Know! (Youtube video)

• 12 Open-source LLMs

• SLMs vs. LLMs, Clearly Explained

• Single Agent vs. Multi-Agent Architecture

• 7 Permission Modes Every Claude Code User Should Know

Claude Fable 5: Everything You Need to Know!

12 Open-source LLMs

Twelve models worth knowing in 2026, each with one standout strength.

1. Llama 4 Scout: Meta's first natively multimodal open-weight model.

2. DeepSeek V4: A Mixture-of-Experts model under MIT license with a native million-token context window. Near-frontier performance at a fraction of the cost per token.

3. Qwen3: Alibaba's flagship open-weight model with switchable thinking and non-thinking modes, all under Apache 2.0.

4. Gemma 4: Google's open-weight family released under Apache 2.0, with the widest language coverage of any model on this list.

5. Phi 4: Microsoft’s compact model trained almost entirely on synthetic, curated data. A practical choice for edge and on-device deployment.

6. Mistral Small 3.1: A VLM with a long context window that fits on a consumer laptop.

7. Nemotron 3 Super: NVIDIA’s hybrid MoE with a million-token context window. Fully open weights, datasets, and recipes, with strong results on agentic coding benchmarks.

8. GLM 5.1: The first open-weight model to top SWE-Bench Pro. Released under MIT with no commercial restrictions.

9. Kimi K2.6: Competitive with leading closed models on coding while costing far less per million tokens. Available on Hugging Face under a Modified MIT license.

10. StarCoder2: One of the most transparent code models available.

11. OLMo 2 (AI2): The most complete example of open-source reproducibility on this list. Weights, training data, code, and full recipes all released under Apache 2.0.

12. Falcon 3: A family of lightweight open-weight models built to run on a single GPU.

Over to you: which open-source model would you add to this list?

FeatureOps Summit 2026 - Feature management in the AI Era (Sponsored)

Speed without control is a false economy. As AI code-generation accelerates software delivery, the FeatureOps Summit 2026 is here to ensure that when we ship more, we break less.This premier virtual event brings together engineers, architects, and product leaders from companies like Samsung, Lloyds Banking Group, Wayfair, Visa, AWS, Allianz and many others, to explore the infrastructure of fearless delivery.

Key Themes:

• AI Safety Nets: Guardrails for the flood of automated code.

• Edge Resilience: Sub-millisecond evaluation at scale.

• Continuous Flow: Moving past the “fixed-release” mindset. Register today to master the tools and patterns required for a fail-safe release environment.

Register Today

SLMs vs. LLMs, Clearly Explained

Big models cost more. Small models do less. Here's how SLMs and LLMs differ across the dimensions that matter in production:

1. Architecture: SLMs are usually under 10B parameters and run on a laptop or phone. LLMs sit at 10B+ with deeper layers and more attention heads, built for broad reasoning across tasks.

2. Task Complexity: SLMs work well on simple tasks but fail on complex multiple reasoning steps. LLMs handle difficult math, multi-step code, and long-horizon planning.

3. Long Context Recall: SLMs lose the thread across long documents or extended conversations. LLMs reliably track and connect information across large inputs.

4. Latency and Cost: SLMs run on consumer hardware with low response times and significantly lower inference costs. LLMs require GPU and carry higher costs per request.

5. Deployment and Privacy: SLMs run on-device or on-premise. LLMs are typically cloud-hosted, which adds data governance complexity.

6. Where each fits:
   SLMs: on-device assistants, real-time classification, or privacy-sensitive applications
   LLMs: complex reasoning, agent workflows, or broad knowledge tasks.

Are you using SLMs, LLMs, or a hybrid setup in production?

Single Agent vs. Multi-Agent Architecture

Some tasks need a single agent. Others need a whole team. Knowing the difference is the skill.

Single-agent system: One reasoning LLM that plans, picks a tool, and loops on its own until the task is done. Use a single agent when:

• the task is a clear, linear sequence

• one agent can hold the whole problem in its head

• you want something simple to build and easy to debug

Multi-agent system: An orchestrator that splits a task into subtasks and routes each one to a specialized agent. Use multi-agent when:

• subtasks can run in parallel

• one agent writes and another independently verifies the work

• the problem is too big for one agent to coordinate alone

Single agents are cheaper and easier to build, but they hit a ceiling on complex work.

Multi-agent systems are more capable and more reliable, but they add coordination cost.

Start with a single agent. Move to multi-agent only when context or reliability become the bottleneck.

Over to you: Are you running single-agent or multi-agent systems in production?

7 Permission Modes Every Claude Code User Should Know

1. plan: The model drafts a plan. Nothing executes until the user approves.

2. default: Standard interactive use. Most tool calls require user approval.

3. acceptEdits: Edits in the working directory are auto-approved. Other shell commands still prompt.

4. auto: An ML classifier decides on requests that miss the fast path.

5. dontAsk: No prompts shown. Deny rules are still enforced.

6. bypassPermissions: Most prompts are skipped. Safety-critical guards still apply.

7. bubble: A subagent escalates its permission request to the parent.

Only 5 modes are user-selectable. “auto” is gated by a feature flag, and “bubble” is internal.

Over to you: Which mode do you reach for most, and what made you pick it?

A running service generates events constantly.

Requests arrive, functions run, errors appear, and each one is a thing that happened at a specific time with a specific context and a specific outcome.

Logs, metrics, and traces are three ways of looking at this same stream. A log captures one event as a line of text, a metric counts or aggregates many events, and a trace links related events as they move across services. Most of the concepts in observability, including cardinality, sampling, and correlation, are consequences of this structure.

In this article, we will look at the basics of observability in detail with concepts like logs, metrics, and traces explained in detail.

Events

Read more

The course kicks off on June 18th, and enrollment closes in less than 24 hours. If you’ve been thinking about leveling up how you and your team work with Claude Code, this is the moment.

Check it out now

A few things you’ll learn:

• The agentic loop, context engineering, and memory layers that make Claude Code useful for real projects

• How to build with Claude Code Skills, MCPs, and hooks to give Claude the tools and feedback loops it needs to self correct

• Parallel development with Git worktrees, subagents, and agent teams

• A capstone project where you ship something real on your own stack

The course includes live sessions, assignments, and office hours, so there’s plenty of room to ask questions and get unstuck.

The second cohort starts in just a few days: June 18 to 19, 2026. If you want to learn everything from the fundamentals of Claude Code to advanced production workflows, including working with large codebases, this could be a great way to level up.

Check it out now

Datadog analyzed tens of thousands of production applications to reveal where risk is actually showing up and what it means for teams handling security today. Download the full report to dive into the key findings, including:

• Why 87% of orgs have exploitable vulnerabilities in production, and how end-of-life runtimes and outdated dependencies are quietly driving that number

• How to cut alert noise by 80% by focusing only on vulnerabilities that pose real business risk

• The hidden dangers of day-one library, AMI, and Docker image updates, and how supply chain attacks exploit them

Get the report

In December 2024, an AI lab called DeepSeek released a 671-billion-parameter language model along with a technical report describing exactly how they built it. Six months later, a different team called Moonshot AI used that report as a starting point. They scaled the design to a trillion parameters, ran into a training instability problem that emerged at that scale, invented a new optimizer to solve it, and shipped their own model. Eight months after that, a third team called Zhipu AI integrated a different DeepSeek innovation into their architecture and contributed a new training framework of their own.

These three teams work for different organizations. However, they were indirectly collaborating in public, through model releases where each company was learning from what its predecessors had done before. This has been made possible by the rise of open-weight models, where even competitors get to learn from each other. The pace of that kind of collaboration has changed over the past eighteen months, and the reasons trace back to the architecture and training choices these teams made in the open.

In this article, we will look at how open-weight models have transformed the AI landscape.

Disclaimer: This post is based on publicly shared details from various sources. Please comment if you notice any inaccuracies.

Open Weight vs Closed Weight

Every modern large language model has two important things behind it:

• The first is the trained parameters, which are the numbers, often hundreds of billions of them, that the model learned during training. The parameters are what make the model “know” things.

• The second is everything that produced those parameters, including the training data and the training code.

A closed-weight model is one that the company keeps behind an API. The user sends some text to the official API endpoint, the company’s servers run the model on their hardware, and a response comes back. The parameters stay with the organization, and running the model on personal hardware or adjusting it for a specific task remains out of reach.

An open-weight model is one where the company has published the trained parameters. Anyone can download them, run the model on their own hardware, and adjust it for their own data. The training data and the full training code, however, usually stay private.

The term is “open weight” rather than “open source” for this reason.

In traditional software, “open source” means the full source code is available to inspect and reproduce. Most AI models marketed as open source are actually open weight, where the trained model is public, while the process that produced it remains closed. This distinction matters because the published weights, paired with detailed technical reports, are what allow other teams to study a design and build on top of it.

Different open-weight models also use different licenses, ranging from very permissive ones like MIT and Apache 2.0 to custom licenses with specific commercial restrictions, so the practical freedoms vary across the ecosystem.

See the diagram below that shows the difference between accessing a closed-weight model and an open-weight model:

The MoE Architecture

Every major open-weight LLM released at the frontier in 2025 and 2026 shares the same architectural skeleton. It is called a Mixture-of-Experts transformer, or MoE for short.

Modern LLMs are built from stacked “blocks.” Each block has two main parts, an attention layer that figures out which previous words matter for the next one, and a feed-forward layer that does the actual computation.

In a regular (”dense”) model, every parameter activates for every word the model processes. Adding more parameters to make a smarter model means the cost of running it scales linearly with that count. With hundreds of billions of parameters, this becomes impractical.

MoE solves this by replacing the single feed-forward layer in each block with several smaller “expert” sub-networks, plus a small routing component that picks which experts to use for each word. The model can store knowledge across many experts while only computing a few of them per word.

This is why two numbers matter for every MoE model:

• The first is total parameters, which represent the model’s full memory footprint and knowledge capacity.

• The second is the active parameters, which represent how much of the model actually computes per word. Active parameters drive inference speed and per-query cost.

DeepSeek V3, for example, has 671 billion total parameters but only 37 billion active per word. Kimi K2 has a trillion total, but 32 billion active. Qwen3 has 235 billion total and 22 billion active. When comparing the cost of running these models, the active counts are what matter, rather than the totals. A trillion-parameter model and a 235-billion-parameter model can cost roughly the same per query if their active counts are similar.

See the diagram below that shows how an MoE block works:

Beginners often assume that experts in MoE specialize by topic, with a math expert, a code expert, and so on. The reality differs from that picture. The router picks experts per word, rather than per question, and the patterns experts specialize in are mostly outside human interpretation. The routing is fine-grained, and a single sentence will pass through many different combinations of experts as it generates.

The MoE architecture explains why every frontier open-weight team is using roughly the same approach. The interesting differences lie in the design choices teams make inside that design approach, and three places are where those choices show up most clearly.

Tips to take AI agents from playground to production (Sponsored)

Every organization is exploring how to adopt AI agents or MCP servers, but how many of them are in production?

And if they aren’t in production, how likely is it that authentication, access control, and agentic identity concerns are the reason?

Watch this on-demand webinar from Descope to learn:

• Real-world MCP and agentic AI use cases

• Identity challenges that prevent production-readiness

• Actionable tips to build secure, scalable AI agents and MCP servers

Watch the webinar

Attention Strategies

The first place is in how teams handle attention.

Every time the model generates a word, it looks back at every previous word in the conversation to figure out what comes next. To avoid recomputing this lookback at every step, the model caches information from earlier words. This cache is called the KV-cache, short for “keys and values,” and it grows as the conversation grows. For long conversations, the KV-cache becomes the main memory bottleneck.

Three different strategies have emerged for managing it.

• Grouped-Query Attention (GQA): Shares cached information across groups of attention heads, which reduces memory usage with a relatively simple implementation. Qwen3 and Llama 4 both use GQA. It is the easiest of the three to engineer and the most widely adopted across the industry.

• Multi-Head Latent Attention (MLA): Compresses the cached information into a smaller latent representation before storing it, then decompresses when the model needs to use it. MLA was introduced by DeepSeek and adopted by Kimi K2. It saves more memory than GQA, while adding computational work for the compression and decompression steps.

• Sparse Attention: Selects only the most relevant previous words to attend to, instead of attending to every one. DeepSeek introduced their version, called DeepSeek Sparse Attention, in V3.2. Zhipu AI’s GLM-5 adopted it shortly after. Sparse attention is most useful when the context is very long, where attending to everything becomes expensive, although it requires careful design to avoid skipping important tokens.

Each strategy is a rational choice depending on what the team is optimizing for, whether that is engineering simplicity, memory efficiency, or context length.

See the diagram below that shows the three attention strategies side by side:

Expert Count and Sparsity

The second point where the teams diverge is in how aggressively they use the MoE pattern.

Across the major open-weight models released in 2025 and 2026, the number of experts ranges from 16 to 384. That wider spread reflects a real disagreement about how far to push sparsity.

At a fixed compute budget, increasing the number of experts can lower training and validation loss, meaning the model learns better from the same amount of compute. The tradeoff is memory. More total experts mean more total parameters that need to live in memory, even if only a few of them fire per word. Kimi K2’s trillion total parameters require a multi-GPU cluster regardless of how few experts activate per token, while Llama 4 Scout’s 109 billion total parameters fit on a single high-memory server. Both belong to the same architectural family, although the deployment realities differ significantly.

A separate disagreement exists around whether to include a “shared expert” that processes every word and provides a baseline capability floor. DeepSeek V3, Llama 4, and Kimi K2 include one. Qwen3 dropped it after using one in their previous Qwen2.5-MoE, and their technical report stays silent on why. Consensus in the field on shared experts remains elusive, which is worth flagging because beginners often assume technical questions in well-resourced labs are settled. Many of them remain open.

Llama 4 takes a particularly distinctive approach. Rather than making every layer in the model an MoE layer, Llama 4 alternates between dense and MoE layers. It also routes each word to only one routed expert, plus the shared one, instead of eight as in DeepSeek V3. The result is fewer active experts per word, with each expert being larger, which represents a different architectural bet from the rest of the field.

See the diagram below that shows how the expert count varies across these models:

Training Approaches

The third point at which teams diverge is in training.

Architecture is one-half of how a model behaves. Training is the other half, and lately it has become where the more meaningful differences live.

Pre-training is the part where the model learns to predict the next word across trillions of tokens of text. Pre-training gives the model its base knowledge of language and the world. The scale varies between teams, with DeepSeek V3 trained on 14.8 trillion tokens and Qwen3 trained on up to 36 trillion. The general approach, however, remains similar across teams.

Post-training is everything that happens after, and post-training is where models now diverge the most. Three techniques deserve attention here.

• The first is reinforcement learning with verifiable rewards. The model produces an output, and the output gets checked for objective correctness. Did the code compile? Did the math answer come out right? The model is rewarded for correct outputs and adjusted away from wrong ones. This was the breakthrough behind DeepSeek R1, and elements of it have been adopted across the open-weight ecosystem.

• The second is distillation. A very large “teacher” model is trained, and its outputs are then used to train smaller “student” models. Llama 4 was co-distilled from a 2-trillion-parameter teacher called Behemoth during pre-training itself. Qwen3 distills from its flagship model down to smaller members of the family.

• The third is synthetic agentic data. Teams build simulated environments loaded with real tools like APIs, shells, and databases, and reward the model for completing tasks in those environments. Kimi K2’s technical report describes a large-scale pipeline that generates tool-use demonstrations across simulated and real-world environments.

Beyond these techniques, the training infrastructure itself has become a meaningful contribution. Two examples from the recent ecosystem stand out.

• MuonClip: Kimi K2’s team developed this optimizer because their training run was hitting instability at the trillion-parameter scale. With MuonClip, they trained on 15.5 trillion tokens without a single loss spike.

• Slime: Zhipu AI built this asynchronous reinforcement learning framework to improve training throughput, which allows more training iterations within the same compute budget.

Both contributions may end up being more reusable across the ecosystem than any specific architectural choice. Architecture is converging while training is now where teams place their different bets.

See the diagram below that shows where these training approaches sit in the overall pipeline:

The Borrow-and-Build Pattern

A pattern runs through all three of these approaches when looked at together.

• DeepSeek V2 introduced MLA.

• DeepSeek V3 kept MLA and refined the MoE design, training the resulting model for approximately 5.5 million dollars on 14.8 trillion tokens.

• Moonshot AI’s Kimi K2 used DeepSeek V3 as a starting point, scaled the design to a trillion parameters, and contributed MuonClip when the scale-up surfaced new training instability.

• DeepSeek V3.2 introduced sparse attention.

• Zhipu AI’s GLM-5 adopted that sparse attention approach and contributed Slime, a new framework for the post-training phase.

Each team built on the previous team’s published innovations, and each added something the next team could build on in turn. These innovations all depend on published weights and detailed technical reports.

See the diagram below that shows how innovations have traveled between teams over the past eighteen months:

This is an observation about the open-weight ecosystem rather than a verdict that open-weight is “winning” or that closed-weight teams have fallen behind. Closed-weight teams are doing different work, optimized for different things, and many of their innovations stay private by choice. The open-weight ecosystem, however, has produced a kind of technical conversation that ran on a smaller scale before.

Conclusion

The specific models covered here will likely be overtaken in months. The framework for reading them, however, will probably hold.

Modern open-weight LLMs share a common skeleton, the MoE transformer, where total parameters and active parameters represent two different costs. Within that skeleton, teams place distinctive bets in three places:

• The first is the attention strategy, choosing between GQA, MLA, or sparse attention.

• The second is how aggressively to use sparsity, which ranges across the field from 16 experts to 384.

• The third is the post-training approach, drawing from reinforcement learning, distillation, synthetic agentic data, or novel training infrastructure.

The license under which a model is released determines what teams and individuals can actually do with it, and “open weight” remains technically narrower than the traditional notion of open source.

The most interesting development in this period of AI engineering is the innovations that open-weight models are inspiring through their published designs

References

• DeepSeek V3 technical report

• DeepSeek V2 technical report (introduces MLA)

• Kimi K2 technical report

• Qwen3 technical report

• Llama 4 announcement (Meta)

• GLM-5 model card and technical details (Z.ai)

• DeepSeek-R1 technical report

• DeepSeek V3.2 (Sparse Attention) announcement

Speed without control is a false economy. As AI code-generation accelerates software delivery, the FeatureOps Summit 2026 is here to ensure that when we ship more, we break less.This premier virtual event brings together engineers, architects, and product leaders from companies like Wayfair, Visa, Mintlify, Lloyds, and many others, to explore the infrastructure of fearless delivery.

Key Themes:

AI Safety Nets: Guardrails for the flood of automated code.
Edge Resilience: Sub-millisecond evaluation at scale.
Continuous Flow: Moving past the “fixed-release” mindset. Register today to master the tools and patterns required for a fail-safe release environment.

Register Today

Every time an LLM generates a response, two operations run in sequence on the same GPU. The first processes the input prompt and emits a single token. The second produces every token after that, one at a time.

From the outside, they look like stages of one process. However, inside the hardware, they have opposite bottlenecks. One is limited by raw compute. The other is limited by how fast data moves through memory. Most of the engineering work that makes production AI systems fast exists because of this split, and the techniques used to handle it are what inference engineering is built around.

Inference engineering is the discipline of running trained AI models in production efficiently. The work spans low-level GPU code, model serving frameworks, and the cloud infrastructure that ties them together. Engineers in this field optimize for some combination of latency, throughput, cost, and quality, with the specific mix depending on the product they support. A few years ago, this work happened almost entirely inside frontier AI labs. Today, it has become a broad specialty that any company running serious AI workloads invests in.

In this article, we will walk through how inference works and why the field’s optimization techniques exist.

Disclaimer: This post is based on publicly shared details from various sources. Please comment if you notice any inaccuracies.

The Rise of Inference Engineering

Three years ago, inference engineering was a specialty practiced almost entirely inside frontier AI labs. The work concerned a small group of engineers building closed models that the rest of the industry consumed through APIs. That picture has shifted dramatically since 2024.

Open models drove the change. Hugging Face, the public registry for AI models, now hosts well over two million open models, roughly 25 times what existed five years ago. Open releases like DeepSeek V3 have closed the capability gap with closed models, giving companies a real choice between paying for a closed API and running an open model themselves.

Self-hosting open models brings three operational advantages over closed APIs:

• Latency profiles can be tuned for the workload pattern of a specific product, where public APIs optimize for general throughput across many customers.

• Uptime can reach four nines or better with dedicated deployments, comparing favorably to the two nines typical of public APIs.

• Costs typically drop by around 80 percent at scale once volume justifies the engineering investment.

The result is that companies across many categories now build serious inference stacks, including AI-native startups, established products integrating AI into existing workflows, and even traditionally cautious sectors like healthcare.

Cursor offers a representative example. The team built Composer 2.0 on top of an open model, applying extensive inference engineering to deliver autocomplete latency below what closed APIs offer.

The Two Phases of LLM inference

Understanding why inference engineering looks the way it does starts with understanding what actually happens when a prompt arrives at an LLM. The process splits into two phases with very different physical demands on the GPU.

A token is the atomic unit that an LLM works with. Roughly, it is a word or word fragment. The word “inference” might be one token, while “engineering” might break into two. Latency metrics that mention tokens per second are counted in this unit.

The first phase is called prefill.

The model takes the entire input prompt and runs it through every layer of weights in parallel. Two outputs come out of this burst, namely the first token of the response and the KV cache, which is a structure that stores intermediate values from the attention mechanism so they can be referenced as more tokens get generated.

Prefill is compute-bound. The GPU’s math units are the limiting factor because every input token gets processed simultaneously through every layer of the model, and throwing more raw computational power at this phase makes it faster. The metric that captures prefill performance is time to first token, or TTFT. That brief pause between sending a prompt to ChatGPT and seeing the first tokens appear is prefill in action.

The second phase is the decode phase. The model generates each subsequent token one at a time, running a full forward pass through every layer of weights for every token. Each new token depends on every token before it, which makes the process fundamentally sequential, and the GPU does this thousands of times for a long response.

Decode is memory-bandwidth-bound. Math throughput sits mostly idle while the GPU spends its cycles reading model weights from memory for each forward pass, with the bottleneck living in data movement rather than arithmetic. The metric that captures decode performance is tokens per second, or TPS. The streaming pace of a long response is the decode phase at work.

Since prefill and decode have opposite bottlenecks, a technique that accelerates one phase often has minimal impact on the other. This is why benchmarks report TTFT and TPS as separate numbers, with performance on each phase measured independently.

This split is also the structural insight that organizes the rest of inference engineering. Once prefill and decode are understood as two distinct operations, the field’s techniques sort themselves into three groups: those that accelerate prefill, those that accelerate decode, and those that rebalance the two against each other.

The picture above is somewhat simplified. Real inference engines run batching, scheduling, and other complexity layered on top, and the prefill-decode split holds underneath all of it, which is why it serves as the foundation for the rest of this article.

Optimization Techniques

With the prefill-decode split in mind, the major techniques in inference engineering become much easier to organize. Each one accelerates a specific phase, attacks both for different reasons, or restructures the system around the split itself.

Let us cover each of the six techniques in detail.

Batching

Batching is the most basic way to scale a single GPU’s output. The inference engine weaves multiple requests together, token by token, so one GPU can serve many users at once. Throughput rises significantly because the GPU’s compute capacity gets fully utilized instead of sitting idle between requests.

The cost is paid in per-user latency.

A single user on an unbatched system gets the lowest possible response time, while the same user on a heavily batched system waits longer because the GPU is also serving other requests. This trade-off is the primary tension that every other technique navigates around, and different products land at different points on the spectrum, with consumer chat tools favoring lower latency and batch processing pipelines favoring higher throughput.

Prefix Caching

Prefix caching accelerates prefill by reusing KV cache values across requests. When two prompts share an opening segment, like a long system prompt that is identical across thousands of requests, the engine computes that prefix once and reads from cache thereafter. This is why API providers charge less for cached input tokens.

The catch is that the cache helps from the start of the sequence up to the first non-matching token. If the very first token differs between two prompts, prefix caching delivers zero savings even when the rest of the sequence is identical. Therefore, prompt structure has direct cost and latency implications, and putting variable user input late in the prompt while keeping shared content early gives the cache something to work with.

Quantization

Quantization helps both phases of inference, though for different reasons.

The basic move is storing model weights in a lower-precision number format. Most modern models train in 16-bit floating-point, and quantization compresses those values down to 8-bit or 4-bit representations, which means smaller weights occupying less memory and requiring less data movement.

Prefill speeds up because lower-precision math operations run faster on the specialized math units inside modern GPUs. Decode speeds up because reduced memory bandwidth pressure means weights are loaded from memory more quickly per forward pass. A typical step down in precision yields roughly 30 to 50 percent better performance, with the exact gain depending on the model and the technique applied.

The cost is potential quality degradation, and different parts of a model tolerate quantization differently.

Linear weights handle it well, activations are somewhat more sensitive, the KV cache is more sensitive still, and attention layers are the most sensitive of all. The reason is that small precision errors in attention layers compound across the sequence of tokens, with each token’s calculation building on the previous ones, so even small errors snowball into meaningful quality loss over a long response.

Most production setups leave attention at full precision for this reason. The bulk of the engineering work in quantization comes down to figuring out which parts to compress and how aggressively.

Speculative Decoding

Speculative decoding accelerates the decode process by exploiting an asymmetry. Generating a token from scratch is expensive, while verifying whether a candidate token matches what the main model would produce is much cheaper. The Sudoku analogy works here, where solving the puzzle takes effort, while checking a finished puzzle is fast.

In speculative decoding, a smaller draft model predicts the next several tokens, and the main model verifies all of them in a single forward pass, accepting the ones that match its own predictions and rejecting the rest. The result is multiple tokens emerging per forward pass through the main model, where one would normally appear.

Speculative decoding improves TPS while leaving TTFT unchanged, because prefill still runs normally. The technique also works best at smaller batch sizes, when the GPU has spare compute capacity to spend on verification. At larger batch sizes, when many requests are being served at once, the GPU is already saturated, and speculation gets dynamically disabled because every cycle is needed for the main workload.

Parallelism

Parallelism techniques let large models run across multiple GPUs when a single one falls short, either because the model is too big to fit in memory or because single-GPU latency is too high. Two main approaches dominate the open model landscape: tensor parallelism and expert parallelism.

Tensor parallelism splits each layer of the model across multiple GPUs. Every GPU holds a fragment of every layer, and the GPUs share the work for each forward pass. This requires high-bandwidth interconnects between the GPUs, like NVIDIA’s NVLink, because results need to be combined after every layer. Tensor parallelism is the default choice for serving very large dense models, where the bandwidth-hungry communication is offset by the speedup from sharing per-layer work.

Expert parallelism applies specifically to mixture-of-experts models, where only a subset of the model’s parameters activate for each token. Different experts get distributed across different GPUs, and tokens get routed to whichever experts they need. The communication overhead is lower than tensor parallelism because experts operate independently, which makes expert parallelism well-suited for multi-node deployments where bandwidth is more limited. Most production deployments combine both, using tensor parallelism within a node and expert parallelism across nodes.

Disaggregation

Disaggregation takes the prefill-decode split literally. The idea is to run prefill on one set of GPUs and decode on another, with the KV cache shipped between them over the network. Each set uses hardware tuned to its specific bottleneck, and each set scales independently based on its own traffic pattern.

The flow becomes a three-step process:

• The prefill engine takes the input sequence and produces both the first token and the KV cache.

• The cache gets sent over a fast interconnect to the decode engine, and the decode engine handles every subsequent token.

• In conditional disaggregation, short or already-cached requests skip the handoff entirely and run on the decode engine alone, which performs better against real-world traffic that includes a mix of long and short prompts.

Disaggregation is the most architectural of the techniques covered here. It treats prefill and decode as separate services with separate operational concerns, giving operators independent levers to scale each one. Companies running large-scale inference often consider this a near-mandatory step once their workload mix is well understood.

When to Invest in Inference Engineering

Putting these techniques into production is a serious task, and combining them adds further complexity. The question every engineering team has to answer is whether to take on this work or whether off-the-shelf APIs are still the right choice. The answer depends on the stage of the product.

Early in building an AI product, off-the-shelf APIs from established providers are almost always the right choice. Meaningful optimization requires real constraints to work against, and early-stage products tend to have fuzzy assumptions about traffic patterns, latency requirements, and unit economics. Engineering effort at this stage is better spent shipping product, since the complexity of running a custom inference stack slows down iteration when iteration speed is what actually matters.

Three signals usually indicate the equation has shifted:

• API costs have grown into a meaningful expense line.

• Latency requirements have moved past what closed APIs can deliver.

• Reliability needs have started to exceed what vendor SLAs offer.

Cursor handled this transition well. Sub-second autocomplete latency was the product itself, and closed APIs aim for general throughput across many customers, while a code completion model demands a specific shape of speed. Self-hosting an open model and applying inference engineering across the stack made the latency target reachable, and the investment paid back because the constraints were real and the workload was well understood.

Conclusion

LLM inference is two operations with opposite physical constraints.

Prefill is compute-bound and runs once per request. Decode is memory-bandwidth-bound and runs once per token. Most of the techniques in inference engineering exist because of this split, and grasping it makes the rest of the field much easier to navigate.

Each technique covered above fits into the prefill-decode framework:

• Batching trades per-user latency for total throughput.

• Prefix caching cuts prefill work when prompts share opening segments.

• Quantization compresses model weights to help both phases.

• Speculative decoding squeezes more tokens out of decode by exploiting idle compute.

• Parallelism scales models across multiple GPUs.

• Disaggregation runs prefill and decode on separate hardware altogether.

Layered on top of all this is the build-versus-buy question. Off-the-shelf APIs remain the right choice for most products in their early stages, while self-hosting starts to make sense when API costs grow into a real expense line, when latency requirements outgrow what closed APIs can deliver, or when reliability needs exceed vendor SLAs.

References:

• Hugging Face Hub Documentation

• DeepSeek-V3 Technical Report

• Cursor — Composer: Building a fast frontier model with RL

• Cursor — Introducing Composer 2

• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

• Efficient Memory Management for Large Language Model Serving with PagedAttention

• Anthropic — Prompt Caching Documentation

• NVIDIA TensorRT Documentation

• Fast Inference from Transformers via Speculative Decoding

• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

• NVIDIA Megatron-Core Developer Guide — Tensor Parallel

• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

• NVIDIA NVLink and NVLink Switch

• Anthropic Claude API Documentation

Coding agents are here to stay, but vibe-coding auth is dangerous business. Connect your AI assistants to the Descope MCP server instead!

This remote MCP server connects agents to the Descope identity platform, giving them the ability to read docs, inspect project config, manage users and tenants, configure authentication flows, review audit logs, and make changes to your identity infrastructure. All through natural language and from a single session.

Descope is trusted by thousands of businesses including GoFundMe, GoodRx, Linktree, and Databricks.

Get started with 100+ prompt examples

This week’s system design refresher:

• How to Run LLMs Locally (Youtube video)

• The Typical AI Agent Stack, Explained

• Understanding Git Reset Modes

• How NAT Works

• Final Week to Enroll: Build with Claude Code

• We’re hiring at ByteByteGo

How to Run LLMs Locally (Great For Learning and Privacy)

The Typical AI Agent Stack, Explained

Most people think an AI agent is just a clever prompt and an LLM. The reality is much deeper. There's an entire architecture working behind the scenes to make it all run.

The diagram below shows the full AI Agent Stack. At the core is the Agent Runtime that runs a ReAct loop, and three other layers feed into it.

AI Agent Runtime: The LLM thinks about what to do, picks a tool, observes the result, then reflects and decides the next step. This loop repeats until the goal is reached.

Model Layer (the brain): The underlying LLMs that power reasoning.

Tool Layer (the hands): How the agent interacts with the real world: search, APIs, code execution, data access.

Memory Layer (the notebook): Short-term working memory for the current task, long-term semantic memory for knowledge, and transactional memory for state.

Wrapping everything is the Observability & Safety Layer. This is what keeps agents debuggable, evaluable, cost-aware, and safe in production.

Over to you: Which layer of the stack do you think is the hardest to get right in production?

FeatureOps Summit 2026 - Feature management in the AI Era (Sponsored)

Speed without control is a false economy. As AI code-generation accelerates software delivery, the FeatureOps Summit 2026 is here to ensure that when we ship more, we break less. This premier virtual event brings together engineers, architects, and product leaders from companies like Wayfair, Visa, Mintlify, Lloyds, and many others, to explore the infrastructure of fearless delivery.

Key Themes:

AI Safety Nets: Guardrails for the flood of automated code.
Edge Resilience: Sub-millisecond evaluation at scale.
Continuous Flow: Moving past the “fixed-release” mindset. Register today to master the tools and patterns required for a fail-safe release environment.

Register Today

Understanding Git Reset Modes

git reset has three modes. Each one moves HEAD, but they differ in what happens to your index and working directory.

1. git reset --soft: Moves HEAD only. Index and working directory stay as-is. Use this when you want to recommit with different changes or a different message.

2. git reset --mixed (default): Moves HEAD and clears the index, but leaves the working directory alone. Your changes become unstaged, still there, just no longer queued for commit.

3. git reset --hard: Moves HEAD, clears the index, and resets the working directory to match the target commit. Any uncommitted changes are gone.

Over to you: Which reset mode do you use the most and has “--hard” ever cost you a day of work?

How NAT Works

Every device in your home probably shares the same public IP, still each one browses, streams, and connects independently. This is handled by NAT (Network Address Translation), a protocol that runs quietly in the background of almost every home network.

It’s the reason IPv4 hasn’t run out completely, and why your router can hide dozens of devices behind a single public IP.

• The Core Idea: Inside your local network, devices use private IP addresses that never leave your home or office. Your router, however, uses a single public IP address when talking to the outside world.

NAT rewrites each outbound request so it appears to come from that public IP address, assigning a unique port mapping for every internal connection.

Outbound NAT (Local to Internet): When a device sends a request,

• NAT replaces the private IP address with the public one

• Assigns a unique port so it can track the connection

• Sends the packet out to the internet as if it originated from the router

Reverse NAT (Internet to Local): When the response returns,

• NAT checks its translation table

• Restores the original private IP address and port

• Delivers the packet to the correct device on the local network

Over to you: Have you ever run into tricky NAT edge cases? Port forwarding? Double NAT? Video calls breaking? Online gaming problems?

Final Week to Enroll: Build with Claude Code

We’re launching a new 2 day intensive, cohort based course called Build with Claude Code, taught by John Kim, who has trained hundreds of engineers at Meta to use Claude Code in real production workflows.

The course kicks off June 18th, and enrollment closes in less than a week. If you’ve been thinking about leveling up how you and your team work with Claude Code, this is the moment.

Check it out now

A few things you’ll learn:

• The agentic loop, context engineering, and memory layers that make Claude Code useful for real projects

• How to build with Claude Code Skills, MCPs, and hooks to give Claude the tools and feedback loops it needs to self correct

• Parallel development with Git worktrees, subagents, and agent teams

• A capstone project where you ship something real on your own stack

The course includes live sessions, assignments, and office hours, so there’s plenty of room to ask questions and get unstuck.

The first cohort starts in just a few days: May 28 to 29, 2026. If you want to learn everything from the fundamentals of Claude Code to advanced production workflows, including working with large codebases, this could be a great way to level up.

Check it out now

We’re Hiring at ByteByteGo

We’re looking for multiple part-time instructors to teach AI and engineering cohort-based live courses.

This is a great fit if you love teaching, enjoy sharing what you know, and want a meaningful side thing alongside your main work.

The role has some upfront time investment to get familiar with the curriculum and prepare, but after that, it’s designed to be a limited commitment (2-5 hours bi-weekly). It offers stable income, good upside, and a chance to share your knowledge while working with ambitious learners.

We’re especially looking for instructors in:

• Building Production-Grade AI Systems

• System Design

• AI Security & LLM Red-Teaming

• AI Evals Intensive

• AI Cost Optimization

• Agentic AI Coding

• Build with Codex

• AI for Engineering Leaders

• AI Automation

• Others, please suggest

Ideal instructors are hands-on, clear communicators, and excited to teach.

If this sounds like you, email us at jobs@bytebytego.com with your background, the topics you’d be excited to teach, and any teaching, writing, or speaking samples.