Really well-structured breakdown — the prefill/decode split is one of those fundamental concepts that clarifies so much downstream confusion about why inference optimization is non-trivial.
One dimension worth adding to the conversation, especially as this space matures: the principles covered here apply beautifully to cloud-scale inference, but they translate in fascinating and constrained ways when you move to edge and on-device deployments. The decode phase being memory-bandwidth-bound is, if anything, even more acute on mobile silicon — where you're dealing with shared LPDDR memory across CPU, NPU, and GPU, and where sustained inference heats up the package fast.
For edge AI deployments — standalone IoT devices, connected hardware, embedded systems with cellular (eSIM-based) connectivity — the build vs. buy calculus flips almost entirely. You're not evaluating API costs vs. self-hosting; you're evaluating whether inference can happen at all without a network connection. Quantization goes from a cost optimization to a hard requirement: Q4 INT8 or bust for most MCU-class targets.
The speculative decoding insight is particularly interesting at the edge: the "draft model + verify" pattern maps well to constrained hardware because the draft model can live entirely on a small local NPU while verification gets offloaded opportunistically when connectivity (via eSIM/cellular) is available. This hybrid local/remote inference pattern is genuinely underexplored and I think it's where the next wave of edge AI product design will live.
Thanks for making these concepts accessible — this is the kind of foundational literacy the field needs as inference moves beyond the datacenter.
We are studying and developing inference cases to identify and extract PII from our customer documents and interactions in multichannel flows. this article provided great points in how we can improve our inference flows.
Hmm you can have both, eg. use open models (eg. llama, kimi) via API in a hosted remote environment like AWS Bedrock. This way you delay building out the infrastructure while using API, and avoid paying frontier model fees.
Renting open weights through Bedrock still means someone else solved the prefill-versus-decode batching for you, so you're buying inference engineering, just unbundled. The shakier part is assuming the open supply stays open: Meta spent three years as the largest open-source champion, then shipped its first proprietary model once returns mattered. Open weights were a catch-up strategy, and catch-up ends.
Best explanation of the prefill/decode split I've read: framing decode as memory-bandwidth-bound makes speculative decoding and quantization click instantly. Saving this as a reference. Thanks for writing it.
Great article about inference and putting it to the engineering lens. This is information that can be easily comprehended and used today. Thank you for making us better.
regarding parallelism strategies, this is also a great source to explore the topic in more depth: https://rocm.blogs.amd.com/software-tools-optimization/vllm-moe-guide/README.html
Love to learn more about parallelism. The KV Caches are killing me, even with the onload/offload I use with my local model.
Very nice explanation.
Really well-structured breakdown — the prefill/decode split is one of those fundamental concepts that clarifies so much downstream confusion about why inference optimization is non-trivial.
One dimension worth adding to the conversation, especially as this space matures: the principles covered here apply beautifully to cloud-scale inference, but they translate in fascinating and constrained ways when you move to edge and on-device deployments. The decode phase being memory-bandwidth-bound is, if anything, even more acute on mobile silicon — where you're dealing with shared LPDDR memory across CPU, NPU, and GPU, and where sustained inference heats up the package fast.
For edge AI deployments — standalone IoT devices, connected hardware, embedded systems with cellular (eSIM-based) connectivity — the build vs. buy calculus flips almost entirely. You're not evaluating API costs vs. self-hosting; you're evaluating whether inference can happen at all without a network connection. Quantization goes from a cost optimization to a hard requirement: Q4 INT8 or bust for most MCU-class targets.
The speculative decoding insight is particularly interesting at the edge: the "draft model + verify" pattern maps well to constrained hardware because the draft model can live entirely on a small local NPU while verification gets offloaded opportunistically when connectivity (via eSIM/cellular) is available. This hybrid local/remote inference pattern is genuinely underexplored and I think it's where the next wave of edge AI product design will live.
Thanks for making these concepts accessible — this is the kind of foundational literacy the field needs as inference moves beyond the datacenter.
We are studying and developing inference cases to identify and extract PII from our customer documents and interactions in multichannel flows. this article provided great points in how we can improve our inference flows.
Hmm you can have both, eg. use open models (eg. llama, kimi) via API in a hosted remote environment like AWS Bedrock. This way you delay building out the infrastructure while using API, and avoid paying frontier model fees.
Renting open weights through Bedrock still means someone else solved the prefill-versus-decode batching for you, so you're buying inference engineering, just unbundled. The shakier part is assuming the open supply stays open: Meta spent three years as the largest open-source champion, then shipped its first proprietary model once returns mattered. Open weights were a catch-up strategy, and catch-up ends.
Best explanation of the prefill/decode split I've read: framing decode as memory-bandwidth-bound makes speculative decoding and quantization click instantly. Saving this as a reference. Thanks for writing it.
Great article about inference and putting it to the engineering lens. This is information that can be easily comprehended and used today. Thank you for making us better.