A noob question. I learned that LLMs are stateless, and here is an entire article on LLM caching. What am I missing? Kindly help me make a distinction.
There is a workaround that makes harness like ChatGPT act stateful, Its called prompt stuffing.
Naturally it forgets everything the moment it generates an output, but with prompt stuffing, all your past conversations are sent together with your new messages in the background, so it looks like it remember your conversations.
but that creates a new problem, without caching, the LLM need to perform mathematical computation of both the past conversations together and your new messages every time, and that consumes too much tokens.
Caching solves that problem, by caching the past conversations , which is a fancy way of saying it stores the last mathematical computation results so the GPU only have to compute the new message.
Please let me know if this helps you understand it better.
The 40 GB example is a good reminder that model size is only part of the bill. During generation, the GPU reads the cache for every new token. Long prompts can slow a system down even when the model fits in memory. That is easy to miss when teams benchmark short chats.
The KV cache cost dynamics directly shape unit economics for Indian AI startups. Prompt caching can cut serving costs by 50-90%, making AI viable for SMBs that otherwise couldn't afford long-context models.
As a planner, I advise founders to model these infrastructure costs alongside burn. Choosing models with smaller context needs or leveraging caching can significantly reduce opex, improving runway and valuation.
The future of AI should increase opportunity, not deepen inequality.
AI promises the potential to reshape health, education, science, manufacturing, finance, public services, and so much more. But without careful governance, equitable access, and investment in people, the gains from AI could be narrowly distributed, excluding some communities and jobs from the transition.
A just AI future depends not just on technical advancements but also on inclusive innovation, accountable governance, skills development and reskilling, the digital infrastructure to enable equitable participation and access, data and privacy safeguards, and widespread access to tools and education. Governments, industry, researchers, and civil society all have a crucial role to play in ensuring AI amplifies rather than concentrates human potential and opportunity.
A Fair AI Framework
Responsible Innovation + Inclusive Access + Human Skills + Transparent Governance + Public Trust = Shared Prosperity
This looks like:
Expanding access to AI and digital education. Investing in reskilling and lifelong learning for workers. Designing and deploying transparent, accountable, and secure systems. Fostering innovation, but ensuring protections for privacy and fundamental rights. Focusing on enabling opportunity and value creation for small businesses, researchers, educators, and health providers-as well as larger institutions.
Whether AI succeeds shouldn't be measured just on better algorithms or higher investment totals. It should be measured by how it lifts lives, expands opportunity, and shares benefits widely.
Ultimately, the future of AI should give every community a chance to innovate and contribute. When opportunity is shared, tech is a lever for inclusive prosperity.
The framing of the KV cache as a bandwidth cost rather than a storage one is what finally made this click for me... it explains why a request can "fit" in memory comfortably and still crawl.
Thanks for detailed and easy to understand post. I noticed that the "Reuse" image has two identical headlines as both says "without" instead of "with" and "without" .
There is a workaround that makes harness like ChatGPT act stateful, Its called prompt stuffing.
Naturally it forgets everything the moment it generates an output, but with prompt stuffing, all your past conversations are sent together with your new messages in the background, so it looks like it remember your conversations.
but that creates a new problem, without caching, the LLM need to perform mathematical computation of both the past conversations together and your new messages every time, and that consumes too much tokens, Caching solves that problem, by caching the past conversations , which is a fancy way of saying it stores the last mathematical computation results so the GPU only have to compute the new message.
Please let me know if this helps you understand it better.
A noob question. I learned that LLMs are stateless, and here is an entire article on LLM caching. What am I missing? Kindly help me make a distinction.
Let me help you with the distinction:
There is a workaround that makes harness like ChatGPT act stateful, Its called prompt stuffing.
Naturally it forgets everything the moment it generates an output, but with prompt stuffing, all your past conversations are sent together with your new messages in the background, so it looks like it remember your conversations.
but that creates a new problem, without caching, the LLM need to perform mathematical computation of both the past conversations together and your new messages every time, and that consumes too much tokens.
Caching solves that problem, by caching the past conversations , which is a fancy way of saying it stores the last mathematical computation results so the GPU only have to compute the new message.
Please let me know if this helps you understand it better.
Hi, thanks a lot for the answer. It helped me do some further research and connect the dots. Now I am quite clear.
Thanks again.
You are welcome,
I'm glad you found it helpful.
The 40 GB example is a good reminder that model size is only part of the bill. During generation, the GPU reads the cache for every new token. Long prompts can slow a system down even when the model fits in memory. That is easy to miss when teams benchmark short chats.
The KV cache cost dynamics directly shape unit economics for Indian AI startups. Prompt caching can cut serving costs by 50-90%, making AI viable for SMBs that otherwise couldn't afford long-context models.
As a planner, I advise founders to model these infrastructure costs alongside burn. Choosing models with smaller context needs or leveraging caching can significantly reduce opex, improving runway and valuation.
The future of AI should increase opportunity, not deepen inequality.
AI promises the potential to reshape health, education, science, manufacturing, finance, public services, and so much more. But without careful governance, equitable access, and investment in people, the gains from AI could be narrowly distributed, excluding some communities and jobs from the transition.
A just AI future depends not just on technical advancements but also on inclusive innovation, accountable governance, skills development and reskilling, the digital infrastructure to enable equitable participation and access, data and privacy safeguards, and widespread access to tools and education. Governments, industry, researchers, and civil society all have a crucial role to play in ensuring AI amplifies rather than concentrates human potential and opportunity.
A Fair AI Framework
Responsible Innovation + Inclusive Access + Human Skills + Transparent Governance + Public Trust = Shared Prosperity
This looks like:
Expanding access to AI and digital education. Investing in reskilling and lifelong learning for workers. Designing and deploying transparent, accountable, and secure systems. Fostering innovation, but ensuring protections for privacy and fundamental rights. Focusing on enabling opportunity and value creation for small businesses, researchers, educators, and health providers-as well as larger institutions.
Whether AI succeeds shouldn't be measured just on better algorithms or higher investment totals. It should be measured by how it lifts lives, expands opportunity, and shares benefits widely.
Ultimately, the future of AI should give every community a chance to innovate and contribute. When opportunity is shared, tech is a lever for inclusive prosperity.
The framing of the KV cache as a bandwidth cost rather than a storage one is what finally made this click for me... it explains why a request can "fit" in memory comfortably and still crawl.
Thanks for detailed and easy to understand post. I noticed that the "Reuse" image has two identical headlines as both says "without" instead of "with" and "without" .
The idea that the KV cache is a bandwidth problem, not just a memory problem, is the part that really stood out to me.
It explains why simply having enough GPU memory to fit a long context doesn't necessarily mean the workload will be fast or cost-effective.
Context length is ultimately an infrastructure and economics tradeoff, not just a model capability.
Let me help you with the distinction:
There is a workaround that makes harness like ChatGPT act stateful, Its called prompt stuffing.
Naturally it forgets everything the moment it generates an output, but with prompt stuffing, all your past conversations are sent together with your new messages in the background, so it looks like it remember your conversations.
but that creates a new problem, without caching, the LLM need to perform mathematical computation of both the past conversations together and your new messages every time, and that consumes too much tokens, Caching solves that problem, by caching the past conversations , which is a fancy way of saying it stores the last mathematical computation results so the GPU only have to compute the new message.
Please let me know if this helps you understand it better.