A large AI model can run on modest hardware only by reducing the memory it occupies, reducing the calculations it performs, or moving some work to slower hardware.
memory bandwidth as the real bottleneck is the part people underestimate. i'm starting to think distillation and compression will decide what runs locally more than the next gpu generation.
The practical trade-off here is that “fits in memory” is a poor success criterion for an interactive tool. Once KV cache and offload are included, the real budget is quality × time-to-first-token × steady-state tokens/sec. I like the article’s closing move toward measuring realistic tasks; I’d add a cold-start and long-context case to catch systems that look cheap in a short benchmark. Which metric do you find most predictive when choosing between a smaller local model and a larger offloaded one?
Wonderful article!
Did you know this about AI?
1 guy controls AI: Under the illusion of competition, is there evidence of a puppet master behind the Data Center pandemic ?
https://scientificprogress.substack.com/p/1-man-controls-ais
CBDC digitatorship infrastructure being built now!
https://scientificprogress.substack.com/p/cbdc-digitatorship-infrastructure
memory bandwidth as the real bottleneck is the part people underestimate. i'm starting to think distillation and compression will decide what runs locally more than the next gpu generation.
Good read , thank you.
Thanks. This was also a great general description of how Ai works.
The practical trade-off here is that “fits in memory” is a poor success criterion for an interactive tool. Once KV cache and offload are included, the real budget is quality × time-to-first-token × steady-state tokens/sec. I like the article’s closing move toward measuring realistic tasks; I’d add a cold-start and long-context case to catch systems that look cheap in a short benchmark. Which metric do you find most predictive when choosing between a smaller local model and a larger offloaded one?
A YEAR of manual outreach vs. one link if this works, I'm never cold emailing by hand again.
The KV cache section catches a lot of people. The 4-bit model loads fine, then someone pastes a long PDF and it runs out of memory halfway through.