Discussion about this post

User's avatar
Raghav Mehta's avatar

The post notes that Mamba layers compress history into a fixed state, which can make precise retrieval harder, while only some layers keep a KV cache. I'd like to know how recall on needle-style long context tests holds up as the share of attention layers shrinks. Is there a context length where the few remaining attention layers stop covering for the fixed state?

Abhishek Pan's avatar

With 2 KV heads for 32 query heads, each attention layer caches 16x less than full multi-head attention, and the Mamba layers only carry a fixed-size state, so on long agent sessions the resident 30B of weights become most of the memory bill. Capacity sheets that size concurrency from per-session KV cache, the way they did for a dense model, will undercount how many sessions one GPU can hold.

4 more comments...

Ready for more?