6 Comments
User's avatar
Raghav Mehta's avatar

The post notes that Mamba layers compress history into a fixed state, which can make precise retrieval harder, while only some layers keep a KV cache. I'd like to know how recall on needle-style long context tests holds up as the share of attention layers shrinks. Is there a context length where the few remaining attention layers stop covering for the fixed state?

Abhishek Pan's avatar

With 2 KV heads for 32 query heads, each attention layer caches 16x less than full multi-head attention, and the Mamba layers only carry a fixed-size state, so on long agent sessions the resident 30B of weights become most of the memory bill. Capacity sheets that size concurrency from per-session KV cache, the way they did for a dense model, will undercount how many sessions one GPU can hold.

Bachu Adharsh Kumar's avatar

Nice to see the training recipe explained simply. Which technique here can a small team copy first?

Nan Deng's avatar

The split I would not collapse is active compute versus the bytes you still have to host. Lightning is about 30 billion parameters with 128 routed experts and 6 selected per token, so a token uses about 3 billion, but the unused experts stay resident for the next token. Grouped-query attention, 32 query heads sharing 2 key-value heads, is the part that actually shrinks a long agent transcript, because the KV cache is what grows across dozens of tool calls. Mamba keeps a fixed state instead of a per-token cache, which is why a long history can get cheaper and also why a precise lookup can fail even when every expert is still in memory.

Rohan's avatar

didn't realise multi-token prediction was part of it too. does it actually speed up generation much?

Travis Wright's avatar

The line that matters most is buried early: a coding agent might make dozens of model calls to solve one problem, so agent economics get decided by inference cost, not benchmark bragging. Thirty billion parameters with about three billion active per token is basically a giant staff where only six people show up to each meeting, which might be the most realistic org chart in AI. Curious how the fixed Mamba state holds up on long agent runs, since precisely recalling a tool result from way earlier in the session is exactly what compressed memory tends to fumble.