Discussion about this post

User's avatar
Raghav Mehta's avatar

The 37.5% lines up exactly with the 5:3 split, since 3 of every 8 layers stop storing their own keys and values. I'm curious whether the quality cost shows up mostly on long context tasks, because all of those later layers attend through one fixed layer's keys. Does it also save prefill time, since those layers can skip their K and V projections entirely?

10 more comments...

Ready for more?