PeekKV: The First Content Token Already Knows Which Heads Matter
The size of the KV cache grows linearly with context length, making it a major memory bottleneck in LLM serving. Motivated by recent progress in identifying the functional roles of attention heads, a growing body of work compresses the KV cache non-uniformly across heads.
arXiv cs.LG2 min read