AbridgeKV: Compressing Long Reasoning into the KV Cache of Learned Pause Tokens
Long chain-of-thought decoding is memory-bound: the KV cache of the generated tokens grows linearly with the test-time compute being spent. Evicting cache entries does not fix this, because a model never trained on a cache with entries removed generates longer, often up to the length limit.
arXiv - cs.CL (NLP)2 min read