1
MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression
长上下文LLM推理提速新方法，MosaicKV通过动态二维KV缓存压缩，显著降低显存占用并保持精度。
arXiv:2607.00760v1 Announce Type: new Abstract: Long-context LLM services now sustain prompts with hundreds of thousands to millions of tokens, making…