Two compressed keys can be numerically close to the originals and still change which passage a model attends to. Reconstruction error is not the same as preserved computation.
BELT's October 13 paper targets attention distributions in its low-rank objective and learns how to allocate rank under a component budget. The authors report quality and throughput gains on their evaluated language models. Read the research paper.
Compression should preserve the use of a number
The key-value cache grows with a language model's context and can become the limiting resource in long conversations or document analysis. Compressing it appears to be a problem of approximating stored tensors. But those tensors exist to support attention. A small numerical reconstruction error can still change which token receives attention, particularly when alternatives have similar scores. Conversely, some numerical differences may barely affect the computation that matters.
BELT chooses an objective closer to that functional role. Its key-side compression targets attention distributions instead of treating faithful reconstruction of the original keys as the primary goal.
Rank is a budget that can move
The framework uses low-rank compression and derives a quadratic surrogate with a closed-form solution for the key-side objective. A learned global allocator distributes rank across components. It uses differentiable soft selection during training and hard selection at deployment to meet a user-specified component budget. This avoids assuming that every component deserves the same compression level.
The mechanism combines two choices that are often discussed separately: what error to minimise and where to spend limited capacity. A behaviour-oriented objective has less value if the rank budget is allocated uniformly to components with very different influence on that behaviour.
The strongest numbers describe different configurations
Across three language models and multiple benchmarks, the authors report up to seventy percent low-rank KV-cache compression, with gains of up to 23.3 points over the strongest baseline on LongBench. Combining the method with quantisation yields up to 17.8-fold overall cache reduction. With Triton GPU kernels and a quantised cache, reported gains reach 2.7-fold for the attention module and 3.3-fold for end-to-end decoding throughput.
These maxima should not be combined into an imagined single universal operating point. Quality, compression and throughput depend on the model, workload and implementation. A deployment decision needs the relevant configuration's complete trade-off rather than the largest number from each experimental category.
Holding the bases fixed isolates allocation
An additional ablation fixes BELT’s key and value bases and varies only how rank is distributed under the same total cache budget. The learned allocation performs better than uniform fractions and the compared allocation methods in that setting. The authors observe a shift of rank from keys toward values, with only about thirty-nine to forty-one percent retained on the key side in the reported allocation.
That result shows why compression quality cannot be explained by choosing a low-rank basis alone. Even with identical bases, the component budget changes useful behaviour. A separate MPT-7B experiment using ALiBi rather than rotary position embeddings also probes the method outside the main positional setup, though it remains another bounded model comparison rather than universal architectural support.
Rare dependencies can expose an allocation error
A learned allocator receives evidence from its training or calibration conditions. Components that appear dispensable there may become useful when a query depends on unusual evidence, conflicting instructions or a rare relationship among distant passages. Testing those cases helps determine whether average benchmark quality hides selective failures introduced by compression.
BELT makes preserved computation a criterion for memory efficiency. A cache exists to help the model distinguish relevant information, not to reproduce every internal number for its own sake. BELT makes that distinction an optimisation target, while its implementation results also acknowledge that an elegant mathematical approximation must still produce a useful layout and execution path on actual hardware.
Questions, news tips or corrections? Contact [email protected].
