An agent removes an obsolete paragraph from its working context. Later cached tokens may still encode information computed while that paragraph was present.
An October 12 paper studies budgeted KV-cache repair. Its Semantic-Unit Refresh recomputes coherent, relevant regions rather than isolated tokens, recovering much of full-recomputation performance in the evaluated deletion and question-answering tasks. Read the research paper.
Removing a sentence does not undo its computation
Agents increasingly edit their working context. They remove obsolete information, replace instructions and compress earlier material to make room for new evidence. The visible text can change immediately, but later cached states were computed while the removed material was still present. Keeping those states creates a mismatch between the current prompt and the history represented by the cache. Recomputing the entire retained suffix resolves the mismatch at a potentially high cost.
The paper studies the middle ground: budgeted cache repair. It asks how much useful task performance can be recovered by recomputing only a chosen fraction of the suffix, and where that limited refresh should be spent.
Coherent regions beat isolated high-value tokens
Comparing allocations under the same relevance scores and token budgets, the authors find that refreshing coherent text regions can substantially outperform refreshing individual positions. Their Semantic-Unit Refresh method uses inexpensive embedding scores to allocate work to query-relevant regions. It needs neither training nor an auxiliary transformer forward pass for that selection.
The regional choice has a computational rationale. Nearby states share dependencies and context. Refreshing one isolated position while leaving its neighbours conditioned on an old version can preserve an inconsistent mix of histories. Treating meaningful regions as units gives the repair a better chance to restore related computations together, rather than simply maximising a list of independent relevance scores.
Execution overhead is part of the repair problem
Selected refreshes can be fragmented across the context. Running each interval separately risks losing the benefit to repeated execution overhead. The paper therefore introduces a layer-packed executor that processes selected states together at each layer while retaining full causal cache access. This addresses how the repair is executed, not merely how its tokens are chosen.
Across five models, the authors report strong recovery on controlled deletion tasks while refreshing two to five percent of the suffix. On four natural multi-hop question-answering datasets, a five-percent budget reaches near-full-recomputation exact match with lower post-edit latency. The executor accelerates fragmented refresh by up to 16.1 times relative to interval-wise execution in the reported comparison.
Identical scores can produce very different repairs
On the controlled deletion task, the authors hold token scores, budget and repair operator fixed. For Qwen3-8B at a two-percent refresh budget, exact match rises from 7.3 percent with individual-token selection to 75.0 percent with the best contiguous window and 89.6 percent with Semantic-Unit Refresh. The comparison isolates the organisation of selected work rather than a better relevance model.
The best window already maximises summed relevance for its width. Distributing work across relevant semantic regions still improves recovery in that experiment. This is the paper’s strongest evidence that support structure is a separate design variable. A system can choose apparently important tokens and still spend its repair budget poorly if their dependent states are left incoherent.
Task recovery is not complete erasure
Those results measure whether the repaired cache supports useful answers on the evaluated tasks. They should not be described as a guarantee that every influence of deleted information has disappeared. Removing a local factual paragraph and removing a broad instruction can affect different parts of the suffix. Exact recomputation remains an important reference when testing edits with wide consequences.
Context management needs a contract linking a visible edit to what happens in the cache. An agent's claim that information has been removed needs to specify whether it means deleted text, repaired computation or a particular task-level outcome. Efficient approximate repair can be valuable without satisfying every interpretation. Making that distinction explicit gives developers a way to choose refresh budgets according to the edit's significance, instead of assuming that a visible prompt change automatically produces an equivalent internal state.
Questions, news tips or corrections? Contact [email protected].
