A recurrent model can carry a fixed amount of history and still spend much of its time moving that history through memory. The mathematical attraction of state-space models—a compact state instead of an ever-growing attention cache—does not remove the hardware cost of repeatedly reading and writing the state at each generated token.
ReplaySSM examines that gap between an elegant recurrence and an efficient serving system. Its proposal changes the schedule on which state is materialised. Recent inputs are retained beside a checkpoint, allowing some work to be deferred and replayed when needed. This is a systems redesign of memory management, rather than a claim that the model has learned a smaller representation.
The checkpoint is not the whole recent past
ReplaySSM's October 9 paper represents recurrence through a checkpoint plus recent inputs. Periodic checkpoint updates reduce state traffic, while speculative rollback becomes a buffer operation. The authors implement the method in vLLM and report serving gains on tested hybrid models. Read the research paper.
A checkpoint plus a buffer of recent inputs can represent the same evolving recurrence without committing a fresh full state on every step. ReplaySSM separates output computation from checkpoint updates, with periodic writes reducing the dominant state-memory traffic. The buffer is doing useful work: it preserves what is needed to reconstruct the intervening evolution.
Speculative decoding benefits because rejected tokens need not be unwound through a chain of fully materialised states. Rollback can instead change a buffer pointer. This makes the representation of tentative work part of the decoding algorithm, rather than an incidental implementation detail.
Tentative tokens should leave reversible state
Speculative decoding asks a model to accept or reject proposed tokens. For a recurrent model, rejecting a proposal is awkward if every tentative step has already overwritten the live state. The cost of restoring that state can eat into the saving from proposing several tokens together.
ReplaySSM retains recent inputs and separates checkpoint updates, so output-only verification can avoid materialising every intermediate state. That is why rollback and reduced memory traffic belong to the same design story. A more reversible representation of speculative work can improve serving even without changing the draft model’s proposal quality. The numerical split extends the same idea: not every temporary approximation deserves to become part of future history. For a practical comparison, acceptance rates, rollback frequency and buffer behaviour should remain visible; a throughput number alone cannot show whether a different workload will exercise the same advantage.
The implementation has a workload
The authors implement the method in vLLM for Mamba-2 and Gated DeltaNet. At serving batch size 256, they report standard-decoding speedups of 1.20–1.47 times across four hybrid models relative to a tuned vLLM baseline. On two 120B-class models, they report roughly 1.8 times the throughput of vLLM’s speculative decoding.
Those conditions belong beside the numbers. A high-batch serving gain does not imply the same improvement for one interactive user, another accelerator or a different mixture of sequence lengths. Buffer overhead and available compute determine whether reducing state traffic is the dominant benefit in a particular deployment.
Rounding should not become history
The split between output state and update state offers a second insight beyond speed. A low-precision copy can help produce a token, while a higher-precision checkpoint remains responsible for future recurrent updates. An approximation used to answer now need not be fed back indefinitely as the basis for later answers.
That is a useful pattern for long-running inference systems: distinguish an economical view of state from the authoritative state that future computation will inherit. ReplaySSM provides evidence for this pattern in tested hybrid models. Its broader value is showing that state precision, write frequency and speculative rollback can be designed together instead of optimised independently.
Questions, news tips or corrections? Contact [email protected].
