A draft model can improve its local predictions while barely changing the number of target-model calls needed to finish an answer. The serving objective and training objective are not identical.
An October 7 paper models speculative decoding as a Markov reward process and derives an Expected Decoding Rounds objective. Its authors use target-model rollouts to optimise the coupled process and report improvements in evaluated parallel drafters. Read the research paper.
Local token quality misses a moving starting point
A speculative decoder accepts a prefix of draft tokens and then begins its next round after that prefix. The next starting position is therefore determined by earlier acceptances. Parallel and semi-autoregressive drafters complicate this further because their distribution at a position depends on where the block begins. Training each block as an independent local prediction problem can ignore the path through which the decoder actually reaches it.
The Expected Decoding Rounds paper turns that path into the object of optimisation. It asks the drafter to reduce the expected number of target verification rounds required by the complete process, rather than relying entirely on a convenient local surrogate.
A Markov view connects errors across rounds
The researchers represent speculative decoding as a Markov reward process. Their Expected Decoding Rounds objective weights local rejection costs by state occupancies and, under the formulation, exactly equals the expected round count. A rejection is important partly because of where it leaves the decoder and how likely that state is to occur, not only because it is a wrong token.
They derive an exact temporal-difference gradient supporting unbiased stochastic optimisation from target-model rollouts. The framework also provides an offline evaluator for comparing round counts on shared target trajectories. That evaluator permits paired drafter comparisons without running the full speculative decoder for every candidate, making the formal model useful beyond a new loss function.
The experiment keeps the target fixed
The authors fine-tune DSpark with a Qwen3-4B target and DFly with Qwen3-8B, using a block size of seven. Thinking mode is disabled in the stated experimental setup. They fine-tune only drafter parameters and compare training objectives across nine benchmarks covering mathematics, code and chat. EDR consistently improves mean accepted length and outperforms the compared objectives in their results.
These details define the demonstrated scope. The study is not evidence that changing a target model's reasoning policy produces the same gain, or that every sampling setup will yield identical margins. The theoretical objective and empirical implementation need to remain linked when translating the result to another decoder.
Shared rollouts make comparisons less noisy
The offline evaluator compares drafters against the same target-model rollouts. That pairing matters because separate decoding runs can visit different starting positions and conflate drafter quality with a different realised path. The formal process gives a way to evaluate expected round counts without repeatedly executing speculative decoding for each candidate.
The paper also avoids auxiliary hyperparameters in the EDR objective, though model training and sampling still have ordinary configuration choices. Its exact temporal-difference gradient is tied to the stated process, not a claim that every practical estimator has zero variance. The useful methodological advance is a clearer connection among objective, optimisation and evaluation, making it possible to ask whether a local improvement reduces the repeated work the decoder is actually expected to perform.
Rounds are useful, but seconds remain the service metric
An exact objective is exact for the quantity it defines. Round count does not include every variation in drafting cost, verification time, batching or hardware utilisation. A decoder with fewer expensive rounds can still be slower than one with more cheap rounds. The paper's framework therefore provides a principled component of evaluation rather than eliminating the need for wall-clock measurements.
The drafter’s training objective should reflect its job inside the decoder. A drafter is not deployed to win an independent language-modelling contest; its predictions change the sequence of states visited by a larger decoder. EDR makes that coupling explicit. It also provides a useful warning for other inference optimisations: a proxy can be convenient and correlated with performance while still missing the repeated process that creates the actual cost.
Questions, news tips or corrections? Contact [email protected].
