A training curve can rise while previously correct answers become wrong. New successes may simply outnumber the losses.
An October 9 self-distillation study associates this degradation with changes in hidden-representation geometry. Its proposed regulariser encourages within-cluster compactness and between-cluster separation, reducing regressions in the authors' evaluated methods. Read the research paper.
A higher score can hide a disappearing capability
Progress in post-training is usually shown as a curve: accuracy rises as optimisation proceeds. That presentation makes the model look as though it is accumulating successes. But each point aggregates different questions. A later checkpoint can solve new items while failing items an earlier checkpoint handled correctly. If the new successes outnumber the losses, the curve still rises. Users relying on the lost capability experience a regression that the headline number can conceal.
The study examines this problem in on-policy self-distillation. It treats the survival of previously solved queries as something to track directly, rather than assuming it follows from overall improvement.
Follow the transitions, not only the totals
In on-policy self-distillation, the same model serves as teacher and student under different conditions. The authors find that training can progressively degrade acquired capabilities even while aggregate performance improves. Paired query-level tracking reveals the transitions from correct to incorrect that an ordinary score does not show. That distinction is especially relevant when a model is repeatedly updated rather than trained once and replaced only after a broad release audit.
The result changes the meaning of a successful update. A higher average may still be desirable, but the losses become an explicit cost to evaluate rather than something silently cancelled by gains on unrelated questions.
Representation geometry offers an intervention
The researchers associate these regressions with reduced compactness within representation clusters and reduced separation between clusters. Their proposed Representation Geometry Regularization encourages both properties during post-training. It can be added to existing self-distillation methods without replacing their core optimisation procedure. Across the evaluated methods and benchmarks, the authors report less capability degradation while retaining gains.
This is evidence for a targeted intervention, not proof that every lost answer has a geometric cause. Hidden representations offer many possible measurements, and associations need careful interpretation. The useful empirical question is whether the regulariser improves retention in additional settings without preventing the model from learning something genuinely new.
The regulariser contains more than a geometry term
RGR includes both a geometry constraint and an entropy-related constraint on the output distribution. The ablation removes each in turn. The full version performs best across the reported AIME24, AIME25 and HMMT25 comparisons, while dropping either component reduces the result. That makes it inappropriate to attribute the entire effect solely to a visual intuition about compact clusters.
The paper also tracks a capability-degradation ratio across training. That query-level measure gives the geometric intervention an outcome to explain: the fraction of previously acquired successes that disappear. A useful follow-up would examine whether improvements in that ratio remain when newly learned items differ more substantially from the earlier ones, rather than relying only on nearby reasoning benchmarks.
A release needs a capability ledger
Teams can report four groups of evaluation items: retained successes, newly acquired successes, persistent failures and regressions. That ledger gives a clearer account of an update than one net score. Important task families can then receive their own retention requirements, rather than being traded away implicitly against a larger benchmark category.
Overly rigid preservation carries a cost, too. A representation may need to change to absorb better knowledge or correct an earlier shortcut. The objective is not to freeze every internal pattern. It is to make the balance between acquisition and destruction measurable, and to determine whether a regulariser improves that balance under the learning conditions the model will actually face.
Questions, news tips or corrections? Contact [email protected].
