Freezing old knowledge protects it, but can leave too little flexibility for new classes. Updating everything restores flexibility while risking the knowledge already acquired.
An October 2 paper proposes Selective Backpropagation, which restricts updates to pre-allocated parameter subsets. It evaluates few-shot incremental learning beyond short, familiar benchmarks, including cross-domain settings and a longer ImageNet stream. Read the research paper.
Ongoing learning is a long-horizon resource problem
Few-shot class-incremental learning asks a model to recognise new classes from very little data while retaining old ones. Each update therefore consumes more than compute. It uses adaptable capacity that may be needed by a later task. Short benchmarks can conceal this resource problem: a strategy that preserves accuracy for several sessions may only be postponing the point at which new learning becomes difficult.
The standard choices expose a familiar tension. Fine-tuning broadly risks interference. Replaying old examples requires storage and additional computation. Freezing most of the network protects past knowledge while narrowing the space available for future adaptation. Selective Backpropagation makes that allocation explicit.
Reserve the parameters before using them
The proposed framework pre-allocates subsets of network parameters for updates. Gradients are restricted to the permitted subset, protecting earlier knowledge while retaining unused capacity for subsequent learning. The authors describe the approach as deterministic parameter budgeting. It avoids an expensive search for masks at each stage, which is relevant when rapid incremental updates are part of the requirement.
The mechanism should not be confused with making a fixed model infinitely expandable. A finite network still has a budget. The research question is whether a simple allocation discipline can achieve a useful balance between stability, adaptability and training cost across the tested stream.
An eighty-session test asks a different question
The authors evaluate standard benchmarks, cross-domain settings and an eighty-session ImageNet-1K stream. They report strong performance with training time close to naive fine-tuning and substantially below prior comparison methods. The extended evaluation is important because it changes which failure modes can appear. Distribution shift tests whether preserved representations remain useful; a longer sequence tests whether the update policy continues working after many allocations.
A method can rank well on an early, distribution-consistent sequence and still become a poor choice for ongoing deployment. The paper's challenge to short-horizon evaluation is therefore part of its contribution, independently of whether a particular parameter partition is ultimately the best design.
Unused capacity still needs to remain independent
The CIFAR-100 ablation identifies free-parameter reinitialisation as another important part of the method. Without it, incremental performance collapses even though the framework nominally leaves parameters available for later tasks. The authors interpret this as implicit co-adaptation: apparently free parameters can become entangled with the current session before being reassigned.
That finding sharpens the budgeting analogy. Reserving a region on paper does not prove that it remains functionally uncommitted. The prototype-based classifier also contributes, although the selective-update-only variant retains much of the full method’s performance. These comparisons make the result more informative than a single aggregate win: parameter isolation is a major contributor, but maintaining the usability of the reserve and adapting the classifier both affect the final trade-off.
Capacity should be measured as well as accuracy
An operator needs to know whether later tasks receive enough useful freedom to learn. Average final accuracy alone can hide a model that retains older classes by sacrificing the newest ones. Per-session acquisition, earlier-task retention, update time and the amount of adaptable capacity remaining describe a more informative trajectory. Uneven task difficulty also matters: equal parameter allocations may not imply equal learning opportunities.
This is especially relevant when future data cannot be assumed to resemble the first training block. A cheap update procedure earns its practical advantage only if it remains effective when the stream becomes longer or less familiar. Continual learning is consequently a budgeting problem over time, with decisions made today changing the options available for tomorrow's data.
References
Questions, news tips or corrections? Contact [email protected].
