Training a deep network from the beginning spends early compute on layers processing immature representations. Adding depth later offers a different allocation of the same learning effort.
An October 12 study examines progressive growth in masked autoencoders and DINOv3. Its experiments report reduced training FLOPs with improved downstream visual performance, alongside changes in intermediate representations and attention patterns. Read the research paper.
The final architecture does not tell the training story
Two vision transformers can finish training with comparable depth while having reached that architecture through different histories. One processes every early example through the full network. Another begins shallow and gains layers later. Progressive growth has attracted renewed interest partly because it can avoid spending computation on a deep network before its representations are mature. The more interesting possibility is that the schedule changes what the final network learns, rather than merely reducing the bill.
This study investigates that possibility in self-supervised vision. It examines both masked autoencoders and DINOv3, giving the analysis two different learning setups rather than assuming that one family's behaviour represents all visual models.
Inserted layers inherit an evolving computation
Progressive depth growth adds layers during training. Those layers arrive after some features and processing patterns have already developed, so their learning conditions differ from layers present at initialisation. The authors examine downstream performance, intermediate representations and internal computational patterns to understand the consequences. That combination matters: a final score can show usefulness, but not whether growth has changed which parts of the network carry it.
The study includes global tasks and dense visual tasks. This avoids defining a useful representation only through one classification endpoint. A representation that helps assign a whole-image label need not be equally useful when the application requires spatially detailed outputs.
The savings accompany representational changes
Across the evaluated settings, the authors report reductions of 13.9 to 24.8 percent in training FLOPs alongside improved downstream performance. ImageNet linear-probe gains reach 3.36 percentage points. Intermediate layers also become more useful in their downstream tests. These findings make the claim stronger than a simple account of paying less for an otherwise equivalent endpoint.
The attention analyses identify recurring alternation between global and local processing in masked autoencoders, and an earlier local-to-global progression in DINOv3. These are observations of changed computation. They provide clues about the effect of growth, while leaving room for other factors behind the downstream improvements.
Layer interventions add evidence beyond attention maps
The full study includes layer-skipping interventions, which indicate stronger early-layer influence and reduced sensitivity to many later-layer interventions after growth. In masked autoencoders, final-layer gains and recurring patterns persist after prolonged full-depth training. This addresses the possibility that the differences are merely temporary consequences of recently adding layers.
The result also creates possible applications that the paper identifies as future directions rather than established deployment gains. Stronger intermediate features might support early exits, cheaper feature extraction or less expensive self-distillation teachers. Those opportunities depend on testing actual exit policies and costs. The existing evidence establishes changed representations and computation; it does not automatically establish a faster early-exit product simply because intermediate probes improve.
Compare histories under the same accounting
A fair deployment decision needs training cost, final architecture and inference cost kept separate. Saving FLOPs during pretraining does not necessarily imply an equal reduction in wall-clock time, and a training strategy should not receive credit for a smaller serving model unless that is actually what it produces. Hardware utilisation and the cost of changing depth can affect practical savings.
Architecture design includes a temporal dimension. Choosing a network's final depth is only one decision; choosing when its layers become available can affect the representations that emerge. For vision researchers, that creates a lever for both efficiency and model quality without treating those goals as inherently opposed.
References
Questions, news tips or corrections? Contact [email protected].
