Fine-tuning can improve a speciality while weakening general capabilities needed around it. A question requiring both then falls between two otherwise capable checkpoints.
SFT-as-Context, posted October 8, gives the specialised model's response to the parent model as context. Across the authors' evaluated pairs and tasks, the arrangement preserves much of each model's strength and can solve combined questions neither handles alone. Read the research paper.
Specialisation can break the surrounding task
A fine-tuned model is often presented as an improved replacement for its parent. In practice it may be better at a narrow activity and worse at the general reasoning around it. A request that mixes both abilities exposes the trade-off. Selecting the specialist for the entire question can lose useful general knowledge; selecting the parent can discard the very capability the fine-tuning was meant to add. This is a problem of how models are composed as well as how they are trained.
SFT-as-context takes the specialist's work seriously without handing it complete control. The specialised response becomes material for another model to consider, rather than necessarily becoming the final answer.
The parent gets the last word
The procedure first obtains a response from the supervised fine-tuned model. It then gives that response, along with the query, to the parent model. No new training is required for this composition. The parent's parameters retain its general capabilities; the specialist supplies task-specific information through the context window. The arrangement therefore operates at answer time rather than merging the two checkpoints or trying to reverse forgetting inside the specialist.
This difference has operational consequences. Teams can test the composition before committing to another training run. They can also swap a specialised component without automatically changing the model responsible for interpreting and presenting its output.
A combination can solve what neither model solves alone
Across nineteen parent-and-specialist pairs and eleven benchmarks, the authors report that the combined method remains near the specialist on fine-tuned capabilities and near the parent on general capabilities. More strikingly, it can answer mixed-capability questions that neither component handles alone. They also report benefits when the specialist is a small open model and the receiving model is a stronger closed system, extending the idea beyond direct checkpoint ancestry.
The paper examines attention patterns and offers a Bayesian analysis. Those analyses support investigation of the mechanism, but the practical claim still rests on the tested model combinations and tasks. A formal bound should not be mistaken for a promise that an arbitrary parent will correctly reject every misleading specialist response.
An extra pass alone does not explain the combination
The ablations ask whether the gains come simply from instructions and another inference pass. Controls give the parent its own response as context and give the specialist its own response as context. On the reported nutrition task, those self-context variants do not recover the same balance as supplying the specialist’s response to the parent.
The mixed-capability tests also compare against an oracle that selects the better of the two separate answers. SFT-as-context can outperform that oracle because selection cannot create an answer combining requirements neither original response satisfies together. This is a useful distinction between routing and composition. Routing chooses a completed answer; composition gives one model information that can change the answer it produces. Neither approach should be credited with the other’s capability without a matching comparison.
Composition adds a new failure surface
The parent has to decide when the supplied answer is helpful and when it is wrong. Plausible errors can anchor its reasoning, particularly when specialist language appears authoritative. An evaluation should therefore include deliberately incorrect specialist outputs as well as successful ones. It should also compare the extra inference against a single larger model or simpler routing strategy under similar latency and cost constraints.
A new specialist need not replace the general model wholesale to be useful. Treating its output as evidence creates another route to combining capabilities, but the quality of that route depends on the receiving model's judgement and on evidence that complementary strengths survive the handoff.
Questions, news tips or corrections? Contact [email protected].
