Reusable skills promise a convenient route to growing agent capability. Instead of enlarging the model, a team adds procedures that can be loaded when relevant. The promise assumes that the agent can still recognise and retrieve the right procedure as the catalogue grows. That access mechanism is itself a source of failure.
The skill-routing study tests growth without changing the original tasks or their relevant skills. By adding irrelevant metadata, it isolates an interference effect that could otherwise look like a decline in reasoning quality. The model has not lost the procedure; the system has made it harder to reach.
The missing skill is often never requested
An October 13 study keeps tasks and golden skills fixed while adding irrelevant skills. Implicit routing deteriorates; explicit preselection performs better. Masking irrelevant descriptions restores much of the loading behaviour in the authors' tests. Read the research paper.
With 1,000 irrelevant skills added, loaded-skill recall in the reported tests falls from 73.6% to 27.4%, while the zero-load rate rises from 9.3% to 51.6%. Explicit selection from the same library retains 73.9% recall. These are controlled results on the evaluated task and skill setup.
The distinction between wrong loading and no loading is revealing. A system that increasingly selects plausible alternatives might need better ranking. A system that stops invoking skills altogether may be losing the routing instruction amid competing descriptions and execution demands. Those behaviours call for different interventions.
Library growth is an intervention on the interface
Holding the task and golden skill fixed makes the experiment unusually diagnostic. If success changes when irrelevant skill metadata is added, the difficulty has been introduced by the access interface rather than by a new task requirement. A benchmark that changes the library and the tasks together could not isolate that effect as cleanly.
The masking experiment strengthens this interpretation by removing irrelevant descriptions while leaving the broader setup in place. It points toward what the agent reads, not merely the number of files that exist. For a team designing a skill protocol, the result motivates careful separation between a selection view and an execution view. That does not prove every library needs a separate model. It shows why the executor should not automatically be required to reason over all available metadata on every task, and why the selector’s own misses need independent measurement.
Descriptions are active inputs
Masking descriptions of irrelevant skills largely restores loading behaviour in the study. That points to semantic and instruction interference rather than a simple inability to open many files. Metadata is not a neutral catalogue when it shares the model’s attention with the task being executed.
This also explains why asking the agent to plan its skill use is not necessarily enough. The planning request still operates in the same crowded input. Explicit preselection changes the information presented to the executor, but introduces another component whose coverage, latency and failure modes must be measured in turn.
An archive can grow faster than usable capability
For a self-learning agent, every completed task may create another skill. A useful evaluation should periodically replay fixed tasks as the library grows, measuring end-to-end success as well as whether the expected skill was loaded. Otherwise a growing count of saved procedures can be mistaken for cumulative improvement.
The broader implication is architectural. Storage, selection and execution are separate responsibilities even when one language model performs all three. A procedure only contributes to operational capability if the right conditions still cause it to be used. Growing memory without maintaining access can make an agent appear richer on paper while becoming less reliable at work.
References
Questions, news tips or corrections? Contact [email protected].
