An agent can learn several basic tasks and improve on a harder combination. Add more examples, however, and improvement may barely move.
EmbodiedCLBench's October 9 paper evaluates compositional continual learning in embodied simulation. The authors find benefits from basic-task experience, alongside limits in how those benefits scale with additional samples. Read the research paper.
More experience is useful only if the agent can extract it
Continual learning promises that an agent's next task will benefit from earlier work. Embodied environments make that promise demanding because a later task may require familiar actions in an unfamiliar combination. Replaying an exact sequence is insufficient. The agent has to identify what is reusable and compose it under new constraints. Simply storing more trajectories does not establish that this extraction has occurred.
EmbodiedCLBench centres evaluation on compositional generalisation. Agents first encounter basic household-style tasks, then attempt advanced tasks built from combinations of those skills. The question is whether experience becomes a resource for new behaviour, and whether that resource grows as more examples are provided.
Basic and advanced tasks have a controlled relationship
The benchmark contains two hundred and sixty-eight tasks: one hundred and sixty-eight basic tasks and one hundred advanced tasks. Basic tasks span six activity categories. Advanced tasks combine atomic actions from two to four basic categories, with longer required sequences. The explicit relationship gives the experiment a way to test recombination rather than merely comparing unrelated easy and hard tasks.
The researchers compare zero-shot execution, in-context learning from full interaction trajectories and skill learning from concise written experience. This distinguishes retaining the raw history from retaining an extracted representation. Both can help, but they impose different context costs and different demands on the agent's ability to decide what should be remembered.
The gain does not scale with the experience supply
Across the tested harnesses and learning approaches, basic-task experience improves advanced-task performance relative to difficult zero-shot execution. The benefit remains when compositional complexity increases. Yet providing more learning samples does not produce a corresponding improvement. The authors conclude that current agents struggle to extract additional meaningful information from the extra experience.
There is also no consistent winner between full-history in-context learning and skill summaries in the reported discussion. That is important because it resists a convenient claim that compression alone solves the bottleneck. A concise skill can be useful, but the evaluated systems have not demonstrated an effective mechanism for turning an expanding experience pool into steadily improving competence.
Experience changes action selection even when scaling stalls
The analysis reports that prior experience helps agents choose more meaningful actions and fewer unexecutable ones. This offers a behavioural signal beneath the final success rate: useful transfer can appear in how the agent navigates the action space before it completes an advanced task. It does not establish why additional samples stop helping.
The complexity analysis extends advanced tasks from three to six atomic actions. Stable relative benefits across that range are encouraging within the tested space, but should not become a claim that task difficulty never matters. More complex plans, unfamiliar environments and differently structured skills can create other limits. The useful result is that the observed experience plateau cannot simply be dismissed as a consequence of testing only one easy composition.
A memory-capacity story is not enough
If added examples fail to help, several mechanisms remain possible: retrieval may choose poorly, summaries may omit relevant conditions, or planning may fail to apply a useful lesson. A final success score does not identify which component is responsible. Separate tests of lesson extraction, selection and execution are needed before attributing the plateau to one cause.
The benchmark operates in simulation and within a defined household task space. It does not establish a universal ceiling on learning from experience. It makes the scaling claim testable. An agent that benefits from a little prior experience is promising; an agent that can convert substantially more experience into substantially better transfer is a different achievement. EmbodiedCLBench shows why that second step should be measured explicitly rather than assumed from the existence of a growing memory.
References
Questions, news tips or corrections? Contact [email protected].
