Textual memory gives agents a cheap way to retain experience without changing their weights. After a task, the model writes a lesson; before the next one, it rereads a playbook. The appealing simplicity hides a selection problem. A plausible lesson may describe an incidental feature of one success, or preserve a mistaken diagnosis of one failure.
Randomized Block Curation evaluates lessons by their effect on later behaviour. Instead of asking the model which advice sounds valuable, it treats the availability of advice as an experimental intervention. The research brings a familiar principle of causal measurement to a part of agent design often governed by the same model’s own judgement.
Include or withhold advice at random
An October 13 study finds large variation in self-written playbooks and proposes Randomized Block Curation. Related lessons are included or withheld at random on practice tasks, allowing their effect on success to be estimated. Curated playbooks outperform the unfiltered and agent-selected alternatives in the reported evaluations. Read the research paper.
The method groups related lessons into blocks, then independently includes or omits blocks on practice tasks. Randomisation makes the groups of tasks exposed to each block comparable, allowing a success-rate difference to estimate the block’s contribution. Grouping provides enough combined effect to measure when individual lessons would be too subtle.
The framework can grow several playbooks and select among curated, whole or absent memory using fresh practice episodes. That last option matters. It allows the system to conclude that the best available memory is no playbook at all, rather than assuming every accumulation of lessons must improve performance.
Whole-playbook selection captures interactions the blocks miss
Randomising lesson blocks estimates their effects over the practice distribution and the combinations presented. It does not automatically identify every interaction among instructions. Two individually useful blocks might conflict, or one might work only when another supplies a prerequisite. Evaluating the resulting complete playbook on fresh episodes is therefore more than a ceremonial final check.
The framework also grows several playbooks because the same memory-writing procedure can produce helpful or harmful advice. This recognises variation in the constructor, not only uncertainty in the selector. An agent’s memory pipeline has at least two questions to answer: what lessons were generated, and which combination should be served? The no-playbook candidate supplies a useful baseline for both. It prevents the procedure from treating the existence of a memory artifact as proof that the artifact deserves influence over the next task.
Self-evaluation is not a substitute for an intervention
Across two memory systems, three models and three benchmarks, the authors report average success of 66.9 with the proposed approach, compared with 55.5 using the whole playbook, 54.1 with no playbook and 53.8 when the agent selects lessons itself. These results support curation in the tested tasks, not an assumption that every lesson can be cheaply assigned a stable causal value.
Blocks can interact, and a group-level effect can conceal a harmful individual instruction. Selection on practice episodes can also overfit if those episodes are reused as final evidence. Fresh evaluation remains necessary even when the curation procedure itself is randomised.
Learning should include permission to forget
An organisation may be tempted to measure agent improvement by the number of reusable lessons saved. This study suggests a better measure: how much held-out success changes when the retained advice is available. A smaller playbook can be more useful if it removes confident but unsupported generalisations.
The wider significance is that self-written memory can be subjected to external behavioural tests. That creates a path between blindly trusting reflection and abandoning textual learning altogether. Advice earns persistence through evidence of usefulness, with its scope tied to the tasks on which that usefulness has actually been measured.
Questions, news tips or corrections? Contact [email protected].
