An assistant answering “I have forgotten that” has made a claim, not demonstrated deletion. The information may remain extractable, or the system may have removed unrelated memories as well.
Bemula's October 13 paper pairs dialogues with and without forget requests, then tests removal, preservation, relearning and resistance to extraction. The evaluated memory systems perform poorly under its demanding end-to-end criterion. Read the research paper.
Forgetting is a selective memory operation
Long-term assistants accumulate facts about people and their changing circumstances. When a user asks to remove one fact, the request is more precise than resetting the whole relationship. The system must stop making the targeted information available while retaining unrelated knowledge and remaining able to learn similar information later. A polite acknowledgement tests none of these operations.
Bemula evaluates forgetting as an end-to-end capability. It is motivated by a gap in memory benchmarks that reward writing, retrieval and organisation while leaving explicit removal largely untested. The benchmark asks whether memory systems can honour a change in what should be retained, rather than only improve how much they can recall.
The control verifies that there was something to forget
The dataset contains fifty multi-session dialogue instances, one hundred and thirty-five forget requests across six types, and 1,742 test questions. Every dialogue has a version without the removal request. A system can earn credit for forgetting only if the paired control shows that it stored the target in the first place. Otherwise a weak memory could look successful simply because it never learned the information.
Five test suites then assess storage, requested removal, preservation of unrelated memories, relearning of similar facts and resistance to extraction attacks. This turns forgetting into a conjunction of requirements. A system that deletes everything may remove the target but fail preservation; one that suppresses a direct answer may still fail when asked indirectly.
The failures survive different memory architectures
The researchers evaluate six memory systems and three backbone models, with full-context baselines where context limits permit. The systems include retrieval, fact extraction, compression and structured note-taking approaches. In the benchmark's demanding end-to-end criterion, no evaluated system passes more than 1.5 percent of forget requests, and most pass none. The authors attribute failures to information remaining extractable or to deletion spreading beyond what was requested.
The result is severe, but its scope should remain clear. These are controlled dialogues and specific implementations evaluated by the study, not a complete audit of every deployed memory product. The benchmark's judge and attack construction are also part of the measurement system.
Fresh sessions test the memory system rather than conversational residue
Before each instance, the experiment resets the system. It ingests the control or forget dialogue turn by turn, then answers evaluation questions in a fresh session. That prevents the assessment from relying only on the immediate conversation in which deletion was acknowledged. The memory mechanism has to produce the changed availability later.
The six implementations include Naive RAG, Mem0, A-MEM, LangMem, LightMem and SimpleMem. Context limits exclude some backbone-and-system combinations, which the paper states rather than filling in missing results. It also uses a model judge and bootstrap procedures clustered by dialogue instance. These details matter because many test questions arise from the same underlying relationship: they are not independent examples of completely different users and deletion histories.
Removal can fail through derived memory
An agent may copy one fact into summaries, extracted notes or linked representations. Deleting an obvious record is insufficient if another representation still supports retrieval. Conversely, removing an entire cluster can destroy useful unrelated facts. The practical test must follow the information's behaviour, not merely inspect whether one database row disappeared.
Paired pre-removal and post-removal tests should include indirect questions, extraction attempts, unrelated personal queries and later relearning. Memory design needs revision and deletion primitives alongside ingestion. A system optimised to preserve every useful connection has not automatically learned how to cut exactly one connection when the user asks. That precision is a separate capability, and Bemula makes its absence difficult to hide behind ordinary recall accuracy.
References
Questions, news tips or corrections? Contact [email protected].
