Restricting retrieved documents to a historical date does not remove later facts stored in a model's weights. Yet finding those facts in a closed-book probe does not establish how they influenced an agent's investigation.
An October 13 study pairs historical company-comparison tasks with recall probes and identity interventions. It finds future-fact accessibility, but not the prespecified shift in evidence gathering toward the better-remembered company. Some observed patterns run in the opposite direction. Read the research paper.
A historical cutoff cannot reach inside model weights
Point-in-time agents are meant to reason using evidence available at a specified historical date. Restricting tools and documents can enforce that rule externally, but a pretrained model may still recall facts from later years. Detecting such recall establishes an information-integrity concern. It does not, by itself, show how the information changed the agent's search or final decision. Those are separate behavioural claims.
The study investigates the distinction through historical company comparisons. It pre-specifies a predicted search direction from independent recall probes, then tests whether normal company identities steer evidence gathering differently from anonymised identities while the external evidence remains historically valid.
The predicted mechanism was fixed before the runs
The task pool draws on public SEC filings and revenue observations, with forty-eight policy tasks selected before agent execution. Closed-book revenue probes identify the better-remembered company on tasks where recall is unequal. That defines the hypothesised direction: if future memory draws investigation toward familiar outcomes, the better-remembered company should receive more evidence gathering.
Paired normal and anonymised runs provide an intervention on identity presentation. They are not a perfect switch for future knowledge. The full paper notes that company identities can remain recoverable from anonymised content, and changing presentation can affect other associations. This limits how specifically a difference between the runs can be attributed to future-fact memory.
The observed search pattern resists the easy story
Both Qwen and DeepSeek recall future facts, but neither supports the pre-specified shift toward the better-remembered company. Qwen's allocation estimates on twenty-eight unequal-recall tasks are negative with intervals spanning zero. DeepSeek shows an inverse pattern on twenty such tasks. The authors investigate the possibility of search substitution: a model may investigate less because it already has an answer available, rather than more because the company is familiar.
A prospective follow-up explicitly supplies future facts. Estimated search shifts are small and in the withdrawal direction, with intervals spanning zero, while final choices move toward the supplied-fact company. Search allocation and decision output therefore need not respond in the same way. The mechanism behind the natural inverse pattern remains unresolved.
The selected tasks share companies and future facts
The frozen pool contains one hundred and ninety-four tasks covering nineteen companies, thirty-four unordered pairs and fiscal years 2018–2024. Forty-eight policy tasks are selected by a deterministic stratified rule before the agent runs. Repeated companies, pairs and future revenue facts create dependence among observations, a limitation the authors discuss explicitly.
The confidence intervals are therefore descriptive summaries of this evaluated task set, not a licence to assume independent draws from all historical financial decisions. The DeepSeek follow-up is prospective, which strengthens the test of the pre-specified direction, but does not remove every selection or identity confound. These design choices help explain why the paper preserves an unresolved mechanism rather than turning the inverse pattern into a new universal theory of search.
Audit the stages without merging their meanings
The evaluated setting covers two model families and a small historical-finance task set with repeated companies and facts. Its estimates should not be treated as a universal law of contamination. Nor does an unexpected search direction certify that future information is harmless. A model can use later knowledge in its answer even when it does not gather evidence in the hypothesised manner.
For historical-agent evaluation, the useful framework is to inspect external evidence integrity, closed-book accessibility, search behaviour and final claims separately. A finding at one stage motivates tests at the others; it does not replace them. The paper's contribution is a more demanding causal standard for leakage explanations, including the willingness to preserve an unresolved result when a plausible story fails its pre-specified behavioural test.
Questions, news tips or corrections? Contact [email protected].
