If an agent improves after seeing past attempts and rewards, it seems natural to credit the rewards. Remove or scramble them, however, and that explanation becomes testable.
An October 8 paper examines direct in-context reinforcement learning across six models and four benchmarks. In the tested setting, altering or removing reward values leaves improvement curves largely unchanged. Corrupted trajectories can also produce similar effects. Read the research paper.
Experience can help for the wrong reason
Agent developers increasingly preserve previous attempts inside a prompt and describe the resulting improvement as learning from experience. The description sounds reasonable when each attempt comes with a reward. But a sequence containing rewards is not automatically a reinforcement-learning procedure. The causal question is whether changing the reward changes what the model learns to do. Without that check, familiar prompt effects can acquire an explanation they have not earned.
This study examines the simplest version of the claim: direct conditioning on raw trajectory-and-reward pairs. It does not ask whether context can be useful. It asks which part of that context is responsible for the improvement attributed to it.
Intervene on the supposed learning signal
The researchers vary rewards while holding the broader setup under control. They flip, randomise or remove reward values, and examine whether the improvement curve changes. Across four benchmarks and six models, it remains largely similar. Explicit prompts telling the model to explore, exploit or reason over rewards do not establish the expected reward-sensitive behaviour in this setting.
They also disturb the trajectories themselves. Shuffled or corrupted trajectories can work as well as genuine ones in the reported tests. That result makes the interpretation more demanding: simply observing performance gains after experience is insufficient evidence that the model extracted the semantic lesson an agent designer intended to supply.
Reading a reward differs from learning from it
A model may notice that a reward token is present without using its value to favour better actions. Formatting, demonstration structure or changes in the input distribution can influence outputs independently of the stated feedback. The authors find patterns resembling ordinary in-context learning and argue that direct in-context RL is better understood within that framework.
For experts, the surprise is methodological. The negative control is simple enough that it can be omitted precisely because the proposed mechanism feels obvious. Yet reward reversal is the kind of intervention that distinguishes a narrative about learning from evidence for it. A better score after more context cannot make that distinction on its own.
Negative feedback does not reliably create avoidance
The full paper also examines whether failed trajectories teach an agent not to repeat an action. Negative credit has little effect in the reported tests, including a prompt explicitly encouraging a different action. This is a more specific failure than not improving the total score: the feedback does not reliably reverse the attraction of an example already present in context.
That matters when prior model preferences conflict with the desired new behaviour. High-reward examples aligned with familiar actions may make direct in-context reinforcement learning appear effective, while negative rewards struggle to redirect it in a genuinely unfamiliar setting. The finding motivates explicit avoidance tests, not merely demonstrations that a positive example can be copied. It also explains why failure histories should not be assumed to function as a parameter-free policy update.
Where the conclusion should stop
These results concern the direct raw-pair setting studied in the preprint. An architecture trained specifically for reward-sensitive adaptation, a system that writes structured lessons, or an agent that updates parameters could behave differently. Extending the finding to all inference-time learning would repeat the same mistake of moving from an observed pattern to an overly broad mechanism.
For a deployed agent, the next useful audit is to keep its experience fixed and alter only the feedback. Then test whether the chosen actions move in the expected direction. If they do not, improving the prose of the reward rubric may be a distraction. Retrieval policy, demonstration selection and the structure of the prompt may be the components actually controlling performance.
Questions, news tips or corrections? Contact [email protected].
