A tool argument may be correct because it follows from a default or several earlier facts. A judge receiving only the final request may reject it because the derivation is absent.
An October 13 study calls this the derivation gap. Controlled source relocation shows judges favouring visible references, including incorrect ones. Its DeRef method constructs a reference from the full history without seeing the candidate argument. Read the research paper.
A judge needs enough evidence to disagree correctly
Tool-call verification is often treated as a second opinion: another model checks whether the proposed argument value is right. But the judge may see less evidence than the acting agent. A correct value can come from a schema default, an inference across earlier turns or a passage omitted from the judge's input. Without that source, the second model may reject a sound decision while favouring an incorrect value stated explicitly nearby.
The DeRef study calls this the derivation gap. It examines not only whether judges make mistakes, but how the visibility of a reference changes their acceptance behaviour. The problem is informational as well as evaluative.
Move the reference and observe the preference
Across seven judges, correct values are accepted less often when their input lacks a textual reference. The effect is not merely a consequence of truncation: defaults show the largest gap on a benchmark without truncation. A controlled relocation test makes the problem sharper. Judges favour a visible incorrect value over a correct value whose source is hidden.
That intervention distinguishes checking truth from matching available evidence. A model can appear strict and consistent while rewarding the answer that is easiest to justify from its limited view. More confident judging does not solve an input that makes the correct derivation inaccessible.
Derive a reference without seeing the candidate
DeRef uses a separate model to read the full history and write a reference for the judge. The deriver does not see the proposed argument, reducing the opportunity to rationalise that candidate. Across three benchmarks, the method improves argument-level AUROC for all tested judges. On the controlled test, the reported AUROC rises from chance or below to 0.9840–0.9959.
Ablations show that writing the reference from the judge's cut input rather than the full history causes the largest loss in the reported comparison. The text of the reference also helps where a parser cannot extract an explicit expected value, explaining why the framework retains a judge instead of reducing verification entirely to string matching.
A correct alternative can look wrong under a single-value label
Each benchmark argument has one gold value. If another value is also valid, a reference naming the gold can encourage a judge to reject the alternative. The study also reports uncertainty in some constructed incorrect labels: an audit cannot confirm three percent in one dataset and four percent in another. Worst-case analyses retain the reported DeRef gains, but the uncertainty remains part of the evidence.
These limitations distinguish argument ranking from complete application correctness. A high AUROC means the scores separate labelled correct and incorrect cases well; it does not guarantee the right threshold for every tool or account for all semantically equivalent values. Practical verification still needs schema-aware handling of alternatives and explicit treatment of uncertain derivations.
The reference is another fallible component
The benchmark assigns one gold value to each argument, which can penalise another acceptable value. Some constructed labels also remain uncertain after audit. The paper reports sensitivity analyses and notes that DeRef does not close every no-source gap. Full-history processing and the extra model call add cost, while a mistaken reference can still mislead the judge.
For deployment, tests should separate explicit values, defaults, multi-step inferences and evidence lost through truncation. A second model becomes a useful check when it can inspect the relevant basis for a decision. A verifier cannot reliably challenge an answer if its context systematically hides why that answer might be correct. DeRef addresses that asymmetry while leaving reference quality itself open to examination.
References
Questions, news tips or corrections? Contact [email protected].
