Data selection methods increasingly inspect a model’s internal activity to decide which examples deserve training time. The attraction is clear: the model’s own responses may reveal lessons that surface text statistics miss. But an internal measurement does not interpret itself. A large signal and a widely distributed signal can identify very different samples.
The neuron-relevance study compares those interpretations using the same underlying attribution profiles. This isolates an often hidden choice in sophisticated curation systems: the rule that compresses a high-dimensional internal explanation into a single ranking. The sophistication of the measurement does not make that aggregation choice inevitable.
Strength and support produce different rankings
An October 12 paper compares effective relevance support with positive relevance magnitude using attribution profiles. The criteria rank examples differently, and support-based selection performs well across the authors' evaluated models and benchmarks. Read the research paper.
Using AttnLRP-guided attribution, the authors construct sample-level relevance profiles. Positive Relevance Magnitude measures average positive strength; Effective Relevance Support measures how broadly positive relevance is distributed. Both start with the same kind of profile, but ask different questions of it.
On Qwen2 models, magnitude has stronger associations with activation-based selection, while support yields substantially different rankings. The differences extend to task composition and preferences among examples within a task. A selector is therefore not merely finding an objective list of important samples; it is operationalising a particular meaning of importance.
Within-task preferences can change without a topic shift
A selector might produce a different dataset simply by preferring different tasks. The paper also examines preferences within tasks, where the distinction between relevance support and magnitude still changes which examples rank highly. This matters because it shows that the aggregation choice is not reducible to a broad topic-balancing policy.
Two instructions on the same task can activate different patterns in the recipient model. One may strongly reinforce a narrow feature, while another draws on a broader collection. Which is useful depends on the adaptation objective and existing training set. The attribution profile is evidence about a computation, not a complete forecast of future learning. Downstream subset experiments provide the needed behavioural check. For a curation team, inspecting selected examples within each task can reveal whether a sophisticated ranking is buying new coverage or merely changing the mix of familiar instruction styles.
Selection has to survive downstream training
Across three language models and five downstream benchmarks, the paper reports practical value from relevance-guided selection, with support-based selection obtaining the best overall average performance rank among compared methods. That training result is important because an attribution ranking alone would not demonstrate useful curation.
The evidence still depends on the backbones, datasets and selection budgets. A sample that engages many features may be useful coverage, but broad activity is not a universal certificate of learning value. Agreement across backbones varies, so transfer of a ranking should be measured rather than inferred from its neuron-level origin.
The aggregation rule is part of the hypothesis
A useful audit would keep the attribution engine fixed, vary the aggregation rule and compare not just final averages but the types of examples and failures each subset preserves. That makes the selection hypothesis visible: is the aim to intensify existing strengths, cover more features or address a known blind spot?
The larger implication reaches beyond neuron relevance. Any pipeline that collapses a rich signal into a score embeds assumptions about what future training needs. Making those assumptions explicit can be more valuable than adding another complicated internal statistic whose ranking is accepted without testing why it should identify the right lessons.
Questions, news tips or corrections? Contact [email protected].
