Target-Oriented Pretraining Data Selection via Neuron-Activated Graph
Zijun Wang, Haoqin Tu, Weidong Zhou, Yiyang Zhou, Xiaohuan Zhou, Bingni Zhang, Weiguo Feng, Taifeng Wang, Cihang Xie, Fengze Liu · 2026
Read Target-Oriented Pretraining Data Selection via Neuron-Activated Graph by Zijun Wang, Haoqin Tu, Weidong Zhou, Yiyang Zhou, Xiaohuan Zhou, Bingni Zhang, Weiguo Feng, Taifeng Wang, Cihang Xie, Fengze Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15706.txt.
Introduction
Wang and nine colleagues pick target pretraining data with Neuron-Activated Graph Ranking, a no-training, see-through scheme. Each target gets a set of high-hit nerve cells from any LLM, drawn as a graph, and data rank by graph likeness to target cases. The scheme lifts target pretraining by 4.9 percent over random picks and tops past bests by 5.3 points on HellaSwag. Knocking out its picked cells, just 0.12 percent of all, drops scores 23.5 percent, showing a thin working spine for learning target traits.
Catalog record
| CREATOR | Zijun Wang, Haoqin Tu, Weidong Zhou, Yiyang Zhou, Xiaohuan Zhou, Bingni Zhang, Weiguo Feng, Taifeng Wang, Cihang Xie, Fengze Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67126 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.15706
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Target-Oriented Pretraining Data Selection via Neuron-Activated Graph Zijun Wang, Haoqin Tu, Weidong Zhou, Yiyang Zhou, Xiaohuan Zhou, Bingni Zhang, Weiguo Feng, Taifeng Wang, Cihang Xie, Fengze Liu Abstract Everyday tasks come with a target, and pretraining models around this target is what turns them into experts. In this paper, we study target-oriented language model (LM) pretraining by introducing N euron- A ctivated G raph Ranking (NAG-based Ranking), a training-free and interpretable framework for target pretraining data selection. Rather than using black-box representations, our approach directly characterizes each target input by a sparse set of high-impact neurons in any off-the-shelf LLMs. Concretely, we quantify neuron impact and select the most influential neurons across layers into a compact N euron- A ctivated G raph (NAG), and rank candidate data by NAG similarity to target examples. We conduct experiments across six benchmarks, where our NAG-based Ranking improves target-oriented pretraining by 4.9% on average over random sampling, and also outperforms state-of-the-art baselines by 5.3% accuracy on HellaSwag. It also remains effective under a more applicable multi-target setting, where our best setup surpasses two baselines by 1.1% and 4.1%, respectively. Furthermore, we provide a comprehensive analysis on why and how our NAG works, e.g . , deactivating NAG-selected neurons (only 0.12% of all) causes a 23.5% performance collapse, and restricting NAG to the final layer incurs a 4.1% average drop, indicating that NAG captures a sparse “functional backbone” for learning target features. We release the code at https://github.com/asillycat/NAG . Figure 1 : General quality-based data selection is often misaligned with specific downstream capabilities ( left ), while prior target-oriented methods rely on shallow similarity to target examples ( middle left ). Our NAG instead aligns pretraining data with target tasks by selecting inputs that activate similar neurons in the LLM, capturing the underlying capability required for the target ( middle right ), even across different domains ( e.g . , economics vs . math). Figure 2 : Overview of Neuron-Activated Graph (NAG) target-oriented data selection. Given a small set of target examples D_{target} , we first characterize each input by its neuron-level NAG features. For a given input, we quantify the impact of individual neurons and select top- K of them per layer to construct a compact NAG. NAGs from the target examples are aggregated into a target neuron-activation profile. Each candidate sample c\in D_{pool} is then mapped to its own NAG and ranked by its similarity to the target profile, i.e . \text{Sim}(c,D_{target}) . Finally, the top- r_{f} ranked samples are selected for LLM pretraining. 1 Introduction Large language models (LLMs) have become increasingly prevalent in everyday tasks, and people usually use these models with specific targets in mind. Selecting high-quality pretraining data is one of the most effective ways to improve model performance within a target domain, yielding great gains ( 26 ; 24 ; 12 ; 34 ) . Despite its importance, what makes “high-quality” data remains surprisingly under-defined ( Figure 1 ). We argue that, in real-world settings, high-quality data should align with targeted scenarios that enable LLMs to acquire desired capabilities efficiently — education, medicine, or specific research domains — while ruling out other irrelevant factors that do not contribute to such capabilities ( 24 ) . However, existing pipelines for data selection make this alignment ambiguous. Many rely on heuristic rules ( 41 ; 29 ; 16 ; 1 ) or implicit assumptions about “quality” ( 30 ; 42 ; 26 ) , leaving a noticeable gap between how data are chosen and the specific capabilities the model ultimately needs to develop. Prior efforts to align LLM pre-training data with explicit targets show that target-oriented pre-training can yield substantial compute multipliers and consistent gains across different scales ( 24 ) . However, most of these approaches achieve task alignment by distilling models’ internal signals, such as embedding heuristics or performance-correlated losses, into auxiliary classifiers for scalability ( 24 ; 38 ; 32 ; 23 ) . This black-box distillation introduces an interpretability bottleneck: the learned signals are opaque and hard to diagnose or refine, thus constraining how effectively the signals can be leveraged for further performance gains. To solve this, we introduce N euron- A ctivated G raph Ranking (NAG-based Ranking) , a neuron-centric framework for target-oriented pretraining data selection. As illustrated in Figure 2 , our key idea is to characterize each text input by identifying which neurons in the LLM matter for processing it, rather than through black-box representations. Specifically, we first quantify neuron impact during inference in an off-the-shelf LLM ( Section 2.1 ) and then organize the most influential neurons across layers into a compact N euron- A ctivated G raph (NAG) ( Section 2.2 ). For the data selection, samples are ranked by the similarity of their NAGs to the target examples, prioritizing training inputs that trigger similar neuron patterns with the target data. Notably, our NAG-based Ranking requires no additional training and relies solely on interpretable signals from any off-the-shelf LLM. Empirical evaluations demonstrate our NAG-based Ranking consistently enhances target-oriented task performance across different settings and benchmarks. Compared to random sampling, NAG-based Ranking yields a substantial 4.9% average improvement in target-oriented pretraining, outperforming both strong classifiers focusing on general data quality ( 26 ) and state-of-the-art methods on target-oriented data selection like BETR ( 24 ) . Moreover, our algorithm presents more general use in the multi-target scenario where baselines often degrade, with the best setup outperforming two baselines by 1.1% and 4.1%, respectively. We also prove the broadened application of NAG by adding its signals to another quality-based data selection method, which further boosts scores. Finally, the performance gains of NAG remain consistent across various backbone models from 4.7% to 5.0%, indicating the robust and model-agnostic nature of NAG. Obtaining better performance is one aspect; explaining how and why NAG works makes our algorithm truly interpretable. We design dedicated experiments to probe from these two perspectives: (i) Why NAG works: we find that NAG works by identifying a sparse “functional backbone” of the LLM — deactivating just 0.12% of model neurons triggers a sharp 23.5% performance collapse ( Section 4.1.1 ). These high-impact neurons can well represent different target information via clustering ( Section 4.1.2 ) — NAG captures the true drivers of the desired targets. Finally, we show a high correlation between NAG-based rankings and the target learning utility ( Section 4.1.3 ). (ii) How NAG operates: we spot the path on how NAG works, specifically, it maps a “computational trajectory” by aggregating signals across all LLM layers. Our analysis shows that restricting NAG to the final layer results in a 4.1% average performance drop ( Section 4.2.2 ). Furthermore, by twitching model components where NAG signals come from, we further identify that extracting signals from the internal projections ( Section 4.2.1 ) at a very sparse neuron ratio ( Section 4.2.3 ) is crucial for capturing effective task-specific signals. 2 Method Prior works ( 25 ; 46 ; 47 ) suggest that model behavior is governed by a highly sparse subset of parameters, with different tasks relying on largely disjoint regions. As a result, inputs attributing to the same capability are expected to induce similar internal parameter usage patterns. Based