UniGeM : Uni fying Data Mixing and Selection via G eometric E xploration and M ining Changhao Wang, Yunfei Yu, Xinhao Yao, Jiaolong Yang, Riccardo Cantoro, Chaobo Li, Qing Cui, Jun Zhou Abstract The scaling of Large Language Models (LLMs) is increasingly limited by data quality. Most methods handle data mixing and sample selection separately, which can break the structure in code corpora. We introduce UniGeM , a framework that unifies mixing and selection by treating data curation as a manifold approximation problem without training proxy models or relying on external reference datasets. UniGeM operates hierarchically: Macro-Exploration learns mixing weights with stability-based clustering; Micro-Mining filters high-quality instances by their geometric distribution to ensure logical consistency. Validated by training 8B and 16B MoE models on 100B tokens, UniGeM achieves 2.0 × \times data efficiency over a random baseline and further improves overall performance compared to SOTA methods in reasoning-heavy evaluations and multilingual generalization. 1 Introduction The generalization of Large Language Models (LLMs) has traditionally relied on scaling parameters and data volume ( 17 ) . However, recent shifts in scaling laws suggest that data quality now constrains model performance more than raw quantity ( 14 ; 9 ) . As high-quality public corpora are nearing depletion ( 33 ) , simply aggregating noisy web data yields diminishing returns ( 12 ) . This motivates a focus on data efficiency : finding subsets that give more gain per compute ( 21 ; 31 ) . Figure 1: Average downstream performance: Random sampling vs. UniGeM for pre-training an 8B MoE model. The inset radar chart shows that UniGeM achieves stronger multilingual performance across programming languages. Most data efficiency approaches fall into two buckets: domain mixing and instance selection ( 36 ; 37 ; 24 ) . Data mixing serves as a macro-distribution balancing mechanism ( 5 ) , but its coarse granularity often treats domains as flat distributions and overlooks structure within each domain ( 6 ) . Meanwhile, many selection methods require training proxy models or reference datasets, which adds substantial overhead and may not track the target model’s behavior at scale ( 20 ; 37 ; 22 ) . Other selection methods, including heuristics and LLM-based scoring, evaluate samples independently and ignore the underlying manifold structure ( 41 ; 35 ) . This decoupling leads to structural blind spots. We either optimize macro-distribution without considering micro-quality, or filter samples at the cost of disrupting the hierarchical dependencies. This matters most for the code corpus , a rigid logical manifold defined by brittle syntax and hierarchical dependencies ( 21 ; 13 ; 8 ) . We thus ask: How can we unify macro-distribution balancing and micro-quality selection to identify the "golden subset" within a structured code corpus? To bridge this gap, we introduce UniGeM (Geometric Exploration and Mining) , a framework that unifies macro-distribution balancing and micro-quality selection. Unlike existing methods that rely on external reference datasets for alignment or require training expensive proxy models to estimate data importance, UniGeM views these tasks as a unified manifold approximation problem. First, Macro-Exploration (Stage-I) discovers semantic manifolds via stability-driven clustering to optimize macro-distribution balancing. Subsequently, Micro-Mining (Stage-II) performs micro-quality selection using intrinsic geometric priors to capture structural and logical dependencies. By measuring manifold deviations, UniGeM distills a golden subset that serves as a faithful approximation of the data manifold, preserving global topology and local dependency structure. We validate UniGeM by training 8B and 16B Mixture-of-Experts (MoE) models (both with 1.4B active parameters) from scratch on a 100B-token code-and-text mixture. Results show that UniGeM achieves superior data efficiency and model performance compared to existing baselines. Our main contributions are: • Unified Framework: We propose a geometry-centric framework to unify macro-distribution balancing and micro-quality selection via manifold approximation. • Proven Data Efficiency: UniGeM achieves 2.0\times data efficiency (vs. a random baseline) and performs better after one epoch. • Broader Gains: UniGeM improves overall performance over SOTA baselines and shows stronger multilingual generalization. • Topological Insights: Our ablations suggest that coverage and local structure matter for reasoning beyond per-sample quality scores, consistent with a manifold-approximation view of curation. 2 Proposed Method: The UniGeM Framework In this section, we detail the implementation of UniGeM and provide a theoretical analysis to demonstrate effectiveness in data curation. Figure 2: Overview of the UniGeM framework. The curation pipeline operates in two hierarchical stages: (Left) Stage-I (Macro-Exploration) identifies the intrinsic manifold resolution K^{*} via topological stability (Algorithm 1 ) and derives global mixing weights \mathbf{r} through a softmax over geometric scores (Eq. 5 ), ensuring comprehensive global coverage across diverse semantic regions. (Right) Stage-II (Micro-Mining) computes sub-cluster sampling weights W(S_{j}) by combining the inherited budget r_{k} , a semantic score modulated by a structural penalty \exp(-\lambda\mathcal{L}_{struct}(S_{j})) (Eq. 6 ), and a cohesion gate [\beta_{S_{j}}+\epsilon] (Eq. 7 ). This stage refines selection within each region to downweight off-manifold outliers while preserving representative local structural dependencies. 2.1 Problem Formulation and Overview Let \mathcal{D}_{raw}=\{x_{i}\}_{i=1}^{N} denote the uncurated corpus. We employ an embedding model f_{\theta} to map each sample x_{i} to a normalized feature vector \mathbf{e}_{i}\in\mathbb{S}^{d-1} . We use this normalized space as a practical latent manifold \mathcal{M} for clustering and selection. We cast data selection not as distributional alignment to an external reference, but as a self-contained manifold approximation problem. Our objective is to identify a subset S\subset\mathcal{D}_{raw} that maximally preserves the topological structure of \mathcal{M} while suppressing off-manifold outliers. Table 1 summarizes the key notations utilized throughout this framework. Table 1: Summary of Key Notations and Symbols. Symbol Definition I. Macro-Exploration (Stage-I) C_{k} Global semantic cluster. \mathbf{z}_{k},s_{k} Geometric features and score. \mathbf{w} Spectral consensus weights. K^{*} Optimal manifold resolution. T_{scale} Alignment scaling factor. r_{k} Global mixing budget. II. Micro-Mining (Stage-II) S_{j} Granular sub-cluster within C_{k} . P_{S_{j}} Semantic scores via probes. \mathcal{L}_{\text{struct}} Structural penalty. \beta_{S_{j}} Geometric cohesion gate. W(S_{j}) Final sampling weight. III. Manifold Theory & Approximation \mathcal{M},d Latent manifold and dimensionality. \mathcal{E}(S) Wasserstein-2 approximation error. \sigma_{k}^{2} Intra-cluster variance of C_{k} in latent space. \alpha_{k} Cluster mass (probability) \mu(C_{k}) . \Delta_{gain}^{(k)} Variance mass rejected by Stage-II pruning. 2.2 Stage-I: Macro Exploration (Global Clustering and Weighting) As illustrated in Figure 2 (Left) , Stage-I performs the Macro-Distribution Balancing . This phase transforms the raw manifold into structured semantic clusters and assigns a sampling budget to each cluster. The process consists of three sequential steps: Geometric Metrics and Scoring. First, we characterize each candidate cluster C_{k} via a feature vector \mathbf{z}_{k}\in\mathbb{R}^{4} . These dimensions act as geometric proxies: • Cohesion ( z_{\text{coh}} ): Inverse intra-cluster distance. It prioritizes tight semantic structures akin to Neural Collapse ( 26 ) to ensure gradient stability. • Cluster Size ( z_{\text{size}} ): Sample volume. Used to mitigate head redundancy (e.g., boilerplate) and shift focus to the information-dense long tail ( 15 ) . • Sequence Length ( z_{\text{len}} ): The average token count, a proxy for verbosity, used to downweight overly long, low-information clusters. • Entropy ( z_{\text{ent}} ): Distributional impurity of language identifiers. It penalizes semantic ambiguity to ensure domain purity. To unify these proxies, we combine the normalized signals with a weighted linear score to derive a scalar Geometric Score s_{k} . This score acts as a global quality metric, formulated as a contrast between structural coherence and statistical instability: s_{k}=w_{\text{coh}}\tilde{z}_{\text{coh}}-\sum_{f\in\mathcal{F}_{neg}}w_{f}\tilde{z}_{f} (1) where \mathcal{F}_{neg}=\{\text{len, ent, size}\} denotes the set of negative factors, and \tilde{z} represents the Z-normalized magnitudes. The weighting vector \mathbf{w} is derived via Spectral Consensus (see Appendix A ). Intrinsic Resolution Selection. Central to this is a Cross-Resolution Soft Alignment . We construct a probabilistic bridge \pi_{K\to K^{\prime}} based on centroid similarity: \pi_{K\to K^{\prime}}(j|i)=\text{Softmax}\left(T_{scale}\cdot\cos(\mu_{i}^{(K)},\mu_{j}^{(K^{\prime})})\right) (2) Using this bridge, we project the geometric scores \mathbf{s}^{(K^{\prime})} (derived in Eq. 1 ) from the finer resolution back to the current layer to obtain the reconstructed scores \mathbf{\hat{s}}^{(K)} : \hat{s}_{i}^{(K)}=\sum\nolimits_{j}\pi(j|i)s_{j}^{(K^{\prime})} (3) Stability is quantified via rank correlation using a Kendall’s \tau (detailed formulation in Appendix B ): J_{stab}(K)=\tau\left(\mathbf{s}^{(K)},\,\mathbf{\hat{s}}^{(K)}\right) (4) The resolution K^{*} that maximizes this stability is selected as the intrinsic resolution (macro-granularity) via Algorithm 1 . 0: Manifold \mathcal{M} , Range \mathcal{K} , Stride \Delta K . 0: Optimal resolution K^{*} . 1: \mathcal{S}\leftarrow\emptyset 2: for K\in\mathcal{K} do 3: K^{\prime}\leftarrow K+\Delta K 4: 1. Extract: Compute centroids \mathcal{C}^{(K)} and scores \mathbf{s}^{(K)} . 5: 2. Align: Construct soft bridge \pi_{K\to K^{\prime}} (Eq. 2 ). 6: 3. Evaluate: Calculate rank stability J_{stab}(K) (Eq. 4 ). 7: \mathcal{S}\leftarrow\mathcal{S}\cup\{(K,J_{stab}(K))\} 8: end for 9: return K^{*}=\underset{K}{\arg\max}\{J:(K,J)\in\mathcal{S}\} Algorithm 1 Intrinsic Resolution via Soft-Alignment Sampling Budget Allocation. With the optimal resolution K^{*} established, we finally allocate the sampling budget \mathbf{r} . Leveraging the geometric scores from Eq. ( 1 ), the probability for each cluster is computed via a standard softmax: r_{k}=\frac{\exp(s_{k})}{\sum_{j=1}^{K^{*}}\exp(s_{j})} (5) This framework ensures the budget concentrates on semantically dense regions (high s_{k} ). 2.3 Stage-II: Micro-Mining (Local Sub-Clustering and Refinement) As illustrated in Figure 2 (Right), Stage-II performs the Micro-Quality Selection via local sub-clustering. This phase decomposes each global cluster C_{k} into fine-grained sub-clusters \{S_{j}\} to capture semantic diversity and topological details. The selection is refined through three coupled mechanisms: Probe-based Semantic Scoring. To assess semantic content efficiently, we extract a small representative Probe Set (indicated by the solid centroids in Figure 2 ) from each sub-cluster S_{j} . We utilize an LLM as a Knowledge Probe to derive a Semantic Score P_{S_{j}} . Implementation details are provided in Appendix D . Relative Structural Consistency. To enforce manifold consistency, we define a Structural Penalty \mathcal{L}_{struct} using a rectified Mahalanobis distance. It penalizes sub-clusters S_{j} where the empirical mean z_{f}(S_{j}) of features (e.g., length, entropy) exceeds the geometric consensus (\mu_{f},\sigma_{f}) of the parent cluster C_{k} : \mathcal{L}_{struct}=\sum_{f\in\{len,ent\}}\left[\frac{z_{f}^{(S_{j})}-\mu_{f}^{(C_{k})}}{\sigma_{f}^{(C_{k})}}\right]_{+}^{2} (6) We introduce \mathcal{L}_{\text{struct}} not only to ensure that the curated subset preserves the underlying manifold topology but also to mitigate the score saturation observed in semantic scores (Fig. 10 and Fig. 11 ). Geometric Cohesion Gate. To ensure topological robustness, we apply a Geometric Cohesion Gate \beta_{S_{j}} , depicted as the gating module in Figure 2 . Leveraging the cohesion metric z_{coh} defined in Stage-I, we modulate sampling confidence based on the sub-cluster’s compactness relative to its parent cluster: \beta_{S_{j}}=\text{Sigmoid}\left(z_{coh}^{(S_{j})}-z_{coh}^{(C_{k})}\right) (7) This gating mechanism suppresses sub-clusters with lower cohesion than their parent ( z_{coh}^{(S_{j})}0 (15) Discordant \displaystyle:(s_{i}-s_{j})(\hat{s}_{i}-\hat{s}_{j})<0 (16) Let N_{conc} and N_{disc} denote the total counts of such pairs: \displaystyle N_{\text{conc}} \displaystyle=\sum_{i