Neural Diversity Regularizes Hallucinations in Small Language Models

Kushal Chakrabarti, Nirmal Balachundhar · 2025

Read Neural Diversity Regularizes Hallucinations in Small Language Models by Kushal Chakrabarti, Nirmal Balachundhar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20690.txt.

Introduction

Kushal Chakrabarti and Nirmal Balachundhar argue that false outputs from language models can be cut by keeping parallel inner representations less alike, a trait they call neural diversity. Larger models and more data alone have not removed made up answers, so a different lever is needed.

The pair give formal bounds on false output chances for grouped models and frame the issue as a reliability problem tied to spread, not just average accuracy. Their ND-LoRA method trains side by side adapters with a decorrelation push and cuts false outputs by up to about a quarter while holding general accuracy. Tests show the two parts help each other, and the best diversity level varies by task.

Catalog record

CREATORKushal Chakrabarti, Nirmal Balachundhar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-20690

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Neural Diversity Regularizes Hallucinations in Small Language Models Kushal Chakrabarti, Nirmal Balachundhar Abstract Language models continue to hallucinate despite increases in parameters, compute, and data. We propose neural diversity — decorrelated parallel representations — as a principled mechanism that reduces hallucination rates at fixed parameter and data budgets. Inspired by portfolio theory, where uncorrelated assets reduce risk by P \sqrt{P} , we prove hallucination probability is bounded by representational correlation: ℙ ⁡ ( H ) ≤ f ⁡ ( σ 2 ​ ( ( 1 − ρ ⁡ ( P ) ) / P + ρ ⁡ ( P ) ) , μ 2 ) \mathbb{P}(\mathrm{H})\leq f(\sigma^{2}((1-\rho(P))/P+\rho(P)),\mu^{2}) , which predicts that language models need an optimal amount of neurodiversity. To validate this, we introduce ND-LoRA (Neural Diversity Low-Rank Adaptation), combining parallel LoRA adapters with Barlow Twins regularization, and demonstrate that ND-LoRA reduces hallucinations by up to 25.6% (14.6% avg.) without degrading general accuracy. Ablations show LoRA adapters and regularization act synergistically, causal interventions prove neurodiversity as the mediating factor and correlational analyses indicate scale: a 0.1% neural correlation increase is associated with a 3.8% hallucination increase. Finally, task-dependent optimality emerges: different tasks require different amounts of optimal neurodiversity. Together, our results highlight neural diversity as a third axis of scaling — orthogonal to parameters and data — to improve the reliability of language models at fixed budgets . Figure 1: Maximizing reliability requires an optimal amount of neural diversity. We vary parallel decorrelated representations ( P\in\{1,2,4,8\} ) and measure performance across 6 hallucination benchmarks. The x-axis shows distance from the performance-maximizing P_{\star} : \Delta P=P-P_{\star} . Aggregating 182,850 evaluation points (LOWESS fit, 80% bootstrap CI, 1000 bootstrap resamplings), the U-shaped curve shows performance improves rapidly with neural diversity, peaking at a specific neurodiversity optimum and then degrading slowly with excessive diversity. 1 Introduction Despite scaling to trillions of parameters, language models hallucinate persistently ( Lin et al. 2021 ) . This reliability crisis is acute for small language models (SLMs) — increasingly favored for edge deployment ( Chen et al. 2024 ) — whose compressed representations make them especially vulnerable to hallucinations, with even well-resourced efforts like GPT-OSS 20B exhibiting 91% hallucination rates on factual benchmarks ( OpenAI 2025 ) . Existing scaling methods — parameters, inference-time compute, and parallel architectures like ParScale ( Chen et al. 2025 ) — predominantly target accuracy (first-moment improvements in perplexity and task performance). Yet reliability (second-moment reduction in hallucinations and factual errors) remains largely unaddressed. While ParScale achieves O(\log P) performance gains with better memory and latency, it provides no explicit mechanism for reducing hallucinations. Without explicit regularization, representational collapse ( Jing et al. 2022 ) drives parallel streams toward identical features, leaving reliability gains unrealized despite added computational resources. Drawing from portfolio theory, we propose that decorrelated representations reduce hallucination risk just as uncorrelated assets reduce financial risk. When streams are perfectly correlated ( \rho=1 ), parallelism provides no benefit; when decorrelated ( \rho\to 0 ), signal-to-noise improves by \sqrt{P} . We formalize this through a neural diversity framework proving hallucination probability is bounded by cross-stream correlation: \mathbb{P}(\text{hallucination})\leq f(\sigma^{2}((1-\rho)/P+\rho),\mu^{2}) . This reveals why existing scaling approaches don’t address reliability and how explicit decorrelation provides a mechanism to reduce hallucinations at fixed parameter budgets. To validate this, we introduce ND-LoRA , combining independent LoRA adapters with Barlow Twins regularization ( Zbontar et al. 2021 ) . Despite adding only 0.004% more computation, ND-LoRA reduces hallucinations by up to 25.6% (14.6% avg.) while preserving general performance. Our experiments establish: (1) neural diversity causally reduces hallucination probability; (2) there is an optimal amount of neural diversity and that optimal amount is task-dependent; and (3) the relationship between neural diversity and hallucination probability is theoretically provable. Neural diversity represents a third scaling axis beyond parameters and data. While traditional scaling asks “how big?” and data scaling “how much?”, diversity scaling asks “how different?” — crucial for achieving reliability without massive computational investment. Our contributions are: • Theory : The first principled connection between neural spectral diversity and hallucination probability that matches empirically observed hallucination rates • Method : ND-LoRA, a parameter-efficient demonstration achieving up to 25.6% hallucination reduction and maintaining general-purpose accuracy at 1.00004 \times training cost • Analysis : Evidence of the concavity of P in industry-standard benchmarks and the presence of task-dependent optimal P revealing when neural diversity helps most (and hurts) • Framework : A general principle for parallel architectures — that maintaining decorrelation is as important as adding capacity — with implications beyond hallucination mitigation 2 A Theory of Neural Diversity Why don’t existing scaling methods improve reliability? Without explicit diversity mechanisms, gradient descent drives parallel streams toward similar representations through representational collapse ( Jing et al. 2022 ) , leaving reliability gains unrealized. We establish the first rigorous connection between architectural diversity and hallucination probability, proving that decorrelated streams reduce hallucinations and providing mathematical foundations for ND-LoRA. 2.1 Preliminaries We develop a framework connecting parallel neural computation to hallucination probability. Language models produce unreliable outputs in several ways: fabricating facts contradicting known knowledge, generating content inconsistent with input context, or creating unsupported claims ( Maynez et al. 2020 ; Ji et al. 2023 ) . While recent surveys provide comprehensive taxonomies ( Huang et al. 2024 ; Tonmoy et al. 2024 ) , we model hallucination probability through a signal-noise analytical proxy that captures the underlying reliability failure common to these manifestations and is analytically tractable. Consider a language model processing input x to produce an output (e.g., answer to a question, summary of a document). For each input, assume an oracle truth T>0 exists. Our architecture employs P parallel computational pathways called streams , each processing the same input through the same model but in perturbed ways. Each stream i produces output \hat{Y}_{i}=T+m_{i} , where m_{i} represents zero-mean noise corrupting the truth. These streams are combined through a weighted average: M\;\triangleq\;\frac{1}{P}\sum_{i=1}^{P}\hat{Y}_{i}\;=\;T+\frac{1}{P}\sum_{i=1}^{P}m_{i}. (1) We assume M has learned an unbiased truth, so \mathbb{E}[M]=T\equiv\mu>0 . Each stream’s noise has mean \mathbb{E}[m_{i}]=0 , variance \mathrm{Var}(m_{i})=\sigma^{2} , and pairwise correlations \rho_{ij}=\mathrm{Corr}(m_{i},m_{j}) measuring representational collapse. The average pairwise correlation captures overall similarity: \rho\;\triangleq\;\frac{2}{P(P-1)}\sum_{1\leq i<j\leq P}\rho_{ij}. (2) We define the hallucination event as \mathrm{H}\equiv\{M\leq 0\} —when aggregated noise overwhelms the true signal, causing incorrect predictions. The key intuition is noise cancellation: when streams are perfectly correlated ( \rho=1 ), noise doesn’t average out and M remains noisy. When decorrelated ( \rho\to 0 ), independent noise cancels and M concentrates around truth T , reducing hallucination probability by \sqrt{P} . 2.2 Neural Diversity Bounds Hallucination We connect noise correlation to feature correlation through linearization: if noise terms depend approximately linearly on representations at a chosen design layer, then reducing feature correlation reduces noise correlation. Design-layer features and diversity. At a chosen design layer , stream i exposes a feature vector z_{i}\in\mathbb{R}^{d} . Let \tilde{z}_{i} denote a whitened version (zero-mean, identity covariance within stream), and define the cross-correlation matrices C_{ij}\triangleq\mathbb{E}[\tilde{z}_{i}\tilde{z}_{j}^{\top}] . Definition 1 (Neural Diversity Index) . The spectral diversity index is \mathcal{D}_{\mathrm{spec}}\;=\;\frac{1}{P(P-1)}\sum_{i\neq j}\,\left\lVert C_{ij}\right\rVert_{2}. (3) Lower \mathcal{D}_{\mathrm{spec}} indicates greater diversity: \mathcal{D}_{\mathrm{spec}}=0 means perfectly decorrelated streams, while \mathcal{D}_{\mathrm{spec}}=1 means complete collapse. We now connect stream correlation to hallucination probability through three steps: variance analysis, linearization, and the main

Read the complete text