Teaching the Teacher: Improving Neural Network Distillability for Symbolic Regression via Jacobian Regularization
Soumyadeep Dhar, Kei Sen Fong, Mehul Motani · 2025
Read Teaching the Teacher: Improving Neural Network Distillability for Symbolic Regression via Jacobian Regularization by Soumyadeep Dhar, Kei Sen Fong, Mehul Motani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22767.txt.
Introduction
Dhar, Fong, and Motani smooth the teacher so the student can learn. Symbolic regression can turn a neural network's knowledge into a readable formula, but the formulas usually predict badly, because networks learn jagged functions while formula search prefers simple ones. Their fix works upstream: train the network with penalties on its Jacobian and Lipschitz constant so the function it learns stays smooth.
Across twenty data sets and fifty trials each, formulas distilled from smoothed teachers score significantly better than the standard pipeline produces. Ablations check which parts of the student algorithm matter. The authors conclude that aligning the teacher's complexity with the student's reach is the missing ingredient in symbolic distillation.
Catalog record
| CREATOR | Soumyadeep Dhar, Kei Sen Fong, Mehul Motani |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22767
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Teaching the Teacher: Improving Neural Network Distillability for Symbolic Regression via Jacobian Regularization Soumyadeep Dhar, Kei Sen Fong, Mehul Motani Abstract Distilling large neural networks into simple, human-readable symbolic formulas is a promising path toward trustworthy and interpretable AI. However, this process is often brittle, as the complex functions learned by standard networks are poor targets for symbolic discovery, resulting in low-fidelity student models. In this work, we propose a novel training paradigm to address this challenge. Instead of passively distilling a pre-trained network, we introduce a Jacobian-based regularizer that actively encourages the “teacher” network to learn functions that are not only accurate but also inherently smoother and more amenable to distillation. We demonstrate through extensive experiments on a suite of real-world regression benchmarks that our method is highly effective. By optimizing the regularization strength for each problem, we improve the R 2 R^{2} score of the final distilled symbolic model by an average of 120% (relative) compared to the standard distillation pipeline, all while maintaining the teacher’s predictive accuracy. Our work presents a practical and principled method for significantly improving the fidelity of interpretable models extracted from complex neural networks. Keywords: Knowledge Distillation, Symbolic Regression, Jacobian Regularization, Explainable AI Note: This is a draft manuscript and has not yet undergone full co-author review. 1 Introduction The remarkable success of deep learning has revolutionized fields from computer vision to natural language processing ( LeCun et al. 2015 ) . These models, particularly deep neural networks (ANNs), have achieved state-of-the-art performance on a vast array of predictive tasks. However, this performance often comes at the cost of transparency. ANNs operate as “black boxes,” with their immense number of parameters forming complex, non-linear relationships that are opaque to human understanding ( Guidotti et al. 2018 ) . This lack of interpretability poses a significant barrier to their adoption in high-stakes domains such as scientific discovery ( Udrescu & Tegmark 2020 ) , medical diagnostics ( Topol 2019 ) , and financial modeling, where the ability to trust, debug, and understand a model’s reasoning is as critical as its predictive accuracy. To bridge this gap, the field of eXplainable AI (XAI) has explored various methods to shed light on these models. One promising paradigm is knowledge distillation ( Hinton et al. 2015 ) , where a large, complex “teacher” model transfers its knowledge to a smaller, simpler “student” model. While often used for model compression, this framework also provides a powerful pathway to interpretability by selecting a student model that is inherently transparent, a concept explored in early work on extracting decision trees from networks ( Craven & Shavlik 1996 ) . Among possible student models, those produced by Symbolic Regression (SR) are particularly compelling. Unlike other models that provide post-hoc explanations, SR aims to discover the underlying mathematical formula that maps inputs to outputs ( Koza 1992 ) . The result is not an explanation, but the model itself —a fully transparent, human-readable equation. A naive two-stage pipeline for achieving this is straightforward: first, train a high-performance teacher ANN on a dataset, and second, train an SR model to mimic the predictions of this teacher. However, the promise of this naive pipeline is seldom realized in practice. In our initial analysis, we find that this approach is brittle and struggles significantly when applied to real-world data. The complex, often chaotic functions learned by standard ANNs are poor targets for SR, resulting in low-fidelity symbolic models whose performance is not competitive with other machine learning baselines (as shown in Section 4 ). This suggests a fundamental problem: the teacher network, while accurate, is a poor instructor. Instead of passively accepting the teacher’s limitations, we ask: can we train the teacher to be more effective at teaching? In this work, we introduce a novel training methodology that actively encourages an ANN to learn functions that are not only accurate but also inherently more amenable to symbolic distillation. Our core contribution is a new Jacobian-based regularization term added to the ANN’s loss function. By penalizing the norm of the network’s Jacobian, we incentivize the model to learn functionally smoother mappings that are easier for SR algorithms to discover and represent. Our main contributions are as follows: • We conduct a comprehensive analysis identifying the brittleness of the naive ANN-to-SR distillation pipeline on real-world benchmark datasets. • We propose a novel, principled regularization method based on the Jacobian norm to explicitly improve the distillability of neural networks. • We demonstrate through extensive experiments that our method dramatically improves the performance of the final symbolic student model—by an average of 120% in R² score—often with minimal impact on the teacher’s own predictive accuracy. • We provide an analysis of our method’s characteristics, including the dataset-dependent nature of the optimal regularization strength, offering a path forward for developing more robust and interpretable AI systems. 2 Related Work Our research is situated at the intersection of several key areas in machine learning: knowledge distillation, explainable AI, symbolic regression, and network regularization. 2.1 Knowledge Distillation First popularized by Hinton et al. ( Hinton et al. 2015 ) , knowledge distillation is a technique where a compact “student” model is trained to reproduce the behavior of a larger, pre-trained “teacher” network. The original motivation was primarily model compression, enabling the deployment of powerful models on resource-constrained devices. However, the paradigm is more general and has been adapted for tasks such as transferring knowledge across different data modalities and for semi-supervised learning. Our work reframes distillation’s purpose, using it not for compression, but as a mechanism to transfer knowledge from a high-performance but opaque model to one that is inherently interpretable. 2.2 Explainable AI (XAI) and Interpretable Models The challenge of understanding black box models has given rise to the field of XAI. Current approaches can be broadly categorized. Post-hoc explanation methods, such as LIME ( Ribeiro et al. 2016 ) and SHAP ( Lundberg & Lee 2017 ) , aim to explain individual predictions of a pre-trained model without altering it. While powerful, these methods provide local explanations and do not reveal the model’s global logic. In contrast, inherently interpretable models are transparent by design. This category includes linear models, decision trees, and Generalized Additive Models (GAMs). Our work aligns with the goal of creating inherently interpretable models. However, instead of training a simple model directly on the data (which may yield suboptimal performance), we aim to extract a simple model from a high-performance complex teacher, seeking to combine the predictive power of deep learning with the transparency of symbolic equations. 2.3 Symbolic Regression At the core of our student model is Symbolic Regression (SR) , a machine learning technique that searches the space of mathematical expressions to find the function that best fits a given dataset. Typically implemented using Genetic Programming (GP) as pioneered by Koza ( Koza 1992 ) , SR has been a powerful tool for automated scientific discovery, rediscovering physical laws from experimental data. While SR is traditionally applied directly to data, we re-purpose it as the “student” in our distillation pipeline, tasking it with a novel goal: to discover the mathematical function implicitly learned by a neural network. 2.4 Regularization in Deep Learning Regularization is a