Feature-Function Curvature Analysis: A Geometric Framework for Explaining Differentiable Models

Hamed Najafi, Dongsheng Luo, Jason Liu · 2025

Read Feature-Function Curvature Analysis: A Geometric Framework for Explaining Differentiable Models by Hamed Najafi, Dongsheng Luo, Jason Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-27207.txt.

Introduction

Hamed Najafi, Dongsheng Luo, and Jason Liu propose a geometric way to explain what a trained model has learned. Standard attribution methods give each feature one static score, which nonlinearity and interactions can distort. Their framework, feature-function curvature analysis, studies the shape of the learned function itself and gives each feature a four-part signature covering impact, volatility, nonlinearity, and interaction.

A dynamic extension tracks those signatures through training, moving from what was learned to how learning unfolded. Experiments give direct evidence of hierarchical learning, where simple linear effects appear before complex interactions. The dynamic view also flags insufficient model capacity and predicts overfitting onset. Together the tools turn explanation from a single number into a fuller, more trustworthy account of learning.

Catalog record

CREATORHamed Najafi, Dongsheng Luo, Jason Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-27207

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Feature-Function Curvature Analysis: A Geometric Framework for Explaining Differentiable Models Hamed Najafi, Dongsheng Luo, Jason Liu Abstract Explainable AI (XAI) is critical for building trust in complex machine learning models, yet mainstream attribution methods often provide an incomplete, static picture of a model’s final state. By collapsing a feature’s role into a single score, they are confounded by non-linearity and interactions. To address this, we introduce Feature-Function Curvature Analysis (FFCA), a novel framework that analyzes the geometry of a model’s learned function. FFCA produces a 4-dimensional signature for each feature, quantifying its: (1) Impact , (2) Volatility , (3) Non-linearity , and (4) Interaction . Crucially, we extend this framework into Dynamic Archetype Analysis , which tracks the evolution of these signatures throughout the training process. This temporal view moves beyond explaining what a model learned to revealing how it learns. We provide the first direct, empirical evidence of hierarchical learning, showing that models consistently learn simple linear effects before complex interactions. Furthermore, this dynamic analysis provides novel, practical diagnostics for identifying insufficient model capacity and predicting the onset of overfitting. Our comprehensive experiments demonstrate that FFCA, through its static and dynamic components, provides the essential geometric context that transforms model explanation from simple quantification to a nuanced, trustworthy analysis of the entire learning process. 1 Introduction The proliferation of complex, “black-box” machine learning models in high-stakes domains has created an urgent need for reliable and insightful explanations of their behavior [ 1 , 2 ] . In response, the field of Explainable AI (XAI) has developed powerful feature attribution methods like LIME [ 3 ] and SHAP [ 4 ] , which assign a single score to quantify a feature’s contribution. While widely used, this approach has a fundamental limitation: a single score cannot disambiguate the nature of a feature’s contribution, conflating direct influence with importance derived from complex non-linearities or interactions [ 5 ] . However, even post-hoc methods—applied after training to interpret a fixed model—though they offer a more detailed, multi-dimensional view of the trained model, still answer only what the model learned. They treat the model as a static artifact, ignoring the rich history of the training process that produced it. This leaves a more profound question unanswered: how does a model arrive at its final, complex representation? Does it learn simple patterns first? Can we observe the moment it discovers a key interaction? Can we detect when its learning process becomes unstable before its performance degrades? These questions matter for actionable ML practice—they let us diagnose reliance on spurious cues, anticipate failures under distribution shift, audit fairness and safety, and decide where to intervene (e.g., data curation, regularization, or model editing). In comparison to decomposition methods (e.g., LRP), Layer-wise Relevance Propagation (LRP) redistributes a prediction backward through the network to assign “relevance” to inputs. In contrast, we introduce Feature–Function Curvature Analysis (FFCA) , a post-hoc, model-dependent probe that examines the local shape of the trained model in input space. We call FFCA geometric because it characterizes how the model’s output changes in the immediate neighborhood of a data point—capturing not only which features matter, but whether their effects are stable or volatile, nonlinear or approximately linear, and whether they act alone or only in combination with others. Accordingly, FFCA reports four diagnostic signatures : (i) impact (how strongly a feature influences the prediction near the point of interest), (ii) volatility (how sensitive that influence is to small input changes), (iii) curvature (the extent to which the relationship is nonlinear), and (iv) interaction (the degree to which a feature’s effect depends on other features). Thus, while both LRP and FFCA are post-hoc and model-dependent, LRP yields a decomposition of a prediction into additive parts, whereas FFCA yields shape-based diagnostics about the model’s local behavior. To address these questions, our FFCA framework analyzes both the trained model’s local behavior and its learning dynamics for any differentiable architecture. FFCA dissects a feature’s role by analyzing the local geometry of the model’s decision surface. It operates in two modes: 1) Static Analysis: At the end of training, FFCA produces a 4-dimensional signature for each feature— Impact (strength of local influence), Volatility (stability of that influence to small perturbations), Non-linearity (departure from a linear response), and Interaction (dependence of a feature’s effect on other features). We group these continuous signatures into an eight-archetype taxonomy to turn raw numbers into actionable labels, enabling rapid triage, cross-model comparison, and clear communication to practitioners and auditors (with recommended interventions per archetype). 2) Dynamic Archetype Analysis: By capturing the 4D signature at regular intervals during training, FFCA provides a temporal view of the learning process. This allows us to observe the evolution of feature roles, providing unprecedented insight into the model’s behavior. FFCA summarizes local model behavior with four complementary diagnostics that capture strength of influence, stability, non-linearity, and cross-feature dependence (formal definitions in Section 3 ). The contributions of this paper are threefold: 1. We propose FFCA, a novel XAI framework that uses a 4D geometric signature and an 8-archetype taxonomy to provide a nuanced characterization of feature roles. 2. We introduce Dynamic Archetype Analysis, a major extension that tracks the evolution of feature roles during training, providing the first direct, empirical evidence of hierarchical learning in neural networks. 3. We provide extensive empirical validation across multiple datasets and architectures, demonstrating that our dynamic analysis is a powerful diagnostic tool for identifying insufficient model capacity and providing early-warning signals for overfitting. 2 Related work FFCA builds on several key XAI research areas. Foundational feature attribution methods like LIME [ 3 ] and SHAP [ 4 ] provide single importance scores but can obscure complex behaviors like non-linearity or interactions, which FFCA aims to disambiguate. Our work is closely related to methods for quantifying feature interactions. While visual tools like PDPs [ 6 ] and ICE plots [ 7 ] can reveal interactions, quantitative measures like the H-statistic [ 8 ] and Shapley Interactions provide model-agnostic scores. FFCA’s Interaction score ( X_{i} ) differs by deriving this information directly from the model’s geometry via the Hessian, offering an efficient, model-native measure. The use of higher-order derivatives is an emerging XAI frontier [ 9 ] . The Hessian matrix mathematically defines non-linearity (diagonal) and interactions (off-diagonal) [ 10 ] . A key challenge is the “zero-Hessian” problem in ReLU networks. FFCA adopts an efficient activation-smoothing strategy, similar to Integrated Hessians [ 11 ] , by temporarily replacing ReLU with Softplus during analysis. While other methods use the Hessian to compute raw interaction values [ 12 , 9 , 13 ] , FFCA’s novelty lies in its systematic framework that translates these geometric properties into the 4D signature and 8-archetype taxonomy for practical diagnosis. Finally, FFCA bridges XAI with deep learning theory on network characterization [ 14 , 15 ] , which uses the same mathematical tools to analyze learning dynamics. FFCA systematizes these tools into an interpretable framework for the applied XAI practitioner. Full discussion in Appendix A . 3 Feature-function curvature analysis (FFCA) FFCA is a model-centric analysis framework designed for any differentiable model. It operates by performing a geometric analysis of the model’s learned function to produce a multi-dimensional characterization of each feature’s

Read the complete text