On Transferring Transferability: Towards a Theory for Size Generalization

Eitan Levin, Yuxin Ma, Mateo Díaz, Soledad Villar · 2025

Read On Transferring Transferability: Towards a Theory for Size Generalization by Eitan Levin, Yuxin Ma, Mateo Díaz, Soledad Villar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23599.txt.

Introduction

Eitan Levin and co-authors build theory for learning models that take inputs of changing size in a 2025 paper. Graphs, sets, and point clouds all vary in size, so models must handle that range. The question is when skill learned on small inputs carries over to large ones.

The authors show that such transfer matches a form of continuity in a limit space that glues small cases to equal large ones. The gluing is set by the data and task at hand. They adapt known architectures to meet the rule and give design principles for new ones. Tests back the theory. Readers get a firm base for size-proof model design.

Catalog record

CREATOREitan Levin, Yuxin Ma, Mateo Díaz, Soledad Villar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23599

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On Transferring Transferability: Towards a Theory for Size Generalization Eitan Levin, Yuxin Ma, Mateo Díaz, Soledad Villar Abstract Many modern learning tasks require models that can take inputs of varying sizes. Consequently, dimension-independent architectures have been proposed for domains where the inputs are graphs, sets, and point clouds. Recent work on graph neural networks has explored whether a model trained on low-dimensional data can transfer its performance to higher-dimensional inputs. We extend this body of work by introducing a general framework for transferability across dimensions. We show that transferability corresponds precisely to continuity in a limit space formed by identifying small problem instances with equivalent large ones. This identification is driven by the data and the learning task. We instantiate our framework on existing architectures, and implement the necessary changes to ensure their transferability. Finally, we provide design principles for designing new transferable models. Numerical experiments support our findings. $\diamond$ $\diamond$ footnotetext: These authors contributed equally to this work 1 Introduction Modern learning problems often involve inputs of arbitrary size: language has text sequences of arbitrary length, graphs encode networks with an arbitrary number of nodes, and particle systems might have an arbitrary number of points. Although traditional models are typically limited to the specific input dimensions they were trained on, dimension-independent architectures have been proposed for several domains. For instance, graph neural networks (GNN) for graphs [ 74 ] , DeepSet for sets [ 97 ] , and PointNet for point clouds [ 68 ] . These models have a desirable feature: they can be trained with low-dimensional samples and, then, deployed on higher-dimensional data. A key question in this context is whether the performance of a model trained with low-dimensional samples transfers across dimensions. This has been thoroughly studied in the context of GNNs [ 74 , 60 , 61 ] , where the term transferability was first coined. 1 1 1 The term “transferability” is not related to transfer learning or meta-learning. It specifically refers to the phenomenon that consistency of outputs is preserved under size changes for structurally similar inputs, permitting direct performance carryover from small to large problems. In turn, the tools used to establish transferability of GNNs are, at first sight, problem-specific, including the concept of a graphon, and certain convergence of homomorphism densities. Thus, the analysis does not readily extend beyond GNNs, and so we ask What properties of the model, data, and learning task ensure that the learning performance transfers well across dimensions? This question is relevant for the current pursuit of graph foundation models [ 55 ] , GNN for combinatorial optimization [ 13 ] , neural PDE solvers [ 69 , 53 ] , and so on, where pre-trained models aim to be adapted to a wide range of tasks involving different-sized objects. In this paper, we answer this question by identifying assumptions on the learning problem that make a model generalize across dimensions, and give a theoretical framework to understand models with such properties. Our framework requires technical notions that we develop throughout the paper, so before diving into the details, we proceed to give the high-level intuition. Our framework considers a machine learning model parametrized by \theta\in\mathbb{R}^{k} , where the number of parameters k is fixed. For any fixed \theta , the model can be evaluated on inputs of any size n . That is, the learned \theta defines a sequence of functions, for example, (f_{n}\colon{V}_{n}\rightarrow\mathbb{R})_{n} with increasingly larger input sizes. 2 2 2 The output size can also depend on n , which is often the case in representation learning. We think of {V}_{n}\subseteq{V}_{\infty} as a subspace of an infinite-dimensional limit space. Transferability then amounts to asking whether for large enough n and m we have f_{n}(x_{n})\approx f_{m}(x_{m}) when the inputs x_{n} and x_{m} are close to each other in the limiting space V_{\infty} . For example, we consider graphs of different sizes to be close if their associated step graphons are close in the cut metric, following standard practice in the GNN literature. Similarly, we view point clouds of varying sizes as close if their corresponding empirical measures are close in the Wasserstein metric. In a nutshell, we identify two properties that guarantee transferability: (i) the functions in the sequence are compatible with one another in a precise sense, and (ii) all of them are continuous with respect to a prescribed norm on {V}_{\infty} . Compatibility ensures that the sequence extends to a function taking infinite-dimensional inputs f_{\infty}\colon{V}_{\infty}\rightarrow\mathbb{R} , while continuity ensures that this extension is continuous. This allows us to relate model outputs on two differently-sized inputs via their extension f_{n}(x_{n})=f_{\infty}(x_{n})\approx f_{\infty}(x_{m})=f_{m}(x_{m}). Moreover, if we evaluate our models on a sequence of inputs x_{n} converging in a suitable sense to an infinite-dimensional input x , then the evaluations f_{n}(x_{n}) converge to the value of the limiting model f_{\infty}(x) . Our framework parallels the ideas from the GNN literature, where graph convergence to a limiting graphon is measured in cut distance—which is equivalent to homomorphism density convergence. However, this framework provides a transparent extension that allows us to analyze other models, e.g., DeepSets and PointNet. Our contributions. Next, we summarize our three core contributions. ( A framework for transferability ) We introduce a class of any-dimensional models and a formal definition of transferability. We show that transferability holds whenever certain notions of compatibility and continuity hold. We further leverage transferability to establish a generalization bound across dimensions. In our theory, the notion of transferability relies on identifying low-dimensional objects with higher-dimensional ones, and measuring similarity of such objects in an appropriate metric. We illustrate how these choices have to align with the data and learning tasks. ( Transferability of existing models ) We instantiate our framework on several models, including DeepSet [ 97 ] , PointNet [ 68 ] , standard GNNs [ 74 ] , Invariant Graph Networks (IGN) [ 58 ] , and DS-CI [ 6 ] . Not all of these models satisfy our assumptions. For those that do, our framework yields new and existing transferability results in a unified fashion. For those that do not, we either modify them to ensure transferability or demonstrate numerically that transferability fails and impedes the performance of these models. ( New transferable models ) We provide design principles for constructing transferable models. Leveraging these ideas, we develop a variant of IGN that is provably transferable, addressing challenges highlighted in previous work [ 58 , 37 ] . We also design a new transferable model for point cloud tasks that is more computationally efficient than existing methods [ 6 ] . Notation. The Kronecker product on matrices is denoted by \otimes . We write \mathbbm{1}_{n} and I_{n} for the all-ones vector and the identity matrix of size n respectively. Given two sequences (a_{n}),(b_{n})\subseteq\mathbb{R} , we say that a_{n}\lesssim b_{n} if there is a constant C>0 so that a_{n}\leq Cb_{n} for all n. See also Appendix A . Outline. The remainder of this section covers related work. Section 2 introduces the theoretical framework. Section 3 defines transferability, which is used in Section 4 to derive a generalization guarantee. Section 5 instantiates our framework on concrete neural network examples, and Section 6 provides supporting experiments. 3 3 3 The

Read the complete text