Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones
A. Zhmoginov, J. Lee, M. Sandler · 2025
Read Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones by A. Zhmoginov, J. Lee, M. Sandler (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05641.txt.
Introduction
Andrey Zhmoginov, Jihwan Lee, and Mark Sandler wrote this paper in two thousand twenty-five, with names in fuller form in its text. It shows how to stamp a small, task-tuned transformer out of a large foundation model in one shot.
Big models know much that a single task does not need, and cost too much for small devices. The method maps the big model's weights to a small one's by a task-set rule. The narrow copy thus holds just the slice of skill the task calls for. On image modeling tasks, the stamped models beat broad conditional models of like size. The work hints at cheap custom models on demand.
Catalog record
| CREATOR | A. Zhmoginov, J. Lee, M. Sandler |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-05641
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones A. Zhmoginov, J. Lee, M. Sandler Abstract Modern Foundation Models (FMs) are typically trained on corpora spanning a wide range of different data modalities, topics, downstream tasks. Utilizing these models can be very computationally expensive and is out of reach for most consumer devices. Furthermore, most of the broad FM knowledge may actually be irrelevant for a specific task at hand. Here we explore a technique for mapping parameters of a large Transformer to parameters of a smaller specialized model. By making this transformation task-specific, we aim to capture a narrower scope of the knowledge needed for performing a specific task by a smaller model. We study our method on image modeling tasks, showing that performance of generated models exceeds that of universal conditional models. † † affiliationnotice: Equal contribution 1 Introduction Transformer-based generative models have recently shown a remarkable success in modeling complex data distributions across a wide spectrum of modalities including images, audio and language. In the language domain, Large Language Models (LLMs) became an extremely powerful tool demonstrating impressive performance across a large scope of language tasks. These models are typically very computationally complex to train and run and require vast amounts of text data that encompass a wide range of topics, facts and downstream applications. The generality of the resulting LLMs can be extremely advantageous, but also prove redundant or even detrimental in narrow applications and in specialized tasks ( Raffel et al. 2020 ) . Here we propose an approach to generating a task-specific Transformer model of a smaller size from a larger Transformer. In our primary setup, similar to that explored in other related publications (see Appendix A ), we maintain a functional universal large Foundation Model (FM) and use it to generate smaller specialized task-dependent Transformers with an emphasis on multi-task learning and personalization scenarios. Following the HyperNetwork approach ( Ha et al. 2016 ) , we generate the specialized models on the fly from the task specification without the need for any fine-tuning. We study this idea experimentally in the visual domain adopting ImageGPT approach ( Chen et al. 2020a ) , in which images are represented as sequences of discrete tokens and an autoregressive Transformer model is used to model this sequence distribution. Here we compare performance of task-specialized generated small Transformer models with universal counterparts of the same size. We also explore zero-shot generalization and knowledge transfer across different tasks. 2 Method In the following, we outline our approach, where we use large Transformer models for generating smaller task-specific Transformers. We simultaneously view this as (a) a model specialization technique, where we can produce a small Transformer best suited for a particular task; (b) model compression technique where a single large model can be used for producing a variety of models of different sizes, and finally (c) a modular network approach, where individual generated task-specialized model implicitly share knowledge via a single large model. A more detailed discussion of weight space manifolds and associated model specialization techniques can be found in Appendix B . Figure 1: Model diagram. Given the task ID \boldsymbol{t} , the weights of the large “source model” \mathcal{M}_{1} (Transformer with the embedding size d ) are mapped to the weights of a smaller “projected model” \mathcal{P}_{s} (Transformer with the embedding size ds ). The loss of \mathcal{P}_{s} on the input sequence is used to train the “projection operator” and tune \mathcal{M}_{1} , while a good performance of \mathcal{M}_{1} can be maintained by continuing to train it with the original objective. 2.1 Matrix Generators In the Transformer architecture, the embedding size typically defines the sizes of all self-attention and MLP weight matrices. Choosing some maximum embedding size d , we define a family of Transformer models \{\mathcal{M}_{s}\}_{s\in S} with the embedding size ds . We typically choose s=2^{-k} with integer k\geq 0 assuming that d/2^{k} is integer for all models of interest. We generally assume that each \mathcal{M}_{s} is a separate model pretrained on some input data distribution p(\boldsymbol{x}) . Given pretrained \mathcal{M}_{1} how can we generate a smaller Transformer with architecture of \mathcal{M}_{s} and the embedding size ds ? One natural way of doing this is to map each linear operator 1 1 1 all dense layers in self-attention and MLP operators \hat{\mathrm{O}}_{1}^{\ell}(\boldsymbol{z}):=\mathrm{W}^{\ell}_{1}\boldsymbol{z}+\boldsymbol{b}^{\ell}_{1} from \mathcal{M}_{1} to a corresponding linear operator 2 2 2 In the following we use a hat to denote operators ( \hat{\mathrm{X}} ) and roman font for matrices ( \mathrm{X} ). \hat{\mathrm{O}}_{s}^{\ell}(\boldsymbol{z}):=\mathrm{W}^{\ell}_{s}\boldsymbol{z}+\boldsymbol{b}^{\ell}_{s} in the small model. Assuming that this transformation is linear, its most general form is simply \displaystyle(\mathrm{W}_{s})_{ij}:=\sum_{k,l}\mathrm{V}^{W}_{ijkl}(\mathrm{W}_{1})_{kl}+\mathrm{V}^{B}_{ij}, \displaystyle(\boldsymbol{b}_{s})_{i}:=\sum_{k}\mathrm{U}^{W}_{ik}(\boldsymbol{b}_{1})_{k}+\mathrm{U}^{B}_{i}. Projections. One natural choice of (\mathrm{V}^{W},\mathrm{V}^{B},\mathrm{U}^{W},\mathrm{U}^{B}) comes from analyzing linear transformations of the embedding space. Specifically, given two linear maps \hat{\mathrm{Q}}:\mathbb{R}^{ds}\to\mathbb{R}^{d} and \hat{\mathrm{P}}:\mathbb{R}^{d}\to\mathbb{R}^{ds} mapping embeddings between embedding spaces of two models, we can naturally define \hat{\mathrm{O}}_{s} via \hat{\mathrm{O}}_{s}(\boldsymbol{z})\equiv\hat{\mathrm{P}}(\hat{\mathrm{O}}_{1}(\hat{\mathrm{Q}}\boldsymbol{z})) . In other words, the action \hat{\mathrm{O}}_{s}(\boldsymbol{z}) is defined as a sequence of several steps: (a) mapping \boldsymbol{z} to a higher-dimensional embedding space via \hat{\mathrm{Q}}\boldsymbol{z} , (b) acting on this activation with \hat{\mathrm{O}}_{1} and finally (c) projecting the result down to the original low-dimensional embedding space with \hat{\mathrm{P}} . Ignoring biases, we can define \hat{\mathrm{Q}}(\boldsymbol{z})=\mathrm{Q}\boldsymbol{z} and \hat{\mathrm{P}}(\boldsymbol{z})=\mathrm{P}\boldsymbol{z} with \mathrm{Q} and \mathrm{P} being two matrices, thus simply obtaining: \displaystyle\mathrm{W}_{s}=\mathrm{P}\mathrm{W}_{1}\mathrm{Q},\qquad\boldsymbol{b}_{s}=\mathrm{P}\boldsymbol{b}_{1}. (1) In the following, we mainly adopt this model. 2.2 Task-Dependent Matrix Generators Our primary goal is generating small task-specific Transformer models, which requires that the generated weight matrices \mathrm{W}_{s} depend on the task. In the following, we assume that each task is uniquely defined by a task identifier \boldsymbol{t}\in T with T being a finite-dimensional vector space of all possible tasks. Given \boldsymbol{t} , we then define a projection operator \rho_{s} that first maps \boldsymbol{t} to \mathrm{V}(\boldsymbol{t}) and \mathrm{U}(\boldsymbol{t}) at every layer as discussed below and then uses these tensors and the large Transformer source model \mathcal{M}_{1} to generate the weights of a smaller projected model \mathcal{P}_{s}=\rho_{s}(\boldsymbol{t};\mathcal{M}_{1}) , in effect defining a function family \mathcal{F}(\rho_{s},\mathcal{M}_{1})=\{\rho_{s}(\boldsymbol{t};\mathcal{M}_{1})|\boldsymbol{t}\in T\} . If the tasks are specified explicitly as a part of the training dataset, we expect that \rho_{s}(\boldsymbol{t};\mathcal{M}_{1}) performs a task \boldsymbol{t} better than a universal task-agnostic model of the same architecture. On the other hand, if task identifiers are not provided, we could design synthetic tasks with the goal of making the resulting \mathcal{F}(\rho_{s},\mathcal{M}_{1}) sufficiently rich to be useful for possible downstream tasks. Figure 2: Projection of the source model weight \mathrm{W}_{1} into the projected model weight \mathrm{W}_{s} . Here \boldsymbol{t} is the task identifier and \mathrm{P}(\boldsymbol{t}) , \mathrm{Q}(\boldsymbol{t}) are “projection matrices” whose diagonal elements are generated using shallow MLPs from \boldsymbol{t} . While there may be numerous choices of defining linear operators \hat{\mathrm{V}}(\boldsymbol{t}) and \hat{\mathrm{U}}(\boldsymbol{t}) , here we adopt the projection matrix approach ( 1 ) by learning \mathrm{P}(\boldsymbol{t}) and \mathrm{Q}(\boldsymbol{t}) matrices (see below). Our experiments suggested that the best way to generate high-quality specialized \mathcal{P}_{s} required that the weights of \mathcal{M}_{1} are also tuned thus increasing the number of trained parameters and modifying a pretrained source model \mathcal{M}_{1} to be compatible with generated projections. We tuned \mathcal{M}_{1} and learned projection operators by optimizing both the original loss on \mathcal{M}_{1} to maintain its quality and the loss on the projected model \mathcal{P}_{s} (see Eq. ( 2 )