Compositional Generalization in Autoregressive Models via Logit Composition

Aakash Kumar · 2026

Read Compositional Generalization in Autoregressive Models via Logit Composition by Aakash Kumar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28304.txt.

Introduction

Aakash Kumar, Maria Sofia Bucarelli, and Emanuele Natale show how to merge self-feeding models without cross-talk. When each model's odds split cleanly by factor, each part keeps charge of its own slice of outputs, and the split holds under smooth rewordings of output space. Merged models also keep length reach when part promises hold evenly at the goal length.

Catalog record

CREATORAakash Kumar
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50882
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.28304

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Compositional Generalization in Autoregressive Models via Logit Composition Aakash Kumar Abstract Composing autoregressive models remains a core challenge in understanding how large language models can combine behaviors or skills learned across tasks. We introduce a new and principled composition strategy for autoregressive systems, inspired by composition methods developed for diffusion models. Under a factorized-conditionals assumption, we show that the resulting composition is projective: each component model preserves control over its own designated subspace of the output distribution avoiding interference between models. This property is further preserved under smooth reparameterizations of the output space, yielding a feature-space theorem. Finally, we show that composition preserves length-generalizing behavior when the factorization assumptions and component guarantees hold uniformly at the target length. These results provide a principled understanding of when model composition and merging succeed in autoregressive systems and identify conditions under which their interactions remain stable. 1 Introduction Composing specialized generative models is an appealing path toward building systems that are more modular, reusable, and controllable than a single monolithic model. If one model captures a particular concept, another captures a different one, and a third provides a shared background, a natural question is whether these components can be combined at inference time to produce a model that behaves as though it had learned all concepts jointly. For Transformer-based autoregressive models, existing approaches to this problem typically operate in either parameter space or architecture space . Parameter-space methods (such as weight interpolation 37 ; 39 , Task Arithmetic 19 , and sparsity-based merging 40 ; 42 ) combine fine-tuned checkpoints by averaging or adding task vectors, but provide limited guarantees on the resulting output distribution. Architecture-space approaches, such as Mixture-of-Experts (MoE) 20 ; 33 ; 13 , combine models at the level of routing and activation, typically without a distinguished background model and with input-dependent selection. Together, these approaches do not provide a static, explicit composition rule at the level of output distributions with guarantees on the resulting autoregressive model. A related line of work in diffusion models has demonstrated that models can be composed by linearly combining their score functions 12 ; 26 . This approach enables compositional generation and can even extrapolate beyond the support of individual models. 4 provided a principled explanation of when such composition is valid, introducing the notion of projective composition and identifying a factorization property as a key sufficient condition. However, this perspective has not been extended to autoregressive models, where generation is inherently sequential and conditioned on an evolving prefix. Inspired by the projective composition framework of 4 , we extend their formalism to autoregressive models such as Transformers. Given a collection of diffusion models (p_{1},\dots,p_{k},p_{b}) , where p_{b} denotes a shared background distribution, 4 studied the composition operator \mathcal{C}[(p_{1},p_{2},\dots,p_{k},p_{b})]\equiv p_{b}\prod_{i=1}^{k}\frac{p_{i}}{p_{b}}. (1) They formalized the notion of correct composition and showed that the operator in Equation 1 yields a correct composition under certain conditions. Our selective composition extends this probability-space composition principle to autoregressive models such as Transformers, which form the basis of large language models (LLMs), by applying an explicit arithmetic rule to conditional next-token distributions. Beyond correctness, we establish a conditional length-generalization guarantee. In particular, when the factorization assumptions and component-model guarantees hold uniformly at a target length, the composed model inherits the corresponding behavior on the combined task. This highlights a key advantage of distribution-level composition: it preserves structural properties of the underlying models under explicit assumptions. To build intuition, we begin with a simple toy example illustrating how composition works in practice. Example: compositional reasoning in trigonometry Consider a collection of trigonometric problems of the form Prove X=Y , where X and Y are trigonometric expressions. Suppose these problems can be decomposed into distinct skills (see Figure 1 ): • algebraic manipulation, • application of the identity \sin^{2}(x)+\cos^{2}(x)=1 , • application of the identity \sec^{2}(x)-\tan^{2}(x)=1 . Let p_{b} be a background Transformer that performs algebraic manipulations, while p_{1} and p_{2} are specialized models that apply the first and second identities, respectively. Each model is trained independently and excels only at its designated skill. The key question is whether we can compose these models to get a model capable of solving problems that require multiple identities in sequence . For example, can the composed model solve a problem that requires first rewriting an expression using \sin^{2}(x)+\cos^{2}(x)=1 and later applying \sec^{2}(x)-\tan^{2}(x)=1 , interleaved with algebraic steps? This setting highlights the central challenge of autoregressive composition: the applicability of each skill depends on the current prefix of the generated solution. As we show in this work, under suitable conditions ensuring that these skills act on disjoint segments of the generation, the composed model successfully integrates all components and produces correct multi-step solutions. Our Contributions • We introduce a principled framework for composing autoregressive models in probability space, extending projective composition to the sequential setting. • We identify conditions (factorized and joint factorizability) under which composition is provably correct. • We show that composition preserves length-generalizing behavior under uniform factorization and component-model guarantees. • We connect our framework to model merging methods, providing insight into when and why such techniques succeed. • We empirically validate our approach on both synthetic and large language model benchmarks, showing exact behavior in the controlled setting and strong coding gains with retention of mathematical performance in LLM evaluations. Paper organization: The rest of the paper is organized as follows. Section 2 reviews related work on model merging and mixture-of-experts methods. Section 3 introduces our compositional framework and presents our main theory. Section 4 further connects our framework to prior model merging approaches. Section 5 presents empirical results on both controlled toy tasks and large language model benchmarks. Finally, we conclude in Section 6 . 2 Related Work Model Merging Recent research has explored how to combine pretrained models without joint retraining, a paradigm known as model merging or parameter arithmetic. Early approaches focused on simple weight-space interpolation between models fine-tuned on different tasks, showing that linear interpolation can sometimes yield models capable of retaining knowledge from both sources 37 ; 39 . A key insight from this line is Task Arithmetic 19 , which introduced the concept of task vectors \tau_{t}=\theta(t)-\theta(0) , where \theta(t) denotes parameters fine-tuned on task t and \theta(0) is the pretrained model. Task vectors are interpreted as directions in parameter space that meaningfully steer model behavior, enabling arithmetic operations across fine-tuned models. Subsequent work has focused on reducing interference between tasks when performing such parameter arithmetic, for example by choosing merging coefficients more effectively 1 ; 41 ; 29 ; 21 . Another line of research uses sparsification to mitigate task interference 40 ; 41 ; 42 ; 9 . Other work uses low-rank subspaces and matrix factorization to extract informative components for merging 15 ; 28 ; 34 . More recent geometric approaches perform fine-tuning and merging in the tangent space of the pretrained model manifold, linearizing updates to better capture local structure, amplify weight disentanglement, and mitigate task interference 31 . Our work connects to the model merging literature because we also compose a pretrained base (background) model with task-specific fine-tuned models, similar to Task Arithmetic 19 and subsequent merging methods. However, unlike parameter-space or task-space techniques, our selective composition operates in probability

Read the complete text