FlexRank: Nested Low-Rank Knowledge Decomposition for Adaptive Model Deployment Riccardo Zaccone, Stefanos Laskaridis, Marco Ciccone, Samuel Horváth Abstract The growing scale of deep neural networks, encompassing large language models (LLMs) and vision transformers (ViTs), has made training from scratch prohibitively expensive and deployment increasingly costly. These models are often used as computational monoliths with fixed cost, a rigidity that fails to leverage overparametrized architectures and largely hinders adaptive deployment across different cost budgets. We argue that nested components, ordered by importance, can be extracted from pretrained models and selectively activated within the available computational budget. To this end, our proposed FlexRank method leverages low-rank weight decomposition with nested, importance-based consolidation to extract submodels of increasing capabilities. Our approach enables a “train-once, deploy-everywhere” paradigm that offers a graceful trade-off between cost and performance without training from scratch for each budget — advancing practical deployment of large models. † † affiliationnotice: \dagger Work done independently of Amazon. 1 Introduction Figure 1: FlexRank takes as input a base model, which is first decomposed by factorizing each linear layer independently. Next, a global ordering is obtained via a dynamic programming subroutine that assumes additivity of errors across layers. This global ordering is then used to extract nested submodels of different sizes, which are stochastically refined through distillation from the base model. Over recent years, the number of parameters and computational demands of modern networks have grown dramatically. Large Language ( 41 ) and Vision ( 8 ) Transformer models now contain billions of parameters and require vast training corpora and compute budgets ( 1 ; 40 ; 16 ; 35 ) . As these models continue to scale, training from scratch has become feasible only for a small number of well-resourced institutions. This has prompted the broader community to reuse publicly released pre-trained models and develop increasingly sophisticated methods for adapting them to downstream tasks ( 20 ; 28 ; 43 ; 26 ; 39 ) . Most adaptation strategies fall under the umbrella of parameter-efficient fine-tuning (PEFT). These methods modify a small auxiliary set of parameters while keeping the original model largely frozen. Although PEFT substantially reduces training costs, it leaves the computational structure of the backbone network unchanged. Consequently, model size and inference cost remain fixed, even when downstream applications could benefit from lighter or more variable configurations. This mismatch between adaptation flexibility and deployment rigidity becomes more pronounced when targeting environments with diverse hardware capabilities and different latency or memory budgets ( 24 ; 47 ; 52 ) . A natural direction for reducing inference cost is to compress the model itself. Two families of techniques have become particularly prominent. Quantization ( 27 ; 13 ; 48 ) reduces numerical precision to shrink memory footprint and improve throughput, but highest-quality results typically rely on quantization-aware training (QAT) ( 29 ; 5 ; 30 ) , which requires modifying the training pipeline. Sparsity-based approaches ( 38 ; 12 ; 23 ) , instead, prune weights or induce structure via penalties, such as \ell_{1} regularization. However, these methods also require retraining the full model to maintain performance and often depend on hardware, kernel support, or specific sparsity patterns to translate sparsity into real speedups ( 32 ; 34 ) . While highly effective, especially for memory-bound workloads, these approaches more often fail to provide a flexible, smoothly varying set of model capacities, which are increasingly needed in heterogeneous deployment environments. For example, consumer device manufacturers typically offer a variety of platforms of varying capabilities that span across different tiers and generations, thus offering different compute and capacity dynamics. Additionally, model inputs vary in difficulty and can be handled by smaller model variants without compromising accuracy. However, training and maintaining submodels of different sizes is costly and cumbersome. This motivates exploring alternative mechanisms for inducing model elasticity . Most current techniques construct elasticity through mechanisms such as joint training of several model sizes via architectural slicing of a pretrained network ( 17 ; 2 ; 7 ) or dynamic routing ( 3 ; 4 ) . Yet the way these methods extract submodels frequently limits the quality of the resulting efficiency–performance trade-offs. First, many methods predetermine a small set of submodel sizes, often chosen uniformly or heuristically, rather than discovering the configurations that truly lie on the Pareto frontier. Second, it is usually assumed that the pretrained model already contains viable subnetworks. Yet, there are no guarantees that these subnetworks were ever encouraged to be performant during training and often compete for representation capacity. To overcome these limitations, we propose FlexRank , a framework that first learns an importance ordering within each layer, then identifies how representational components contribute to the behavior of the full model. This yields a structured way of shrinking the network: smaller submodels correspond to truncating less important components while preserving the most influential ones. Building on this ordering, we then efficiently search for nested submodels that approximate the performance Pareto frontier. These submodels inherit their structure from the full model, rather than being independently trained, leveraging parameter sharing. We subsequently apply a refinement stage, allowing each chosen submodel to recover much of the accuracy of the original network. This two-stage process enables principled, nested elasticity without training multiple models from scratch or relying on architectural heuristics. Concretely, we make the following contributions: • We identify a key limitation of existing elasticity methods: subnetworks extracted from pretrained models are not encouraged to share useful representations, leading to severe degradation under aggressive compression. • We propose FlexRank , a principled framework that learns an importance ordering over representational components of a pretrained network and uses it to construct nested subnetworks with shared weights. • We introduce an efficient search and refinement procedure that discovers submodels lying close to Pareto frontier. • Through extensive empirical evaluation and supporting analysis, we demonstrate that FlexRank achieves smoother accuracy degradation and consistently outperforms state-of-the-art compression and elasticity approaches across DNNs, ViTs, and LLMs. 2 Preliminaries We first formally define the notion of an elastic model and discuss its desired properties. We then introduce the underlying objective of elastic training . 2.1 Elastic Models Versatile deployment requires models that can adapt to varying hardware constraints without storing separate parameters for every possible configuration. We formalize this requirement as follows. Let \mathcal{D} denote a data distribution and \mathcal{B} a set of computational budgets corresponding to realizable configurations, i.e . , \mathcal{B}=\{\beta_{i}\}_{i=1}^{N} . A model is defined as a tuple (f,\theta) , where f(\mathbf{d};\theta) is a function parameterized by shared weights \theta and \mathbf{d}\sim\mathcal{D} denotes the input. From a single set of shared parameters \theta , we derive a family of model realizations via \mathcal{T}_{\beta}(\cdot) , an algorithm-dependent transformation parameterized by the budget \beta\in(0,1] . We denote by \mathcal{R}(\cdot) and \mathcal{C}(\cdot) the performance and cost operators, respectively, subject to the constraint \mathbb{E}_{\mathbf{d}\sim\mathcal{D}}\left[\mathcal{C}(f(\mathbf{d};\mathcal{T}_{\beta}(\theta)))\right]\leq\beta . To illustrate, consider two common compression strategies. For sparsification, \mathcal{B} represents sparsity ratios and \mathcal{T}_{\beta}(\theta)=M_{\beta}\odot\theta , where M_{\beta}\in\{0,1\}^{d} is a binary mask with density proportional to \beta . For quantization, \mathcal{B} defines a set of relative bit-widths, and \mathcal{T}_{\beta} is the corresponding quantization operator. In this work, we focus on low-rank approximations of weight matrices. Specifically, each weight matrix \{W_{i}\in\mathbb{R}^{m_{i}\times n_{i}}\}\subset\theta is factorized as W_{i}=U_{i}V_{i}^{\top} , with factors U_{i}\in\mathbb{R}^{m_{i}\times r_{i}} and V_{i}\in\mathbb{R}^{n_{i}\times r_{i}} . The budget parameter \beta controls the total number of preserved parameters via a transformation \mathcal{T}_{\beta} that selects, for each layer, a subset of indices \mathcal{S}_{i}\subseteq[r_{i}]=\{1,2,\ldots,r_{i}\} and retains only the corresponding columns of U_{i} and V_{i} . The subsets \{\mathcal{S}_{i}\} are chosen across layers to satisfy a global budget constraint induced by \beta . We discuss how standard neural network layers can be parameterized within this framework in Section D.3 . More generally, \mathcal{T}_{\beta}(\cdot) may be any transformation. Ideally, we would use the optimal transformation \displaystyle\textstyle\mathcal{T}^{\star}_{\beta}(\theta)\in\arg\min_{\mathcal{T}_{\beta}(\theta)}\mathbb{E}_{\mathbf{d}\sim\mathcal{D}}\left[\mathcal{R}\big(f(\mathbf{d};\mathcal{T}_{\beta}(\theta))\big)\right]. (1) However, solving ( 1 ) is largely intractable, as it entails a combinatorial optimization problem, different for each \theta . Consequently, additional assumptions are required to render the problem tractable. Ultimately, our goal is to find \theta^{\star} that is Pareto optimal in the joint space of performance and cost. Definition 2.1 (Pareto Elastic Model) . An elastic model (f,\theta^{\star}) is optimally elastic if each configuration f(\cdot;\mathcal{T}^{\star}_{\beta}(\theta)) lies on the Pareto front \mathcal{P} of the objective space (\mathcal{R},-\mathcal{C}) . Formally, the model (f,\theta^{\star}) is optimally elastic iff for each \beta\in\mathcal{B} , there exists no \tilde{\theta} and \tilde{\mathcal{T}}_{\beta} such that: \displaystyle\textstyle\mathbb{E}_{\mathbf{d}\sim\mathcal{D}}\left[\mathcal{R}\big(f(\mathbf{d};\tilde{\mathcal{T}}_{\beta}(\tilde{\theta}))\big)\right]>\mathbb{E}_{\mathbf{d}\sim\mathcal{D}}\left[\mathcal{R}\big(f(\mathbf{d};\mathcal{T}^{\star}_{\beta}(\theta^{\star}))\big)\right]. Optimal elasticity characterizes an ideal weight-sharing regime in which a single parameter vector \theta simultaneously encodes optimal representations for all budgets. This formulation decouples representation learning—encoded in the shared parameters \theta —from budget-specific realization governed by the transformation \mathcal{T}_{\beta} . Achieving this property constitutes the central challenge addressed in this work. 2.2 Training Elastic Models The Pareto elastic model is idealized and serves as a conceptual upper bound rather than an attainable objective; in practice, we therefore turn to a more tractable optimization formulation that approximates Pareto elasticity, defined as \textstyle\hat{\theta}^{\star}\in\arg\min_{\theta}\,\,\sum_{\beta_{k}\in\mathcal{B}}\alpha_{k}\mathbb{E}_{\mathbf{d}\sim\mathcal{D}}\left[\mathcal{L}(f(\mathbf{d};\mathcal{T}^{\star}_{\beta_{k}}(\theta))\right].\vskip-5.69046pt (2) where \mathcal{L}(\cdot) is a task-specific loss, and \{\alpha_{k}>0\} are coefficients prioritizing different budget regimes. Note that ( 2 ) defines an implicit bilevel optimization problem, since the optimal transformation \mathcal{T}^{\star}_{\beta} itself depends on \theta . This problem is difficult to solve as gradient-based optimization cannot be directly applied due to the complexity of computing \mathcal{T}^{\star}_{\beta_{k}}(\theta) . Consequently, existing approaches typically either (i) optimize only the full model and then extract submodels from frozen parameters, or ii) optimize multiple submodels for different budgets and select the best-performing ones post hoc. As we argue in Section 4 , both strategies are suboptimal, motivating the need for a novel approach. 3 FlexRank We are now ready to introduce FlexRank , a principled framework for low-rank knowledge decomposition that enables elastic model scaling. As previously discussed ( Section 2.2 ), directly solving the resulting bilevel optimization is intractable for large models. We therefore focus on a practical and scalable setting in which a high-capacity pretrained base model is available, an assumption well aligned with modern deployment pipelines ( 1 ; 40 ; 16 ; 35 ) . Specifically, FlexRank takes as input a pre-trained network (f,\theta) and a calibration dataset \mathcal{Z} 1 1 1 In practice, this can be a representative subset of pretraining or downstream data, at the scale of 10^{3} samples. . Rather than jointly optimizing elastic submodels end-to-end, we leverage the base model to efficiently identify high-quality submodels that act as proxies for determining the transformation operator \mathcal{T}^{\star}_{\beta} . We make the key approximation that, although further optimization is required to obtain strong deployable models, the relative importance ordering induced by these submodels is preserved; that is, the optimal mask structure produced by \mathcal{T}^{\star}_{\beta} is fixed, allowing subsequent optimization to focus solely on the parameters. An overview of FlexRank is shown in Figure 1 . The method proceeds in three stages. First, each layer of the base model is independently decomposed via low-rank factorization, exploiting the fact that optimal per-layer decompositions admit a natural ordering ( Section 4.4 ). Second, assuming approximate additivity of approximation errors across layers, we compute a global importance ordering by solving a dynamic program, yielding a collection of nested submodels across budgets. This nested structure is crucial for limiting weight-sharing interference, as analyzed in Section 4 . Finally, since the decomposed submodels ignore cross-layer dependencies, we refine all configurations through knowledge distillation from the base model, producing a deployable elastic model with shared weights. We detail each component in the following sections. 3.1 Layer Decomposition To initialize the shared decomposed parameters \theta , we first perform a layer-wise factorization of the pretrained weights \{W_{i}\} . We seek an initialization that preserves the functional behavior of the original model and admits a closed-form solution, which we refer to as DataSVD . Specifically, we compute low-rank factors U_{i} and V_{i} by minimizing the output reconstruction error \textstyle\min_{U_{i},V_{i}}\;\mathbb{E}_{\mathbf{x}_{i}\sim\mathcal{X}_{i}}\left[\left\|(W_{i}-U_{i}V_{i}^{\top})\mathbf{x}_{i}\right\|_{2}^{2}\right], (3) where \mathbf{x}_{i} denotes the input activations to layer i and \mathcal{X}_{i} is the corresponding distribution. In practice, ( 3 ) can be approximated by sampling a matrix \mathbf{X}_{i}\in\mathbb{R}^{n_{i}\times N} containing activation vectors collected from a calibration dataset, with N chosen large enough to capture the principal directions of the data. As we show in Section C.1 and the space complexity can be made independent of N , scaling as \mathcal{O}(n_{i}^{2}) . Solving the objective via SVD crucially induces a natural ordering of rank components within each layer, which enables a tractable global selection of components across layers via dynamic programming, as described next. Remark 3.1 . Data-aware decompositions of this form have been considered in prior work (e.g., ( 6 ) ). However, in our method it serves only as initialization to extract candidate submodels from a pretrained network. Importantly, this initialization alone is not sufficient to recover good submodels, as we show in Section 5 . 3.2 Identifying the Pareto Front (a) Post-Training Selection (PTS) (b) All Submodel Learning (ASL) (c) Nested Submodel Learning (NSL) Figure 2: Nested trained submodels are Pareto Elastic: Comparison of the considered submodels training strategies on the synthetic setting described in Section D.1 . Blue points visualize all 1023 submodels, the red line represents the best models and the green one the true Pareto Front. The difference between the red and green lines is the best submodel optimality gap as per Equation 8 , and is zero only for NSL. Following the notation of Section 2.1 , for a given budget \beta_{k}\in\mathcal{B} , the transformation operator \mathcal{T}_{\beta_{k}} selects a rank r_{k,i}\leq r_{i} for each layer i , such that the overall budget constraint is satisfied. We denote the corresponding configuration vector by \mathbf{m}_{k}=\{r_{k,i}\}_{i=1}^{L} . We further denote by \mathcal{T}_{\mathbf{m}_{k}} the transformation induced by \mathbf{m}_{k} . To obtain an optimally elastic model, we seek a set of configurations \mathcal{M}:=\{\mathbf{m}_{k}\}_{k=1}^{K} that solve the following problem \displaystyle\textstyle\min_{\{\mathbf{m}_{k}\}_{k=1}^{K}}\sum_{k=1}^{K}\mathbb{E}_{\mathbf{d}\sim\mathcal{D}}\left[\mathcal{L}\bigl(f(\mathbf{d};\mathcal{T}_{\mathbf{m}_{k}}(\theta^{0}))\bigr)\right], (4) where \theta^{0} denotes the shared parameters obtained after layer-wise decomposition. Importantly, we optimize only over the masks \{\mathbf{m}_{k}\} , not over the parameters themselves. While \theta^{0} is not intended as a final deployable model, we assume it is sufficient to identify configurations whose relative optimality is preserved throughout subsequent training. Nestedness. We further impose a nestedness constraint \mathbf{m}_{k-1}\preceq\mathbf{m}_{k} in Equation 4 . This constraint is critical for limiting weight-sharing interference, as inconsistent rank selections across layers would prevent the shared parameters \theta from converging to a coherent representational hierarchy. We provide theoretical evidence for this phenomenon in the simplified setting of a single-layer network in Section 4 . Identifying optimal \mathcal{M} is a combinatorial challenge. Even when restricting to K candidate budgets (e.g., using linear spacing) across L layers, the search space contains K^{L} possible submodels, rendering brute-force evaluation infeasible. To make the problem tractable, we introduce additional structure. Since the initial decomposition is performed independently per layer, we implicitly assume that layers are approximately independent under \theta^{0} . Accordingly, we assume that the error incurred by low-rank approximations is additive across layers. While this is a strong assumption, it is standard in the literature (e.g., ( 21 ; 11 ) ) and enables an efficient dynamic programming solution with complexity \mathcal{O}(L\cdot K) , linear in both the number of layers and budgets. Implementation details are provided in Section C.2 . 3.3 Elastic Training Objective Once the set of optimal configurations \mathcal{M}^{\star}=\{\mathbf{m}^{\star}_{k}\}_{k=1}^{K} is identified, we fix these mappings for each budget level \beta_{k}\in\mathcal{B} . Thus, for the remainder of the training phase, the operator \mathcal{T}_{\mathbf{m}^{\star}_{k}} is fixed independent of the current \theta according to the obtained rank assignments in \mathbf{m}^{\star}_{k} . We optimize \theta by minimizing the distillation loss ( \mathcal{L}_{\text{KD}} ) between the elastic submodels and the original, non-decomposed pretrained model f(\cdot;\theta_{\text{orig}}) . Since a strong pretrained base model is available, training from teacher logits provides a substantially richer supervision signal than ground-truth labels. We define the loss for the k -th budget level as \ell_{k}(\theta)=\mathbb{E}_{\mathbf{d}\sim\mathcal{D}}\left[\mathcal{L}_{\text{KD}}\big(f(\mathbf{d};\mathcal{T}_{\mathbf{m}^{\star}_{k}}(\theta)),f(\mathbf{d};\theta_{\text{orig}})\big)\right]. (5) Therefore, the final objective becomes \textstyle\min_{\theta\in\mathbb{R}^{d}}\sum_{k=1}^{K}\alpha_{k}\ell_{k}(\theta),\quad\text{s.t. }\sum_{k=1}^{K}\alpha_{k}=1, (6) which can be efficiently solved by standard gradient-based optimization techniques, where in each step, we sample objective \ell_{k} proportionally to \alpha_{k} . 3.4 Recovering the True Pareto Front We empirically validate the central approximation underlying FlexRank that nested configurations identified from a fixed layer-wise optimal initialization are sufficient to recover Pareto-optimal submodels after training. In fact, we verify a stronger claim: that FlexRank recovers the true Pareto front obtained by independently training all possible submodels. This experiment is feasible only for small models due to the combinatorial search space. We therefore consider a four-layer network (two CNNs and two MLPs trained on MNIST, with K=10 rank levels per layer, yielding K^{L}=10{,}000 possible submodels. This allows us to exhaustively evaluate the attainable Pareto front by independently training all submodels. Figure 3: FlexRank recovers the true Pareto Front in DNNs: points represent independently trained nested submodels, starting from (i) a random weights (red) or (ii) from the DataSVD (green) of a pretrained model (yellow star), with best models highlighted with dashed lines. At convergence, FlexRank recovers the (in advance unknown) Pareto front within a single set of shared weights. Figure 3 compares three settings: independently trained submodels initialized at random and from DataSVD, and FlexRank . While nestedness and weight sharing impose additional constraints, the submodels learned by FlexRank consistently outperform the best models trained from scratch and, under extended training, converge to the Pareto front of independently trained DataSVD-initialized models. Here, extended training refers to training the elastic model until convergence, which requires more iterations since multiple submodels are optimized jointly; in contrast, the independently trained models are already converged and do not benefit from additional training. Notably, this result is stronger than required by our formulation: even though the independently trained submodels do not share parameters, FlexRank recovers their Pareto-optimal performance without incurring any loss due to weight sharing. 3.5 Efficient Low-Rank Inference Low-rank elastic models enable efficient inference by reducing the cost of matrix–vector multiplication from \mathcal{O}(mn) to \mathcal{O}((m+n-r)r) once the target rank r is fixed, which is strictly less than the \mathcal{O}(mn) FLOPs required for dense multiplication for any r<\min(m,n) . While the standard inference cost of a rank- r factorization is \mathcal{O}((m+n)r) , it can be further reduced by exploiting the non-uniqueness of the factorization. For any invertible G\in\mathbb{R}^{r\times r} , UV^{\top}=(UG)(VG^{-1})^{\top} . At inference time, choosing G such that the top r\times r block of UG is the identity reduces the cost of the U multiplication from \mathcal{O}(mr) to \mathcal{O}((m-r)r) . As a result, inference-time cost scales linearly with r , enabling computational gains without aggressive rank reduction. The matrix G is computed once per layer after rank selection via an r\times r matrix inversion, costing \mathcal{O}(r^{3}) FLOPs and negligible compared to SVD-based preprocessing. 4 The Need for Nestedness In this section, we discuss how to formulate elastic training in the rank space to obtain Pareto-optimal submodels. For linear models we prove: (i) why training only the full model does not recover optimal submodels; and that (ii) training all possible submodels leads to degradation of the Pareto Front. We finally prove the core result our method is based on: Pareto-optimal submodels are found by training only “nested” submodels. Proofs are deferred to Appendix B . 4.1 Setup Consider a linear model represented by a matrix M\in\mathbb{R}^{m\times n} , parameterized as M=UV^{\top} with U\in\mathbb{R}^{m\times k} , V\in\mathbb{R}^{n\times k} , and k=\min(m,n) . Let M^{\star} be the optimal solution, and assume it has SVD P\Sigma Q^{\top} satisfying: \Sigma=\mathrm{diag}(\sigma_{1},\dots,\sigma_{k}),\quad\sigma_{i}>\sigma_{i+1}>0\quad\forall i