Latent Flow Transformer

Yen-Chen Wu Feng-Ting Liao Meng-Hsi Chen Pei-Chen Ho Farhang Nabiei Da-shan Shiu MediaTek Research · 2025

Read Latent Flow Transformer by Yen-Chen Wu Feng-Ting Liao Meng-Hsi Chen Pei-Chen Ho Farhang Nabiei Da-shan Shiu MediaTek Research (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14513.txt.

Introduction

Yen-Chen Wu and co-authors wrote a 2025 paper on shrinking the layer stacks of large language models. Such models stack tens to hundreds of layers, and more layers cost more to run. Work on image models showed smooth, flow based layers can beat stacks of fixed ones.

Their design swaps a block of layers for one learned move operator trained by flow matching. It stays compatible with the base model's shape. A new walk method keeps the links between steps that past flow methods lost. On a mid-size open model, the swapped form beat plain layer skips at the same cut size. With the walk method, a dozen layers folded into one with a small quality loss. The authors see a bridge between the two model styles.

Catalog record

CREATORYen-Chen Wu Feng-Ting Liao Meng-Hsi Chen Pei-Chen Ho Farhang Nabiei Da-shan Shiu MediaTek Research
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14513

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Latent Flow Transformer Yen-Chen Wu Feng-Ting Liao Meng-Hsi Chen Pei-Chen Ho Farhang Nabiei Da-shan Shiu MediaTek Research Abstract Transformers, the standard implementation for large language models (LLMs), typically consist of tens to hundreds of discrete layers. While more layers can lead to better performance, this approach has been challenged as far from efficient, especially given the superiority of continuous layers demonstrated by diffusion and flow-based models for image generation. We propose the Latent Flow Transformer (LFT), which replaces a block of layers with a single learned transport operator trained via flow matching, offering significant compression while maintaining compatibility with the original architecture. Additionally, we address the limitations of existing flow-based methods in preserving coupling by introducing the Flow Walking (FW) algorithm. On the Pythia-410M model, LFT trained with flow matching compresses 6 of 24 layers and outperforms directly skipping 2 layers (KL Divergence of LM logits at 0.407 vs. 0.529), demonstrating the feasibility of this design. When trained with FW, LFT further distills 12 layers into one while reducing the KL to 0.736 surpassing that from skipping 3 layers (0.932), significantly narrowing the gap between autoregressive and flow-based generation paradigms. 1 Introduction Transformer-based large language models (LLMs) have become foundational to progress in NLP. While their scale enables remarkable capabilities, it also introduces substantial memory and compute demands, limiting their accessibility. Although evidence suggests many layers are overparameterized zafrir2021prune , mainstream compression techniques like pruning and knowledge distillation lu_reassessing_2024 ; xu2024surveyknowledgedistillationlarge have yet to deliver the scale of reduction required to fully address these limitations. In image generation, diffusion and flow matching (FM) models lipman2022flow have demonstrated remarkable efficiency, with recent approaches generating high-quality images in as few as one sampling pass frans_one_2024 . Inspired by this success, several works have explored adapting FM and diffusion models for language modeling sun2019patientknowledgedistillationbert ; xu2024surveyknowledgedistillationlarge . While these approaches offer promising directions, their performance still trails behind SOTA conventional transformer models—likely due to the discrete and compositional nature of language, where token-level generation resists continuous approximation. Recent studies frame transformer forward passes as discretized continuous dynamical processes chen2018neural ; li-etal-2022-ode ; feinashley2025flowinglayerscontinuousdynamical , and show that latent trajectories across layers can be nonlinear and complex. Concurrently, in image modeling, it has been shown that efficient surrogate models can be trained to approximate the latent mapping of deep networks using straighter transport dynamics das2023imagegenerationshortestpath ; song2023consistencymodels . These developments motivate exploring whether flow-based approaches can learn more efficient inter-layer mappings in transformers, offering a new path to structural compression. Building upon these insights, We propose the Latent Flow Transformer (LFT) , a novel transformer variant designed for parameter-and-compute efficient language modeling by replacing a contiguous block of transformer layers with a single learned transport operator trained via flow matching lipman2022flow . To guide which layers are suitable for replacement, we introduce the Recoupling Ratio , an interpretable Optimal Transport-based metric that accurately predicts flow matching quality. Our analysis finds that middle layers are particularly compressible, supporting the intuition that early LLM layers function qualitatively differently from later ones. In our experiments training the LFT, with only 10^{8} training tokens, we show that standard flow matching enables effective LFT training, with perturbations smaller than those caused by removing a single transformer layer Sajjad_2023 . However, performance plateaus as flow path proximity introduces velocity ambiguity kim_simulation-free_2024 ; park2024constantaccelerationflow —a known limitation in FM models. While reflow techniques are a common strategy to mitigate this issue, they are inherently incompatible with the LFT’s objective of preserving the original input-output latent relationship. To address this limitation, while advanced flow matching techniques offer potential solutions, we introduce Flow Walking (FW) , a scalable algorithm for learning latent transport using numerical integration. By training a transformer-like policy to explore paths with clearer view to the target, FW enables the model to navigate and resolve local ambiguities arising from multiple nearby flow paths, thereby significantly improving transport alignment and overcoming the performance plateau observed with standard flow matching. We conduct extensive experiments on the Pythia-410M model to validate the proposed framework. First, we find that layer selection is critical: LFTs constructed by replacing the layers identified by the Recoupling Ratio consistently outperform those using arbitrary layer selections. Second, we demonstrate that LFT trained via flow matching can compress 12 out of 24 layers into a single transport layer while achieving a lower performance drop (KL divergence of LM logits at 0.407) than simply skipping two layers (KL divergence of LM logits at 0.529), thereby establishing the feasibility of our architecture. Finally, when trained with FW under the same compression setting, the LFT achieves a substantially lower KL divergence of LM logits at 0.736 than that with skipping three layers at 0.932, overcoming the primary obstacle in aligning latent transport across distant transformer layers and effectively bridging autoregressive and flow-based modeling paradigms. In summary, our contributions are as follows. • We introduce the Latent Flow Transformer, a novel architecture for efficient language modeling that replaces a block of transformer layers with a single learned transformer-like layer based on flow-matching principles. • We develop an interpretable predictor Recoupling Ratio that estimates the suitability of different transformer layers for replacement. Our analysis with the predictor suggests that the middle layers of transformer models are particularly amenable to this form of compression, supporting hypotheses about the differentiated roles of layers in LLMs. • We demonstrate that while standard flow matching can learn the LFT’s transport operator with a modest data budget, it encounters a performance plateau stemming from challenges such as flow path proximity in the latent space. • We propose Flow Walking, a scalable explorative algorithm for learning latent transport trajectories, designed to effectively navigate and resolve flow path proximity issues. • We empirically validate the LFT trained with both FM and FW on the Pythia -410M model, demonstrating substantial parameter reduction with limited initial performance degradation and significant capability recovery after fine-tuning. We release our code for the community to reproduce our results at https://github.com/mtkresearch/latent-flow-transformer 2 Preliminary 2.1 Continuous-Time perspective of transformer layers A neural net with L discrete layers, especially with a residual connection, can be seen as refining its input hidden state h_{l-1} at layer l , l=1,2,...,L , to a better state at its output, h_{l} . A continuous-time counterpart can be expressed by the following dynamics chen2018neural ; chen2023contiformer ; papamakarios2021normalizing : \displaystyle v_{t}=\frac{dh_{t}}{dt}=u_{\theta}(h_{t},t) (1) where \theta parameterizes the ordinary differential equation (ODE), and h_{t} matches with h_{l} whenever t matches l/L . Learning \theta involves conducting expensive simulation kidger2022neural : \displaystyle h_{t_{2}}=h_{t_{1}}+\int_{t_{1}}^{t_{2}}u_{\theta}(h_{t},t)\,dt (2) 2.2 Flow Matching for Simulation-Free Training Flow-based approaches albergo2022building ; lipman2022flow ; liu2022rectified provide a simulation-free solution to the ODE defined in Equation 1 . Flow matching moves a particle at initial position x_{0} to its target position x_{1} via a predefined path in unit time, and learns the corresponding velocity field lipman2022flow conditioned on a set of (x_{0},x_{1}) pairs by minimizing the following loss, \mathcal{L}_{FlowMatching}=\mathbb{E}_{t}\left[\|u_{\theta}(x_{t},t)-v_{t}\|^{2}\right], (3) where x_{t} and v_{t} are the position and the velocity of the particle at time t , respectively. A straight line, constant speed trajectory is a popular and practical choice for flow matching. With such a target trajectory, x_{t} and v_{t} are simply x_{t}=(1-t)\cdot x_{0}+t\cdot x_{1} ,

Read the complete text