FAR: Function-preserving Attention Replacement for IMC-friendly Inference

Yuxin Ren, Maxwell D Collins, Miao Hu, Huanrui Yang · 2025

Read FAR: Function-preserving Attention Replacement for IMC-friendly Inference by Yuxin Ren, Maxwell D Collins, Miao Hu, Huanrui Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21535.txt.

Introduction

Ren, Collins, Hu, and Yang propose in twenty twenty-five a way to make transformer AI models run well on memory-based chips. The attention step at the heart of transformers clashes with such chips, needing many cross-multiplications and far-flung memory access. That mismatch adds delay and data traffic on edge devices where these chips promise big energy savings.

Their framework swaps every attention block in a trained image model for a step-by-step module that suits the chip layout. Training by block-wise copying keeps the swap faithful to the original function. Extra pruning trims models to fit small chip arrays. Tests showed accuracy close to the original models, with fewer settings and lower delay.

Catalog record

CREATORYuxin Ren, Maxwell D Collins, Miao Hu, Huanrui Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-21535

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FAR: Function-preserving Attention Replacement for IMC-friendly Inference Yuxin Ren, Maxwell D Collins, Miao Hu, Huanrui Yang Abstract . While transformers dominate modern vision and language models, their attention mechanism remains poorly suited for in-memory computing (IMC) devices due to intensive activation-to-activation multiplications and non-local memory access, leading to substantial latency and bandwidth overhead on ReRAM-based accelerators. To address this mismatch, we propose FAR, a Function-preserving Attention Replacement framework that substitutes all attention in pretrained DeiTs with sequential modules inherently compatible with IMC dataflows. Specifically, FAR replaces self-attention with a multi-head bidirectional LSTM architecture via block-wise distillation to retain functional equivalence while enabling linear-time computation and localized weight reuse. We further incorporate structured pruning on FAR models, enabling flexible adaptation to resource-constrained IMC arrays while maintaining functional fidelity. Evaluations on the DeiT family demonstrate that FAR maintains comparable accuracy to the original attention-based models on ImageNet and multiple downstream tasks with reduced parameters and latency. Further analysis shows that FAR preserves the semantic token relationships learned by attention while improving computational efficiency, highlighting its potential for energy-efficient transformer inference on IMC-based edge accelerators. Conference: Great Lakes Symposium on VLSI 2026; June 22–24, 2026; Canandaigua, NY, USA Great Lakes Symposium on VLSI 2026 (GLSVLSI ’26), June 22–24, 2026, Canandaigua, NY, USA DOI: 10.1145/3787109.3816372 ISBN: 979-8-4007-2431-2/2026/06 † † cc-license: by 1. Introduction IMC crossbar illustration Figure 1. IMC crossbar illustration IMC crossbar illustration Transformer architectures have become the dominant backbone in multimodal, vision, and language tasks ( Vaswani et al., 2017 ; Devlin et al., 2019 ; Dosovitskiy et al., 2021 ; et al., 2021a ) . Their effectiveness largely due to self-attention, which enables each token to integrate global information and to build rich contextual representations during large-scale pretraining ( Cordonnier et al., 2020 ) . Despite high expressive power, self-attention introduces substantial overhead: its pairwise token interactions require \mathcal{O}(T^{2}) computation and generate heavily data-dependent memory traffic ( Tay et al., 2022 ) . These properties align well with highly parallel digital processors such as GPUs and TPUs, but they fundamentally conflict with the computation model of emerging in-memory computing (IMC) accelerators. IMC systems, particularly ReRAM crossbars, excel when computation can be expressed as weight-stationary vector–matrix multiplications. As illustrated in Fig. 1 , applying input voltages along wordlines and sensing accumulated currents on bitlines naturally realizes analog GEMM with minimal data movement and high energy efficiency ( Yang et al., 2020b ; Wolters et al., 2024 ) . However, attention is dominated not by weight-stationary GEMM, but by activation-to-activation multiplications such as QK^{\top} , softmax normalization, and per-token dynamic mixing. These operations require repeatedly reading spatially distributed activations and routing intermediate results across arrays, leading to fragmented analog operations, extensive ADC/DAC conversions, and poor crossbar utilization. Consequently, many IMC accelerators offload attention to digital compute units, while only mapping feed-forward layers onto ReRAM, leaving the quadratic attention bottleneck unresolved at inference time. Meanwhile, recent analyses of transformers consistently show that attention layers exhibit substantial redundancy at inference time ( et al., 2021b ; He et al., 2025 ) . Across depth, attention maps tend to become progressively smoother and more compressible, and many heads can be well-approximated by low-rank or weakly varying interaction patterns. Based on these observations, we posit that the functional role of an attention block during inference is often closer to a stable, smooth sequence-to-sequence mapping than to a fully dynamic all-pairs interaction. This motivates our investigation into whether such structured behavior can be captured by sequential modules whose computation patterns naturally align with the locality and weight-stationarity properties preferred by IMC hardware. Motivated by these observations, we introduce FAR , a framework that replaces every attention block in a pretrained transformer with a sequential module whose computation aligns with IMC execution, and trains them through head-level, block-wise distillation so that it faithfully replicates the original attention behavior. All transformer components outside the replaced blocks remain frozen during distillation, ensuring that FAR functions as a drop-in architectural conversion rather than training from scratch. The resulting model thus preserves both the pretrained representation quality and the downstream transfer performance of the original transformer. Block-wise replacement of attention. Each replaced module is supervised by a similarity loss, while a classification loss is applied at the output. During distillation, only the replacement blocks are updated. Figure 2. Block-wise replacement of attention. Each replaced module is supervised by a similarity loss, while a classification loss is applied at the output. During distillation, only the replacement blocks are updated. Block-wise replacement of attention. Each replaced module is supervised by a similarity loss, while a classification loss is applied at the output. During distillation, only the replacement blocks are updated. We adopt an LSTM-based module as the sequential substitute because its computation pattern is inherently more compatible with IMC execution: LSTMs ( Hochreiter and Schmidhuber, 1997 ) reuse a fixed set of weights across time steps, avoid activation-to-activation multiplications, and follow a strictly sequential dataflow with high weight locality. These properties map naturally onto ReRAM crossbars, in sharp contrast to attention’s non-local activation mixing and quadratic interaction pattern, which incur significant data movement on IMC hardware. To retain the inductive bias of multi-head attention, where different heads operate on distinct subspaces and learn complementary token-mixing patterns, we design a multi-head bidirectional LSTM (BiLSTM) structure in which each head processes a projected slice of the embedding. This organization enables the recurrent heads to specialize in different interaction patterns, and the bidirectional recurrence provides forward and backward contextual coverage that reflects the contextual behavior of self-attention. To further reduce the computational footprint for IMC deployment, we apply sparsity-inducing regularization ( Yang et al., 2020a ) on the hidden dimensions of each LSTM head, enabling layer-adaptive compression that removes redundant recurrent channels while respecting LSTM gate coupling. This yields compact sequential modules whose capacity matches the functional complexity of each layer and better fits the limited array sizes and bandwidth of IMC hardware. We validate FAR on the DeiT ( Touvron et al., 2021 ) vision transformer family and show that it preserves the accuracy scaling behavior of attention-based models across model sizes. FAR slightly improves performance on DeiT-Tiny and remains competitive on DeiT-Small and DeiT-Base, demonstrating that sequential substitutes can retain the representational behavior learned during large-scale pretraining even without all-pairs token interactions. FAR models also show good generalizability when finetuned to downstream classification tasks, indicating that the learned sequence-to-sequence mappings remain effective across domains. Beyond algorithmic quality, hardware-oriented ReRAM simulations show that FAR enables a sequential and weight-stationary execution pattern that substantially reduces memory traffic and improves inference latency on IMC accelerators. These results indicate that attention can be replaced end-to-end at inference time while maintaining practical accuracy, providing a viable and hardware-efficient alternative for deploying pretrained transformers on IMC-based systems. 2. Related Work Efficient Alternatives to Self-Attention. The quadratic cost of self-attention has motivated both attention-free and attention-efficient architectures. Attention-free models such as MLP-Mixer ( et al., 2021c ) , RetNet ( Sun et al., 2024 ) , and Mamba ( Gu and Dao, 2024 ) replace attention with MLP-based, recurrent, or state-space token mixing, achieving linear-time complexity and improved hardware efficiency. However, these models are trained from scratch and cannot leverage pretrained transformers, limiting

Read the complete text