I-Segmenter: Integer-Only Vision Transformer for Efficient Semantic Segmentation

Jordan Sassoon, Michal Szczepanski, Martyna Poreba · 2025

Read I-Segmenter: Integer-Only Vision Transformer for Efficient Semantic Segmentation by Jordan Sassoon, Michal Szczepanski, Martyna Poreba (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-10334.txt.

Introduction

Jordan Sassoon, Michal Szczepanski, and Martyna Poreba squeeze vision transformers onto small devices in this 2025 paper. They introduce I-Segmenter for semantic segmentation. That task labels an image pixel by pixel. Transformers segment well, but they demand heavy memory and compute. Simply quantizing them to low precision usually breaks accuracy. Errors pile up through the network.

I-Segmenter is, by the authors' account, the first fully integer-only framework of its kind. It swaps every floating-point operation for an integer counterpart. It adds a new activation function. That function tames long-tailed value distributions, a known weak point of uniform quantization. Two decoder steps get integer-friendly replacements as well. The result stays within a modest margin of the full-precision baseline. It shrinks the model up to about fourfold, and it runs faster. It even holds up when calibrated with a single image.

Catalog record

CREATORJordan Sassoon, Michal Szczepanski, Martyna Poreba
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-10334

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

I-Segmenter: Integer-Only Vision Transformer for Efficient Semantic Segmentation Jordan Sassoon, Michal Szczepanski, Martyna Poreba Abstract Vision Transformers (ViTs) have recently achieved strong results in semantic segmentation, yet their deployment on resource-constrained devices remains limited due to their high memory footprint and computational cost. Quantization offers an effective strategy to improve efficiency, but ViT-based segmentation models are notoriously fragile under low precision, as quantization errors accumulate across deep encoder–decoder pipelines. We introduce I-Segmenter, the first fully integer-only ViT segmentation framework. Building on the Segmenter architecture, I-Segmenter systematically replaces floating-point operations with integer-only counterparts. To further stabilize both training and inference, we propose λ \lambda -ShiftGELU, a novel activation function that mitigates the limitations of uniform quantization in handling long-tailed activation distributions. In addition, we remove the L2 normalization layer and replace bilinear interpolation in the decoder with nearest-neighbor upsampling, ensuring integer-only execution throughout the computational graph. Extensive experiments show that I-Segmenter achieves accuracy within a reasonable margin of its FP32 baseline ( 5.1 % 5.1\% on average), while reducing model size by up to 3.8 × 3.8\times and enabling up to 1.2 × 1.2\times faster inference with optimized runtimes. Notably, even in one-shot PTQ with a single calibration image, I-Segmenter delivers competitive accuracy, underscoring its practicality for real-world deployment. I Introduction By enabling global context modeling, Vision Transformers (ViTs) have pushed semantic segmentation to new state-of-the-art results [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 ] . These modern frameworks typically adopt ViTs as encoders combined with either CNN-based or Transformer-based [ 12 ] decoders, surpassing the precision of fully CNN-based architectures. Achieving accurate segmentation, however, requires preserving fine local details across high-resolution inputs. Meeting the need for dense patch-level information places a heavy burden on ViTs, as the self-attention mechanism scales quadratically with input size, causing sharp increases in memory and computation. To cope with this complexity, several research directions have been investigated [ 13 , 14 , 15 , 16 ] . Compact architectures [ 17 , 18 , 19 , 20 , 21 , 22 ] redesign Transformer components to balance accuracy with efficiency. Model compression techniques such as pruning [ 23 , 24 , 25 , 26 , 27 , 28 , 29 ] , quantization [ 30 , 31 , 32 , 33 , 34 , 35 , 36 ] , and knowledge distillation [ 37 , 38 ] aim to reduce computational cost without severely impacting accuracy. Fig. 1 : Overview of our proposed model, I-Segmenter. The figure summarizes both the architectural modifications and the main efficiency gains introduced by I-Segmenter. On the top, we illustrate the layer replacement strategy where each FP32 operator in Segmenter is mapped to its integer-only counterpart. This design enables full integer execution throughout the encoder and decoder. On the bottom, we report the impact of this quantization strategy across model scales, benchmarked on the TVM inference framework. Quantization reduces numerical precision by representing values and operations with fewer bits, mapping them from high-precision formats to lower-precision ones. This yields faster, smaller, and more memory-efficient models. Integer quantization offers substantial advantages for model deployment. By converting floating-point representations (FP) into compact integer formats (INT), it reduces memory footprint and bandwidth requirements, accelerates inference on hardware optimized for integer arithmetic, and lowers energy consumption. Such gains are crucial for edge deployment on constrained hardware, including ARM Cortex-M microcontrollers, Qualcomm Snapdragon CPUs, and dedicated NPUs such as Google Edge TPU or ARM Ethos-U. Two main strategies dominate: post-training quantization (PTQ) and quantization-aware training (QAT). PTQ converts a pretrained model to low precision using a small calibration set, enabling fast deployment but frequently degrading accuracy. In contrast, QAT simulates quantization effects during training, allowing the model to adapt its weights to low-precision constraints, often achieving higher accuracy at the cost of increased training complexity. In general, 8- or 6-bit quantization results in about a 3% drop in Top-1 accuracy on ImageNet-1k [ 39 ] . At 4 bits, this degradation becomes much more pronounced, with early methods showing losses of up to 40%. At 3 bits, the loss is even more severe, with accuracy collapsing dramatically and some models becoming nearly unusable [ 35 ] . A central challenge in ViT quantization comes from non-linear operations. Small numerical errors can be disproportionately amplified. In Softmax, tiny logit perturbations cause large shifts in attention, distorting token relationships. GELU adds further difficulty, its smooth non-linearity and long-tailed activations make it hard to preserve both small values and outliers under low precision. Although quantization has shown promising results for ViTs in image classification, its application to semantic segmentation remains largely unexplored. Segmentation is particularly sensitive to quantization errors, as it requires dense pixel-level predictions. While individual ViT components can tolerate reduced precision, errors accumulate across the many encoder blocks, making the model highly vulnerable under low-bit quantization. The addition of a decoder head further complicates the problem, since multi-scale aggregation and nonlinear operations can amplify errors propagated from the backbone. In this work, we present the first systematic study of quantizing ViT-based models for semantic segmentation. Our contributions are the following: • We propose I-Segmenter, which, to the best of our knowledge, is the first ViT-based semantic segmentation model supporting full integer-only inference, extending the Segmenter architecture [ 4 ] with integer-friendly components inspired by I-ViT [ 40 ] . • We introduce \lambda -ShiftGELU, a novel integer-compatible activation function that stabilizes QAT and PTQ for dense prediction tasks. • We remove the L2 normalization step and replace the floating-point bilinear upsampling with nearest neighbor interpolation, a lightweight and integer-friendly alternative widely supported in inference engines, and perform an ablation study to assess their impact on segmentation accuracy. • Through a comprehensive evaluation across datasets, model scales, and input sizes, we show that I-Segmenter maintains accuracy within a negligible margin of its FP32 counterpart (no more than 6.7\% loss), while reducing model size by up to 3.8\times and achieving up to 1.2\times faster inference with optimized runtimes such as TensorRT [ 41 ] and TVM [ 42 ] (Figure 1 ). • We evaluate I-Segmenter in the one-shot PTQ setting, where quantization is performed using a single calibration image. This approach is highly practical but also challenging. Remarkably, we achieve competitive accuracy with just 1 second of calibration time. • We analyze the role of inference backends (ONNX Runtime [ 43 ] , TensorRT, and TVM) in exploiting integer-only execution, highlighting both performance benefits and engineering challenges for deployment on edge devices. By leveraging TVM, we demonstrate that I-Segmenter can be executed entirely with integer arithmetic, down to the level of individual kernel computations. II Background and Related Work II-A Segmentation Meets Vision Transformers ViTs were originally designed for image classification, where a single global representation, extracted from the [CLS] token, is used by a multilayer perceptron (MLP) head to output one class label for the entire image. In segmentation, however, the goal shifts to assigning multiple class labels across an image at the pixel or region level. Different ViT-based models achieve this objective through distinct decoding strategies. SETR [ 1 , 2 ] was a pioneering effort that demonstrated the viability

Read the complete text