Not All Tasks Quantize Equally: Fisher-Guided Quantization for Visual Geometry Transformer

Yipu Zhang, Jintao Cheng, Weilun Feng, Jiehao Luo, Chuanguang Yang, Zhulin An, Yongjun Xu, Wei Zhang · 2026

Read Not All Tasks Quantize Equally: Fisher-Guided Quantization for Visual Geometry Transformer by Yipu Zhang, Jintao Cheng, Weilun Feng, Jiehao Luo, Chuanguang Yang, Zhulin An, Yongjun Xu, Wei Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15828.txt.

Introduction

Yipu Zhang and seven colleagues address quantization of feed-forward 3D reconstruction models like VGGT. Such models predict depth, camera pose and point clouds through a shared backbone. Yet different blocks and channels aid each task unevenly, so equal treatment overemphasizes insensitive tasks and harms sensitive ones. Their Fisher-Guided Quantization uses the diagonal Fisher information matrix to measure sensitivity across tasks, blocks and channels. It feeds this into a learnable affine transformation. Tests show it beats baselines on VGGT, with up to 39 percent relative improvement under 4-bit quantization.

Catalog record

CREATORYipu Zhang, Jintao Cheng, Weilun Feng, Jiehao Luo, Chuanguang Yang, Zhulin An, Yongjun Xu, Wei Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49993
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.15828

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Not All Tasks Quantize Equally: Fisher-Guided Quantization for Visual Geometry Transformer Yipu Zhang, Jintao Cheng, Weilun Feng, Jiehao Luo, Chuanguang Yang, Zhulin An, Yongjun Xu, Wei Zhang Abstract Feed-forward 3D reconstruction models, represented by Visual Geometry Grounded Transformer (VGGT), jointly predict multiple visual geometry tasks such as depth estimation, camera pose prediction, and point map reconstruction in a single forward pass. They have been widely adopted in 3D vision applications, but their billion-scale parameters bring substantial memory and computation overhead, posing challenges for on-device deployment. Post-Training Quantization (PTQ) is an effective technique to reduce this overhead. Existing PTQ methods for feed-forward 3D models mainly focus on handling heavy-tailed activation distributions and constructing diverse calibration datasets. However, we observe that feed-forward 3D models predict multiple geometric attributes through a shared backbone, where different transformer blocks and hidden channels contribute distinctly to each task, resulting in substantially different sensitivities to quantization errors across tasks, blocks, and channels. Consequently, treating all tasks equally over-emphasizes insensitive tasks and causes significant accuracy loss on the sensitive ones. To address this issue, we propose Fisher-Guided Quantization ( FGQ ) for feed-forward 3D reconstruction models. Specifically, FGQ uses the diagonal Fisher information matrix to quantify the different sensitivities across tasks, blocks, and channels, and incorporates these sensitivities into the Learnable Affine Transformation during calibration to better preserve the channels and blocks most critical to each task. Extensive experiments across camera pose estimation, point map reconstruction, and depth estimation show that FGQ consistently outperforms state-of-the-art quantization baselines on VGGT, achieving up to 39% relative improvement under the 4-bit quantization. 2 2 footnotetext: Equal contribution. 1 1 footnotetext: Corresponding author:Wei Zhang< [email protected] >. Figure 1: FGQ effectively preserves the point map reconstruction fidelity under 4-bit quantization, surpassing previous state-of-the-art VGGT (QuantVGGT ( 2 ) ) quantization. 1 Introduction Feed-forward 3D reconstruction methods, represented by Visual Geometry Grounded Transformer (VGGT) ( 21 ) , have achieved significant progress by jointly predicting depth, camera poses, and point maps in a single forward pass. They have been widely adopted in downstream applications such as spatial reasoning ( 5 ) , robotics ( 25 ) , and autonomous driving ( 10 ) . However, their billion-scale parameters introduce prohibitive computation and memory overhead during inference, limiting real-time processing and on-device deployment ( 26 ) . A variety of methods have been proposed to address this efficiency bottleneck, including quantization ( 2 ; 26 ; 13 ) , token merging ( 16 ; 18 ) , and architectural modifications that redesign the attention or computation flow for better efficiency ( 23 ; 19 ) . Existing PTQ methods for feed-forward 3D models primarily address two challenges: (i) mitigating activation outliers associated with special tokens (e.g., camera and register tokens), and (ii) capturing the geometric and semantic complexity of 3D representations during calibration dataset construction. However, these approaches overlook a critical characteristic of these models: their inherent multi-task architecture. Since depth, pose, and point map prediction operate at vastly different scales and are evaluated with distinct metrics, they exhibit markedly different sensitivities to quantization-induced perturbations. Figure 2: Normalized Performance Degradation (NPD) across various tasks. As illustrated in Fig. 2 , applying a uniform quantization strategy causes disproportionately severe degradation in point map reconstruction compared to camera pose and depth estimation. Beyond this task-level disparity, Fig. 3 (a)&(b) further shows that the per-block and per-channel quantization loss follows markedly different patterns across tasks. Together, these observations confirm that different transformer blocks and hidden channels contribute distinctly to each task, leading to substantially different quantization sensitivities across tasks, blocks, and channels. To this end, we propose Fisher-Guided Quantization ( FGQ ), a task-aware PTQ framework for feed-forward 3D reconstruction models. FGQ leverages the diagonal Fisher information matrix to jointly quantify quantization sensitivity across three granularities: tasks (depth, pose, and point map), transformer blocks, and hidden channels. These sensitivities are then incorporated into the Learnable Affine Transformation during calibration, guiding the transformation to preserve the channels and blocks most critical to each task. Our contributions are summarized as follows: • We identify the multi-task sensitivity problem in PTQ for feed-forward 3D reconstruction, showing that different transformer blocks and hidden channels contribute distinctly to each task, leading to substantially different quantization sensitivities across tasks, blocks, and channels. • We propose Fisher-Guided Quantization ( FGQ ), which uses the diagonal Fisher information matrix to quantify per-task, per-block, and per-channel sensitivity, and integrates these sensitivities into the Learnable Affine Transformation objective to guide calibration toward the most quantization-sensitive components. • Extensive experiments on camera pose estimation, point map reconstruction, and depth estimation demonstrate that FGQ improves task performance by up to 39% over state-of-the-art PTQ baselines under 4-bit quantization, with the largest gains on point map reconstruction, where prior PTQ methods incur the most significant degradation. 2 Related Work Efficient Feed-Forward 3D Reconstruction. Recent work targets the high compute and memory cost of feed-forward 3D reconstruction models such as VGGT. Token merging methods like FastVGGT ( 16 ) and LiteVGGT ( 18 ) fuse similar tokens by exploiting redundancy in global attention. Architectural redesigns restructure attention itself: FlashVGGT ( 23 ) approximates global attention as cross-attention to a compact set of descriptor tokens generated by spatial resampling, while AVGGT ( 19 ) converts early global layers into frame attention and subsamples K/V over patch tokens. Closest to our work, recent PTQ methods ( 2 ; 26 ) address two key challenges in quantizing these models: (i) mitigating activation outliers from data-independent special tokens (e.g., camera and register tokens), which yield heavy tails and extreme channel and token variance unfriendly to low-bit quantization; and (ii) dealing with calibration data that captures the geometric and semantic complexity of 3D multi-view inputs. However, prior PTQ methods apply uniform quantization across all heads and treat feed-forward 3D models as single-task models, overlooking the distinct quantization sensitivities of their depth, pose, and point map predictions. Model Quantization. Post-training quantization (PTQ) reduces the memory and compute cost of large language models. Weight-only methods such as GPTQ ( 3 ) and AWQ ( 9 ) compress weights to low bit-widths with minimal accuracy loss. To further quantize activations and the KV cache, a recent line of work applies affine transformations to suppress outliers before quantization. QuaRot ( 1 ) uses fixed random Hadamard rotations, while SpinQuant ( 11 ) and FlatQuant ( 20 ) instead learn the transformation matrices during calibration: SpinQuant optimizes rotation matrices via Cayley optimization, and FlatQuant learns lightweight per-layer affine transformations that flatten weight and activation distributions. To better involve the sensitivity in the quantization process, we adopt the learnable affine transform based method as detailed in Sec. 3.2.3 . 3 Method 3.1 Prelinminary Visual Geometry Grounded Transformer (VGGT) ( 21 ) is a feed-forward transformer that maps a set of N input images (I_{i})_{i=1}^{N} to their 3D attributes (camera parameters, depth maps, point maps, and tracking features) in a single pass. Each image I_{i} is first patchified by a DINOv2 ( 12 ) encoder into K tokens t_{i}\in\mathbb{R}^{K\times C} , and the full token set is denoted \mathbf{T}=[t_{1};t_{2};\ldots;t_{N}]\in\mathbb{R}^{N\times K\times C} . The backbone consists of L Alternating-Attention (AA) blocks, each interleaving two self-attention operations: \mathbf{T}_{i}^{(\ell+\tfrac{1}{2})}=\mathrm{FrameAttention}\big(\mathbf{T}_{i}^{(\ell)}\big),\quad i=1,\ldots,N, (1) \mathbf{T}^{(\ell+1)}=\mathrm{GlobalAttention}\big(\mathbf{T}^{(\ell+\tfrac{1}{2})}\big), (2) where \ell\in\{0,1,\ldots,L-1\} indexes the

Read the complete text