VersaQ‑3D: A Reconfigurable Accelerator Enabling Feed‑Forward and Generalizable 3D Reconstruction via Versatile Quantization

Yipu Zhang, Jintao Cheng, Xingyu Liu, Zeyu Li, Carol Jingyi Li, Jin Wu, Lin Jiang, Yuan Xie, Jiang Xu, Wei Zhang ∗ · 2026

Read VersaQ‑3D: A Reconfigurable Accelerator Enabling Feed‑Forward and Generalizable 3D Reconstruction via Versatile Quantization by Yipu Zhang, Jintao Cheng, Xingyu Liu, Zeyu Li, Carol Jingyi Li, Jin Wu, Lin Jiang, Yuan Xie, Jiang Xu, Wei Zhang ∗ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20317.txt.

Introduction

Yipu Zhang and ten colleagues present VersaQ-3D, a reconfigurable accelerator chip for fast 3D reconstruction. New transformer models can rebuild a 3D scene directly from images, with no per-scene tuning. But a billion-parameter model demands too much memory and compute for instant on-device use.

VersaQ-3D uses versatile quantization, representing numbers at mixed precision, to slim the model for hardware. A reconfigurable design lets one chip serve both feed-forward reconstruction and related tasks. Fast on-device 3D would help augmented reality, robot perception, and digital twins respond in real time. That is the gap this chip is built to close.

Catalog record

CREATORYipu Zhang, Jintao Cheng, Xingyu Liu, Zeyu Li, Carol Jingyi Li, Jin Wu, Lin Jiang, Yuan Xie, Jiang Xu, Wei Zhang ∗
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-20317

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VersaQ‑3D: A Reconfigurable Accelerator Enabling Feed‑Forward and Generalizable 3D Reconstruction via Versatile Quantization Yipu Zhang, Jintao Cheng, Xingyu Liu, Zeyu Li, Carol Jingyi Li, Jin Wu, Lin Jiang, Yuan Xie, Jiang Xu, Wei Zhang ∗ Abstract 3D reconstruction and view synthesis are fundamental to emerging applications such as AR/VR, robotic perception, and digital twins. The Visual Geometry Grounded Transformer (VGGT) has demonstrated strong feed‑forward 3D reconstruction capability by directly predicting essential geometric attributes without per‑scene optimization. However, its billion‑parameter scale results in high memory and computation demands, limiting instant on‑device deployment. While quantization provides a natural path to efficiency, methods designed for large language models fail on VGGT due to two key challenges: saturated activation channels that limit low-bit quantization, and diverse 3D semantics that cause unreliable calibration and overfitting. Furthermore, VGGT poses hardware challenges arising from precision‑sensitive nonlinear operators and a global attention mechanism that incurs excessive memory and computation demands due to the long‑sequence problem. To address these challenges, we propose VersaQ‑3D, an algorithm-architecture co‑design framework for efficient VGGT execution. At the algorithm level , we present the first calibration‑free, scene‑agnostic quantization method for VGGT that supports low-bit execution down to 4 bits. Our method leverages orthogonal transforms to decorrelate features, suppress outliers, and preserve structural weight features, enabling robust low‑bit inference without calibration data. At the architecture level , we design a unified, reconfigurable accelerator with a hierarchical multi‑precision compute fabric that supports BF16, INT8, and INT4 execution. Specifically, a unified systolic datapath executes both linear and nonlinear operators on the same hardware with configurable precision modes to balance accuracy and efficiency, which reduces end‑to‑end latency by 60%. Moreover, our two‑stage recomputation‑based tiling further cuts runtime by an additional 7% by alleviating on‑chip memory pressure for long‑sequence multi‑head attention. Evaluations show that VersaQ‑3D preserves 98-99% of full‑precision accuracy at W4A8 and achieves acceptable accuracy at W4A4 that outperforms prior methods by 1.61 × \times -2.39 × \times across diverse scenes. Its co‑designed accelerator further delivers 5.2 × \times -10.8 × \times speedup over edge GPUs with low area and power consumption, enabling instant and energy‑efficient feed‑forward 3D reconstruction. I Introduction 3D reconstruction and view synthesis have long been the fundamental components in various computer vision and graphics tasks. They aim to recover and represent the geometry and appearance of real-world scenes from visual observations, enabling machines and humans alike to see, understand, and interact with 3D environments. Their importance extends across a diverse range of applications from immersive experiences in augmented and virtual reality (AR/VR) [ 60 , 33 ] to precise environmental understanding in robotics and autonomous systems [ 63 , 65 ] , and large-scale digital twins used in simulation and design [ 41 ] , even to the smart city modeling [ 9 ] . As these applications scale to ever-larger and more dynamic environments, there is a growing demand for methods to be not only accurate but also computationally efficient and robust for diverse scenarios. Traditional 3D reconstruction methods, such as Structure‑from‑Motion (SfM) [ 50 ] and Multi‑View Stereo (MVS) [ 18 ] , rely on iterative multi‑stage optimization to recover camera poses and dense geometry. Although accurate, their high computational cost, slow convergence, and sensitivity to noise and occlusion make them impractical for large‑scale and/or real‑time applications [ 59 ] . Learning‑based approaches, including Neural Radiance Fields (NeRF) [ 42 ] and 3D Gaussian Splatting (3DGS) [ 26 ] , improve scene representation and rendering quality but require per‑scene optimization and high computational cost. Moreover, these methods focus primarily on view synthesis rather than explicit geometric reconstruction, limiting their applicability to downstream perception and robotics tasks. In contrast, feed-forward approaches aim to produce explicit 3D outputs, such as camera poses, depth maps, point maps, and 3D correspondences, directly from input images without costly iterative refinement. The Visual Geometry Grounded Transformer (VGGT) [ 55 ] exemplifies this shift toward scalable, end-to-end 3D reconstruction. VGGT is a feed-forward model capable of performing 3D reconstruction from one, a few, or even hundreds of input images within a single forward pass. It predicts a full set of geometric attributes in tens of seconds and often matches or even surpasses the accuracy of traditional methods, without requiring any post‑processing. Despite its remarkable reconstruction capability, the billion‑scale parameterization of VGGT poses substantial challenges to achieving instant 3D reconstruction [ 33 , 44 ] , where high‑fidelity results are expected within only a few seconds of visual input [ 43 , 45 , 3 ] . However, current transformer‑based 3D models fall far short of such latency targets due to their extreme memory and compute demands. VGGT consists of over 1.2B parameters, requiring > 4 GB of weights in FP16 precision and more than 7 TFLOPs of computation per inference for multi‑view reconstruction. These resource requirements drastically exceed the capabilities of typical edge GPUs, such as the NVIDIA Jetson Orin NX [ 12 ] , which offers limited LPDDR5 memory with 102.4 GB/s bandwidth and 3.76 FP16 TFLOPS. Profiling results (Sec. II-D ) reveal that VGGT is constrained primarily by memory capacity and bandwidth, while the quadratic cost of global attention further amplifies latency as the number of input views increases. Consequently, the combination of massive parameter size, bandwidth sensitivity, and quadratic computation cost makes direct on‑edge deployment of VGGT impractical without model compression and hardware support. Fig. 1: Llama and VGGT activation value distributions and corresponding percentile distributions. The red band (25th-75th percentiles) denotes the inter-quartile range containing the middle 50% of activations. Unlike LLMs, which show isolated spiking outliers, VGGT exhibits saturated activation channels where many channels maintain persistently high activation values. Quantization, which reduces model size and computational cost by lowering numerical precision, appears a natural path to improve VGGT’s efficiency. Since VGGT shares the transformer backbone with modern large language models (LLMs), many existing LLM‑oriented quantization techniques appear applicable. These methods generally depend on a calibration dataset to capture representative activation statistics [ 10 , 34 , 57 ] . However, such approaches are ineffective for VGGT due to two primary challenges. First , VGGT exhibits saturated activation channels , where many channels maintain persistently high activation values rather than isolated spiking outliers. This saturation behavior, illustrated in Fig. 1 , contrasts sharply with the isolated spiking outliers observed in LLMs and makes existing outlier‑smoothing techniques [ 2 , 38 ] ineffective at maintaining model accuracy after quantization. Second , the diverse and scene‑dependent semantics of 3D multi‑view data hinder effective calibration: small datasets overfit to specific scenes, while large ones remain costly and fail to cover unseen geometries. These challenges prevent standard post‑training quantization pipelines from maintaining VGGT’s reconstruction accuracy or generalizing well across dynamic 3D environments. In addition, the nonlinear operators in the model exhibit high sensitivity to quantization, often necessitating dedicated hardware units to ensure computational accuracy. This results in non‑linear precision constraints , where low‑bit quantization is insufficient and BF16‑level precision must be preserved. However, such customized units introduce additional hardware resource overhead and may lead to workload imbalance between the linear and nonlinear computation pipelines. Fixed‑precision architectures further restrict the ability to exploit the benefits of different precisions, as they cannot be configured to meet real‑time latency or high‑fidelity requirements.

Read the complete text