HEART-ViT: Hessian-guided Efficient Dynamic Attention and Token Pruning in Vision Transformers
Mohammad Helal Uddin, Liam Seymour, Sabur Baidya · 2025
Read HEART-ViT: Hessian-guided Efficient Dynamic Attention and Token Pruning in Vision Transformers by Mohammad Helal Uddin, Liam Seymour, Sabur Baidya (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20120.txt.
Description
Mohammad Helal Uddin, Liam Seymour, and Sabur Baidya prune vision transformers with a calculus-aware scalpel. Transformers pay quadratic attention costs and carry much redundant work. That blocks deployment where latency is tight. Past pruning treats tokens or heads apart. It leans on heuristics or first-order signals. HEART-ViT gauges second-order, curvature-weighted sensitivity for both at once. It uses fast Hessian-vector products to do so.
Pruning calls are made per input, under set loss budgets. Token and head cuts are unified in one frame. A structural lesson emerged. Token cuts give most of the savings, while head cuts polish what is left. On ImageNet tests, the method cut compute by near half and latency by over a third. Accuracy matched or beat the base models after fine-tuning. Gains held on edge hardware too.
Catalog record
| CREATOR | Mohammad Helal Uddin, Liam Seymour, Sabur Baidya |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-20120