Focus on What Matters: Enhancing Medical Vision-Language Models with Automatic Attention Alignment Tuning
Aofei Chang, Le Huang, Alex James Boyd, Parminder Bhatia, Taha Kass-Hout, Cao Xiao, Fenglong Ma · 2025
Read Focus on What Matters: Enhancing Medical Vision-Language Models with Automatic Attention Alignment Tuning by Aofei Chang, Le Huang, Alex James Boyd, Parminder Bhatia, Taha Kass-Hout, Cao Xiao, Fenglong Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18503.txt.
Description
Aofei Chang and six co-authors teach medical vision-language models to look at the right part of the image. These models answer questions about scans and draft reports, but their attention often drifts to irrelevant regions, producing confident errors. Fixes applied at run time are limited or need extra labels.
Their 2025 framework, A3Tune, instead retrains the model's focus. It starts with rough region labels from a segmentation model, sharpens them with a biomedical image-text model, and then adjusts only the attention heads that matter most for vision. A mixture-of-experts part adapts the tuning to each prompt and image. On question answering and report writing benchmarks, the tuned models beat leading baselines and showed cleaner attention maps.
Catalog record
| CREATOR | Aofei Chang, Le Huang, Alex James Boyd, Parminder Bhatia, Taha Kass-Hout, Cao Xiao, Fenglong Ma |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18503