Focus on What Matters: Enhancing Medical Vision-Language Models with Automatic Attention Alignment Tuning

Aofei Chang, Le Huang, Alex James Boyd, Parminder Bhatia, Taha Kass-Hout, Cao Xiao, Fenglong Ma · 2025

Read Focus on What Matters: Enhancing Medical Vision-Language Models with Automatic Attention Alignment Tuning by Aofei Chang, Le Huang, Alex James Boyd, Parminder Bhatia, Taha Kass-Hout, Cao Xiao, Fenglong Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18503.txt.

Description

Aofei Chang and six co-authors teach medical vision-language models to look at the right part of the image. These models answer questions about scans and draft reports, but their attention often drifts to irrelevant regions, producing confident errors. Fixes applied at run time are limited or need extra labels.

Their 2025 framework, A3Tune, instead retrains the model's focus. It starts with rough region labels from a segmentation model, sharpens them with a biomedical image-text model, and then adjusts only the attention heads that matter most for vision. A mixture-of-experts part adapts the tuning to each prompt and image. On question answering and report writing benchmarks, the tuned models beat leading baselines and showed cleaner attention maps.

Catalog record

CREATORAofei Chang, Le Huang, Alex James Boyd, Parminder Bhatia, Taha Kass-Hout, Cao Xiao, Fenglong Ma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18503

Browse the full library · About the Tower of Babel