Do Image–Text Metrics Respect Semantic Invariances?

Amit AgarwalHitesh Laxmichand PatelMeizhu LiuJyotika Singh Karan DuaHansa MeghwaniMatthew RoweMichael AvendiYassi AbbasiTao ShengSujith RaviDan Roth Oracle AI Correspondence: [email protected] · 2026

Read Do Image–Text Metrics Respect Semantic Invariances? by Amit AgarwalHitesh Laxmichand PatelMeizhu LiuJyotika Singh Karan DuaHansa MeghwaniMatthew RoweMichael AvendiYassi AbbasiTao ShengSujith RaviDan Roth Oracle AI Correspondence: [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24702.txt.

Description

Amit Agarwal and others probe whether image-text metrics respect meaning-preserving changes. Caption scorers grade image-text alignment. The team tests five evaluators under spatial, object, and framing shifts across detection and caption suites. Harmless spatial tweaks and plain rewordings move scores by 6 to 9 percent on average. For matched systems, shifts flip rankings in up to about 37 percent of cases, mainly under spatial changes, and people judge perturbed pairs equally correct. A post-hoc calibration roughly halves median sensitivity while keeping correlation with learned evaluators.

Catalog record

CREATORAmit AgarwalHitesh Laxmichand PatelMeizhu LiuJyotika Singh Karan DuaHansa MeghwaniMatthew RoweMichael AvendiYassi AbbasiTao ShengSujith RaviDan Roth Oracle AI Correspondence: [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE94826
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24702

Browse the full library · About the Tower of Babel