Vision-Grounded Machine Interpreting: Improving the Translation Process through Visual Cues
Claudio Fantinuoli · 2025
Read Vision-Grounded Machine Interpreting: Improving the Translation Process through Visual Cues by Claudio Fantinuoli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23957.txt.
Description
A machine interpreter that hears only words misses the scene that would settle what a word means. Claudio Fantinuoli, the sole author, introduces Vision-Grounded Interpreting, which primes speech-to-speech translation with visual context. A prototype feeds webcam images and speech into a vision-language model, so the translation process starts already informed by the situation. A hand-built diagnostic corpus targets three kinds of ambiguity.
Visual grounding sharply improved lexical disambiguation in the evaluation. Gains for gender resolution were modest and less stable. Syntactic ambiguities gained nothing. Fantinuoli argues that multimodality is a necessary next step for machine interpreting quality.
Catalog record
| CREATOR | Claudio Fantinuoli |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-23957