Vision-Grounded Machine Interpreting: Improving the Translation Process through Visual Cues

Claudio Fantinuoli · 2025

Read Vision-Grounded Machine Interpreting: Improving the Translation Process through Visual Cues by Claudio Fantinuoli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23957.txt.

Description

A machine interpreter that hears only words misses the scene that would settle what a word means. Claudio Fantinuoli, the sole author, introduces Vision-Grounded Interpreting, which primes speech-to-speech translation with visual context. A prototype feeds webcam images and speech into a vision-language model, so the translation process starts already informed by the situation. A hand-built diagnostic corpus targets three kinds of ambiguity.

Visual grounding sharply improved lexical disambiguation in the evaluation. Gains for gender resolution were modest and less stable. Syntactic ambiguities gained nothing. Fantinuoli argues that multimodality is a necessary next step for machine interpreting quality.

Catalog record

CREATORClaudio Fantinuoli
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-23957

Browse the full library · About the Tower of Babel