Enhancing Visual Forced Alignment with Local Context-Aware Feature Extraction and Multi-Task Learning

Yi He, Lei Yang, Shilin Wang · 2025

Read Enhancing Visual Forced Alignment with Local Context-Aware Feature Extraction and Multi-Task Learning by Yi He, Lei Yang, Shilin Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-03286.txt.

Description

Yi He, Lei Yang, and Shilin Wang teach a computer to lip-read the timing of speech from video alone. The task, called visual forced alignment, lines up each word and sound with the lip movements that made it. No audio track is used. Past tools often drifted on fine movements of the mouth. Their 2025 method reads both the broad scene and the local lip zone at once.

Two tasks are trained as one, so each sharpens the other. A cleaned-up search step then fixes the lining up of sounds in time. On a known test set of TV speech, word match rose by about 6 percent over past tools. Sound match rose by about 27 percent, a far bigger leap. The team points to automatic captions for shows and short clips as a prime use.

Catalog record

CREATORYi He, Lei Yang, Shilin Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-03286

Browse the full library · About the Tower of Babel