Enhancing Visual Forced Alignment with Local Context-Aware Feature Extraction and Multi-Task Learning
Yi He, Lei Yang, Shilin Wang · 2025
Read Enhancing Visual Forced Alignment with Local Context-Aware Feature Extraction and Multi-Task Learning by Yi He, Lei Yang, Shilin Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-03286.txt.
Description
Yi He, Lei Yang, and Shilin Wang teach a computer to lip-read the timing of speech from video alone. The task, called visual forced alignment, lines up each word and sound with the lip movements that made it. No audio track is used. Past tools often drifted on fine movements of the mouth. Their 2025 method reads both the broad scene and the local lip zone at once.
Two tasks are trained as one, so each sharpens the other. A cleaned-up search step then fixes the lining up of sounds in time. On a known test set of TV speech, word match rose by about 6 percent over past tools. Sound match rose by about 27 percent, a far bigger leap. The team points to automatic captions for shows and short clips as a prime use.
Catalog record
| CREATOR | Yi He, Lei Yang, Shilin Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-03286