Classifier-Guided Captioning Across Modalities
Ariel Shaulov Tel Aviv University arielshaulov @mail.tau.ac.il Tal Shaharabany Tel Aviv University shaharabany @mail.tau.ac.il Eitan Shaar Bar-Ilan University [email protected] Gal Chechik Bar-Ilan University NVIDIA, Israel [email protected] Lior Wolf Tel Aviv University [email protected] · 2025
Read Classifier-Guided Captioning Across Modalities by Ariel Shaulov Tel Aviv University arielshaulov @mail.tau.ac.il Tal Shaharabany Tel Aviv University shaharabany @mail.tau.ac.il Eitan Shaar Bar-Ilan University [email protected] Gal Chechik Bar-Ilan University NVIDIA, Israel [email protected] Lior Wolf Tel Aviv University [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-03183.txt.
Description
Ariel Shaulov and four co-authors present classifier-guided captioning that works across modalities, in work from 2025. Most captioning systems are trained for one setting, such as images labeled by crowd workers, and adapt poorly to audio or video. Their method adapts a frozen captioning system at inference time, with no new training of the base model.
A text classifier, trained on data generated by GPT-4 with tailored prompts, steers captions toward the cues that matter in the new setting. For audio, that means naming sounds and their sources. Combined with an existing zero-shot audio captioning system, the method set a new best result, and tests spanned several models and modalities.
Catalog record
| CREATOR | Ariel Shaulov Tel Aviv University arielshaulov @mail.tau.ac.il Tal Shaharabany Tel Aviv University shaharabany @mail.tau.ac.il Eitan Shaar Bar-Ilan University [email protected] Gal Chechik Bar-Ilan University NVIDIA, Israel [email protected] Lior Wolf Tel Aviv University [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-03183