Visual Mod ality Prompt for Adapting Vision-Language Object Detectors
Heitor R. Medeiros .Atif BelalSrikanth MuralidharanEric GrangerMarco PedersoliLIVIA, Dept. of Systems Engineering, ETS Montreal, CanadaInternational Laboratory on Learning Systems (ILLS) · 2024
Read Visual Mod ality Prompt for Adapting Vision-Language Object Detectors by Heitor R. Medeiros .Atif BelalSrikanth MuralidharanEric GrangerMarco PedersoliLIVIA, Dept. of Systems Engineering, ETS Montreal, CanadaInternational Laboratory on Learning Systems (ILLS) (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00622.txt.
Description
Medeiros and co-authors teach vision-language object detectors to see in infrared and depth without forgetting how to see in daylight. Such detectors shine at zero-shot finding, yet their skill drops on other sensor types, and full fine-tuning erases the very flexibility worth keeping. The catalog title and creator fields are garbled with fused names and affiliations, so the clean head governs here. Their 2024 method, ModPrompt, adds a learned visual prompt instead of retraining.
An encoder-decoder prompt reshapes each image for the new modality, with a residual design that stays cheap at run time. On two leading detectors and hard infrared and depth data sets, ModPrompt matched full fine-tuning while the zero-shot skill stayed intact. The authors present it as a light adapter: one small add-on per sensor type, rather than a new model each time.
Catalog record
| CREATOR | Heitor R. Medeiros .Atif BelalSrikanth MuralidharanEric GrangerMarco PedersoliLIVIA, Dept. of Systems Engineering, ETS Montreal, CanadaInternational Laboratory on Learning Systems (ILLS) |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-00622