Visual Program Distillation with Template-Based Augmentation
Michal Shlapentokh-Rothman, Yu-Xiong Wang, Derek Hoiem · 2024
Read Visual Program Distillation with Template-Based Augmentation by Michal Shlapentokh-Rothman, Yu-Xiong Wang, Derek Hoiem (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08564.txt.
Description
Michal Shlapentokh-Rothman, Yu-Xiong Wang, and Derek Hoiem teach small models to write visual programs in a 2024 paper. Big models can answer image questions by writing code, but cost and labeling needs block wider use. Their distillation method needs no human written program examples. It splits programs into reusable skill templates and their arguments, then makes synthetic training data from that split.
With modest question and answer data, models of at most a billion parameters learned to write good specialized programs. Inference also runs much faster than prompting a giant model. Cheap, quick visual programming could bring code based image reasoning to small devices and niche domains alike.
Catalog record
| CREATOR | Michal Shlapentokh-Rothman, Yu-Xiong Wang, Derek Hoiem |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-08564