Visual Program Distillation with Template-Based Augmentation

Michal Shlapentokh-Rothman, Yu-Xiong Wang, Derek Hoiem · 2024

Read Visual Program Distillation with Template-Based Augmentation by Michal Shlapentokh-Rothman, Yu-Xiong Wang, Derek Hoiem (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08564.txt.

Description

Michal Shlapentokh-Rothman, Yu-Xiong Wang, and Derek Hoiem teach small models to write visual programs in a 2024 paper. Big models can answer image questions by writing code, but cost and labeling needs block wider use. Their distillation method needs no human written program examples. It splits programs into reusable skill templates and their arguments, then makes synthetic training data from that split.

With modest question and answer data, models of at most a billion parameters learned to write good specialized programs. Inference also runs much faster than prompting a giant model. Cheap, quick visual programming could bring code based image reasoning to small devices and niche domains alike.

Catalog record

CREATORMichal Shlapentokh-Rothman, Yu-Xiong Wang, Derek Hoiem
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-08564

Browse the full library · About the Tower of Babel