OSCAR: Object Status and Contextual Awareness for Recipes to Support Non-Visual Cooking Conference: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CH
Franklin Mingzhe Li, Kaitlyn Ng, Bin Zhu, Patrick Carrington · 2025
Read OSCAR: Object Status and Contextual Awareness for Recipes to Support Non-Visual Cooking Conference: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CH by Franklin Mingzhe Li, Kaitlyn Ng, Bin Zhu, Patrick Carrington (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05962.txt.
Description
Franklin Mingzhe Li and co-authors built this 2025 assistant for blind and low-vision cooks following recipes. Cooking from a recipe is hard without sight: you cannot glance to check whether the onions are chopped or the sauce has thickened. Their system, OSCAR, tracks the status of objects in the recipe, using language and vision models to read steps, watch video frames, and log progress.
Tests on 173 online cooking videos plus twelve real non-visual cooking recordings showed the object-status approach beat baselines by over twenty percent across several vision models. The team also released their annotated non-visual cooking set as a benchmark. The work treats progress tracking, not just step reading, as the core aid.
Catalog record
| CREATOR | Franklin Mingzhe Li, Kaitlyn Ng, Bin Zhu, Patrick Carrington |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-05962