Empirical Evaluation of Multi-Modal Touch Detection in Over-the-Shoulder Video Surveillance
Mohammadreza Rashidi · 2026
Read Empirical Evaluation of Multi-Modal Touch Detection in Over-the-Shoulder Video Surveillance by Mohammadreza Rashidi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29504.txt.
Description
Rashidi tests whether keystrokes on a phone can be reconstructed from over-the-shoulder video alone, without recording the screen. His framework combines four detection routes. Hand landmarks come from MediaPipe, skin colour filtering uses HSV values, frame differencing catches motion, and Canny edge analysis traces outlines. Detected touch points then map onto a reference screen layout.
Evaluation uses a staged 120-frame first-person video of passcode entry. MediaPipe and skin detection fail under partial hand occlusion and ambient noise. Motion-only and edge-only setups reach F1-scores, a balance of precision and recall, of only about eighteen percent. The combined setup scores slightly lower still, and its sequence similarity stays near three percent. On five real third-person videos, the detector fires far too often, up to hundreds of points per frame. Rashidi concludes the staged result does not survive uncontrolled footage.
Catalog record
| CREATOR | Mohammadreza Rashidi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.29504