Empirical Evaluation of Multi-Modal Touch Detection in Over-the-Shoulder Video Surveillance

Mohammadreza Rashidi · 2026

Read Empirical Evaluation of Multi-Modal Touch Detection in Over-the-Shoulder Video Surveillance by Mohammadreza Rashidi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29504.txt.

Description

Rashidi tests whether keystrokes on a phone can be reconstructed from over-the-shoulder video alone, without recording the screen. His framework combines four detection routes. Hand landmarks come from MediaPipe, skin colour filtering uses HSV values, frame differencing catches motion, and Canny edge analysis traces outlines. Detected touch points then map onto a reference screen layout.

Evaluation uses a staged 120-frame first-person video of passcode entry. MediaPipe and skin detection fail under partial hand occlusion and ambient noise. Motion-only and edge-only setups reach F1-scores, a balance of precision and recall, of only about eighteen percent. The combined setup scores slightly lower still, and its sequence similarity stays near three percent. On five real third-person videos, the detector fires far too often, up to hundreds of points per frame. Rashidi concludes the staged result does not survive uncontrolled footage.

Catalog record

CREATORMohammadreza Rashidi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29504

Browse the full library · About the Tower of Babel