One Patch is All You Need: Joint Surface Material Reconstruction and Classification from Minimal Visual Cues
Sindhuja Penchala, Gavin Money, Gabriel Marques, Samuel Wood, Jessica Kirschman · 2025
Read One Patch is All You Need: Joint Surface Material Reconstruction and Classification from Minimal Visual Cues by Sindhuja Penchala, Gavin Money, Gabriel Marques, Samuel Wood, Jessica Kirschman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-20784.txt.
Introduction
Sindhuja Penchala and colleagues ask in 2025 how much of a surface a vision model needs to see. Their model, SMARC, receives a single contiguous patch covering just 10 percent of an image. From that sliver it must do two jobs: reconstruct the full colour surface and classify which material it is. The design pairs a partial-convolution U-Net, built for incomplete images, with a classification head.
On the Touch and Go dataset of real-world surface textures, SMARC beat five rival models, including a Vision Transformer, a Masked Autoencoder, and DETR. It reconstructed surfaces at a PSNR of 17.55 decibels, a standard fidelity measure, while classifying materials with 85.10 percent accuracy. The authors credit partial convolutions, which track which pixels are real, for the spatial reasoning under extreme sparsity. They also report the fastest inference of the group, a practical edge for robots that must judge surfaces from glimpses.
Catalog record
| CREATOR | Sindhuja Penchala, Gavin Money, Gabriel Marques, Samuel Wood, Jessica Kirschman |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-20784
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
One Patch is All You Need: Joint Surface Material Reconstruction and Classification from Minimal Visual Cues Sindhuja Penchala, Gavin Money, Gabriel Marques, Samuel Wood, Jessica Kirschman Abstract Understanding material surfaces from sparse visual cues is critical for applications in robotics, simulation, and material perception. However, most existing methods rely on dense or full-scene observations, limiting their effectiveness in constrained or partial view environment. To address this challenge, we introduce SMARC, a unified model for Surface MAterial Reconstruction and Classification from minimal visual input. By giving only a single 10% contiguous patch of the image, SMARC recognizes and reconstructs the full RGB surface while simultaneously classifying the material category. Our architecture combines a Partial Convolutional U-Net with a classification head, enabling both spatial inpainting and semantic understanding under extreme observation sparsity. We compared SMARC against five models including convolutional autoencoders Zhang 2018 , Vision Transformer (ViT) Verma et al. 2023 , Masked Autoencoder (MAE) He et al. 2022 , Swin Transformer Liu et al. 2021 , and DETR Ban et al. 2022 using Touch and Go dataset Yang et al. 2022 of real-world surface textures. SMARC achieves state-of-the-art results with a PSNR of 17.55 dB and a material classification accuracy of 85.10%. Our findings highlight the advantages of partial convolution in spatial reasoning under missing data and establish a strong foundation for minimal-vision surface understanding. 1 Introduction Surface material reconstruction and categorization are essential for robotic perception and physical contact with the environment Dzedzickis et al. 2021 . These abilities let robots to recognize textures, hardness, and reflectivity,important clues that influence how a robot perceives, manipulates, and responds to real-world surfaces Xie et al. 2019 . Understanding whether a surface is metallic, soft, or granular has a direct impact on applied force, grip stability, and tool trajectory in robotic manipulation activities like grasping, drilling, and polishing Kawaharazuka et al. 2024 . However, visual inputs obtained in unstructured or congested surroundings are frequently insufficient due to occlusion, sensor noise, or limited views. Such limited observations make it difficult to derive precise material properties or reconstruct surface appearance from incomplete data. As a result, there is a growing need for perception frameworks that can reason beyond limited seen pixels and infer missing information in order to accomplish accurate surface understanding and reconstruction in robotic systems Tian et al. 2024 . Early image reconstruction methods relied heavily on convolutional architectures, which excel at modeling local spatial patterns and capturing fine-grained texture details. Classical frameworks such as Convolutional Autoencoders Zhang 2018 and U-Net established the foundation for dense image restoration tasks including denoising, inpainting, and semantic segmentation. These architectures progressively extract hierarchical features, enabling them to model localized geometric cues and edge continuity effectively. However, their dependence on local receptive fields restricts global reasoning, making them less effective when large portions of an image are missing or corrupted Simonyan and Zisserman 2014 . The emergence of transformer-based vision models Dosovitskiy et al. 2020 has significantly expanded the scope of image understanding. Vision Transformers (ViT) Verma et al. 2023 introduced self-attention mechanisms that learn long-range dependencies and global feature interactions, while hierarchical extensions such as the Swin Transformer Liang et al. 2021 enhanced scalability by integrating multi-resolution windowed attention. Similarly, Masked Autoencoders (MAE) He et al. 2022 advanced self-supervised representation learning by reconstructing missing patches from visible regions, leading to strong generalization across downstream tasks. DETR Ban et al. 2022 further applied the transformer paradigm to object recognition and localization, reformulating perception as a set-based prediction problem through end-to-end attention modeling. Collectively, these advances have demonstrated powerful performance in complete-view settings, where sufficient visual context is available for learning structure and semantics jointly. Despite their success, these architectures remain limited in scenarios where the input is highly sparse or incomplete. Convolution based models tend to produce over-smoothed and textureless output because local kernels cannot infer long-range context beyond visible regions Suganuma et al. 2018 . Transformer based models, though effective at global reasoning, rely on dense token embeddings and struggle when visual evidence is insufficient to establish spatial correspondence Dosovitskiy et al. 2020 . The random masking strategy employed by MAE disrupts spatial continuity, resulting in coarse reconstructions that lack fine structural fidelity. Consequently, existing frameworks are not well-suited for perception tasks that require robust reasoning under heavy occlusion such as surface material reconstruction and classification in robotic environments, where accurate inference from minimal visual cues is essential. To overcome these limitations, we developed SMARC, a dual purpose framework that integrates partial convolutions and mask propagation within a U-Net-like architecture. Unlike prior methods, SMARC operates exclusively on valid, continuous (unmasked) pixels and dynamically updates the mask at each layer to maintain spatial consistency. This design enables the network to reconstruct missing regions with high structural fidelity while performing joint surface classification. Through this mask aware learning strategy, SMARC bridges the gap between incomplete perception and reliable material understanding. The key contributions of this work are as follows: • SMARC is proposed as a unified framework for joint surface reconstruction and classification from only 10% visible input. • A mask aware Partial Convolutional U-Net is designed to dynamically update visibility masks, guiding both reconstruction and classification. • The effectiveness of SMARC is demonstrated on the Touch and Go dataset, achieving 17.55% PSNR and 85.10% accuracy, outperforming five state of the art models. • SMARC exhibits superior computational efficiency, achieving the fastest inference speed of 19.1M parameters per second, demonstrating its scalability and real time processing capability in robotics. The rest of paper is structured as follows. Section 2 introduces the proposed SMARC architecture, detailing the partial convolutional U-Net backbone, classification head, and training strategy. Section 3 presents experimental results on the Touch and Go dataset, comparing reconstruction and classification metrics. Finally, Section 4 concludes the paper and discusses potential directions for extending minimal view surface understanding in robotics and vision applications. 2 Methodology We propose SMARC, a dual-purpose architecture built upon a modified U-Net framework with partial convolutions to jointly perform image inpainting and surface classification. The model is designed to restore images with missing or occluded regions while simultaneously identifying the underlying surface type such as concrete, grass, wood, or rock. SMARC follows an encoder–bottleneck–decoder structure enhanced with skip connections, a dedicated mask propagation path, and a multi-scale classification head (shown in Figure 1 ). Within this design, partial convolutions ensure that feature extraction and reconstruction operate exclusively on valid pixels, thereby minimizing artifacts and preserving structural integrity in masked areas. The decoder progressively restores the complete RGB image, while the classification branch fuses multi level semantic features to enable accurate and relevant surface recognition. 2.1 Encoder Architecture The encoder consists of four hierarchically arranged blocks, each designed to extract semantically rich features from partially masked input images. At each level, the encoder receives both a feature map and a corresponding binary mask as input that identify valid (unmasked) regions. This mask is propagated and refined in parallel with the feature map, enabling robust learning from incomplete observations. Each block operates using partial convolutional layers (PConv) and incorporates channel wise attention via Squeeze-and-Excitation (SE) modules. Let the input image be x\in\mathbb{R}^{H\times W\times 3} , and the respective binary mask be m\in\{0,1\}^{H\times W\times 1} . The encoder