LLM-Guided Material Inference for 3D Point Clouds

Nafiseh Izadyar, Teseo Schneider · 2025

Read LLM-Guided Material Inference for 3D Point Clouds by Nafiseh Izadyar, Teseo Schneider (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-03237.txt.

Introduction

A 3D scan shows shape, but not what things are made of. Nafiseh Izadyar and Teseo Schneider tackle that gap in this 2025 work. Material traits decide how objects look, yet most shape data sets ignore them.

The authors use large language models in two stages to infer materials from point clouds. The clouds come with rough segment labels. Knowing material from shape alone could help robots grasp objects safely. It could also improve virtual scenes, where surfaces must react to light in a realistic way. Coarse segments are enough to start, which suits scans made by fast capture tools.

Catalog record

CREATORNafiseh Izadyar, Teseo Schneider
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-03237

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLM-Guided Material Inference for 3D Point Clouds Nafiseh Izadyar, Teseo Schneider Abstract Most existing 3D shape datasets and models focus solely on geometry, overlooking the material properties that determine how objects appear. We introduce a two-stage large language model (LLM) based method for inferring material composition directly from 3D point clouds with coarse segmentations. Our key insight is to decouple reasoning about what an object is from what it is made of. In the first stage, an LLM predicts the object’s semantic; in the second stage, it assigns plausible materials to each geometric segment, conditioned on the inferred semantics. Both stages operate in a zero-shot manner, without task-specific training. Because existing datasets lack reliable material annotations, we evaluate our method using an LLM-as-a-Judge implemented in DeepEval Ip and Vongthongsri 2025 . Across 1,000 shapes from Fusion Willis et al. 2022 /ABS Izadyar et al. 2025 and ShapeNet Chang et al. 2015 , our method achieves high semantic and material plausibility. These results demonstrate that language models can serve as general-purpose priors for bridging geometric reasoning and material understanding in 3D data. 1 Introduction The past decade has witnessed the creation of numerous large-scale 3D shape datasets, such as ShapeNet, ABC, Fusion, ABS, and PointNet Chang et al. 2015 ; Koch et al. 2019 ; Willis et al. 2022 ; Izadyar et al. 2025 ; Qi et al. 2017 . These datasets have driven significant progress in data-driven shape analysis; most notably in 3D reconstruction Huang et al. 2023 , segmentation Fu et al. 2023 ; Kolodiazhnyi et al. 2024 , and shape generation Yang et al. 2024 ; Liu et al. 2025 ; Li et al. 2025 . Despite these advances, a crucial dimension of real 3D shape remains overlooked: material appearance . Most publicly available shape datasets lack annotations on how the object looks or what it is made of, limiting their ability to support applications that depend on material and appearance understanding. For instance, it is fundamental for photorealistic rendering necessary for robotics reinforcement learning, where reflectance and texture determine visual realism. While several recent works have explored visual material recognition from RGB images Fang et al. 2024 , estimating material composition directly from 3D geometry shapes remains an open challenge. The difficulty stems from the ambiguity of 3D data without colour or material information, the diversity of materials, and the limited availability of annotated 3D datasets. This makes it hard to apply traditional supervised methods that rely on dense, per-part material labels. Fortunately, many practical 3D shape representations already include coarse structural segmentation (e.g., CAD models contain shells, meshes contain face groups). This segmenting information often corresponds to material boundaries and can be exploited for material estimation. Figure 1 : A pot-shaped object misclassified without semantic reasoning. Directly inferring materials from geometry assigns incorrect materials (i.e., foam, fabric), left. Our pull pipeline first identifies the object as a pot and then assigns plausible materials such as metal, plastic, and rubber. Figure 2 : Overview of our method: Stage 1 uses an LLM to infer high-level semantic information about the object from multi-view depth and raster renderings; Stage 2 combines the semantic label with the input coarse segmentation to predict a material for each part. In this paper, we propose a language-model-driven approach to infer material composition directly from this coarse 3D representation. Our key insight is that large language models (LLMs) can recognize shapes and can reason about plausible material assignments based on object semantics and part relationships (Fig. 1 ). Given a point cloud segmented into a small number of parts, our method predicts plausible materials for each part. This enables robust, interpretable material inference even in the absence of explicit training labels and supervision. We validate our method (Section 4 ) on over 1,000 models from the Fusion/ABS and ShapeNet datasets, using the LLM as the judge Zheng et al. 2023 . Across diverse object categories and origins, our approach achieves high accuracy in recovering the correct material composition. We also provide qualitative visualizations that show our predictions align closely with human perception of material plausibility. In summary, our contributions are twofold: 1. We introduce the first LLM-based framework for inferring material composition directly from segmented 3D geometry. 2. We demonstrate that our approach achieves high material classification accuracy, establishing a new baseline for material-aware 3D understanding. 2 Related Work 3D Shape Understanding and Representation Learning. Large 3D datasets such as ShapeNet Chang et al. 2015 , ModelNet Wu et al. 2015 , and the ABC dataset Koch et al. 2019 have driven rapid progress in geometry-centric learning for classification, segmentation, and reconstruction. Point-cloud and mesh encoders including PointNet Qi et al. 2017 , DGCNN Wang et al. 2019 , and subsequent transformer-based architectures Guo et al. 2021 have become standard for geometric reasoning tasks. While these approaches learn powerful shape descriptors, they capture only geometric structure omitting appearance, material, and physical context that are crucial for realistic perception and interaction. Recent works such as PointNeXt Qian et al. 2022 and SparseConvNet Graham et al. 2017 further improve geometric fidelity but remain limited to purely spatial features. Material and Appearance Estimation. Estimating material properties has a long history in computer vision. Classical inverse-rendering methods recover reflectance and BRDF parameters from RGB images Rematas et al. 2015 ; Gardner et al. 2017 ; Deschaintre et al. 2018 ; Guo et al. 2020 ; Shen et al. 2025 , while material recognition from photographs uses convolutional or multimodal networks Bell et al. 2015 ; Hu et al. 2020 . For 3D data, however, most methods assume textured meshes or images, not bare geometry. Some works attempt to infer material categories from surface shape cues Schwartz and Nishino 2018 ; Ayaz et al. 2022 , but rely on curated datasets with aligned appearance supervision. Our work addresses a different setup: infer materials directly from untextured, coarsely segmented point clouds. Language-Driven and Multimodal 3D Understanding. Recent advances in vision-language modeling extend material and semantic reasoning beyond pixels. Models such as CLIP Radford et al. 2021 and BLIP-2 Li et al. 2023 enable zero-shot recognition by aligning visual and textual representations. Their 3D counterparts ULIP Xue et al. 2023 , CLIP-Forge Sanghi et al. 2022 , and OpenShape Liu et al. 2023 project 3D features into shared language spaces for zero-shot shape retrieval and captioning. These models reason about category-level semantics but not physical materials. More recently, large language models (LLMs) have been used as general reasoning engines for 3D understanding (e.g., LLM-Grounder Yang et al. 2023 and Chat-3D Wang et al. 2023 ) where text-based reasoning augments 3D perception. Our approach follows this trend, employing an LLM not as a language–vision aligner but as a semantic prior to infer plausible materials from geometric context. Part-Aware 3D Reasoning. Part-level understanding underpins both semantic segmentation and physical reasoning. Large-scale part-annotated datasets such as PartNet Mo et al. 2018 enable hierarchical decomposition of shapes into functional substructures, inspiring recent neural architectures for part segmentation Yu et al. 2021 ; Uy et al. 2019 . Although part-aware learning captures object organization, prior methods treat parts as purely geometric entities. We extend this line by associating parts with material plausibility, bridging shape decomposition and

Read the complete text