Zero-shot Autonomous Microscopy for Scalable and Intelligent Characterization of 2D Materials

Jingyun Yang · 2025

Read Zero-shot Autonomous Microscopy for Scalable and Intelligent Characterization of 2D Materials by Jingyun Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10281.txt.

Introduction

Jingyun Yang and co-authors present ATOMIC, a system that runs a microscope on its own to study atom-thin materials, in a 2025 study. Naming the layers in a new material takes trained experts and slow hand work, and fresh materials trip up even skilled staff.

ATOMIC links a vision foundation model, large language models, grouping tools, and shape analysis to scan samples and read the images with no new training. Prompts, not labeled sets, steer the work. On a standard test material the system marked single layers with 99.7 percent accuracy, on par with experts. Labs gain a path to fast, steady checks of two-dimensional materials as new ones appear.

Catalog record

CREATORJingyun Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-10281

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Zero-shot Autonomous Microscopy for Scalable and Intelligent Characterization of 2D Materials Jingyun Yang Abstract Abstract Characterization of atomic-scale materials traditionally requires human experts with months to years of specialized training. Even for trained human operators, accurate and reliable characterization remains challenging when examining newly discovered materials such as two-dimensional (2D) structures. This bottleneck drives demand for fully autonomous experimentation systems capable of comprehending research objectives without requiring large training datasets. In this work, we present ATOMIC (Autonomous Technology for Optical Microscopy & Intelligent Characterization), an end-to-end framework that integrates foundation models to enable fully autonomous, zero-shot characterization of 2D materials. Our system integrates the vision foundation model ( i.e. , Segment Anything Model), large language models ( i.e. , ChatGPT), unsupervised clustering, and topological analysis to automate microscope control, sample scanning, image segmentation, and intelligent analysis through prompt engineering, eliminating the need for additional training. When analyzing typical MoS 2 samples, our approach achieves 99.7% segmentation accuracy for single layer identification, which is equivalent to that of human experts. In addition, the integrated model is able to detect grain boundary slits that are challenging to identify with human eyes. Furthermore, the system retains robust accuracy despite variable conditions including defocus, color temperature fluctuations, and exposure variations. It is applicable to a broad spectrum of common 2D materials—including graphene, MoS 2 , WSe 2 , SnSe—regardless of whether they were fabricated via top-down or bottom-up methods. This work represents the implementation of foundation models to achieve autonomous analysis, establishing a scalable and data-efficient characterization paradigm that fundamentally transforms the approach to nanoscale materials research. Main Accurate and efficient characterization is critical for the research and applications of materials at the atomic scale. Advanced characterization tools, including microscopy, have made significant contributions to exploring new 2D materials and gaining fundamental understanding li2020general ; ko2023operando ; li2018vapour ; najmaei2013vapour . However, current microscopy methods involve manual inspection and interpretation, creating bottlenecks in research workflows and thus limiting the pace of nanomaterials research. Recent adoption of supervised learning algorithms for materials research, including analyzing microscopic images, has demonstrated high accuracy raccuglia2016machine ; zhao2023learning ; Xie2018 ; zeni2025generative ; granda2018controlling ; Kandel2023 ; schutt2017quantum ; lu2018accelerated ; Zhu2022 ; Masubuchi2019 , but these algorithms have a high demand for a large amount of high-quality training data, which is infeasible in emergent research fields, particularly for newly discovered atomic materials. Foundation Models (FMs) including large language models (LLMs) openai2023gpt4 and vision foundation models (VFMs) have recently emerged as a transformative class of AI models, characterized by large-scale pre-training on diverse datasets and remarkable capabilities in natural language processing, knowledge reasoning, image analysis, and task generalization radford2021learning ; bommasani2021opportunities ; hou2024assessing ; wang2023chatcad ; fu2024drive ; boiko2023emergent . Due to their extensive pre-training on diverse datasets, FMs demonstrate zero-shot (no training) or few-shot (small dataset training) capabilities in various tasks kojima2022large . This feature is particularly valuable for analyzing scientific results, where tasks often involve scarce experimental data and high annotation costs that make traditional supervised learning methods impractical. In this work, we developed an integrated framework, ATOMIC, that integrates FMs to enable fully autonomous optical microscopy (OM) for zero-shot 2D materials characterization. Our system integrates the VFM ( i.e. , Segment Anything Model, SAM) with LLMs ( i.e. , ChatGPT) to create a comprehensive solution that encompasses microscope control, sample scanning, image segmentation, and intelligent analysis. Unlike conventional ML approaches, our method requires only prompts, but no labeled training data, while offering universal analysis across diverse materials with seamless integration into various experimental systems. Large Model Controlled Microscopy ATOMIC processes natural language to generate microscopic analysis, as illustrated in Fig. 1a. The system comprises three synergistic core modules: Physical Control, which autonomously adjusts settings for high-quality image acquisition (Fig. 1b); Spatial Segmentation, which employs zero-shot SAM to generate masks representing potential material domains (Fig. 1c); and LLM-Supervised Clustering, which classifies segmentation masks using RGB channels to distinguish between monolayer, multilayer, and impurity regions (Fig. 1d). ATOMIC demonstrates high accuracy on CVD-grown MoS 2 flake size with a 0.026 Jensen-Shannon divergence against expert annotations (Fig. 1e). The model achieves 99.7% pixel-wise accuracy and 99.9% pixel-wise recall for monolayer MoS 2 (Fig. 1f), confirming its capability for autonomous materials characterization. The Physical Control Module utilizes GPT to autonomously manage microscope operations, dynamically adjusting focus, exposure, and stage movement in real time to optimize imaging conditions. ZEISS Micro Toolbox (MTB) serves through a Flask-based API to GPT, facilitating seamless communication between the microscope and ATOMIC, thereby enabling precise and adaptive control. Within this framework, a GPT-based agent (GPT-4o-mini, July 2024 release) interprets user-defined task instructions, decomposes them into actionable subtasks, and issues corresponding control commands (Fig. S1). This interaction allows for real-time decision-making, ensuring that the microscope adjusts imaging conditions without human intervention. By leveraging MTB’s hardware control capabilities and the GPT agent’s high-level reasoning, the system adaptively optimizes imaging parameters based on real-time feedback. A structured system prompt provides contextual awareness, defining the research domain and the GPT agent’s role in operating the microscope. Additionally, structured task prompts specify scanning paths and region of interest to enable inch-scale sample inspection. For instance, a typical field of view is captured in approximately 6 seconds, allowing a 1\text{ cm}\times 1\text{ cm} substrate to be scanned in 6 minutes under 10× magnification. Spatial Segmentation Following GPT-controlled OM image acquisition, the Spatial Segmentation Module applies SAM for initial region segmentation. Specifically, SAM operates in mask generation mode, configured with thresholds for Intersection over Union (IoU), stability, and mask refinement, producing a set of candidate masks across the entire image without training. With its high precision in detecting material boundaries, SAM effectively distinguishes monolayer MoS 2 from substrate regions, even when edge contrast is exceptionally low. Fig. 2a–d compare conventional Canny edge detection with SAM’s segmentation output. Unlike Canny detection, which struggles with continuous boundaries and primarily captures high-contrast multilayer regions, SAM successfully identifies all relevant flakes, including monolayers with low-contrast edges, and even intricate fractal-like structures challenging to human eyes. Moreover, SAM avoids issues including impurity misclassification, missing multilayer, and missing small flakes, encountered in conventional computer vision methods (see more discussion in Supplementary S3). SAM’s zero-shot performance was evaluated by comparing the model’s segmentation performance against human-labeled ground truth data. SAM generating initial candidate masks, followed by a filtering process to eliminate superfluous masks based on SAM-estimated IoU values. These IoU metrics were estimated by SAM’s internal prediction head, which processes the feature correlation between the generated mask and the input image kirillov2023segment . To validate SAM’s zero-shot mask selection capability, we compared SAM-predicted IoU scores against human-labeled ground truth. Figure S2 demonstrates that SAM achieves 95 % mean accuracy for selected masks, confirming the reliability of SAM’s IoU-based selection mechanism. These results establish that zero-shot SAM effectively identifies 2D materials flakes without requiring domain-specific training. SAM exceeds human performance in specific tasks. Fig. 2e–h demonstrate SAM’s capability to detect subtle grain boundary slits that are imperceptible to human eyes. While human require contrast enhancement and zooming in to discern some of these internal structures, SAM directly detects them from the full original image (Fig. 2g). These slits indicate areas where growth originated from separate nucleation sites that later merged, suggesting

Read the complete text