Transcending Dimensions using Generative AI: Real-Time 3D Model Generation in Augmented Reality

Majid Behravan, Maryam Haghani, Denis Gračanin · 2025

Read Transcending Dimensions using Generative AI: Real-Time 3D Model Generation in Augmented Reality by Majid Behravan, Maryam Haghani, Denis Gračanin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-21033.txt.

Introduction

Majid Behravan, Maryam Haghani, and Denis Gračanin build a tool that makes 3D models inside augmented reality in a 2025 paper. Classic 3D modeling needs skill, special software, and hours of work. Their system lets a user generate, move, and reshape a model live, while wearing an AR view of the room. A generative model called Shap-E turns images into 3D forms.

The hard parts are cutting the object cleanly from its background and keeping interaction smooth. A detection network masks the object before conversion. Thirty-five people tested the system and rated its usability. The overall score sat just under seventy on a hundred-point scale. People who used AR or VR often rated it much higher, near eighty-one. The authors point to games, teaching, and online shopping as natural homes for the tool.

Catalog record

CREATORMajid Behravan, Maryam Haghani, Denis Gračanin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-21033

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Transcending Dimensions using Generative AI: Real-Time 3D Model Generation in Augmented Reality Majid Behravan, Maryam Haghani, Denis Gračanin Abstract Traditional 3D modeling requires technical expertise, specialized software, and time-intensive processes, making it inaccessible for many users. Our research aims to lower these barriers by combining generative AI and augmented reality (AR) into a cohesive system that allows users to easily generate, manipulate, and interact with 3D models in real time, directly within AR environments. Utilizing cutting-edge AI models like Shap-E, we address the complex challenges of transforming 2D images into 3D representations in AR environments. Key challenges such as object isolation, handling intricate backgrounds, and achieving seamless user interaction are tackled through advanced object detection methods, such as Mask R-CNN. Evaluation results from 35 participants reveal an overall System Usability Scale (SUS) score of 69.64, with participants who engaged with AR/VR technologies more frequently rating the system significantly higher, at 80.71. This research is particularly relevant for applications in gaming, education, and AR-based e-commerce, offering intuitive, model creation for users without specialized skills. 1 Introduction The accelerated development of generative AI has significantly impacted many areas, leading to notable advancements natural language processing, image generation, and more. One of the most exciting developments is the ability of generative AI to create three-dimensional (3D) objects. Generative AI models have proven to be incredibly versatile, enabling the generation of 3D models from descriptive text and visual inputs. This advancement is particularly crucial for AR, where 3D objects play a significant role in creating immersive experiences. By leveraging generative AI, we can develop AR applications more efficiently, reducing both time and cost. However, manually creating 3D content is a time-consuming process that requires a high level of expertise [ 9 , 6 ] . The power of generative AI models lies in their ability to overcome the limitations of pre-designed objects. Users can now create an unlimited number of 3D objects, tapping into their creativity to design innovative spaces and experiences. This capability democratizes 3D model creation, making it accessible to a broader audience and fostering a new wave of creativity and innovation [ 5 , 12 ] . The integration of generative AI with AR has the potential to revolutionize various industries, from gaming and entertainment to education and retail. By enabling the creation of realistic and customizable 3D objects, we can enhance user experiences and open up new possibilities for interactive and immersive applications [ 18 ] . Our research focuses on the technical aspects of this integration, exploring the algorithms and methodologies required to achieve high-quality 3D model generation from diverse sources. We delve into the challenges and solutions associated with converting images into accurate and visually appealing 3D models in AR. The implications of our work extend beyond the realm of AR, as the techniques developed can be applied to virtual reality (VR) and other immersive technologies [ 20 , 21 ] . The ability to generate 3D models on-the-fly enhances the flexibility and adaptability of these technologies, making them more responsive to user needs and preferences. In this research, our contribution is the enhanced capability of generative AI models to convert images into 3D models within AR environments. We present an approach that allows for the seamless transformation of real-world objects and items from online stores web pages into the AR space. This approach bridges the gap between the physical and digital worlds, providing users with a powerful tool to create and interact with 3D objects in real-time. With tools like these, designers and developers can achieve new heights of productivity and creativity, fundamentally transforming how we interact with digital content. Ultimately, our goal is to push the boundaries of what is possible with generative AI and AR, creating a seamless and intuitive user experience. By harnessing the power of these technologies, we aim to transform the way people interact with digital content, paving the way for a future where the physical and digital worlds coexist harmoniously (Figure 1 ). Figure 1 : From a 2D image to an AR environment. Left half: From a 2D image on a computer screen to a 3D object in an AR environment. Right half: From a 2D image of a real world object to a 3D object in an AR environment. 2 Related Works The landscape of image-to-3D generation has rapidly evolved, driven by advancements in generative AI technologies. Researchers have developed a range of innovative methods to tackle the challenges of reconstructing 3D models from two-dimensional images, particularly focusing on single-view 3D reconstruction. We provide an overview of significant contributions in this field. 2.1 Image to 3D Generation The field of image-to-3D generation using generative AI has seen substantial progress, with numerous methodologies contributing to its rapid development. We describe the key works and models that have advanced this domain, particularly in the context of single-view 3D reconstruction. Early advancements in 3D generative modeling largely focused on the development and application of Generative Adversarial Networks (GANs) [ 11 ] . This approach was pivotal in shaping the early landscape of 3D generation, enabling the creation of models from representations like point clouds [ 1 ] and implicit representations [ 7 ] . Building on this foundation, recent studies have integrated GANs with differentiable rendering methods, which utilize multiple rendered views as the basis for the loss signal, significantly enhancing the quality of generated 3D models [ 6 ] . Building on these advancements, Behravan et al. proposed leveraging vision-language models (VLMs) for real-time 3D object generation in AR, emphasizing contextual understanding for enhanced object recommendation [ 2 , 3 ] . Similarly, Dasgupta et al. introduced a framework for real-time object recognition using edge computing, addressing hardware limitations in mixed reality [ 8 ] . These approaches advance AI-driven decision-making and object interaction in AR and MR. The advent of diffusion models marked a significant leap forward in high-quality image generation, with growing interest in extending these models to the 3D domain. Typically, this involves training a Vector-Quantized Variational Autoencoder (VQ-VAE) on 3D representations such as triplane [ 25 ] and point cloud [ 31 ] , followed by applying the diffusion model in the latent space. Although direct training on 3D representations is less common, it is gaining traction, particularly with point clouds [ 17 ] , voxels [ 32 ] , and neural wavelet coefficients [ 16 ] . Among the innovative methods, ShapeClipper stands out by reconstructing 3D object shapes from single-view RGB images using CLIP-based shape consistency and geometric constraints. This approach leverages the semantic consistency of CLIP embeddings to enhance 3D shape learning, showing superior performance on datasets like Pix3D, Pascal3D+, and OpenImages [ 15 ] . Similarly, Shap-E employs autoencoders trained on explicit 3D representations combined with a diffusion model in the latent space, offering a flexible approach to representing complex 3D structures. This method addresses the limitations of fixed-resolution explicit outputs, providing a scalable solution for 3D model generation [ 1 ] . Voxel-based methods have also seen significant advancements. These approaches benefit from the regularity of voxel representation and the application of 2D convolutional neural networks (CNNs). However, they often face a trade-off between resolution and computational efficiency. For example, voxel-grid methods

Read the complete text