The context of new media art for photography generation based on diffusion models Authors: Chenye Zhang Published: 2026 ABSTRACT To further improve the controllability and artistic expressiveness of photography generation in new media art creation, this study proposes a multi-level conditional control generation framework based on diffusion models. The framework constructs three-level conditional fusion methods of semantics, layout and style. It integrates Contrastive Language-Image Pre-training (CLIP) text guidance, spatial structure feature mapping and style loss optimization into the Latent Diffusion Model (LDM). Meanwhile, this study introduces self-attention mechanism and consistency loss to enhance the spatial coherence of generated images. Experimental results show that the Fréchet Inception Distance (FID) value of the patterns generated by the model is 15.3, the Inception Score (IS) value is 28.7, and the CLIP Score is 0.312, which are better than those of baseline models such as Stable Diffusion 1.5. The results of ablation experiments show that the layout control and style control modules play a key role in the generation quality of the model. After removing the layout control, the FID value of the model increases to 17.2. After removing the style control, the visual realism of the generated patterns decreases. The above results indicate that the proposed framework can generate photography works with both high visual quality and high artistic matching degree. This provides a new technical path for new media art creation, and also promotes the development of generation models in semantic understanding and narrative expression. Introduction The rise of new media art is profoundly influencing traditional visual art creation modes, and artificial intelligence (AI)-generated content technologies represented by diffusion models have provided new possibilities for the digital creation of photographic art1,2. However, due to real-world environments, equipment costs, and the randomness of real-time shooting, traditional “text-image” generation modes have problems such as poor controllability, difficulty in matching artistic intentions, rough image details, and unreasonable structures3–5. Therefore, how to establish an image model that can accurately understand and complete complex instructions while producing high-quality and high-aesthetic effects has become a major research focus in realizing innovation in AI and new media art. At present, researchers in China and other countries have conducted some preliminary explorations on the above issues. In terms of technical theories, denoising diffusion probabilistic models and Latent Diffusion Model (LDM) are important foundations for image generation, while models such as Contrastive Language-Image Pre-training (CLIP) have provided possibilities for cross-modal state regulation6,7. In terms of applications, open-source models like Stable Diffusion have shown strong scalability, and models such as DALL·E2 have achieved certain progress in commercial applications8–10. However, existing studies mainly focus on image generation for general scenarios, so they have not well met the requirements of photography as a special art form (e.g., composition, light and shadow, depth of field, style). In addition, current modeling methods have not effectively solved problems such as precise control of spatial layout, stable reproduction of specific styles, and consistency of visual elements in complex scenes during their construction. To address the above issues, this study attempts to propose a diffusion model framework for new media art photography creation. The specific research contents are as follows: (1) Construct a multi-level conditional input method based on semantic understanding, layout control and style matching, which is used for precise guidance in the generation process. (2) Propose an optimization method based on spatial consistency to enhance the consistency of high-resolution images in terms of structure and details. (3) Carry out relevant experimental design and performance evaluation on the proposed method, and compare it with existing mainstream models. This study will theoretically extend the application methods of controllable generation models in artistic creation, and provide artists with an efficient and intuitive creation method, which plays a positive role in promoting the further integration of AI and cultural industries. Literature review Basic theory of diffusion model In terms of the basic theory of diffusion models, Khader et al. (2023) explored the application of denoising diffusion probabilistic models in medical image generation. By designing diffusion and denoising processes suitable for voxel data, they proved that the model could generate high-quality synthetic medical images11. Hagemann et al. (2025) proposed a multi-level diffusion framework, which handled data generation problems in function spaces by introducing infinite-dimensional score-based diffusion models. This method had shown certain advantages in complex image generation tasks12. Ho et al. (2022) proposed a cascaded diffusion model for high-fidelity image generation. By concatenating multiple diffusion models to gradually improve resolution, the image generation quality was enhanced13. Kidder (2024) proposed a method for generating synthetic cancer images using diffusion models in the field of cancer research, providing a new approach for medical research and education14. Application of diffusion-based model in cross-modal generation task In terms of cross-modal generation tasks based on diffusion models, Li et al. (2024) proposed the SwinV2-Imagen model, which combined hierarchical vision Transformers with diffusion models. It handled text-to-image generation tasks through an optimized framework, and the results showed that this method could generate visual content highly consistent with text descriptions15. Wang et al. (2025) extended their research to the field of video generation, adopting a cascaded latent space diffusion model approach to improve the quality and temporal coherence of generated videos16. Zhang et al. (2024) applied diffusion models to human motion generation tasks and proposed the Motion Diffuse model, which processed text-driven human motion sequence generation through diffusion processes. This framework had certain effectiveness in modeling complex temporal data17. In addition, Yang et al. (2023) comprehensively summarized the methods and applications of diffusion models, systematically outlined the theoretical foundations and main innovations of different branch methods, and prospected future research directions, providing background support for subsequent research in this field18. Research status of the generation of artistic photography In recent years, scholars have begun to pay attention to the application and influence of AI in photographic art creation. Meyer (2025) discussed how to redefine the concept of “photography” by generative AI, and pointed out that the image generated by AI was not a photo in the traditional sense, but a visual product based on data synthesis, which brought new thinking to the essence of photography and the identity of the author19. Hanna (2023) studied the Midjourney platform, analyzed the actual use of AI art generator in artistic creation and advertising design, and found that such tools could quickly generate visually attractive images, but there were still shortcomings in composition logic and detail authenticity20. Hausken (2024) compared the difference between photographic realism and AI-generated images from the perspective of visual false information, and thought that the photo-level images generated by current AI models lacked the optical and physical characteristics and immersion of real photography, although they were highly realistic21. To sum up, the existing research mostly discusses the problems of AI photography from the theoretical and sociological perspectives, but there are still few related engineering studies on how to improve the controllability of the generated images in composition, light and shadow and style from the technical level. This further shows that the multi-level conditional control method proposed here has practical significance. Existing studies mainly focus on fields such as medicine and video, there is little work devoted to the generation of artistic photography and lack control methods targeting the unique composition, light and shadow, and stylistic expression of photographic art. This makes it difficult to meet the high demand for aesthetic control in new media art creation. To address this, this study proposes a multi-level conditional control diffusion model for photographic works, providing a new technical path for new media art creation. Research model The Construction of diffusion model for the generation of photographic works Diffusion model The working principle of diffusion model can be divided into two processes: forward diffusion and reverse denoising. In the forward process, the original image will be added with random noise in turn, and after many iterations, it will become a pure noise image. This process can be expressed by the following conditional probability cauculation:1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:Q\left({x}_{t}\right|{x}_{t-1})=N({x}_{t};\sqrt{1-{\beta\:}_{t}}{x}_{t-1},{\beta\:}_{t}R)$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{x}_{t}$$\end{document} is the image with noise in step t. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{x}_{t-1}$$\end{document} is the image of the previous step, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\beta\:}_{t}$$\end{document} is a decimal in the interval (0,1) to control the intensity of noise added in each step. R is identity matrix, which means that the noise is independent and identically distributed Gaussian noise. The function of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\beta\:}_{t}$$\end{document} is to adjust the noise added in each step. The reverse process is to let the model learn how to restore the image from pure noise step by step, that is, to predict the noise that should be removed at each step, and its equation is:2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{P}_{\theta\:}\left({x}_{t-1}\right|{x}_{t})=N({x}_{t-1};{\mu\:}_{\theta\:}({x}_{t},t),{{\Sigma\:}}_{\theta\:}({x}_{t},t))$$\end{document} In Eq. (2), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\theta\:$$\end{document} represents the model parameter. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mu\:}_{\theta\:}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{{\Sigma\:}}_{\theta\:}$$\end{document} are the mean and variance of the model prediction, respectively. The training goal of diffusion model is to make the predicted noise as close as possible to the real noise. Model process design Currently, generating images that meet photographic aesthetic standards in the context of new media art has become a major demand among a wide range of photography enthusiasts22,23. To address this, this study proposes a photographic image generation framework based on diffusion models. The core principle of the framework is to simulate the real-world creative process of photographers, “conception, composition, shooting, and post-processing”. Its purpose is to accurately integrate artists’ subjective ideas into the generation process through various conditional signals, enabling comprehensive control over the content, composition, and style of the generated works. The basic architecture of the entire model adopts the LDM, with optimizations made to its conditional control capability and visual consistency. The main operation steps of the whole image generation are shown in Fig. 1. Fig. 1Diffusion model generation process for photographic works generation. Diffusion model generation process for photographic works generation. From Fig. 1, the proposed model mainly consists of three modules: a conditional encoder, a denoising network, and an image decoder. The conditional encoder is used to process various forms of input conditions. The denoising network receives the conditional information integrated by the conditional encoder. The image decoder is used to up sample the latent representation and reconstruct it into a high-resolution Red Green Blue (RGB) image space. Multilevel conditional control Based on the framework construction in the previous section, how to enable the model to generate photographic works that align with the predefined artistic concept is a problem to be solved. For the generated model, if only a text description is given, the model needs to randomly select one of a large number of possible results, which usually makes the generated result quite different from the user’s expected result. To reduce this uncertainty, more constraints can be added in the generation process, such as the position of the object and the texture style of the picture. That is, on the basis of the conditional probability Eq. (1) of the diffusion model, layout and style are further added to become Q (image | text, layout, style). Every time there is one more condition, the space that the model needs to guess will become smaller, and the generated results will be easier to control24,25. Based on the above ideas, this study puts forward three levels of conditional control methods: semantics, layout and style. Its purpose is to guide the image generation process from three perspectives: semantics, layout, and style. The core principle is to uniformly map control signals of different modalities to the conditional latent space and introduce them into the denoising network through distinct approaches26. In terms of semantic control, a pre-trained CLIP text encoder is mainly used to process natural language descriptions and convert them into a high-dimensional semantic vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{c}_{text}$$\end{document}. This vector serves as a global condition, guiding the overall content and style of the image at each step of the denoising process27. Within the U-Net, this condition functions primarily through a cross-attention mechanism. Therefore, the intermediate feature map z of the decoding block in the U-Net is taken as Query(q), and the text embedding \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{c}_{text}$$\end{document} is taken as Key(k) and Value(v) to calculate the attention weight. This process can be expressed as:3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathrm{A}\mathrm{t}\mathrm{t}(q,k,v)=soft[\left(q{k}^{T}\right)/\sqrt{d}]v$$\end{document} d denotes the projected dimension. This method enables the model to better generate each local region of the image based on the global semantic information of the text, thereby ensuring consistency between the generated content and the description. In terms of layout control, the goal of this study is to allow users to control the positions and approximate shapes of main objects in the image. Therefore, the model will enable users to input simplified layout diagrams, including bounding boxes or segmentation maps m. The layout diagram will be processed through a lightweight convolutional encoder \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{E}_{layout}$$\end{document}, generating a spatial structure feature map \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{f}_{layout}={E}_{layout}\left(m\right)$$\end{document}. This feature map is concatenated with the middle-layer features of the U-Net encoder channel by channel, enabling the model to control the generation process more accurately in space28,29. The combination process of the two can be expressed as follows:4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{z}_{enc}^{{\prime\:}}=Concat({z}_{enc},{f}_{layout})$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{z}_{enc}$$\end{document} refers to the feature map of a certain layer in the U-Net encoder. This spatial alignment enables the model to place objects strictly according to the layout provided by users, improving the controllability of the generated works in terms of composition. Correspondingly, the training objective of the conditional diffusion model is transformed from unconditionally predicting noise \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\epsilon\:$$\end{document} to predicting noise under given multiple conditions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:c=\{{c}_{text},{f}_{layout}\}$$\end{document}, and its loss function L can be expressed as:5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:L={E}_{{z}_{0},c,\epsilon\:\sim\:N\left(\mathrm{0,1}\right),t}\left[\parallel\:\epsilon\:-{\epsilon\:}_{\theta\:}({z}_{t},t,c){\parallel\:}_{2}^{2}\right]$$\end{document} In the above equation, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{z}_{0}$$\end{document} is a clean latent code, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{z}_{t}$$\end{document} refers to its noisy version at time step t, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\epsilon\:}_{\theta\:}$$\end{document} is a denoised U-Net with parameter θ. In terms of style control, this study adopts a reference image-based style adaptation method30. Given a target style image \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{I}_{style}$$\end{document}, a pre-trained Visual Geometry Group Network (VGGNet) is used to extract feature statistics from multiple of its convolutional layers, namely Gram matrices \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{G}^{l}$$\end{document}, which are employed to represent the texture style of the image31,32. For the l-th layer of VGGNet, the calculation equation of its Gram matrix is as follows:6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{G}^{l}\left(I\right)=\frac{1}{{C}_{l}{H}_{l}{W}_{l}}{\phi\:}^{l}\left(I\right)\cdot\:{\phi\:}^{l}(I{)}^{T}$$\end{document} In Eq. (6), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\phi\:}^{l}\left(I\right)$$\end{document}= denotes the feature map of image I at the l-th layer, with a dimension of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{C}_{l}\times\:{H}_{l}\times\:{W}_{l}$$\end{document}. During the generation process, a style loss \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{style}$$\end{document} is added to the loss function to make subtle adjustments to the generation process, ensuring that the generated image \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{I}_{gen}$$\end{document} is stylistically close to the reference image. The style loss is calculated as the weighted sum of the differences between Gram matrices of each layer:7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{style}=\sum\:_{l}\:{\lambda\:}_{l}\parallel\:{G}^{l}\left({I}_{gen}\right)-{G}^{l}\left({I}_{style}\right){\parallel\:}_{F}^{2}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\lambda\:}_{l}$$\end{document} denotes the weight coefficient of the l-th layer, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\parallel\:$$\end{document}represents the Frobenius norm. Meanwhile, to avoid excessive damage to the content semantics of the generated image, a content loss \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{content}$$\end{document} needs to be added to the model. This loss is generally calculated using the Mean Squared Error (MSE) of the feature maps themselves:8\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{content}=\parallel\:{\phi\:}^{l}\left({I}_{gen}\right)-{\phi\:}^{l}\left({I}_{content}\right){\parallel\:}_{2}^{2}$$\end{document} In Eq. (8), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{I}_{content}$$\end{document} refers to an image generated primarily by semantics and layout conditions. After the above operations, the overall objective function of the style control link is the combination of the above losses:9\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{total}=L+\alpha\:{L}_{style}+\beta\:{L}_{content}$$\end{document} In the above equation, α and β both represent hyperparameters, which are used to balance the importance of different loss terms. On the basis of the above multi-level condition control, this study uses the dynamic weight adjustment method to improve the coordination between different conditions in the generation process. At different time steps in the denoising process, the model will automatically adjust the weight coefficients of semantic, layout and style conditions. Assuming that the current time step is t, it is first encoded as a position embedding vector\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\:{e}_{t}$$\end{document}. Then, the weight coefficients of three conditions are calculated by using the lightweight fully connected network:10\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{[{\omega\:}_{sem},{\omega\:}_{lay},{\omega\:}_{sty}]}^{T}=softmax(\varphi\:{e}_{t}+b)$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\varphi\:$$\end{document} and b are learnable network parameters. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\omega\:}_{sem}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\omega\:}_{lay}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\omega\:}_{sty}$$\end{document} respectively represent the weight coefficients of semantic, layout and style conditions in time step t, and the softmax function ensures that the sum of the three is 1. In the training process, these weights are jointly optimized with other parameters of the model. In the early stage ( t is large), the layout weight \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\omega\:}_{lay}$$\end{document} is high, which ensures the reasonable image structure. In the later stage ( t is small), the style weight \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\omega\:}_{sty}$$\end{document} gradually increases to strengthen the texture details. Optimization of generation based on spatial consistency In high-resolution image generation tasks, it is necessary to ensure that the image has certain structural rationality and overall coherence at the spatial level, as this directly affects the visual realism and artistic appeal of the presented works33. Therefore, the model requires the assistance of diffusion models34. However, when generating complex images, diffusion models generally suffer from problems such as object distortion, inconsistent details, or scattered overall composition. In addition, diffusion models may lack coordination between local regions when processing large-sized images. To mitigate such issues, this study integrates a spatial consistency-based optimization strategy into the LDM. The goal is to optimize the model architecture and loss function to enhance the internal consistency and detail stability of the generated images. The spatial consistency-based optimization method mainly includes two aspects. One is using an enhanced self-attention mechanism to capture long-range dependencies in the image, and the other is adopting an additional consistency loss function for regularization. In the mechanism of self-attention, the standard self-attention is optimized: it adds relative position bias to the calculation of attention weight, that is, it adds a learnable parameter related to the position distance between query and key to the original scaled dot product result, so that the model can understand the relative spatial relationship between different image blocks more clearly. Let the intermediate feature map in the U-Net still be \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:F\in\:{R}^{H\times\:W\times\:C}$$\end{document}, where H and W represent the height and width of the feature map respectively, and C is the number of channels. In specific operations, it is first reshaped into a matrix form \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:X\in\:{R}^{N\times\:C}$$\end{document}, where N = H×W is the total number of spatial positions. Then, the projections of query q, key k, and value v are calculated through learnable weight matrices:11\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:q=X{W}_{q},K=X{W}_{k},V=X{W}_{v}$$\end{document} In the above equation, W denotes the projection matrix. The calculation of attention weights is realized through scaled dot-product, and the output features are obtained through weighted sum, with the calculation method consistent with Eq. (1). It should be noted that although the self-attention mechanism proposed in this section has the same calculation method as that in the previous section, their functions are different. The attention mechanism proposed in the previous section aims to align the external conditions of the image (text description) with the image being generated, while the self-attention mechanism proposed in this section is used to coordinate the internal conditions of the generated image to prevent the image from generating distorted, unreasonable or contradictory objects. In addition to optimizing the model architecture, other constraints are introduced through the loss function to enhance consistency. During the training process, a consistency loss \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{consist}$$\end{document} based on feature similarity is defined to maintain the stability of the internal structure of the generated image at the spatial level. First, a pre-trained VGG network is used as the feature extractor \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\phi\:$$\end{document} to extract high-level features of the generated image \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{I}_{gen}$$\end{document} and its randomly augmented version \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{I}_{gen}^{{\prime\:}}$$\end{document}, and the difference between the two is calculated as follows:12\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:L_{{consist}} = \left\| {\phi \:\left( {I_{{gen}} } \right) - \phi \:\left( {I_{{gen}}^{{\prime \:}} } \right)} \right\|_{2}^{2}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left\| \cdot \right\|_{2}^{2}$$\end{document} denotes L2 norm, which is used to encourage the model to maintain consistency in the feature space. In the overall training objective, the loss is combined with the standard diffusion loss \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{diff}$$\end{document} to form a total loss function:13\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{total}={L}_{diff}+\lambda\:{L}_{consist}$$\end{document} λ is a hyperparameter, which is used to control the strength of consistency optimization. Experimental design and performance evaluation Datasets collection To meet the high requirements of new media art for visual aesthetics and composition, this study selects the Large-scale Artificial Intelligence Open Network-Aesthetics (LAION-Aesthetics) dataset as the main data source for experiments. LAION is a large-scale image-text pair dataset constructed after filtering by the CLIP model, and LAION-Aesthetics is a subset of it. Each image in this subset is assigned an aesthetic score predicted by a neural network model. To ensure the images align with the research’s target of “photographic works”, only those with a predicted aesthetic score higher than 6.5 are retained. In addition, this dataset contains a large number of images covering diverse scenes, objects, and styles, each paired with a corresponding text description. These descriptions provide semantic condition support for the text-to-image generation task (https://laion.ai/blog/laion-aesthetics). In the data preprocessing stage, the following operations are performed: (1) Resize the longest side of all images to 512 pixels while maintaining the aspect ratio, and use center cropping to obtain 512 × 512 pixel square images. This step aims to unify the input size and reduce the computational load of the model. (2) Standardize the text descriptions, including converting to lowercase, removing special characters, and truncating excessively long sequences, to ensure effective processing by the text encoder. After data collection and cleaning, the entire dataset is split into a training set, validation set, and test set in an 8:1:1 ratio. The training set is used for model parameter learning, the validation set for monitoring model performance during training and hyperparameter tuning, and the test set for evaluating the model’s generalization ability and generation effect. The specific division ratio is shown in Table 1. Table 1Division of image dataset of “photographic works”.Dataset partitionNumber of images (sheets)Proportion of total datasetTraining set400080%Verification set50010%Test set50010%amount to5000100% Table 1Division of image dataset of “photographic works”.Dataset partitionNumber of images (sheets)Proportion of total datasetTraining set400080%Verification set50010%Test set50010%amount to5000100% Division of image dataset of “photographic works”. Experimental environment and parameters setting This section details the hardware and software environments, baseline models, evaluation metrics, and key hyperparameter settings adopted in the experiments. The experiments are mainly conducted on a high-performance computing server equipped with 4 NVIDIA A100 graphics cards (each with 40GB of video memory). The specific environment configuration is shown in Table 2. Table 2Experimental environment (including hardware and software).CategoryComponents and specific configurationHardwareProcessor: 2x Intel Xeon Gold 6230RGraphics card: 4x NVIDIA A100 (40GB)Memory: 512GB DDR4SoftwareOperating system: Ubuntu 20.04 LTSDeep learning framework: PyTorch 1.13.1CUDA version: 11.7Python version: 3.9.18Key libraryDiffusers 0.21.4, Transformers 4.26.1 Table 2Experimental environment (including hardware and software).CategoryComponents and specific configurationHardwareProcessor: 2x Intel Xeon Gold 6230RGraphics card: 4x NVIDIA A100 (40GB)Memory: 512GB DDR4SoftwareOperating system: Ubuntu 20.04 LTSDeep learning framework: PyTorch 1.13.1CUDA version: 11.7Python version: 3.9.18Key libraryDiffusers 0.21.4, Transformers 4.26.1 Experimental environment (including hardware and software). In the whole training process of diffusion model, lightweight fine-tuning is used. Firstly, the pre-training weights of Stable Diffusion 1.5 are loaded, most of the underlying parameters are frozen, and only the cross-attention layer and the newly added conditional coding module in U-Net are updated. This can reduce the training parameters and accelerate the convergence speed, and it is not easy to appear over-fitting on small-scale datasets. In the process of model training, the key hyperparameters are set, as shown in Table 3. Table 3Hyperparameter setting of model.HyperparameterSetting valueOptimizerAdamInitial learning rate5*10− 5Learning rate schedulerLinear warm-up and attenuationWarm-up steps1000Batch size32Number of training cycles50Latent space dimension4 × 64 × 64Diffusion time step1000Gradient clipping threshold1.0Weight attenuation coefficient0.01 Table 3Hyperparameter setting of model.HyperparameterSetting valueOptimizerAdamInitial learning rate5*10− 5Learning rate schedulerLinear warm-up and attenuationWarm-up steps1000Batch size32Number of training cycles50Latent space dimension4 × 64 × 64Diffusion time step1000Gradient clipping threshold1.0Weight attenuation coefficient0.01 Hyperparameter setting of model. In baseline models, five mainstream models from the past three years are selected: ControlNet, Stable Diffusion 1.5, Stable Diffusion eXtended Large‌ (SDXL), DALL·E2, and Guided Language to Image Diffusion for Generation and Editing (GLIDE). These models are chosen because they have shown excellent performance in public benchmark tests, and their code and pre-trained models are publicly available, which facilitates subsequent comparative experiments. In addition, because the selected ControlNet model does not support the input of layout pictures and other baseline models do not support the input of layout pictures and style reference pictures, it will mainly verify the effects of different models at the text input level in the subsequent comparative experiments. In evaluation metrics, a combination of objective quantitative metrics and subjective human evaluation is adopted. The objective metrics include Fréchet Inception Distance (FID) (used to measure the distribution difference between generated images and real images in the feature space; a lower value indicates higher generation quality), Inception Score (IS) (used to evaluate the clarity and diversity of generated images), and CLIP Score (calculates the semantic matching degree between generated images and input text descriptions via the CLIP model; a higher score indicates better image-text relevance). In addition, subjective scores from professionals on the generated images in terms of authenticity, artistry, and consistency with text are collected. Performance evaluation Qualitative result analysis The model’s performance is evaluated from two aspects: visual perception and artistic expression. First, the generation results of the proposed model and the selected baseline models under the same text prompts are compared. Then, an in-depth analysis is conducted on the proposed model’s performance in complex semantic understanding and artistic style adaptation. To this end, five professionals with more than three years of experience in photography or digital media art are invited to form an evaluation panel. The panel scores the images generated by the models on different dimensions (on a 1–5 scale). Figures 2 and 3 show the average expert scores of the proposed model and baseline models (ControlNet, Stable Diffusion 1.5, SDXL and DALL·E2) for images generated under two different text prompts. Text Prompt 1 is “A robin with dewdrops on its feathers, perched on a blooming cherry branch with a blurred background”. Text Prompt 2 is “The interior of a Gothic cathedral, where sunlight casts colorful rays through stained glass windows”. Fig. 2Expert score comparison of images generated by different models under text prompt 1 (average score). Expert score comparison of images generated by different models under text prompt 1 (average score). Fig. 3Expert score comparison of images generated by different models under text prompt 2 (average score). Expert score comparison of images generated by different models under text prompt 2 (average score). Figures 2 and 3 indicate that the proposed model outperforms the baseline models under both text prompts. Under Text Prompt 1, the proposed model achieves an average score of 4.6 for detail quality, 4.8 for color coordination, and 4.5 for composition rationality, all higher than those of the other baseline models. Under Text Prompt 2, the proposed model still performs better than the other models. Among these metrics, the score for composition rationality (4.6) is the most prominent, demonstrating the model’s superior capability in handling complex spatial structures. Under Text Prompt 1, a comparison of the generation results is made among this model, Stable Diffusion 1.5, and DALL·E2. Compared with the Stable Diffusion 1.5 and DALL·E 2 models, the proposed model demonstrates better overall performance in terms of detail quality, color harmony, and composition. The image generated by Stable Diffusion 1.5 appears somewhat monotonous in overall color tone and fails to reflect the keyword “morning.” Although the image generated by the DALL·E 2 model exhibits richer colors, problems remain in composition and detail, as the flowers partially obscure the bird’s chest. From the perspective of professional photographic composition, the robin in the image generated by Stable Diffusion 1.5 is slightly overemphasized relative to the overall layout, lacking the commonly used photographic technique of negative space. In contrast, the image generated by the proposed model adopts the rule-of-thirds composition, with the robin perched on a flowering branch in the lower-right area. The subject size and placement are relatively balanced, while substantial open space is reserved in the upper-left area, satisfying the aesthetic requirements of dynamism and visual breathing space in photographic works. Regarding light and shadow rendering, the background colors generated by Stable Diffusion 1.5 appear somewhat monotonous. Although DALL·E 2 provides richer colors, the lighting tones on the feathers are inconsistent, resulting in insufficient overall harmony. The proposed model more effectively simulates the soft side-backlighting effect of early morning, with dew drops on the feathers showing natural reflections and smooth tonal transitions. In terms of depth-of-field control, the background blur generated by the proposed model exhibits an out-of-focus transition effect, clearly distinguishing the foreground flowers from the background layers. By contrast, the background blur produced by the other models is relatively uniform and lacks layered variation. The above analysis indicates that the proposed model has clear advantages in key elements of photographic art and better satisfies the aesthetic standards of professional photography. Additionally, 10 text prompts containing different scenes, lighting conditions, and levels of complexity were selected, and the images generated by each model were rated by the same group of professionals to avoid bias in results caused by using only two prompt texts. The average scores of each model under 10 prompts are shown in Table 4. Table 4Average expert ratings of different models under 10 text prompts (1–5 points).ModelsDetail qualityColor coordinationComposition rationalityThe proposed model4.54.64.4Stable Diffusion 1.53.94.03.8DALL·E24.14.24.0ControlNet3.73.93.6SDXL4.04.13.9 Table 4Average expert ratings of different models under 10 text prompts (1–5 points).ModelsDetail qualityColor coordinationComposition rationalityThe proposed model4.54.64.4Stable Diffusion 1.53.94.03.8DALL·E24.14.24.0ControlNet3.73.93.6SDXL4.04.13.9 Average expert ratings of different models under 10 text prompts (1–5 points). The data in Table 4 show that in the additional 10 text scenes, the average scores of the proposed model in the dimensions of detail quality, color coordination, and composition rationality are still better than other comparison models, and are basically consistent with the rating trends suggested by the two examples in Figs. 3 and 4. This indicates that the proposed model also has good generalization ability under a wider range of text conditions. Quantitative result analysis Building on the qualitative analysis in the previous section, this study further conducts quantitative evaluation. The purpose is to verify the differences in generation performance between the proposed model and mainstream baseline models through objective data. Three metrics, FID, IS, and CLIP Score, are adopted in the experiment, and the results are shown in Fig. 4. Fig. 4Comparison of performance indexes of different models on test set. Comparison of performance indexes of different models on test set. In Fig. 4, the proposed model achieves an FID value of 15.3, an IS value of 28.7, and a CLIP Score of 0.312, all outperforming the other baseline models. Specifically, its FID is 1.8 lower than that of DALL·E2, indicating better visual authenticity of the generated images. Its IS is 1.5 higher than that of DALL·E2, suggesting better diversity and quality of the generated content. The reasons for the above results are twofold: (1) Multi-level conditional control maps semantics, layout, and style uniformly to the hidden space, reducing the conditional uncertainty of the denoising network at each time step and making noise prediction more accurate; (2) The self-attention mechanism in spatial consistency optimization enhances long-range dependency modeling, reduces local distortion, and makes the feature distribution of generated images closer to the true image distribution. To further evaluate the model’s performance under different levels of semantic complexity, this study divides the test set into three subsets based on the length of text prompts and the detail of descriptions. The analysis results are shown in Fig. 5. Fig. 5Comparison of CLIP scores under different text complexity levels. Comparison of CLIP scores under different text complexity levels. In Fig. 5, the proposed model achieves CLIP Scores of 0.325, 0.315, and 0.296 under simple, medium, and complex text prompts, respectively. This is because the layout control module in the proposed model encodes the spatial relationships in the text into structural feature maps and concatenates them with U-Net features channel by channel, allowing the model to accurately place objects according to the text description. Meanwhile, the dynamic weight adjustment mechanism strengthens the layout weight in the early denoising step and the style weight in the later stage, thus maintaining a high image text matching degree even under complex semantic descriptions. In addition, its score under complex text prompts is 0.007 higher than that of DALL·E2. This indicates that the proposed model has superior semantic understanding ability. It also demonstrates that the model can maintain generation quality when processing high-complexity texts, a finding of great value for practical application scenarios. Ablation experiment The layout control module and style control module in the proposed model are removed separately. The performance changes of the model were observed on the same sub-test set, and the results are shown in Fig. 6. Fig. 6Influence of different modules on the overall performance of the model. Influence of different modules on the overall performance of the model. In Fig. 6, after removing the layout control module, the model’s FID value increases while the CLIP Score decreases. This indicates that the lack of spatial layout guidance leads to reduced structural quality of generated images and lower matching degree with the spatial relationships described in the text. After removing the style control module, although the impact on the IS value is negligible, the FID still decreases to a certain extent. This demonstrates that the style control module can effectively improve the overall authenticity and visual effect of the generated images. The comparative experiments of different module combinations are carried out on the same test set to verify the individual contribution of each module and whether there is interaction effect between modules. The experimental results obtained are shown in Table 5. Table 5Experimental results of ablation with different module combinations.Module combinationFIDISCLIP ScoreSemantic only (no layout, no style)18.226.50.295Semantics +layout16.127.80.308Semantic +style17.027.20.303Semantics +Layout+ Style (complete model)15.328.70.312 Table 5Experimental results of ablation with different module combinations.Module combinationFIDISCLIP ScoreSemantic only (no layout, no style)18.226.50.295Semantics +layout16.127.80.308Semantic +style17.027.20.303Semantics +Layout+ Style (complete model)15.328.70.312 Experimental results of ablation with different module combinations. In Table 5, the model exhibits the poorest performance when only semantic control is used, with an FID of 18.2. After adding layout control, the FID decreases by 2.1, while adding style control reduces the FID by 1.2, indicating that the layout module contributes more substantially to the improvement of structural quality. When both layout and style controls are incorporated simultaneously, the FID further decreases to 15.3, which is slightly better than the additive effect achieved when each is used separately. This suggests that the functions of different modules in the multi-level conditional control framework are complementary and effective. Discussion In-depth analysis of the previous experimental results shows how the model works. The proposed approach is a multi-level conditional-control diffusion model. It fuses multi-modal information: semantic cues, layout guidance, and style references. This fusion boosts visual realism, improves compositional soundness, and enhances artistic expression. Together these gains raise the overall quality of the generated photographs. It outperforms mainstream baseline models in processing complex semantic descriptions and imitating specific styles, demonstrating the practical value of the refined conditional fusion method in new media art creation. However, the model still lacks sufficient expressiveness when handling highly abstract artistic texts or those with profound narrative content. This indicates that current Generative Artificial Intelligence (GAI) still faces challenges in deep semantic understanding and creative expression. Meanwhile, it provides important insights for future research to integrate symbolic reasoning and data-driven methods, enabling more efficient human-machine collaborative art creation. Conclusion Research contribution At the theoretical level, this study proposes and verifies a diffusion model framework integrating multi-level conditional control mechanisms for new media art photography creation. This framework effectively combines semantic understanding, spatial layout guidance, and visual style adaptation, further advancing research on the controllability of generative models in the art field. At the practical application level, the model provides artists with an efficient and intuitive creative method. It can generate photographic works with high visual consistency and aesthetic quality according to abstract concepts and reference styles, which is expected to lower the technical threshold for professional-level visual content creation. This plays a positive role in promoting the application of AI technology in the creative industry. Future works and research limitations The limitations of this study are as follows: The model’s ability to understand artistic texts involving deep narratives and highly abstract semantics needs improvement. Additionally, when generating high-resolution images, the model may encounter issues such as detail distortion and high computational costs. Future research will focus on integrating external knowledge graphs to enhance the model’s semantic reasoning capability. Meanwhile, more efficient adaptive sampling algorithms will be explored to improve generation efficiency, thereby extending the model’s ability to generate dynamic visual content.