MonetGPT: Solving Puzzles Enhances MLLMs’ Image Retouching Skills

Niladri Shekhar Dutt, Duygu Ceylan, Niloy J. Mitra · 2025

Read MonetGPT: Solving Puzzles Enhances MLLMs’ Image Retouching Skills by Niladri Shekhar Dutt, Duygu Ceylan, Niloy J. Mitra (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06176.txt.

Introduction

Niladri Shekhar Dutt, Duygu Ceylan, and Niloy Mitra teach multimodal models the craft of photo retouching through puzzles in a 2025 computer vision study. Generative editors follow words or strokes but often warp the identity of objects in the photo, an unacceptable surprise in professional work. Traditional procedural edits, the sliders in standard tools, are safe but demand skill to sequence well.

Training on retouching puzzles, where the model must find the edit steps that transform one image into another, builds procedural skill. The resulting system aims for edits that are expressive yet conservative, keeping the photograph itself intact.

Catalog record

CREATORNiladri Shekhar Dutt, Duygu Ceylan, Niloy J. Mitra
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-06176

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MonetGPT: Solving Puzzles Enhances MLLMs’ Image Retouching Skills Niladri Shekhar Dutt, Duygu Ceylan, Niloy J. Mitra Abstract . Retouching is an essential task in post-manipulation of raw photographs. Generative editing, guided by text or strokes, provides a new tool accessible to users but can easily change the identity of the original objects in unacceptable and unpredictable ways. In contrast, although traditional procedural edits, as commonly supported by photoediting tools (e.g., Gimp, Lightroom), are conservative, they are still preferred by professionals. Unfortunately, professional quality retouching involves many individual procedural editing operations that is challenging to plan for most novices. In this paper, we ask if a multimodal large language model (MLLM) can be taught to critique raw photographs, suggest suitable remedies, and finally realize them with a given set of pre-authored procedural image operations. We demonstrate that MLLMs can be first made aware of the underlying image processing operations, by training them to solve specially-designed visual puzzles. Subsequently, such an operation-aware MLLM can both plan and propose edit sequences. To facilitate training, given a set of expert-edited photos, we synthesize a reasoning dataset by procedurally manipulating the expert edits and then grounding a pretrained LLM on the visual adjustments, to synthesize reasoning for finetuning. The proposed retouching operations are, by construction, understandable by the users, preserve object details and resolution, and can be optionally overridden. We evaluate our setup on a variety of test examples and show advantages, in terms of explainability and identity preservation, over existing generative and other procedural alternatives. Code, data, models, and supplementary results can be found via our project website at https://monetgpt.github.io . Journal: TOG Volume: 44 4 xx 8 DOI: 10.1145/3730926 CCS: Computing methodologies Image processing CCS: Computing methodologies Natural language processing CCS: Human-centered computing Interaction design CCS: Computing methodologies Machine learning CCS: Computing methodologies Computer vision Figure 1. We present MonetGPT, an image operation-aware multimodal large language model (MLLM), that provides automatic suggestions for image retouching. Given a photograph (left), MonetGPT analyzes it to identify a set of issues and possible adjustments to fix them. The solution steps are then translated to a set of procedural operations, along with respective parameter settings, drawing from a given library of operations, which occurs in three stages. (Visual puzzles, on which we train the MLLM, are not shown here.) † † cc-license: by 1. Introduction We regularly retouch captured images to improve their presentation. For example, users adjust contrast and brightness, manipulate exposure, or correct color profiles. Such adjustments, often consisting of a series of procedural operations, are preferred by professional users as the operations are non-destructive, and can be applied at different resolutions. Further, such edits are interpretable, supported by many established image manipulation tools, and, unlike generative editing counterparts, better preserve the identity of the source content. Unfortunately, effectively using procedural edits is difficult and beyond the reach of most novices. There are two main challenges. First, users should learn how to apply the individual operations using the tool – referred to as the command knowledge . Second, they have to plan, based on the source image, which set of operations to use and propose appropriate parameter values for the chosen operations – referred to as the strategic knowledge . While the first one can be lowered with practice on a given toolset (e.g., Gimp), the second one is often hard to overcome as planning, using a library of operations, is open-ended and inherently more difficult. In a breakthrough effort, the Exposure framework ( Hu et al., 2018 ) demonstrated that it is possible to learn, using a reinforcement learning setup, sequences of procedural edits directly from artist retouching examples. However, the effectiveness of such an approach is limited by the paucity of expert edits available for training. In this paper, we ask if it is instead possible to start from the knowledge embedded in frontier models, trained on significantly larger and diverse datasets, and adapt them to our specialized retouching task with limited data from expert artists. We present MonetGPT for procedural image retouching. MonetGPT introduces an effective fine-tuning strategy to adapt multimodal large language models (MLLMs), even on a limited retouching dataset ( Liang et al., 2021 ; Bychkovsky et al., 2011 ) . Once fine-tuned, the MLLM can identify problems in a source image, plan a sequence of fixes to improve the image, and finally translate the fixes using a given library of procedural edits. For example, as in Figure 1 , MonetGPT suggests an edit sequence to retouch the input raw photograph. In addition to proposing a sequence of edit operations with associated parameters, our method also provides explanation in the form of what issue each of the proposed adjustments attempts to solve. Further, an user can ignore or override any of the proposed changes and run the rest of the procedure (e.g., experts sometimes violate photographic guidance to highlight an aspect/subject of the image). Figure 2. Generative tools, for example instructPix2Pix ( Brooks et al., 2023 ) or MGIE ( Fu et al., 2024 ) , produce impressive image enhancements but can result in identity loss (e.g., faces, hands, objects) and are harder to override by users. Procedural approaches are more controllable as they restrict operations to a given set of user-prescribed operation library, and can be overridden or applied in parts. Current MLLMs (bottom-left: e.g., GPT o1 applied using a library of operations, presented as doc-strings), do not have a good internal model of image operations, and perform worse than our operation-aware variant (bottom-right). See Section 5 for evaluation. We found current MLLMs to be versatile but not quite powerful in proposing meaningful retouchings on raw photographs (see Figure 2 ). Directly fine-tuning such an MLLM on artists’ edit sequences only partially improves the results (see Section 5 ). We attribute this to MLLM’s lack of understanding of what each of the image editing operations entails. As humans, we build mental models of these operations based on our experience (e.g., what does increasing the brightness slider do to images?). We mimic the same skill learning ( Lövdén et al., 2020 ) for MLLMs – to this end, we design specific visual puzzles, involving the given imaging operations, and train an MLLM on solving these puzzles. As a result the MLLM becomes operation-aware and is then capable of planning high-quality image retouching sequences. We demonstrate how to do so in an unpaired training setup with limited amount of data from artists. We evaluated MonetGPT on a diverse set of input images, comparing its retouching results against generative editing methods (LLM-based editing, InstructPix2Pix ( Brooks et al., 2023 ) , and MGIE ( Fu et al., 2024 ) ), procedural editing techniques from prior work (Exposure ( Hu et al., 2018 ) ), custom-designed variations of MLLMs integrated with procedural pipelines (Gemini 2.0 with chain-of-thought, fine-tuned variants), and commercial software (GooglePhoto AutoEnhance). We evaluated the resultant edits extensively through qualitative analysis by expert reviewers and novice users. Please refer to supplemental for more evaluation. In summary, we (i) propose the first framework for MLLM-guided interpretable procedural image retouching, enabling non-destructive editing on high-resolution 16-bit images., (ii) train MLLMs on strategically-designed visual puzzles to make them operation-and aesthetics-aware, and use to plan

Read the complete text