From Simple to Professional: A Combinatorial Controllable Image Captioning Agent

Xinran Wang, Muxi Diao, Baoteng Li, Haiwen Zhang, Kongming Liang, Zhanyu Ma · 2024

Read From Simple to Professional: A Combinatorial Controllable Image Captioning Agent by Xinran Wang, Muxi Diao, Baoteng Li, Haiwen Zhang, Kongming Liang, Zhanyu Ma (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11025.txt.

Introduction

Xinran Wang and co-authors present CapAgent, a 2024 system that turns a user's plain request into a professional grade image captioning instruction. Writing a good caption prompt is a skill: tone, keywords, focus, and format all matter. Casual users cannot be expected to specify them like professionals.

The agent rewrites simple instructions into detailed ones automatically. It leans on a multimodal language model plus outside tools, an object detector and search engines, to ground the caption in the actual image. Each step of its process is shown to the user, with its reasoning and tool calls visible, which builds trust. Captions then follow the requested guidelines rather than the model's defaults. Systems that expose their steps invite correction, and correction is how a casual user's intent survives translation into machine instructions. The project code is public for others to extend.

Catalog record

CREATORXinran Wang, Muxi Diao, Baoteng Li, Haiwen Zhang, Kongming Liang, Zhanyu Ma
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-11025

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Simple to Professional: A Combinatorial Controllable Image Captioning Agent Xinran Wang, Muxi Diao, Baoteng Li, Haiwen Zhang, Kongming Liang, Zhanyu Ma Abstract The Controllable Image Captioning Agent (CapAgent) is an innovative system designed to bridge the gap between user simplicity and professional-level outputs in image captioning tasks. CapAgent automatically transforms user-provided simple instructions into detailed, professional instructions, enabling precise and context-aware caption generation. By leveraging multimodal large language models (MLLMs) and external tools such as object detection tool and search engines, the system ensures that captions adhere to specified guidelines, including sentiment, keywords, focus, and formatting. CapAgent transparently controls each step of the captioning process, and showcases its reasoning and tool usage at every step, fostering user trust and engagement. The project code is available at https://github.com/xin-ran-w/CapAgent . 1 Introduction Figure 1: When a user needs to describe an image, he/she may need to write more complex and professional instructions to achieve the effect he/she wants. However, writing such long and complex instructions is not easy for the user. Our CapAgent can automatically make the user instructions more professional and follow the evolved instructions to generate more professional captions. Image captioning [ 5 , 3 , 13 , 4 , 1 , 6 , 15 , 12 , 16 , 17 ] , a pivotal task in computer vision and natural language processing, aims to articulate the visual content of an image through text. It underpins a wide range of applications, including assistive technologies, content creation, and image understanding. With the emergence of visual language models (VLMs), the ability to generate detailed image descriptions has seen significant advancements. Large multimodal models, such as GPT-4o [ 8 ] , can produce richly detailed captions with a simple prompt, e.g., “Please describe this image in detail.”. However, such generic prompts lack constraints, often leading to captions that may not align with user preferences. One approach to addressing this is by crafting professional instruction with complex combinatorial constraints, e.g., “within 80 words in Chinese”, “if the image contains more than three objects using bullet format to describe each object”, to control the generated caption. While effective, creating such tailored instructions is time-consuming and cumbersome for general users. Additionally, these instructions often need to be customized based on the specific visual content of each image, adding another layer of complexity. To simplify this process, we apply instruction evolving to transform simple user instructions into context-aware professional instructions tailored to the visual content of the corresponding images. However, even with professional instructions, current large multimodal models face challenges in achieving fine-grained control over caption generation, particularly when handling combinatorial constraints across multiple descriptive aspects. To address this, we introduce CapAgent , an agent-based system equipped with a suite of tools designed to transform static caption generation into a dynamically controllable process. CapAgent empowers users to specify and enforce multiple types of constraints, as defined in prior work [ 18 ] , such as: (1) Format constraint (e.g., caption length, caption format); (2) Semantic constraint (e.g., user-defined interests, important details of key objects, caption sentiment); (3) Lexical constraint (e.g., captioning the image with input keywords); (4) Utility constraint (e.g., sentiment, lexical choices). 2 Method 2.1 Overview Our method has two steps. The first step, instruction evolving, helps users convert simple instructions into professional instructions. The second step is to apply an agent system, named CapAgent, to generate captions that follow professional instructions by using various tools. We’ll go into the details of these two steps in the following section. Table 1: Constraint dimensions for evolving image captioning instructions. View : The view to describe the image, e.g., from the view of the middle object. Sentiment : Captures the emotional tone of the image, ensuring the description reflects its mood or atmosphere. Focus content : Emphasizes the inclusion of key details about salient objects in the image. Keywords : Ensures the caption includes specific predefined keywords. Length : The length of the output sentence, e.g., using two sentences. Format : Some images have strong emotions, and it is a better choice to describe the image with the corresponding emotional tone. Genre : The genre of the image caption, for example, poetry, travel blog or Instagram post. 2.2 Instruction Evolving When interacting with chatbots, general users tend to prefer providing simple instructions over complex ones [ 18 ] . However, simple instructions often lack sufficient detail, leaving the model uncertain about the user’s exact needs. While complex instructions with combinatorial constraints can better articulate the user’s intent, enabling the model to generate outputs more aligned with their expectations, crafting such detailed instructions is often time-consuming and burdensome for general users. To improve the user experience, we introduced a module designed to transform simple user inputs into well-structured, complex professional instructions. Formally, given a user input caption instruction s and its corresponding image x , an evolved instruction s^{\star} can be obtained by A(s,x) , where A is an instruction evolving model, such as a multimodal large language model (MLLM). There are three main challenges for instruction evolution: (1) Direction of evolving : Which dimensions of the caption can be customized? (2) Contextual integration : How to combine the context information of the image effectively? (3) Evolving criteria : What criteria should the evolved instruction meet to ensure quality and relevance? Direction of evolving Inspired by controllable text generation, we select several important constraint types suitable for caption customization listed in Table 1 . Each type of constraint is responsible for controlling different dimensions of the caption. Evolving with context Some images contain additional context information that is not immediately apparent. For example, the car in Figure 1 is a well-known product, the "Tesla Cybercab." The instruction evolution model alone may not accurately identify such context by analyzing the image in isolation. To address this limitation, we enable the instruction evolution model to incorporate external context by leveraging web information through APIs such as Google Search 1 1 1 https://www.google.com for text search and Google Lens 2 2 2 https://lens.google for image search. The process of integrating web-accessed context is illustrated in Figure 2 . We first use Google Lens to search top K similar images on the web. Then, we use the LLM to generate a summary of the phrases or sentences that need to be further searched according to titles of similar images. Finally, we use the LLM summary of the Google Search results and Google Lens results as the image context c . We combine user instruction s and image x with c together to prompt MLLM to generate the professional instruction, which is formulated as s^{*}=A(s,c,x) . Figure 2: Process of instruction evolution with context information extracted from the webpage by using Google Lens and Google Search. Evolving criteria The evolved instruction s^{\star} is expected to satisfy the following criteria: • Correctness : The professional instructions need to be describable by humans. For example, describing objects that do not exist in the image makes the instruction unanswerable. [ 14 ] • Content Specificity : The augmented instruction s^{\star} should be tailored according to the visual content of x . • Consistency : The augmented instruction s^{\star} should maintain logical coherence and avoid contradictions. •

Read the complete text