TECCI: Tricky Edits of Collected and Curated Images

Aishwarya Agrawal, Roy Hirsch, Yasumasa Onoe, Sherry Ben, Jason Baldridge · 2026

Read TECCI: Tricky Edits of Collected and Curated Images by Aishwarya Agrawal, Roy Hirsch, Yasumasa Onoe, Sherry Ben, Jason Baldridge (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01213.txt.

Introduction

Agrawal and co-authors build TECCI, a hard test for image edits. Text-guided editors still fail on many kinds of edits. Hard cases span place, motion, view, scale, and creative change. TECCI holds 7,550 pairs of images and edit prompts. Its images span seven groups, picked to expose weak spots. Gemini wrote prompts for five edit types per image.

The set also has 530 images with hand-written hard prompts. Five top models face human grades on three points. These are prompt fit, small change, and image quality. No model tops a 22% success rate. Nano Banana Pro does best overall. An auto-rater built on Gemini matches human grades 74.7% of the time.

Catalog record

CREATORAishwarya Agrawal, Roy Hirsch, Yasumasa Onoe, Sherry Ben, Jason Baldridge
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01213

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TECCI: Tricky Edits of Collected and Curated Images Aishwarya Agrawal, Roy Hirsch, Yasumasa Onoe, Sherry Ben, Jason Baldridge Abstract Despite tremendous recent progress, current text-guided image editing methods still struggle with many aspects of editing involving instruction following, minimally editing the source image, and ensuring high visual quality. These problems are especially apparent when the requested edit is challenging, such as those that involve position, motion, viewpoint, scale and creative edits. To systematically test generative image editors, we propose a novel image editing benchmark – TECCI : T ricky E dits of C ollected and C urated I mages. TECCI consists of a completely new set of images we are releasing. The images in TECCI span 7 image categories. The images and these categories were curated intentionally to target weaknesses of existing methods. The edit instructions in TECCI are automatically generated by Gemini, covering 5 edit types per source image. We also curated a set of 530 images for which we created challenging manually written edit instructions. Overall, TECCI contains 7550 pairs of images and edit instructions. We conduct human evaluations of five leading image editing models on TECCI . Humans judge outputs along three dimensions: 1) instruction following, 2) minimality of the edits, and 3) visual quality. To scale-up the evaluation, we also build an auto-rater using Gemini that achieves 74.7 % 74.7\% accuracy in matching human evaluations. Our evaluations reveal that: 1) none of the models exceed a 22% overall success rate, demonstrating the challenging nature of TECCI , 2) Nano Banana Pro is the best performing model overall, 3) models perform significantly better at instruction following compared to minimal edits and visual quality, 4) models struggle with editing architecture and nature images which require strong understanding of spatial layout and intricate visual details. 5) reasoning and creative edits are the most difficult, whereas color and appearance edits are the easiest. 1 Introduction Figure 1 : Overview of TECCI dataset. Representative samples from the different subsets and edit categories. Text guided image editing is being increasingly used to benchmark the capabilities of generative image models [ 21 , 9 , 1 , 16 , 18 , 17 , 14 ] . This task evaluates whether these models can – 1) faithfully follow edit instructions (specified via a text prompt), 2) minimally edit the source image, i.e., do not make any changes in the source image beyond what is specified in the instruction, 3) maintain the visual quality of the source image in the generated edited image. Thus, this task serves as a good benchmark to evaluate visio-linguistic understanding as well as controlled generation capabilities of generative models. We present a novel text-guided image editing benchmark to systematically evaluate generative models on challenging edit types: T ricky E dits of C ollected and C urated I mages ( TECCI ). TECCI targets image and edit types that have been observed (qualitatively) to be difficult for models, such as changing the text displayed on images, changing the time shown on clocks, structural and viewpoint changes to buildings, pose and action changes to animals and art subjects, state changes to vehicles, and scale changes to landscapes. TECCI also includes humorous creative edits that require models to carry out some degree of imagination. Unlike most existing image editing dataset [ 21 , 9 , 1 , 16 , 18 , 17 , 14 ] , the images TECCI have been taken and donated by the authors, ensuring that these images have not been seen by any current models in their training. TECCI spans 7 image categories: Text, Clock, Vehicle, Architecture, Art, Animal, and Nature. The edit instructions in TECCI are automatically generated by Gemini covering 5 edit types per source image. We also curated a smaller set (530) of challenging manually written edit instructions. Overall, TECCI consisting of 7550 pairs of images and edit instructions (see Fig. 1 for representative examples). We conduct human evaluations of five state-of-the-art image editing models on our proposed benchmark. Humans judge each edited sample along three dimensions: 1) faithfulness to the language instruction, 2) minimality of the edits, 3) visual quality of the edited image. In order to scale-up the evaluation, we also build an auto-rater using Gemini that achieves 74.7\% accuracy in matching human evaluations. Our evaluations reveal that: 1) none of the models exceed a 22% overall success rate, demonstrating the challenging nature of TECCI , 2) Nano Banana Pro is the best performing model overall, 3) models perform significantly better at instruction following compared to minimal edits and visual quality, 4) models struggle with editing architecture and nature images which require strong understanding of spatial layout and intricate visual details. 5) reasoning and creative edits are the most difficult, whereas color and appearance edits are the easiest. Contributions. We make the following contributions: \circ We curate and release a large-scale challenging image editing benchmark – TECCI – consisting of 7550 pairs of images and edit instructions. The images in TECCI are completely new, manually captured and donated by the authors. \circ We conduct a large-scale human evaluation of five state-of-the-art editing models on TECCI . In total, we collect 32,875 human judgements. \circ We conduct a comprehensive analysis of model performance using the collected human judgements revealing gaps in model capabilities and directions for future research. \circ We build an auto-rater to scale up the model evaluation on the full set of TECCI . We show that our auto-rater achieves an accuracy of 74.7\% in matching human evaluations. 2 Related Work Several benchmarks have been proposed for the task of text guided image editing [ 21 , 9 , 1 , 16 , 18 , 17 , 14 ] . These benchmarks differ along the following axes: \bullet Localized edits vs global edits : Localized edits involve edits to a local region of an image, spanning simple edits such as object addition and removal, attribute changes to an object, and complex edits such as counting, spatial relations and action edits. Global edits involve making global changes to an image such as stylistic changes, changing the weather, daytime into nighttime etc. TECCI covers all these types of edits. \bullet Image source : The images in the existing benchmarks are sourced either from existing natural image datasets such as COCO [ 3 ] , OpenImages [ 11 ] , Visual Genome [ 8 ] or are generated synthetically using image generation models. TECCI differs from all existing benchmarks in that TECCI uses completely new set of images manually taken by one of the authors. Thus, TECCI includes natural images that do not already exist on the internet. \bullet Edit instruction source : The edit instructions in the existing benchmarks are either manually written or automatically generated using LLMs. TECCI includes a larger set of automatically generated instructions as well as a smaller challenging set of instructions manually written by one of the authors. \bullet Dataset size : The sizes of the existing datasets range from 250 to 10,000. TECCI lies on the higher end of this spectrum, with 7.5K samples. \bullet Availability of human judgements : Some existing benchmarks collect human judgements on the model generated edits, either using single-sided (judging each model output independently)

Read the complete text