STaD: Scaffolded Task Design for Identifying Compositional Skill Gaps in LLMs

Sungeun An, Swanand Ravindra Kadhe, Shailja Thakur, [email protected] · 2026

Read STaD: Scaffolded Task Design for Identifying Compositional Skill Gaps in LLMs by Sungeun An, Swanand Ravindra Kadhe, Shailja Thakur, [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18177.txt.

Introduction

An, Kadhe, Thakur, DeLuca, and Patel propose STaD, a framework for exposing compositional skill gaps in large language models. Aggregate benchmark scores hide which reasoning skills a model lacks and how to fix them. STaD generates controlled task variations that add structured, step-by-step support, a technique called scaffolding. Tests on six models across three reasoning benchmarks reveal distinct failure points and skill gaps for each model.

Catalog record

CREATORSungeun An, Swanand Ravindra Kadhe, Shailja Thakur, [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE90702
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.18177

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

STaD: Scaffolded Task Design for Identifying Compositional Skill Gaps in LLMs Sungeun An, Swanand Ravindra Kadhe, Shailja Thakur, [email protected] Abstract Benchmarks are often used as a standard to understand LLM capabilities in different domains. However, aggregate benchmark scores provide limited insight into compositional skill gaps of LLMs and how to improve them. To make these weaknesses visible, we propose S caffolded Ta sk D esign (STaD) framework. STaD generates controlled variations of benchmark tasks based on the concept of scaffolding , which introduces structured, incremental support in a step-by-step manner. Rather than inspecting failures individually, this approach enables systematic and scalable probing of model behavior by identifying the specific reasoning skill compositions they lack. Treating the LLM as a black box, our experiments on six models of varying sizes reveal multiple failure points in three reasoning benchmarks and highlight each model’s unique and distinct skill gaps. Source Code: Datasets: 1 Introduction Figure 1: Scaffolded tasks for identifying compositional skill gap s2 ( Subtraction : Eggs remaining for sale). Benchmarks are standardized tests designed to evaluate the strengths and limitations of large language models (LLMs) across domains. However, most existing evaluations often treat complex tasks as monolithic 22 ; 12 . Like a single math test score, aggregate benchmark scores may suggest that a model is "bad at math" without explaining why. In particular, two models may achieve the same overall accuracy on the same benchmarks yet have entirely different skill gaps: one may lack a specific reasoning ability, another may struggle when multiple skills must be combined, and a third may have the necessary knowledge but fail to situate the knowledge in the context of the task. Addressing such skill gaps requires examining a model’s intermediate reasoning processes. One line of prior work has aimed to make model reasoning more explicit through Chain-of-Thought prompting, which elicits intermediate reasoning steps 25 . While this can surface intermediate traces of reasoning, these approaches are difficult to scale or analyze systematically 12 , and the generated explanations may not faithfully reflect the model’s actual internal reasoning process 18 . Another line of work focuses on task decomposition, which breaks complex tasks into simpler sub-tasks and makes intermediate reasoning steps more explicit and configurable 15 ; 5 ; 27 ; 21 ; 13 ; 17 . This provides a more structured representation of the reasoning process. However, existing decomposition-based methods are primarily used to improve task performance or to examine reasoning step by step, rather than to systematically diagnose failure modes. From a cognitive science perspective, competence is not only what a learner or a model can do independently, but also what it can achieve with appropriate guidance 24 . In this view, scaffolding refers to targeted support that is gradually removed as proficiency develops 23 . We adapt this idea to LLM evaluation, using scaffolding not to improve performance, but as a diagnostic tool to reveal where and why models fail. In this work, we propose S caffolded Ta sk D esign (STaD) framework to combine task decomposition and scaffolding to systematically identify the steps at which LLMs fail and the specific reasoning skill compositions they lack. In a nutshell, STaD breaks down benchmark tasks into sub-tasks representing individual reasoning steps; it then creates controlled variations of the original task by providing scaffolding at specific sub-tasks. By measuring performance across scaffolded variations, STaD can pinpoint exactly where the model struggles. Unlike isolated skill evaluation, we focus on compositional skill gaps that arise when multiple reasoning skills must be combined. For example, consider the task in Figure 1 , where a model must calculate Janet’s daily earnings from selling eggs, which involves Addition, Subtraction, and Multiplication ( s_{1} , s_{2} , and s_{3} ). Suppose the model fails on the original task ( q ). We first scaffold s_{1} (Addition: total eggs used daily), reducing the task to Subtraction and Multiplication. The model still fails, suggesting that the limitation is not confined to Addition. We then additionally scaffold s_{2} (Subtraction: eggs remaining for sale), leaving only Multiplication. In this setting, the model succeeds and produces the correct answer of $18. This reveals the model’s compositional skill gap: it cannot independently perform Subtraction ( s_{2} ) in combination with Multiplication ( s_{3} ), even though it can handle Multiplication alone. We evaluate STaD on three public reasoning benchmarks: ToT Arithmetic (Temporal Reasoning) 6 , GSM8K (Grade School Math) 4 , and Math-Hard (Difficult Math Problems) 11 . Our results show that failures are multifaceted: models with similar overall performance can exhibit very different bottlenecks, some skills are present but fail in context, and compositional skill interaction frequently create challenges. Scaffolding further distinguishes tasks that can be reliably solved with targeted support from those that remain fundamentally intractable. STaD provides actionable insights, including: (1) Identifying skill combinations that cause failures; (2) Distinguishing weaknesses that can be mitigated with scaffolding from those that are fundamentally difficult; (3) Highlighting skill-level areas for further training or targeted interventions. Our main contributions are threefold: • Concept : We formalize scaffolded competence for LLMs, capturing not only independent task performance but also the abilities a model can demonstrate with support. This distinguishes between tasks that become solvable with scaffolding and those that remain unsolved, revealing compositional skill gaps and failure modes. • Framework : We introduce S caffolded Ta sk D esign (STaD) framework that generates scaffolded variations of benchmark tasks to systematically probe reasoning and compositional skill gaps in LLMs. We release the codebase and datasets, enabling application of STaD to existing benchmarks for constructing scaffolded task designs. • Empirical Study : We apply our framework to three reasoning benchmarks. Our results reveal distinct skill gaps across models and demonstrate the value of detailed evaluation beyond aggregate scores. 2 Scaffolded Task Design (STaD) Given a benchmark, our goal is to construct scaffolded variations of its tasks that progressively introduce or remove support for intermediate reasoning steps. These scaffolded variations reveal latent competence and identify compositional skill gaps in the LLM. At a high level, STaD takes each benchmark question with its ground-truth answer and produces (1) a decomposition of the reasoning process into intermediate steps with corresponding partial answers, and (2) a set of scaffolded task variants that selectively provide support for these steps. We then evaluate models on both the original and scaffolded tasks to quantify scaffolded competence and estimate the minimum level of scaffolding required for successful reasoning. Formally, we define a benchmark as a set of questions and corresponding answers, Q=\{q_{1},q_{2},\dots,q_{n}\} and A=\{a_{1},a_{2},\dots,a_{n}\} . STaD operates using three functional LLM roles, which we describe in detail below. Teacher model. The teacher model, f_{\text{teach}} , is responsible for generating scaffolded variations of each task including: (i) decomposing q into minimal reasoning units, (ii) computing corresponding intermediate answers, and (iii) constructing scaffolded task variants. The teacher model can be implemented using different instantiations, including a single LLM, multiple models, or human-in-the-loop supervision. Judge model. The judge model, f_{\text{judge}} , evaluates correctness for both intermediate and final predictions by comparing model outputs to the ground truth a . It returns a binary correctness score. Target model. The target model, f_{\text{target}} , is the model we aim to diagnose. It is

Read the complete text