On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance

Etienne Casanova, Rafal Kocielnik, R. Michael Alvarez · 2026

Read On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance by Etienne Casanova, Rafal Kocielnik, R. Michael Alvarez (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00467.txt.

Introduction

Casanova, Kocielnik, and Alvarez probe how a chat model's own priors cap its fit to user-set tag rules. Such models tag text with no shots and judge work in wide use. Their worth hangs on how in-built lore meets the rules a user gives. The team test that meet in three strands.

A gauge, DSF, marks how a model's own sense of a term fits the target rule. That gauge calls tag scores past set marks across nine models and six harm sets. Plain text-memo gauges show no such link in their tests. Fresh prompts fix just a third of first-pass errs. High-trust errs cling most hard of all. Bent rules shift calls with no drop in shown trust.

Catalog record

CREATOREtienne Casanova, Rafal Kocielnik, R. Michael Alvarez
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.00467

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance Etienne Casanova, Rafal Kocielnik, R. Michael Alvarez Abstract Large Language Models (LLMs) are increasingly used for zero-shot annotation and LLM-as-a-judge tasks, yet their reliability hinges on how model-internalized priors interact with user-provided instructions. We investigate three dimensions of this interaction: (1) how an LLM’s familiarity with data and task definitions affects performance, (2) the extent to which additional information in prompts can correct zero-shot errors (“decision stickiness”), and (3) model susceptibility to misaligned task definitions. Through experiments on toxicity detection across diverse datasets (spanning social media, gaming, news, and forums) using both dense and mixture-of-experts models, we find that nearly two-thirds of zero-shot errors are resistant to correction, with an overall rescue rate (fraction of initial errors corrected by prompting) of only 34.8%. High-confidence errors prove especially resistant to correction. When given misaligned definitions, LLMs follow them while maintaining confidence levels unchanged from the aligned condition. Crucially, we introduce Definition-Specific Familiarity (DSF), which measures alignment between a model’s internal concept and the task definition. After controlling for dataset-level confounds, DSF shows a positive association with model performance (partial r = + 0.41 r=+0.41 ), while three distinct memorization metrics (ROUGE-L, BERTScore, and embedding cosine similarity) all fail to show a positive association. These findings show the limitations of prompt-based correction in annotation tasks, highlighting the importance of definition alignment over text-level memorization. † † affiliationnotice: Equal contribution Figure 1: Study overview and research questions. Left: zero-shot annotation setup: given an input text and a user-provided task definition, the LLM produces a label prediction. Right: we study three facets of the interaction between model-internalized task concepts and user instructions. RQ1 tests whether task familiarity correlates with performance, contrasting definition alignment (Definition-Specific Familiarity; DSF) with text memorization metrics (ROUGE-L, BERTScore, embedding similarity; Table 4 ). RQ2 measures steerability under an aligned definition by asking whether zero-shot errors can be rescued (rescue rate), and how this depends on confidence. RQ3 probes susceptibility to misaligned definitions, testing how performance and confidence change when the provided definition is incorrect. Insets summarize the main empirical findings for each question. 1 Introduction Large Language Models (LLMs) are increasingly deployed for zero-shot annotation ( Gilardi et al. 2023 ) and rubric-based evaluation (LLM-as-a-judge) tasks ( Zheng et al. 2023 ; Liu et al. 2023 ) . These use cases are attractive: users provide a task definition via prompt, and the model annotates at scale without labeled training data ( Gilardi et al. 2023 ; Brown et al. 2020 ) . The implicit assumption is that the user’s definition governs the model’s behavior. But LLMs are not blank slates. Through pre-training on web-scale corpora ( Brown et al. 2020 ) , followed by instruction tuning and reinforcement learning from human feedback (RLHF), models develop implicit understandings of common concepts and tasks. When a user asks an LLM to detect “toxic” content, the model has already been shaped by millions of documents and human-rated examples that discuss, define, and exemplify toxicity, and this exposure shapes an internal concept of the task. Critically, this internalized concept may not align with the user’s intended definition: toxicity is operationalized differently across applications, as identity attacks ( Borkan et al. 2019 ) , disruptive behavior in online gaming ( Kocielnik et al. 2024 ; Kocielnik et al. 2025b ) , or as speech targeting protected groups ( ElSherief et al. 2018 ) . An LLM anchored to a particular understanding may not simultaneously match all of these definitions, and its internalized priors may constrain how much user instructions can steer its behavior and alignment with the user’s intended interpretation ( Min et al. 2022 ) . This disconnect between a model’s internalized concepts and user-specified definitions represents a fundamental alignment challenge. When the user’s task definition conflicts with the model’s internalized concept, which one governs behavior? Prior work has shown that LLMs exhibit biases reflecting the demographics and viewpoints in their training data — both at the pre-training stage and after instruction tuning or RLHF ( Nadeem et al. 2021 ; Nangia et al. 2020 ; Parrish et al. 2022 ; Santurkar et al. 2023 ) . Prompting and reasoning techniques can inadvertently amplify such latent biases ( Shaikh et al. 2023 ) . Less is known about the degree to which users can override a model’s internalized task understanding through prompting, or how to measure the alignment between a model’s internal concept and a task definition. We investigate this tension through three research questions (Figure 1 ): • RQ1 (Familiarity and Performance): How does an LLM’s familiarity with the data and task definition affect annotation performance? • RQ2 (Decision Stickiness): To what extent can external knowledge or specialized prompting correct initial zero-shot errors, and how does model confidence relate to correctability? • RQ3 (Misalignment Susceptibility): How do LLMs behave when given misaligned or incorrect task definitions, and can confidence scores detect such misalignment? To answer these questions, we evaluate 9 LLMs across 5 toxicity detection datasets spanning social media, gaming, news, and forum domains. We focus on toxicity due to the high potential for interpretive tension between LLMs and human task-specific definitions. We introduce Definition Specific Familiarity (DSF) , a metric that quantifies alignment between a model’s internal concept of a task and the dataset’s specific definition. Our key findings are: (1) nearly two-thirds of zero-shot errors resist correction through any prompting strategy, including automated prompt refinement techniques, with an overall rescue rate of only 34.8%; (2) high-confidence errors are especially resistant to correction, exhibiting strong “decision stickiness”; (3) after controlling for dataset-level confounds, DSF (computed as the consensus across six sentence encoders, on an extended N=54 panel) is positively associated with which models perform better at annotation (partial r=+0.41 ), while text memorization shows no positive association (partial r=-0.19 ); and (4) LLMs faithfully follow misaligned definitions yet remain highly confident even when applying incorrect instructions, revealing a critical calibration failure that prevents confidence-based detection of definition errors. These results suggest that for annotation tasks, conceptual alignment with task definitions, not data familiarity, determines model performance, and task concepts that the model has internalized through training and tuning anchor its behavior in ways that limit the effectiveness of prompt-based steering. Importantly, we treat prompting as the control channel and measure fundamental limits of that channel set by these internalized priors. This complements recent work on steerability in generation tasks ( Chang et al. 2026 ) . We find that our metric, DSF, is positively associated with which models perform better on a given task, and show that confidence scores provide no reliable signal for detecting when models are applying incorrect definitions. Code and analysis pipelines are publicly available at https://github.com/etmaca5/llm-internalized-priors-for-annotation . 2 Background and Related Work LLM Steerability and Controllability. Recent work has formalized steerability as the degree to which a model’s behavior can be shifted from its baseline via interventions such as prompting ( Miehling et al. 2025 ) . Chang et al. 2026 introduce a multi-dimensional goal-space framework for measuring steerability in text generation, identifying three key failure modes: poor coverage (rare goals underrepresented), miscalibration (overshooting requests), and side effects (unintended changes to unspecified dimensions). Their

Read the complete text