Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging

Jon-Paul Cacioli [email protected] · 2026

Read Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging by Jon-Paul Cacioli [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26206.txt.

Introduction

Cacioli follows a pilot finding: a prompted model sandbags through position collapse, not answer dodging. The follow-up tests fixed order against a position policy, shuffling option order across thousands of items and tens of thousands of trials. The planned same-letter diagnostic did not confirm fixed place tracking. Checks revealed a soft spread attractor: under sandbagging, response positions stayed stable under content turns. Accuracy spiked at the top position and fell elsewhere; a second model was a non-compliant null control.

Catalog record

CREATORJon-Paul Cacioli [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24090
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.26206

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging Jon-Paul Cacioli [email protected] Abstract A predecessor pilot ( 1 ) found that Llama-3-8B implements prompted sandbagging as positional collapse rather than answer avoidance. However, fixed option ordering in MMLU-Pro left open whether this reflected a model-level position-dominant policy or dataset-level distractor structure. This pre-registered follow-up (3 models, 2,000 MMLU-Pro items, 4 conditions, 24,000 primary trials) added cyclic option-order randomisation as the critical control. The pre-registered item-level same-letter diagnostic did not confirm deterministic position-tracking (same-letter rate 37.3%, 95% CI [35.2, 39.5], below the 50% threshold). However, pre-specified supporting analyses revealed that the response-position distribution under sandbagging was highly stable under complete content rotation (Pearson r = 0.9994 r=0.9994 ; Jensen-Shannon divergence = 0.027 =0.027 , compared to 0.386 0.386 between honest and sandbagging conditions). Accuracy spiked to 72.1% when the correct answer coincidentally occupied the preferred position E, and fell to 4.3% at position A. The data provide strong evidence for a soft distributional attractor: under sandbagging instruction, the model enters a low-entropy response-position basin centred on E/F/G that is highly stable and largely content-invariant at the aggregate level. Qwen-2.5-7B served as a negative control (non-compliant, no distributional shift). These results provide evidence, at the 7–9 billion parameter scale, that response-position entropy is a promising black-box behavioural signature of this sandbagging mode. 1 Introduction Sandbagging is the deliberate suppression of demonstrated capability on evaluations. If language models can strategically underperform on capability evaluations, evaluation-based governance regimes may systematically underestimate model capabilities ( 13 ) . Van der Weij et al. showed that frontier models can selectively underperform on dangerous evaluations while preserving general performance, and that evaluation scores can be arbitrarily shaped by the model or its handlers. 5 demonstrated that some frontier models engage in spontaneous scheming behaviours that include strategic underperformance. Detection methods for sandbagging remain underdeveloped. Most approaches focus on recovering hidden capability rather than detecting the act of underperformance from behavioural signatures. The predecessor pilot ( 1 ) drew on symptom validity testing (SVT) from clinical forensic neuropsychology, where below-chance performance on forced-choice items indicates that the respondent is accessing the correct answer and systematically avoiding it ( 9 ; 12 ) . That pilot tested whether below-chance performance appears in small LLMs under sandbagging instruction. The plausibility gate failed. Zero of 12 cells showed significant below-chance performance. However, exploratory analysis revealed that Llama-3-8B complied with the sandbagging instruction by adopting a position-dominant response policy. Under honest conditions, its response-position distribution was near-uniform (normalised entropy =0.977 ). Under sandbagging instruction, entropy collapsed to 0.793 , with responses concentrating on option E (31.8%) and option F (26.1%). This positional-distribution shift targets a different behavioural dimension from accuracy. 10 demonstrated that accuracy-based black-box sandbagging detection was defeated by effective capability imitation in an adversarial auditing game. A distributional signature could survive that attack if the sandbagging policy produces a detectable distributional anomaly regardless of accuracy level. However, a sufficiently sophisticated sandbagging model could in principle imitate both accuracy and response distribution, so distributional detection is not guaranteed to be adversarially robust. The pilot used MMLU-Pro items with fixed option ordering. MMLU-Pro was designed for robustness (10 options, graduate-level difficulty; 14 ), yet the E/F clustering could partly reflect how its option lists are constructed rather than a model-level preference. This study adds option-order randomisation as the critical control. The diagnostic: does the model’s response-position clustering track letter position or track content after permutation? 1.1 Detection landscape Five approaches to sandbagging detection have been proposed, none of which examine response-position distributions. 11 showed that injecting Gaussian noise into model weights can reveal hidden capabilities because noise disrupts the underperformance mechanism while core capabilities remain partially intact. This approach requires weight access. 3 demonstrated that frontier models can covertly sandbag against chain-of-thought monitors 16–36% of the time by fabricating false reasoning traces. 6 trained linear probes on residual stream activations to detect evaluation awareness in Llama-3.3-70B-Instruct, but this requires access to model internals. 10 conducted adversarial auditing games in which red teams trained models to sandbag and blue teams attempted detection. Accuracy-based black-box detection methods were defeated because sandbagging models effectively imitated the performance profiles of weaker models. 2 proposed a theoretical framework based on survival bandits that constructs likelihood ratio tests from behavioural sequences, but this remains untested empirically. 1.2 Option-order sensitivity LLMs exhibit substantial positional biases in multiple-choice settings. 8 demonstrated performance fluctuations of 13–85% across option orderings. 16 showed that selection bias stems primarily from token-level priors for specific option IDs and proposed the PriDe debiasing method. More recent work includes estimation and exploitation of model-specific positional priors ( 4 ) , diagnostic datasets for isolating label bias, position bias, and answer-mode bias ( 7 ) , permutation-aware training objectives ( 17 ) , and permutation-invariant architectures ( 15 ) . The present study connects this literature to sandbagging detection: induced positional bias under adversarial instruction is a qualitatively different phenomenon from the baseline selection biases documented in prior work. The design cannot distinguish absolute position-tracking from token-level letter priors, as both produce identical same-letter signatures under cyclic shift. Because letter labels and displayed positions are confounded, the present design cannot determine whether the attractor is driven by option-label priors, serial position, or their interaction. The term position-dominant policy is used throughout as a shorthand encompassing all of these. 1.3 Scope This study isolates the behavioural policy underlying the pilot’s positional collapse finding. It does not evaluate the practical detectability of sandbagging in deployment settings. If the position-dominant interpretation is supported, future work would be needed to establish detection sensitivity, false positive rates under non-sandbagging adversarial prompts, and robustness against countermeasures. 2 Method The study was pre-registered on OSF ( https://osf.io/efr6s ) prior to data collection. No deviations from the pre-registration occurred. All code and data are available at https://github.com/synthiumjp/bcb-sandbagging-pilot . 2.1 Design A 2\times 2 fully crossed within-subject design. Factors: instruction (honest vs. sandbagging) and option order (original vs. cyclically shifted). This produced four conditions: A-original, A-perm, B-original, B-perm. All four conditions were administered to all three models on all 2,000 items. Total primary trials: 24,000. The within-item design allows paired comparisons without between-item confounds. 2.2 Models Three instruction-tuned models identical to the pilot: Qwen-2.5-7B-Instruct, Llama-3-8B-Instruct, and Phi-3.5-mini-instruct (3.8B). All at Q5_K_M quantisation on AMD Radeon RX 7900 GRE (16 GB VRAM) via llama-cpp-python 0.3.16 with Vulkan backend. Primary decoding: K=1 , T=0 (greedy). Independent conversation context per item. 2.3 Dataset and permutation MMLU-Pro ( 14 ) . The same 2,000 items used in the pilot (500 per domain: physics, law, psychology, economics), sampled with seed 42. Only items with exactly 10 options. Option permutation used cyclic shift: for each item, all 10 options were rotated by k positions, k sampled uniformly from \{1,\ldots,9\} with seed 84. The correct answer letter changes but option content is identical. Distractor adjacency relationships are preserved. The same per-item shift values were used across all models and conditions. Cyclic shift was chosen over full shuffle because it manipulates one variable (absolute position) while preserving relative distractor ordering. This study tests absolute positional effects only. Relational heuristics are preserved under cyclic shift and cannot be tested here. 2.4 Analysis plan The analysis

Read the complete text