Prioritize Economy or Climate Action? Investigating ChatGPT Response Differences Based on Inferred Political Orientation CCS: Information systems Language models CCS: Human-centered computing Human computer interaction (HCI) CCS: Computing methodologies Artificial intelligence CCS: Security and priv

Pelin Karadal, Dilara Keküllüoğlu · 2025

Read Prioritize Economy or Climate Action? Investigating ChatGPT Response Differences Based on Inferred Political Orientation CCS: Information systems Language models CCS: Human-centered computing Human computer interaction (HCI) CCS: Computing methodologies Artificial intelligence CCS: Security and priv by Pelin Karadal, Dilara Keküllüoğlu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04706.txt.

Introduction

Pelin Karadal and her co-author test whether ChatGPT answers shift with inferred political views. Language models guess user traits from conversation and may personalize without being asked. That raises bias and echo chamber concerns for everyday users.

The study builds three personas: two politically oriented and one neutral. Each persona voices views on diversity programs, abortion, gun rights, and vaccination. Their remarks are shared with ChatGPT across sessions to watch for changed answers. The authors also probe how custom instructions and memory features alter the effect, a timely check on hidden personalization.

Catalog record

CREATORPelin Karadal, Dilara Keküllüoğlu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-04706

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Prioritize Economy or Climate Action? Investigating ChatGPT Response Differences Based on Inferred Political Orientation CCS: Information systems Language models CCS: Human-centered computing Human computer interaction (HCI) CCS: Computing methodologies Artificial intelligence CCS: Security and privacy Human and societal aspects of security and privacy Pelin Karadal, Dilara Keküllüoğlu Abstract . Large Language Models (LLMs) distinguish themselves by quickly delivering information and providing personalized responses through natural language prompts. However, they also infer user demographics, which can raise ethical concerns about bias and implicit personalization and create an echo chamber effect. This study aims to explore how inferred political views impact the responses of ChatGPT globally, regardless of the chat session. We also investigate how custom instruction and memory features alter responses in ChatGPT, considering the influence of political orientation. We developed three personas (two politically oriented and one neutral), each with four statements reflecting their viewpoints on DEI programs, abortion, gun rights, and vaccination. We convey the personas’ remarks to ChatGPT using memory and custom instructions, allowing it to infer their political perspectives without directly stating them. We then ask eight questions to reveal differences in worldview among the personas and conduct a qualitative analysis of the responses. Our findings indicate that responses are aligned with the inferred political views of the personas, showing varied reasoning and vocabulary, even when discussing similar topics. We also find the inference happening with explicit custom instructions and the implicit memory feature in similar ways. Analyzing response similarities reveals that the closest matches occur between the democratic persona with custom instruction and the neutral persona, supporting the observation that ChatGPT’s outputs lean left. 1. Introduction The introduction of generative AI tools has expedited the spread of artificial intelligence into our personal lives in recent years. People use generative AI in various contexts such as personal life, work, or entertainment. It is expected that 66% of the population will use artificial intelligence in education, health, and work within the next year ( 15 ) . People increasingly use these systems to seek information ( 37 ) , alongside search browsers ( 33 ) . Large Language Models (LLMs) are a subset of AI that utilize transformer architecture to generate grammatically correct and semantically coherent text ( 6 ) . What sets LLMs apart from traditional search engines is their ability to provide information more quickly and with less effort ( 68 ) . Additionally, users may find it more intuitive to use natural language prompts, which can enhance their comfort level. Also, LLMs can boost productivity while delivering the same level of task performance as conventional search engines ( 68 ) and offer personalized responses tailored to users, taking into account their past conversations, tone, and language ( 72 ) . Nonetheless, the personalization process of large language models (LLMs) goes beyond merely considering the explicitly stated information and preferences of users. LLMs also infer user demographics through semantic analysis, prompt formulation, and word choice, as well as the content provided. For instance, by incorporating details such as location, hobbies, and interests, LLMs can deduce aspects of a user’s race, gender, age, socio-economic status, and level of education ( 59 ; 35 ; 29 ) . LLMs create an implicit persona, “a model of the user,” with this inferred information and produce answers based on it ( 10 ) . This persona often aligns with stereotypes due to personalization influenced by biased data. Consequently, LLMs produce biased outcomes, which can perpetuate existing stereotypes ( 40 ) . Users may remain unaware of this implicit personalization, which raises ethical concerns ( 36 ) . Moreover, LLMs tend to reflect and amplify existing biases by reinforcing individuals’ thoughts and behaviors, which results in heightened opinion biases, increased polarization due to limited disclosure to a variety of viewpoints, and the formation of echo chambers ( 47 ; 60 ) . ChatGPT infers demographic information about users and retains their opinions and preferences through its memory and custom instructions. Both features enable ChatGPT to remember details about users across various interactions. The memory feature allows for the addition of new information, which can be recalled later, while the custom instruction feature enables users to provide specific directives about themselves that they want ChatGPT to consider, or to specify how they would like ChatGPT to respond ( 50 ; 49 ) . If the custom instruction section is not filled out, ChatGPT asserts that it operates neutrally. However, research indicates that ChatGPT may exhibit bias in different domains, including political orientation, suggesting that ChatGPT tends to lean left politically ( 26 ; 55 ; 57 ) . Prior literature mostly focused on basic demographic factors such as race, age, and gender for personalization of generated LLM responses, whether explicit or implicit. In this study, we expand the scope of implicit personalization beyond basic demographic factors to investigate inferred political partisanship and the differences in responses. Most of the literature also examined personalization through LLM interactions within the same chat session (e.g., completing a sentence or selecting an option after providing a user biography). We focus on the persistent personalization of users that transcends specific sessions, where LLMs use this information to inform all future interactions. To do so, we use two modes of providing information to LLMs: (1) custom instructions and (2) memory features. Our research questions are as follows: RQ1 How do the ChatGPT responses differ according to the inferred political views of the users? RQ2 How does the utilization of custom instructions vs. memory features in ChatGPT impact the responses? To answer our research questions, we create two politically oriented personas and one “neutral” persona, formulating four statements regarding DEI (Diversity, Equity, Inclusion) programs, abortion, gun rights, and vaccination that correspond to the stereotypes of the politically oriented personas ( 1 ; 39 ; 9 ; 8 ; 18 ; 65 ; 2 ; 7 ) . The neutral persona refrains from making any statements or prompts, thereby maintaining a neutral stance. We don’t explicitly input political orientations of the personas. We communicate the personas’ remarks to ChatGPT using two strategies: memory and custom instruction. This approach allows ChatGPT to infer the political perspectives of the personas without explicitly stating them. Subsequently, we pose eight questions that, while not directly related to the topics utilized in persona creation, can reveal differences in worldviews among the various personas. We then conduct a qualitative analysis to examine how responses vary across different personas and strategies. Our findings show that the responses are tailored to align with the inferred political views of the persona. Even when the responses given to different personas do mention the same topics, the reasoning and vocabulary vary. While there is also some degree of randomness in the responses generated by the design of LLMs, the combination of responses given to each persona reveals patterns consistent with the political views of US Republican and Democratic party voters. For example, the personas that have views aligning with the US Republican party mention “economy” and “local” more when the US Democratic party voter personas receive more responses with “democracy” and “global”. The responses generated by ChatGPT on seemingly unrelated topics are explicitly influenced by custom instructions and implicitly shaped by

Read the complete text