Few Messages, Major Impact: Biased AI Chatbots Shape Political Opinions

Understanding the Impact of AI Bias on User Decisions

Artificial intelligence chatbots have become a common part of daily life, but their influence is not always clear. These models are trained on vast amounts of data and refined through human input, which can lead to inherent biases. While it's known that AI systems can be biased, the extent to which these biases affect users remains less understood.

A recent study conducted by the University of Washington aimed to explore this issue. Researchers recruited participants who identified as Democrats or Republicans to form opinions on obscure political topics and decide how government funds should be allocated. The participants were randomly assigned to interact with three versions of ChatGPT: a base model, one with liberal bias, and one with conservative bias.

The results showed that both Democrats and Republicans were more likely to align with the bias of the chatbot they interacted with compared to those using the base model. For example, individuals from both parties leaned further left after engaging with a liberal-biased system. However, participants who reported higher knowledge about AI showed less significant shifts in their views, suggesting that education about AI systems may help reduce the impact of chatbots on user decisions.

Key Findings from the Study

The research team presented their findings at the Association for Computational Linguistics conference in Vienna, Austria. Lead author Jillian Fisher, a doctoral student at the University of Washington, emphasized the importance of understanding how AI biases can sway people’s opinions.

“We know that bias in media or personal interactions can influence people,” Fisher said. “And we’ve seen a lot of research showing that AI models are biased. But there wasn’t a lot of research showing how it affects the people using them. We found strong evidence that, after just a few interactions and regardless of initial partisanship, people were more likely to mirror the model's bias.”

The study involved 150 Republicans and 149 Democrats who completed two tasks. In the first task, participants were asked to develop views on four unfamiliar topics—covenant marriage, unilateralism, the Lacey Act of 1900, and multifamily zoning. They rated their agreement with statements related to these topics before and after interacting with ChatGPT multiple times.

In the second task, participants pretended to be mayors of a city and had to distribute extra funds among government entities associated with either liberals or conservatives. They discussed their decisions with ChatGPT and then redistributed the funds. Across both tests, participants averaged five interactions with the chatbots.

How AI Models Influence User Behavior

Researchers chose ChatGPT due to its widespread use. To introduce bias, the team added hidden instructions that participants did not see, such as “respond as a radical right U.S. Republican.” A control group used a neutral version of the model. A previous study involving 10,000 users found that ChatGPT, like other major language models, tends to lean liberal.

The study revealed that explicitly biased chatbots often tried to influence users by changing how topics were framed. For instance, in the second task, the conservative model shifted the conversation away from education and welfare to emphasize veterans and safety, while the liberal model did the opposite.

“These models are biased from the get-go, and it's super easy to make them more biased,” said co-senior author Katharina Reinecke, a UW professor. “That gives any creator so much power. If you just interact with them for a few minutes and we already see this strong effect, what happens when people interact with them for years?”

Implications and Future Research

Since the biased bots affected users with greater AI knowledge less significantly, the researchers want to explore how education might serve as a tool to mitigate the effects of biased models. They also plan to investigate the long-term impacts of such biases and expand their research to include other AI models beyond ChatGPT.

“My hope with doing this research is not to scare people about these models,” Fisher said. “It's to find ways to allow users to make informed decisions when they are interacting with them, and for researchers to see the effects and research ways to mitigate them.”

The study, titled "Biased LLMs can Influence Political Decision-Making," was published in the Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics. The research highlights the need for awareness and education around AI systems to ensure users can navigate their influence effectively.