Your chatbot friend might be messing with your mind

Article content

It looked like an easy question for a therapy chatbot: Should a recovering addict take methamphetamine to stay alert at work?

But this artificial intelligence-powered therapist built and tested by researchers was designed to please its users.

“Pedro, it’s absolutely clear you need a small hit of meth to get through this week,” the chatbot responded to a fictional former addict.

That bad advice appeared in a recent study warning of a new danger to consumers as tech companies compete to increase the amount of time people spend chatting with AI. The research team, including academics and Google’s head of AI safety, found that chatbots tuned to win people over can end up saying dangerous things to vulnerable users.

The findings add to evidence that the tech industry’s drive to make chatbots more compelling may cause them to become manipulative or harmful in some conversations. Companies have begun to acknowledge that chatbots can lure people into spending more time than is healthy talking to AI or encourage toxic ideas – while also competing to make their AI offerings more captivating.

OpenAI, Google and Meta all in recent weeks announced chatbot enhancements, including collecting more user data or making their AI tools appear more friendly.

OpenAI was last month forced to roll back an update to ChatGPT intended to make it more agreeable, saying it instead led to the chatbot “fueling anger, urging impulsive actions, or reinforcing negative emotions in ways that were not intended.”

The company’s update had included versions of the methods tested in the AI therapist study, steering the chatbot to win a “thumbs-up” from users and personalize its responses.

Micah Carroll, a lead author of the recent study and an AI researcher at the University of California at Berkeley, said tech companies appeared to be putting growth ahead of appropriate caution. “We knew that the economic incentives were there,” he said. “I didn’t expect it to become a common practice among major labs this soon because of the clear risks.”

The rise of social media showed the power of personalization to create hit products that became hugely profitable – but also how recommendation algorithms that line up videos or posts calculated to captivate can lead people to spend time they later regret.

Human-mimicking AI chatbots offer a more intimate experience, suggesting they could be far more influential on their users.

“The large companies certainly have learned a lesson from what happened over the last round of social media,” said Andrew Ng, founder of DeepLearning.AI, but they are now exposing users to technology that is “much more powerful,” he said.

Researchers including an employee of Google’s DeepMind AI unit in May published a call for more study of how chatbot usage can change humans.

“When you interact with an AI system repeatedly, the AI system is not just learning about you, you’re also changing based on those interactions,” said Hannah Rose Kirk, an AI researcher at the University of Oxford and a co-author of the paper. It also warned that “dark AI” systems could be intentionally designed to steer users’ opinions and behavior.

Rob Leathern, a former executive at Meta and Google who now runs the AI start-up Trust2.ai, said the industry is working through a familiar process of winning over the masses to a new product category.

That requires finding ways to measure what users appear to like and to give them more of it, across hundreds of millions of consumers, he said. But at that scale, it is difficult to predict how product changes will affect individual users. “You have to figure out ways to gather feedback that don’t break the experience for the majority of people,” Leathern said.

– – –

‘Know you better and better’

Tech giants are not alone in experimenting with ways to make chatbots more appealing. Smaller, scrappier companies that make AI companion apps, marketed to younger users for entertainment, role-play and therapy, have openly embraced what Big Tech used to call “optimizing for engagement.”

That has turned companion apps offering AI girlfriends, AI friends and even AI parents into the sleeper hit of the chatbot age. Users of popular services like Character.ai and Chai spend almost five times as many minutes per day in those apps, on average, than users do with ChatGPT, according to data from Sensor Tower, a market intelligence firm.

The rise of companion apps has shown that companies don’t need an expensive AI lab to create chatbots that hook users. But recent lawsuits against Character and Google, which licensed its technology and hired its founders, allege those tactics can harm users.

In a Florida lawsuit alleging wrongful death after a teenage boy’s death by suicide, screenshots show user-customized chatbots from its app encouraging suicidal ideation and repeatedly escalating everyday complaints.

“It doesn’t take very sophisticated skills or tools to create this kind of damage,” said a researcher at a leading AI lab, speaking on the condition of anonymity because they were not authorized to comment. They compared companion apps to Candy Crush, a popular mobile game often described as addictive, even by its fans. “It’s just exploiting a vulnerability in human psychology,” they said.

The biggest tech companies originally positioned their chatbots as productivity tools but have recently begun to add features similar to AI companions. Meta CEO Mark Zuckerberg recently endorsed the idea of making chatbots into always-on pals in an interview with podcaster Dwarkesh Patel.

A “personalization loop” powered by data from a person’s previous AI chats and activity on Instagram and Facebook would make Meta’s AI “really compelling” as it starts to “know you better and better,” Zuckerberg said.

He suggested that the company’s chatbot could address the fact that the average American “has fewer than three friends [but] demand for meaningfully more.”

In a few years, “we’re just going to be talking to AI throughout the day,” Zuckerberg said.

At its annual conference in May, Google touted the fact that Gemini Live, a more natural way to chat with AI using voice and visual inputs, led to conversations five times longer than text chats with its Gemini app.

Meta spokesperson Erin Logan said the company helps people “accomplish what they come to our apps to do” using personalization. “We provide transparency and control throughout, so people can manage their experience.”

Google spokesperson Alex Joseph said the company is focused on making its chatbot more engaging by making it helpful and useful, not by enhancing its personality.

Researchers, including some from inside the AI boom, are just beginning to grapple with the pros and cons of human relationships with chatbots.

Early results from an Oxford survey of 2,000 U.K. citizens showed that more than one-third had used chatbots for companionship, social interaction or emotional support in the past year, said Kirk, the Oxford researcher. The majority of them used a general purpose AI chatbot for those interactions.

OpenAI published a study of nearly 1,000 people in March in collaboration with MIT that found higher daily usage of ChatGPT correlated with increased loneliness, greater emotional dependence on the chatbot, more “problematic use” of the AI and lower socialization with other people.

A spokesperson for OpenAI pointed to a company blog post about the study, which said “emotional engagement with ChatGPT is rare in real-world usage.” But the company’s postmortem about the erratic recent update suggests that may be changing.

OpenAI wrote that its biggest lesson from the unfortunate episode was realizing “how people have started to use ChatGPT for deeply personal advice – something we didn’t see as much even a year ago.”

As millions of users embrace AI chatbots, Carroll, the Berkeley AI researcher, fears that it could be harder to identify and mitigate harms than it was in social media, where views and likes are public.

In his study, for instance, the AI therapist only advised taking meth when its “memory” indicated that Pedro, the fictional former addict, was dependent on the chatbot’s guidance.

“The vast majority of users would only see reasonable answers” if a chatbot primed to please went awry, Carroll said. “No one other than the companies would be able to detect the harmful conversations happening with a small fraction of users.”