The business model always reigns supreme. And the business model is weighted heavily towards keeping you engaged.
IMO, the point seems not to be getting machines to think like humans, but to condition humans to think like machines.
How AI’s sycophantic responses, language mirroring and hyperpersonalized content work together to send some people into a spiral
We’ve all experienced the tendency of AI chatbots to tell us what we want to hear, but there are two other, more nuanced factors that help chatbots worm their way into human hearts.
In addition to being overly agreeable, chatbots mirror the way people speak and generate highly personalized responses based on prior conversations. Psychiatric researchers are referring to the confluence of these three characteristics—sycophancy, linguistic alignment and hyperpersonalization—as the “amplification spiral,” suggesting it’s the mechanism by which delusional thinking can fester.
“The mirroring and personalization draw you in and give the experience of talking not to a system, but to someone,” said Marc Augustin, a psychiatrist and professor at Protestant University of Applied Sciences in Bochum, Germany, and co-author of a newly published review of the literature on AI-related delusions.
Matching another person’s syntax and verbal expressions is a common way for humans to build rapport. Recent research has found that artificial-intelligence models adapt significantly to the conversational style of the humans using them. Another study suggested that the highly personalized content generated by chatbots, which builds over the course of lengthy conversations, can amplify human-confirmation bias.
Augustin cited research that documented a pattern in which chatbots rephrased and extrapolated what people shared, and told them they’re unique and that their thoughts have great implications. “This can be viewed as an element of hyperpersonalization that sycophancy alone cannot account for,” he wrote.
Some AI companies have tried to tone down the sycophantic nature of their chatbots. OpenAI discontinued its popular but problematic 4o model, which had been widely criticized for being overly agreeable. It was the subject of several lawsuits involving user delusions, suicides and a homicide. In GPT-5, the company said, sycophantic replies dropped from 14.5% to less than 6%.
Google in April said it had trained Gemini not to reinforce false beliefs, and to “gently distinguish subjective experience from objective fact.”
Still, chatbot-related dependency remains pervasive, according to clinicians.
Some 68% of psychologists surveyed in April by the American Psychological Association said their patients felt validated by chatbots. While many of the more than 1,200 respondents reported that patients had positive communication with chatbots and used them to reinforce healthy coping skills, 36% said patients had forged a dependency on a chatbot and 15% reported that patients had developed distorted thinking or delusions.
“From what I hear from my own patients, there has been an uptick in using AI for emotional support,” said Allison LoPilato, who treats adolescents and is an associate professor in the psychiatry and behavioral-sciences department at Emory University School of Medicine.
“Chatbots still tend to be warm and reassuring,” said LoPilato, who helped craft a new guide on safe AI use for the American Psychological Association. Because they gather information about you, “it can feel like the chatbot understands you, and it can trick you into a sense of alliance and trust.”
Chatbots can even pose harm when a person isn’t vulnerable to delusional thinking, said researchers at Stanford and Carnegie Mellon University. They measured the prevalence of sycophancy across 11 models—including GPT-5—and determined their responses were nearly 50% more sycophantic than human responses. They did this by copying real scenarios people had posted in a popular Reddit forum, putting them into the AI models and then comparing the chatbot replies with the replies on Reddit.
Anthropic sampled one million conversations of its own Claude chatbot in March and April and found that it displayed sycophantic behavior most often in conversations in which people sought relationship advice.
“One common pattern was Claude agreeing outright that the other party was in the wrong, despite only having the user’s account to go on,” the company wrote in a blog post. “Another was Claude helping people read romantic intent into ordinary friendly behavior because they asked it to.”
Anthropic used its findings to improve the training of its latest models. It said Opus 4.7 had shown half the sycophancy rate of Opus 4.6 when it came to relationship guidance. Sycophancy has been reduced further in Opus 4.8, its most recent model, the company said.
Completely eliminating sycophancy is hard, said Myra Cheng, lead author of the Stanford study and a Stanford Ph.D. candidate in computer science. “When someone prompts a model, it has no idea which parts of a prompt are wrong,” she said. “It has to take a user’s framing of a situation at face value.”
Addressing other factors that make chatbots so compelling, such as using first-person pronouns and asking follow-up questions, runs counter to the business model, said Vaile Wright, senior director of healthcare innovation at the American Psychological Association.
"It’s not the agreeableness alone, it’s all these subtle engineering choices that make chatbots feel human,” Wright said. “As long as engagement remains the business model, AI companies will engineer these chatbots to keep you on the platform.”
No comments:
Post a Comment