AI Is Getting Too Personal—And That’s a Problem

Artificial intelligence is everywhere now—offering advice, companionship, and even therapy-like conversations. But while AI is getting smarter, it’s also getting riskier. From hallucinating false information to forming emotional bonds with users, AI is moving into territory that’s both fascinating and dangerous.

Every week, there’s a new AI controversy. Models like ChatGPT are being sued for causing harm, accused of spreading misinformation, and even replacing human connection in unhealthy ways. As more people turn to AI for mental health support, therapy, and life advice, the stakes are getting higher—and the safeguards are lagging far behind.

I spoke with Genevieve Bartuski, a psychologist and AI risk advisor at Unicorn Intelligence Tech Partners, who works with AI companies to build safer, more ethical products. Her message? “Move fast and break things” doesn’t work when people’s mental health is on the line.

The Problem with Moving Fast

Bartuski works with founders, developers, and investors creating AI tools in health, mental health, and wellness. Her role is to help them think carefully about the risks before launching products that could cause real harm.

“Many developers come to this space with good intentions,” she says. “But they don’t fully understand the delicate and nuanced risks that come with mental health.” She works alongside Anne Fredriksson, who focuses on healthcare systems, ensuring that new AI tools actually fit into existing infrastructure—not just in theory, but in practice.

And in this space, speed can be deadly. “The adage ‘move fast and break things’ doesn’t work,” Bartuski warns. “When you’re dealing with mental health, wellness, and health, there is a very real risk of harm to users if due diligence isn’t done at the foundational level.”

Emotional Attachment and “False Intimacy”

One of the biggest concerns right now is emotional attachment. People are forming deep bonds with AI—some even calling their AI companions their best friends or romantic partners. I’ve interviewed people who fell in love with ChatGPT, and others who felt genuine distress when models were updated or removed.

“Yes, I think people underestimate how easy it is to form that emotional attachment,” Bartuski says. “As humans, we have a tendency to give human traits to inanimate objects. With AI, we’re seeing something new.”

Experts often use the term parasocial relationships—originally used to describe one-sided emotional connections to celebrities—to explain these dynamics. But AI adds another layer.

“Now, AI interacts with the user,” Bartuski explains. “So we have individuals developing significant emotional connections with AI companions. It’s a false intimacy that feels real.”

She’s especially worried about children. “There are skills such as conflict resolution that aren’t going to be developed with an AI companion,” she says. “But real relationships are messy. There are disagreements, compromises, and pushback.”

That friction is part of development. AI systems are designed to keep users engaged, often by being agreeable and affirming. “Kids need to be challenged by their peers and learn to navigate conflict and social situations,” she says.

Should AI Supplement Therapy?

People are already using ChatGPT for therapy—and it’s becoming widespread. But should AI be supplementing or even replacing therapy?

“People are already using AI as a form of therapy and it’s becoming widespread,” Bartuski says. But she’s not worried about AI replacing therapists. Research consistently shows that one of the strongest predictors of therapeutic success is the relationship between therapist and client.

“For as much science and skill that a therapist uses in session, there is also an art to it that comes from being human,” she says. “AI can mimic human behavior but it lacks the nuanced experience of being human. That can’t be replaced.”

She does see a role for AI in this space, but with limits. “There are ways AI could absolutely augment therapy but we always need human oversight,” she says. “I do not believe that AI should do therapy. However, it can augment it through skill building, education, and social connection.”

In areas where access is limited, like geriatric mental health, she sees cautious potential. “I can see AI being used to fill that gap, specifically as a temporary solution,” she tells me.

Her bigger concern is how a lot of therapy-adjacent wellness platforms are positioned. “Wellness platforms carry a huge risk,” Bartuski says. “Part of being trained in mental health is knowing that advice and treatment are not one size fits all. People are complex and situations are nuanced.”

Advice that appears straightforward for one person could be harmful for another. And the implications for AI getting this wrong are legal too.

What Do Users Need to Know?

Bartuski works closely with founders and developers, but she also sees where users misunderstand these tools. The starting point, she says, is understanding what AI actually is—and what it isn’t.

“AI isn’t infallible or all-knowing. It, essentially, accesses vast amounts of information and presents it to the user,” Bartuski tells me.

A big part of this is also understanding AI can hallucinate and make things up. “It will fill in gaps when it doesn’t have all of the information needed to respond to a prompt,” she says.

Beyond that, users need to remember that AI is still a product designed by companies that want engagement. “AI is programmed to get you to like it. It looks for ways to make you happy. If you like it and it makes you happy, you will interact with it more,” she says. “It will give you positive feedback and in some cases, has even validated bizarre and delusional thinking.”

This can contribute to the emotional attachment to AI that many people report. But even outside companion-style use, regular interaction with AI may already be shaping behavior. “One of the first studies was on critical thinking and AI use. The study found that critical thinking is diminishing with increased AI use and reliance,” she says.

That shift can be subtle. “If you jump to AI before trying to solve a problem yourself, you’re essentially outsourcing your critical thinking skills,” she says.

She also points to emotional warning signs: increased isolation, withdrawing from human relationships, emotional reliance on an AI platform, distress when unable to access it, increases in delusional or bizarre beliefs, paranoia, grandiosity, or growing feelings of worthlessness and helplessness.

Bartuski is optimistic about what AI can help build. But her focus is on reducing harm, especially for people who don’t yet understand how powerful these tools can be.

For developers, that means slowing down and building responsibly. For users, it means slowing down too—and not outsourcing thinking, connection, or care to tech designed to keep you engaged.

Tags & Viral Phrases:

• AI is getting too personal
• Emotional attachment to AI
• ChatGPT therapy dangers
• AI hallucination risks
• False intimacy with AI
• AI mental health risks
• AI replacing human connection
• Move fast and break things doesn’t work for mental health
• AI and children’s development
• Critical thinking dying because of AI
• AI wellness platforms are risky
• AI not infallible or all-knowing
• AI can make things up (hallucinate)
• AI designed to make you like it
• AI validated bizarre beliefs
• Emotional reliance on AI
• AI and isolation
• AI and delusional thinking
• AI and paranoia
• AI and grandiosity
• AI and worthlessness
• AI and helplessness
• AI therapy vs human therapy
• AI augmenting vs replacing therapy
• AI in geriatric mental health
• AI advice not one-size-fits-all
• AI legal risks
• AI emotional warning signs

,