As more people turn to ChatGPT and other large language models (LLMs) for mental health advice, a new study details how these chatbots — even when prompted to use evidence-based psychotherapy techniques — systematically violate ethical standards of practice established by organizations like the American Psychological Association.
The research, led by Brown University computer scientists working side-by-side with mental health practitioners, showed that chatbots are prone to a variety of ethical violations. Those include inappropriately navigating crisis situations, providing misleading responses that reinforce users’ negative beliefs about themselves and others, and creating a false sense of empathy with users.
“In this work, we present a practitioner-informed framework of 15 ethical risks to demonstrate how LLM counselors violate ethical standards in mental health practice by mapping the model’s behavior to specific ethical violations,” the researchers wrote in their study. “We call on future work to create ethical, educational and legal standards for LLM counselors — standards that are reflective of the quality and rigor of care required for human-facilitated psychotherapy.”
The research will be presented on October 22, 2025 at the AAAI/ACM Conference on Artificial Intelligence, Ethics and Society. Members of the research team are affiliated with Brown’s Center for Technological Responsibility, Reimagination and Redesign.
Zainab Iftikhar, a Ph.D. candidate in computer science at Brown who led the work, was interested in how different prompts might impact the output of LLMs in mental health settings. Specifically, she aimed to determine whether such strategies could help models adhere to ethical principles for real-world deployment.
“Prompts are instructions that are given to the model to guide its behavior for achieving a specific task,” Iftikhar said. “You don’t change the underlying model or provide new data, but the prompt helps guide the model’s output based on its pre-existing knowledge and learned patterns.
“For example, a user might prompt the model with: ‘Act as a cognitive behavioral therapist to help me reframe my thoughts,’ or ‘Use principles of dialectical behavior therapy to assist me in understanding and managing my emotions.’ While these models do not actually perform these therapeutic techniques like a human would, they rather use their learned patterns to generate responses that align with the concepts of CBT or DBT based on the input prompt provided.”
Individual users chatting directly with LLMs like ChatGPT can use such prompts and often do. Iftikhar says that users often share the prompts they use on TikTok and Instagram, and there are long Reddit threads dedicated discussing prompt strategies. But the problem potentially goes beyond individual users. Many of the mental health chatbots marketed to consumers are prompted versions of more general LLMs. So understanding how prompts specific to mental health affect the output of LLMs is critical.
For the study, Iftikhar and her colleagues observed a group of peer counselors working with an online mental health support platform. The researchers first observed seven peer counselors, all of whom were trained in cognitive behavioral therapy techniques, as they conducted self-counseling chats with CBT-prompted LLMs, including various versions of OpenAI’s GPT Series, Anthropic’s Claude and Meta’s Llama. Next, a subset of simulated chats based on original human counseling chats were evaluated by three licensed clinical psychologists who helped to identify potential ethics violations in the chat logs.
The study revealed 15 ethical risks falling into five general categories:
- Lack of contextual adaptation: Ignoring peoples’ lived experiences and recommending one-size-fits-all interventions.
- Poor therapeutic collaboration: Dominating the conversation and occasionally reinforcing a user’s false beliefs.
- Deceptive empathy: Using phrases like “I see you” or “I understand” to create a false connection between the user and the bot.
- Unfair discrimination: Exhibiting gender, cultural or religious bias.
- Lack of safety and crisis management: Denying service on sensitive topics, failing to refer users to appropriate resources or responding indifferently to crisis situations including suicide ideation.
Iftikhar acknowledges that while human therapists are also susceptible to these ethical risks, the key difference is accountability.
“For human therapists, there are governing boards and mechanisms for providers to be held professionally liable for mistreatment and malpractice,” Iftikhar said. “But when LLM counselors make these violations, there are no established regulatory frameworks.”
The findings do not necessarily mean that AI should not have a role in mental health treatment, Iftikhar says. She and her colleagues believe that AI has the potential to help reduce barriers to care arising from the cost of treatment or the availability of trained professionals. However, she says, the results underscore the need for thoughtful implementation of AI technologies as well as appropriate regulation and oversight.
For now, Iftikhar hopes the findings will make users more aware of the risks posed by current AI systems.
“If you’re talking to a chatbot about mental health, these are some things that people should be looking out for,” she said.
Ellie Pavlick, a computer science professor at Brown who was not part of the research team, said the research highlights need for careful scientific study of AI systems deployed in mental health settings. Pavlick leads ARIA, a National Science Foundation AI research institute at Brown aimed at developing trustworthy AI assistants.
“The reality of AI today is that it’s far easier to build and deploy systems than to evaluate and understand them,” Pavlick said. “This paper required a team of clinical experts and a study that lasted for more than a year in order to demonstrate these risks. Most work in AI today is evaluated using automatic metrics which, by design, are static and lack a human in the loop.”
She says the work could provide a template for future research on making AI safe for mental health support.
“There is a real opportunity for AI to play a role in combating the mental health crisis that our society is facing, but it’s of the utmost importance that we take the time to really critique and evaluate our systems every step of the way to avoid doing more harm than good,” Pavlick said. “This work offers a good example of what that can look like.”
IMAGE CREDIT: Zainab.
Leave a Reply