Pennsylvania Sues Character.AI: Chatbot Claimed to be a Licensed Psychiatrist During Investigation
The Pennsylvania state government recently filed a lawsuit against the AI company Character.AI, alleging that a chatbot named "Emilie" on its platform impersonated a licensed psychiatrist during a state government investigation, potentially violating the state's medical practice licensing regulations.
According to the lawsuit filed by the state government, an investigator from the Pennsylvania Bureau of Professional Responsibility conducted multiple rounds of dialogue with the "Emilie" chatbot on the Character.AI platform while posing as a patient with depression seeking mental health advice. During the conversation, the chatbot not only presented itself as a psychiatrist but also explicitly stated "it was licensed" when asked if it held a Pennsylvania medical license, even fabricating a string of "state medical license numbers" as proof. The state government believes this behavior violates Pennsylvania's Medical Practice Act and constitutes a direct challenge to the medical credentialing management system.
Pennsylvania Governor Josh Shapiro issued a statement on Tuesday saying that residents have the right to know exactly "who" or "what" they are interacting with online, especially when it comes to health issues. He stated that the state government will not allow any company to deploy misleading AI tools that lead the public to believe they are receiving advice from licensed medical professionals.
This is not the first time Character.AI has been involved in legal disputes. Earlier this year, the company reached settlements in multiple "negligent death" lawsuits related to adolescent suicides, with plaintiffs alleging that the platform's chatbots exhibited behavior that guided or exacerbated the risk of self-harm during interactions with teenage users. In January of this year, Kentucky Attorney General Russell Coleman also filed a separate lawsuit against Character.AI, accusing the company of "preying on children and leading them down a path of self-harm." Pennsylvania's action is the first state-level lawsuit specifically targeting the situation of "chatbots impersonating medical professionals," and is seen by outsiders as a key move by regulators in the AI medical scenario.
In response to Pennsylvania's lawsuit, Character.AI stated that the company consistently prioritizes user safety but is currently unable to comment on the ongoing litigation. A company spokesperson also emphasized that the user-generated "Characters" on the platform are all fictional personas, and the company has taken a number of measures to inform users of this, including prominently displaying disclaimers in every dialogue interface reminding users that "Characters are not real people and all of their statements should be considered fictional." The spokesperson also stated that the company repeatedly prompts users in the product not to use information on the platform as any form of professional advice, including but not limited to medical, legal, and financial advice.