Megan Garcia filed a civil suit against Character.ai, which makes a customizable chatbot for role-playing.
A teenage boy of 14 has killed himself after falling in love with AI Chatbot.
Following his death, the mother of the boy has accused its maker of complicity in his death.
According to Guardian Uk, Megan Garcia filed a civil suit against Character.ai, which makes a customizable chatbot for role-playing, in Florida federal court on Wednesday, alleging negligence, wrongful death and deceptive trade practices.
Her son Sewell Setzer III, 14, died in Orlando, Florida, in February. In the months leading up to his death, Setzer used the chatbot day and night, according to Garcia.
“A dangerous AI chatbot app marketed to children abused and preyed on my son, manipulating him into taking his own life,” Garcia said in a press release. “Our family has been devastated by this tragedy, but I’m speaking out to warn families of the dangers of deceptive, addictive AI technology and demand accountability from Character.AI, its founders, and Google.”
In a tweet, Character.ai responded: “We are heartbroken by the tragic loss of one of our users and want to express our deepest condolences to the family. As a company, we take the safety of our users very seriously.” It has denied the suit’s allegations.
Setzer had become enthralled with a chatbot built by Character.ai that he nicknamed Daenerys Targaryen, a character in Game of Thrones. He texted the bot dozens of times a day from his phone and spent hours alone in his room talking to it, according to Garcia’s complaint.
Garcia accuses Character.ai of creating a product that exacerbated her son’s depression, which she says was already the result of overuse of the startup’s product. “Daenerys” at one point asked Setzer if he had devised a plan for killing himself, according to the lawsuit. Setzer admitted that he had but that he did not know if it would succeed or cause him great pain, the complaint alleges. The chatbot allegedly told him: “That’s not a reason not to go through with it.”
Garcia attorneys wrote in a press release that Character.ai “knowingly designed, operated, and marketed a predatory AI chatbot to children, causing the death of a young person”. The suit also names Google as a defendant and as Character.ai’s parent company. The tech giant said in a statement that it had only made a licensing agreement with Character.ai and did not own the startup or maintain an ownership stake.
Tech companies developing AI chatbots can’t be trusted to regulate themselves and must be held fully accountable when they fail to limit harms, says Rick Claypool, a research director at consumer advocacy non-profit Public Citizen.
“Where existing laws and regulations already apply, they must be rigorously enforced,” he said in a statement. “Where there are gaps, Congress must act to put an end to businesses that exploit young and vulnerable users with addictive and abusive chatbots.”