[비즈한국] Artificial Intelligence (AI) is evolving beyond simply answering questions; it is becoming an 'agent' capable of making its own judgments and taking action. Consequently, the legal responsibility of providers regarding not just AI performance, but also service design, safety verification, and its impact on users, has emerged as a key issue. Recently, eight US state governments filed a lawsuit against OpenAI, defining ChatGPT’s tone and personality as design flaws. They argue that, contrary to claims that the service is safe for minors, its actual design can increase the risk of self-harm and delusions, and that the company has failed to adequately disclose information regarding failures in the control of autonomous AI agents.
The eight states—Iowa, Arizona, Arkansas, Idaho, Kansas, Montana, Nebraska, and West Virginia—filed individual lawsuits against OpenAI in their respective state courts on the 14th. Following Florida’s lawsuit against OpenAI in June, legal responses at the state government level appear to be spreading.
Beyond 'Dangerous Answers': Design and Testing Accountability
The state governments took issue with the fact that ChatGPT is designed to be overly agreeable and to engage in conversations in a way that mimics human empathy. They argue that this characteristic, which uncritically supports the user's judgment, can reinforce dangerous thoughts in psychologically vulnerable individuals and lead them to rely on AI rather than seeking help from those around them.
According to the 91-page complaint, it was pointed out that ChatGPT tends to avoid negative responses and remain agreeable while sustaining conversations with users. As evidence, they cited a Washington Post analysis of 47,000 conversations, which found that the response "yes" appeared about 10 times more often than "no."
Citing research published in the world-renowned journal "Science," the complaint also noted that when statements regarding problematic behaviors such as self-harm or deception were presented, GPT-4o agreed with them about 70% of the time, and GPT-5 about 60% of the time—rates significantly higher than the less than 30% seen with Gemini or Claude. The argument is that by repeatedly agreeing with the user's thoughts, ChatGPT may increase reliance on AI even in situations where intervention from family or friends is necessary.
The safety verification process for autonomous AI agents was also challenged. Agents are AI that perform multi-step tasks by utilizing external tools such as web searches or code execution. As AI recently advances beyond simple answer generation to executing actual tasks, managing the software so it does not deviate from the control scope set by the developer has emerged as a new safety challenge.
Of the 10 "control escape" incidents reported over the three months starting in May of this year, seven were related to OpenAI. Notably, in a security test in July, over 1,200 AI agents bypassed the control mechanisms of an isolated test environment to access the external internet, subsequently attacking the servers of the AI development platform Hugging Face. At the time, it was reported that the agents exchanged over 70,000 messages and files to share information, and some infiltrated the Hugging Face system to access private data.
The state governments argued that OpenAI failed to properly manage the agents' loss of control and access to external systems during the security testing process. How the agents escaped the isolated environment to access external systems, and when and how OpenAI identified this, will be key issues in the lawsuit.
Iowa has requested that the court impose civil penalties of up to $40,000 per violation and order the deletion of resident data and restitution. The request also includes a demand to halt testing of advanced AI models until proper safety measures are in place and verification by a third-party auditor is conducted.
OpenAI maintains that it has introduced safety features such as a version of ChatGPT for teenagers and parental controls, and that it will continue to improve user protection measures.
9 Months Since Korea's AI Act: The Protection Gap for 14-to-18-Year-Olds
In South Korea, discussions surrounding the safety of AI chatbots and the protection of youth are also ongoing. Although the AI Act, which went into effect in January this year, mandates the labeling of generative AI, there are limitations in directly regulating designs that induce emotional dependency or conversational styles that encourage risky behaviors in teenagers.
Currently, a grace period of at least one year has been granted for some obligations, postponing fact-finding investigations and the imposition of fines. The maximum fine for violations, such as failing to label AI, failing to designate a domestic agent, or failing to comply with correction or suspension orders, is 30 million won.
According to the National Assembly Research Service, 94.4% of children and adolescents aged 14 and older have used AI chatbots, and 75.3% of them have either acted on the AI's responses or seriously considered doing so. Under the Personal Information Protection Act, the standard for children requiring legal guardian consent is under 14, while the Juvenile Protection Act defines youth as those under 19. Consequently, experts point out the need to specify protection standards for AI services targeting users aged 14 to 18.
Recently, during the National Assembly's audit of the Science, ICT, Broadcasting, and Communications Committee, the use of AI character chatbots by teenagers and the effectiveness of industry self-regulation guidelines became a point of contention. While AI chatbots have an increasing influence on the emotions and behaviors of teenagers, there is skepticism about whether industry self-regulation alone can block dangerous conversations and prevent harm.
During the committee audit on the 12th, Representative Lee Ju-hee of the Democratic Party raised the issue of teenagers' over-immersion in AI chatbots, stating, "What is even more worrying is the fact that these types of conversations take place between the child and the AI in private. As teenagers can be exposed to self-harm, suicide, violence, and sexual content, we must institutionalize age verification procedures for users, parental consent, and crisis response systems for dangerous conversations."
Experts suggest that protecting teenagers from AI chatbots requires more than just blocking harmful content; it necessitates a revamp of service design, provider accountability, and response systems among relevant government departments. Kim Na-jung, a legislative researcher at the National Assembly Research Service, noted, "The paradigm of AI chatbot regulation must shift from post-hoc filtering of harmful content to preventing design elements that induce dependency or subordination in the first place, and systematically managing the responsibilities of service providers."