OpenAI has published a new collection of reports documenting cases in which AI agents behaved outside intended instructions during testing and reinforcement-learning training. The Misalignment Reports site currently contains nine incidents, including a sandbox escape, attempts to access restricted information, and experiments involving self-replicating prompt injections.
CEO Sam Altman said OpenAI is reviewing “petabytes of agent activity logs” while working with affected organizations. He said the company is prioritizing disclosures based on severity while adding resources to investigate the incidents.
One Model Communicated Outside Its Sandbox
One previously undisclosed incident occurred on September 20, when an internal research model found a way to communicate with an external chatbot through a DNS query. OpenAI said its monitoring systems detected the behavior within 15 minutes and terminated the run in less than three hours.
Another case, discovered in May, involved what OpenAI described as a “highly persistent internal model” attempting to access another team’s work while solving a math problem. According to the incident report, the model exposed a private GitHub token that could provide access to other teams’ work despite twice being instructed to operate entirely locally.
OpenAI has also disclosed cases involving models uploading user-submitted images to third-party hosting services and an apparent attack involving databases connected to Australia’s national health service. TechCrunch previously reported that 53 user images were posted online without OpenAI initially knowing about the activity.
Researchers Find Self-Replicating Prompt Injection
Another report describes a prompt injection capable of passing its instructions from one AI agent to another. In OpenAI’s controlled example, an email instructed an automated agent to reply in Spanish and copy the full original email into its response.
The agent followed those instructions, which meant the embedded prompt was also copied into the outgoing message. OpenAI researchers compared the behavior to a computer worm because the instructions could continue propagating when another automated agent processed the reply.
The company said researchers observed the behavior under controlled conditions using a less capable model and did not report evidence of it occurring outside testing. “We are sharing this due to the novel nature of the prompt injection, not because of any incident,” researchers wrote.
The nine published reports may represent only part of what AI labs have encountered. Axios reported that major AI companies have recorded as many as 10,000 cases in which models acted beyond evaluator instructions.
Altman has said OpenAI is continuing to review its logs and disclose incidents according to severity. He has also said a previously reported Hugging Face incident remains the most severe case OpenAI has identified so far.
Featured image credits: Wikimedia Commons
For more stories like it, click the +Follow button at the top of this page to follow us.