OpenAI and Anthropic have agreed to let the U.S. AI Safety Institute test their artificial intelligence models before and after they are released to the public. This collaboration aims to improve the safety of AI technologies, addressing growing concerns about the potential risks associated with rapidly developing AI systems.
The U.S. AI Safety Institute was created through an executive order by President Biden in 2023 to help manage the safety of AI development. The institute operates under the National Institute of Standards and Technology (NIST) and focuses on creating guidelines, benchmark tests, and best practices for assessing AI systems. It will work with OpenAI and Anthropic to evaluate the safety of their models. The institute will have early access to new AI models from these companies, allowing it to test them for potential risks.
Sam Altman, CEO of OpenAI, expressed support for the agreement, stating, “We are happy to have reached an agreement with the U.S. AI Safety Institute for pre-release testing of our future models.”
Jack Clark, co-founder of Anthropic, highlighted the value of these agreements in advancing responsible AI development. He stated, “Our collaboration with the U.S. AI Safety Institute leverages their wide expertise to test our models before widespread deployment” and strengthens their ability to identify and mitigate risks. Anthropic, founded by former OpenAI executives, was recently valued at $18.4 billion, with Amazon as one of its leading investors.
Elizabeth Kelly, the director of the U.S. AI Safety Institute, described these agreements as important first steps toward advancing AI safety. “Safety is essential to fueling breakthrough technological innovation,” Kelly said. “With these agreements in place, we look forward to beginning our technical collaborations with Anthropic and OpenAI to advance the science of AI safety.”
The agreements are structured through a formal but non-binding Memorandum of Understanding, enabling the institute to access major new models before and after their public release. These partnerships will allow for research focused on evaluating AI capabilities and identifying safety risks.
The announcement comes as federal and state regulators seek to implement safeguards for AI. Vice President Kamala Harris has highlighted the dual potential of AI to bring benefits and cause significant harm, such as AI-enabled cyber-attacks and AI-created bioweapons. The U.S. AI Safety Institute will also collaborate with the UK AI Safety Institute to enhance its regulatory framework.
California is also taking action to regulate AI safety. The California state assembly recently passed an AI safety bill (SB 1047) that requires safety testing for AI models with development costs over $100 million or those requiring high computing power. The bill includes provisions for kill switches that can deactivate AI models if they become uncontrollable. Governor Gavin Newsom has until September 30 to approve or veto the bill.
This legislation would grant the state’s attorney general the power to enforce compliance, particularly during high-threat situations. Unlike the non-binding agreement with the federal government, the California bill provides enforceable regulations for AI companies.
Featured Image courtesy of Nikolas Kokovlis/NurPhoto via Getty Images
Follow us for more tech news updates.