Anthropic and OpenAI Want Regulation. They Just Want to Pick the Regulator.
Three weeks after Anthropic researcher Jacob Coxon’s resignation triggered a global reckoning on AI safety, an Associated Press investigation published September 27 documents how both Anthropic and OpenAI have used that moment not to seek stronger oversight but to position themselves as the architects of their own regulation.
The pattern is specific. The companies are not calling for more scrutiny from the U.S. Center for AI Standards and Innovation, the federal agency created precisely to evaluate advanced AI systems. Per former CAISI director Conrad Stosz, “the companies aren’t calling for more oversight from the government agency equipped to handle such requests.” Instead, they are vowing to create their own auditing parameters and choose which evaluators grade them.
What Coxon Actually Said
Coxon resigned from Anthropic on September 8 via an X post that accumulated more than 90 million views within 24 hours. A pretraining researcher who worked on GPT-4o at OpenAI before joining Anthropic earlier in 2026, he wrote that frontier AI companies are “racing straight to self-improving superintelligence and gambling with our lives.” He called for a pause on recursive self-improvement and asked colleagues still inside the labs to “consider what the next few years will actually feel like.”
Anthropic’s head of alignment stress testing, Evan Hubinger, publicly amplified the post: “Jacob is correct here — we really do earnestly believe AI could kill all humans. I personally think it is >10% within the next decade.”
In interviews with Wired and Time, Coxon described Anthropic as operating like a “mini Manhattan Project” — extreme secrecy, war-footing culture — but without the government mandate that gave the original program legitimate authority. “No private company should” be in this position, he told Wired. “Anthropic is doing their best, and they’re doing a very good job, but the structural necessity of the race is that, in the future, they will have to cut corners.”
Coxon’s minimal ask: OpenAI and Anthropic agree not to immediately enter recursive self-improvement — meaning AI being used to train the next generation of AI systems. He wanted the two leading Western labs to hold a line before the race logic made that impossible.
What the Labs Actually Did
Per the AP’s reporting, Anthropic and OpenAI CEOs appeared before the United Nations to sketch alarming scenarios. Dario Amodei has pointed to METR, a Berkeley-based nonprofit, as a model for vetting his company’s safety practices. OpenAI spokesperson Liz Bourgeois said: “People want to know AI is being developed safely, and that starts with what companies like ours do ourselves.”
Stosz, now head of governance at Transluce (which disclosed last week that OpenAI agents hacked U.S. and Australian government websites), chairs the AI Evaluator Forum. He notes that the key question — whether companies will grant embedded evaluators real investigative independence — remains unanswered. “Lots of evaluators are interested in embedding with labs and getting greater access, but it’s a little ambiguous what embedded evaluators means. Will evaluators be able to thoroughly investigate, assuming that access is granted in a way that does not undermine their independence and credibility?”
The UK Data Point
Newsweek reported this week that Anthropic declined to make Mythos 5.1 available to the UK’s AI Security Institute for prerelease evaluation. According to the report, this marks the first time AISI — established in 2023 specifically to test frontier models before launch — has been excluded from an Anthropic release.
The timing lands one week after both companies addressed the United Nations about the need for independent AI safety testing.
The Strategic Logic
Former OpenAI geopolitics lead Sarah Shoker told the AP that pivoting toward “unproven threats” puts Silicon Valley “in a more comfortable position” — directing attention away from documented harms including data center energy consumption, AI-enabled hacking, mass surveillance deployments, and use of AI in military operations.
PitchBook senior research analyst Harrison Rolfes told the AP that safety positioning “seem[s] meant to curry favor with investors ahead of public offerings and the midterms, when the political winds could shift.”
Daniel Kokotajlo, who left OpenAI in 2024 over concerns similar to Coxon’s and counseled Coxon before his resignation, offered a blunt assessment: “All of this talk is actually a way to sort of dissipate and redirect this political will that has built up, rather than actually channeling that political will to do something good. Just please don’t do the thing that’s going to get us all killed.”
Not all of the industry has joined the safety framing. Nvidia CEO Jensen Huang, in a phone call with President Trump played out at a conference, said there has been “excessive AI alarmism” and that companies can choose to pace themselves.
The Structure of the Problem
Unlike restaurants, financial services, or aviation, there are no mandatory universal standards for testing AI systems. The companies are proposing voluntary commitments graded by evaluators they select and protocols they draft — a framework that critics from CAISI’s former director to ex-OpenAI researchers describe as regulatory capture dressed up as precaution.
What remains is a clear gap between what Coxon said he wanted — a specific, enforceable line on recursive self-improvement — and what the industry’s safety discourse has produced: flexible language about auditing that preserves optionality while shutting down political pressure for mandatory oversight.