7 hours ago
7
Joe Benton calls for greater transparency from AI companies.
Days after Anthropic researcher Jacob Coxon resigned, warning that AI companies were “gambling with our lives” in their race to build self-improving superintelligence, another employee from the company’s safety team has also stepped down, raising concerns over the risks posed by increasingly capable AI systems.Joe Benton, the former manager of Anthropic’s Scalable Oversight team, said he left the company’s safety team two weeks ago because he believes frontier AI companies are moving too quickly towards systems that could become far more capable than humans, without adequately addressing the risks.“I left Anthropic's safety team two weeks ago. Now feels like a good moment to explain why,” Benton wrote in a post on X.
‘We may not survive this’
Benton said AI companies were racing to build machines “much smarter than any human” and warned that humanity may not survive if the development of such systems continues without stronger safeguards.“AI companies are racing to build machines that are much smarter than any human, and we may not survive this,” he wrote. Benton said he wanted to work outside the company to better inform the public about the risks and help navigate the transition responsibly.
In a longer Substack post, Benton said frontier AI companies were racing towards “superintelligence”, which he described as AI systems capable of recursively improving themselves.
He warned that such systems could eventually have drives and desires that diverge from those of human overseers, while developing capabilities that cannot be effectively constrained.“Humanity may not survive this transition,” Benton wrote, arguing that greater preparation was needed to make the development of advanced AI safer.He also warned that AI companies could lose control of their systems or experience an “intelligence explosion” without the public ever knowing.
According to Benton, competition between frontier AI firms creates pressure to spend less on safety because companies fear falling behind rivals.
Benton calls for independent AI safety checks
Benton called for greater transparency from AI companies, including disclosure of progress towards recursive self-improvement and reporting of safety incidents and near-misses.He also said companies should have to meet minimum safety standards and undergo independent assessments to verify that they are meeting them.Benton is set to join METR, an independent organisation that evaluates the safety and risks of AI systems. He said independent checks could help change the incentives facing AI companies and encourage them to address safety concerns.His comments came after Anthropic researcher Jacob Coxon announced his resignation earlier this week.
Why did Jacob Coxon quit Anthropic?
Coxon, who previously worked at OpenAI before joining Anthropic, said he resigned because he believed neither company was acting responsibly in its pursuit of advanced AI.“I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly,” Coxon said in an X post.He accused the companies of “racing straight to self-improving superintelligence” and “gambling with our lives”.Coxon also warned against underestimating the capabilities of increasingly advanced AI systems, saying they could soon be capable of hacking systems, transforming fields rapidly and acquiring real-world power and resources.Coxon’s resignation drew attention from other Anthropic researchers. Samuel Marks, the company’s scalable-oversight lead, said AI developers believe the technology could cause human extinction or similarly catastrophic outcomes, adding that such an event “could happen in the next few years”. He also said that “the more senior the employee, the more concerned they are.”Benton also cited his former manager Evan Hubinger, who has said he believes there is a greater than 10% chance that AI could kill humanity.
Benton said many of his Anthropic colleagues were “terrified” by the risks posed by the systems they were building.Benton’s exit adds to a list of AI safety researchers leaving major labs over concerns about safety and the direction of AI development. At OpenAI, former Superalignment co-lead Jan Leike resigned in May 2024, saying he disagreed with the company’s priorities and that safety had taken a back seat to product development.Benton has called for greater transparency from AI companies, including disclosure of AI capability gains, safety incidents and near-misses, along with independent checks to ensure companies meet minimum safety standards.
