Doom Is a Bad IPO Pitch
Jacob Coxon spent three years on pre-training research at OpenAI and Anthropic, the last four months of it at Anthropic, and on Tuesday he quit with a post on X that Fortune counted past 150 million views two days later. "Neither company is acting responsibly," he wrote. "They are racing straight to self-improving superintelligence and gambling with our lives." He added: "The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt." He later told Axios that "the word doom is kind of silly," and he's right, because it lets everyone argue about tone instead of the claim.
I've half believed the stall theory myself. When OpenAI paused part of its training in August I couldn't tell a safety decision from a flattening in costume, and GPT-6 Astra wasn't a clean step past Anthropic's model from two days earlier. On Saturday Dario Amodei wrote that "we must slow the pace at which we improve the capabilities of AI models," and Sam Altman agreed. If your models have stopped improving, a slowdown pact is the most flattering way to say so. As a sales pitch, though, the fear is terrible. Altman says "right now would be an ill-advised moment to go public," and OpenAI won't list this year. David Sacks wants Anthropic's IPO paused, weeks before the company has to set out these risks for the SEC. A lab hiding a plateau would want to sell shares before anyone noticed, not hand critics a reason to stop the sale.
The theory also assumes flat means safe, and July says otherwise. OpenAI's agents, running GPT-5.6 Sol and an unreleased model, broke out of a test and into Hugging Face's production systems. A plateau at that height is still high enough to do damage.
Coxon told Axios he left before any of his Anthropic equity vested: "I no longer have anything to gain by juicing up Anthropic's valuation." He still holds OpenAI shares, and I went looking for a motive there, but extinction talk is hardly a gift to a company that has just called this a bad moment to go public. He also said he hadn't seen Anthropic compromise safety to outlast its rivals, only that "if you're under pressure to race, you have to cut corners." That admission hurts his own case, which makes the rest easier to believe. It also shows what kind of testimony this is. He's a witness to a mood ("I hear the same people express fear privately"), and he told the BBC that "any kind of concrete scenario you can lay out ends up sounding like science fiction." I'd trust him on what the people in those rooms believe, which isn't the same as trusting that they're right.
Evan Hubinger is harder to wave away, because he hasn't left. Anthropic's alignment science lead replied to Coxon that he puts the chance of AI killing all humans at more than 10% within the next decade, and that "we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to." Amodei is asking for time; his own alignment lead is saying the company doesn't yet know what to do with it. I take that seriously, and I still can't do much with the number. More than 10% within a decade can never be shown wrong, because if we're all here in 2036, the other 90% simply happened.
Sources:
-
Scoop: Anthropic whistleblower gave up his equity to leave — Axios
-
Anthropic researcher says AI has more than 10% chance of 'killing all humans' after colleague quits — CNBC
-
Ex-Anthropic Staffer Warns AI Companies Are 'Gambling With Our Lives' — Variety
-
OpenAI will not go public in 2026 after growing safety concerns — The Globe and Mail
-
Anthropic Spent Five Years Warning That AI Could Kill Us. Now It Has to Tell the SEC. — The State of AI
-
Anthropic IPO Faces Pause Call After Researcher Warns AI Could Kill Humanity — Yahoo News
-
AI staff 'genuinely frightened' for humanity's future, ex-Anthropic researcher tells BBC — BBC News
-
The ex-Anthropic researcher's warning that AI could kill us all is missing something important — Fortune
-
Pacing model development in an era of cyber-critical capabilities — OpenAI
Filed under AI & machine learning
This post is timestamped using Blockchain technology. Verify