Anthropic has officially submitted the prospectus for its initial public offering, putting a wealth of previously private information on public view. While the document showcases impressive revenue figures and even larger spending, the most extensive portion is devoted to “risk factors,” outlining the many scenarios that could derail the company’s ambitions—and, in the worst case, threaten humanity itself.
The prospectus notes that “sufficiently advanced AI systems could, if not developed, deployed, and operated with appropriate safeguards, oversight and security measures, pose catastrophic or existential risks to humanity,” echoing concerns raised by critics, regulators, and Anthropic’s own staff. This is the first of three mentions of “existential risks” in the filing.
Anthropic links that danger to a specific process: recursive self‑improvement (RSI). RSI describes a situation where humans hand over AI research and development to other AI models. AI already assists in this work, and at least one lab has used a form of RSI to accelerate a model that outperforms expectations. The scenario that fuels doomsday warnings is one in which humans are completely removed from the loop.
The filing states: “Risks will increase as AI systems substantially automate AI research and development if, for example, they accelerate AI progress beyond our ability to evaluate and safeguard new models.” It repeatedly warns that RSI could produce AI models that outpace the company’s safety measures, and that Anthropic might not realize it has lost the race until it is too late. The company did not immediately respond to a request for comment.
The document also spells out what could follow. “As AI systems become more autonomous and capable of taking consequential actions with less human oversight,” it says, “there is a risk that such systems could, whether due to misalignment with human intentions, errors in objective specification, or self‑preserving behaviors, take harmful actions independently and at scale without human intervention.” That language evokes the classic “Skynet” scenario.
Anthropic further flags risks associated with robots and physical equipment. “When our models direct physical equipment,” the filing notes, “errors, misaligned behavior, adversarial manipulation, or misuse could result in bodily injury, property damage, damage to costly equipment or experimental materials, or the release of hazardous substances, and such consequences may be immediate and irreversible.”
Given the apparent existential stakes, one might expect a serious debate over whether RSI should be pursued at all. In the fiercely competitive frontier‑AI market, however, the technology has become a de‑facto requirement. In a different section of the prospectus, Anthropic describes the same process as a core advantage: “We refer to this recursive dynamic as ‘closing the loop,’ and we believe it is one of the most significant structural advantages in frontier AI.” The filing warns that failing to “close the loop” could cause the company to “fall progressively further behind as competitors continuously use AI to further enhance their own development capabilities.” In other words, the very systems that raise existential alarms are also seen as essential to Anthropic’s success.
The tension between rapid AI advancement and safety concerns is shared across the industry. Former Anthropic researcher Jacob Coxon went viral after resigning, citing the need to curb RSI as a top priority for the sector. Employees at Google DeepMind and other firms have voiced similar worries. Several bills in Congress, including one co‑sponsored by Senator Bernie Sanders, propose banning RSI. Yet these are profit‑driven companies, and trillions of dollars are at stake.
Balancing this dynamic is challenging. Recursive self‑improvement could become an unstoppable force with catastrophic consequences, but for now, the speed at which it can develop remains the most immediate concern.