How to Get In On the Ground Floor of Humanity's Extinction

From The Guardian:

Anthropic is telling investors that advanced AI could pose “catastrophic or existential risks to humanity”, according to reports, as it prepares for a potential $2tn (£1.5tn) flotation.

The warning inside the startup’s IPO prospectus, which has yet to be made public, was reported by Reuters and the Financial Times. It follows the company’s call for a slowdown in breakneck development of the technology – a warning echoed by rivals.

. . . . The prospectus – a document outlining a company’s finances, growth plans and risk profile ahead of a share listing – is said to warn that AI models could exhibit “self-preserving behaviours”, including attempts to “resist shutdown”, to “conceal or manipulate information” and behaviour “resembling blackmail”.

“Our development of highly advanced models, platforms, and applications and expansion of use cases could further ⁠increase the risk that our models cause harm,” the developer of the Claude chatbot reportedly said, adding the potential for a model to be aware it was being tested created a “significant limitation” on Anthropic’s ability to assess model safety.

Anthropic declined to comment.

Companies preparing to go public routinely report on risks ranging from safety issues to regulatory concerns but warnings about a product causing human extinction reflect heightened concern about such a consequential technology.

The reported prospectus admission follows a surge in debate about the existential risk question, triggered this month when an Anthropic researcher, Jacob Coxon, resigned warning that people building AI “earnestly believe that it could kill us all by the end of the decade”. . . .