“Artificial intelligence could kill off humanity within the decade.”
Many of us will have read variants of that sentence in the last few days in a wide range of news reports following the resignation of Jacob Coxon, a researcher at Anthropic. Coxon left Anthropic accusing the company and OpenAI, a previous employer, of acting irresponsibly and “gambling with our lives” by racing straight to self-improving superintelligence.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” he claimed. “This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear privately. No other human activity poses this level of danger.”
His predictions were supported by two other current employees at Anthropic, Evan Hubinger and Samuel Marks.
Hubinger, who works in Anthropic’s alignment division, wrote: “We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”
For good measure, Anthropic’s “scalable oversight lead”, Marks added: “AI developers believe their technology could cause human extinction (or similarly bad outcomes). This could happen in the next few years. In general, the more senior the employee, the more concerned they are.”
Anthropic responded to the dire warnings by stressing it was building “models with some of the strongest safeguards in the industry”. This may not have been quite as reassuring as the company intended after it emerged Anthropic had refused to submit its latest model Mythos 5.1, to the UK’s AI Security Institute for pre-release testing. And it may have sounded even more hollow in the wake of the publicised instances of AIs going rogue over the summer, such as when OpenAI agents launched an attack on Hugging Face.
Those looking for encouragement among all the negative stories surrounding AI could try and draw some comfort from the fact that companies like Anthropic and OpenAI seem to be aware of the potential dangers and would like to be seen to be doing something about it. Anthropic, for example, argues that the industry needs to adopt a” lawful, verifiable way to work together to pace how we release powerful models”. OpenAI’s chief scientist, Jakub Pachocki, believes that “international coordination on future AI development needs to become a top priority for governments around the world”.
The difficulty here, however, is that the AI companies calling for greater government responsibility are, often, also the companies arguing for a huge expansion in the construction of data centres to support AI that would occur at the expense of ordinary people in terms of access to vital resources, such as electricity and water. They are also the ones seeking to create an aura of “forced inevitability” around AI, often at the expense of the safeguards which would provide for slower, more considered and regulated AI development.
It’s only a couple of weeks, let’s not forget, since Mark Zuckerberg, Elon Musk, Sam Altman and Jensen Huang took part in a gathering of technology ministers from G20 members in North Carolina where the US government sought to encourage the countries to take a ‘hands off’ approach to AI regulation under the ‘Carolina Principles’.
As Reuters noted: “The US goals largely align with the thinking of the world’s biggest AI companies, nearly all of which are American. They want less regulation around the world for their rapidly growing businesses, or to shape the rules as they are written. Federal government requirements could hurt the industry’s profits if they slow the release of new models or prompt the companies to change how their products perform to address security concerns.”
You could be forgiven for wondering if companies like OpenAI are pursuing two contradictory aims by seeking “international coordination on future AI development” while also being part of the US government’s lobbying for minimal oversight of AI development, but it doesn’t take superintelligence to understand how they can be reconciled.
You can have “international coordination” based on an agreement by governments for minimal oversight of future AI development, for example.
What I think we shouldn’t lose sight of here is that it will be decisions taken by humans, not AI, that will create the conditions for the apocalyptic scenario put forward by Coxon. If AI does end up killing off humanity, don’t blame AI. It won’t be AI that causes it. The culprits will be the people promoting, developing and imposing AI upon us all in their irresponsible headlong rush to win the race. You might argue they’ve already started with themselves because they don’t see the consequences for humanity clearly enough to stop it. It is they who will be the ones that kill all humanity.
Ltd
