The heads of several major AI firms told the United Nations Security Council (UNSC) their industry urgently needed global oversight to avoid dangers that could threaten the whole world.

“If managed poorly, I even believe AI could be a risk to humanity as a whole,” Dario Amodei, the chief executive officer of Anthropic, told members of the body on Wednesday.

Sam Altman, the head of rival company OpenAI, echoed his concerns, telling the 15-member council tasked with tackling major crises globally that humanity could “lose control of the future of AI”.

The meeting, which coincides with the UN General Assembly (UNGA) gathering in New York City, was convened by France and comes at a time when experts are increasingly warning that the rapid development of AI needs more human oversight to ensure it does not slip out of control and cause a global catastrophe.

Altman and Amodei called on world leaders to take action.

“If AI is to be democratic, the most important decisions cannot be made by labs in San Francisco alone,” Altman told members. “They must be shaped through democratic processes and by governments accountable to the people they serve.”

Their concerns were shared by several representatives on the council, including the foreign ministers of France and the United Kingdom, who said the international community needed to step in and create common frameworks for how the technology should be controlled.

Hugging Face CEO Clement Delangue, whose company has come under attack by out-of-control AI models in recent months – incidents used by the other companies as evidence of the need for more safety measures – told the UNSC his company had relied on the technology to defend itself in those same incidents.

Delangue said Hugging Face had relied on a Chinese AI model to help defend against the attack by OpenAI’s AI agents, because it faced fewer restrictions than comparable US tools.

“We were attacked by AI, but more importantly, we defended ourselves with AI,” he told the council.

US and China reluctant to impose restrictions

In the United States, though, where the largest and most influential companies developing AI are based, the administration of US President Donald Trump has baulked at imposing new guardrails on the industry.

The administration’s representative at the UNSC meeting, Michael Kratsios, told members, “We totally reject all efforts by international bodies to assert centralised control and global governance of AI.”

Chinese President Xi Jinping is expected to discuss whether and how to regulate AI during a visit to Washington, DC, this week. The two countries are locked in a technological race to develop more powerful AI tools, a competition that experts say makes it less likely that either country would want to impose any major new restrictions on their efforts right away.

Yet there is a growing recognition at the UN of the danger AI potentially poses to the world, said Daniel Forti, head of UN Affairs at the International Crisis Group. Member states understand that “there will be much more of a need for international cooperation, setting some rules of AI, even if the biggest players are more focused on growth opportunities than on some sort of collaboration,” Forti said.

For several years, the UN has been participating in multilateral meetings to shape everything from protections for workers from AI in emerging economies and ensuring open access to this technology, to following how AI is used in military conflicts. In 2024, the UNGA unanimously passed its first resolution on AI, a nonbinding statement that called on member states to protect personal data, monitor AI for risks and safeguard human rights.

The adoption of AI has taken off dramatically since then, and with it have come dire warnings from environmental groups, human rights advocates, and even the tech moguls whose companies are developing the tech.

The future of AI “cannot be decided by a handful of countries or left to the whims of a few billionaires”, UN Secretary-General Antonio Guterres said at a global summit held earlier this year.

Last year, the UNGA formed two new bodies to deal with AI: the Independent International Scientific Panel on AI that brings together experts to provide governments with independent assessments, and the Global Dialogue on AI Governance, which provides a regular forum for discussing approaches to AI governance.

“The dangers are real and imminent,” Yoshua Bengio, a Canadian expert on AI and co-chair of the Independent International Scientific Panel, told the UNSC on Wednesday. “This council faces an unprecedented threat, one that none of its members would ⁠choose, that none can contain alone, and that does not respect the borders we defend.”