"It is clear that AI will have an impact on every area of our lives," Guterres said at the first UN Security Council meeting on the topic.

"Generative AI has enormous potential for good and evil at scale," he added, noting that while it could help end poverty or cure cancer, the technology could also have "very serious consequences for global peace and security."

British Foreign Secretary James Cleverly, whose country currently holds the rotating Security Council presidency, said AI "will affect the work of this council. It could enhance or disrupt global strategic stability."

"It challenges our fundamental assumptions about defense and deterrence. It poses moral questions about accountability for lethal decisions on the battlefield," said Cleverly, whose government will host an AI summit later this year.

Guterres asked member states to put in place a legally binding pact to "prohibit lethal autonomous weapons systems" by the end of 2026.

While AI can be used to identify patterns of violence or to monitor ceasefires, Guterres said, he warned that its use by terrorists or governments with ill intent could cause "horrific levels of death and destruction."

He also warned that malfunctioning AI could wreak havoc, especially if the technology is used in connection with nuclear weapons systems or biotechnology.

"I urge agreement on the general principle that human agency and control are essential for nuclear weapons and should never be withdrawn," he said.

He called for a working group to develop options for global AI governance by year's end.

And he reiterated his support for a United Nations entity to support efforts to govern the use of AI, modeled after the International Atomic Energy Agency or the International Civil Aviation Organization.