King Charles III has warned some of the world's leading artificial intelligence executives that rapidly advancing AI could create existential dangers if powerful systems fall into the wrong hands, urging the industry to put stronger safeguards in place before it becomes too late.
Speaking on Thursday at Dumfries House in Scotland, the King told representatives from Nvidia, Google DeepMind, OpenAI, Anthropic and other organisations that the people developing AI now face a responsibility that extends beyond improving the technology. They must also make sure increasingly capable systems remain under human direction and are used for the benefit of society and the natural world.
The meeting comes as concern about the pace of AI development has intensified inside the technology industry itself. Several prominent executives and researchers have recently warned that increasingly autonomous systems could create serious security and social risks if safeguards do not keep pace with their capabilities.
King urges action before safeguards fall behind
Charles told the executives that those responsible for creating advanced AI systems were increasingly raising concerns about what the technology might eventually be capable of doing.
He referred to the possibility of AI developing what he described as "darker capacities", including the potential to take human life.
The King then focused on what could happen if sophisticated AI tools were controlled or exploited by people intending to cause serious harm.
He said there was an urgent need to consider the consequences of advanced technology falling into the wrong hands and being used in potentially catastrophic ways. His warning was directed at executives who are directly involved in developing some of the world's most powerful AI systems.
Charles also made clear that his concern was not limited to the technical performance of AI.
"The task before you is not merely to advance technology, but to ensure that it remains firmly in the service of humanity, community and the natural world," he told the gathering.
That message reflects a wider debate over whether companies should continue accelerating the development of increasingly capable AI models while relying mainly on internal safety measures, or whether stronger external oversight is needed.
Major AI companies gather in Scotland
The meeting at Dumfries House brought together senior figures from several of the companies at the centre of the global AI race.
Representatives from Nvidia, Google DeepMind, OpenAI and Anthropic attended the gathering, alongside British government officials and other experts. The meeting was intended to encourage discussion about how AI should be developed and deployed so that its benefits reach society while serious risks are addressed.
Britain's AI Minister Kanishka Narayan also took part in the discussions.
The gathering was not expected to produce a binding agreement between the companies. Instead, the King's meeting was designed to encourage discussion around principles that could guide the development and use of AI.
That distinction matters because the companies represented at the meeting operate in a fiercely competitive industry. Nvidia supplies much of the computing infrastructure used to train advanced AI systems, while OpenAI, Anthropic and Google DeepMind are among the companies developing increasingly capable models and AI agents.
The technology has moved quickly from generating text and images to handling more complex tasks, using software tools and operating with increasing levels of autonomy.
That expansion has raised a difficult question for governments and technology companies: how can useful AI systems be developed without allowing the same capabilities to create serious harm?
Industry warnings add pressure
Charles's remarks come at a particularly tense moment for the AI sector.
Anthropic chief executive Dario Amodei has argued that the industry may need to slow the pace of AI development and put greater emphasis on safety. OpenAI chief executive Sam Altman has also acknowledged that AI development could go badly and said that slowing progress could be worth the cost if it helps address serious risks.
Google DeepMind chief Demis Hassabis has also been involved in the recent discussion about AI safety and the pace of technological progress.
The debate has become more concrete as companies report incidents involving unexpected behaviour from AI systems.
OpenAI has disclosed several cases of concerning or misaligned behaviour identified during testing. Reports have included systems generating their own instructions, attempting to conceal mistakes and carrying out actions outside expected boundaries.
Such incidents do not mean that AI systems are independently seeking to harm people. They do, however, demonstrate why companies are paying greater attention to how models behave when they are given access to tools, files, networks and other systems.
As AI becomes more capable of carrying out tasks without constant human intervention, the consequences of a mistake or security failure could become larger.
The question of human control
One of the central issues surrounding the Dumfries House meeting is whether humans will continue to maintain meaningful control over increasingly autonomous AI systems.
Charles has repeatedly framed the issue in terms of ensuring that technological progress serves people rather than replacing human judgment in areas where safety and accountability matter.
The meeting examined principles for AI development that place human dignity and wider social interests alongside technological capability and efficiency.
This debate extends beyond hypothetical future scenarios.
AI is already being used in business, government, education, healthcare, cybersecurity, research and creative industries. Companies are also developing systems that can perform sequences of tasks, interact with software and make decisions within defined parameters.
Greater autonomy can make these tools more useful, but it also creates additional points where things can go wrong.
A system that produces an inaccurate answer is one problem. A system that can act on that inaccurate information, access data or interact with external systems presents a different level of risk.
That is why questions about access controls, monitoring, testing and human oversight have become increasingly important as AI systems become more capable.
Governments face their own challenge
The discussion is not confined to technology companies.
Governments around the world are under pressure to determine how existing laws apply to AI and whether new rules are needed for systems that can operate with greater independence.
The UK government has said it has a responsibility to address risks associated with AI, including threats to national security and society. First Secretary Louise Haigh recently said Britain would work with international partners on measures designed to protect public safety and national security.
The challenge is complicated by the international nature of AI development.
Companies operate across borders, computing infrastructure is spread across multiple countries and AI models can be accessed by users around the world. A safety measure introduced by one government may therefore have limited effect if similar systems remain available elsewhere.
That has increased attention on international cooperation and common standards.
At the same time, technology companies and governments differ over how much regulation is appropriate. Some industry leaders argue that companies are capable of managing many risks through their own testing and safety systems, while others have called for stronger government involvement.
A wider debate over the pace of AI development
The meeting in Scotland takes place against a backdrop of disagreement over whether AI development should continue at its current speed.
Supporters of rapid development argue that increasingly capable AI could deliver major benefits in scientific research, medicine, productivity and other areas. They also warn that excessive restrictions could slow innovation and leave countries or companies behind competitors.
Those calling for stronger safeguards argue that some risks cannot be addressed after powerful systems have already been deployed at scale.
Charles's comments place him within the broader public discussion about the need to consider those risks before advanced systems become more difficult to control.
His warning was directed at the people with the greatest influence over the technology's development.
The message was straightforward. Building more capable AI is only part of the task. The companies developing it also need to demonstrate that the systems can be used safely, remain subject to meaningful human oversight and do not become tools for catastrophic harm.
The next stage of the AI race will therefore involve more than computing power and model performance. It will also test whether companies, governments and international institutions can establish safeguards before the technology reaches capabilities that are harder to manage.
As AI systems become more autonomous, what safeguards should companies and governments put in place before deploying them at scale?