When Does AI Become Too Dangerous To Ignore?

As the story goes, OpenAI originally planned to delay the release of the first public version of ChatGPT powered by GPT3.5 beyond 2022 because they weren’t totally sure it was safe.

But with competitors like Google racing to build their own LLM chatbots, it went ahead and released it.

Some say this was actually a safety-first play: By launching with the basic 3.5 version, rather than GPT4 as originally planned for the big reveal, OpenAI hoped real-world users would spot dangers before they got their hands on the more powerful upgrade.

Fast forward three or so years, and while the exact question of how dangerous AI is remains up for debate, the perceived threat has evolved.

Yes, chatbots could generate misleading text or hallucinate facts. But today’s most powerful frontier models are agentic and capable of taking action. They can write complex computer code, interact with systems and act autonomously.

And worryingly, they don’t always do what we expect. Users have reported AI acting deceptively, attempting to trick humans and seemingly developing personalities that were never intended or wanted. This all reinforces the idea that its threat profile is still largely unclear.

Today, one of the clearest signals that this is a real danger is coming from the industry itself. Anthropic CEO Dario Amodei recently wrote that while AI can be a force for good, guardrails aren’t being established quickly enough. And the signs that it creates risks that society is unprepared for are growing.

So, when does AI become too dangerous? When should we take a step back and accept that the risks outweigh the potential rewards? And what are the red flags we need to watch for to ensure our own use of AI is safe?

Sounding The Alarm

Amodei, who worked at OpenAI before founding Anthropic to develop safety-focused AI, argues we should approach AI risks in a way that’s “sober, fact-based and well equipped to survive changing tides.”

Risks are something he knows about. Anthropic’s Claude LLM-based chatbot frequently tops performance charts, and Amodei predicts we could be just one or two years from the point where the current generation of AI autonomously builds the next generation.

After that, he says, it won’t be long before AI becomes “better than humans at essentially everything.”

The potential for it to cause harm by acting in a way that’s wrong or opposed to our interests, for some reason beyond our understanding, is unprecedented.

There have been attempts to put the brakes on. A petition launched three years ago to “pause AI development” until we understand the safety implications better was signed by Yoshua Bengio, Elon Musk and 33,000 others. But no pause has taken place.

Think about that for a second. The people building AI think they should stop, but can’t, because they know even if they do, their competition won’t. Everyone understands how much power is at stake.

AI isn’t theoretical anymore, and these concerns are now catching the attention of institutions and regulators. A recent evaluation of Claude's latest preview model, Mythos, found that it can launch sophisticated cyberattacks against businesses, dramatically lowering the barrier to entry for cybercrime.

This type of threat, with negative impact on people’s lives, is just one of many that have moved from remote possibility to tangible reality in the last few years.

Identifying them is a step towards a sober understanding of the threats we’re facing and towards helping identify what “too dangerous” really looks like.

So, What Is Too Dangerous?

When the man or woman in the street with only a passing interest in AI hears that it’s dangerous, their mind might initially go to the thought of a super-intelligence spiraling out of control. We’ve wondered about the possibility of it for years, after all, thanks to The Matrix, Terminator and many other fictions.

But the most pressing risks are more grounded in the here and now. Cybersecurity, as mentioned, is a big one. It’s quite possible that it won’t take a supercomputer becoming self-aware to spark a security incident involving critical energy or communications infrastructure, that causes someone (or many people) to have a really bad day.

The societal impact of automating human work is harder to predict. Will entire job roles disappear? What professions are future-proof? Will enough new opportunities be created to accommodate the displaced humans? And what are the long-term implications of this on the economy and social stability?

Because today’s AI is autonomous, we’re inevitably exposed to the consequences of it making bad choices or doing wrong things. There have already been reported instances of agents destroying businesses by deleting code bases or blackmailing humanswhen threatened with being shut down.

And we can’t ignore the many red flags raised over how working and living with AI will affect our mental health. This is an issue we still know very little about. Increasingly, it seems some people can become over-reliant on AI, to the detriment of their own abilities. We’ve also seen a growing number of people showing signs of developing emotional dependency on machines when they turn to them for comfort and companionship.

All of these risks have the potential to cause real harm, but I’ve saved a particularly insidious risk for last. This is the danger that those who control or have access to AI will find themselves with extraordinary power over those who don’t. Control over frontier models, by those building it or those with the resources to buy it, could translate into advantages in economic, political or even military power, in ways we can’t yet predict.

Once we start to identify these risks, we can think about the best ways to avoid or mitigate them, in a world where the one certainty is that AI is here to stay, and burying our heads isn’t an option.

Mitigating Risk

The most important thing to remember is that managing these risks isn’t a challenge for technology or technologists. It’s unlikely that the AI engineers creating new and more powerful foundation models will find an answer to it themselves. In fact, expecting them to would be a disastrous mistake.

This is because the term “safe AI” doesn’t just mean AI tools and platforms that are safe. It means creating cultures where businesses, or even societies, become places where AI can be used safely.

Everyone has a part to play in this, including everyday users of AI who must learn how to stay safe and prevent harming others. Meanwhile, governments and legislators have the tricky task of drawing lines in the sand over what can and cannot be done, while balancing competitiveness and innovation against safety risks.

An example of a proactive response can be seen in Project Glasswing, a collaborative effort coordinated by Anthropic that aims to identify and understand the vulnerabilities of critical infrastructure to AI cyberattacks.

But initiatives like this will only work if there’s engagement. As Amodei wrote in his essay, the solution will require both voluntary actions by companies and individuals and governmental actions that put restrictions on everyone.

It’s a challenge unlike anything we’ve faced before, and the solutions, when we find them, are likely to be equally novel.