Why No One Really Understands AI, And Why That Should Worry Us

AI may be the most powerful technology most of us have ever used without really understanding how it works.

That should make us pause. AI is already being built into healthcare, government services, financial decisions, business operations and tools millions of people use every day. It can write, code, analyze, summarize and recommend at astonishing speed. Yet behind the impressive results sits an uncomfortable truth: even the people building the most advanced AI systems cannot always explain why they behave the way they do.

One of the clearest examples is hallucination, the tendency of AI systems to produce confident answers that are simply wrong. This is not a minor inconvenience when AI is being used to support medical judgments, compliance processes, hiring decisions or financial advice. OpenAI itself has acknowledged that hallucinations remain a difficult and unresolved problem.

Then there is the wider “black box” problem. AI systems can produce useful outputs, but the path from input to answer is often opaque. We can see what goes in and what comes out, but the reasoning in between can be hard to inspect, explain or trust.

For businesses, that creates a very practical question. If AI has to be checked, verified and monitored before its outputs can be trusted, how much productivity are we really gaining and where does the risk become too high?

The Gamble

OpenAI’s widely discussed paper Why Language Models Hallucinate offers the insight that these mistakes and hallucinations could simply be a by-product of the probabilistic way LLMs operate.

In other words, no more a “bug” than a human’s propensity to make mistakes, guess wrongly or jump to illogical conclusions. This makes it very hard to engineer around the problem.

This creates a challenge for businesses. The productivity gains promised by AI are based on the assumption that it can do work that we’d otherwise have to do ourselves. But if everything created by AI has to be checked and verified before it can be trusted, how efficient is it really?

Of course, this will vary by task. A certain error rate might be tolerable in return for huge productivity gains when creating targeted marketing campaigns, for example. But for medical diagnosis, financial decision-making or automating compliance procedures, it probably isn’t.

Fundamentally, huge amounts of money being poured into AI by businesses could turn out to be wasted if we never manage to understand or adapt to this problem.

Opening The Black Box

This isn’t to say nothing is being done about the problem. Leading AI labs are increasingly focused on understanding what’s actually happening within their systems.

This field of study is known as AI interpretability, and discoveries have helped researchers make connections between activations of artificial neurons and LLM output. This unearths clues about how they work, in much the same way that neuroscientists make discoveries about the human brain.

Another study based on Claude Sonnet led to the discovery that certain LLM behaviors known as “features” can be turned on or off by artificially influencing variables.

And another looks for explanations of why AI sometimes chooses “manipulative” behavior like lying or concealing its intentions, and appears to have made some headway.

By identifying patterns and decision-making processes hidden within the billions of parameters that make up AI systems, it's hoped they will become more useful, but also more predictable and understandable.

All of this is positive, but it doesn't change the fact that AI is already being widely integrated into critical systems today, while many unanswered questions still remain.

Uncertainty Principle

The fact that we don’t fully understand something has never stopped us from using it before. Quantum mechanics, for example, underpins many of the most significant scientific advancements of the last century. Many aspects of it aren’t fully understood, but that hasn’t stopped us from building models of how the entire universe works based on it.

The key, both for individuals thinking about how AI will integrate with their lives and businesses using it to drive growth and innovation, will be learning to live with this uncertainty.

This means maintaining guardrails for when things go wrong, but probably also means not automating simply for the sake of it. The possible consequences of unpredictable AI behavior have to be carefully thought through, planned for and weighed against every project’s value proposition.

Perhaps the biggest unknown is that we still don’t understand the long-term societal implications of AI. When it comes to the way that jobs, businesses, politics and our lives in general are affected, many of the effects may only become visible years from now.

Understanding its limits at least as much as we understand its potential is critical to managing these challenges and making informed decisions involving AI safety.