The Next Giant Leap For AI Is Called World Models

In recent years, we’ve become used to seeing text, pictures, videos and even computer code generated by AI. But what if it could go a step further and create entire worlds?

Well, for developers working on a category of generative tools capable of simulating full 3D environments, that’s the aim.

World models are designed to generate immersive 3D environments, complete with inhabitants and working physics systems, that we can explore and manipulate, as if we were really there.

Think of the kind of 3D spaces we move around in video games or virtual reality, but rather than being meticulously crafted by humans, they’re entirely built by machines.

It’s thought they will have enormous implications for everything from engineering and architecture to robotics and medicine, by creating simulations that help us understand the real world.

So let’s take a bit of a deeper look into exactly what these world models are, who’s building them, and why they’re one of the most important areas of AI research today.

How Do World Models Work?

There are two different methods that AI models can use to create virtual worlds and environments right now.

The first is to model everything dynamically, on the fly, as the user interacts with the world. This is similar to the way generative video models work—by predicting how each pixel should change over time based on what it’s learned about physics and object behavior.

Only world models respond to the user’s input as they navigate around the world by moving the camera, or interacting with people and objects it contains, rather than just interpreting prompts to decide what video should be generated.

Using this method, the entire world is continuously generated, frame-by-frame, based on the model’s internal understanding of how the environment and objects should behave.

This method allows the creation of highly flexible, realistic and unique environments. Imagine a video game world, for example, where literally anything can happen. The possibilities aren’t limited to situations and choices that have been written into the code by a game programmer, because the model generates visuals and sounds to match any choice the player makes.

One major drawback is that this approach is hugely compute-intensive. This means the most sophisticated of the real-time world models available today are limited to maintaining world consistency for just a few minutes, due to the high CPU overhead.

This is why other models take a different approach to solving the problem of world generation. Rather than generating a world frame by frame on the fly, they take a prompt and transform it into persistent geometric models, digital assets, and physics metadata.

This data can then be downloaded and imported into other software tools where it can be manipulated, edited and explored.

Who Is Building World Models?

Some of the biggest names in AI are currently developing their own world models.

These include Google, with its Genie 3 platform, currently in research preview and capable of creating worlds that remain persistent for several minutes.

Meta (Facebook) is also developing its own world model, following the same dynamic-generation technique as Google. Its platform, Habitat 3, is designed to create virtual environments where embodied AI (physical robots) can be trained to navigate, manipulate objects, and interact safely with humans before being deployed in the real world.

World Labs, led by AI pioneer Fei-Fei Li, however, takes a different approach with its Marble world model, which creates persistent, downloadable 3D environments from text, image or video prompts.

Even Elon Musk is getting involved, with his xAI development group working on a so-far unnamed world model that will reportedly be used for both video games and training robots.

What Will They Do?

From a business perspective, the use cases for world models are potentially as unlimited as those for text, image and sound generation.

Leaving aside their obvious utility in video games and entertainment, potential use cases include healthcare settings where they can be used to create immersive digital twins of clinical environments to simulate patient interactions.

These environments will also be used to create virtual training grounds for industrial robots, autonomous vehicles, and other embodied AI objects that will operate in real-world environments.

They’ll let manufacturers test new layouts, equipment placements and workflows in plants and factories, modelling for safety, energy efficiency, and reduced downtime.

Architects will use them to model, view and interact with buildings, testing how they react to physics, lighting, airflow and the movement of people before a single brick is put in place.

And because they will model on a micro as well as macro-scale, they can be used to simulate the human body environment and the molecular reactions that determine the efficacy of new medicines and treatments.

Why Is This So Important?

I believe world models have the potential to be integral to the wider genAI-driven transformation of business and society that’s currently ongoing.

And I’m not alone there — in fact, Jack Parker-Holder and Shlomi Fruchter of Google DeepMind have stated they believe it will be a key stepping stone on the road to artificial general intelligence.

AGI, the current “holy grail” of AI development, is usually summarized as referring to machines capable of applying their knowledge and abilities to any task, regardless of whether they’ve been specifically trained to do it, just as humans can.

In order to navigate and understand the world, AI needs to know how it’s built, what it’s made from, and how it holds together.

World models promise to give it the capacity to do this, in a way that augments its abilities with language and vision.

This is why I believe they are one of the most exciting and vital areas of AI development right now, and a field that anyone wanting to understand how AI will influence and shape the future should be following with interest.