AI Developers Shift Focus from Chatbots to 'World Models' for Physical Understanding

Computer scientists are increasingly pivoting from the development of large language models (LLMs) that power chatbots like ChatGPT and Claude to a new frontier: "world models." These models aim to equip AI with the ability to understand and navigate physical environments, a significant departure from an AI's current capability to process and generate text.

Researchers like Louis Castricato, who previously studied LLMs at Brown University, feel that fundamental research in language models has reached a plateau, shifting focus to applications. He has since founded Overworld, a company dedicated to developing AI that can comprehend and interact with the real world, not just language.

While LLM-based chatbots continue to attract substantial investment, many AI entrepreneurs see physical world understanding as the next major development. Prominent figures in AI, including Fei-Fei Li, founder of startup World Labs, describe world models as a crucial concept for achieving true AI intelligence. Li, often called the "Godmother of AI," explains that world models learn the statistical structures of space and time—how light interacts with surfaces, how objects behave according to physics, and how environments appear from various perspectives—unlike LLMs, which learn the statistical structure of text.

Yann LeCun, an AI pioneer and founder of Advanced Machine Intelligence Labs, also views world models as essential for enabling AI agents to predict the consequences of their actions. Different definitions of world models exist, often tied to specific technological goals, such as creating more advanced robots or interactive video games.

Limitations of Language Models

Existing generative AI models, trained on vast amounts of text and visual media, excel at tasks like generating dialogue, images, and code by predicting the next word or pixel. However, experts note their limitations in real-world interaction. Martial Hebert, dean of computer science at Carnegie Mellon University, points out that chatbots cannot perform physical tasks, such as picking up a coffee mug. He argues that understanding the geometry of the world, human movement, and physical interactions is far more complex than predicting text.

Hebert, a robotics researcher for over four decades, sees world models as a pathway to developing "physical AI" or embodied AI more quickly and affordably. He likens the general, adaptive capabilities of the human body, such as balancing and walking, to the internal models the nervous system uses to navigate the world, suggesting that similar general models could power robots.

Applications Beyond Robotics

Beyond robotics, world models have potential applications in areas like video game development. Castricato's company, Overworld, is creating video game environments that dynamically adapt to player interactions, emphasizing interactivity with detailed virtual worlds.

Venture capitalists are showing interest in this emerging field. Steve Jang, co-founder and managing partner at Kindred Ventures, noted that his firm is investing in companies like Overworld, Causal Labs (developing AI for weather prediction), and Extropic (building specialized chips for world models). Jang anticipates a future with diverse AI models rather than a single, all-encompassing one.

Categorizing World Models

To clarify the diverse approaches, Fei-Fei Li proposed a taxonomy of world models. She identified three main categories:

• Renderers: These models prioritize visual fidelity in virtual environments but may not provide robust learning for physical interaction.
• Simulators: They create virtual training grounds that accurately represent physical laws and structures.
• Planners: These models focus on predicting the actions an AI agent or robot should take in unstructured environments.

Li suggests that the development of robots capable of planning is a key goal for the industry, indicating a race towards creating functional and adaptable robotic systems.