World Models
Theoretical and practical foundations of world models — from control theory, neuroscience, and RL framings (POMDPs, belief MDPs, transducers) to modern systems (Ha & Schmidhuber, Dreamer, Genie, JEPA) and the role of symmetries in building abstractions.
By Fernando Rosas (University of Sussex)
What you’ll learn
- Obtain a theoretical understanding of how world models are understood and used in AI, and also in other disciplines including control theory, neuroscience, and cognitive science.
- Develop a conceptual understanding of how world models are used in modern AI systems, including Ha & Schmidhuber, Dreamer, Genie, and JEPA.
- Have a critical understanding of why world models matter — how agents can use world models, and what can they do with world models that they couldn't do without them.
- Become familiar with the notion of transducer / stochastic automata as a general architecture of world models, and how it generalises various kinds of world models in reinforcement learning (e.g. POMDPs and beliefs MDPs).
- Reason about how agents can build abstractions as coarse-grainings of world models.
- Understand the role of symmetries in the construction of abstractions in world models.
Overview
World models are a key feature of advanced agents, which allows them to plan and consider counterfactuals without actually taking actions. This module focuses on developing a formal understanding of what world models are and what they are not, and how agents can use them to their advantage. The module is structured into three units: (a) a general introduction to world models and how they are used in modern AI, (b) an exploration of how world models can be formalised in the context of RL, and (c) and overview of how agents can use world models to build abstractions. In doing so, the module adopts an interdisciplinary approach combining ideas from computer science with principles from statistical physics, neuroscience, and cognitive science.
Prerequisites
-
Basic understanding of learning theory (as provided by e.g. the "Principles of Learning" module) and reinforcement learning (as provided by the module on it), and familiarity with the corresponding terminology.
-
Having some knowledge of statistical physics would help to more easily internalise the contents related with abstraction, although this is not mandatory.
Content
Fast track
The slides should provide a basic understanding of the main ideas. From there, students can choose to go deeper into specific topics by following the notes or papers referred to in various slides.
Main content
The session is divided into three parts: (a) general introduction to world models, (b) formal definition of world models in reinforcement learning, and (c) operationalisation of abstractions within world models. Each of these units build in the previous one.
-
The introductory lecture in world models was done following the content within these rough notes. The session used these slides.
-
The session on a formal approach to world models for reinforcement learning agents used these slides, which are closely related to this paper (see also this LW post)
-
The last session on abstractions in world models used these slides, which are closely related to this preprint.
Learn more
Related work on world models:
-
Separation principle from optimal control theory as a foundation of why agents build beliefs and plan upon them.
Related work on abstractions on world models