---
title: World Models
cluster: D
contributors:
  - Fernando Rosas (University of Sussex)
summary: Theoretical and practical foundations of world models — from control
  theory, neuroscience, and RL framings (POMDPs, belief MDPs, transducers) to
  modern systems (Ha & Schmidhuber, Dreamer, Genie, JEPA) and the role of
  symmetries in building abstractions.
learningOutcomes:
  - Obtain a theoretical understanding of how world models are understood and
    used in AI, and also in other disciplines including control theory,
    neuroscience, and cognitive science.
  - Develop a conceptual understanding of how world models are used in modern AI
    systems, including Ha & Schmidhuber, Dreamer, Genie, and JEPA.
  - Have a critical understanding of why world models matter — how agents can
    use world models, and what can they do with world models that they couldn't
    do without them.
  - Become familiar with the notion of transducer / stochastic automata as a
    general architecture of world models, and how it generalises various kinds
    of world models in reinforcement learning (e.g. POMDPs and beliefs MDPs).
  - Reason about how agents can build abstractions as coarse-grainings of world
    models.
  - Understand the role of symmetries in the construction of abstractions in
    world models.
---
## Overview

World models are a key feature of advanced agents, which allows them to plan and consider counterfactuals without actually taking actions. This module focuses on developing a formal understanding of what world models are and what they are not, and how agents can use them to their advantage. The module is structured into three units: (a) a general introduction to world models and how they are used in modern AI, (b) an exploration of how world models can be formalised in the context of RL, and (c) and overview of how agents can use world models to build abstractions. In doing so, the module adopts an interdisciplinary approach combining ideas from computer science with principles from statistical physics, neuroscience, and cognitive science.

## Prerequisites

-   Basic understanding of learning theory (as provided by e.g. the "Principles of Learning" module) and reinforcement learning (as provided by the module on it), and familiarity with the corresponding terminology.
    
-   Having some knowledge of statistical physics would help to more easily internalise the contents related with abstraction, although this is not mandatory.
    

## Content

### Fast track

The slides should provide a basic understanding of the main ideas. From there, students can choose to go deeper into specific topics by following the notes or papers referred to in various slides.

### Main content

The session is divided into three parts: (a) general introduction to world models, (b) formal definition of world models in reinforcement learning, and (c) operationalisation of abstractions within world models. Each of these units build in the previous one.

-   The introductory lecture in world models was done following the content within these rough [notes](/uploads/world-models/world_models_report.pdf). The session used [these slides](/uploads/world-models/intro_world_models.pdf).
    
-   The session on a formal approach to world models for reinforcement learning agents used [these slides](/uploads/world-models/FROSAS_worldmodels_afternoon1.pdf), which are closely related to [this paper](https://arxiv.org/abs/2504.04608) (see also this [LW post](https://www.lesswrong.com/posts/L6Z6K8qXJhrSNMN4L/ai-in-a-vat-fundamental-limits-of-efficient-world-modelling))
    
-   The last session on abstractions in world models used [these slides](/uploads/world-models/FROSAS_abstractions_afternoon2.pdf), which are closely related to this [preprint](https://arxiv.org/abs/2512.00984).
    

### Learn more

Related work on world models:

-   [Geniel 3](https://deepmind.google/blog/genie-3-a-new-frontier-for-world-models/)
    
-   [Latest Dreamer paper](https://www.nature.com/articles/s41586-025-08744-2)
    
-   [Robust agents learn world models](https://deepmind.google/research/publications/49666/)
    
-   [Separation principle](https://en.wikipedia.org/wiki/Separation_principle) from optimal control theory as a foundation of why agents build beliefs and plan upon them.
    
-   [Deep dive into JEPA](https://rohitbandaru.github.io/blog/JEPA-Deep-Dive/)
    

Related work on abstractions on world models

-   [Abstractions as emergent properties](https://www.quantamagazine.org/the-new-math-of-how-large-scale-order-emerges-20240610/)
    
-   [Towards a unified theory of abstractions in MDPs](/uploads/world-models/P21.pdf)
    
-   [A Theory of Abstraction in Reinforcement Learning](/uploads/world-models/thesis.pdf)