Iliad Intensive Curriculum

• 0Prerequisites

  The prerequisites for the Iliad Intensive program.

• A.1AI Alignment Introduction

  A conceptual introduction to AI alignment: choosing an alignment target, ensuring trained systems actually pursue it, added challenges with goal-directedness, and views on risks and their mitigations.

• A.2Alignment in Practice

  What do labs do to make their models aligned in practice? Covers the different stages in the training pipeline, deployment and methods labs use to stress-test the result to see if the alignment holds.

• A.3Alignment in Practice 2

  The alignment in practice day discusses a schematic for the transformer architecture, transformer training, alignment issues seen during transformer training, and methods used to correct those issues including AI control.

• A.4Reward Learning Theory

  RLHF grades a policy by a proxy for the true reward: what the evaluator can see. We derive by hand, in a toy example with partial observability, how a policy can hide its failures from the evaluator and inflate the evaluator's perception of its return.

• B.2Mysteries of Deep Learning

  Why deep learning works so well — approximation, generalization and optimization — and the parts nobody can explain yet.

• B.3Singular Learning Theory

  Singular learning theory (SLT) places degeneracy as a core part of understanding how neural networks learn. We cover the parameter-function map, the meaning of degeneracy through the local learning coefficient, to Watanabe's free energy formula and Bayesian phase transitions.

• B.4Training Dynamics

  Exact learning dynamics of deep linear networks -- loss-landscape geometry, balanced gradient flow and the NTK, the rich (saddle-to-saddle) and lazy regimes, their mixed unification, and the implicit bias of SGD noise.

• B.5Data Attribution

  Which training examples caused a model to behave the way it does? These notes develop the main approaches to data attribution -- counterfactuals and Shapley values, classical and Bayesian influence functions, and training-dynamics unrolling -- with derivation exercises throughout.

• C.1Intro to ML Engineering

  Crash course in ML and LLM basics in PyTorch, how to setup machines for ML research and engineering, and making effective use of Claude Code.

• C.2Mechanistic Interpretability

  What can we learn about how a model works by looking at its internals? We cover the main methods for reverse engineering neural networks, from features and circuits to sparse autoencoders and auto-interp of activations, and discuss the relevance to safety.

• C.3Computational Mechanics

  What must a next-token predictor remember about the past? Causal states, belief states and their geometry, from HMMs to generalised HMMs, ending with the evidence that transformers learn this structure.

D.1Decision Theory and Reinforcement Learning

• D.1.1Preferences to Rewards

  Building from preferences and a minimal set of axioms to a utility function expressible as a sum of discounted rewards: the familiar framing in reinforcement learning.

• D.1.2Reinforcement Learning

  The Bellman equations and what follows from them: the existence of optimal policies, the policy improvement theorem, the rate of convergence of Bellman updates, and the convergence of Q-learning.

• D.2Policy Gradients and Misgeneralization

  Implement Vanilla Policy Gradient to train a neural network using reinforcement learning, and explore emperically the concepts of goal misgeneralisation in a toy environment.

D.3AIXI

• D.3.1Solomonoff Induction

  How an idealized agent should predict. A Bayesian mixture over a countable class of computable hypotheses learns to predict any sequence, with total error bounded by the description length of the truth -- a formal Occam's razor.

• D.3.2AIXI

  Exploring the Bayesian optimal policy for history based reinforcement learning.

D.4Agent Foundations

• D.4.1Agent Foundations

  Superintelligence alignment may allow only one critical try, with no trial and error on a system more capable than anything yet observed. Agent foundations seeks research directions that survive extreme optimization and self-modification.

• D.4.2Optimization and Thermodynamics

  An optimizer funnels a broad range of starting conditions into a narrow target; the entropy it removes measures how much. Physics charges for every bit, so an embedded agent pays for optimization with what it knows.

D.5Decision Theory

• D.5.1Decision Theory

  Safety proofs require decision-theoretic assumptions invariant under self-modification. Covers ideal choice in exotic environments (copies, simulations) and multiagent cooperation via open-source game theory and safe Pareto improvements.

• D.5.2Open-Source Game Theory

  When agents can read each other's code, cooperation needs neither repetition nor trust — but the same transparency starts races to commit first. Conditional commitment, safe Pareto improvements, and a tournament.

• D.6Instrumental Convergence

  We discuss a simple mathematical formalization of what it means to "seek power" in a Markov decision process (MDP), and conditions under which such behavior emerges.

• E.1Scalable Oversight and Debate

  Debate is a form of scalable oversight: it seeks to extend a reward signal for honest solutions to difficult problems while reducing the amount of human judgment needed.

• E.2Steganography & Backdoors

  Steganography is the study of hiding messages in plain sight. We run a demo for intuition, show that perfect undetectability is expensive but computational undetectability is cheap, and hide a backdoor in a model by tampering with weight initialization.