Physics of Deep Learning

Contents

• -1. Prerequisites
• 0. A one-day teaching plan
  • 0.1 Schedule
  • 0.2 Main references
• 1. Intro reading: learning mechanics
• 2. Statistical mechanics of learning
  • 2.1 The Bayesian model of learning
  • 2.2 Why the exponential of a loss is a likelihood
  • 2.3 Partition functions, free energy, and cumulants
  • 2.4 A physics example: the Ising model
  • 2.5 Phases and phase transitions
  • 2.6 The Ising perceptron: a first-order transition in learning
  • 2.7 Comparison with QFT
• 3. Large width at initialization: the Gaussian process limit
  • 3.1 Notation
  • 3.2 Normalizing the network
  • 3.3 Gaussian processes
  • 3.4 The NNGP correspondence
  • 3.5 NNGP via functional integrals
  • 3.6 L/N
  • 3.7 Edge of chaos
• 4. The NTK and large width during training
  • 4.1 Gradient flow and the NTK
    • 4.1.1 Geometry of gradient flows
  • 4.2 Features
  • 4.3 Constant NTK at large width
  • 4.4 Lazy learning
  • 4.5 Summary
  • 4.6 Bayesian learning at large width: kernel learning
• 5. Mean-field scaling
  • 5.1 Mean-field theories in physics
    • 5.1.1 Warm-up: the Curie–Weiss magnet
    • 5.1.2 The general shape of a mean-field model
  • 5.2 Mean-field scaling for one-hidden-layer networks
    • 5.2.1 Initialization
    • 5.2.2 The loss is a mean-field energy
    • 5.2.3 Equations of motion and the mean field
    • 5.2.4 From particles to measures: the continuity equation
    • 5.2.5 The McKean–Vlasov equation and self-consistency
  • 5.3 The NTK is no longer constant: feature learning
  • 5.4 The scaling dial and Maximal Update Parameterization
  • 5.5 What is this framework good for?
  • 5.6 Grokking as a lazy-to-rich transition: a reading exercise
  • 5.7 Bayesian grokking: a first-order transition (alternative reading)
• 6. Non-equilibrium physics, MSRJD, and DMFT
  • 6.1 From stochastic dynamics to path integrals: the MSRJD formalism
  • 6.2 Double descent as a dynamical phase transition
    • 6.2.1 Model definition and formalism
    • 6.2.2 Overview of results via training and generalization error
  • 6.3 Dynamical mean-field theory (DMFT)
    • 6.3.1 Self-consistency
    • 6.3.2 Recent applications in deep learning
• 7. Scaling laws
  • 7.1 The empirical laws
  • 7.2 Joint laws, bottlenecks, and the Chinchilla revision
  • 7.3 Where do the laws come from?
  • 7.4 A survey of theoretical proposals
  • 7.5 The quantization model: exponents from the statistics of skills
  • 7.6 Emergence: smooth laws from staggered cliffs
  • 7.7 Exponents from geometry and from spectra
    • 7.7.1 Geometry: the data-manifold argument
    • 7.7.2 Spectra: scaling laws from kernel eigenvalues
  • 7.8 A solvable large-N field theory
  • 7.9 All the knobs at once: a dynamical theory
  • 7.10 Beyond pretraining: post-training and test-time scaling
• A. The replica method
  • A.1 What the annealed approximation gets wrong
  • A.2 Replicas
  • A.3 A variant: the spherical perceptron
• B. From linear regression to kernel regression
  • B.1 Linear regression
  • B.2 Linear regression with nonlinear features
  • B.3 Kernel regression
  • B.4 Kernel ridge regression
  • B.5 Linearized networks are kernel ridge regressors
• C. Solutions to exercises

-1. Prerequisites

We don't assume any physics background, though reading some of the suggested reference material may be helpful. We do assume:

• a basic understanding of probability theory and Bayesian inference;
• familiarity with simple neural networks (MLP's);
• undergraduate-level calculus and linear algebra;
• and, in a few places, differential geometry.

What you’ll learn

By the end of these notes, you should be able to:

• Use the statistical-mechanics dictionary for Bayesian learning — energy, entropy, partition function, free energy, cumulants, order parameters, quenched averages — and explain why the exponential of a loss function is a likelihood.
• Solve the Ising model in the mean-field approximation; explain what a phase transition is, why finite systems show smooth sigmoids and growing susceptibility peaks instead of true kinks, and how the Ising perceptron realizes a first-order transition to perfect generalization, metastable lag included.
• Derive the Gaussian-process description of a wide network at initialization, and explain the roles of the fan-in normalization, the depth-to-width ratio, and the edge of chaos.
• Derive the NTK, and explain why it freezes at large width — so that training reduces to kernel regression, with no feature learning.
• Derive, heuristically, the mean-field limit of one-hidden-layer networks — neurons as interacting particles, self-consistency, the McKean–Vlasov equation — and explain how its moving kernel encodes feature learning; place the NTK and mean-field limits on the scaling dial, and connect it to $\mu$P and hyperparameter transfer.
• Compare the two theoretical accounts of grokking — an equilibrium first-order transition vs. delayed lazy-to-rich dynamics — and propose an experiment that would distinguish them.
• Recognize the MSRJD/DMFT path-integral formulation of training dynamics, the role of correlation and response functions, and double descent as a dynamical phase transition.
• Derive the three scaling laws of the quantization model, explain "emergence" as the aggregation of staggered cliffs, and survey where scaling exponents come from: skill statistics, data geometry, and kernel spectra.

0. A one-day teaching plan

These notes contain far more than can be taught in a day. This section records one selection of material that was taught in a single day at the Iliad Intensive (September 2026, day B.6), as a worked example of how the notes can be used. The day alternates three lectures, given from the slide deck that accompanies these notes, with exercise sessions in which participants pick one exercise from the notes and work on it, alone or in small groups. It closes with a short reading and a discussion.

The deck is the file slides.tex in the same folder as these notes. The website build compiles it and offers it in the "Slides" row at the top of this chapter's web page; it can also be opened directly at iliad-team.github.io/iliad-intensive/downloads/qft/qft-slides.pdf. Its three parts are the three lectures below: frames 1–20 (Part I), 21–41 (Part II), and 42–52 (Part III). The lectures deliberately leave derivations to the exercise sessions, so the deck is sparse by design; the corresponding sections of these notes fill in what the slides state.

0.1 Schedule

time	session	where in these notes
10:00–11:00	Lecture, Part I: Bayesian learning and phase transitions	Sections 2.1–2.5
11:00–11:15	Break
11:15–12:00	Exercises, Part I (choose one)	Exercises 2.1, 2.2 and 2.3
12:00–12:30	Lecture, Part IIa: large width at initialization; the NTK	Sections 3.1–3.4, Section 3.6, Sections 4.1–4.2
12:30–1:30	Lunch
1:30–2:00	Lecture, Part IIb: large width during training; mean-field scaling	Sections 4.3–4.4, Sections 5.2–5.4
2:00–3:00	Exercises or lab, Part II (choose one)	Exercises 3.3, 4.5, 4.4 and 5.3
3:00–3:45	Lecture, Part III: neural scaling laws, in theory and practice	Sections 7.1–7.5
3:45–4:15	Break
4:15–5:00	Exercise, Part III	Exercise 7.1
5:00–5:30	Reading on "learning mechanics"	Section 1
5:30–5:50	Discussion in small groups	the prompts of Section 1
5:50–6:00	Feedback

0.2 Main references

The day is based on the following references, all of which appear elsewhere in these notes. For Part I: Tong's lectures on statistical field theory and on kinetic theory (Tong 2012; Tong 2012), the physics behind the lecture; Engel and Van den Broeck (Engel & Van den Broeck 2001) for the perceptron tradition; and Gyorgyi (Gy"orgyi 1990) for the Ising perceptron transition. For Part II: Hanin's lectures (Hanin 2026) and Roberts, Yaida, and Hanin (Roberts et al. 2022) on large-width limits and the $L/N$ expansion; Ringel et al. (Ringel et al. 2025) on statistical field theory in deep learning; Spiliopoulos, Sowers, and Sirignano (Spiliopoulos et al. 2025) for rigorous NTK and mean-field limits; Lin (Lin 2025) for experiments identifying features through the empirical NTK; and the two accounts of grokking (Kumar et al. 2024; Rubin et al. 2024), which were not lectured but are the natural follow-up reading (Sections 5.6 and 5.7). For Part III: Kaplan et al. (Kaplan et al. 2020) and Henighan et al. (Henighan et al. 2020) for the empirical laws; Hoffmann et al. (Hoffmann et al. 2022) and Porian et al. (Porian et al. 2024) for Chinchilla and the reconciliation with Kaplan; Michaud et al. (Michaud et al. 2023) for the quantization model; and Maloney, Roberts, and Sully (Maloney et al. 2022) and Bordelon, Atanasov, and Pehlevan (Bordelon et al. 2024) for the solvable field theories.

A complementary introduction to the physics of learning, taught at the August 2026 Intensive by Renshaw-Whitman, can be found in (Renshaw-Whitman 2026); Appendix B is adapted from it.

1. Intro reading: learning mechanics

The review (Simon et al. 2026) argues that a scientific theory of deep learning — a "mechanics of learning" — is already emerging, and it organizes the evidence into five converging strands: analytically solvable settings, insightful limits, simple empirical laws, disentangled hyperparameters, and universal phenomena. The body of the paper develops the first two strands in detail and samples the remaining three. These notes will do much the same, so it is worth keeping the five strands in mind as a map of everything that follows.

The paper's starting observation deserves to be savored, because it sets the tone for the whole subject. A deep learning system is completely specified by four explicit ingredients: an architecture (a composition of simple linear and nonlinear maps), a dataset, an objective, and a learning rule (gradient-based updates from a random initialization). Nothing is hidden: every parameter, every gradient, every intermediate activation can be measured, at every step, to whatever precision we like. In the language of the reading, deep learning directly exposes its own equations of motion — the central challenge is not opacity but complexity. This places the field in very different territory from the genuinely opaque sciences: in neuroscience or cell biology the microscopic equations of motion are unknown and the state is largely unmeasurable, while in high-energy physics we must infer the microscopic laws indirectly, from scattering amplitudes. The better analogy is fluid dynamics: there, too, we possess the microscopic law — for deep learning, the update equation is five lines of code — and what is missing is the effective description, the analogue of Navier–Stokes.

Read Section 2 (only) of (Simon et al. 2026), and think about the following:

1. Rank the physics analogies in Table 1 of the reading (solvable models $\leftrightarrow$ hydrogen atom; limits $\leftrightarrow$ thermodynamic limit; empirical laws $\leftrightarrow$ Kepler and Planck; hyperparameters $\leftrightarrow$ Reynolds number; universality $\leftrightarrow$ critical phenomena). Which are the most compelling? Which do you think are most useful in understanding deep learning?
2. Discretization and other approximations. The reading's Discretization Hypothesis proposes that practical networks are best understood as noisy, finite approximations to models of infinite size — like a finite-difference discretization of a PDE — so that finite-size effects typically cost performance rather than enable it. Given what you've seen so far, does this seem accurate? The hypothesis is falsifiable: it would be refuted by a finite-size effect that delivers a general benefit unachievable in any limit. What are some possible experiments/setups that could falsify it?
3. What actually matters for alignment? The reading wants falsifiable, quantitative, average-case predictions rather than worst-case bounds — physics rather than classical learning theory. What is lost? E.g. for alignment, does one want worst-case guarantees or forecasting of typical capabilities? Which of the five lines of evidence is most relevant for alignment?

You may want to revisit these questions at later stages as you go through these notes, and learn more about physics-inspired approaches in deep learning.

2. Statistical mechanics of learning

There is a large and well-developed body of work applying physics to learning theory. Much of it consists of statistical mechanics applied to Bayesian learning. The classic textbook is Engel and Van den Broeck (Engel & Van den Broeck 2001); modern reviews include (Zdeborov'a & Krzakala 2016; Bahri et al. 2020); and an excellent reference for the physics itself is David Tong's lecture course on statistical field theory (Tong 2012), which we follow closely in Sections 2.4–2.5.

In this section we focus on the Bayesian model of learning. We first recall what it is, why it is a reasonable idealization of how networks are actually trained, and why the exponential of a loss function deserves to be called a likelihood (Sections 2.1–2.2). We then switch to physics. We introduce partition functions, free energies, and cumulants (Section 2.3), and work through the Ising model in the mean-field approximation, building up to first- and second-order phase transitions (Sections 2.4–2.5). The payoff is a learning problem that is the Ising magnet's twin, the Ising perceptron, which shows a first-order transition to perfect generalization; it is left as a self-contained walk-through exercise (Section 2.6). Section 2.7 is a translation guide for readers arriving from quantum field theory. The ideas introduced here — order parameters, free energies, saddle points, quenched averages, phase transitions — will keep returning throughout the notes.

2.1 The Bayesian model of learning

Consider a network $f_{\theta }$ with parameters $\theta \in {\mathbb{R}}^P$, trained on $n$ samples $(x^{\mu },y^{\mu }{)}_{\mu =1}^n$ to minimize a loss

a sum over samples of a per-sample loss $\ell$. In practice, the network is initialized at random, ${\theta }_0\sim \pi (\theta )$, and trained with some version of gradient descent,

$$\Delta \theta \approx -\eta {\nabla }_{\theta }\mathcal{L}(\theta ).$$

A common idealization of this process, and the one this section is about, is Bayesian learning. It says that after training, the parameters should be regarded as drawn from the posterior distribution

Here the initialization distribution $\pi (\theta )$ plays the role of the Bayesian prior, and $e^{-\beta \mathcal{L}(\theta )}$ is interpreted as the likelihood of the training data given the parameters $\theta$ (Section 2.2 explains why this is reasonable). The positive constant $\beta$, the inverse temperature, controls how strongly the data constrain the parameters. We will see in a moment that it can be read as a rough measure of how much training has been done.

An interpretation of $\beta$. It is convenient to write the loss as $n$ times an average over samples,

$$\mathcal{L}(\theta )=n{L}_n(\theta ),L_n(\theta ):=\frac{1}{n}\sum _{\mu =1}^n\ell (f_{\theta }(x^{\mu }),y^{\mu }).$$

The empirical loss $L_n$ stays finite as $n\to \infty$, converging to the population loss $L(\theta ):={\mathbb{E}}_{(x,y)}\ell (f_{\theta }(x),y)$, the expectation over the true data distribution. Thus for large $n$ the posterior is

and the sample size $n$ always accompanies the inverse temperature, in the combination $n\beta$. Increasing the number of samples is the same as lowering the temperature. If, moreover, samples are drawn fresh at each training step, then $n$ is also a rough measure of training time. Altogether: more data $=$ more training $=$ larger $n\beta$ $=$ colder. We will often write $e^{-\beta L}$ for short, but it is worth remembering that the effective $\beta$ grows with $n$.

Bayes vs. gradient descent. We do not train networks by sampling from (2.3); we train them by gradient descent. Why, then, is the Bayesian model a useful idealization? There are two connections.

The first is heuristic: the analogy just described, $n\to \infty$ (more samples, longer training) $\leftrightarrow$ $\beta \to \infty$ (lower temperature). As we will see, sweeping a parameter like $\beta$ can take a system through phase transitions, and the analogy predicts that training can do the same. It is only an analogy. In particular, when a system is driven dynamically through a first-order transition, the transition is delayed by hysteresis (Section 2.5), so timing predictions from the Bayesian model should be treated with care. A genuinely dynamical formalism for training, the MSRJD path integral, appears in Section 6.

The second connection is exact, and is what makes the Bayesian model testable in practice. Replace the batch noise of SGD by constant, isotropic Gaussian noise; the result is called Langevin dynamics:

Here $\xi$ is Gaussian noise with mean $0$ and covariance $I$, $\eta$ is the learning rate, and $T$ is a parameter called the temperature (or diffusion constant). Each discrete algorithm on the left has a continuous-time limit on the right, obtained by shrinking the step size and scaling up the number of steps: gradient descent becomes gradient flow, and Langevin dynamics becomes the Langevin equation, a stochastic differential equation. The key fact is that if one starts from any reasonable distribution of parameters and evolves with the Langevin equation (for a sufficiently nice $\mathcal{L}$), the distribution converges at late times to the Gibbs distribution

(See (Tong 2012, Chapter 3) for a physics-focused discussion of this convergence.) A prior can be included by training on the modified loss $\mathcal{L}(\theta )-{\beta }^{-1}\log \pi (\theta )$, which reproduces (2.3). Thus one can engineer the Bayesian posterior by Langevin training, and this is how theoretical predictions derived from (2.3) are usually tested. Note that the Langevin route gives a different Bayes-to-gradient connection from the first: here training time $t\to \infty$ at fixed $T$ gives the posterior at $\beta =1/T$, while $T\to 0$ at fixed $t$ gives gradient flow.

2.2 Why the exponential of a loss is a likelihood

The Bayesian model treats $e^{-\beta \mathcal{L}(\theta )}$ as the probability of the training data given the parameters. For the loss functions actually used in deep learning, this is not a metaphor: the losses were designed as negative log-likelihoods, so that minimizing the loss is maximum-likelihood estimation. Two examples cover most of practice.

Mean squared error. Suppose the labels are the network's output plus Gaussian noise, $y=f_{\theta }(x)+ϵ$ with $ϵ\sim \mathcal{N}(0,{\sigma }^2)$. Then the probability density of a label given the input and the parameters is

$$p(y|x,\theta )=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp [-\frac{1}{2{\sigma }^2}(y-f_{\theta }(x){)}^2],$$

and for $n$ independent samples,

The MSE loss is the negative log-likelihood of a Gaussian noise model, and the inverse temperature is the inverse noise variance. Noisier labels mean a hotter posterior, which constrains the parameters less.

Cross-entropy. For classification, the network outputs a probability distribution $p_{\theta }(y|x)$ over $K$ classes, typically the softmax of its $K$ output values. The cross-entropy loss is $\ell =-\log p_{\theta }(y|x)$, so that

exactly, with $\beta =1$: the exponential of the cross-entropy loss is the likelihood, with no noise model to choose. The same is true of the next-token loss of a language model, which is exactly the log-probability the model assigns to the training corpus.

The general pattern. Whenever a loss is a negative log-likelihood, $e^{-\mathcal{L}}$ is the likelihood and (2.3) at $\beta =1$ is the textbook Bayesian posterior. A value $\beta \ne 1$ gives a tempered posterior. It arises naturally when the noise level is not known (as in (2.9)), and it is also a useful theoretical knob: as the interpretation of $n\beta$ above shows, temperature and sample size are interchangeable in the Bayesian model. There is also a useful limiting case. Some losses are hard constraints, $\ell =0$ if the prediction is right and $\ell =\infty$ if it is wrong. Then $e^{-\ell }$ is a step function, the posterior is uniform on the parameters that fit every training sample, and no $\beta$ appears at all. This is the zero-temperature limit, and it is the setting of the Ising perceptron in Section 2.6.

2.3 Partition functions, free energy, and cumulants

The Bayesian posterior (2.3) looks exactly like a thermal system in equilibrium with a heat bath, the basic object of statistical mechanics. The dictionary is:

Bayesian learning	statistical mechanics
parameters $\theta \in {\mathbb{R}}^P$	microscopic state of a system
	(e.g. the spins of $P$ electrons)
loss $\mathcal{L}(\theta )$	energy $E(\theta )$, or Euclidean action $S(\theta )$
posterior $p(\theta )\propto e^{-\beta \mathcal{L}(\theta )}$	Boltzmann distribution at temperature $T=1/\beta$
number of samples $n$, or training time	inverse temperature $\beta$
#parameters $P$	#degrees of freedom $N$

To include the prior $\pi (\theta )$, simply redefine the energy as $E(\theta )=\mathcal{L}(\theta )-{\beta }^{-1}\log \pi (\theta )$, so that $e^{-\beta E}=e^{-\beta \mathcal{L}}\pi$. In what follows we write $E(\theta )$ for whatever sits in the exponent, and $N$ for the number of degrees of freedom, as physicists do; for a network, $N=P$.

The thermodynamic limit. Statistical mechanics is about systems with very many degrees of freedom, and one studies their behavior as $N\to \infty$: the thermodynamic limit. In this limit one distinguishes extensive quantities, which grow in proportion to $N$ (the total energy of a gas), from intensive ones, which stay finite (the temperature, or the energy per particle). Both notions transfer directly to learning: with $P$ parameters and $n$ samples, the loss (2.1) is extensive in $n$ and the empirical loss $L_n$ is intensive. Many of the approximations in these notes are thermodynamic limits of some kind, sending the number of parameters, the width, or the number of samples to infinity.

Partition function and free energy. The normalization constant in (2.3),

is called the partition function. In Bayesian terms it is the marginal likelihood of the data. It depends on $\beta$ and on any other hyperparameters of the problem. Its logarithm defines the free energy¹ and the free energy density,

There are three reasons to care about $F$ rather than $Z$. First, $F$ is extensive and $\phi$ intensive: the free energy density has a limit as $N\to \infty$, while $Z$ itself is exponentially large or small. Second, "$F$ is minimized": at large $N$ the integral (2.11) is dominated by the regions of parameter space with the smallest local free energy, so the system's macroscopic state is found by minimizing $F$. Section 2.4 makes this precise. Third, derivatives of $F$ compute expectation values, as follows.

Sources and cumulants. Suppose we care about the expectation values of some observables ${\mathcal{O}}_i(\theta )$, functions of the parameters. Add source terms $J^i$ to the exponent,

$$Z(\beta ,J)=\int d\theta e^{-\beta E(\theta )+{\sum }_i{J}^i{\mathcal{O}}_i(\theta )},F(\beta ,J)=-\log Z(\beta ,J).$$

Differentiating under the integral,

and in general the $p$-th derivative is the $p$-th cumulant,

$${\kappa }_p({\mathcal{O}}_{i_1},\dots ,{\mathcal{O}}_{i_p})=-\frac{{\partial }^{p}F}{\partial J^{i_1}\cdots \partial J^{i_p}}{|}_{J=0}:$$

the mean, the covariance, the third central moment, and from $p=4$ on, combinations of moments that vanish for Gaussian variables. The sourced free energy is the generating function of cumulants. Cumulants will return in field-theory clothing in Section 3.3, where they measure how far a network at initialization is from a Gaussian process.

If the only observable we care about is the energy itself, no source is needed: $\beta$ already couples to $E$, and

We will meet these derivatives again shortly: discontinuities in them are how phase transitions are defined.

Exercise 2.1 (★) (Partition functions, sources, and cumulants). Take a Boltzmann distribution $p_{\beta }(\theta )\propto e^{-\beta E(\theta )}$, an observable $\mathcal{O}(\theta )$, and the sourced partition function $Z(\beta ,J)=\int d\theta e^{-\beta E(\theta )+J\mathcal{O}(\theta )}$, with $F=-\log Z$. All expectations are in $p_{\beta }$, that is, at $J=0$.

(a) Show that $\mathbb{E}(\mathcal{O})=-{\partial }_{J}F{|}_{J=0}$ and $\mathrm{Var}(\mathcal{O})=-{\partial }_J^{2}F{|}_{J=0}$.

(b) The third cumulant is defined by continuing the pattern, ${\kappa }_3(\mathcal{O}):=-{\partial }_J^{3}F{|}_{J=0}$. Compute it in terms of the moments $\mathbb{E}({\mathcal{O}}^3)$, $\mathbb{E}({\mathcal{O}}^2)$, $\mathbb{E}(\mathcal{O})$, and show that it equals the third centered moment $\mathbb{E}[(\mathcal{O}-\mathbb{E}\mathcal{O}{)}^3]$.

(c) Show that for a Gaussian $\mathcal{O}$, $\log Z(J)$ is exactly quadratic in $J$, so that all cumulants with $p\ge 3$ vanish. Compute ${\kappa }_4$ in terms of centered moments and check that it is not the fourth centered moment. (Moral: cumulants measure the failure of higher moments to be determined by lower ones in the Gaussian pattern.)

(d) Verify (2.16).

Solution

(a) Differentiating under the integral, ${\partial }_J\log Z=\frac{1}{Z}\int d\theta \mathcal{O}{e}^{-\beta E+J\mathcal{O}}={\mathbb{E}}_J(\mathcal{O})$, which at $J=0$ is $\mathbb{E}(\mathcal{O})$; since $F=-\log Z$, this is $-{\partial }_{J}F{|}_0$. Once more, ${\partial }_J^2\log Z={\mathbb{E}}_J({\mathcal{O}}^2)-{\mathbb{E}}_J(\mathcal{O}{)}^2={\mathrm{Var}}_J(\mathcal{O})$.

(b) Using ${\partial }_J{\mathbb{E}}_J({\mathcal{O}}^k)={\mathbb{E}}_J({\mathcal{O}}^{k+1})-{\mathbb{E}}_J({\mathcal{O}}^k){\mathbb{E}}_J(\mathcal{O})$ on ${\partial }_J^2\log Z={\mathbb{E}}_J({\mathcal{O}}^2)-{\mathbb{E}}_J(\mathcal{O}{)}^2$ gives ${\kappa }_3=\mathbb{E}({\mathcal{O}}^3)-3\mathbb{E}({\mathcal{O}}^2)\mathbb{E}(\mathcal{O})+2\mathbb{E}(\mathcal{O}{)}^3$. Expanding $\mathbb{E}[(\mathcal{O}-\mathbb{E}\mathcal{O}{)}^3]=\mathbb{E}({\mathcal{O}}^3)-3\mathbb{E}({\mathcal{O}}^2)\mathbb{E}(\mathcal{O})+3\mathbb{E}(\mathcal{O})\mathbb{E}(\mathcal{O}{)}^2-\mathbb{E}(\mathcal{O}{)}^3$ gives the same expression.

(c) For $\mathcal{O}\sim \mathcal{N}(\mu ,{\sigma }^2)$ (or more generally when $\mathcal{O}$ is Gaussian under $p_{\beta }$), $\mathbb{E}(e^{J\mathcal{O}})=e^{J\mu +J^2{\sigma }^2/2}$, so $\log Z(J)=\log Z(0)+J\mu +\frac{1}{2}{J}^2{\sigma }^2$ is quadratic and every derivative beyond the second vanishes. One more derivative of the pattern in (b), with $\bar{\mathcal{O}}:=\mathcal{O}-\mathbb{E}\mathcal{O}$, gives ${\kappa }_4=\mathbb{E}({\bar{\mathcal{O}}}^4)-3\mathbb{E}({\bar{\mathcal{O}}}^2{)}^2$; for a Gaussian $\mathbb{E}({\bar{\mathcal{O}}}^4)=3{\sigma }^4$, so ${\kappa }_4=0$ while the fourth centered moment is not.

(d) ${\partial }_{\beta }\log Z=-\frac{1}{Z}\int E{e}^{-\beta E}=-\mathbb{E}(E)$, so ${\partial }_{\beta }F=\mathbb{E}(E)$; ${\partial }_{\beta }^2\log Z=\mathbb{E}(E^2)-\mathbb{E}(E{)}^2$, so $-{\partial }_{\beta }^{2}F=\mathrm{Var}(E)$; and ${\partial }_{\beta }^3\log Z=-{\kappa }_3(E)$ by the same computation as (b) with $J\to -\beta$, so ${\kappa }_3(E)={\partial }_{\beta }^{3}F$.

Quenched averages. In a learning problem there is a second source of randomness besides the posterior: the training set itself. Quantities like the free energy depend on which $n$ samples were drawn, and at large $n$ one expects them to concentrate around their average over draws. Physicists call randomness that is fixed before the system equilibrates quenched disorder, by analogy with impurities frozen into an alloy when it is quenched in water; the training data are quenched disorder for the posterior. The quenched free energy is

and it is the free energy, not the partition function, that should be averaged, because it is the free energy that is extensive and whose derivatives give expectation values. Averaging $\log Z$ is usually hard, and the annealed approximation swaps the order,

which is an upper bound by Jensen's inequality and often a good approximation. We will use it in Section 2.6. The exact treatment goes through the replica method, sketched in Appendix A; the classic reference is (Engel & Van den Broeck 2001).

2.4 A physics example: the Ising model

The analogy with thermal physics opens up a bag of well-developed approximation methods. We illustrate two of them, the mean-field approximation and the saddle-point approximation at large $N$, on the most famous model in statistical mechanics. For the physics of this example, well beyond what we cover here, see (Tong 2012, Chapter 1).

The model. Let $\Lambda$ be a graph with $N$ vertices and undirected edges $⟨ij⟩$; typically $\Lambda$ is a regular lattice in $D$ dimensions, and every vertex has the same number $d$ of neighbors ($d=2D$ for a hypercubic lattice). At each vertex $i$ sits a spin $s_i\in \{\pm 1\}$. A state of the system is a choice of all $N$ spins, so there are $2^N$ states, and the energy of a state is

The coupling $J>0$ rewards neighboring spins for aligning with each other, and the external magnetic field $B$ rewards spins for aligning with it. At inverse temperature $\beta$ the Boltzmann distribution is $p_{\beta }(s)=e^{-\beta E(s)}/Z$ with $Z(\beta ,J,B)={\sum }_{s\in \{\pm 1{\}}^N}{e}^{-\beta E(s)}$. Dimensionally, only the combinations $\beta J$ and $\beta B$ matter.

The simplest macroscopic quantity describing a state is its magnetization, the average spin,

It is our first example of an order parameter: a single number that summarizes an $N$-dimensional microscopic state for the purpose of the questions we want to ask. The question here is whether the spins align ($|m|$ close to $1$, an ordered phase) or not ($m$ near $0$, a disordered phase), and how this depends on $\beta$, $J$, and $B$.

Mean-field approximation. We would like to push the distribution $p_{\beta }(s)$ over $2^N$ states forward to a distribution over the single variable $m$. The interaction term stands in the way, since it couples each spin to its particular neighbors. The mean-field approximation replaces the neighbors of each spin by their average: suppose that each spin feels only the average field of all the others, so that in the neighbor sum we may replace each $s_j$ by $m$,

$$\sum _{⟨ij⟩}{s}_i{s}_j=\frac{1}{2}\sum _i{s}_i\underset{\approx dm}{\underbrace{\sum _{j\sim i}{s}_j}}\approx \frac{dm}{2}\sum _i{s}_i=\frac{Nd}{2}{m}^2.$$

(The factor $\frac{1}{2}$ compensates for counting each edge twice.) The energy becomes a function of $m$ alone,

This is an uncontrolled approximation for a lattice in low dimension, where a spin's few neighbors fluctuate along with it, but it turns out to be qualitatively right for $D\ge 2$ and quantitatively right for $D\ge 4$ (Tong 2012, Chapter 1). It becomes exact for a model in which every spin couples weakly to every other; that model, the Curie–Weiss magnet, returns in Section 5.1 as the prototype of mean-field limits of neural networks.

Entropy, and the reduction to one variable. With the energy depending only on $m$, the sum over states can be organized by the value of $m$:

$$Z\approx \sum _m\Omega (m)e^{N\beta \left(\frac{dJ}{2}{m}^2+Bm\right)},\Omega (m):=\#\{s:m(s)=m\}=\left(\genfrac{}{}{0ex}{}{N}{\frac{1+m}{2}N}\right),$$

since a state with magnetization $m$ has exactly $\frac{1+m}{2}N$ spins up. Stirling's formula gives

the entropy per site at magnetization $m$: the log of the number of microscopic states compatible with the macroscopic value $m$. It is largest at $m=0$, where $h=\log 2$ and almost all of the $2^N$ states live, and it vanishes at $m=\pm 1$, where exactly one state lives. Replacing the sum over $m$ by an integral (the spacing is $2/N$), the partition function becomes a one-dimensional integral,

The function $s(m)$ is called the effective action, or the free-energy density at fixed $m$: energy minus temperature times entropy, per site. We have reduced the problem from $N$ variables to one. This is the general shape of every calculation in this section: push the microscopic distribution forward to an order parameter, at the price of an entropy term that counts how many microstates sit at each value.

Saddle points. At large $N$ the integrand $e^{-N\beta s(m)}$ is exponentially peaked at the minima of $s$, and the integral is dominated by a window of width $\sim 1/\sqrt{N}$ around the lowest one. This is the saddle-point approximation (also called Laplace's method), and it gives

The free energy is minimized, in exactly the sense promised in Section 2.3. The minimizer $\bar{m}$ is the magnetization the system actually displays. Setting $\partial s/\partial m=0$ and using $h^{\prime }(m)=-{\tanh }^{-1}(m)$,

This transcendental equation is best solved graphically (Figure 1). Depending on $\beta$, $J$, and $B$ it has one solution, or three; when there are three, the outer two are minima of $s$ and the middle one is a maximum.

Self-consistency (an aside). There is another route to (2.27), which is the form in which mean-field theory usually appears and which we will use in Section 5. Look at the effective field felt by a single spin $s_i$:

$$B_i^{\mathrm{eff}}=-\frac{\partial E}{\partial s_i}=J\sum _{j\sim i}{s}_j+B\approx dJm+B.$$

A single spin in this field has partition function $Z_i={\sum }_{s_i=\pm 1}{e}^{\beta B^{\mathrm{eff}}{s}_i}=2\cosh \beta (dJm+B)$ and mean $\mathbb{E}(s_i)={\beta }^{-1}{\partial }_B\log Z_i=\tanh \beta (dJm+B)$. If the approximation is to be consistent, the average that comes out must equal the average we put in, $m=\mathbb{E}(s_i)$, and that is (2.27). Solve the one-body problem in the average environment, then demand that the average be reproduced: this is the prototype of every mean-field theory.

2.5 Phases and phase transitions

Now we analyze the structure of the minima of $s(m)$, which control where the probability concentrates, as $\beta$, $J$, and $B$ are varied. Taylor-expanding the entropy, $h(m)=\log 2-\frac{m^2}{2}-\frac{m^4}{12}-\dots$, the action is

This expansion of a free energy in powers of an order parameter is called a Landau theory, and the qualitative analysis below depends only on its form, not on the microscopic model behind it.

The ordered phase and a first-order transition. Take $dJ>{\beta }^{-1}$: the coupling beats the temperature, the quadratic coefficient in (2.29) is negative, and $s(m)$ is a double well (Figure 2). The system wants to be ordered, with neighboring spins aligned; but aligned which way? The field decides. For $B<0$ the global minimum is near $m=-1$, for $B>0$ near $m=+1$, and at $B=0$ the two minima are exactly degenerate. As $B$ is swept through $B_c=0$, the magnetization the system displays jumps from $\bar{m}\approx -1$ to $\bar{m}\approx +1$. This is a first-order phase transition.

The structure of this transition is universal, in the sense that it follows from two isolated minima crossing and not from any detail of the model. Consider the expected magnetization, which by (2.14) is a derivative of the free energy with respect to the field:

$$\mathbb{E}(m)=\frac{1}{N}\sum _i\mathbb{E}(s_i)=-\frac{1}{\beta }\frac{\partial \phi }{\partial B}.$$

At finite $N$, the two minima of $s$ are two competing phases, and $\mathbb{E}(m)$ is a probability-weighted average of their magnetizations. As $N\to \infty$ the saddle-point approximation puts all the weight on the lower minimum, so $\mathbb{E}(m)$ develops a discontinuity at $B_c$, and $\phi$ itself develops a kink (Figure 3, left). A discontinuity in a first derivative of the free-energy density, in the thermodynamic limit, is the definition of a first-order transition. Its second derivative, the susceptibility

$$\chi :=\frac{\partial \mathbb{E}(m)}{\partial B}=-\frac{1}{\beta }\frac{{\partial }^2\phi }{\partial B^2}=N\beta \mathrm{Var}(m),$$

is a peak whose height grows like $N$ and whose width shrinks like $1/N$: a nascent delta function (Figure 3, right). Genuine non-analyticity can occur only at $N=\infty$: at finite $N$ the partition function is a finite sum of analytic functions of $B$, so everything is smooth, and what one sees is a rapid crossover that sharpens as $N$ grows. Watching the peak in $\chi$ grow with $N$, rather than saturate, is the standard numerical diagnostic that separates a true transition from a crossover. Exercise 2.2 works out the shapes of these curves for a general first-order transition; they are worth remembering whenever "emergent capabilities" in models of growing size are under discussion (Section 7.6).

Metastability and hysteresis. The suppressed minimum in Figure 2 does not disappear when it stops being the global one. A system that starts there and evolves by local dynamics (flipping one spin at a time) must climb over the barrier to reach the true minimum, which takes a time exponential in $N$. In practice the system stays in the metastable phase until the field is pushed well past $B_c$, and the transition observed dynamically lags the thermodynamic one. This is hysteresis, familiar from real magnets. It is the reason the "$\beta \leftrightarrow$ training time" analogy of Section 2.1 must be handled with care at first-order transitions, and we will meet it again in the guise of delayed generalization (grokking) in Section 5.7.

A second-order transition. An $n$-th order phase transition is a discontinuity in the $n$-th derivative of the free-energy density as $N\to \infty$. The Ising model has a second-order transition too, at fixed $B=0$, as the temperature $T={\beta }^{-1}$ crosses $T_c=dJ$ (Figure 4). Here nothing crosses anything: the single basin at $m=0$ flattens and splits into two symmetric ones. The quadratic coefficient in (2.29) changes sign at $T_c$, and the minima sit at

so the order parameter $\bar{m}\propto (T_c-T{)}^{1/2}$ moves continuously but not smoothly; $\phi$ and ${\partial }_T\phi$ are continuous, and the discontinuity sits in ${\partial }_T^2\phi$ (the heat capacity). The point $(T,B)=(T_c,0)$ is a critical point. Below it the two ordered states $\pm \bar{m}$ are exact mirror images, and the system must pick one: this is spontaneous symmetry breaking, the ${\mathbb{Z}}_2$ symmetry $s\to -s$ of the energy at $B=0$ being broken by the state. The exponent $\frac{1}{2}$ in (2.32) is an example of a critical exponent. Mean-field theory predicts it in every dimension, and it is wrong in low dimension: the exact value is $\frac{1}{8}$ in $D=2$ (Onsager), and there is no transition at all in $D=1$. Mean-field exponents become exact for $D\ge 4$. Understanding why is the subject of the renormalization group, for which see (Tong 2012).

Back to learning. In the Bayesian model of Section 2.1, the natural control parameter is $n\beta$, the amount of data or training. The Ising analogue is sweeping $\beta$ at fixed $B=0$, which sees the second-order transition. First-order transitions in learning arise when two qualitatively different families of solutions compete, one entropically favored and one favored by the data; the Ising perceptron of Section 2.6 is the cleanest example. Before turning to it, the following exercise establishes what every first-order transition looks like at finite $N$.

Exercise 2.2 (★) (Anatomy of a first-order transition). Consider any system whose partition function has the form $Z(B)=\int d\theta e^{-NE(\theta ,B)}$, depending on some hyperparameter $B$. Here $B$ could be the inverse temperature, an external field, or the load $\alpha$ of Section 2.6; we have absorbed $\beta$ into $E$ to keep the setup general. Suppose that $E(\theta ,B)$ has exactly two isolated minima, at ${\theta }_1$ and ${\theta }_2$. Then at large $N$ the integral is dominated by two competing phases. Letting $U_1,U_2$ be small neighborhoods of ${\theta }_1,{\theta }_2$,

$$Z(B)\approx {\int }_{U_1}d\theta e^{-NE(\theta ,B)}+{\int }_{U_2}d\theta e^{-NE(\theta ,B)}=e^{-N{\phi }_1(B)}+e^{-N{\phi }_2(B)},$$

with ${\phi }_i(B)=-\frac{1}{N}\log {\int }_{U_i}d\theta e^{-NE(\theta ,B)}=E({\theta }_i,B)+o(1)$ the free-energy densities of the two basins. Suppose further that as $B$ crosses a critical value $B_c$ the two minimum values cross: ${\phi }_1(B_c)={\phi }_2(B_c)$ with ${\phi }_1^{\prime }(B_c)\ne {\phi }_2^{\prime }(B_c)$, where a prime denotes ${\partial }_B$.

(a) Occupation. Show that the probability of finding the system in basin $i$ is

$$p_i(B)=\frac{e^{-N{\phi }_i(B)}}{e^{-N{\phi }_1(B)}+e^{-N{\phi }_2(B)}},$$

and that, expanding to first order in $\Delta B:=B-B_c$,

$$p_2(B)\approx \frac{1}{1+e^{-N\Delta {\phi }^{\prime }\Delta B}},\Delta {\phi }^{\prime }:={\phi }_1^{\prime }(B_c)-{\phi }_2^{\prime }(B_c),$$

a logistic function of $N\Delta {\phi }^{\prime }\Delta B$. Over what window of $B$ does the system switch allegiance?

(b) First derivative. Show from (2.33) that

$$\frac{\partial \phi }{\partial B}\approx p_1(B){\phi }_1^{\prime }(B_c)+p_2(B){\phi }_2^{\prime }(B_c),$$

the probability-weighted average of the branch slopes. Sketch it at finite $N$ and at $N=\infty$: a sigmoid of height $\Delta {\phi }^{\prime }$ sharpening into a jump. The kink in $\phi$ itself is the defining discontinuity of a first-order transition. Show also, directly from the integral, that ${\partial }_B\phi =\mathbb{E}[{\partial }_{B}E(\theta ,B)]$: the observable that jumps is the one conjugate to the control parameter. When $B=\beta$ it is the energy, and the jump is the latent heat; for the Ising model with $B$ the field, it is the magnetization.

(c) Second derivative. Using ${\partial }_B{p}_1=-{\partial }_B{p}_2=-N{p}_1{p}_2\Delta {\phi }^{\prime }$ (verify this), show that

$$-\frac{{\partial }^2\phi }{\partial B^2}=N{p}_1{p}_2(\Delta {\phi }^{\prime }{)}^2-p_1{\phi }_1^{\prime \prime }-p_2{\phi }_2^{\prime \prime }.$$

Sketch this curve. Near $B_c$, where $p_1{p}_2\approx \frac{1}{4}$, the first term is a peak of height $\sim N(\Delta {\phi }^{\prime }{)}^2/4$ and width $\sim 1/(N\Delta {\phi }^{\prime })$, which converges to a delta function as $N\to \infty$. Show also that when $\mathcal{O}(\theta )={\partial }_{B}E(\theta ,B)$ does not depend on $B$, $-{\partial }_B^2\phi =N\mathrm{Var}(\mathcal{O})\ge 0$. (For $B=\beta$ this is the heat capacity; for the Ising field it is the susceptibility.)

(d) Plot it. Take the minimal model ${\phi }_1(B)=a(B-B_c)$, ${\phi }_2=0$ and plot $\phi$, ${\partial }_B\phi$, and $-{\partial }_B^2\phi$ for a few values of $N$. (Use a numerically stable $\log (e^x+e^y)$.) Watching the peak grow with $N$ rather than saturate is the standard diagnostic separating a true transition from a smooth crossover.

Solution

(a) $p_i$ is the fraction of $Z$ contributed by basin $i$. With ${\phi }_1-{\phi }_2\approx \Delta {\phi }^{\prime }\Delta B$ near the crossing, $p_2=(1+e^{-N({\phi }_1-{\phi }_2)}{)}^{-1}\approx (1+e^{-N\Delta {\phi }^{\prime }\Delta B}{)}^{-1}$. The switch happens over the window $|\Delta B|\lesssim 1/(N\Delta {\phi }^{\prime })$, which shrinks with system size.

(b) Differentiating $\phi =-\frac{1}{N}\log (e^{-N{\phi }_1}+e^{-N{\phi }_2})$ gives ${\partial }_B\phi =p_1{\phi }_1^{\prime }+p_2{\phi }_2^{\prime }$ exactly; near $B_c$ the slopes are $\approx {\phi }_i^{\prime }(B_c)$. Directly from the integral, ${\partial }_B\phi =-\frac{1}{N}{\partial }_B\log Z=\frac{1}{Z}\int d\theta {\partial }_{B}E{e}^{-NE}=\mathbb{E}[{\partial }_{B}E]$: the control parameter is itself a source, coupling to $\mathcal{O}={\partial }_{B}E$. Sketch: at finite $N$, ${\partial }_B\phi$ is a sigmoid interpolating from ${\phi }_1^{\prime }(B_c)$ to ${\phi }_2^{\prime }(B_c)$ over the window of (a); as $N\to \infty$, $\phi \to \min ({\phi }_1,{\phi }_2)$, continuous with a kink at $B_c$, and ${\partial }_B\phi$ becomes a step of height $\Delta {\phi }^{\prime }$.

(c) ${\partial }_B{p}_1={\partial }_B\frac{e^{-N{\phi }_1}}{e^{-N{\phi }_1}+e^{-N{\phi }_2}}=-N{p}_1{p}_2({\phi }_1^{\prime }-{\phi }_2^{\prime })$ and ${\partial }_B{p}_2=-{\partial }_B{p}_1$. Then ${\partial }_B^2\phi =({\partial }_B{p}_1){\phi }_1^{\prime }+({\partial }_B{p}_2){\phi }_2^{\prime }+p_1{\phi }_1^{\prime \prime }+p_2{\phi }_2^{\prime \prime }=-N{p}_1{p}_2(\Delta {\phi }^{\prime }{)}^2+p_1{\phi }_1^{\prime \prime }+p_2{\phi }_2^{\prime \prime }$. Sketch: a smooth background from the ${\phi }_i^{\prime \prime }$ terms plus a peak at $B_c$ of height $\approx N(\Delta {\phi }^{\prime }{)}^2/4$ and width $\sim 1/(N\Delta {\phi }^{\prime })$, the nascent delta function of the $N=\infty$ kink. For the variance identity: with $\mathcal{O}={\partial }_{B}E$ independent of $B$, differentiating ${\partial }_B\phi =\mathbb{E}(\mathcal{O})$ once more gives ${\partial }_B^2\phi =-N[\mathbb{E}({\mathcal{O}}^2)-\mathbb{E}(\mathcal{O}{)}^2]$, so $-{\partial }_B^2\phi =N\mathrm{Var}(\mathcal{O})\ge 0$, which is why the curve is a peak and not a dip. The two-branch formula exhibits the leading piece of this variance: $\mathcal{O}\approx {\phi }_1^{\prime }$ in one basin and $\approx {\phi }_2^{\prime }$ in the other, and the variance of that two-point mixture is exactly $p_1{p}_2(\Delta {\phi }^{\prime }{)}^2$.

(d) See Figure 14. The peak in the right panel grows in proportion to $N$; a crossover's bump would saturate.

2.6 The Ising perceptron: a first-order transition in learning

We close the physics part of this section with a complete statistical-mechanics analysis of a learning problem, built to run in parallel with the Ising magnet. It is the simplest model we know in which more data produces a genuine phase transition to perfect generalization, and it goes back to Gyorgyi (Gy"orgyi 1990); see (Engel & Van den Broeck 2001, Chapter 7). It is presented as a walk-through exercise; solutions are in Appendix C, but try each part before reading them.

Setup. A teacher is a fixed vector of binary weights ${\theta }^{\ast }\in \{\pm 1{\}}^N$. Inputs are Gaussian, $x\sim \mathcal{N}(0,I_N)$, and the teacher labels them by

A student is another binary vector $\theta \in \{\pm 1{\}}^N$, guessing labels the same way, $f_{\theta }(x)=\mathrm{sign}(\theta \cdot x)$. This model is called the Ising perceptron, because its weights take the values of Ising spins. We observe $n$ examples $(x^{\mu },y^{\mu })$ with $y^{\mu }=y(x^{\mu })$, and do Bayesian learning at zero temperature with a uniform prior, in the sense of Section 2.2: the per-sample loss is $0$ if the student classifies the example correctly and $\infty$ otherwise, so the posterior is uniform on the students that classify every example correctly. Since teacher and student agree on $x$ exactly when $y(x)(\theta \cdot x)>0$,

with $\Theta (u)=1$ for $u>0$ and $0$ otherwise. With the factor $1/2^N$, $Z$ is the fraction of all students consistent with the data. Both $N$ and $n$ will be taken large, at fixed ratio: the load

the number of examples per parameter, is the control parameter of the problem.

Exercise 2.3 (★) (The Ising perceptron). (a) The order parameter is a magnetization. Define the overlap $R(\theta )=\frac{1}{N}\theta \cdot {\theta }^{\ast }\in [-1,1]$ and the alignment variables $s_i:={\theta }_i{\theta }_i^{\ast }\in \{\pm 1\}$. Show that $R=\frac{1}{N}{\sum }_i{s}_i$: the overlap is the magnetization of the spins $s_i$. Learning the teacher means magnetizing them to $R=1$.

(b) Generalization error. Show that a student at overlap $R$ misclassifies a fresh Gaussian input with probability

$$\epsilon (R)=\frac{\arccos R}{\pi }.$$

[Hint: project $x$ onto the plane spanned by $\theta$ and ${\theta }^{\ast }$; the projection is an isotropic two-dimensional Gaussian, and the two decision boundaries are diameters meeting at angle $\arccos R$. When do the signs disagree?]

(c) Entropy: the same $h$ as the magnet. Show that the number of students at overlap $R$ is $\left(\genfrac{}{}{0ex}{}{N}{\frac{1+R}{2}N}\right)$, so that by Stirling's formula

$$\begin{array}{rl}\frac{1}{N}\log \#\{\theta :R(\theta )=R\}\to h(R)& =\log 2-\frac{1-R}{2}\log (1-R)\\ & -\frac{1+R}{2}\log (1+R),\end{array}$$

identical to the magnet's entropy (2.24) with $m\to R$.

(d) Push forward to one variable. Average $Z$ over datasets. The examples are independent, and each is classified correctly by a student at overlap $R$ with probability $1-\epsilon (R)$, so

$${\mathbb{E}}_{x}Z=\frac{1}{2^N}\sum _{\theta }(1-\epsilon (R(\theta )){)}^n.$$

Trade the sum over $\theta$ for an integral over $R$ using (c), and show that at fixed load $\alpha$ and large $N$,

$${\mathbb{E}}_{x}Z\approx {\int }_{-1}^{1}dR{e}^{-Ns(R)},s(R)=\log 2-h(R)-\alpha \log (1-\epsilon (R)).$$

This is the exact analogue of the magnet's (2.25): an entropy term counting states, and a "field" term rewarding alignment. The aligning field is now the data, and its strength is the load $\alpha$. (Averaging $Z$ rather than $\log Z$ is the annealed approximation (2.18). It preserves all the qualitative structure here and shifts the numbers only slightly; Appendix A explains how to do the exact calculation.)

(e) Plot $s(R)$ for several values of $\alpha$ and locate its critical points. Do you expect a phase transition? At which $\alpha$?

(f) The teacher basin. $R=1$ is the perfectly aligned state, where the student is the teacher. Show that $s(1)=\log 2$ for every $\alpha$ (zero entropy, but zero data cost), and that $R=1$ is always a local minimum: writing $\delta =1-R$, deviating from the teacher gains entropy $\sim \frac{\delta }{2}\log (1/\delta )$ but costs data-fit $\alpha \epsilon \sim \alpha \sqrt{2\delta }/\pi$ (expand $\arccos$), and $\sqrt{\delta }\gg \delta \log (1/\delta )$. This differs from the magnet, where the aligned state sat at the bottom of a smooth well and was not always a minimum.

(g) The interior basin, the crossing, and the transition. At $\alpha =0$, $s$ is minimized at $R=0$ with $s(0)=0<s(1)$: the disordered cloud of typical students dominates. As $\alpha$ grows, this interior minimum $R^{\ast }(\alpha )$ drifts toward $1$ and its value rises. Numerically, find the load ${\alpha }_c$ at which the two basins exchange dominance, $s(R^{\ast }({\alpha }_c))=s(1)$. You should find ${\alpha }_c\approx 1.45$, with the interior minimum still at $R^{\ast }\approx 0.70$, that is, generalization error ${\epsilon }^{\ast }\approx 0.25$. Conclude that at ${\alpha }_c$ the typical student jumps from $25\%$ error to zero error: a first-order transition to perfect generalization. Then connect to Exercise 2.2 with $\alpha$ as the control parameter: what do ${\partial }_{\alpha }\phi$ and $-{\partial }_{\alpha }^2\phi$ look like at finite $N$, and what plays the role of the latent heat? Finally, find the load at which the interior minimum disappears altogether, and explain what happens in between in the light of the metastability discussion of Section 2.5.

(h) The dictionary. Complete the table:

Ising magnet	Ising perceptron
spins $s_i\in \{\pm 1\}$	alignments $s_i={\theta }_i{\theta }_i^{\ast }\in \{\pm 1\}$
magnetization $m$	overlap $R$
entropy $h(m)$	entropy $h(R)$ (the same function)
aligning field $B$	?
knob swept across the transition	?
ordered phase $\mathbb{E}(m)\approx \pm 1$	?
jump in $\mathbb{E}(m)$ as a function of $B$	?

Also articulate one structural difference: in the magnet the two competing minima are symmetry partners; what are they here, and which ingredient of (c) made a sharp transition possible at all? (What would change for spherical weights $\theta \in {\mathbb{R}}^N$, $|\theta {|}^2=N$, where the slice at overlap $R$ has volume $\propto (1-R^2{)}^{N/2}$? Appendix A.3 works this case out.)

Solution

(a) $R=\frac{1}{N}{\sum }_i{\theta }_i{\theta }_i^{\ast }=\frac{1}{N}{\sum }_i{s}_i$ since $({\theta }_i^{\ast }{)}^2=1$. The map $\theta \mapsto s$ is a bijection of the hypercube (a "gauge transformation"), so the uniform measure goes to the uniform measure, and the perceptron's order parameter is literally a magnetization.

(b) Both labels depend only on the projection of $x$ onto $\mathrm{span}(\theta ,{\theta }^{\ast })$, which is an isotropic two-dimensional Gaussian. The decision boundaries are two diameters at angle $\vartheta =\arccos R$; the signs disagree exactly on the double wedge between them, of angular fraction $2\vartheta /2\pi =\vartheta /\pi$. Note that the answer does not depend on $N$, and holds for any $\theta$ with $|\theta {|}^2=N$, binary or not.

(c) A student at overlap $R$ agrees with the teacher on exactly $\frac{1+R}{2}N$ coordinates ($s_i=+1$), giving the binomial count; Stirling gives $h(R)$, the binary entropy at magnetization $R$. Note $h(0)=\log 2$ and $h(1)=0$: the latter is finite, because exactly one state sits at $R=1$.

(d) Independence over examples gives the $n$-th power. Trading the sum for an integral, $\frac{1}{2^N}{\sum }_{\theta }\to \int dR{e}^{N[h(R)-\log 2]}$, and $(1-\epsilon {)}^n=e^{N\alpha \log (1-\epsilon )}$; collecting exponents gives $s(R)$ as stated. (The $\log 2$ normalizes $s(0)=0$ at $\alpha =0$.)

(e) The curves are shown in Figure 15. At $\alpha =0$, $s=\log 2-h(R)$ has a single minimum at $R=0$, the bulk of the hypercube. As $\alpha$ grows, the interior minimum drifts toward larger $R$ and its value rises (note $s(0)=\alpha \log 2$), while $R=1$ stays pinned at $s(1)=\log 2$. For $\alpha$ large enough there are two competing local minima separated by a barrier: exactly the two-basin structure of Exercise 2.2. One should therefore expect a first-order transition at the load where the interior minimum's value crosses $\log 2$, near $\alpha \approx 1.45$ from the plot; at $\alpha =1.7$ the interior minimum survives only as a metastable dip above $\log 2$. No transition is expected at small $\alpha$, where the interior minimum is the only relevant one and moves smoothly.

(f) At $R=1$: $h(1)=0$ and $\epsilon (1)=0$, so $s(1)=\log 2$ for all $\alpha$. Near $R=1$ with $\delta =1-R$: $h(1-\delta )=\frac{\delta }{2}\log \frac{2}{\delta }+O(\delta )$, while $\arccos (1-\delta )\approx \sqrt{2\delta }$ gives data cost $-\alpha \log (1-\epsilon )\approx \alpha \sqrt{2\delta }/\pi$. Since $\sqrt{\delta }\gg \delta \log (1/\delta )$ as $\delta \to 0$, $s(1-\delta )>s(1)$: a basin around the teacher at every $\alpha >0$. It is protected by the singular cost of the first errors, which is what a smooth well cannot provide.

(g) Minimizing $s$ over the interior numerically (Figure 16, left), the interior branch value crosses $s(1)=\log 2$ at ${\alpha }_c\approx 1.45$, with minimizer $R^{\ast }\approx 0.70$ and ${\epsilon }^{\ast }=\arccos (0.70)/\pi \approx 0.25$. The order parameter jumps from $0.70$ to $1$ (middle panel). With $\alpha$ as the control parameter of Exercise 2.2: $\phi (\alpha )={\min }_{R}s(R)$ has a kink at ${\alpha }_c$; at finite $N$, ${\partial }_{\alpha }\phi =\mathbb{E}[-\log (1-\epsilon (R))]$ is a sigmoid interpolating between the branch slopes $-\log (1-\epsilon (R^{\ast }))\approx 0.29$ and $0$, over a window of width $\sim 1/(N\Delta {\phi }^{\prime })$ (right panel), and $-{\partial }_{\alpha }^2\phi$ is a peak of height $\propto N$. The role of the latent heat is played by the jump in $-\log (1-\epsilon )$, the per-example log-likelihood released when the student snaps onto the teacher. Past ${\alpha }_c$ the interior minimum survives as a local minimum until the spinodal $\alpha \approx 1.73$, where it merges with the barrier and disappears. In the window $1.45\lesssim \alpha \lesssim 1.73$ a local learning dynamics (flipping a few weights at a time to improve alignment) can stay stuck at $\epsilon \approx 0.25$ even though perfect generalization is thermodynamically dominant: the information needed to identify the teacher is present in the data before the dynamics can find it, and generalization arrives abruptly and later than the data first permits. This is the metastability of Section 2.5, and a prototype of the delayed transitions of Section 5.7. (The exact, quenched treatment (Gy"orgyi 1990; Engel & Van den Broeck 2001) moves the transition to ${\alpha }_c\approx 1.245$ but changes nothing structurally.)

(h) Field $B\leftrightarrow$ the data term, the per-example log-likelihood $\log (1-\epsilon (R))$, with strength $\alpha$; the knob swept $=$ the load $\alpha =n/N$, the amount of data; the ordered phase $\leftrightarrow$ perfect generalization $R=1$; the jump in $\mathbb{E}(m)$ as a function of $B$ $\leftrightarrow$ the discontinuous drop of $\epsilon (R^{\ast })$ from $0.25$ to $0$ as a function of $\alpha$. The structural difference: in the magnet the two competing minima are symmetry partners, while here they are an entropy-dominated basin and a data-dominated basin. The ingredient that makes the sharp transition possible is $h(1)=0$ being finite: discreteness gives the perfectly aligned state a finite entropy, so it can compete as a separate basin. For spherical weights the slice entropy $\frac{1}{2}\log (1-R^2)$ diverges to $-\infty$ as $R\to 1$, the perfectly aligned state is infinitely penalized, the minimum always sits in the interior and slides continuously with $\alpha$, and there is no transition; the generalization error then decays smoothly, as $\epsilon \simeq 0.625/\alpha$ in the exact treatment (Appendix A.3).

2.7 Comparison with QFT

This short subsection is a translation guide, aimed mainly at readers arriving from high-energy or condensed-matter physics — if the words "action" and "path integral" mean more to you than "posterior" and "marginal likelihood," read on; everyone else may safely skip ahead.

Nearly everything in these notes lives, in the first instance, in Euclidean equilibrium statistical mechanics: probability distributions of Boltzmann–Gibbs form $P\propto e^{-\beta E}$, partition functions, free energies. For a field theorist, a partition function with sources,

$$Z[J]=\int \mathcal{D}\varphi e^{-S[\varphi ]+\int J\varphi },$$

is a moment generating functional; its logarithm generates connected correlation functions (cumulants); the action $S$ encodes the microscopic rules of the system, and a quadratic action means a free (Gaussian) theory, with everything else treated by perturbing around it. In these notes the role of the field $\varphi$ is played variously by the weights $\theta$ of a network, by an order parameter such as the magnetization $m$ or the overlap $R$ of Sections 2.4 and 2.6, or — most usefully — by the network function $f(x)$ itself, with the input $x$ playing the role of the spacetime coordinate. Functional integrals over the network function are developed carefully in Sections 3.3 and 3.5, and the reader coming from QFT will find many familiar ingredients there dressed up as learning theory: Wick's theorem, sources, cumulants, effective actions.

If your instincts were trained on Lorentzian signature, picture everything here as already Wick-rotated. There is no $i/\hslash$: the weights $e^{-S}$ are real and positive, as after the standard continuation

$$Z_{\mathrm{QFT}}=\int \mathcal{D}\varphi e^{\frac{i}{\hslash }S[\varphi ]}\stackrel{\tau =it}{\to }{Z}_{\mathrm{E}}=\int \mathcal{D}\varphi e^{-S_E[\varphi ]/\hslash }.$$

The role of $\hslash$ — the parameter controlling the size of fluctuations around saddle points — is played here by temperature, or by $1/N$ at large width, or by $1/n$ at large sample size.

When we eventually study the dynamics of training as a field theory in Section 6, a genuinely real-time formalism does appear — a path integral over trajectories with a causal structure and a response field. That's the closest object in these notes to a Lorentzian field theory, and is precisely the Martin–Siggia–Rose construction of non-equilibrium statistical mechanics.

For readers who would like physics references alongside the learning-theory ones: David Tong's lecture notes (Tong 2012; Tong 2012; Tong 2012; Tong 2006) are a friendly introduction to statistical physics, statistical field theory, kinetic theory, and QFT respectively; Lancaster and Blundell (Lancaster & Blundell 2014) is a gentle conceptual route into QFT; Altland and Simons (Altland & Simons 2010) bridges toward condensed-matter field theory (and contains the Langevin/MSRJD material used in Section 6); Pathria (Pathria & Beale 2011) is a standard graduate statistical mechanics text; Goldenfeld (Goldenfeld 2018) is the classic on phase transitions and the renormalization group; and Zinn-Justin (Zinn-Justin 2002) is the encyclopedic reference for the functional formalism.

3. Large width at initialization: the Gaussian process limit

The next few sections are all about different limits in neural networks, in which some parameter(s) become large. Such limits are where physical/statistical analyses really shine. Studying them teaches some basic lessons about how we should (and should not) initialize networks, and how to choose other hyperparameters. It also leads to analytically solvable models that display many properties we see in practice: lazy vs. rich learning, double descent, phase transitions, scaling laws for optimal learning. For further pedagogical references on much of this material, we recommend (Roberts et al. 2022; Spiliopoulos et al. 2025; Ringel et al. 2025) and the online lectures (Hanin 2026).

We'll begin in this section with a classic result (Neal 1996; Lee et al. 2018; Yang 2019; Hanin 2023) about increasing the width of a network at fixed depth and training-sample size.

Modern neural networks are more sophisticated than the simple MLP's discussed here, but they share many of the same features. Layer norms solve the problem of trainability at large depth, but the analysis of rich/lazy regimes, feature learning, and hyperparameter scaling is still relevant. See e.g. the discussion of $\mu P$ in Section 5.5.

3.1 Notation

We'll use the following notation for neural networks and training data.

• $\theta \in {\mathbb{R}}^P$ trainable network parameters (weights, biases)
• $(x^{\mu },y^{\mu }{)}_{\mu =1}^n$ training data samples, with $x^{\mu }\in {\mathbb{R}}^d$ and $y^{\mu }\in \mathbb{R}$ (unless otherwise stated)
• $f_{\theta }:{\mathbb{R}}^d\to \mathbb{R}$ network function

For MLP's in particular (also called FCN's = fully connected networks)

• $\{z_i^{(\ell )}{\}}_{i=1}^{N_{\ell }}$ pre-activations in layer $\ell \in \{0,...,L+1\}$
• $L=$ number of hidden layers (depth); $N_{\ell }=$ width of $\ell$'th layer; $N_0=d$ and $N_{L+1}=1$. We will often take constant hidden depth, $N_1=...=N_L=N$.
• $\varphi :\mathbb{R}\to \mathbb{R}$ activation function (assumed the same in all layers except the first)
• network recursion $z_i^{(\ell )}=\{\begin{array}{ll}\frac{1}{\sqrt{N_{\ell -1}}}{\sum }_{j=1}^{N_{\ell -1}}{w}_{ij}^{(\ell )}\varphi (z_j^{(\ell -1)})+b_i^{(\ell )}& \ell \ge 2\\ \frac{1}{\sqrt{d}}{\sum }_{j=1}^d{w}_{ij}^{(1)}{x}_j+b_i^{(1)}& \ell =1\end{array}$ with $z^{(0)}=x$ and $z^{(L+1)}=f$; parameters are $\theta =\{w^{(\ell )},b^{(\ell )}{\}}_{\ell =1}^{L+1}$ . The explicit factors of $1/\sqrt{N_{\ell -1}}$ are a normalization convention, explained in Section 3.2: with them in place, all weights and biases can be taken of order one.

3.2 Normalizing the network

Consider a fully connected network of depth $L$ with layer widths $N_{\ell }$ as in Section 3.1. Suppose we keep $L$ fixed but make the $N_{\ell }$ very large. Suppose also that the network is initialized randomly, with all weights and biases drawn independently from normal distributions of order-one variance,

with ${\sigma }_w,{\sigma }_b$ fixed as the widths grow. (Here and throughout, $\mathcal{N}(0,\sigma )$ denotes a Gaussian of variance $\sigma$. Letting the variances depend on the layer changes nothing below.) The question is how to normalize the sums in the network recursion so that preactivations stay of order one from layer to layer. From

$$z_i^{(\ell )}=\frac{1}{\sqrt{N_{\ell -1}}}{w}_i^{(\ell )}\cdot \varphi (z^{(\ell -1)})+b_i^{(\ell )}$$

(where we've written $w_i=(w_{i,1},...,w_{i,N_{\ell -1}})$ and $z^{(\ell -1)}=(z_1^{(\ell -1)},...,z_{N_{\ell -1}}^{(\ell -1)})$ as vectors) we get

$$\begin{array}{rl}\text{Var}(z_i^{(\ell )})& =\frac{1}{N_{\ell -1}}\sum _{j=1}^{N_{\ell -1}}\text{Var}\left(w_{ij}^{(\ell )}\right)\mathbb{E}\left[\varphi \right(z_j^{(\ell -1)}{)}^2]+\text{Var}(b_i^{(\ell )})\\ & ={\sigma }_w\cdot \frac{1}{N_{\ell -1}}\sum _{j=1}^{N_{\ell -1}}\left(\text{Var}\right(\varphi (z_j^{(\ell -1)}))+\mathbb{E}[\varphi (z_j^{(\ell -1)}){]}^2)+{\sigma }_b.\end{array}$$

Here we've conditioned on the previous layer, and used that for independent mean-zero $X$ and arbitrary $Y$, $\text{Var}(XY)=\text{Var}(X)\mathbb{E}[Y^2]$, along with additivity of variance for independent summands. The average in the second line involves only the activation function and the order-one preactivations of the previous layer, so it stays of order one as $N_{\ell -1}\to \infty$. (We'll sharpen this estimate in Section 3.7.) Thus

The prefactor $1/\sqrt{N_{\ell -1}}$, the square root of the number of inputs (the "fan-in") feeding each neuron, is exactly what keeps the preactivations of order one at large width. We call this the fan-in normalization; it goes back to LeCun and collaborators in the 90's (LeCun et al. 1998). In Section 4 it will also be called the NTK normalization, and in Section 5 we will meet the alternative mean-field normalization, in which the output layer carries $1/N$ instead of $1/\sqrt{N}$.

3.3 Gaussian processes

A Gaussian process $f:{\mathbb{R}}^m\to \mathbb{R}$ with mean function $\mu$ and covariance kernel $K$ is a random function such that on any $p$ inputs $(x^1,...,x^p)\in ({\mathbb{R}}^m{)}^p$ the vector of function values is Gaussian

$$\left(\begin{array}{c}f(x^1)\\ ⋮\\ f(x^p)\end{array}\right)\sim \mathcal{N}\left(\left(\begin{array}{c}\mu (x^1)\\ ⋮\\ \mu (x^p)\end{array}\right),\left(\begin{array}{ccc}K(x^1,x^1)& \cdots & K(x^1,x^p)\\ ⋮& \ddots & ⋮\\ K(x^p,x^1)& \cdots & K(x^p,x^p)\end{array}\right)\right).$$

Here $\mu :{\mathbb{R}}^m\to \mathbb{R}$ and $K:{\mathbb{R}}^m\times {\mathbb{R}}^m\to \mathbb{R}$ are fixed, deterministic functions that control the mean and covariance of any tuple of values of $f$. The function $K$, often called the kernel of the process, is symmetric: $K(x,x^{\prime })=K(x^{\prime },x)$.

Let `$dx$' denote a measure on ${\mathbb{R}}^m$ (it need not be the usual Lebesgue measure). If $K$, as integral kernel, has an inverse $K^{-1}$ satisfying

$$\int d{x}^{\prime }{K}^{-1}(x,x^{\prime })K(x^{\prime },x^{\prime \prime })=\delta (x-x^{\prime \prime })$$

then there's also a "physics definition" of a Gaussian process. It's not entirely rigorous, as it invokes multi-dimensional functional integrals. Mathematically, it's best thought of as a heuristic tool, which comes with a package of axiomatic manipulations whose actual results can (and should) all be fully justified by other means. The manipulations are all quite intuitive, though, and familiar to quantum field theorists, so we mention this formalism here. The idea is that, just like a single Gaussian variable `$z$' has a probability measure

$$P(z)=\frac{1}{Z}{e}^{-\frac{1}{2}{K}^{-1}(z-\mu {)}^2},Z=\int dz{e}^{-\frac{1}{2}{K}^{-1}(z-\mu {)}^2}=\sqrt{2\pi K},$$

a Gaussian process $f$ has a (non-rigorous) measure

$$\begin{array}{rl}P[f]& =\frac{1}{Z}{e}^{-\frac{1}{2}\int dxd{x}^{\prime }(f(x)-\mu (x))K^{-1}(x,x^{\prime })(f(x^{\prime })-\mu (x^{\prime }))},\\ Z& \text{‘=’}\int Df{e}^{-\frac{1}{2}\int dxd{x}^{\prime }(f(x)-\mu (x))K^{-1}(x,x^{\prime })(f(x^{\prime })-\mu (x^{\prime }))}.\end{array}$$

If the kernel and the measure theory on the input space ${\mathbb{R}}^m$ are nice enough to allow $K$ to be diagonalized, with a discrete basis of orthonormal eigenfunctions $\{{\varphi }_k(x){\}}_{k=1}^{\infty }$ and eigenvalues ${\lambda }_1\ge {\lambda }_2\ge \dots >0$ (so that $K^{-1}$ has eigenvalues $1/{\lambda }_k$), then $P[f]$ can be understood as an infinite-dimensional Gaussian. Writing $f(x)-\mu (x)={\sum }_{k=1}^{\infty }{a}_k{\varphi }_k(x)$,

$$P[f]=\frac{1}{Z}{e}^{-\frac{1}{2}{\sum }_{k=1}^{\infty }{a}_k^2/{\lambda }_k},Z=\prod _{k=1}^{\infty }(2\pi {\lambda }_k{)}^{1/2}.$$

One virtue of the physics formalism is that it organizes expectation values of products of $f(x)$'s. Let $\mu =0$ for simplicity (to generalize to any $\mu$, just shift $f\mapsto f-\mu$ below). Let $J:{\mathbb{R}}^m\to \mathbb{R}$ be an arbitrary (nice...) function and define

The integral evaluates formally to

Then expectation values can be expressed via derivatives, which by (3.12) become

The sum is over partitions of $\{1,...,p\}$ into pairs, and vanishes if $p$ is odd. Eqn. (3.14) is called Wick's Theorem in QFT, Isserlis's Theorem in probability (and it's correct, despite the non-rigorous derivation here).

Exercise 3.1 (Wick's theorem). If you haven't seen this before, derive (3.12) by completing the square in (3.11), and then derive (3.14) by differentiating. Check the first two cases explicitly: $\mathbb{E}[f(x^1)f(x^2)]=K(x^1,x^2)$, and $\mathbb{E}[f(x^1)f(x^2)f(x^3)f(x^4)]=K^{12}{K}^{34}+K^{13}{K}^{24}+K^{14}{K}^{23}$ (with $K^{ij}=K(x^i,x^j)$). How many terms appear at $p=6$?

Solution

Complete the square: shifting $f\to f+KJ$ (schematically, $f(x)\to f(x)+\int d{x}^{\prime }K(x,x^{\prime })J(x^{\prime })$) decouples the source, leaving $Z[J]=Z[0]e^{\frac{1}{2}\int JKJ}$. Each derivative $\delta /\delta J(x^i)$ acting on the exponential either brings down a factor $\int K(x^i,\cdot )J$ (which must later be hit by another derivative, else it dies at $J=0$) or closes a pair by hitting a previously produced factor, contributing $K(x^i,x^j)$. Every surviving term therefore corresponds to a perfect pairing of $\{1,\dots ,p\}$. For $p=2$ this gives $K(x^1,x^2)$; for $p=4$, the three pairings $K^{12}{K}^{34}+K^{13}{K}^{24}+K^{14}{K}^{23}$; for $p=6$ there are $5!!=15$ terms; in general $(p-1)!!$.

Another virtue of the physics formalism is that it neatly connects higher cumulants to deviations from Gaussianity. In general, the $p$-th cumulant ${\kappa }_p(X_1,...,X_p)$ of random variables $X_1,...,X_p$ is defined recursively via

$$\mathbb{E}(X_1\cdots X_p)=\sum _{\text{partition Π of {1,...,p}}}\prod _{\text{blocks {i1,...,ik}⊂Π}}{\kappa }_k(X_{i_1},...,X_{i_k})$$

For example, ${\kappa }_1(X)=\mathbb{E}(X)$, ${\kappa }_2(X_1,X_2)=\text{Cov}(X_1,X_2)=\mathbb{E}(X_1{X}_2)-\mathbb{E}(X_1)E(X_2)$. In physics, the ${\kappa }_p$'s are called connected correlation functions; they measure the part of a joint expectation that's not captured by expectations of subsets of the variables involved. For a Gaussian process,

$${\kappa }_2(f(x^1),f(x^2))=K(x^1,x^2);{\kappa }_p(f(x^1),...,f(x^p))=0\text{for all p>2},$$

as can easily be seen directly from (3.14), where joint expectations for $p>2$ all factor into products of pairs. Thus, nonvanishing higher ${\kappa }_p$'s are a measure of non-Gaussianity.

This appears in the functional-integral formalism as follows. Suppose that $f:{\mathbb{R}}^m\to \mathbb{R}$ is a random function that's governed by a functional probability distribution

$$P[f]=\frac{1}{Z}{e}^{-S[f]},$$

for some functional $S:\{\text{functions }f\}\to \mathbb{R}$. (In physics $S$ is called the action.) Then just as in (3.11), we can introduce the generating function

$$\begin{array}{rl}Z[J]& :=\int Df{e}^{-S[f]+\int dxJ(x)f(x)}\\ \Rightarrow \mathbb{E}[f(x^1)...f(x^p)]& =\frac{1}{Z[0]}\frac{\delta }{\delta J(x^1)}...\frac{\delta }{\delta J(x^p)}Z[J]{|}_{J=0}\end{array}$$

The cumulants appear naturally as derivatives of $\log Z[J]$, otherwise known as the (sourced) free energy:

$${\kappa }_p(f(x^1),...,f(x^p))=\frac{\delta }{\delta J(x^1)}...\frac{\delta }{\delta J(x^p)}\log Z[J]{|}_{J=0},$$

which in turn means (formally) that

$$Z[J]=\exp \sum _{p=1}^{\infty }\frac{1}{p!}\int d{x}^1...d{x}^p{\kappa }_p(f(x^1),...,f(x^p))J(x^1)...J(x^p),$$

a generating function of cumulants. The original $S[f]$ is the inverse Legendre transform of $\log Z[J]$ (by definition of $Z[J]$). Expanded as a formal series in the cumulants (assuming they are small), and assuming $\mathbb{E}[f(x)]=0$ for simplicity, it takes the form

$$\begin{array}{rl}S[f]=& \frac{1}{2}\int d{x}^{1}d{x}^2{\kappa }_2^{-1}(x^1,x^2)f(x^1)f(x^2)\\ & -\int d{x}^{1}d{x}^{2}d{x}^{3}d{y}^{1}d{y}^{2}d{y}^3{\kappa }_2^{-1}(x^1,y^1){\kappa }_2^{-1}(x^2,y^2)\\ & \times {\kappa }_2^{-1}(x^3,y^3){\kappa }_3(y^1,y^2,y^3)f(x^1)f(x^2)f(x^3)\\ & -\int d{x}^{1}d{x}^{2}d{x}^{3}d{x}^4({\kappa }_2^{-1}{)}^4[{\kappa }_4-{\kappa }_3{\kappa }_2^{-1}{\kappa }_3\text{terms}]\\ & \times f(x^1)f(x^2)f(x^3)f(x^4)-\dots \end{array}$$

Physicists might recognize this as a sum of Feynman diagrams, with cumulants giving the vertices. A key structure is that the $p$-th order term in the effective action contains only ${\kappa }_p$ and lower cumulants. In particular, $f$ is a Gaussian process if and only if $S[f]$ is quadratic; and small deviations from Gaussianity, measured by cumulants, provide the higher-order terms in $S[f]$.

3.4 The NNGP correspondence

A classic result in the theory of neural networks is that, at fixed depth and large width, with the fan-in normalization of Section 3.2, the network function at initialization converges to a Gaussian process. The theorem appeared in the 90's (Neal 1996) for a single hidden layer, and was generalized more recently to deeper networks and different architectures, cf. (Lee et al. 2018; Yang 2019; Hanin 2023).

In this theorem, the source of stochasticity in the network function comes from the random initializations of network parameters. One can think of randomly initializing a swarm of networks and asking what sorts of values the network function outputs. The result is significant because of its universality, and because it feeds into (and greatly simplifies) further analyses of training.

The derivation proceeds via induction. We'll just sketch it here, for MLP's.

²

For simplicity we'll assume that biases have been set to zero, and that all weights have the same order-one variance ${\sigma }_w$, as in (3.1), with the fan-in normalization of Section 3.2.

In the first network layer, the preactivations are

$$z_i^{(1)}(x)=\frac{1}{\sqrt{d}}\sum _{j=1}^d{w}_{ij}^{(1)}{x}_j,w_{ij}^{(1)}\sim \mathcal{N}(0,{\sigma }_w).$$

At fixed $x$, each $z_i^{(1)}(x)$ is a sum of $d$ i.i.d. Gaussian variables (with coefficients $x_j$), and thus is Gaussian itself, with $\mathbb{E}[z_i^{(1)}]=0$ and

$$\text{Cov}(z_i^{(1)}(x),z_i^{(1)}(x^{\prime }))=\frac{1}{d}\sum _{j,j^{\prime }=1}^d\underset{{\delta }_{j{j}^{\prime }}{\sigma }_w}{\underbrace{\text{Cov}(w_{ij},w_{i{j}^{\prime }})}}{x}_j{x}_{j^{\prime }}^{\prime }={\sigma }_w\frac{x\cdot x^{\prime }}{d},$$

thus $z_i^{(1)}$ is a Gaussian process (GP) with

$$z_i^{(1)}\sim \mathcal{N}(0,K^{(1)}),K^{(1)}(x,x^{\prime })={\sigma }_w\frac{x\cdot x^{\prime }}{d}.$$

In the second layer, we've got

If we condition on the preactivations of the first layer, the same argument gives

Thus we again get a Gaussian process. The problem is that if we don't condition on the preactivations, the kernel of this GP is not deterministic, but becomes a random variable itself: $\varphi (z^{(1)})\cdot \varphi (z^{(1)}{}^{\prime })$. This is where the infinite-width limit comes in. Heuristically, the central limit theorem implies that the sum of products of (non-Gaussian!) random variables on the LHS of (3.25) will become Gaussian as $N_1\to \infty$. More concretely, we can calculate and bound the variance

$$\begin{array}{rl}\text{Var}(K^{(2)}(z^{(1)},z^{(1)}{}^{\prime }))& =\frac{{\sigma }_w^2}{N_1^2}\text{Var}(\varphi (z^{(1)})\cdot \varphi (z^{(1)}{}^{\prime }))\\ & =\frac{{\sigma }_w^2}{N_1^2}{N}_1\cdot \text{Var}(\varphi (z_1^{(1)})\varphi (z_1^{(1)}{}^{\prime }))\to 0\text{as N1→∞}\end{array}$$

as long as the first-layer preactivations are order one (independent of $N_1$), and the activation function is reasonable (any standard choice will be). Thus, as $N_1\to \infty$, we can replace the stochastic kernel $K^{(2)}$ by its expectation,

$${\bar{K}}^{(2)}(x,x^{\prime }):={\sigma }_w{\mathbb{E}}_{z_1^{(1)}(x),z_1^{(1)}(x^{\prime })\sim \mathcal{N}(0,K^{(1)})}[\varphi (z_1^{(1)}(x))\varphi (z_1^{(1)}(x^{\prime }))]$$

Each subsequent layer is the same. At infinite width, the kernels become deterministic, and obey a recursion

with ${\bar{K}}^{L+1}(x,x^{\prime })$ ultimately governing a GP for the network function $f(x)$.

Exercise 3.2 (The NNGP correspondence, empirically (lab)). Fix two unit-norm inputs $x,x^{\prime }\in {\mathbb{R}}^d$ with, say, $d=8$ and $x\cdot x^{\prime }=0.5$. For widths $N\in \{3,30,300\}$, sample $M=10^4$ independent initializations of the one-hidden-layer erf network $f(x)=\frac{1}{\sqrt{N}}{\sum }_{i=1}^N{c}_i\mathrm{erf}(a_i\cdot x/\sqrt{d})$ with $c_i,a_{ij}\sim \mathcal{N}(0,1)$ (that is, ${\sigma }_w=1$).

(a) Make a histogram of the values $f(x)$ across the ensemble, and overlay the Gaussian density with the variance ${\bar{K}}^{(2)}(x,x)$ predicted by (3.32); watch the visibly non-Gaussian $N=3$ histogram converge.

(b) Make a scatter plot of the pairs $(f(x),f(x^{\prime }))$ and compare the sample $2\times 2$ covariance with the predicted kernel matrix.

(c) For each initialization, record the stochastic kernel $K^{(2)}$ of (3.26), and verify that its variance across initializations decays like $1/N$ (that's the mechanism by which the kernel becomes deterministic).

Solution

Everything is closed-form thanks to the erf kernel (3.32), which is the point of choosing erf. Expected observations:

(a) at $N=3$ the histogram has visibly heavy tails and excess kurtosis (the output is a sum of only three i.i.d. non-Gaussian terms); at $N=30$ deviations are still detectable; at $N=300$ the histogram is indistinguishable from the predicted Gaussian at this sample size.

(b) The sample covariance matches the $2\times 2$ kernel matrix built from (3.32) to within the Monte Carlo error $O(M^{-1/2})$.

(c) A log–log plot of $\mathrm{Var}(K^{(2)})$ against $N$ has slope $-1$, as in the variance computation of Section 3.4. The whole lab is $\sim$20 lines of NumPy; the one subtlety is to histogram over initializations at fixed inputs, not over inputs.

3.5 NNGP via functional integrals

One can also analyze large width at initialization via functional integrals. This just reorganizes the above steps. We'll illustrate how it works for a network with one hidden layer and zero biases, $N_2=1,N_1=N,N_0=d$,

$$f(x)=\frac{1}{\sqrt{N}}\sum _{i=1}^N{c}_i\varphi (\frac{1}{\sqrt{d}}{a}_i\cdot x),c_i\in \mathbb{R},a_i,x\in {\mathbb{R}}^d.$$

The initial distribution of weights, all of variance ${\sigma }_w$, is

$$P(c,a)=\frac{1}{Z}{e}^{-\frac{1}{2{\sigma }_w}|c{|}^2-\frac{1}{2{\sigma }_w}|a{|}^2}.$$

To push this forward to a distribution on the first-layer preactivations $z_i=\frac{1}{\sqrt{d}}{a}_i\cdot x$, we integrate against a delta function:

Here the $z_i(x)$ are arbitrary functions of $x$, and are only tied to the network values $\frac{1}{\sqrt{d}}{a}_i\cdot x$ via the delta functions. Eqn. (3.35) represents a functional probability density on the functions $z_i$. To evaluate it, introduce a second set of "Lagrange multiplier" functions ${\tilde{z}}_i$, and write

$$\delta (z_i(x)-\frac{1}{\sqrt{d}}{a}_i\cdot x)=\int D{\tilde{z}}_i{e}^{i\int dx{\tilde{z}}_i(x)(z_i(x)-\frac{1}{\sqrt{d}}{a}_i\cdot x)},$$

and then do the exact Gaussian integral over $a$'s, and then over $\tilde{z}$'s:

$$\begin{array}{rl}P(c,z)& =\frac{1}{Z}\int daD\tilde{z}\exp [-\frac{1}{2{\sigma }_w}|c{|}^2-{\sum }_i\frac{1}{2{\sigma }_w}|a_i{|}^2\\ & +i{\sum }_i\int dx{\tilde{z}}_i(x)(z_i(x)-\frac{1}{\sqrt{d}}{a}_i\cdot x)]\\ & =\frac{1}{Z}\int daD\tilde{z}\exp [-\frac{1}{2{\sigma }_w}|c{|}^2+{\sum }_i[-\frac{1}{2{\sigma }_w}(a_i+\frac{i{\sigma }_w}{\sqrt{d}}\int dx{\tilde{z}}_i(x)x{)}^2\\ & -\frac{{\sigma }_w}{2d}\int dxd{x}^{\prime }{\tilde{z}}_i(x)(x\cdot x^{\prime }){\tilde{z}}_i(x^{\prime })+i\int dx{\tilde{z}}_i(x)z_i(x)\left]\right]\\ & =\frac{1}{Z^{\prime }}\int D\tilde{z}\exp [-\frac{1}{2{\sigma }_w}|c{|}^2-\frac{{\sigma }_w}{2d}{\sum }_i\int dxd{x}^{\prime }{\tilde{z}}_i(x)(x\cdot x^{\prime }){\tilde{z}}_i(x^{\prime })\\ & +i{\sum }_i\int dx{\tilde{z}}_i(x)z_i(x)]\\ & =\frac{1}{Z^{\prime \prime }}\exp [-\frac{1}{2{\sigma }_w}|c{|}^2-\frac{1}{2{\sigma }_w}{\sum }_i\int dxd{x}^{\prime }{z}_i(x)(\frac{x\cdot x^{\prime }}{d}{)}^{-1}{z}_i(x^{\prime })].\end{array}$$

We clearly see the first-layer GP kernel $K^{(1)}(x,x^{\prime })=\frac{{\sigma }_w}{d}x\cdot x^{\prime }$ appearing.

To push-forward to the network outputs $f=\frac{1}{\sqrt{N}}c\cdot \varphi (z)$, we just do this again:

$$\begin{array}{rl}P(f)& =\int dcDzP(c,z)\delta (f(x)-\frac{1}{\sqrt{N}}c\cdot \varphi (z(x)))\\ & =\frac{1}{Z^{\prime \prime }}\int dcDzD\tilde{f}\exp [-\frac{1}{2{\sigma }_w}|c{|}^2\\ & -\frac{1}{2{\sigma }_w}{\sum }_i\int dxd{x}^{\prime }{z}_i(x)(\frac{x\cdot x^{\prime }}{d}{)}^{-1}{z}_i(x^{\prime })\\ & +i\int dx\tilde{f}(x)(f(x)-\frac{1}{\sqrt{N}}c\cdot \varphi (z(x)))]\\ & =\frac{1}{Z^{\prime \prime \prime }}\int DzD\tilde{f}\exp [-\frac{1}{2{\sigma }_w}\int dxd{x}^{\prime }(\frac{x\cdot x^{\prime }}{d}{)}^{-1}z(x)\cdot z(x^{\prime })\\ & -\frac{{\sigma }_w}{2N}\int dxd{x}^{\prime }\tilde{f}(x)[\varphi (z(x))\cdot \varphi (z(x^{\prime }))]\tilde{f}(x^{\prime })+i\int dx\tilde{f}(x)f(x)]\end{array}$$

The problem is that it's now impossible to do the integral over the $z$'s exactly, because (for general activation function $\varphi$) the action is no longer quadratic in them. However, as $N\to \infty$, we can treat $\exp [-\frac{{\sigma }_w}{2N}\int dxd{x}^{\prime }\tilde{f}(x)[\varphi (z(x))\cdot \varphi (z(x^{\prime }))]\tilde{f}(x^{\prime })]$ as a perturbation, and approximate the exponent $[\varphi (z(x))\cdot \varphi (z(x^{\prime }))]$ by its expectation value under the Gaussian measure $\exp [-\frac{1}{2{\sigma }_w}\int dxd{x}^{\prime }(\frac{x\cdot x^{\prime }}{d}{)}^{-1}z(x)\cdot z(x^{\prime })]$ (with kernel $K^{(1)}$) on the $z$'s. To be clear, the approximation that's happening is of the form

$$\mathbb{E}{e}^{\frac{1}{N}X}=e^{\frac{1}{N}\mathbb{E}X}+O(\text{Var}(X)/N^2)$$

Up to $O(1/N)$ corrections, that produces the deterministic kernel of the previous section,

$${\bar{K}}^{(2)}(x,x^{\prime })={\sigma }_w{\mathbb{E}}_{z\sim \mathcal{N}(0,K^{(1)})}[\varphi (z(x))\varphi (z(x^{\prime }))],$$

and we get

$$\begin{array}{rl}P(f)& =\frac{1}{Z^{\prime \prime \prime \prime }}\int D\tilde{f}\exp [-\frac{1}{2}\int dxd{x}^{\prime }\tilde{f}(x){\bar{K}}^{(2)}(x,x^{\prime })\tilde{f}(x^{\prime })\\ & +i\int dx\tilde{f}(x)f(x)]+O(\frac{1}{N})\\ & =\frac{1}{Z^{\prime \prime \prime \prime \prime }}\exp [-\frac{1}{2}\int dxd{x}^{\prime }f(x){\bar{K}}^{(2)}(x,x^{\prime }{)}^{-1}f(x^{\prime })]+O(\frac{1}{N}).\end{array}$$

3.6 L/N

The inductive argument of Section 3.4 does not hold at infinite $L$. There are two potential problems.

First, a factor of ${\sigma }_w$ appears with every layer. These factors will accumulate, and can lead either to vanishing kernels (which also turns out to imply vanishing gradients during training), in which case the network will collapse; or to unbounded kernels (and increasingly large gradients), in which case the network will be chaotic and again untrainable. We'll revisit this in Section 3.7. With careful tuning, the issue can be avoided. For example, in a deep linear network (DLN) with $\varphi (z)=z$, one can just take ${\sigma }_w=1$, known as LeCun initialization (LeCun et al. 1998). In a ReLU network, the correct choice instead is ${\sigma }_w=2$, known as He initialization (He et al. 2015).

Even with ${\sigma }_w$ tuned, according to the activation function, so that on average the GP kernels remain order-one from layer to layer, non-Gaussianities can still accumulate. The upshot is that it is not width alone, say of typical size $N$, but rather the ratio $L/N$ of depth-to-width that controls the effective behavior of a network — including its departure from pure Gaussianity at large $N$.

This behavior is analyzed in Chapters 3-4 of (Roberts et al. 2022) using functional-integral methods. In particular, Chapter 3 considers the simple example of DLN's, and produces exact recursion relations for higher cumulants of preactivations in each layer. It is shown that in a network of constant hidden width $N$, the leading non-Gaussianity is

$$\begin{array}{rl}{\kappa }_4^{(L+1)}& =\left[\right(1+\frac{2}{N}{)}^L-1]({\kappa }_2^{(L+1)}{)}^2\\ & =[(e^{2L/N}-1)-\frac{1}{N}\frac{2L}{N}{e}^{2L/N}+O(N^{-2})]({\kappa }_2^{(L+1)}{)}^2,\end{array}$$

where the expansion on the RHS is at finite $L/N$ and $L,N\to \infty$. For small $L/N$, the non-Gaussianity remains small but finite, of size $L/N$. For large $L/N$, ${\kappa }_4^{(L+1)}$ blows up, as do all the other cumulants. The network outputs will fluctuate chaotically from one instantiation to another, and the network again becomes untrainable.

Here's a more direct analysis, in a special case, following (Hanin 2026, Lecture 2). Consider a DLN of length $L$ with input and hidden layers all of width $N$:

$$\begin{array}{rl}{z}^{(\ell )}& =\frac{1}{\sqrt{N}}{w}^{(\ell )}{z}^{(\ell -1)},z^{(0)}=x,\\ f(x)& =z^{(L+1)}=N^{-\frac{L+1}{2}}{w}^{(L+1)}\cdots w^{(1)}x,\end{array}$$

with $w_{ij}^{(\ell )}\sim \mathcal{N}(0,1)$, i.e. ${\sigma }_w=1$ (and input dimension $d=N$). Conditional on $z^{(\ell -1)}$, the vector $z^{(\ell )}$ is Gaussian $z^{(\ell )}\sim \mathcal{N}(0,\frac{1}{N}{I}_N|z^{(\ell -1)}{|}^2)$, so its norm satisfies the recursion

$$|z^{(\ell )}{|}^2\sim |z^{(\ell -1)}{|}^2\frac{{\chi }_N^2}{N}.$$

(Recall that ${\chi }_N^2$ is the distribution of the sum of squares of $N$ i.i.d. standard Gaussian variables.) Let's normalize the input so that $|x|=1$. Then the recursion implies that

$$|f(x){|}^2\stackrel{d}{=}\prod _{k=1}^{L+1}\frac{{\chi }_N^2}{N}=\exp \sum _{k=1}^{L+1}(\log {\chi }_N^2-\log N),$$

with each factor of ${\chi }_N^2$ independent. At large $N$, we have $\log {\chi }_N^2-\log N=\mathcal{N}(-\frac{1}{N},\frac{2}{N})+O(\frac{1}{N^2})$, so at large $N,L$ with $L/N$ fixed

$$|f(x){|}^2\stackrel{d}{=}{e}^{\mathcal{N}(-\frac{L}{N},2\frac{L}{N})},$$

up to higher $O(\frac{1}{N^2}$ corrections. Thus, as $L/N$ grows, the expectation of $\mathbb{E}|f(x)|\to 0$ but the variance and higher cumulants blow up.

The lesson once again is

• $N\to \infty$, $L/N=0$: Gaussian process, no correlations across layers (and, as we'll see in Section 4, no "feature learning").
• $N,L\to \infty$, $L/N$ small: controlled correlations, trainable network (and, as we'll see, feature learning during training). In open-weight LLM's today, $L/N\approx 0.005$–$0.01$; see Exercise 3.3.
• $N,L\to \infty$, $L/N$ large: chaotic and untrainable.

Exercise 3.3 (★) (Depth-to-width ratios in the wild). Look up the depth $L$ (number of transformer blocks) and the width $N$ (model/embedding dimension) for a few families of open-weight language models — e.g. GPT-2, the Llama family, the Qwen family — and compute $L/N$ for each. How much does the ratio vary across families, across sizes within a family, and across the years? Is everyone training in the "controlled correlations" window?

Solution

Representative numbers at the time of writing (take the width $N$ to be the model/embedding dimension): GPT-2 small has $L/N=12/768\approx 0.016$ and GPT-2 XL $48/1600=0.030$; Llama-2-7B has $32/4096\approx 0.008$; Llama-2-70B, $80/8192\approx 0.010$; Llama-3-405B, $126/16384\approx 0.008$. The striking finding is how stable the ratio is: modern open-weight families cluster around $L/N\sim 0.005$–$0.01$ across two orders of magnitude in parameter count, comfortably inside the "controlled correlations" window, with the earliest (GPT-2-era) models running noticeably deeper for their width. Whether this stability reflects tuned wisdom or copied convention is a fair discussion question.

3.7 Edge of chaos

We briefly remark that the analysis of how to tune initialization variances ${\sigma }_w,{\sigma }_b$ (etc.) leading to stable network behavior over a large number of layers (the first point in Section 3.6) is sometimes called finding the "edge of chaos."

The term dates back to dynamical systems theory from the 80's and 90's, referring to the boundary between ordered and chaotic phases. It was adapted to neural networks around 2016 (Poole et al. 2016; Schoenholz et al. 2017). The idea was to treat the progression through layers of a network as a dynamical system, and to ask: given two nearby inputs $x,x^{\prime }$, for what range of hyperparameters (${\sigma }_w,{\sigma }_b$, etc.) and to what depth do they remain nearby, without either a) collapsing; or (b) blowing up chaotically (with no correlation). The analysis roughly amounts to making sure the kernel recursion of (3.26), (3.30) leads to kernels that neither collapse nor blow up. Doing it carefully, theoretically, can produce elementary trainable networks (with no layer norms!) over 10,000 layers deep (Xiao et al. 2018).

For MLP's with ReLu activations, the edge of chaos is just the one point ${\sigma }_w=2,{\sigma }_b=0$. Moving slightly away from this point, one can estimate the "critical depth" $L_c$ at which the network either collapses or blows up – see (Schoenholz et al. 2017). For MLP's with other activation functions, the edge of chaos is more interesting. For $\varphi =\tanh$, the phase diagram is shown in Figure 5.

The quantity `${\chi }_1$' that controls the critical depth ($L_c=-1/\log {\chi }_1$) is the derivative of the layer-to-layer kernel recursion at its fixed point, which measures how fast nearby inputs separate. Importantly, the same quantity ${\chi }_1$ also controls the size of gradients as they back-prop through a network during training. There's a direct link between behavior at initialization and behavior during training. We come to training next.

4. The NTK and large width during training

In this section, we consider training networks that are randomly initialized as in Section 3.2, and their behavior at large width. The lesson will be that as width$\to \infty$ these networks effectively behave like linear models: their parameters $\theta$ never stray far from their values ${\theta }_0$ at initialization, hence

Under suitable conditions, the network itself also converges exponentially fast to a "memorizing network" that fits the training data perfectly. Heuristically, this happens because small changes in many parameters, as in (4.1), can add up to an order-one change in the network values themselves. Nevertheless, the network never experiences phase transitions or feature learning — we'll make this more precise below. The large-width regime (at finite depth and training data) is thus known as the lazy regime, or the NTK regime. (For a self-contained review of models that are linear in their parameters, and of why training them amounts to kernel ridge regression, see Appendix B.)

Although this regime does not describe the feature learning that makes practical networks interesting, it is important to understand for at least two reasons: it is an honest, analytically solvable model of training; and it is a precise lesson in what must be avoided if one wants genuine feature learning — a lesson we act on in Section 5.

We also emphasize the connection between Section 3 and the current analysis. Both involve the same large-width limit. At initialization, it leads to a Gaussian process. During training, we'll see that it leads to a linear model with lazy learning, converging once again to a GP at late training times.

4.1 Gradient flow and the NTK

Throughout this section, we approximate SGD with continuous-time gradient flow (cf. Section 2.1). This involves two approximations. First we update on the entire training dataset $\{(x^{\mu },y^{\mu }){\}}_{\mu =1}^n$ at each step, rather than on batches, so that SGD becomes GD:

$${\theta }_{k+1}-{\theta }_k=-{\eta }_{GD}{\nabla }_{\theta }{\mathcal{L}}_{\theta }(\{x^{\mu },y^{\mu }\}){|}_{\theta ={\theta }_k}$$

where

is the full empirical loss.

Second, we scale down the learning rate ${\eta }_{GD}={\lambda }^{-1}\eta$ and scale up the number of training steps, setting $k=\lfloor \lambda t]$ where $t\ge 0$ is a continuous "time" variable. As $\lambda \to \infty$, the gradient descent converges to a gradient flow

$$\dot{\theta }:=\frac{d}{dt}\theta =-\eta {\nabla }_{\theta }{\mathcal{L}}_{\theta }.$$

All theorems in this section can be extended to SGD, adding additional error terms to bound the difference between SGD and GF. (See for example (Spiliopoulos et al. 2025, Chapter 19).)

4.1.1 Geometry of gradient flows

Suppose that $X$ is a smooth Riemannian manifold, with nondegenerate metric $g=g_{ab}d{x}^{a}d{x}^b$, inverse metric $g^{-1}=g^{ab}\frac{\partial }{\partial x^a}\frac{\partial }{\partial x^b}$, and a function $h:X\to \mathbb{R}$. Then one can define gradient flow with respect to $(h,g)$ as

Note that this equation does not make sense (it's not coordinate-free) without an inverse-metric (a bilinear form on each cotangent space $T^{\ast }X$). The inverse-metric is used to relate the differential $dh=({\partial }_{b}h{)}_{b=1}^{\text{dim}X}$ on the RHS to a vector field $g^{-1}dh$, which is the right sort of geometric object to induce a flow on $X$.

In neural networks, $X={\mathbb{R}}^P$ is the space of parameters $\theta$ and $h=\mathcal{L}$ is the loss function. One always assumes a constant Euclidean metric in parameter space $g={\sum }_{i=1}^P(d{\theta }^i{)}^2={\delta }_{ij}d{\theta }^{i}d{\theta }^j$, in order to define SGD/GD/GF/etc. This is the source of some of the simplest inductive biases of trained models, and also why one expects "features" to have some linear structure. Eqn. (4.5) with the metric looks like

$${\dot{\theta }}^i=-\eta {\delta }^{ij}{\partial }_j{\mathcal{L}}_{\theta }.$$

Now return to the general setting of Riemannian manifolds. Suppose we have a smooth map $f:X\to Y$ between two manifolds, and moreover that $h:X\to \mathbb{R}$ is the pull-back of a function $h^Y:Y\to \mathbb{R}$, meaning $h(x)=h^Y(f(x))$. Let $\{f^{\alpha }{\}}_{\alpha =1}^{\text{dim}Y}$ be the components of the map $f$, in local coordinates $y^{\alpha }$ on $Y$. Then the inverse-metric on $X$ induces an inverse-metric on $Y$ (called its push-forward):

$$f_{\ast }(g^{-1})=f_{\ast }\left[g^{ab}\frac{\partial }{\partial x^a}\frac{\partial }{\partial x^b}\right]=\underset{g^{\alpha \beta }}{\underbrace{g^{ab}\frac{\partial f^{\alpha }}{\partial x^a}\frac{\partial f^{\beta }}{\partial x^b}}}\frac{\partial }{\partial y^{\alpha }}\frac{\partial }{\partial y^{\beta }}.$$

Schematically, $g^{\alpha \beta }=(J{g}^{-1}{J}^T)$, where $J^{\alpha }{}_a=\frac{\partial f^{\alpha }}{\partial x^a}$ is the Jacobian matrix of $f$. Then the gradient flow on $X$ induces a gradient flow on $Y$, given by

$${\dot{y}}^{\alpha }=-\eta g^{\alpha \beta }\frac{\partial h^Y}{\partial y^{\beta }}.$$

Let's apply this to neural networks, where $f:{\mathbb{R}}^P\to {\mathbb{R}}^n$ is the function from parameters to network outputs on the training data $(x^{\mu },y^{\mu })$, i.e. $f^{\mu }(\theta )=f_{\theta }(x^{\mu })$ in our standard notation. The loss function is indeed a pullback of a function on ${\mathbb{R}}^n$, since it depends on parameters $\theta$ only through the network values on training data,

$${\mathcal{L}}_{\theta }={\mathcal{L}}^{(f)}(f(\theta )),{\mathcal{L}}^{(f)}(f):=\sum _{\mu =1}^n\ell (f^{\mu },y^{\mu }).$$

The push-forward of the gradient flow in parameter space becomes

The inverse-metric ${\Theta }^{\mu \nu }$ on the space of function values is called the "neural tangent kernel" or NTK. The name was coined in (Jacot et al. 2018); it also appeared in (Du et al. 2019) under a different name. Here we are emphasizing that it's a completely natural geometric object, for any neural network (in fact, for any ML model).

We mention one final bit of geometry. Let ${\delta }_{\mu \nu }$ be a standard Euclidean metric in the space of function values, with inverse ${\delta }^{\mu \nu }$. The gradient of the loss ${\delta }^{\mu \nu }\partial {\mathcal{L}}^{(f)}/\partial f^{\nu }$ that appears in (4.10) (contracted with an inverse metric) is sometimes called the residual, and denoted ${\Delta }^{\mu }$ or $r^{\mu }$. For MSE loss, we've got ${\mathcal{L}}^{(f)}=\frac{1}{2}{\sum }_{\mu =1}^n(f^{\mu }-y^{\mu }{)}^2=\frac{1}{2}{\delta }_{\mu \nu }(f^{\mu }-y^{\mu })(f^{\nu }-y^{\nu })$. Then the residual is just the difference between function values and truth for the training data:

$${\Delta }^{\mu }:={\delta }^{\mu \nu }\partial {\mathcal{L}}^{(f)}/\partial f^{\nu }=f^{\mu }-y^{\mu }.$$

Noting that $y^{\mu }$ are constants, the gradient-flow equation can be recast as

Those who have studied gauge theory may recognize this as an equation for parallel transport, with respect to an $n\times n$ matrix-valued connection $\eta {\Theta }^{\mu }{}_{\nu }$. If one knows the dependence of $\Theta$ on time (which in general is far from trivial — see the exercise below) the solution is a time-ordered exponential

$${\Delta }^{\mu }(t)=\left(P\exp \right[-\eta \int \Theta (t)dt]{)}_{\nu }^{\mu }{\Delta }^{\nu }(0).$$

Exercise 4.1 (The NTK from the chain rule). Derive the flow equation (4.10) for the function values $f^{\mu }$ directly, using only the chain rule — no geometry required. (The geometric derivation explains why an object like ${\Theta }^{\mu \nu }$ had to appear; the chain rule shows it appears in three lines.)

Solution

By the chain rule and the gradient-flow equation,

$$\begin{array}{rl}\frac{d}{dt}{f}^{\mu }& ={\nabla }_{\theta }{f}^{\mu }\cdot \dot{\theta }=-\eta {\nabla }_{\theta }{f}^{\mu }\cdot {\nabla }_{\theta }\mathcal{L}\\ & =-\eta {\nabla }_{\theta }{f}^{\mu }\cdot \sum _{\nu }{\nabla }_{\theta }{f}^{\nu }\frac{\partial {\mathcal{L}}^{(f)}}{\partial f^{\nu }}=-\eta {\Theta }^{\mu \nu }\frac{\partial {\mathcal{L}}^{(f)}}{\partial f^{\nu }},\end{array}$$

using in the middle step that $\mathcal{L}$ depends on $\theta$ only through the function values $f^{\nu }$.

Exercise 4.2 (How fast does the kernel move?). Show that the time derivative of the NTK satisfies

$$\frac{d}{dt}{\Theta }^{\mu \nu }=-\eta [{\nabla }_{\theta }{f}^{\mu }\cdot {\nabla }_{\theta }^2{f}^{\nu }\cdot {\nabla }_{\theta }{f}^{\rho }]\frac{\partial {\mathcal{L}}^{(f)}}{\partial f^{\rho }}+(\mu \leftrightarrow \nu ),$$

where ${\nabla }_{\theta }^2{f}^{\nu }={\nabla }_{\theta }^2{f}_{\theta }(x^{\nu })$ is the Hessian. Bounding the Hessian ultimately leads to the constant-NTK theorem in Section 4.3 below.

Solution

Differentiate ${\Theta }^{\mu \nu }={\nabla }_{\theta }{f}^{\mu }\cdot {\nabla }_{\theta }{f}^{\nu }$ along the flow: $\frac{d}{dt}{\Theta }^{\mu \nu }=\dot{\theta }\cdot {\nabla }_{\theta }^2{f}^{\mu }\cdot {\nabla }_{\theta }{f}^{\nu }+{\nabla }_{\theta }{f}^{\mu }\cdot {\nabla }_{\theta }^2{f}^{\nu }\cdot \dot{\theta }$. Substituting $\dot{\theta }=-\eta {\nabla }_{\theta }{f}^{\rho }\partial {\mathcal{L}}^{(f)}/\partial f^{\rho }$ gives the stated formula. Taking operator norms, $\Vert \dot{\Theta }\Vert \lesssim \eta \Vert {\nabla }_{\theta }^{2}f\Vert \Vert {\nabla }_{\theta }f{\Vert }^2|\partial \mathcal{L}/\partial f|\sim \eta \Vert {\nabla }_{\theta }^{2}f\Vert \Vert \Theta \Vert |\Delta |$: the kernel can only move as fast as the Hessian of the network function (not of the loss) allows — which is why bounding that Hessian at large width freezes the kernel.

4.2 Features

The Jacobian matrix $J^{\mu }{}_i=\partial f^{\mu }/\partial {\theta }^i$ measures how function values on the training data depend on small variations of the network weights. Assuming there are more parameters than training data, its singular value decomposition looks like

$$J=US{V}^T,S=[\text{diag}(s_1,...,s_n)0_{n\times (P-n)}].$$

The right singular vectors (columns of $V$) define natural modes in parameter space, while the left singular vectors (columns of $U$) are the natural modes in function-value space — where "natural" means with respect to how network outputs depend on weights. The $n$ left singular vectors (together with their singular values $s_i$) can be thought of, abstractly, as the features that a network has learned. A common definition of feature learning is that these singular vectors+values change during training.

The analysis of features can equivalently be performed via the NTK. As a matrix, we've got

with eigenvectors = the feature vectors, and eigenvalues = $\{s_i^2\}$ . Thus feature learning happens when the NTK changes during training.

4.3 Constant NTK at large width

Now we get to the main theoretical result and simplification of this section. Suppose we've got an MLP of fixed depth $L$, and fixed training data size $n$, and take the width of all hidden layers to infinity. We'll assume the hidden widths all scale in a similar way $N_L,N_{L-1},...,N_1\approx N\to \infty$, and we initialize the network the standard way from Section 3.2. We want to show that for almost all samples of training data,

• The NTK ${\Theta }^{\mu \nu }$ remains constant throughout training, up to corrections that vanish as $N\to \infty$.
• The network itself linearizes: at any training time $t$, $f_{\theta (t)}(x)=f_{{\theta }_0}(x)+(\theta (t)-{\theta }_0)\cdot {\nabla }_{\theta }{f}_{{\theta }_0}(x)$

A rigorous derivation is somewhat technical, and we refer the reader to (Du et al. 2019; Jacot et al. 2018), (Roberts et al. 2022, Ch. 11), (Spiliopoulos et al. 2025, Ch. 20) for complete arguments, explained from a variety of different perspectives. We'll just sketch the key ideas behind the result. See also (Hanin 2026, Lecture 3).

We'll simplify life by setting biases to zero, so the network becomes

$$f_{\theta }(x)=z^{(L+1)}(x),z_i^{(\ell )}=\frac{1}{\sqrt{N_{\ell -1}}}\sum _j{w}_{ij}^{(\ell )}\varphi (z_j^{(\ell -1)}),w_{ij}^{(\ell )}\sim \mathcal{N}(0,{\sigma }_w),$$

in the fan-in normalization of Section 3.2, with order-one weights.

Size of the NTK. The first task is to estimate how large the NTK is at large width, because that decides how the learning rate must scale. By the chain rule, the gradient with respect to the layer-$\ell$ weights is an outer product,

$$\frac{\partial f}{\partial w_{ij}^{(\ell )}}=\frac{1}{\sqrt{N_{\ell -1}}}{\beta }_i^{(\ell )}\varphi (z_j^{(\ell -1)}),{\beta }_i^{(\ell )}:=\frac{\partial f}{\partial z_i^{(\ell )}},$$

where the backpropagated sensitivities ${\beta }^{(\ell )}$ obey ${\beta }^{(L+1)}=1$ and

$${\beta }_i^{(\ell )}={\varphi }^{\prime }(z_i^{(\ell )})\frac{1}{\sqrt{N_{\ell }}}\sum _k{w}_{ki}^{(\ell +1)}{\beta }_k^{(\ell +1)}.$$

The NTK is a sum over layers,

The second bracket is an average of $N_{\ell -1}$ terms. At initialization and large width, the law of large numbers replaces it by ${\bar{K}}_{\mu \nu }^{(\ell )}/{\sigma }_w$, the Gaussian-process kernel of (3.30) evaluated on the two inputs, which is of order one. (For $\ell =1$ the bracket is exactly $x^{\mu }\cdot x^{\nu }/d$, which is ${\bar{K}}_{\mu \nu }^{(1)}/{\sigma }_w$.) The first bracket is handled by the same recursion run backwards. Define

$${\dot{\bar{K}}}_{\mu \nu }^{(\ell )}={\dot{\bar{K}}}^{(\ell )}(x^{\mu },x^{\nu }):=\mathbb{E}[{\varphi }^{\prime }(z_i^{(\ell -1)}(x^{\mu })){\varphi }^{\prime }(z_i^{(\ell -1)}(x^{\nu }))]\text{(for any fixed i)},$$

which is of order one for reasonable activation functions. Using the independence at initialization of the weights $w^{(\ell +1)}$ from everything below them, one finds $B_{\mu \nu }^{(\ell )}\approx {\sigma }_w{\dot{\bar{K}}}_{\mu \nu }^{(\ell +1)}{B}_{\mu \nu }^{(\ell +1)}$ with $B_{\mu \nu }^{(L+1)}=1$, and hence

This is a sum of $L+1$ terms, each of order one, with no factors of $N$ anywhere: in the fan-in normalization the NTK is of order one at large width, and every layer contributes to it at the same order. (Equation (4.23) is the standard recursion for the infinite-width NTK (Jacot et al. 2018).) By (4.10), an order-one learning rate $\eta$ then moves the function values $f^{\mu }$ by an order-one amount per unit time, which is what we want, and no rescaling of $\eta$ with width is needed in this convention. For the one-hidden-layer network $f=\frac{1}{\sqrt{N}}{\sum }_i{c}_i\varphi (a_i\cdot x)$ one can check this by hand:

an average of $N$ order-one terms. (Here and below we absorb the first-layer factor $1/\sqrt{d}$ into the inputs $x$, since $d$ is fixed.)

Caution: the estimate (4.23) is at fixed depth $L$ and fixed data size $n$. The NTK grows like $L$ with depth, and the operator norm of the $n\times n$ matrix $\Theta$ grows with $n$, so for deep networks or large datasets the learning rate must shrink accordingly, and the constant-NTK statements below need $n$ and $L$ to stay small compared with $N$.

The Hessian is small. To see that the NTK is approximately constant during training, we consider its time derivative, which by Exercise 4.2 is bounded by the operator norm of the Hessian of the network function,

The Hessian is a very large but very sparse matrix, and in the fan-in normalization its operator norm is small. We'll sketch how this works for one hidden layer ($L=1$) with one-dimensional output, following (Hanin 2026, Lecture 3). Writing

the Hessian splits into blocks, one per hidden neuron,

The operator norm of a block-diagonal matrix is the largest operator norm of its blocks. Each block has fixed size $(d+1)\times (d+1)$, contains only order-one quantities (at least at initialization), and carries an explicit factor $1/\sqrt{N}$. Thus

It's reasonable to expect that this bound continues to hold during training, and with a bit more work it can be proven (though we won't do it here). It also holds for networks of arbitrary fixed depth. A more careful analysis, keeping track of cancellations among the fluctuating terms, improves the resulting bound on the kernel drift to $O(1/N)$; the crude $O(1/\sqrt{N})$ is enough for the argument.

Putting the pieces together. First, the parameters do not move far in total. From $\dot{\theta }=-\eta {\sum }_{\mu }{\Delta }^{\mu }{\nabla }_{\theta }{f}^{\mu }$,

$$|\dot{\theta }{|}^2={\eta }^2|\sum _{\mu =1}^n{\Delta }^{\mu }{\nabla }_{\theta }{f}^{\mu }{|}^2\le {\eta }^2|\Delta {|}^2|{\nabla }_{\theta }f{|}^2\le {\eta }^{2}n|\Delta {|}^2\Vert \Theta \Vert =O(1),$$

where $|{\nabla }_{\theta }f{|}^2={\sum }_{\mu }{\Theta }^{\mu \mu }=\mathrm{tr}\Theta$. So by the mean value theorem, $\theta (t)-{\theta }_0=t\dot{\theta }(\bar{t})$ for some $\bar{t}\in [0,t]$, and

An order-one total displacement, shared among $P\approx Nd$ parameters, means that each individual weight moves only by $O(1/\sqrt{N})$: relative to its order-one initial value, every weight barely changes. Second, the kernel freezes. By (4.25) and (4.28),

so the order-one kernel $\Theta$ becomes constant as $N\to \infty$. Third, the network linearizes. By Taylor's theorem with remainder,

$$f_{\theta }(x)-f_{{\theta }_0}(x)={\nabla }_{\theta }f(x){|}_{{\theta }_0}\cdot (\theta -{\theta }_0)+\frac{1}{2}(\theta -{\theta }_0)\cdot {\nabla }_{\theta }^{2}f(x){|}_{\bar{\theta }}\cdot (\theta -{\theta }_0)$$

for some $\bar{\theta }$ on the segment between ${\theta }_0$ and $\theta$. Setting $\theta =\theta (t)$ and using (4.30) and (4.28), the quadratic term is bounded by

$$|\frac{1}{2}(\theta -{\theta }_0)\cdot {\nabla }_{\theta }^{2}f(x){|}_{\bar{\theta }}\cdot (\theta -{\theta }_0)|\le \frac{1}{2}\Vert {\nabla }_{\theta }^{2}f\Vert |\theta -{\theta }_0{|}^2=O(\frac{1}{\sqrt{N}})\cdot O(1),$$

so the linear approximation becomes exact as $N\to \infty$, even though the linear term, and with it the network function, moves by an order-one amount. In words: each of the $N$ neurons has $O(1/\sqrt{N})$ leverage on the output, so an order-one change in $f$ can be assembled from $O(1/\sqrt{N})$ changes in every weight, and changes that small leave the features ${\nabla }_{\theta }f$ untouched.

4.4 Lazy learning

When the NTK is constant during training — confusingly often called the NTK regime — a network undergoes what is known as "lazy learning." It is not learning features, since the eigenvalues $s_{\mu }^2$ and eigenvectors $u_{\mu }$ of the NTK are (by construction) constant. Nevertheless, the network function converges to its training data exponentially fast: from (4.12) with constant $\Theta$, we have

where we recall that ${\Delta }^{\mu }(t)=f^{\mu }-y^{\mu }=f_{\theta (t)}(x^{\mu })-y^{\mu }$. Writing the residuals in an eigenbasis for the NTK, $\Delta ={\hat{\Delta }}^{\mu }{u}_{\mu }$ we see that

Each feature is learned at a characteristic timescale $t_{\mu }\approx 1/(\eta s_{\mu }^2)$ .

Exercise 4.3 (Lazy predictions on test points). Derive that on an arbitrary test point $x$, not necessarily part of the training data,

$$f_{\theta (t)}(x)=f_{{\theta }_0}(x)-\Theta (x,x^{\mu }){\Theta }_{\mu \nu }^{-1}(11-e^{-\eta t\Theta }{)}^{\nu }{}_{\rho }{\Delta }^{\rho }(0)$$

where $\Theta (x,x^{\mu })={\nabla }_{\theta }f(x)\cdot {\nabla }_{\theta }f(x^{\mu })$ is the NTK for a test-train pair (which remains constant during training, just like the train-train NTK ${\Theta }^{\mu \nu }$).

Solution

In the linearized regime, $f_{\theta (t)}(x)=f_{{\theta }_0}(x)+{\nabla }_{\theta }f(x)\cdot (\theta (t)-{\theta }_0)$ with all gradients at ${\theta }_0$. The parameter motion integrates the flow: $\dot{\theta }=-\eta {\sum }_{\mu }{\nabla }_{\theta }{f}^{\mu }{\Delta }^{\mu }(t)$ with $\Delta (t)=e^{-\eta t\Theta }\Delta (0)$ from (4.34), so

$$\begin{array}{rl}\theta (t)-{\theta }_0& =-\eta \sum _{\mu }{\nabla }_{\theta }{f}^{\mu }{\int }_0^{t}ds(e^{-\eta s\Theta }{)}^{\mu }{}_{\rho }{\Delta }^{\rho }(0)\\ & =-\sum _{\mu }{\nabla }_{\theta }{f}^{\mu }\left[{\Theta }^{-1}\right(11-e^{-\eta t\Theta }){]}^{\mu }{}_{\rho }{\Delta }^{\rho }(0).\end{array}$$

Contracting with ${\nabla }_{\theta }f(x)$ and recognizing $\Theta (x,x^{\mu })={\nabla }_{\theta }f(x)\cdot {\nabla }_{\theta }f(x^{\mu })$ yields (4.36). (The test–train kernel is constant during training for the same reason the train–train one is: it is built from the frozen gradients at ${\theta }_0$.)

From Exercise 4.3, we see that at the end of training $t\to \infty$ the network obeys

$$f_{{\theta }_{\infty }}(x)=f_{{\theta }_0}(x)-\Theta (x,x^{\mu }){\Theta }_{\mu \nu }^{-1}(f_{{\theta }_0}(x^{\nu })-y^{\nu }).$$

This is an affine-linear transformation of the Gaussian process $f_{{\theta }_0}(x)$ at initialization, so it is again a Gaussian process. It's straightforward to check that the new mean and covariance are

where $K^{x{x}^{\prime }}=K(x,x^{\prime })$, ${\Theta }^{x\mu }=\Theta (x,x^{\mu })$, etc.

4.5 Summary

So far we've seen that in the fan-in normalization (Section 3.2), with order-one weights and an order-one learning rate $\eta$, at large width $N$ we've got

$$\begin{array}{rl}\Vert \Theta \Vert \sim |{\nabla }_{\theta }f{|}^2\sim 1,\Vert {\nabla }_{\theta }^{2}f\Vert & \sim \frac{1}{\sqrt{N}},|f_{\theta }-f_{{\theta }_0}|\sim 1,\\ |\theta -{\theta }_0|& \sim 1\text{(i.e. }{N}^{-1/2}\text{ per weight)},\end{array}$$

which forces $\Theta$ to be constant and the network to linearize as a function of its parameters.

All this breaks down if

• Depth $L$ is large as well, with $L/N$ nonzero. (See (Roberts et al. 2022, Ch. 11).)
• Data samples $n$ become large, with $n/N$ nonzero.
• Learning rate becomes large.

Both $L$ and $n$ large are interesting cases for modern neural networks!

Exercise 4.4 (★) (Numerics: the lazy network (lab)). This lab, and its continuation in Exercise 5.3, trains one small network in both the fan-in and the mean-field normalization and tests the claims of Sections 4 and 5 numerically. The setup runs in seconds, in about sixty lines of NumPy; reproduce it, then modify it.

The starter model. Inputs are one-dimensional: $n=30$ equispaced training points $x^{\mu }\in [-2,2]$, with targets from a two-neuron teacher, $y(x)=2\tanh (3x)-\frac{3}{2}\tanh (x)$. The student is a one-hidden-layer $\tanh$ network

$$f(x)=\frac{1}{N^{\gamma }}\sum _{i=1}^N{c}_i\tanh (a_{i}x),c_i,a_i\sim \mathcal{N}(0,1),N=300,$$

trained by full-batch gradient descent on $\mathcal{L}=\frac{1}{2}{\sum }_{\mu }(f(x^{\mu })-y^{\mu }{)}^2$, with residuals ${\Delta }^{\mu }=f(x^{\mu })-y^{\mu }$. Here take the fan-in exponent $\gamma =\frac{1}{2}$, learning rate $\eta =0.02$, and $4\times 10^4$ steps. The empirical NTK to monitor is the $n\times n$ matrix

$$\begin{array}{rl}{\Theta }^{\mu \nu }=\frac{1}{N^{2\gamma }}\sum _{i=1}^N[& \tanh (a_i{x}^{\mu })\tanh (a_i{x}^{\nu })\\ & +c_i^2(1-{\tanh }^2(a_i{x}^{\mu }))(1-{\tanh }^2(a_i{x}^{\nu }))x^{\mu }{x}^{\nu }].\end{array}$$

(a) It fits. Train and check that the loss becomes small. (The network is overparametrized; fitting is not the interesting part.)

(b) Initialization. Over about $200$ seeds, histogram $f_0(x)$ at a fixed input, say $x=1$. You should see an order-one Gaussian, the Gaussian process of Section 3.4, and the sample covariance across two inputs should match the $\tanh$ GP kernel.

(c) The kernel freezes. For $N\in \{30,100,300,1000\}$ (a few seeds each), record $\Vert {\Theta }_{\mathrm{end}}-{\Theta }_0\Vert /\Vert {\Theta }_0\Vert$ and plot it against $N$ on log–log axes. Expected: it decays, with slope between $-\frac{1}{2}$ (the guaranteed bound (4.31)) and $-1$ (the typical measurement). Also record the relative parameter displacement $|{\theta }_{\mathrm{end}}-{\theta }_0|/|{\theta }_0|$ and compare with (4.30).

(d) Mode by mode. Diagonalize ${\Theta }_0$ and plot $|(U^T\Delta {)}_k(t)|$ on a semilog axis for a few $k$: straight lines with slopes $-\eta s_k^2$, the features being learned at $t_k=1/(\eta s_k^2)$, largest eigenvalue first, as in (4.35). Compare with the Bayesian version, Exercise 4.5(b).

(e) Prediction. Compare the trained function on fresh test points with the frozen-kernel prediction (4.36).

Solution

Reference results for the setup as specified (single seeds; small-$N$ numbers fluctuate).

(a) Final loss $\sim 10^{-5}$–$10^{-4}$. Keep $\eta <2/s_{\max }^2({\Theta }_0)$ for stability; here $s_{\max }^2\approx 30$, so $\eta =0.02$ is safe.

(b) At $x=1$, $f_0\sim \mathcal{N}(0,O(1))$, and the sample covariance across two inputs matches the $\tanh$ GP kernel.

(c) Measured $\Vert \Delta \Theta \Vert /\Vert {\Theta }_0\Vert \approx 0.25,0.14,0.06$ at $N=100,300,1000$: decaying, with a log–log slope between $-1/2$ and $-1$; average a few seeds to clean up the trend. The relative parameter displacement decays like $N^{-1/2}$, since the total displacement is $O(1)$ by (4.30) while $|{\theta }_0|\sim \sqrt{P}\sim \sqrt{N}$. Compute $\Theta$ from per-example gradients (an $n\times P$ Jacobian contracted with itself), in float64.

(d) The residual modes are straight lines on a semilog plot, with slopes $-\eta s_k^2$, until they hit the small nonlinear floor; the smallest-eigenvalue modes are visibly not yet learned at the end of training. This is spectral bias on display, and the dynamical twin of Exercise 4.5(b).

(e) At $N=300$ the trained function on test points agrees with (4.36) to within a few percent; the agreement degrades as $N$ shrinks, which is feature learning beginning to stir. The continuation is Exercise 5.3.

4.6 Bayesian learning at large width: kernel learning

So far, we have considered MLP's at large width, trained via gradient flow (as an approximation to SGD/GD). We found that their NTK remains constant and that their late-time distribution is a Gaussian process given by (4.38).

Suppose that we consider the same large-width network, but "train" Bayesian inference instead, on the empirical MSE loss $\mathcal{L}(\theta )=n{L}_n(\theta )=\frac{1}{2}{\sum }_{\mu =1}^n|f(x^{\mu })-y^{\mu }{|}^2$. (Recall from Section 2.1 that, practically speaking, this means training via Langevin dynamics.) Then we expect a Bayesian posterior

$$P(\theta )=\frac{1}{Z}{e}^{-\beta \mathcal{L}(\theta )}\pi (\theta ),\pi (\theta )\propto e^{-\frac{1}{2{\sigma }_w}|\theta {|}^2},$$

where the prior $\pi (\theta )$ is the Gaussian of order-one variance from (3.1), and the network carries the fan-in normalization of Section 3.2.

We'd like to push the distribution forward to the network function $f_{\theta }(x)$. At large-width, the push-forward of the prior is our familiar Gaussian process from Section 3.4; expressed via a functional integral as in Section 3.5 it becomes

$$\pi [f]=\frac{1}{Z^{\prime }}{e}^{-\frac{1}{2}\int dxd{x}^{\prime }f(x)K^{-1}(x,x^{\prime })f(x^{\prime })}$$

with $K(x,x^{\prime })={\bar{K}}^{(L+1)}(x,x^{\prime })$ the final-layer GP kernel. Since the loss is explicitly a function of network values, we immediately get the late-time trained distribution

$$P[f]=\frac{1}{Z^{\prime \prime }}{e}^{-\frac{1}{2}[\beta {\sum }_{\mu =1}^n(f(x^{\mu })-y^{\mu }{)}^2+\int dxd{x}^{\prime }f(x)K^{-1}(x,x^{\prime })f(x^{\prime })]}$$

Note that the action here is quadratic in function values, so this is still a Gaussian process!

Exercise 4.5 (★) (Bayesian lazy learning: kernel ridge regression from the Gaussian posterior). Lazy gradient flow at large width solved itself in Section 4.3: the NTK froze, and the $k$-th kernel eigenmode of the residual decayed as $e^{-\eta s_k^{2}t}$, so that it is learned at the characteristic time $t_k=1/(\eta s_k^2)$. Here you derive the Bayesian-learning version of the same statement, with $\beta$ in the role of training time. Restricted to the training values $\mathbf{f}=(f(x^1),\dots ,f(x^n))$, the prior is Gaussian with covariance the kernel matrix ${\hat{K}}^{\mu \nu }:=K(x^{\mu },x^{\nu })$, so the posterior on training values is

$$p(\mathbf{f})\propto \exp [-\frac{1}{2}{\mathbf{f}}^T{\hat{K}}^{-1}\mathbf{f}-\frac{\beta }{2}|\mathbf{f}-\mathbf{y}{|}^2].$$

(a) Posterior mean. The exponent is quadratic, so the posterior is Gaussian and everything follows from completing the square. Show that the posterior mean of the training values is $\mathbf{\mu }=\hat{K}(\hat{K}+{\beta }^{-1}{I}_n{)}^{-1}\mathbf{y}$.

(b) Eigenmodes. Diagonalize the kernel, $\hat{K}{u}_k={\lambda }_k{u}_k$, and show that mode by mode

$$u_k\cdot \mathbf{\mu }=\frac{{\lambda }_k}{{\lambda }_k+{\beta }^{-1}}{u}_k\cdot \mathbf{y}=\frac{\beta {\lambda }_k}{1+\beta {\lambda }_k}{u}_k\cdot \mathbf{y}.$$

Conclude that the eigenmodes of the GP kernel at initialization are learned one by one as $\beta$ grows, mode $k$ switching on at the characteristic scale ${\beta }_k=1/{\lambda }_k$, large-eigenvalue modes first (spectral bias).

(c) Compare with gradient flow. Place this alongside (4.35), ${\hat{\Delta }}^k(t)=e^{-\eta s_k^{2}t}{\hat{\Delta }}^k(0)$, that is $t_k=1/(\eta s_k^2)$. What is the same, and what differs? Two things to notice: the correspondence $\beta \leftrightarrow \eta t$ between the amount of data (or inverse noise) and training time; and the different shapes of the two filters, the power law $1/(1+\beta {\lambda }_k)$ in $\beta$ versus the exponential $e^{-\eta s_k^{2}t}$ in $t$ (compare Exercise B.5). Also: the Bayesian calculation uses the GP kernel $K$, the dynamical one the NTK $\Theta$, two different frozen kernels for the same network.

(d) Prediction. Show that the full posterior over functions is again a Gaussian process,

$$P[f]\propto e^{-\frac{1}{2}\int dxd{x}^{\prime }[f(x)-\mu (x)]\Sigma (x,x^{\prime }{)}^{-1}[f(x^{\prime })-\mu (x^{\prime })]},$$

with mean and covariance

$$\begin{array}{rl}\mu (x)& =\sum _{\mu ,\nu }K(x,x^{\mu })(\hat{K}+{\beta }^{-1}{I}_n{)}_{\mu \nu }^{-1}{y}^{\nu },\\ \Sigma (x,x^{\prime })& =K(x,x^{\prime })-\sum _{\mu ,\nu }K(x,x^{\mu })(\hat{K}+{\beta }^{-1}{I}_n{)}_{\mu \nu }^{-1}K(x^{\nu },x^{\prime }).\end{array}$$

The posterior mean is kernel ridge regression with ridge ${\beta }^{-1}$ (Appendix B.4), and $\beta \to \infty$ interpolates the data exactly. [Hint: the exponent is quadratic in $f$; complete the square, and use the Woodbury identity to relate the two natural forms of the answer.]

Solution

(a) Completing the square, the exponent is $-\frac{1}{2}{\mathbf{f}}^T{A}^{-1}\mathbf{f}+\beta {\mathbf{y}}^T\mathbf{f}+\mathrm{const}$ with $A^{-1}={\hat{K}}^{-1}+\beta I_n$, so the posterior is Gaussian with covariance $A$ and mean $\mathbf{\mu }=\beta A\mathbf{y}=\beta ({\hat{K}}^{-1}+\beta I_n{)}^{-1}\mathbf{y}=\hat{K}(\hat{K}+{\beta }^{-1}{I}_n{)}^{-1}\mathbf{y}$ (multiply through by $\hat{K}$).

(b) In the eigenbasis all matrices are diagonal, giving (4.46). The filter $\beta {\lambda }_k/(1+\beta {\lambda }_k)$ is $\approx 0$ for $\beta \ll 1/{\lambda }_k$ and $\approx 1$ for $\beta \gg 1/{\lambda }_k$: a switch at ${\beta }_k=1/{\lambda }_k$. On a $\log \beta$ axis it is a sigmoid of universal shape, pleasant to compare with the logistic occupations of Section 2.5, though nothing here is a phase transition: the switches are $n$ smooth crossovers, one per mode.

(c) The same: frozen-kernel linear structure, mode-by-mode learning, spectral bias with scale $1/{\lambda }_k$, and the dictionary "$\beta \approx$ training time" of Section 2.1 made quantitative as $\beta \leftrightarrow \eta t$. Different: the suppression of the unlearned part is $1/(1+\beta {\lambda }_k)$, a power law in $\beta$, versus $e^{-\eta s_k^{2}t}$, exponential in $t$ (the two filters of Figure 13); and the kernels differ. Bayesian learning of the frozen network is governed by the GP kernel $K$, a property of the function distribution at initialization, while gradient flow is governed by the NTK $\Theta$, a property of the parametrization. Both are fixed, data-independent kernels: in neither picture does the lazy network adapt its features.

(d) Varying the exponent of $P[f]$ with respect to $f(x)$ gives the stationarity equation $\int d{x}^{\prime }{K}^{-1}(x,x^{\prime })\mu (x^{\prime })+\beta {\sum }_{\mu }(\mu (x^{\mu })-y^{\mu })\delta (x-x^{\mu })=0$, that is $\mu (x)=-\beta {\sum }_{\mu }K(x,x^{\mu })(\mu (x^{\mu })-y^{\mu })$. Evaluating at the training points gives a closed linear system whose solution is (a), and substituting back yields the stated $\mu (x)$. The covariance is the inverse of the quadratic form, $\Sigma =(K^{-1}+\beta {\delta }_{\mathrm{data}}{)}^{-1}$, which the Woodbury identity converts into the stated form. As $\beta \to \infty$ this pins $\mu (x^{\mu })=y^{\mu }$: noiseless GP interpolation; finite $\beta$ is ridge regularization with ridge ${\beta }^{-1}$.

These equations embody what's sometimes known as kernel regression, with ridge parameter ${\beta }^{-1}=T$. (Appendix B.4 derives the same formula for a finite linear model, with no Bayesian input.) If the training samples are sufficiently dense, the mean can be an approximated by an integral in terms of the untrained kernel

$$\mu (x)\approx \int d{x}^{\prime }d{x}^{\prime \prime }K(x,x^{\prime })(K+(n\beta {)}^{-1}\text{id}{)}^{-1}(x^{\prime },x^{\prime \prime })y(x^{\prime \prime }).$$

Writing the data "truths" $y(x)=\sum y_k{\phi }_k(x)$ in an eigenbasis for $K$, with ordered eigenvalues ${\lambda }_1\ge {\lambda }_2\ge ...$, we find

$$\mu (x)=\sum _{k=1}^{\infty }\frac{{\lambda }_k}{{\lambda }_k+(n\beta {)}^{-1}}{y}_k{\phi }_k(x)\approx \sum _{k\text{s.t.}{\lambda }_k\gtrsim (n\beta {)}^{-1}}{y}_k{\phi }_k(x).$$

This is the Bayesian version of lazy learning. For given $n,\beta$, the "modes" of the data distribution with ${\lambda }_k\gtrsim (n\beta {)}^{-1}$ are learned by the network, and the rest are projected out.

Recall that in the Bayesian context, the rescaled inverse-temperature $n\beta$ should play a role somewhat analogous to training time in SGD/GD. We can see this explicitly here. For example, in SGD/GD lazy learning, the characteristic time to learn an NTK mode $s_{\mu }^2$ is $t\approx 1/(\eta s_{\mu }^2)$ (see (4.35)). Correspondingly, in Bayesian lazy learning, the minimum inverse-temperature required to learn a GP mode is $\beta n\approx 1/{\lambda }_k$.

5. Mean-field scaling

Sections 3 and 4 analyzed one and the same large-width limit: the fan-in normalization (Section 3.2) with order-one weights and an order-one learning rate, and width $N\to \infty$ at fixed depth and data. At initialization the network converges to a Gaussian process; during training the NTK freezes, the network linearizes, and (by the definition of Section 4.2) no features are learned. It is tempting to conclude that infinite width implies laziness. The lesson of this section is that it does not, and that laziness was a consequence of a choice of parametrization, rather than of width. Changing the normalization of the output layer by a single extra factor of $\sqrt{N}$ produces a different, equally well-defined infinite-width limit in which the network starts at zero, individual weights move an order-one distance, the NTK evolves in time, and features are genuinely learned.

This alternative limit is called the mean-field limit. The name comes by analogy to a classic setup in statistical physics: an interacting particle system with weak all-to-all couplings, whose $N\to \infty$ behavior is governed by a self-consistent "mean field." We'll first explain what physicists mean by a mean-field theory (Section 5.1), then run the analogy in detail for one-hidden-layer networks (Section 5.2), see how the NTK story of Section 4 changes (Section 5.3). Finally we'll discuss some practical applications, including interpolating scalings, $\mu$P, and what this framework is actually used for (Sections 5.4–5.5).

The mean-field limit of one-hidden-layer networks was constructed independently, and essentially simultaneously, by four groups (Mei et al. 2018; Chizat & Bach 2018; Rotskoff & Vanden-Eijnden 2022; Sirignano & Spiliopoulos 2020); our treatment draws on all of them. Pedagogical accounts may be found in (Spiliopoulos et al. 2025, Ch. 20) and (Hanin 2026, Lecture 4).

5.1 Mean-field theories in physics

"Mean-field theory" is one of the oldest and simplest approximation schemes in statistical mechanics: replace the fluctuating environment felt by each degree of freedom with its average (a macroscopic order parameter), then demand self-consistency upon recalculating this average. For most systems this is an uncontrolled approximation, though often a remarkably successful one cf. (Tong 2012, Chapter 1). There is, however, a family of models — those with weak, all-to-all couplings — for which it becomes exact as $N\to \infty$; and networks in mean-field scaling will turn out to be precisely such a model.

5.1.1 Warm-up: the Curie–Weiss magnet

The Curie–Weiss magnet is the Ising model of Section 2.4 on the fully connected graph: every spin couples to every other, with the coupling scaled down to $J/N$ so that the total force on any one spin, and the total energy, stay of order $N^0$ and $N^1$ respectively. Setting $J=1$, the energy (2.19) becomes

The second equality is exact, not an approximation: on the complete graph the energy depends on the spins only through the magnetization. Compare the mean-field energy (2.22) of the lattice model, with $dJ\to 1$. Everything that was approximate in Section 2.4 is therefore exact here. The partition function reduces to the one-dimensional integral (2.25) with the same entropy $h(m)$, the saddle-point equation is

and the model has the second-order transition of Figure 4 at ${\beta }_c=1$ and the first-order transition of Figure 2 in $B$, exactly as analyzed in Section 2.5.

The reason mean-field theory is exact here is worth stating in a form that carries over to networks. The field felt by a single spin is $-\partial E_N/\partial s_i=m+B$: spin $i$ does not care about any individual other spin, only about the empirical average $m$, the "mean field." The classic argument then has three features:

1. Each degree of freedom couples to the others only through an average. (True by construction here.)
2. The average barely fluctuates. If the spins are roughly independent, the central limit theorem makes the fluctuations of $m$ of size $O(1/\sqrt{N})$; as $N\to \infty$ each spin sees a deterministic effective field $m+B$.
3. The average is fixed self-consistently. A single spin in the field $m+B$ has $\mathbb{E}(s_i)=\tanh (\beta (m+B))$; since $m$ was defined as the average spin, consistency requires (5.2).

For a finite-dimensional lattice, where each spin has a few strongly coupled neighbors, steps 1–2 are approximations. For the $1/N$ all-to-all model they become exact as $N\to \infty$: the mean field is an average of $N$ weakly correlated terms, and the law of large numbers kills its fluctuations. Networks in mean-field scaling will turn out to be models of exactly this kind.

Exercise 5.1 (Curie–Weiss by fixed-point iteration). Solve (5.2) numerically by the fixed-point iteration $m^{(k+1)}=\tanh (\beta (m^{(k)}+B))$, for $\beta$ below and above $1$ and for a few values of $B$, and watch the iteration converge. Which fixed points are stable? Keep this iteration in mind: we will meet its infinite-dimensional cousin below.

Solution

The iteration converges geometrically to a stable fixed point, at rate set by the slope $\beta (1-m^2)$ of the right-hand side there. For $\beta <1$ the only fixed point is $m=0$ and it is stable. For $\beta >1$ the fixed point $m=0$ has slope $\beta >1$ and is unstable: any seed away from it flows to one of the two nonzero fixed points $\pm m^{\ast }$, which are stable, and a small $B$ selects the one with the sign of $B$ (the other survives as a stable fixed point too, the metastable state of Section 2.5). Its infinite-dimensional cousin is the fixed-point iteration for the McKean–Vlasov equation in Section 5.2.

5.1.2 The general shape of a mean-field model

Nothing in the story above needed binary spins. More abstractly, one can take $N$ "particles" ${\theta }_i\in {\mathbb{R}}^p$ with an extensive energy of the form

with a single-particle potential $V$ and a symmetric pair interaction $U$, again with the characteristic $1/N$ coupling. The force felt by particle ${\theta }_i$ is $-{\nabla }_{{\theta }_i}{E}_N$. More so, for particles dominated by drag (highly viscous motion), the dynamics at zero temperature are governed by familiar gradient-descent equations

where ${\nabla }_1$ differentiates the first slot of $U$. Once more each particle feels only an empirical average of forces.

It is worth packaging this observation in a way that will carry over verbatim to networks. Define the empirical measure

a probability measure on ${\mathbb{R}}^p$ recording where the collection of $N$ particles sit. Then (5.4) says that every particle descends one and the same effective single-particle potential, shaped by where everyone currently is:

The particles are coupled only through ${\rho }_N$ — the exact analogue of "every spin feels only $m$." The same three-step argument as before then suggests that as $N\to \infty$ the empirical measure becomes deterministic and self-consistent. We will carry this out carefully, for networks, in the next subsection.

5.2 Mean-field scaling for one-hidden-layer networks

Return to the one-hidden-layer network (4.26) of Sections 3.5 and 4.3, but now normalized with $1/N$ out front instead of $1/\sqrt{N}$, with the same order-one weights:

with $c_i\in \mathbb{R}$, $a_i,x\in {\mathbb{R}}^d$ (as in (4.26), the first-layer factor $1/\sqrt{d}$ has been absorbed into $x$). Compared with the fan-in normalization of Sections 3–4, mean-field scaling suppresses the output by one extra factor of $1/\sqrt{N}$. The function class is identical — only the coordinates on it have changed. However, gradient flow is not invariant under reparametrization: as emphasized in Section 4.1.1 and in the callout of Section 4.3, gradient descent presupposes the Euclidean metric on parameter space, and rescaling coordinates rescales that metric. Two parametrizations of the same function class can (and here will) flow along completely different trajectories.

For a mean-field analysis, our $N$ particles are going to be the $N$ hidden neurons, with parameters

$${\theta }_i=(c_i,a_i)\in {\mathbb{R}}^{d+1},i=1,...,N.$$

Their values are encoded in the empirical measure (5.5), and the network (5.7) depends on its parameters only through ${\rho }_N$,

We think of the network value $f_{\rho }$ as a linear functional of $\rho$. Neurons are viewed as exchangeable particles, and the measure is their state.

5.2.1 Initialization

At initialization, the summands $c_i\varphi (a_i\cdot x)$ in (5.7) are i.i.d. with mean zero (since $c_i$ is independent of $a_i$ and $\mathbb{E}[c_i]=0$). With the $1/N$ prefactor, the law of large numbers gives, for each fixed $x$,

$$f_{{\theta }_0}(x)=\underset{=0}{\underbrace{\mathbb{E}[c\varphi (a\cdot x)]}}+O(1/\sqrt{N})\stackrel{N\to \infty }{\to }0.$$

Contrast this with Section 3.4: with the $1/\sqrt{N}$ normalization, the same sum sits exactly at the scale of its fluctuations, and the CLT produces an order-one Gaussian process. Mean-field scaling instead sits at the scale of the mean.

The network begins training as (essentially) the zero function; there is no random-function prior to speak of, and whatever the trained network becomes is built by training rather than inherited from initialization.

5.2.2 The loss is a mean-field energy

Let's train on MSE loss over the data $(x^{\mu },y^{\mu }{)}_{\mu =1}^n$, as in Section 4:

$$\mathcal{L}(\theta )=\frac{1}{2}\sum _{\mu =1}^n(f_{\theta }(x^{\mu })-y^{\mu }{)}^2.$$

Inserting (5.7) and expanding the square, we get

with

Comparing (5.12) with (5.3), we see that up to an overall factor of $N$, the loss is precisely a mean-field energy. In particular, $E_N:=N\mathcal{L}$ has the canonical normalization from (5.3) and is extensive, while the loss itself can be interpreted as the intensive energy per neuron.

We can also identify physical roles for the two potentials! $V$ rewards neurons for aligning their output function $c\varphi (a\cdot \cdot )$ with the data, playing the role of the external field $B$; while $U$ is a data-weighted interaction between pairs of neurons that penalizes redundant features, playing the role of the spin-spin coupling.

5.2.3 Equations of motion and the mean field

Because $\partial f/\partial {\theta }_i=O(1/N)$ in this parametrization, raw gradients of $\mathcal{L}$ are $O(1/N)$, and gradient flow at fixed learning rate would freeze as $N\to \infty$. We must therefore scale the learning rate up,

with $\tilde{\eta }$ fixed. (Equivalently, we could run vanilla gradient flow on the extensive energy $E_N=N\mathcal{L}$.) This is the rule of Section 4.3 at work: the NTK, computed below in (5.24), is $O(1/N)$, so keeping $\frac{d}{dt}{f}^{\mu }=-\eta {\Theta }^{\mu \nu }{\Delta }^{\nu }$ of order one requires $\eta \propto N$. Note the contrast with Section 4.3: there the NTK was of order one and the learning rate needed no rescaling; here the NTK shrinks like $1/N$ and we scale up. Recall also the caveat from that section about changes of variables. The mean-field network could equally be written without a prefactor, $f={\sum }_i{\tilde{c}}_i\varphi (a_i\cdot x)$ with ${\tilde{c}}_i=c_i/N\sim \mathcal{N}(0,1/N^2)$; but then the same dynamics (5.15) requires a learning rate $\tilde{\eta }/N$ for the readout weights ${\tilde{c}}_i$ and $N\tilde{\eta }$ for the hidden weights $a_i$ — one learning rate per layer, differing by a factor $N^2$. Keeping all weights of order one is what lets a single $\eta$ do the job, and it is the reason we adopted that convention in Section 3.2.

With (5.14), the flow $\dot{\theta }=-\eta {\nabla }_{\theta }\mathcal{L}$ becomes, per neuron,

where ${\Delta }^{\mu }=f_{{\rho }_N}(x^{\mu })-y^{\mu }$ is the residual of Section 4.1. Equivalently, in the potential form (5.6),

with ${\Delta }_{\rho }^{\mu }=f_{\rho }(x^{\mu })-y^{\mu }$.

As anticipated, the velocity of neuron $i$ depends on its own state $(c_i,a_i)$, and on all the other neurons only through the average $f_{{\rho }_N}=\frac{1}{N}{\sum }_j{c}_j\varphi (a_j\cdot \cdot )$, via the residual ${\Delta }^{\mu }$. The network function $f$ is the mean field here! We arrive at a dictionary:

Curie–Weiss	one-hidden-layer network, mean-field scaling
spin $s_i\in \{\pm 1\}$	neuron ${\theta }_i=(c_i,a_i)\in {\mathbb{R}}^{d+1}$
magnetization $m=\frac{1}{N}{\sum }_j{s}_j$	network function $f_{\rho }=\frac{1}{N}{\sum }_j{c}_j\varphi (a_j\cdot \cdot )$
external field $B$	data term $V$ (the targets $y^{\mu }$)
pair coupling $-\frac{1}{N}{s}_i{s}_j$	neuron-neuron interaction $\frac{1}{N}U({\theta }_i,{\theta }_j)$
effective field $m+B$	residual ${\Delta }_{\rho }^{\mu }=f_{\rho }(x^{\mu })-y^{\mu }$
extensive energy $E_N$	$N\mathcal{L}$ (the loss is intensive)
zero-temperature dynamics	gradient flow
Langevin at temperature $T$	noisy training, Section 2.1
$m=\tanh (\beta (m+B))$	self-consistent Gibbs measure, Equation 5.22 below

Exercise 5.2 (Structure of the mean-field equations). (a) Derive (5.12)–(5.13) and (5.15), and check that (5.16) reproduces them, using the symmetry of $U$.

(b) (Balancedness.) For a positively homogeneous activation such as ReLU, ${\varphi }^{\prime }(z)z=\varphi (z)$. Show that in this case (5.15) conserves $c_i^2-|a_i{|}^2$ for every neuron, exactly, along the flow. The two layers of a given neuron therefore remain coupled for all time. (On the other hand, distinct neurons will decouple from each other, as we'll see momentarily.)

(c) Observe that for MSE loss ${\dot{c}}_i$ in (5.15) does not depend on $c_i$ (only on $a_i$ and the mean field). Where does the linearity of $f_{\rho }$ in $c$ enter?

Solution

(a) Expanding the square in the loss and collecting single-sum and double-sum terms gives (5.12)–(5.13) directly. For the equations of motion, $\partial f_{{\rho }_N}(x^{\mu })/\partial c_i=\frac{1}{N}\varphi (a_i\cdot x^{\mu })$ and $\partial f_{{\rho }_N}(x^{\mu })/\partial a_i=\frac{1}{N}{c}_i{\varphi }^{\prime }(a_i\cdot x^{\mu })x^{\mu }$; the chain rule plus $\eta =N\tilde{\eta }$ gives (5.15). Writing the same gradient through $\Psi$, the symmetry $U(\theta ,{\theta }^{\prime })=U({\theta }^{\prime },\theta )$ ensures the double sum contributes twice the one-sided derivative, matching (5.16).

(b) From (5.15): $c_i{\dot{c}}_i=-\tilde{\eta }{c}_i{\sum }_{\mu }{\Delta }^{\mu }\varphi (a_i\cdot x^{\mu })$ and $a_i\cdot {\dot{a}}_i=-\tilde{\eta }{c}_i{\sum }_{\mu }{\Delta }^{\mu }{\varphi }^{\prime }(a_i\cdot x^{\mu })(a_i\cdot x^{\mu })$. For positively homogeneous $\varphi$, ${\varphi }^{\prime }(z)z=\varphi (z)$, so the two right-hand sides coincide and $\frac{d}{dt}(c_i^2-|a_i{|}^2)=2(c_i{\dot{c}}_i-a_i\cdot {\dot{a}}_i)=0$.

(c) Because $f_{\rho }$ is linear in the output weights, $\partial f/\partial c_i$ contains no $c_i$; the only route by which $c_i$ influences ${\dot{c}}_i$ is through the mean field ${\Delta }^{\mu }$ — i.e. through the population average, weighted by $1/N$. This linearity is also what makes the loss a quadratic (rather than higher-order) polynomial in each particle, i.e. what limits the interaction to be pairwise.

5.2.4 From particles to measures: the continuity equation

We now pass to the $N\to \infty$ limit, in distinct steps.

Step 1: rewrite the $N$ coupled ODEs as one equation for the empirical measure. (This remains exact at finite $N$.) Suppose particles move deterministically with some velocity field, ${\dot{\theta }}_t=v_t({\theta }_t)$, and let ${\rho }_t$ denote the law of one such particle.

The expectation of a smooth test function $g$ obeys

$$\begin{array}{rl}\frac{d}{dt}\int gd{\rho }_t& =\frac{d}{dt}\mathbb{E}[g({\theta }_t)]=\mathbb{E}[\nabla g({\theta }_t)\cdot v_t({\theta }_t)]\\ & =\int \nabla g\cdot v_{t}d{\rho }_t=-\int g\nabla \cdot ({\rho }_t{v}_t),\end{array}$$

integrating by parts in the last step. Since $g$ was arbitrary,

This is known as the continuity equation, and it simply expresses conservation of probability. Applied to our system, with $v_t=-\tilde{\eta }\nabla \Psi (\cdot ;{\rho }_{N,t})$ from (5.16), the empirical measure satisfies (weakly, since it is a sum of delta functions)

An important consequence of the mean-field ($1/N$, all-to-all) coupling here is that (5.19) is a closed equation in ${\rho }_N$.

Step 2: the limit. At finite $N$, ${\rho }_{N,t}$ is a random, atomic measure. (Atomic means it's a sum of delta functions; it's random due to random initialization.) As $N\to \infty$ with i.i.d. initialization ${\theta }_i\sim {\rho }_0$, the law of large numbers suggests — and the theorems of (Mei et al. 2018; Chizat & Bach 2018; Rotskoff & Vanden-Eijnden 2022; Sirignano & Spiliopoulos 2020) confirm — that ${\rho }_{N,t}$ converges (weakly) to ${\rho }_t$, a deterministic curve of smooth probability measures solving the limiting PDE

There are quantitative $O(1/\sqrt{N})$ error bounds at any fixed time horizon.

5.2.5 The McKean–Vlasov equation and self-consistency

In this context, the analogue of the Curie–Weiss self-consistency (5.2), namely $m=\tanh (\beta (m+B))$, is called the McKean–Vlasov equation. To get it, focus on a single representative neuron ${\theta }_i$. As $N\to \infty$, we can replace the force on it by the ensemble average:

The constraint ${\rho }_t=\mathrm{Law}({\bar{\theta }}_t)$ is the entire content of the equation: the particle is driven by a field built from a distribution, and that distribution is its own law.

(Probabilists call (5.21) a nonlinear Markov process, nonlinear not in $\theta$ but in the law.) The McKean–Vlasov equation is equivalent to the continuity equation; the particle trajectories of (5.21) are the characteristic curves of the PDE (5.20).

We remark on a few structural features:

• Propagation of chaos. Since neurons interact only through a quantity (${\rho }_N$, equivalently $f_{{\rho }_N}$) that becomes deterministic as $N\to \infty$, they become asymptotically independent: any fixed collection of $k$ neurons converges to $k$ i.i.d. copies of (5.21), with cross-neuron correlations of size $O(1/N)$. Thus, independence at initialization propagates instead of degrading (Sznitman 1991).
• Gradient-flow structure. Define the $N\to \infty$ loss as a functional of the measure, $\mathcal{L}[\rho ]=\frac{1}{2}{\sum }_{\mu }(f_{\rho }(x^{\mu })-y^{\mu }{)}^2$. Since $f_{\rho }$ is linear in $\rho$, this functional is convex in $\rho$.

This ultimately leads to global-convergence theorems. For example, it was shown in (Chizat & Bach 2018) that for homogeneous activations (e.g. ReLU), if training converges at all, it converges to a global minimizer of the convex functional $\mathcal{L}[\rho ]$.

• Temperature. If one trains instead with the Langevin dynamics of Section 2.1, Eqn. (2.6), at temperature $T={\beta }^{-1}$, one finds convergence to a stationary distribution that satisfies (Mei et al. 2018)

This is another version of the "$m=\tanh (\beta (m+B))$" consistency condition, now for noisy training.

How would one actually solve the self-consistent system (5.21)? The same way one solves (5.2) (Exercise 5.1): freeze a guessed environment $({\rho }_t)$, solve the now-ordinary ODE, compute the law of the solution, feed it back, iterate. On short time horizons this map is a contraction, which is how existence, uniqueness, and propagation of chaos are all proven in one stroke (Sznitman 1991). Alternatively, solve the PDE (5.20) on a grid, which is feasible (only) for small $d$.

In Figure 6 we instead solve the system by simply training a network with $d=1$. In the left panel, each dot is a neuron. Training is a flow of the cloud of neurons, and the cloud condenses onto the low-dimensional structures in weight space that realize the target — learning its features. Exercise 5.3 reproduces this figure, and sets it beside the frozen cloud of the lazy network.

Exercise 5.3 (★) (Numerics: the mean-field network (lab)). Continue the lab of Exercise 4.4: same data, same teacher $y(x)=2\tanh (3x)-\frac{3}{2}\tanh (x)$, same student (4.40), but now with the mean-field exponent $\gamma =1$ and the rescaled updates (5.15),

$$\begin{array}{rl}{c}_i& -=\Delta \tau \sum _{\mu }{\Delta }^{\mu }\tanh (a_i{x}^{\mu }),\\ a_i& -=\Delta \tau c_i\sum _{\mu }{\Delta }^{\mu }(1-{\tanh }^2(a_i{x}^{\mu }))x^{\mu },\end{array}$$

with $\Delta \tau =5\times 10^{-4}$ for $4\times 10^4$ steps, so that the rescaled time $\tau =\tilde{\eta }t$ runs to $20$. Note that there is no $1/N$ in these updates: that is the required $O(N)$ learning rate (5.14), already built in ($\eta =N\Delta \tau$). Monitor the NTK (4.41) with $\gamma =1$.

(a) It fits. Train and check that the loss becomes small, if more slowly in rescaled time than the lazy run.

(b) Initialization. Over about $200$ seeds, histogram $f_0(x)$ at $x=1$ for several $N$: the distribution collapses to $0$ with width $\sim 1/\sqrt{N}$. Check the scaling.

(c) The kernel moves. For $N\in \{30,100,300,1000\}$, record $\Vert {\Theta }_{\mathrm{end}}-{\Theta }_0\Vert /\Vert {\Theta }_0\Vert$ and plot it together with the lazy curve of Exercise 4.4(c): one plot, two curves, the whole dichotomy.

(d) Watch the particles. Scatter-plot the neuron cloud $(a_i,c_i)$ at $\tau \approx 0,0.4,2,20$ for both runs. Lazy: the cloud is static (each weight moves $O(1/\sqrt{N})$). Mean-field: the cloud transports and condenses onto thin filaments, the configurations realizing the teacher, as in Figure 6. Repeat at $N=3000$: the pattern is the same, because you are looking at a deterministic limiting measure.

(e) Now play. The starter teacher is itself a two-neuron $\tanh$ network, so the student's features can align exactly with the teacher's. Is that feature learning, or feature copying? Try teachers outside the model class and watch what the cloud does instead: $y(x)=\sin (2x)$; $y(x)=|x|-1$; a ReLU student on the $\tanh$ teacher (and vice versa); or move to $d=2$ with $y(x)=\tanh (x_1+x_2)$ and plot the cloud of directions $a_i/|a_i|$. Which aspects of (c)–(d) are universal, and which were special to the aligned case?

Solution

Reference results (single seeds).

(a) Final loss $\sim 5\times 10^{-3}$ at $\tau =20$ and still slowly falling; the mean-field run is measured in rescaled time, so run longer to go lower.

(b) At $x=1$, $f_0\sim \mathcal{N}(0,O(1/N))$, with standard deviation falling like $1/\sqrt{N}$ across $N$: the Gaussian process of Section 3 survives only as the $O(1/\sqrt{N})$ fluctuation around zero.

(c) Measured $\Vert \Delta \Theta \Vert /\Vert {\Theta }_0\Vert \approx 1.7,0.8,0.7,0.9$ at $N=30,100,300,1000$: order one, with no decay, against the decaying lazy curve. Use the same $\Theta$ formula with the appropriate $\gamma$ in both runs.

(d) The mean-field cloud reproduces Figure 6: neurons stream along order-one trajectories and pile onto one-dimensional filaments through $(a,c)\approx$ (teacher directions, matched output weights), with the point symmetry $(a,c)\to (-a,-c)$ from the oddness of $\tanh$. The lazy cloud is visually identical at $\tau =0$ and $\tau =20$ (displacements $\sim 1/\sqrt{N}$).

(e) For $y=\sin (2x)$ or $|x|-1$ the loss still falls (more slowly), and the cloud condenses onto different, task-shaped structures, with more filaments spread over a range of slopes $a$, since no finite set of teacher neurons exists to copy; the kernel still moves by $O(1)$. That is the cleaner demonstration that mean-field training builds features rather than merely locating the teacher's. Mismatched activations behave similarly. The lazy runs, by contrast, change in only one way, the quality of the fit (the random features either happen to span the target well or do not); the kernel and the cloud never adapt.

5.3 The NTK is no longer constant: feature learning

Section 4.3 showed that in the fan-in normalization the NTK freezes as $N\to \infty$, and Section 4.2 defined feature learning as change of the NTK during training. Let's redo that computation in mean-field scaling. From (5.7), $\partial f/\partial c_i=\frac{1}{N}\varphi (a_i\cdot x)$ and $\partial f/\partial a_i=\frac{1}{N}{c}_i{\varphi }^{\prime }(a_i\cdot x)x$, so the NTK of Section 4.1 is

with the per-neuron kernel

$$k_{\theta }(x,x^{\prime }):=\varphi (a\cdot x)\varphi (a\cdot x^{\prime })+c^2{\varphi }^{\prime }(a\cdot x){\varphi }^{\prime }(a\cdot x^{\prime })x\cdot x^{\prime }.$$

Thus $\Theta =O(1/N)$, recovering the previous observation that gradients are small, which was the reason for the rescaling (5.14). The estimates of Section 4.3 also explain why the kernel is now free to move. The Hessian blocks (4.27) carry $1/N$ instead of $1/\sqrt{N}$, so $\Vert {\nabla }_{\theta }^{2}f\Vert =O(1/N)$; but the learning rate is $\eta =N\tilde{\eta }$, so the bound (4.25) gives for the relative drift $\Vert \dot{\Theta }\Vert /\Vert \Theta \Vert \lesssim \eta \Vert {\nabla }_{\theta }^{2}f\Vert |\Delta |=O(1)$. The bound no longer forces the kernel to freeze, and the equations of motion (5.15) will show that it does move. As $N\to \infty$, the meaningful object is $N\Theta =\int k_{\theta }d{\rho }_N$, and the function-space dynamics (4.10) now becomes a kernel gradient flow with a time-dependent kernel:

At initialization the LLN makes $N\Theta \to {\mathbb{E}}_{{\rho }_0}[k_{\theta }]$, a deterministic kernel — the same limiting kernel (4.24) as in the fan-in normalization. The two limits start from the same place.

During training, however, they part ways. In Section 4.3, each parameter moved only $O(1/\sqrt{N})$, so ${\rho }_N$ stayed within $O(1/\sqrt{N})$ of ${\rho }_0$ and the kernel was pinned, (4.31). Here, the equations of motion (5.15) move each particle an order-one distance. (Each neuron has only $O(1/N)$ leverage, so fitting the data forces the average neuron to move a distance $O(1)$.) Thus ${\rho }_t$ transports macroscopically, and by (5.26) the kernel is dragged along at order one as well.

5.4 The scaling dial and Maximal Update Parameterization

It's quite instructive to look at an entire one-parameter family of scalings, which has fan-in/NTK scaling and mean-field scaling as its endpoints. The family can be encoded via a prefactor

where $c_i,a_{ij}\sim \mathcal{N}(0,1)$. For each $\gamma$, as $N\to \infty$, the learning rate must be simultaneously be scaled as $\eta =N^{2\gamma -1}\tilde{\eta }$ so that the function-space dynamics stays $O(1)$ (one can derive that from $\Theta \sim N^{1-2\gamma }$). Repeating the estimates above gives (see Exercise 5.4):

	$f_{{\theta }_0}$ at init	per-neuron motion	limit
$\gamma =\frac{1}{2}$	$O(1)$ (GP, CLT)	$O(N^{-1/2})$	lazy: Sections 3 and 4
$\frac{1}{2}<\gamma <1$	$O(N^{\frac{1}{2}-\gamma })\to 0$	$O(N^{\gamma -1})\to 0$	lazy (frozen kernel, zero init)
$\gamma =1$	$O(N^{-1/2})\to 0$	$O(1)$	mean-field: this section

Notably, every intermediate $\gamma$ collapses onto the kernel limit: the initialization GP is scaled away, but the neurons still move vanishingly little, so training is again linear in an (appropriately rescaled) frozen kernel (Chizat et al. 2019; Mei et al. 2018). Mean-field, $\gamma =1$, is the unique scaling in this family with both a well-defined limit and order-one feature movement; it's the "maximal update" that one can take without blowing up. This uniqueness can be generalized from one hidden layer to arbitrary depth (where consistency forces layer-dependent learning-rate and initialization scalings), and is known as the Maximal Update Parametrization, $\mu$P, in Yang and Hu's line of work on tensor programs (Yang & Hu 2021). In particular, mean-field scaling is $\mu$P for $L=1$.

Exercise 5.4 (Lazy vs. rich along the dial). For the family (5.27):

(a) show $\Theta =O(N^{1-2\gamma })$ and hence that $\eta =N^{2\gamma -1}\tilde{\eta }$ is the right co-scaling;

(b) show the initialization output is $O(N^{1/2-\gamma })$ and per-neuron velocities are $O(N^{\gamma -1})$;

(c) conclude that the (normalized) kernel drift over any fixed training horizon is $O(N^{\gamma -1})$, vanishing for every $\gamma <1$;

(d) reconcile $\gamma =\frac{1}{2}$ with the bounds of Section 4.3. Optional: relate $\gamma$ to the output-scale parameter $\alpha$ of (Chizat et al. 2019), where multiplying any model by $\alpha \to \infty$ produces lazy training.

Solution

(a) $\partial f/\partial c_i=N^{-\gamma }\varphi (a_i\cdot x)$ and similarly for $a_i$, so $\Theta ={\sum }_i{N}^{-2\gamma }{k}_{{\theta }_i}=N^{1-2\gamma }\cdot \frac{1}{N}{\sum }_i{k}_{{\theta }_i}=O(N^{1-2\gamma })$; demanding $\eta \Theta =O(1)$ forces $\eta =N^{2\gamma -1}\tilde{\eta }$.

(b) At initialization $f=N^{-\gamma }{\sum }_i{c}_i\varphi (a_i\cdot x)$ is $N^{-\gamma }$ times a sum of $N$ i.i.d. mean-zero terms $=O(\sqrt{N})$, so $f_{{\theta }_0}=O(N^{1/2-\gamma })$. Per-neuron velocities: ${\dot{c}}_i=-\eta \partial \mathcal{L}/\partial c_i=O(N^{2\gamma -1}\cdot N^{-\gamma })=O(N^{\gamma -1})$, and likewise for ${\dot{a}}_i$.

(c) The normalized kernel $N^{2\gamma -1}\Theta =\frac{1}{N}{\sum }_i{k}_{{\theta }_i}$ is an empirical average over particles, so over any fixed horizon its drift is of the order of the per-particle displacement, $O(N^{\gamma -1})$ — vanishing for every $\gamma <1$, order one exactly at $\gamma =1$.

(d) At $\gamma =\frac{1}{2}$: per-neuron motion $O(N^{-1/2})$, matching (4.30) (an $O(1)$ total displacement shared among $N$ neurons); initialization output $O(1)$, the GP of Section 3. For the optional part: relative to the mean-field parametrization (5.7), $f_{\gamma }=\alpha f_{\mathrm{MF}}$ with $\alpha =N^{1-\gamma }$, so in the language of (Chizat et al. 2019) every $\gamma <1$ sends the laziness $\alpha \to \infty$ with width — lazy — while $\gamma =1$ keeps $\alpha$ of order one.

5.5 What is this framework good for?

Nobody solves the McKean–Vlasov PDE (5.20) for a real network! There are practical payoffs, though, of three different kinds: scaling prescriptions, mechanism identification, and reduced descriptions that downstream theories build on.

Hyperparameter transfer ($\mu$P) is one key application. The mean-field analysis is, fundamentally, a theory of which joint scalings of initialization and learning rate keep the training dynamics nondegenerate and learning features as $N\to \infty$. Its deep-network generalization (Yang & Hu 2021) turns this into an engineering recipe, $\mu$Transfer (Yang et al. 2022): parametrize the network so that its infinite-width limit is the feature-learning one, and then optimal hyperparameters become approximately width-independent, so one can tune a small proxy model and transfer the settings to the large one. In the original demonstration this retuned GPT-3 6.7B at a few percent of pretraining cost and outperformed the published model; variants of this parametrization are now standard in large-scale training.

The dial of Section 5.4 also lets one interpolate between regimes on real architectures, and measure where feature learning actually pays off. Kernel-regime behavior tends to be competitive on small datasets, while on CIFAR-scale image tasks trained networks beat their own frozen NTKs by large margins (Geiger et al. 2020). Diagnostics from this circle of ideas (kernel distance/alignment during training, e.g. the silent-alignment effect (Atanasov et al. 2022)) are now routine tools for asking "is my network actually learning features, or just doing kernel regression?"

The mean-field limit here holds data fixed while sending $N\to \infty$, which unfortunately does not yet make it a quantitative theory of generalization at realistic data/width ratios. The dynamical mean-field theory of Section 6 (Bordelon & Pehlevan 2022) will rectify this somewhat; it is a field-theoretic refinement of the current analysis, whose extensions do reproduce compute-optimal neural scaling laws, in particular at large dataset size (Bordelon et al. 2024).

5.6 Grokking as a lazy-to-rich transition: a reading exercise

We finish this section with two guided readings that apply various concepts introduced above to one of the most interesting and fundamental phenomenons in learning: grokking (Power et al. 2022). Grokking refers to the following behavior. One trains a small network on an algorithmic dataset — the canonical example is addition mod $P$, presented as pairs $(n,m)$ with label $(n+m)\mathrm{mod}P$, with only a fraction of all $P^2$ facts shown. The network reaches perfect training accuracy quickly, while test accuracy sits at chance: it has simply memorized. Then, orders of magnitude later in training and long after the train loss has flattened, test accuracy jumps abruptly to near $100\%$. Such delayed, abrupt generalization, with emergent capabilities, demands a mechanism.

Theory has produced (at least) two quantitative explanations, and this subsection and the next walk through one each: the dynamical lens of Kumar–Bordelon–Gershman–Pehlevan (Kumar et al. 2024) (this subsection) — grokking as a delayed transition from lazy to rich dynamics — and the equilibrium lens of Rubin–Seroussi–Ringel (Rubin et al. 2024) (Section 5.7) — grokking as a first-order phase transition with metastable lag. The two walkthroughs are parallel in structure and use largely disjoint toolkits from these notes. The dynamical one presumes comfort with Sections 4 and 5.4; the Bayesian one is a denser read, and combines scaling analysis with a mean-field approximation.

The work of Kumar, Bordelon, Gershman, and Pehlevan (Kumar et al. 2024) invokes no statistical-mechanics ensemble at all. There is only gradient descent, and the lazy/rich dial of Section 5.4, unfolding in time. The paper claims that a network groks when 1) it first fits the training data through its cheap, fast channel (kernel regression with the frozen NTK of its initialization), and that lazy fit memorizes without generalizing because the initial kernel is misaligned with the task; then 2) on a slower timescale controlled by the network's laziness, the features finally move (the kernel rotates, as in Section 5.3), a generalizing solution is found, and test loss falls long after train loss did. Two beautiful features of this work are that its clearest examples grok under vanilla gradient descent with zero weight decay, and that the parameter norm increases during grokking. Both are direct counterexamples to earlier accounts of grokking built on weight decay and shrinking norms.

The paper is written almost entirely in the language these notes have already built (Sections 4 and 5), so the dialect box below is short. Read the introduction, the two-layer polynomial-regression model, and the experiments (modular arithmetic, MNIST, a one-layer transformer); skip the dynamical mean-field asymptotics on a first pass — that machinery belongs to Section 6. Then work through Exercise 5.5.

Exercise 5.5 (Grokking as a lazy-to-rich transition). Read (Kumar et al. 2024) as assigned above, and answer the following, citing where in the paper each answer lives. (Parts (a)–(e) have exact counterparts in Exercise 5.6, the other walkthrough; part (f) is the shared comparison question.)

(a) What regime is the analysis set in, and what dial controls it? There is no thermodynamic limit and no equilibrium ensemble here, so what plays the role that a "scaling limit" plays elsewhere in this section? How is the laziness parameter $\alpha$ implemented, and how does it relate to the $\gamma$-dial of Section 5.4? Section 5.2's scaling prescription had two halves, an output scale and a learning-rate scale. Which half is doing the work here? (The other walkthrough meets the opposite half: see Exercise 5.6(a).)

(b) What plays the role of the order parameter? What quantity distinguishes the memorized state from the generalizing one, and how would you measure it in a real training run? Compare with the Fourier overlap $\Phi (w)$ that plays this role in the other walkthrough (Exercise 5.6(b)): what information do the two quantities share?

(c) Necessary conditions. The paper states three conditions for grokking to occur. Find them, and translate each into the language of these notes (Sections 4.6, 4.3 and 5.4).

(d) What parameter is tuned? What makes grokking more dramatic, and what makes it vanish entirely? What does the theory predict in the two extremes $\alpha \to \infty$ and $\alpha$ small? Contrast with the control parameters of Exercise 5.6(d).

(e) Where does the delay come from? Both theories must explain the gap between the train-loss drop and the test-loss jump. Here there is no metastability and no noise — so what produces two well-separated timescales in a single deterministic flow? What does each theory require to produce a delay (noise? weight decay? neither?), and how do the paper's zero-weight-decay, growing-norm experiments bear on earlier explanations of grokking?

(f) Synthesis. Are the equilibrium mechanism of Section 5.7 and the dynamical mechanism of this section incompatible, or two corners of one phenomenon? Design at least one experiment whose outcome would favor one over the other — consider, e.g., how the grokking delay should depend on training noise at fixed hyperparameters, or how the delay should fluctuate from seed to seed, under each mechanism.

Solution

(a) Finite network, deterministic gradient descent/flow, no limit required: the analysis is organized around the linearization of Section 4.3 and departures from it. The dial is an output multiplier $\alpha$ (with appropriate co-scaling of the loss/learning rate), exactly the scale parameter of lazy training (Chizat et al. 2019): at large $\alpha$ an $O(1)$ change of $f$ requires only an $O(1/\alpha )$ change of weights, so features barely move — a continuous, finite-$N$ version of the $\gamma$-dial, where $N^{\gamma -1}$ played the role of $1/\alpha$ (Exercise 5.4). And this paper is precisely about the half of the scaling story that the Bayesian treatment of Exercise 5.6(a) could not see: there, only the induced function prior survived and the learning-rate/dynamics half dropped out; here there is no prior at all, and everything is the dynamics half — the two walkthroughs literally partition Section 5.2's prescription between them.

(b) The alignment between the (current) NTK and the task — equivalently the kernel's distance from its initialization, or the paper's low-dimensional sufficient statistics: in the memorized state the kernel is still (close to) its initialization and the fit lives in misaligned modes; in the generalizing state the kernel's top eigenspaces have rotated onto the target. Measurable in any run by tracking the empirical NTK (Section 4.2). The Fourier overlap $\Phi (w)$ of Exercise 5.6(b) carries the same information — features aligned with the task's structure — read off the weights and their posterior rather than off the kernel; for modular addition, a rotated kernel and condensed Fourier modes are two descriptions of the same event.

(c) (i) The top eigenvectors of the initial NTK are misaligned with the labels: in the language of Section 4.6, the target's weight sits in modes with small eigenvalues, so the lazy (kernel-regression) solution fits the training points without generalizing — memorization, in the precise mode-filtering sense. (ii) The dataset is in a Goldilocks window: large enough that a generalizing solution exists (cf. the critical sample size of Exercise 5.6(d)), small enough that train and test loss do not simply track each other. (iii) Training starts lazy (large $\alpha$, or large initialization scale): feature learning is initially suppressed, as in Section 4.3, so the kernel fit happens first.

(d) The laziness $\alpha$ (and the initialization scale, which acts similarly): increasing it stretches the delay between the lazy fit and feature learning, making grokking more dramatic; decreasing it makes the network rich from the start, so train and test fall together and grokking vanishes. In the extreme $\alpha \to \infty$ the network is a pure kernel machine forever: by condition (i) it never generalizes, and the grokking time diverges. Contrast Exercise 5.6(d): there the knobs were sample size and temperature moving the system across a phase boundary; here the knob does not change where the system ends up so much as how long the misaligned detour lasts.

(e) From a separation of timescales built into the flow: fitting the train set within the frozen-kernel subspace is fast (the exponential convergence of Section 4, eq. (4.34)), while rotating the kernel is slow — suppressed by the laziness, since feature motion is $O(1/\alpha )$-small per unit of function change. Train loss bottoms out on the fast clock; test loss must wait for the slow one. Nothing stochastic is needed: no noise, no weight decay, no barrier crossing — and that is the force of the paper's counterexamples, since grokking with zero weight decay and growing weight norm rules out explanations in which regularization-driven norm shrinkage causes the transition. The equilibrium mechanism of Section 5.7, by contrast, needs temperature: its delay is the escape from a metastable free-energy basin, which pure gradient flow would never make.

(f) They live in different corners and need not compete: Rubin et al. describe noisy (Langevin/Bayesian) training near equilibrium, Kumar et al. deterministic training far from it; both say the cheap channel is used first — kernel fit or metastable memorizing phase — and features arrive late. Distinguishing experiments:

1. noise dependence — an activated (metastable-escape) delay should shorten rapidly with increasing training noise, roughly Arrhenius-like, while a timescale-separation delay survives at zero noise and depends on it only weakly; measure grokking time vs. Langevin temperature at fixed $\alpha$, $n$.
2. Seed-to-seed statistics — nucleation out of a metastable state is a rare-event process with broad (roughly exponential) waiting-time distributions across seeds; a deterministic mechanism predicts narrow, initialization-scale-set delays.
3. The $\alpha$-dial itself — the lazy-to-rich account makes a sharp prediction that the delay is monotonically stretched by $\alpha$ with grokking eliminated at small $\alpha$; the equilibrium account ties the delay to the free-energy landscape at fixed ensemble, not to $\alpha$. Running these on modular addition is a very feasible project.

5.7 Bayesian grokking: a first-order transition (alternative reading)

The alternative walkthrough takes the opposite starting point. Delayed, abrupt generalization should by now also trigger a reflex from Section 2.5: competing branches of a free energy, a first-order transition, and a metastable lag, exactly as dissected for the Ising perceptron in Section 2.6. The paper (Rubin et al. 2024) makes this reflex precise, and it is a fitting capstone for these notes: it assembles, in one real (if small) deep-learning phenomenon, nearly every tool we have developed: the Gibbs posterior of noisy training (Section 2.1), GP kernels and kernel regression (Sections 3.3 and 4.6), mean-field scaling and self-consistency (this section), and the anatomy of first-order transitions (Section 2.5). It is, however, a genuinely dense paper, written in an idiosyncratic dialect of the "adaptive kernel" school (Seroussi et al. 2023). This subsection is therefore structured as a reading exercise in the style of Section 1: the goal is to recognize the structures, not to verify the computations. Read the paper's introduction, setup, and modular-arithmetic results (do not read the appendices on a first pass), then work through Exercise 5.6.

Warm-up. Before any statistical mechanics, it helps to know concretely what the network is trying to become. Gromov (Gromov 2023) writes down exact weights for a two-layer network with quadratic activation that solve modular addition: the first-layer weights are plane waves on ${\mathbb{Z}}_P$ — entries $\cos (2\pi kn/P)$, $\sin (2\pi kn/P)$ at various frequencies $k$ — and the quadratic activation's product-to-sum identities convert them into the required functions of $(n+m)\mathrm{mod}P$. Skim (Gromov 2023) just long enough to believe this. The lesson is that "the Fourier coefficients of the first-layer weights" becomes the obvious coordinate system for measuring how much structure a network has actually learned.

Exercise 5.6 (Grokking as a first-order phase transition). Read (Rubin et al. 2024) as assigned above, and answer the following, citing where in the paper each answer lives. (Parts (a)–(e) have exact counterparts in Exercise 5.5, the other walkthrough; its part (f) is the shared comparison question.)

(a) What scaling limit is taken? Look for where the output-layer weight decay is fixed, and for the phrase "mean-field scaling." How does the paper's scaling relate to the parametrization (5.7) and to the $\gamma$-dial of Section 5.4? Careful — this is subtler than it looks: in an equilibrium (Bayesian) framework there is no learning rate, so which part of Section 5.2's scaling prescription survives, and in what combination?

(b) What is the order parameter for the modular-addition task? What does the posterior distribution of a single neuron's input weights look like on each side of the transition, and which figure of Section 5.2 does the ordered side resemble?

(c) What are the main computational techniques? List them, and match each to the section of these notes where its ancestor appears.

(d) What parameter is tuned to cross the phase transition in the paper's phase diagrams?

(e) Where did the time go? Grokking is a statement about training time, yet the paper computes a static equilibrium ensemble. How does a first-order transition in an equilibrium free energy produce delayed, abrupt generalization in a training run? Compare, line by line, with the metastability discussion for the Ising perceptron in Section 2.6.

Solution

(a) A Bayesian two-layer network $f(x)={\sum }_{i=1}^N{a}_i\varphi (w_i\cdot x)$ whose readout prior (set by the output-layer weight decay) has variance $\mathbb{E}[a_i^2]={\sigma }_a^2/N$; the limit taken is $N\to \infty$ together with ${\sigma }_a^2\to 0$ — the paper's "mean-field scaling" — plus the EK (dense-data) approximation. The subtlety: at equilibrium, only the induced prior on functions matters. For the linear readout, a prefactor $\kappa$ in $f=\kappa {\sum }_i{a}_i\varphi$ and a prior variance ${\sigma }^2$ for $a_i$ enter the function-space prior only through the product ${\kappa }^2{\sigma }^2$ (a Gaussian change of variables). So the mean-field parametrization (5.7) — prefactor $1/N$, order-one weights — corresponds to bare readout variance $\propto 1/N^2$, while variance ${\sigma }_a^2/N$ at fixed ${\sigma }_a^2$ would be precisely the standard GP scaling of Section 3. The mean-field regime is the additional suppression ${\sigma }_a^2\to 0$. Meanwhile the other half of Section 5.2's prescription — the learning-rate rescaling (5.14) — has no equilibrium counterpart at all: the learning rate drops out of the stationary distribution (Section 2.1). In the language of the $\gamma$-dial: gradient-flow scalings are classified by (prefactor, initialization, learning rate) jointly, but their Bayesian shadows are classified by the single combination ${\kappa }^2{\sigma }^2$.

(b) The overlap $\Phi (w)$ of a hidden neuron's input weights with the teacher-relevant directions — for modular addition, the Fourier coefficients of $w$ on ${\mathbb{Z}}_P$ (the warm-up (Gromov 2023) says which ones matter). The sharper statement is about the single-neuron posterior $P(w)$: in the GFL phase it is a single Gaussian centered at $\Phi =0$; in the mixed phase it becomes a mixture, with a finite fraction of neurons condensed at $\Phi \ne 0$ and the rest still Gaussian. The order parameter jumps discontinuously: first order. The ordered side is a finite-temperature cousin of Figure 6, where the measure over neurons condenses onto the low-dimensional structures realizing the target.

(c) (i) Noisy training $\to$ Gibbs posterior (Section 2.1); (ii) mean-field decoupling of the posterior into readout $\otimes$ hidden-layer factors, with neurons coupled only through averages (this section); (iii) the self-consistent "adaptive" kernel $Q_{\mu \nu }={\sigma }_a^2⟨\varphi (w\cdot x^{\mu })\varphi (w\cdot x^{\nu }){⟩}_{\mathrm{posterior}}$, whose defining equation has exactly the structure of (5.22) — the kernel depends on a posterior that depends on the kernel; (iv) saddle-point evaluation (Section 2.4); (v) a variational Gaussian-mixture ansatz, needed precisely because a first-order transition means several competing saddles; (vi) the EK approximation for the data average (Section 4.6). Note what is absent: in the main text the data averaging is done by EK smoothing, not by the replica trick of Appendix A.2.

(d) Primarily the sample size $n$ at fixed temperature: below a critical number of training facts the disordered (memorizing) phase is dominant and the network never generalizes; above it, the mixed phase wins. (The transition line lives in the $(n,\text{temperature})$ plane, so it can equally be crossed by tuning the noise level or weight decay at fixed $n$ — but the grokking narrative crosses it in $n$.)

(e) In equilibrium there is no time; the free-energy crossing itself is instantaneous. The time-dependence of grokking is the dynamical lag of a first-order transition: Langevin training equilibrates quickly within the metastable disordered basin — which is enough to drive the train loss to zero (memorization is easy there) while test accuracy stays at chance — and only much later escapes to the thermodynamically dominant mixed phase, whereupon test accuracy jumps. This is precisely the Ising-perceptron story of Section 2.6: the information needed to generalize is present in the data well before the dynamics finds the ordered state, so capability arrives "abruptly and later than it should."

6. Non-equilibrium physics, MSRJD, and DMFT

So far our statistical-mechanics toolkit has been an equilibrium one: Gibbs measures, partition functions, and free energies (Section 2), together with limits in which training simplifies to kernel regression (Section 4) or to a deterministic flow of measures (Section 5). This section introduces the formalism that confronts the dynamics of learning head-on. The strategy is borrowed from non-equilibrium statistical physics: write the stochastic dynamics itself as a path integral — the Martin–Siggia–Rose–Janssen–De Dominicis (MSRJD) construction — then average over the randomness (noise, initialization, and, crucially, the training data, treated as quenched disorder exactly as in Section 2.3), and finally extract closed, self-consistent equations for a small set of two-time order parameters: correlation functions $C(t,t^{\prime })$ and response functions $R(t,t^{\prime })$. The result is known as dynamical mean-field theory (DMFT), and it is the third and final incarnation of "mean field" in these notes: the statics of Section 2 determined scalars self-consistently, the McKean–Vlasov limit of Section 5 determined a measure, and DMFT determines whole functions of two times.

The payoffs in this section are a complete worked application — double descent as a dynamical phase transition in linear regression (Section 6.2) — and a guided tour of the modern DMFT program for deep learning (Section 6.3), whose scaling-law application closes the notes in Section 7. Throughout, we follow the spirit of Altland and Simons (Altland & Simons 2010), which motivates the formalism in condensed-matter language with a good balance of detail and intuition, and we borrow freely from two pedagogical tutorials written for neuroscientists, (Chow & Buice 2012) and (Stapmanns et al. 2020).

6.1 From stochastic dynamics to path integrals: the MSRJD formalism

When we model a system with fluctuations — or deliberately add noise to a deterministic model, as in the Langevin dynamics of Section 2.1 — differential equations become stochastic, and the difficulty of solving them jumps. A stochastic differential equation, at first order for example, takes the form

$$\frac{dx}{dt}=f(x)+\xi ,$$

where $\xi (t)$ is a (Gaussian) white noise process — additive noise, $\xi =\xi (t)$, as opposed to multiplicative noise $\xi =\xi (x,t)$ — and $x=x(t)$ is now a random trajectory. The noise is entirely defined by the first and second moments,

$${⟨\xi (t)⟩}_{\xi }=0,{⟨\xi (t)\xi (t^{\prime })⟩}_{\xi }=D\delta (t-t^{\prime }),$$

where the expectations are with respect to the noise ensemble³. Obtaining closed-form solutions for the time evolution of the probability distribution for $x(t)$ is usually challenging. These evolutions can be attacked by the traditional routes — the Langevin and Fokker–Planck descriptions — for which the standard references are Risken (Risken 1996), Gardiner (Gardiner 2004), and van Kampen (van Kampen 2007). The objects of interest here are instead the moments of $x(t)$ and the probability density function $p(x,t).$ Methods developed in statistical physics (in particular those used in perturbative quantum field theory and classical non-equilibrium systems) using path integrals provide a way to calculate these statistical objects. The path integral, or functional integral, approach is convenient for perturbative calculations (expanding around an exactly solvable theory with a small parameter), but also permits importing other tools like renormalization group methods or other explicitly non-perturbative approaches when there is no small parameter, or the expansion breaks down (as they often do).

The conceptual recipe of the MSRJD formalism is short, and worth internalizing before the derivation:

• write down a discrete SDE, fixing a convention (Ito) that enforces causality;
• express the transition probability between time steps as a delta function imposing the update rule;
• assemble the product over infinitesimal steps into a path integral;
• write each delta function in Fourier representation, introducing an auxiliary response field;
• average over the noise, then add sources and differentiate to extract statistics.

It turns out that the historic approach of Onsager and Machlup (Onsager & Machlup 1953) is the "Lagrangian" version of this "Hamiltonian"-like approach; the two are related by a Legendre transform, or equivalently by integrating out the auxiliary field. That auxiliary field is aptly named the response field: as we will see below, its correlation functions compute how the system responds to external perturbations, and it is the genuinely new ingredient of the non-equilibrium formalism.

We now give the derivation, quickly but completely. The standard first step is to discretize time for a continuous equation, or simply start with a discrete update rule. Under this discretization $x(t)\to x_i$, where $i=1,...,N,$ so that,

$$x_i=x_{i-1}+\Delta t(f(x_{i-1})+{\xi }_{i-1}),$$

where $\Delta t$ is the temporal spacing. We can rearrange this so that the RHS is zero and write the transition probability between steps as delta function, and the product of these representing a trajectory solution. Defining a solution of the stochastic differential equation as $x[\xi ]$, an expectation value of an observable $\mathcal{A}[x]$ is formally,

$${⟨\mathcal{A}[x]⟩}_{\xi }=\int \mathcal{D}x\mathcal{A}[x]{⟨\delta (x-x[\xi ])⟩}_{\xi }=\int \mathcal{D}x\mathcal{A}[x]\left|\frac{\delta X}{\delta x}\right|\delta (X).$$

We used the shorthand $X_i\equiv x_i-x_{i-1}-\Delta t[f(x_{i-1})+{\xi }_{i-1}]$ for the constraint imposed at each step. The measure here, is a typical functional path integral measure $\mathcal{D}x={\prod }_{i}d{x}_i$ with $\delta (x-x[\xi ])$ similarly factorized across steps. In particular we also picked a first order finite difference and the convention that the noise is evaluated at the previous time-step, which is called Ito discretization or convention. Many common processes are manifestly Ito, like SGD for example, where the noise drawn at a batch doesn't affect the weights at that iteration. There are other conventions, the most common alternative is Stratonovich (which is more like a midpoint rule), however, Ito has a unit (volume-preserving) functional determinant and other schemes require careful treatment to avoid anomalous terms from the determinant. In our case the matrix is triangular with a unit diagonal. Substituting in the definition of $X$,

$${⟨\mathcal{A}[x]⟩}_{\xi }=\int \mathcal{D}x\mathcal{A}[x]{⟨\delta ({\partial }_{t}x-f(x)-\xi )⟩}_{\xi }.$$

Finally, we can use the Fourier representation of the delta, introducing an auxiliary variable $\tilde{x}$, which in the continuum is

$${⟨\mathcal{A}[x]⟩}_{\xi }=\int \mathcal{D}x\mathcal{D}\tilde{x}\mathcal{A}[x]e^{i\int dt\tilde{x}({\partial }_{t}x-f(x)-\xi )}.$$

The final step is the payoff: the noise now appears linearly in the exponent, so the Gaussian average over $\xi$ can be done exactly, $⟨e^{-i\int dt\tilde{x}\xi }{⟩}_{\xi }=e^{-\frac{D}{2}\int dt{\tilde{x}}^2}$, leaving the MSRJD functional integral

Four structural comments, each of which does real work later:

• The partition function is one. Setting $\mathcal{A}=1$, the integral is just $\int \mathcal{D}x$ of a normalized probability: $Z_{\mathrm{dyn}}=1$, with no sources. Unlike in equilibrium, all the physics lives in correlation functions, obtained by coupling sources to both $x$ and $\tilde{x}$ and differentiating — compare the $Z_{\mathrm{dyn}}=1$ starting point of Section 6.2.
• The response field earns its name. Perturbing the drift, $f\to f+h(t)$, changes the action by $i\int \tilde{x}h$, so

Correlators of $\tilde{x}$ compute responses to kicks. Causality is built in: $R(t,t^{\prime })=0$ for $t<t^{\prime }$ (the Ito discretization is what guarantees this, along with $⟨\tilde{x}\tilde{x}⟩=0$).

• Two-time observables. The natural objects of the formalism are the correlation function $C(t,t^{\prime })=⟨x(t)x(t^{\prime })⟩$ and the response $R(t,t^{\prime })$. In equilibrium they are not independent: the fluctuation–dissipation theorem (FDT) locks them together, $TR(t,t^{\prime })=-{\partial }_{t^{\prime }}C(t,t^{\prime })$ for $t>t^{\prime }$ (with $T=D/2$ in our normalization) (Kubo 1957; Kubo 1966). A measured violation of the FDT is therefore the standard diagnostic that a system is genuinely out of equilibrium — the warning light used to great effect in Section 6.2.
• Onsager–Machlup. The action (6.7) is quadratic in $\tilde{x}$, so the response field can be integrated out exactly, giving the probability of a trajectory as $P[x]\propto e^{-\frac{1}{2D}\int dt({\partial }_{t}x-f(x){)}^2}$: the Lagrangian form promised above, in which the most probable path extremizes a classical action.

Exercise 6.1 (The Ornstein–Uhlenbeck process and the FDT). (a) Fill in the noise average leading to (6.7).

(b) For the linear drift $f(x)=-kx$ (the Ornstein–Uhlenbeck process), compute the stationary correlation and response functions, either by Fourier transforming the quadratic action or by solving the SDE directly: $C(\tau )=\frac{D}{2k}{e}^{-k|\tau |}$ and $R(\tau )=\theta (\tau )e^{-k\tau }$.

(c) Verify the fluctuation–dissipation theorem $TR(\tau )=-{\partial }_{\tau }C(\tau )$ for $\tau >0$, with $T=D/2$.

(d) Explain, from the discrete-time construction, why $⟨\tilde{x}(t)\tilde{x}(t^{\prime })⟩=0$ and why $R$ is causal.

Solution

(a) The white-noise measure is Gaussian with $⟨\xi (t)\xi (t^{\prime })⟩=D\delta (t-t^{\prime })$, so $⟨e^{-i\int \tilde{x}\xi }⟩=e^{-\frac{D}{2}\int {\tilde{x}}^2}$, and collecting terms in the exponent gives (6.7).

(b) The linear SDE is solved by $x(t)={\int }_{-\infty }^{t}ds{e}^{-k(t-s)}\xi (s)$ (stationary state). Then $C(\tau )={\int }_{-\infty }^{t}ds{\int }_{-\infty }^{t^{\prime }}d{s}^{\prime }{e}^{-k(t-s)-k(t^{\prime }-s^{\prime })}D\delta (s-s^{\prime })=\frac{D}{2k}{e}^{-k|\tau |}$ with $\tau =t-t^{\prime }$. Adding a perturbation $h$ to the drift shifts the solution by ${\int }_{-\infty }^{t}ds{e}^{-k(t-s)}h(s)$, so $R(\tau )=\theta (\tau )e^{-k\tau }$.

(c) For $\tau >0$: $-{\partial }_{\tau }C=\frac{D}{2}{e}^{-k\tau }=\frac{D}{2}R(\tau )$, i.e. $TR=-{\partial }_{\tau }C$ with $T=D/2$.

(d) In the discrete construction each delta function propagates information strictly forward in time (Ito: the noise at step $i-1$ affects $x_i$, never earlier variables). Diagrammatically, $⟨x\tilde{x}⟩$ is a retarded propagator; $⟨\tilde{x}\tilde{x}⟩$ would be a closed loop of retarded propagators, which vanishes — equivalently, "a response to no perturbation is not an observable." Causality of $R$ is the same statement.

6.2 Double descent as a dynamical phase transition

As a complete worked application of the formalism, we describe recent work of Li and Goldenfeld (Li & Goldenfeld 2026), which uses dynamical mean-field theory to analyze the double-descent phenomenon and classifies it as a dynamical phase transition, all within the simplest possible setting: a teacher–student model of linear regression. Even this very simple model requires a long calculation, but the paper is quite readable despite the overhead. The headline: in the under-parametrized regime the optimization process is ergodic and relaxes to a finite-error solution, while the over-parametrized regime is a non-ergodic phase with genuinely non-equilibrium dynamics — an initial memorization stage leaves a persistent memory of the initial conditions and confines the dynamics to a subspace around the initialization. The diagnostic is the response function $R(t,t^{\prime })$ of Section 6.1 — how a perturbation at time $t^{\prime }$ affects the model's state at time $t$; the same object appears as a Green function in quantum mechanics, signal analysis, and information theory. In the under-parametrized regime the response function decays: no memory. In the over-parametrized regime it saturates at a finite value: persistent memory of initialization. Consistently, the first regime obeys the fluctuation–dissipation theorem (Kubo 1957; Kubo 1966; Goldenfeld 2018), while in the second the FDT breaks — the standard warning light, flagged in Section 6.1, that a system has gone out of equilibrium. The model here is quite simple, and the authors propose that the frameworks here may generalize across architectures, with different universality classes based on the variations of the task, proposing random feature models with multiplicative (state-dependent) noise as an extension, with the hopes of eventually attaining an RG framework for interpreting phase transitions in neural networks.

6.2.1 Model definition and formalism

Notation. We transcribe the paper into the conventions of these notes: $P$ trainable parameters, $n$ training samples, learning rate $\eta$, teacher weights $w^{\ast }$, student weights $\hat{w}$ (with response fields $\tilde{w}$), and the overparametrization ratio

$$\nu :=P/n$$

as control parameter — the reciprocal of Section 2's load $\alpha =n/N$, so that the over-parametrized phase is $\nu >1$. For readers consulting the paper, the dictionary is:

these notes	Li–Goldenfeld (Li & Goldenfeld 2026)
$P$ (parameters)	$d$
$n$ (samples)	$N$
$\nu =P/n$	$\alpha =d/N$
$w^{\ast }$, $\hat{w}$, $\tilde{w}$	$\beta$, $\hat{\beta }$, $\tilde{\beta }$
$\eta$ (learning rate)	$\zeta$

The batch-inclusion probability $\gamma$, the ridge $\lambda$, and the label-noise variance $D$ are as in the paper.

Explicitly, the student and teacher share the same architecture

$$y^{\mu }=\frac{1}{\sqrt{P}}\sum _{i=1}^P{w}_i^{\ast }{x}_i^{\mu }+{ϵ}^{\mu }$$

where $P$ is the number of model parameters, $\mu =1,\dots ,n$ indexes the dataset with inputs $x^{\mu }$, the $w_i^{\ast }$ are the teacher's weights, and ${ϵ}^{\mu }$ is label noise. The thermodynamic limit takes $n\to \infty$ and $P\to \infty$ with the overparametrization ratio $\nu =P/n$ — the paper's complexity parameter — held fixed. The three main regimes are now definable as under-parametrized: $\nu <1$, near-criticality: $\nu \approx 1$, and over-parametrized: $\nu >1$. We note their optimization procedure, which is SGD with both regularization controlled by $\lambda$ (set to zero for most of the work) and batching controlled by the binary indicator variable $s_{\mu }$ which is drawn for each point in the dataset with probability $\gamma$,

From here they construct the MSRJD integral exactly as in Section 6.1; taking functional derivatives and averaging over the disorder (the dataset, the initialization, the batch selection, and the label noise) yields the correlation and response functions that are the main objects of study. We skip most of the details, but it is worth displaying the starting point — the dynamical partition function for gradient flow, equal to one just as promised in Section 6.1:

We want to highlight the general structure of this equation as a path integral over an action with sources coupled to both the "physical" parameter fields and the response fields,

$$\begin{array}{rl}{Z}_{\mathrm{dyn}}[\mathbf{j},\tilde{\mathbf{j}}]=⟨\int \mathcal{D}\hat{\mathbf{w}}(t)\mathcal{D}\tilde{\mathbf{w}}(t)\exp (S_{\mathrm{dyn}}& +\sum _{i=1}^P\int {\tilde{j}}_i(t){\hat{w}}_i(t)dt\\ & +\sum _{i=1}^n\int j_i(t){\tilde{w}}_i(t)dt)⟩,\end{array}$$

which is a very common object to see in statistical field theory! In general $S_{\text{dyn}}$ will hold all of the microscopic details of the theory, and this is a unique case with a fairly tractable model. In general we are not so lucky. The expectation value denoted by angled brackets is averaging over the input dataset $\{{\mathbf{x}}^{\mu }{\}}_{\mu =1}^n$, the initial condition $\hat{\mathbf{w}}(0)$, the stochastic sampling $s_{\mu }(t)$, and the added label noise $\{{ϵ}^{\mu }{\}}_{\mu =1}^n$. For comedic relief we also write their full dynamical function integrated over quenched disorder:

$$\begin{array}{rl}{Z}_{\mathrm{dyn}}& \propto \int \mathcal{D}\mathcal{B}(t)\int \mathcal{D}\mathcal{Q}(t,t^{\prime })\exp (-i\tilde{\mathbf{w}}{\partial }_t\hat{\mathbf{w}}-\lambda P{Q}_2(t,t)\\ & -P{Q}_1\cdot {\hat{Q}}_1-P{Q}_2\cdot {\hat{Q}}_2-P{Q}_3\cdot {\hat{Q}}_3-P{Q}_4\cdot {\hat{Q}}_4\\ & -\frac{n}{2}\log \det \left(\begin{array}{cc}\gamma Q_2^{\top }-\gamma Q_4+\delta & {\gamma }^2{Q}_3\\ Q_1+D& \gamma Q_2-\gamma Q_4^{\top }+\delta \end{array}\right)+\tilde{\mathbf{j}}\hat{\mathbf{w}}+\mathbf{j}\tilde{\mathbf{w}}\\ & +\int \int dtd{t}^{\prime }\sum _k\left[\right({\hat{w}}_k(t)-w_k^{\ast }){\hat{Q}}_1(t,t^{\prime })({\hat{w}}_k(t^{\prime })-w_k^{\ast })\\ & +i{\hat{w}}_k(t){\hat{Q}}_2(t,t^{\prime }){\tilde{w}}_k(t^{\prime })+i{\tilde{w}}_k(t){\hat{Q}}_3(t,t^{\prime })i{\tilde{w}}_k(t^{\prime })]\\ & +\int dt\sum _{k}i{\tilde{w}}_k(t){\hat{Q}}_4(t)w_k^{\ast }),\end{array}$$

where $\mathcal{D}\mathcal{B}(t)$ abbreviates the measure $\mathcal{D}\hat{\mathbf{w}}(t)\mathcal{D}\tilde{\mathbf{w}}(t)$ over the dynamical fields, and the new measure $\mathcal{D}\mathcal{Q}(t,t^{\prime })$ is

$$\begin{array}{rl}\mathcal{D}\mathcal{Q}(t,t^{\prime })={\left(\frac{P}{2\pi }\right)}^4\mathcal{D}{Q}_1(t,t^{\prime })\mathcal{D}{\hat{Q}}_1(t,t^{\prime })& \mathcal{D}{Q}_2(t,t^{\prime })\mathcal{D}{\hat{Q}}_2(t,t^{\prime })\\ \times \mathcal{D}{Q}_3(t,t^{\prime })\mathcal{D}{\hat{Q}}_3(t,t^{\prime })& \mathcal{D}{Q}_4(t)\mathcal{D}{\hat{Q}}_4(t).\end{array}$$

Each of the $\mathcal{Q}$ variables represents a particular correlation function between two variables.

6.2.2 Overview of results via training and generalization error

They define the generalization error as a mean squared error (MSE)

$${\epsilon }_g(t)=⟨{\left(\frac{1}{\sqrt{P}}\sum _{i=1}^P{x}_i^0[{\hat{w}}_i(t)-w_i^{\ast }]-{ϵ}^0\right)}^2⟩$$

After taking the thermodynamic limit, the generalization error is expressed in terms of the equal time correlator $C(t,t)$ determined from the DMFT equations and $D$, the noise variance:

$${\epsilon }_g(t)=C(t,t)+D$$

They find two forms of this equation and propose a third, a scaling relation near the critical point. At $\lambda =0$ for $\nu <1$, the asymptotic error is:

Note the pole as criticality is approached. The quantity $\chi ={\int }_0^{\infty }R(\tau )d\tau$ is the susceptibility; it is well-defined only when the response function decays, i.e. only in equilibrium. Its appearance here signals that in this phase the response to perturbations of the initial conditions really does decay: asymptotically, the system carries no memory of its initialization.

The non-ergodic phase with persistent memory is captured by the asymptotic generalization error depending explicitly on the retention of the initial condition $C(0,0)$ and still showing sensitivity to $\nu \to 1$ analogous to the $\nu <1$ regime near criticality:

where fully, $C(0,0)=⟨{\hat{w}}^2(0)⟩+r$. The term $⟨{\hat{w}}^2(0)⟩$ is the variance of the initialization and $r$ is the mean-squared norm of the ground-truth parameters per dimension. Note that here, the response function does not decay, because we carry memory of the initial conditions. It saturates at $R(\tau \to \infty )=\frac{\nu -1}{\nu }$, which they call the ergodicity-breaking degree, quantifying the persistent memory of a perturbation applied to the initialization.

They also show similar trends with the training error, namely for $\nu <1$ the error is asymptotically finite, as the model is still attempting to approximate the solution, while for $\nu >1$ the training error vanishes as initialization-dependent memorization ensures the model always performs accurately on the training data. They write training error as

$$\underset{t\to \infty }{\lim }{\epsilon }_t(t)=(\gamma \chi +1{)}^{-2}{\epsilon }_g(t\to \infty ).$$

For $\nu <1$ it simplifies to $D(1-\nu )$, while again, for $\nu >1$, the system is no longer at equilibrium and the susceptibility $\chi ={\int }_0^{\infty }R(\tau )d\tau$ diverges, which drives the training error to zero.

To characterize the critical behavior of the model near $\nu \approx 1$, they calculate the critical exponents $(\sigma ,\varphi )$ and propose a scaling function. The first part is relatively straightforward, where from Eqs. (6.18) and (6.19), one finds ${\epsilon }_g(t\to \infty )$ diverges from both sides $\nu \to 1^{\pm }$. This suggests the scaling ${\epsilon }_g(t\to \infty )\sim |1-\nu {|}^{-\sigma }$ with an exponent $\sigma =1$ in DMFT. We remind the reader that these are asymptotic long-time steady-state results and both training and generalization error are time-dependent. Since the saturation time is $\nu$-dependent, a general feature of phase transitions is a dynamic scaling data collapse should be present near the transition (Goldenfeld 2018). They make this argument and propose a scaling ansatz of form:

where they introduce a new critical exponent $\varphi$ and write the universal scaling function $F(z)$, and $\gamma$ is still the probability of samples being included. They make two observations about the form of $F$ near and at the critical point. First, near the point, $\nu \to 1^{\pm }$ corresponds to $z\to 0$, they argue the singularity in front must be absorbed into $F$ and it takes the asymptotic form $F(z)\sim z^{\sigma /\varphi }$. Second, at $\nu =1$, since the generalization error grows in time instead of saturating, ${ϵ}_g(t)\sim (\gamma t{)}^{\sigma /\varphi }$. The exponent $\varphi$ therefore characterizes this growth in training time. Notably, the form of Eq. (6.21) implies a characteristic time scale, which they denote $\tau (\nu )=(\gamma |1-\nu {|}^{\varphi }{)}^{-1}$ to describe convergence in the scaling of the error. Since the error grows, we would expect this to diverge as $\nu \to 1$, which it does, and is an iconic signature of a phase transition, called critical slowing down. In MC calculations for lattice systems for example, this shows up in the autocorrelation time increasing, and requiring more time to get uncorrelated samples as we approach the critical point. A great deal of energy goes into designing methods to circumvent this slowing down when attempting to measure precise values of critical exponents using MC techniques, and motivates the choice of entirely different paradigms like RG flow or conformal bootstrap where applicable (Goldenfeld 2018).

Exercise 6.2 (Double descent in the lab). Simulate the model of this subsection: teacher $w^{\ast }\sim \mathcal{N}(0,r{I}_P)$, inputs $x^{\mu }\sim \mathcal{N}(0,I_P)$, labels $y^{\mu }=w^{\ast }\cdot x^{\mu }/\sqrt{P}+{ϵ}^{\mu }$ with noise variance $D$; student $\hat{w}$ trained by the update (6.11) with ridge $\lambda =0$ and full batches ($\gamma =1$), from $\hat{w}(0)\sim \mathcal{N}(0,{\sigma }_0^2{I}_P)$. Take $P=200$ and $\nu =P/n\in \{0.5,0.9,1.1,2\}$.

(a) Plot training and generalization error against time, and verify the asymptotics (6.18) and (6.19) — including the divergence of both branches as $\nu \to 1$: double descent, resolved in time.

(b) Probe ergodicity breaking: run two clones from initializations differing by a small random $\delta \hat{w}(0)$, and track the normalized separation $|\Delta \hat{w}(t){|}^2/|\Delta \hat{w}(0){|}^2$. Verify that it decays to zero for $\nu <1$ but saturates for $\nu >1$ — at the predicted plateau $(\nu -1)/\nu$, the paper's ergodicity-breaking degree, measured in twenty lines of code.

(c) Near $\nu =1$, extract the relaxation time of ${\epsilon }_g(t)$ and check for the critical slowing down $\tau \sim |1-\nu {|}^{-\varphi }$ of Eq. (6.21).

Solution

(a) With full batches and $\lambda =0$, gradient descent converges for $\eta <2/{\lambda }_{\max }(X^{T}X/P)$; keep $\eta$ a safe factor below. The late-time errors match (6.18) and (6.19), and plotting ${\epsilon }_g(t\to \infty )$ against $\nu$ traces the double-descent peak at $\nu =1$ from both sides.

(b) The clone separation obeys the linear dynamics $\Delta \hat{w}(t+1)=(I-\eta X^{T}X/P)\Delta \hat{w}(t)$: it decays iff the sampled covariance $X^{T}X$ has full rank, i.e. iff $n\ge P$ ($\nu \le 1$). For $\nu >1$ the covariance has a null space of dimension $P-n$, and the projection of $\Delta \hat{w}(0)$ onto it survives forever; for an isotropic perturbation the expected surviving fraction is $(P-n)/P=(\nu -1)/\nu$ — precisely the paper's ergodicity-breaking degree $R(\tau \to \infty )$.

(c) The relaxation time is set by the smallest nonzero eigenvalue of $X^{T}X/P$, which by Marchenko–Pastur closes like $(1-\sqrt{1/\nu }{)}^2\sim |1-\nu {|}^2/4$ near threshold, so one expects $\varphi \approx 2$; compare with the paper's measured exponent and with the collapse ansatz Eq. (6.21).

6.3 Dynamical mean-field theory (DMFT)

Dynamical mean-field theory, of which Section 6.2 was one instance, is a general strategy for reducing the number of effective degrees of freedom in an interacting dynamical system: one follows a single representative degree of freedom, moving in a fluctuating background that stands in for all the others — and one demands that the statistics of the background be determined, self-consistently, by the statistics of the representative trajectory itself. This is the same trichotomy as every other mean field in these notes (couple through averages; argue fluctuations are controlled; close self-consistently), upgraded from scalars and measures to two-time functions. An excellent intuition-building tutorial in a biophysics context is (Blumenthal 2025), and (Cugliandolo 2023) is a broad modern review. In quantum many-body theory the same idea has been enormously effective for otherwise intractable systems (Georges et al. 1996; Kotliar et al. 2006). Below we first clarify what self-consistency means in a dynamical setting, then survey the recent deep-learning literature; a natural technical entry point for the latter is the out-of-equilibrium DMFT of the perceptron (Agoritsas et al. 2018).

6.3.1 Self-consistency

The term self-consistency describes the demand that the predictions of a theory agree with the approximations used to obtain them. A simple example, taken from (Stapmanns et al. 2020), makes the issue vivid (and is worked in detail in Exercise 6.3). Consider the stochastic process $dx(t)=f(x(t))dt+dW(t)$, with $W$ a Wiener process (so that $\dot{W}$ is the white noise of Section 6.1), and suppose $f(x_0)=0$ with $f^{\prime }(x_0)<0$: a stable fixed point of the noiseless dynamics. One can expand around it, writing $x(t)=x_0+\delta x(t)$, to obtain the approximation

$$\frac{d}{dt}\delta x(t)=f^{\prime }(x_0)\delta x(t)+\xi (t)+\mathcal{O}({\delta }^{2}x(t)),$$

where $\xi$ is also centered gaussian white noise. The mean and variance of these fluctuations can be computed straightforwardly via Fourier transform, and in the temporal basis the variance is $⟨\delta x^2⟩=\frac{D}{-2f^{\prime }(x_0)}$, however using these results with the Taylor expansion of $f(x)$ to second order gives a correction to the mean

$$⟨x⟩\approx x_0+\frac{1}{2}\frac{f^{\prime \prime }(x_0)}{f^{\prime }(x_0)}\frac{D}{2f^{\prime }(x_0)}$$

which is inconsistent with the value $x_0$ assumed for the mean when constructing the approximation: the noise, filtered through the curvature of $f$, shifts the very point we expanded around.

Exercise 6.3 (Self-consistency in the smallest possible example). (a) Derive the stationary variance $⟨\delta x^2⟩=D/(-2f^{\prime }(x_0))$ of the linearized dynamics (compare Exercise 6.1(b)).

(b) Carrying the Taylor expansion of $f$ to second order, derive the quoted correction to the mean.

(c) Propose the self-consistent repair: expand instead around a point $x_{\ast }$, chosen so that the effective drift vanishes, $f(x_{\ast })+\frac{1}{2}{f}^{\prime \prime }(x_{\ast }){⟨\delta x^2⟩}_{\ast }=0$, with ${⟨\delta x^2⟩}_{\ast }=D/(-2f^{\prime }(x_{\ast }))$ determined simultaneously. Note the structure: two coupled equations, each an input to the other — a miniature of the Curie–Weiss condition of Section 5.1, and of the DMFT equations to come.

Solution

(a) The linearized equation $\dot{\delta x}=f^{\prime }(x_0)\delta x+\xi$ is an Ornstein–Uhlenbeck process with $k=-f^{\prime }(x_0)>0$; by Exercise 6.1(b) its stationary variance is $D/(2k)=D/(-2f^{\prime }(x_0))$.

(b) Taking the stationary average of $\dot{x}=f(x)+\xi$ with $f$ expanded to second order about $x_0$: $0=f(x_0)+f^{\prime }(x_0)⟨\delta x⟩+\frac{1}{2}{f}^{\prime \prime }(x_0)⟨\delta x^2⟩$, and using $f(x_0)=0$, $⟨x⟩=x_0-\frac{f^{\prime \prime }(x_0)}{2f^{\prime }(x_0)}⟨\delta x^2⟩=x_0+\frac{1}{2}\frac{f^{\prime \prime }(x_0)}{f^{\prime }(x_0)}\cdot \frac{D}{2f^{\prime }(x_0)}$, the quoted shift — nonzero, contradicting the assumption $⟨x⟩=x_0$.

(c) Demand instead that the expansion point $x_{\ast }$ satisfy the effective stationarity condition $f(x_{\ast })+\frac{1}{2}{f}^{\prime \prime }(x_{\ast })⟨\delta x^2{⟩}_{\ast }=0$ together with $⟨\delta x^2{⟩}_{\ast }=D/(-2f^{\prime }(x_{\ast }))$. These are two equations determining $(x_{\ast },⟨\delta x^2{⟩}_{\ast })$ jointly: the fluctuations shape the effective potential, and the effective potential shapes the fluctuations. Structurally this is $m=\tanh (\beta (m+h))$ again — and DMFT is the same fixed-point demand imposed on entire two-time functions $C(t,t^{\prime })$, $R(t,t^{\prime })$ instead of scalars.

A framework for resolving this, and also attempting to enforce that the small parameter we expand about is actually the expansion parameter involves using n-particle irreducible effective actions (nPI). A full discussion of this is well beyond the scope of this document, but is covered very well in a pedagogical introduction (Berges 2004), where the main idea is enforcing self-consistency via increasingly multi-local sources that generate n-point correlation functions. The first section emphasizes that performing standard perturbation theory with an anharmonic oscillator yields so-called secular terms which are powers of time $t$ that appear next to the actual expansion parameter $ϵ$, and using the nPI framework effectively resums these terms properly at each order in $ϵ$. The 1PI approach is directly related to a local approximation with mean field theory, while in the 2PI approach, the object determined self consistently is the 2-point function, which in the local approximation is related to the local propagator and is the basis of dynamical mean field theory.

6.3.2 Recent applications in deep learning

Fortunately, one rarely needs the general nPI machinery in practice. The working route — structurally the same trick by which large-$N$ models in physics (e.g. the $G$–$\Sigma$ formulation of the SYK model) are solved — is to insert bilocal collective fields for the correlation and response functions via delta functions, exactly as the $Q$'s of Section 6.2, and then to evaluate their path integral by saddle point in a suitable $N\to \infty$ limit. This is how the deep-learning DMFT of (Bordelon & Pehlevan 2022; Bordelon et al. 2024) is constructed.

A brief, opinionated tour of the literature, oldest first.

Spin-glass origins. The mean-field dynamics of disordered systems is where all of this machinery was forged. Amit, Gutfreund, and Sompolinsky (Amit et al. 1985) brought spin-glass methods into neural networks (in the guise of Hopfield-style associative memories), and Sompolinsky, Crisanti, and Sommers (Sompolinsky et al. 1988) derived the DMFT of random recurrent networks, discovering a transition to chaos. The structural point to retain: an ordered system like the Ising magnet has one global coupling $J{\sum }_{⟨ij⟩}{\sigma }_i{\sigma }_j$ and is already hard; a disordered system has random couplings ${\sum }_{ij}{J}_{ij}{\sigma }_i{\sigma }_j$, and it is precisely the all-to-all randomness that makes mean-field methods exact — the same structural feature that data-dependent couplings give learning problems, starting with the perceptron of Section 2.

The Bordelon–Pehlevan program. For deep networks trained by gradient descent at fixed dataset, the DMFT of (Bordelon & Pehlevan 2022) takes the infinite-width limit and finds that the order parameters are precisely the feature kernels $\Phi (t,s)$ and gradient kernels $G(t,s)$ — two-time generalizations of the NNGP kernel and NTK of Sections 3–4. The lazy regime of Section 4 is recovered when these kernels freeze; rich, feature-learning dynamics is their genuine evolution, computed self-consistently. (Fair warning: the notation is heavy — response fields and conjugate order parameters abound — but the structure is exactly MSRJD plus a saddle point.) The program now includes finite-width kernel fluctuations (Bordelon & Pehlevan 2023), hyperparameter transfer across depth (Bordelon et al. 2023), and the dynamical theory of neural scaling laws (Bordelon et al. 2024) that we meet again in Section 7.9.

The Mignacco–Urbani line. A complementary series of works treats the data as quenched disorder at proportional scalings of dimension and sample size, in the spirit of Section 2: DMFT for Gaussian-mixture classification (Mignacco et al. 2021), and a DMFT characterization of the effective noise of SGD at late times (Mignacco & Urbani 2022) — the dynamical refinement of the batch-noise discussion of Section 2.1.

Rigor. What began as a physics heuristic is increasingly a theorem: rigorous derivations of DMFT for first-order methods now exist (Celentano et al. 2021; Gerbelot et al. 2024), with a growing line of extensions — e.g. to adaptive Langevin diffusions, with propagation-of-chaos guarantees and convergence of the linear response (Fan et al. 2025). A recent tour de force in this direction (Montanari & Urbani 2025) solves the dynamics of a two-layer network at scale — a long calculation, but a well-organized paper — finding: the emergence of a slow timescale tied to the growth of the network's complexity (in the Gaussian/Rademacher sense); an inductive bias toward low complexity when the initialization has low complexity; a dynamical decoupling of the feature-learning and overfitting regimes; and a non-monotone test error, with a "feature unlearning" regime at very late times.

The connective tissue of this whole section, worth stating once more: the MSRJD path integral is to training dynamics what the Gibbs measure was to Bayesian learning — the object on which every physics technique (saddle points, disorder averages, collective fields, perturbation theory) can be brought to bear.

7. Scaling laws

In this final section we return to the theme of Section 1 and discuss the neural scaling laws. Over many orders of magnitude, the test loss of large models falls as a clean power law in each resource spent on it: parameters, data, and compute. These laws were found by fitting curves to measurements, and they have proved to be a remarkably simple, robust, and consequential empirical observation, one that any successful model of deep learning should be able to capture. Many models do capture their form, and several go further and make quantitative predictions for the exponents, based on features of the data distribution.

The organizing question of the section is: where do the exponents come from? We first assemble the empirical picture (Sections 7.1–7.2): the three laws, what each one holds fixed, and the joint laws that govern compute-optimal training, where the Kaplan–Chinchilla discrepancy lived and was resolved. We then turn to theory. Section 7.3 states the mechanism common to every proposal — power laws in the data produce power laws in the loss — and Section 7.4 surveys the main proposals in one table. The rest of the section works out four rows of that table, in order of increasing machinery: the frequency statistics of skills (Section 7.5, run as a derive-it-yourself exercise, with its payoff for "emergent capabilities" in Section 7.6); the geometry of data and the spectra of kernels (Section 7.7); and solvable large-$N$ field theories, static and then dynamical (Sections 7.8–7.9). We close with the newest measurements, for which there is no theory yet: post-training and test-time scaling (Section 7.10).

Notation. We keep the conventions of these notes: $P$ is the parameter count and $n$ the dataset size. Be warned that the scaling-law literature writes $N$ for our $P$ and $D$ for our $n$ (their $N$ is not our width!), so the famous exponents ${\alpha }_N,{\alpha }_D$ are our ${\alpha }_P,{\alpha }_n$. For reading the sources:

these notes	scaling-law literature (Kaplan et al. 2020; Hoffmann et al. 2022; Michaud et al. 2023; Maloney et al. 2022)
$P$ (parameters)	$N$
$n$ (dataset size)	$D$
${\alpha }_P$, ${\alpha }_n$ (exponents)	${\alpha }_N$, ${\alpha }_D$
$m$ (quanta learned, Section 7.5)	$n$

Note also that in this section $\alpha$ (no subscript) denotes the Zipf exponent of the quantization model, not the load of Section 2.

7.1 The empirical laws

Kaplan et al. (Kaplan et al. 2020) established, for autoregressive language models, that when a model is not bottlenecked by the other resources its test loss obeys

where $P$ is the number of (non-embedding) parameters, $n$ the dataset size in tokens, and $C$ the training compute in FLOPs. The measured exponents were ${\alpha }_P\approx 0.076$, ${\alpha }_n\approx 0.095$, and ${\alpha }_C\approx 0.050$, with the trends holding over up to seven orders of magnitude. Three points of bookkeeping are usually left implicit. They are worth making explicit before asking where the laws come from.

Why FLOPs, and why $C\approx 6Pn$. Compute is measured in floating-point operations because FLOPs are the only measure of training effort that does not depend on how the job is run. Wall-clock time depends on the hardware and the parallelism. The number of training steps depends on the batch size: the same $n$ tokens can be one giant batch or a million small ones, and below the "critical batch size" (which (Kaplan et al. 2020) measure separately) the loss depends only on the number of tokens processed. FLOPs count the arithmetic itself, which is also what energy and money are proportional to. The standard estimate is

Per token, the forward pass costs about $2P$ FLOPs: essentially every parameter sits in some matrix multiplication and is used once, for one multiply and one add. The backward pass costs about twice the forward pass, since each layer needs the gradient with respect to its activations and the gradient with respect to its weights, two matrix multiplications of the same size as the forward one. That gives $6P$ FLOPs per token, times $n$ tokens. The $\approx$ absorbs attention (whose per-token cost grows with context length rather than parameter count), embeddings, and optimizer overhead.

The single-epoch assumption. The estimate (7.2) assumes that every token processed is a fresh token, so that "tokens processed" and "dataset size" are the same number. This was true of the runs on which the laws were measured, which used well under one epoch of the corpus. Conceptually the two are different resources. The statistical resource is the number of distinct tokens $n$; the computational resource is the number of tokens processed, $T=En$ over $E$ epochs, so that in general $C\approx 6PT$. Fixing $n$ and growing $C$ by repetition buys optimization, not statistics: the loss converges to the $n$-limited floor and then overfits. In the language of Section 2.1, more epochs means larger $\beta$ at a fixed quenched sample, while fresh data changes the quenched free energy itself. Empirically the one-epoch idealization is forgiving: for up to about four epochs, repeated tokens are nearly as valuable as fresh ones, after which their value decays quickly (Muennighoff et al. 2023). This matters increasingly, now that frontier training is data-constrained.

The compute law is not independent. Since $C\approx 6Pn$ depends only on the other two resources, how can compute ever be the unbottlenecked one? The answer is that $L(C)$ is measured by a different protocol. It is the lower envelope

traced out by training many model sizes and keeping the best loss at each value of compute. On this curve compute is the binding constraint by construction, and "not bottlenecked" means "optimally allocated." This makes ${\alpha }_C$ a derived exponent. Minimizing an additive joint law $L=A{P}^{-{\alpha }_P}+B{n}^{-{\alpha }_n}$ at fixed $Pn$ balances the two terms, and gives

The harmonic sum makes ${\alpha }_C$ smaller than either single-resource exponent, since a factor of $C$ must be split between growing $P$ and growing $n$. Kaplan's values give ${\alpha }_C\approx 0.042$, against the measured $0.050$: roughly consistent, with the gap traceable to the form of the fit and to early-stopping details.

Universal form; data-dependent exponents. The same power-law form holds far beyond language. Henighan et al. (Henighan et al. 2020) trained autoregressive transformers on many kinds of data and found power laws everywhere, but with exponents that depend on the data:

modality	${\alpha }_P$	${\alpha }_C$
language	$0.070$	$0.048$
images $32\times 32$	$0.13$	$0.10$
images $8\times 8$	$0.24$	$0.19$
video	$0.24$	$0.14$
math (procedural)	$0.16$	$0.17$

("Math" here means procedurally generated problems, in the style of the DeepMind mathematics dataset. They are emitted by a known program, so the supply is effectively unlimited. Remember this case: it is the one modality whose data distribution is known in closed form.) The exponents vary by a factor of $3$–$5$ across modalities. Architecture, by contrast, barely matters: within reasonable architectures, changing the shape (depth vs. width at fixed size) or even the family moves the constants and the plateaus, not the slopes (Kaplan et al. 2020). This asymmetry is the central clue of the section: whatever sets the exponent lives in the data. Section 7.7 makes it quantitative.

7.2 Joint laws, bottlenecks, and the Chinchilla revision

Each single-resource law assumes the other resources are unlimited. Joint fits describe the crossover to being bottlenecked, with the loss governed by whichever resource is scarcest. Two parametric forms dominate the literature. Kaplan et al. used

while the Chinchilla analysis of Hoffmann et al. (Hoffmann et al. 2022) refit with the additive ansatz

The additive form is easy to read. $E\approx 1.69$ is the irreducible entropy of text, the noise floor that no amount of scale removes; the two deficits decay independently; and the bottleneck is simply the larger deficit. Both forms reduce to the single-resource laws (7.1) in the appropriate corners.

The difference between the fits mattered a great deal in practice, because of what they say about the compute-optimal allocation in (7.3). Kaplan et al.'s analysis gave $P\propto C^{0.73}$ and $n\propto C^{0.27}$: with more compute, grow the model. For two years the field trained ever-larger models on comparatively little data. Chinchilla's fit gives $P\propto C^{0.5}$ and $n\propto C^{0.5}$: grow both equally, or as a rule of thumb about $20$ tokens per parameter. The prediction was tested directly. At the compute budget of DeepMind's 280B-parameter Gopher, the new rule said to train a model four times smaller on four times more data, and the resulting 70B-parameter Chinchilla beat Gopher across the board. (Both models, and the law, are named in DeepMind's animal series.) Chinchilla's allocation is now the accepted baseline. The discrepancy was later dissected by Porian et al. (Porian et al. 2024): Kaplan's fit was skewed by three small-scale artifacts — how last-layer and embedding compute were counted, over-long warmup, and optimizer settings that were not retuned with scale — and with these corrected, the two experiments agree. One coda: frontier models now deliberately over-train relative to Chinchilla, on many more than $20$ tokens per parameter, because a smaller model at equal loss is cheaper at inference time, a cost the training-compute-optimal analysis does not see.

With the empirical picture assembled, here is what a theory of (7.1) owes us: (i) why power laws at all, over so many decades; (ii) the values of the exponents (why so small? why do they depend on the modality but not on the architecture?); (iii) relations between exponents (is ${\alpha }_n$ determined by ${\alpha }_P$?); and (iv) when the laws break. The models surveyed below each answer a subset.

7.3 Where do the laws come from?

The basic answer, common to every proposal surveyed below, is that the loss curve reads off the coarse distribution of structure in the data. Power laws go in with the data and come out in the loss.

The prototype of "power laws in" is Zipf's law: rank the words of a large corpus by frequency, and the frequency of the word of rank $k$ falls off as $f_k\propto 1/k$, over about four decades of rank (Figure 7). It is a scale-free tail of ever rarer structure, and it is not special to words. Sequences of $n$ consecutive words ($n$-grams, the unit of pre-neural language modeling) are Zipf-like at every $n$, with slopes that flatten as $n$ grows. Facts, meaning discrete pieces of world knowledge attested in text, inherit the heavy-tailed popularity of the entities they mention, and a model's ability to answer a factual question tracks the number of training documents that support it (Kandpal et al. 2023). The covariance and kernel spectra of text and image data are measured power laws (Sections 7.7–7.8). A learner working through such data never runs out of structure to absorb. It moves ever further down the tail, at ever slower returns. That is, qualitatively, why the loss is a power law rather than an exponential (there is structure at all scales) and why it does not simply plateau (the tail does not end).

The exponents of these tails differ, and the differences matter, because in every model below the loss exponent is inherited from the data exponent. Pure Zipf is in fact a marginal case. Write the tail of skill frequencies as $p_k\propto k^{-(\alpha +1)}$. The quantization model of Section 7.5 gives a loss exponent equal to $\alpha$, and exact Zipf ($f_k\propto 1/k$, that is, $\alpha =0$) is the boundary at which the tail sum decays only logarithmically and power-law scaling degenerates. Read this way, the measured ${\alpha }_P\approx 0.076$ for language says that its effective skill distribution is $p_k\sim k^{-1.08}$, barely steeper than pure Zipf. Language scaling is slow because its structure sits just above the marginal $1/k$ tail. Image data has steeper tails and scales faster; compare the modality table of Section 7.1.

Where does the theory stand? The single-resource laws have been derived from this mechanism by many models of quite different flavor, and a few models derive multi-resource laws; the survey below keeps score. Three things are missing. First, no theory computes the data's exponents from first principles. Every model converts measured data statistics into loss curves, and the statistics themselves are an empirical input. This is true even for procedurally generated math, where the generating program is known exactly but the computation has not been carried out. Second, there are no joint laws with genuine feature learning. The best dynamical derivations live in lazy, random-feature settings (Section 7.9), while in solvable cases feature learning is known to improve exponents for structured targets, so the corrections need not be small. Third, the bridge from the smooth loss to capabilities is missing. Benchmark performance can jump within a decade of scale while the loss glides smoothly (Wei et al. 2022); this has been contested as an artifact of the metric (Schaeffer et al. 2023), and the quantization model of Section 7.6 gives it a sharper reading. A fourth gap is newer: the post-training and test-time scaling laws of Section 7.10 have no theory at all.

7.4 A survey of theoretical proposals

The table below organizes the most prominent derivations of neural scaling laws by four questions: what models the data (equivalently, what a "feature" is and how features are distributed); what models the network, if anything; whether actual exponents are predicted, and in terms of what; and whether multi-resource laws are obtained.

proposal	data model / "features"	NN framework	exponents?	multi-resource?
quantization; Michaud et al. (Michaud et al. 2023)	discrete skill "quanta," Zipf-used, learned in order	none (a counting argument)	in terms of the Zipf exponent $\alpha$; plus the relation $\frac{1}{{\alpha }_n}=\frac{1}{{\alpha }_P}+1$	separate $P$, $n$, $S$ laws
data manifold; Sharma–Kaplan (Sharma & Kaplan 2020)	smooth target on a $d$-dimensional manifold	piecewise-linear interpolation (function class only; no training)	yes: ${\alpha }_P\approx 4/d$	no ($P$ only)
percolation; Brill (Brill 2024)	latent subtask clusters at criticality, with power-law cluster sizes	none: a generic nonparametric learner $+$ regression estimates	from percolation critical exponents	two regimes; not joint
kernel eigenlearning; Bahri et al. (Bahri et al. 2024), Bordelon–Canatar–Pehlevan (Bordelon et al. 2020; Canatar et al. 2021; Simon et al. 2023)	power-law kernel spectra $+$ target coefficients	frozen kernel: infinite-width NTK, or GP/Bayesian	in terms of the spectral exponents	partially: resolution- vs. variance-limited regimes
solvable field theory; Maloney–Roberts–Sully (Maloney et al. 2022)	latent power-law covariance, extended by random features	linear model on fixed random features (frozen kernel); large-$N$ RMT, statics	in terms of the data's spectral exponent	yes: joint $L(P,n)$, with duality $P\leftrightarrow n$
DMFT; Bordelon–Atanasov–Pehlevan (Bordelon et al. 2024)	power-law (source/capacity) spectra	DMFT of gradient-flow training on fixed random features (frozen kernel; dynamics)	yes, including the time exponent	fully joint $L(t,P,n)$ $+$ compute frontier

Five remarks for reading the table. First, everyone agrees on the origin. In every row the loss exponent is inherited from a power-law tail in the data, whether of skill frequencies, cluster sizes, manifold volume, or covariance spectra. The rows differ in what the tail is a tail of, and in how much network machinery is retained. Second, and this deserves emphasis: every row that models the network at all does so with a frozen kernel. The eigenlearning row works with the infinite-width NTK or GP kernel of Sections 4.3 and 4.6; the two field-theory rows train only the readout of a fixed set of random features, which is a frozen kernel by construction (Exercise B.6(b)). The manifold row models only the network's function class, as a count of linear pieces, and never trains it; the quantization and percolation rows contain no network. So none of the current derivations involves feature learning, the very mechanism that Sections 5–6 argued is what makes finite networks differ from kernels. Third, equilibrium routes (Bayesian GP learning curves and their finite-width refinements) and dynamical routes (gradient flow, DMFT) give matching exponents wherever they overlap. This is the "two limits of one system" correspondence of Sections 2.1 and 6 at work again. Fourth, only the two field-theory rows currently predict joint laws, and only the dynamical one prices training time, deriving a Chinchilla-like compute-optimal frontier rather than assuming it. Fifth, no row computes the data's own exponent. The closest attempt is the percolation proposal, which derives power-law cluster-size statistics from a critical model of the data distribution; its two criticality regimes (power-law-distributed discrete subtasks, and a dominant data manifold) map onto the quantization and manifold rows respectively. We will not develop the percolation model further. The remaining subsections work through the other rows, in order of machinery.

Do the predicted exponents agree with experiment? Where the comparison has been made, the answer is encouraging but not yet sharp. The quantization model's authors extracted "quanta" from a small language model by clustering gradients, measured how often each is used in the training distribution, and found a power law whose exponent roughly matches the model's scaling exponent, as the model requires (Michaud et al. 2023); on the other hand, the model's parameter-free relation between ${\alpha }_P$ and ${\alpha }_n$ is violated by the measured ordering of Kaplan's exponents (Exercise 7.1(e)). The manifold prediction ${\alpha }_P\approx 4/d$ was confirmed in teacher–student experiments, where $d$ and ${\alpha }_P$ can be measured independently, and tested more loosely on image classifiers and GPT-type language models (Sharma & Kaplan 2020); for language, taking ${\alpha }_P\approx 0.076$ at face value gives $d\approx 50$ (Exercise 7.2). The kernel predictions have been tested on standard architectures and datasets, with exponents computed from measured kernel spectra, and the predicted duality between the $P$- and $n$-exponents holds up (Bahri et al. 2024). The two field-theory rows take the data's spectral exponent as input. Their qualitative predictions — exponents independent of architecture, ${\alpha }_P\approx {\alpha }_n$, and equiparameterization as the compute-optimal rule — match the Chinchilla-era measurements, but a quantitative test needs the covariance spectrum of the training data, which is rarely reported for the large runs.

The honest summary is that these theories convert an exponent measured in the data into an exponent measured in the loss, and the conversion has held up wherever both have been measured.

7.5 The quantization model: exponents from the statistics of skills

The quantization model of Michaud, Liu, Girit, and Tegmark (Michaud et al. 2023) is a minimal model of scaling. It has no architecture, no optimizer, and essentially no mathematics beyond a tail sum, and it produces all three laws of (7.1), plus a sharp point of view on "emergent capabilities." The name is deliberately provocative: the proposal is that what a network learns comes in discrete units, dubbed quanta.

⁴

The assumptions, in the form we'll use:

• QH1 (discreteness). The computations and knowledge relevant to the prediction task decompose into enumerable quanta: basic skills or facts (e.g. "carry the one," "Paris follows `the capital of France is'," ...). Each quantum is either fully learned or not learned at all, with no partial credit.
• QH2 (ordered acquisition). Networks learn quanta roughly in order of decreasing usefulness, so a given model has learned the first $m$ quanta of a fixed sequence.
• QH3 (Zipfian use). Quantum $k$ is used on a fraction

of samples ($\alpha >0$ a constant), and on those samples, knowing it reduces the loss by a constant $\Delta$ (independent of $k$).

Consequently, a model that has learned the first $m$ quanta has expected loss

That is the entire model. Everything else is asking how $m$ is limited by each resource.

Exercise 7.1 (★) (Three scaling laws from three assumptions). (a) Estimate the tail sum in (7.8) to show $L_m-L_{\infty }\propto m^{-\alpha }$.

(b) Parameter-limited (data and training unlimited). Suppose each quantum costs a constant capacity of $c$ parameters to learn. Derive ${\alpha }_P$ in terms of $\alpha$.

(c) Data-limited (capacity and training unlimited). Suppose a quantum is learnable only if it is used by at least $\tau$ training samples, for constant $\tau$. Given $n$ i.i.d. samples, which quanta are learnable? Derive ${\alpha }_n$.

(d) Training-limited, single epoch. Suppose instead that learning quantum $k$ requires $\tau$ gradient updates on batches containing it. After $S$ steps of online training (fresh batches), derive ${\alpha }_S$.

(e) Falsifiability. Parts (b)–(c) predict a parameter-free relation between ${\alpha }_P$ and ${\alpha }_n$: write it down, and note which of the two exponents must be larger. Compare with the measured values quoted after (7.1). Something is wrong — what? List the assumptions you used and discuss which are most suspect. (This is a discussion question with no settled answer; see the solution for candidates.)

Solution

(a) ${\sum }_{k>m}{k}^{-(\alpha +1)}\approx {\int }_m^{\infty }{k}^{-(\alpha +1)}dk=m^{-\alpha }/\alpha$, so $L_m-L_{\infty }\propto m^{-\alpha }$. (The same replace-sum-by-integral move as the entropy estimates of Section 2.4.)

(b) $m=P/c$ quanta fit, so $L(P)-L_{\infty }\propto (P/c{)}^{-\alpha }$: $\boxed{{\alpha }_P=\alpha }$.

(c) Quantum $k$ is seen $\approx n{p}_k$ times, so it is learnable iff $n{p}_k\gtrsim \tau$, i.e. $k\lesssim m(n)\propto (n/\tau {)}^{1/(\alpha +1)}$ by (7.7). Then $L(n)-L_{\infty }\propto m(n{)}^{-\alpha }$: $\boxed{{\alpha }_n=\frac{\alpha }{\alpha +1}}$.

(d) Identical threshold logic with $S$ in place of $n$: $\boxed{{\alpha }_S=\frac{\alpha }{\alpha +1}}$. Note the model predicts ${\alpha }_n={\alpha }_S$ — data and single-epoch training time are interchangeable bottlenecks, a foreshadowing of the joint treatment in Section 7.9.

(e) Eliminating $\alpha$: $\frac{1}{{\alpha }_n}=\frac{1}{{\alpha }_P}+1$, hence ${\alpha }_n<{\alpha }_P$ always. But the measured LLM values have ${\alpha }_n\approx 0.095>{\alpha }_P\approx 0.076$: the predicted ordering is violated, not just the numbers. Candidate culprits, roughly in order of suspicion: (i) the measured ${\alpha }_n$ comes from multi-epoch training with early stopping, not the clean "capacity-unlimited, data-limited" protocol assumed in (c); (ii) the threshold $\tau$ and per-quantum cost $c$ need not be $k$-independent (rare quanta may be intrinsically harder or easier); (iii) $\Delta$ constant across quanta is crude; (iv) real capabilities may not decompose into independent quanta at all — they share structure, so learning one changes the cost of others (see the lab of Section 7.6, where exactly this happens). The instructive point is not which repair is right but that the model is sharp enough to be wrong — compare the falsifiability discussion of Section 1.

7.6 Emergence: smooth laws from staggered cliffs

The quantization model's deepest payoff is conceptual. Under QH1, the learning of any single quantum is a step function: a capability is absent, then abruptly present. Yet the aggregate loss (7.8) is a smooth power law. There is no contradiction. A smooth law is a sum of many staggered cliffs, each weighted by $p_k$ and individually invisible in the aggregate, since quantum $k$ contributes only the small fraction $p_k$ of the loss. Michaud et al. sharpen this with a genetics metaphor. A test sample is monogenic if its loss is governed by a single quantum, so that its loss curve during training is a plateau followed by a cliff; it is polygenic if many quanta contribute, in which case its loss falls smoothly. Both kinds of per-token curves are observed in real LLMs. From this perspective, "emergent capabilities" (Wei et al. 2022) are neither magic nor mirage (Schaeffer et al. 2023): any benchmark dominated by a few quanta must show abrupt arrival, while any sufficiently aggregated metric must look smooth. Emergence is a property of what you measure, not (necessarily) of a phase transition inside the model.

Lab: multitask sparse parity. The entire mechanism fits in a laptop-scale experiment, shown in Figure 8; students are encouraged to reproduce and modify it (all details are in the caption). A network is trained online on $T$ separate parity subtasks with Zipfian frequencies. The input is a one-hot task indicator concatenated with random bits, and the label is the parity of a small task-specific subset of the bits. Parity is a needle-in-a-haystack skill with no partial credit, i.e. an honest hand-built quantum. The result: each subtask's test loss is a staggered cliff (frequent tasks fall first), while the frequency-weighted aggregate loss decays smoothly. Note also the honest failure visible in the right panel. The measured aggregate decays more steeply than the idealized quanta slope $-\alpha /(\alpha +1)$ of Exercise 7.1(d), because the cliffs bunch up rather than spreading Zipf-like; the rare tasks are learned sooner than the threshold rule $S{p}_k\gtrsim \tau$ predicts. The subtasks share a hidden layer and an adaptive optimizer, so they are not learned independently, and assumption (iv) in the solution to Exercise 7.1(e) fails.

7.7 Exponents from geometry and from spectra

The quantization model takes the Zipf exponent $\alpha$ as input. The next two arguments produce exponents as output, from properties of the data.

7.7.1 Geometry: the data-manifold argument

Sharma and Kaplan (Sharma & Kaplan 2020) propose the simplest continuous mechanism. A network with ReLU activations computes a piecewise-linear function, so regression on a data manifold of intrinsic dimension $d$ is a problem of piecewise-linear interpolation. A network with $P$ parameters can afford $\sim P$ linear pieces; tiling a $d$-dimensional manifold with $P$ pieces gives spacing $s\sim P^{-1/d}$; a linear patch approximates a smooth target to accuracy $\sim s^2$ (Taylor); and MSE is the square of the error. Hence

Exponents are small because data manifolds are high-dimensional. The formula is testable, since $d$ can be measured with intrinsic-dimension estimators, which is exactly what (Sharma & Kaplan 2020) do.

Exercise 7.2 (Exponents from the data manifold). (a) Fill in the derivation of (7.9), stating precisely where smoothness of the target is used.

(b) How does the answer change for mean-absolute-error loss?

(c) Taking the measured ${\alpha }_P\approx 0.076$ for language models at face value, estimate the effective dimension of the "manifold of text." Discuss whether the input dimension, the embedding dimension, or something else is being measured.

(d) What does this mechanism predict for the data exponent ${\alpha }_n$, if $n$ samples are what limits the resolution of the tiling?

Solution

(a) As in the text; smoothness (bounded second derivatives) enters in the per-patch error $\sim s^2$.

(b) Mean-absolute error is the error itself, not its square: $L\sim s^2\sim P^{-2/d}$.

(c) $d\approx 4/0.076\approx 50$: an effective, intrinsic dimension of the distribution of text as seen by the model — vastly smaller than the nominal input dimension (vocabulary $\times$ context), which is the whole point.

(d) If data limits the tiling, $n$ samples resolve spacing $s\sim n^{-1/d}$ and ${\alpha }_n\approx 4/d\approx {\alpha }_P$: this mechanism predicts equal exponents — closer to what is observed than the quanta prediction, and the same conclusion the field theory of Section 7.8 reaches from spectra.

7.7.2 Spectra: scaling laws from kernel eigenvalues

The second mechanism uses machinery we already own: the Bayesian kernel regression of Section 4.6, whose mode-filtering formula said that with $n$ samples (and inverse temperature $\beta$), the modes of the GP kernel that get learned are those with ${\lambda }_k\gtrsim (n\beta {)}^{-1}$. Suppose the kernel spectrum and the target's decomposition are power laws,

$${\lambda }_k\sim k^{-a}(a>1),y_k^2\sim k^{-b}(b>1),$$

in a shared eigenbasis $\{{\phi }_k\}$. The unlearned modes dominate the test error:

giving $L(n)\sim n^{-(b-1)/a}$: the scaling exponent is a ratio of spectral decay rates. Everything about the data and the architecture enters through two measurable power laws. This is the starting point of the "eigenlearning" analyses (Bordelon et al. 2020; Canatar et al. 2021; Simon et al. 2023), which sharpen (7.11) into precise learning curves (including the overfitting factor whose divergence produces the double-descent peak of Section 6). Bahri et al. (Bahri et al. 2024) organize the possibilities into a clean $2\times 2$ taxonomy: resolution-limited regimes (exponents from spectra or geometry, as here and in (7.9)) versus variance-limited regimes (universal exponent $1$), in each of $P$ and $n$.

Exercise 7.3 (Spectral scaling laws, and quanta as eigenmodes). (a) Derive (7.11) from the mode-filtering formula of Section 4.6, and check that the crossover modes $k\sim k_{\ast }$ do not change the exponent.

(b) Show that the quantization model's data-scaling law is the special case $b=a$ of (7.11) under the dictionary: quanta $=$ eigenmodes, Zipf law $=$ spectrum, $a=\alpha +1$. (Hint: in the quanta model, $p_k$ plays both roles, setting the learning threshold and the contribution $\Delta p_k$ to the loss, so the two power laws coincide.)

(c) In the quanta model, what would it mean for $b\ne a$? Invent a modification of QH3 that realizes it.

Solution

(a) The learned modes contribute $((n\beta {)}^{-1}/({\lambda }_k+(n\beta {)}^{-1}){)}^2{y}_k^2\approx 0$ for $k\ll k_{\ast }$; the unlearned ones contribute $y_k^2$; the crossover window contributes $O(k_{\ast }^{1-b})$ as well, so the exponent is unchanged. Hence $L\sim n^{-(b-1)/a}$.

(b) With $b=a=\alpha +1$: $L\sim n^{-\alpha /(\alpha +1)}$, i.e. the exponent ${\alpha }_n$ of Exercise 7.1(c). The threshold ${\lambda }_k\gtrsim (n\beta {)}^{-1}$ is the threshold $n{p}_k\gtrsim \tau$; the tail sum of $y_k^2$ is the tail sum of $\Delta p_k$. Kernel regression is a quantization model whose quanta are eigenmodes.

(c) $b\ne a$ decouples how often a skill is tested (its loss weight) from how easy it is to learn (its threshold) — e.g. let quantum $k$ require ${\tau }_k\propto k^{\theta }$ occurrences. Any monotone relation between frequency and difficulty generates a two-exponent family, and (7.11) is its universal form.

7.8 A solvable large-N field theory

Maloney, Roberts, and Sully (Maloney et al. 2022) supply what the arguments above lack: a microscopic, exactly solvable model in which the joint law $L(P,n)$ is computed, not modeled. The ingredients are: a generative data model in which observed inputs are (nonlinear functions of) random projections of a latent space, engineered so that the data covariance has a power-law spectrum ${\lambda }_k\sim k^{-(1+\zeta )}$, as measured in real image and text data (the spectral exponent is the paper's $\gamma$; we reserve $\gamma$ for the scaling dial of Section 5.4); a random-feature linear student, so we are in the lazy limit of Section 4, whose $P$ random features are its trainable parameters, trained on $n$ samples; and the joint thermodynamic limit $P,n\to \infty$ at fixed ratio. The computation proceeds at large $P$ and $n$, where expectations over data and features become Gaussian integrals, the test loss becomes a product of resolvents (Green's functions), and the $1/P$ expansion organizes into planar ('t Hooft) diagrams that can be resummed exactly.

⁵

Frozen, but random. A natural first reaction is that this model should be trivial. The student is a linear model on fixed random features. Its kernel never moves, so we are squarely in the lazy regime of Section 4, and in the strict infinite-width lazy limit the loss would not depend on $P$ at all. The $P$-scaling law is a finite-size effect on the lazy limit. At finite $P$ the feature Gram matrix is a random matrix, correlated with the random sample, and its rank caps how many modes of the data spectrum the student can resolve. All of the physics lives in the disorder average over this randomness, which is what the resolvents and the planar diagrams compute. Frozen does not mean deterministic.

Three notable results:

• Where exponents come from: the loss inherits the data's spectral exponent, $L\sim \min (P,n{)}^{-\zeta }$ up to logarithms. Remarkably, generic nonlinear feature maps extend the data's power-law spectrum rather than reshaping it. This explains why exponents are roughly architecture-independent and can only be improved by changing the data (or the task).
• Duality and Chinchilla: the loss is (to leading order) symmetric under $P\leftrightarrow n$. This predicts ${\alpha }_P\approx {\alpha }_n$, consistent with observation, and opposite in spirit to the quanta model's asymmetric prediction (Exercise 7.1(e)). It makes equiparameterization $P\propto n$ compute-optimal, a first-principles Chinchilla rule.
• Where laws break: scaling stops when $\min (P,n)$ exceeds the latent dimensionality of the data model, and the loss plateaus. Power laws are a statement about the resolvable structure in the data, and they end when the structure is exhausted.

7.9 All the knobs at once: a dynamical theory

Everything so far is statics, or one knob at a time. The current frontier is the dynamical mean-field theory of Section 6 applied to scaling. Bordelon, Atanasov, and Pehlevan (Bordelon et al. 2024) solve a random-feature model trained by gradient flow, with power-law data spectra, in the joint limit of large width, large data, and long time. They obtain a single self-consistent prediction for $L(t,P,n)$, with the correlation and response functions of Section 6 as the order parameters. Each resource, when it is the binding constraint, contributes its own power law, recovering the resolution-limited exponents of Section 7.7 in the appropriate corners. Moreover, the theory predicts the full compute-optimal frontier: minimizing $L$ at fixed compute $C\propto Pt$ yields Chinchilla-like allocation rules and the frontier exponent, rather than assuming them.

A fair question: the features here are again frozen and random, exactly the setting of Section 7.8, so is dynamical mean-field theory overkill? No, for two reasons. First, time is a genuine third resource. Predicting $L(t,P,n)$ and the compute frontier requires the whole training trajectory, and the statics only sees the $t\to \infty$ corner. Second, the naive mode-by-mode solution of the lazy regime, in which each kernel eigenmode of the residual decays as $e^{-\eta {\lambda }_{k}t}$ as in (4.34), applies to a deterministic kernel. Here the kernel's eigenvalues and eigenvectors are random and correlated with the sampled data, so the disorder average couples the modes. The DMFT correlation and response functions are the dynamical generalization of the static resolvents of Section 7.8, and at $t\to \infty$ the equations collapse back onto the static answer, as they must. What the theory is not yet doing is feature learning. The adaptive-kernel version of the formalism exists (Bordelon & Pehlevan 2022), but it has not been pushed through power-law data to closed-form scaling laws; these are the "joint laws with genuine feature learning" missing from Section 7.3. Since feature learning is known to improve exponents for structured targets in solvable cases, the correction need not be small.

7.10 Beyond pretraining: post-training and test-time scaling

Frontier models are no longer just pretrained. They are fine-tuned on curated demonstrations (SFT), optimized against reward models or verifiable rewards (RLHF, RLVR), and allowed to spend variable compute at inference time. Do scaling laws extend to these stages? Increasingly, yes. But the functional forms change, and the changes are instructive.

Transfer and fine-tuning still look Kaplan-like. The value of pretraining for a downstream task can be quantified as the effective data transferred, meaning the amount of fine-tuning data that the pretrained initialization is worth, and it scales as a power law in both the fine-tuning dataset size and the model size (Hernandez et al. 2021).

Reward-model optimization introduces a new scaling axis: not data or parameters, but the distance moved from the base policy, $d=\sqrt{\mathrm{KL}(\pi \Vert {\pi }_0)}$. Gao, Schulman, and Hilton (Gao et al. 2023) found that the true ("gold") reward follows simple laws in $d$, namely $d(a-b\log d)$ for best-of-$n$ sampling and $d(a-bd)$ for RL, with coefficients that scale smoothly with the size of the reward model. Note the shape: not a power law but a rise and a fall. Optimizing a learned proxy eventually degrades the true objective. This is a Goodhart peak, quantified.

RL on verifiable rewards, the training recipe of the reasoning models, scales by saturating sigmoids rather than power laws. In the largest systematic study to date, performance as a function of RL compute is predictable, but by a fit with an asymptote, an efficiency, and a midpoint, and most recipe choices move the efficiency without changing the asymptote (Khatri et al. 2025). A physics gloss suggests why the form differs, though it is a hypothesis rather than a result. Pretraining mines an effectively unbounded Zipf tail of new structure, hence power laws. RL post-training mostly elicits and reweights capabilities the base model already contains, on a bounded task distribution. In quanta language, pretraining acquires quanta, while RL sharpens the policy over quanta it already has, and one runs out. Test-time compute is the other new axis: the coverage achieved by repeated sampling grows roughly as a power law in the number of samples (Brown et al. 2024), and train-time and test-time compute can be traded against each other along a compute-optimal frontier (Snell et al. 2024).

What does not yet exist is a joint law over all four budgets (pretraining, SFT, RL, inference), and any theory at all, even at the level of the quantization model, that derives the sigmoid or the Goodhart peak from a model of the data and skill distribution. The empirical forms are waiting for a statistical-mechanics treatment.

That every ingredient of the pretraining synthesis — Gibbs measures, kernels, mean-field limits, MSRJD actions — appeared earlier in these notes as a physics import is, we hope, a fair summary of the state of the field. A theory of the scaling laws is not finished, but it is recognizably being built, and it is being built out of statistical physics. The post-training laws are younger still: their regularities are only now being settled, and there is no theory of them yet. For a student entering the field, that is a map of where the open territory begins.

A. The replica method

Section 2.3 introduced the quenched free energy $-{\mathbb{E}}_{\mathrm{data}}\log Z$ and the annealed approximation (2.18) that replaces it by $-\log {\mathbb{E}}_{\mathrm{data}}Z$, and Section 2.6 used that approximation to analyze the Ising perceptron. This appendix explains how the quenched average is computed exactly, by the replica method, and what the annealed approximation misses. We keep the Ising perceptron of Section 2.6 as the running example, in the notation and normalization of that section: $N$ binary weights $\theta \in \{\pm 1{\}}^N$, a binary teacher ${\theta }^{\ast }$, Gaussian inputs, $n=\alpha N$ examples, and the partition function (2.39). The classic reference for everything here is Engel and Van den Broeck (Engel & Van den Broeck 2001).

A.1 What the annealed approximation gets wrong

Jensen's inequality gives $\log \mathbb{E}Z\ge \mathbb{E}\log Z$, so the annealed free energy $-\log \mathbb{E}Z$ is always a lower bound on the quenched one. To see what drives the gap, write $Z=e^{-N\phi }$, where the free-energy density $\phi$ fluctuates from dataset to dataset. Exponentiation turns modest fluctuations of $\phi$ into enormous fluctuations of $Z$: a dataset whose $\phi$ is below the typical value by a constant is exponentially rare, but it contributes an exponentially large $Z$, and the two exponentials compete at the same order in $N$. The average $\mathbb{E}Z$ can therefore be dominated by vanishingly rare datasets, while the typical dataset, which is what $\mathbb{E}\log Z$ describes, behaves differently. (This is the familiar behavior of lognormal random variables, whose mean sits far in the upper tail. The mean is a poor summary of a heavy-tailed quantity.)

For the Ising perceptron the direction of the error follows. The datasets that dominate $\mathbb{E}Z$ are those that constrain the student unusually weakly, and weakly constrained students sit at low overlap. The annealed average therefore overweights poorly aligned students, and overestimates the amount of data needed to learn: it puts the transition to perfect generalization at ${\alpha }_c^{\mathrm{ann}}\approx 1.45$, where the exact calculation sketched below gives ${\alpha }_c\approx 1.245$ (Gy"orgyi 1990). The structure of the transition, two basins exchanging dominance, is the same in both.

A.2 Replicas

The full calculation computes ${\mathbb{E}}_{\mathrm{data}}\log Z$ through the identity

The strategy is to compute $\mathbb{E}{Z}^Q$ for integer $Q$, where it has a transparent meaning, obtain a formula analytic in $Q$, and continue it to $Q\to 0$. For integer $Q$, $Z^Q$ is the partition function of $Q$ independent copies, or replicas, of the student, all judged against the same dataset. For the Ising perceptron, from (2.39),

The data average is easy, because the inputs are Gaussian. For fixed replicas and teacher, each example enters only through the $Q+1$ numbers $u^a={\theta }^a\cdot x^{\mu }/\sqrt{N}$ and $v={\theta }^{\ast }\cdot x^{\mu }/\sqrt{N}$, which are exactly jointly Gaussian (they are linear in $x^{\mu }$), with mean zero and a covariance determined entirely by the overlap matrix

The product of step functions asks that every $u^a$ have the sign of $v$, and the probability of that event is some function $e^{-e(q,R)}$ of the overlaps alone: a Gaussian orthant probability in $Q+1$ dimensions. Since the examples are independent, the data average of the product over $\mu$ is the $n$-th power, $e^{-ne(q,R)}=e^{-\alpha Ne(q,R)}$. For $Q=1$ the only overlap is $R$, the orthant is a wedge, and $e(R)=-\log (1-\epsilon (R))$ is the energy term of Section 2.6.

The disorder is gone. In its place, the replicas are coupled through the overlaps $q^{ab}$, and the sum over the $Q$ binary vectors can be organized by the value of the overlaps, exactly as the sum over one binary vector was organized by $R$ in Exercise 2.3(c). Writing $e^{Nh(q,R)}$ for the number of $Q$-tuples of students with prescribed overlaps (for $Q=1$ this is the binomial count of Exercise 2.3(c)), and replacing the sum over overlaps by an integral,

This is the replicated version of the one-variable action (2.44) of Section 2.6, to which it reduces at $Q=1$. At large $N$ the integral is dominated by the minimum of $s_Q$ over the overlaps, and the replica identity (A.1) then gives the quenched free-energy density

Compare the annealed approximation, which is ${\min }_R{s}_1(R)$: the annealed calculation keeps a single replica, and so never sees the coupling $q^{ab}$ between replicas that the shared data induce. That coupling is precisely the information about dataset-to-dataset fluctuations that Appendix A.1 found missing.

Replica symmetry. To make (A.5) concrete one needs $h$ and $e$ as functions of the $Q(Q+1)/2$ overlaps, and a way to continue the result in $Q$. The standard ansatz, replica symmetry, is to take $q^{ab}=q$ for all $a\ne b$ and $R^a=R$. It is natural, since the saddle-point equations are symmetric under permuting replicas, and it reduces the problem to two scalars. Under it, both $h$ and $e$ become one-dimensional Gaussian integrals: the orthant probability collapses to an integral over a single shared Gaussian "field" of the Gaussian tail function $H(x)={\int }_x^{\infty }\frac{dz}{\sqrt{2\pi }}{e}^{-z^2/2}$, and the count $h$ is handled by introducing conjugate variables for the constraints, which reduces it to a one-site problem in the manner of the mean-field Ising model of Section 2.4. The resulting expression is analytic in $Q$ and can be differentiated at $Q=0$. As a caution, symmetric equations can have asymmetric solutions, and this happens in physical systems, famously in spin glasses; the assumption must be checked, and for the perceptron problems considered here it holds.

The result. For the Ising perceptron, the replica-symmetric calculation (Gy"orgyi 1990) reproduces the two-basin structure found in Exercise 2.3: an interior solution with $R^{\ast }(\alpha )<1$ that drifts toward the teacher as data accumulate, and the teacher solution $R=1$, which exchange dominance in a first-order transition. Only the number moves, from ${\alpha }_c^{\mathrm{ann}}\approx 1.45$ to ${\alpha }_c\approx 1.245$. A guided walk through the calculation follows.

Exercise A.1 (The quenched calculation, via replicas). Work through the following program for the Ising perceptron, in as much detail as your appetite allows; full details are in (Engel & Van den Broeck 2001, chs. 2 and 7).

(a) Replicate. For integer $Q$, write $\mathbb{E}{Z}^Q$ as in (A.2): a $Q$-fold copy of the system, coupled only through the shared data.

(b) Average over data. Show that for fixed replicas and teacher, $u^a={\theta }^a\cdot x/\sqrt{N}$ and $v={\theta }^{\ast }\cdot x/\sqrt{N}$ are exactly jointly Gaussian with the covariance (A.3), and that the data average factorizes across examples, each contributing the same function $e^{-e(q,R)}$ of the overlaps. Check that at $Q=1$, $e(R)=-\log (1-\epsilon (R))$ with $\epsilon$ the wedge formula (2.41).

(c) Push forward to the order parameters. Insert delta functions fixing $q^{ab}$ and $R^a$ (in integral representation, with conjugate variables ${\hat{q}}^{ab}$, ${\hat{R}}^a$), so that the sum over the binary vectors factorizes over sites $i=1,\dots ,N$ and becomes the $N$-th power of a $Q$-spin sum. Conclude that $\mathbb{E}{Z}^Q$ takes the form (A.4), and that the annealed action (2.44) is its $Q=1$ case.

(d) Replica symmetry. Take $q^{ab}=q$ for $a\ne b$ and $R^a=R$ (and likewise for the conjugates). Show that the orthant probability becomes $e^{-e_{\mathrm{RS}}}=2\int DtH(\cdot {)}^Q$ for an appropriate argument of the tail function $H$, built from $q$, $R$, and a single Gaussian variable $t$, and that the site sum becomes a one-dimensional Gaussian integral of $(2\cosh (\cdot ){)}^Q$. Continue both in $Q$ and expand to first order at $Q=0$.

(e) Use the symmetry of Gibbs learning. A student drawn uniformly from the consistent set is statistically indistinguishable from the teacher, since the teacher is itself a uniformly random binary vector consistent with the data. Argue that this forces $q=R$ at the saddle: the overlap between two Gibbs students equals their overlap with the teacher. One scalar remains, with one conjugate.

(f) Solve. The saddle-point equations for $R(\alpha )$ are transcendental. Solve them numerically and show that there are two competing solutions, an interior one and $R=1$, whose free energies cross at ${\alpha }_c\approx 1.245$. Compare with the annealed ${\alpha }_c^{\mathrm{ann}}\approx 1.45$ of Exercise 2.3(g), and explain the direction of the shift using Appendix A.1.

Solution

The six steps are their own outline; we record only the checkpoints. (Step 2) Conditioned on the ${\theta }^a$ and ${\theta }^{\ast }$, the $u^a$ and $v$ are linear in the Gaussian vector $x$, hence exactly Gaussian, with the covariances (A.3); independence over examples gives the $n$-th power, so the per-example factor is $e^{-e(q,R)}$ with $e=-\log P[\text{all }Q\text{ replicas agree with the teacher}]$, a $(Q+1)$-dimensional Gaussian orthant probability. At $Q=1$ the orthant is the double wedge of Exercise 2.3(b), of probability $1-\epsilon (R)$. (Step 3) With $\delta (N{q}^{ab}-{\theta }^a\cdot {\theta }^b)=\int \frac{d{\hat{q}}^{ab}}{2\pi }{e}^{i{\hat{q}}^{ab}(N{q}^{ab}-{\sum }_i{\theta }_i^a{\theta }_i^b)}$ and likewise for $R^a$, the exponent is a sum over sites of identical terms, so ${\sum }_{\{\theta \}}=({\sum }_{{\theta }_i^1,\dots ,{\theta }_i^Q=\pm 1}{e}^{-i{\sum }_{a<b}{\hat{q}}^{ab}{\theta }^a{\theta }^b-i{\sum }_a{\hat{R}}^a{\theta }^a{\theta }^{\ast }}{)}^N$; the saddle point over the conjugates then defines $h(q,R)$. At $Q=1$ only $\hat{R}$ survives and the site sum is $2\cosh \hat{R}$, reproducing the binomial entropy after the saddle in $\hat{R}$. (Step 4) Under replica symmetry, the $Q$ correlated Gaussians $u^a$ can be written as $\sqrt{q}t+\sqrt{1-q}{z}^a$ with $t$ shared and the $z^a$ independent, and $v$ correlated with $t$ through $R$; integrating out the $z^a$ gives the orthant probability as $2\int DtH(\cdot {)}^Q$, where the argument of $H$ is linear in $t$ with coefficients built from $q$ and $R$. The site sum becomes $\int Dz(2\cosh (\sqrt{\hat{q}}z+\hat{R}){)}^Q$ after decoupling the $\hat{q}{\theta }^a{\theta }^b$ term by the same trick. Both are analytic in $Q$, and $\frac{1}{Q}\log \int (\cdots {)}^Q\to \int (\cdots )\log (\cdots )$ as $Q\to 0$. (Step 5) The teacher is a uniformly random binary vector consistent with every example, exactly as a Gibbs student is, so the two are exchangeable, and $\mathbb{E}[{\theta }^a\cdot {\theta }^b]=\mathbb{E}[{\theta }^a\cdot {\theta }^{\ast }]$ at the saddle: $q=R$. (Step 6) Solving the remaining pair of equations numerically, the interior solution exists for all $\alpha$ up to a spinodal and the teacher solution $R=1$ always exists; their free energies cross at ${\alpha }_c\approx 1.245$ (Gy"orgyi 1990), below the annealed $1.45$. The annealed average is dominated by rare, weakly constraining datasets, which overweight poorly aligned students and so require more data before the teacher basin wins; removing that bias moves the transition earlier.

A.3 A variant: the spherical perceptron

The replica method is easiest to push all the way to closed form in a variant of the problem with continuous weights, which is also the classic case in which the annealed and quenched learning curves can be compared analytically. Keep everything in Section 2.6 the same, but let the weights be real, $\theta ,{\theta }^{\ast }\in {\mathbb{R}}^N$. Since $f_{\theta }=\mathrm{sign}(\theta \cdot x)$ is invariant under $\theta \to \lambda \theta$ with $\lambda >0$, remove the redundant scale by putting the weights on a sphere,

normalized so that individual weights stay of order one, and let the prior be the uniform measure $\pi (\theta )d\theta$ on the sphere. This is the spherical perceptron. The wedge formula (2.41) for the generalization error is unchanged, since its derivation used only that $(\theta \cdot x,{\theta }^{\ast }\cdot x)/\sqrt{N}$ is a Gaussian pair with correlation $R$, which holds for any $\theta$ with $|\theta {|}^2=N$. What changes is the entropy: the number of students at overlap $R$ becomes the volume of the slice of the sphere at overlap $R$. Decompose $\theta =R{\theta }^{\ast }+{\theta }_{\perp }$ with ${\theta }_{\perp }\perp {\theta }^{\ast }$. The constraint (A.6) gives $|{\theta }_{\perp }{|}^2=N(1-R^2)$, so the slice is a sphere of radius $\sqrt{N(1-R^2)}$ in the $(N-1)$-dimensional orthogonal complement, of volume $\propto (N(1-R^2){)}^{(N-2)/2}$, and

Repeating the annealed calculation of Exercise 2.3(d) with this entropy gives

and the annealed free-energy density is

Figure 9 shows $S_{\mathrm{ann}}$ for several loads. The entropy–energy competition is the same as for the Ising perceptron, but the outcome is different: the entropy $-\frac{1}{2}\log (1-R^2)$ diverges as $R\to 1$, so the perfectly aligned state is infinitely penalized, the minimum always sits in the interior, and it slides continuously from $R=0$ toward $R=1$ as $\alpha$ grows. There is no phase transition. This is the answer to the closing question of Exercise 2.3(h): the binary weights of the Ising perceptron give the aligned state finite entropy, $h(1)=0$, and that is what allows it to compete as a separate basin.

The learning curve. The large-$\alpha$ behavior follows by expanding near $R=1$. Writing $R=\cos (\pi \epsilon )$, so that $\epsilon$ is the generalization error itself, $1-R^2\approx (\pi \epsilon {)}^2$ and

which is minimized at

The generalization error decays as a power law in the amount of data, with exponent one: the simplest instance of the scaling laws of Section 7. The exact, quenched treatment (Seung et al. 1992; Engel & Van den Broeck 2001) runs through the replica steps of Exercise A.1 with the volume entropy (A.7) in place of the count (here replica symmetry can be justified by the convexity of the region of the sphere consistent with the data), and yields

the same exponent, with a different constant. As in Appendix A.1, the annealed average overweights poorly aligned students and overestimates the error, $1/\alpha >0.625/\alpha$; and since the model is realizable, the exponent survives the approximation even though the constant does not.

Exercise A.2 (The annealed learning curve of the spherical perceptron). (a) Derive the slice entropy (A.7), and assemble the annealed action (A.8) by repeating the steps of Exercise 2.3(d).

(b) Derive the saddle-point condition for (A.9), and show that $R=1$ is never a minimum, in contrast with Exercise 2.3(f).

(c) Verify the large-$\alpha$ learning curve (A.11), and show that at small $\alpha$ the annealed theory predicts $R\propto \alpha$. Sketch the full learning curve $\epsilon (\alpha )$.

(d) Which steps of Exercise A.1 change when the count of binary students is replaced by the volume (A.7), and which do not?

Solution

(a) Decompose $\theta =R{\theta }^{\ast }+{\theta }_{\perp }$. The spherical constraint $|\theta {|}^2=N$ forces $|{\theta }_{\perp }{|}^2=N(1-R^2)$, so the slice at fixed $R$ is a sphere of radius $\sqrt{N(1-R^2)}$ in the $(N-1)$-dimensional orthogonal complement, of volume $\propto (N(1-R^2){)}^{(N-2)/2}=e^{\frac{N}{2}\log (1-R^2)+O(\log N)}$, giving $h(R)=\frac{1}{2}\log (1-R^2)$ up to constants. The data factor $(1-\epsilon (R){)}^n=e^{N\alpha \log (1-\epsilon )}$ is as in Exercise 2.3(d), and collecting exponents gives (A.8).

(b) Setting $S_{\mathrm{ann}}^{\prime }(R)=0$ with $\epsilon (R)=\arccos (R)/\pi$ gives

$$\frac{R}{1-R^2}=\frac{\alpha }{\pi \sqrt{1-R^2}(1-\epsilon (R))}⟺\frac{R}{\sqrt{1-R^2}}=\frac{\alpha }{\pi (1-\epsilon (R))}.$$

As $R\to 1$ the entropy term $-\frac{1}{2}\log (1-R^2)\to +\infty$ while the energy term stays finite, so $S_{\mathrm{ann}}\to +\infty$ and $R=1$ is never a minimum; the minimizer is always interior.

(c) With $R=\cos (\pi \epsilon )$ and $\epsilon \to 0$: $1-R^2={\sin }^2(\pi \epsilon )\approx (\pi \epsilon {)}^2$, so $S_{\mathrm{ann}}\approx -\log (\pi \epsilon )+\alpha \epsilon +\text{const}$, minimized at $\epsilon =1/\alpha$. For small $R$ the entropic part is $\approx R^2/2$ and $\epsilon (R)\approx \frac{1}{2}-R/\pi$, so $-\alpha \log (1-\epsilon )\approx -\alpha \log \frac{1}{2}-\frac{2\alpha }{\pi }R$, and minimizing $R^2/2-\frac{2\alpha }{\pi }R$ gives $R^{\ast }=2\alpha /\pi \propto \alpha$. The learning curve rises linearly from $R=0$, bends over, and approaches $R=1$ with the power-law tail $\epsilon \simeq 1/\alpha$.

(d) Steps 1, 2, 5, and 6 are unchanged: the data average sees only the overlaps, and the Gibbs-symmetry argument holds for any prior invariant under the symmetries of the problem. Step 3 changes: the sum over binary vectors becomes an integral over $Q$ points of the sphere, and the count $h(q,R)$ becomes the log-volume of the set of $Q$-tuples with prescribed overlaps, a Gaussian integral with a $\log \det$ of the overlap matrix; under replica symmetry it is $\frac{1}{2}[(Q-1)\log (1-q)+\log (1-q+Qq-Q{R}^2)]$ up to constants, whose $Q\to 0$ derivative reproduces (A.7) at $q=R$. Step 4 is then the same $H$-function integral. The outcome differs because this entropy diverges at $R\to 1$: no competing basin, no transition, and a learning curve $\epsilon \simeq 0.625/\alpha$ (Seung et al. 1992).

B. From linear regression to kernel regression

This appendix is a self-contained review of models that are linear in their parameters: linear regression, regression on fixed nonlinear features, kernel regression, and kernel ridge regression. It ends with the observation that a neural network linearized around its initialization — the ansatz (4.1) of Section 4 — is exactly such a model, so that training it amounts to kernel ridge regression. Nothing here requires large width, gradient-flow asymptotics, or moving features; those are the subjects of Sections 3–5, and this appendix supplies the linear algebra they rest on. Readers who found the kernel formulas of Sections 4.1 and 4.6 appearing out of thin air may want to read it first. ⁶

We use the notation of Section 3.1: training data $(x^{\mu },y^{\mu }{)}_{\mu =1}^n$ with $x^{\mu }\in {\mathbb{R}}^d$ and $y^{\mu }\in \mathbb{R}$; a model $f_{\theta }:{\mathbb{R}}^d\to \mathbb{R}$ with parameters $\theta \in {\mathbb{R}}^P$; and $f^{\mu }:=f_{\theta }(x^{\mu })$ for its values on the training inputs. It is convenient to stack the data into matrices,

so that $X^{\mu }{}_j=x_j^{\mu }$ is the $j$-th coordinate of the $\mu$-th input, and the inputs and their labels are stacked in the same way. Throughout, the loss is the MSE of Section 4.1.1,

with $\Delta \in {\mathbb{R}}^n$ the residual. Many references (including the notebook this appendix is drawn from) normalize the loss as $\frac{1}{2n}|f-Y{|}^2$ instead; this rescales the ridge parameter $\lambda$ below by a factor of $n$ and the learning rate $\eta$ by a factor of $1/n$, and changes nothing else.

B.1 Linear regression

The simplest model is linear in the inputs,

so that $P=d$ and the vector of training predictions is $f=X\theta$. (An intercept can be included by appending a constant coordinate $x_{d+1}=1$ to every input, as in Exercise B.1 below.) The loss ${\mathcal{L}}_{\theta }=\frac{1}{2}|X\theta -Y{|}^2$ is a convex quadratic function of $\theta$, with gradient

This has the shape that every gradient in these notes has: the transpose of the Jacobian $\partial f^{\mu }/\partial {\theta }^j$ — here the constant matrix $X$ — acting on the residual, cf. (4.10). Setting the gradient to zero gives the normal equations

Since the loss is convex, every solution of (B.5) is a global minimizer, and there are no other critical points. What the solutions look like depends on how $P=d$ compares with $n$.

Fewer parameters than data, $P<n$. If the columns of $X$ are linearly independent — the generic situation when $d\le n$ — then $X^{T}X$ is positive definite, since $v^T{X}^{T}Xv=|Xv{|}^2>0$ for $v\ne 0$. There is a unique minimizer,

The normal equations say that the residual ${\Delta }_{\ast }=X{\theta }_{\ast }-Y$ is orthogonal to every column of $X$. Equivalently, the prediction $X{\theta }_{\ast }$ is the orthogonal projection of $Y$ onto the column space $\mathrm{col}(X)\subset {\mathbb{R}}^n$, the set of all prediction vectors the model can produce (Figure 10). Fitting is projecting. The projection $X{\theta }_{\ast }$ is unique even in cases where ${\theta }_{\ast }$ is not.

It is worth pausing on the word "orthogonal." Least squares minimizes the Euclidean ($L^2$) distance between $X\theta$ and $Y$, and the projection is orthogonal with respect to the Euclidean metric ${\delta }_{\mu \nu }$ on ${\mathbb{R}}^n$ — the same metric on the space of function values that was made explicit in Section 4.1.1. This metric is a choice, hidden inside the phrase "mean squared error." It declares that an error of a given size counts the same at every data point, and that errors at different data points are independent. Any other positive-definite metric $g_{\mu \nu }$ on ${\mathbb{R}}^n$ defines a different loss ${\mathcal{L}}^g=\frac{1}{2}{g}_{\mu \nu }{\Delta }^{\mu }{\Delta }^{\nu }$, with a different minimizer

which is the $g$-orthogonal projection of $Y$ onto $\mathrm{col}(X)$ ("weighted least squares"). Unless $Y$ already lies in $\mathrm{col}(X)$, different metrics give genuinely different fitted functions; see Exercise B.1(d). Note that $g_{\mu \nu }$ is a metric on the space of function values indexed by the data points, not a metric on the input space ${\mathbb{R}}^d$.

More parameters than data, $P>n$. If $d>n$, or more generally if the columns of $X$ are linearly dependent, then $X^{T}X$ is singular. The normal equations still have solutions (Exercise B.1(b)), but now infinitely many: an affine subspace of ${\mathbb{R}}^P$ of dimension $P-\mathrm{rank}(X)$. If moreover $X$ has full row rank $n$, then $Y\in \mathrm{col}(X)$ and every solution interpolates the data exactly, $X\theta =Y$, with zero loss. The loss has no way to prefer one of them. Something else must make the choice; that is the subject of Appendix B.3, and it is where kernels come from. Far from being an edge case, $P>n$ is the normal situation for neural networks.

Exercise B.1 (Least squares by hand). (a) Differentiate ${\mathcal{L}}_{\theta }=\frac{1}{2}|X\theta -Y{|}^2$ with respect to a single coordinate ${\theta }^j$, assemble the components into (B.4), and derive the normal equations (B.5). Explain geometrically why they say that the residual is orthogonal to every column of $X$.

(b) Show that if $X$ has full column rank then $X^{T}X$ is positive definite, so that (B.6) is the unique minimizer. Show that the normal equations are always consistent (i.e. $X^{T}Y\in \mathrm{col}(X^{T}X)$), and that if $P>n$ they have infinitely many solutions.

(c) Fit the affine function $f(x)={\theta }^1+{\theta }^{2}x$ to the three points $x\in \{-1,0,1\}$ with labels $y=x^2$; that is, take

$$X=\left(\begin{array}{cc}1& -1\\ 1& 0\\ 1& 1\end{array}\right),Y=\left(\begin{array}{c}1\\ 0\\ 1\end{array}\right).$$

Find ${\theta }_{\ast }$, the predictions, the residual, and the loss. Is the remaining error a failure of optimization or of representation?

(d) Repeat (c) with the weighted loss (B.7) for $g=\mathrm{diag}(1,4,1)$, which trusts the middle data point four times as much as the others. Check that the fitted function changes.

Solution

(a) $\partial \mathcal{L}/\partial {\theta }^j={\sum }_{\mu }{X}^{\mu }{}_j({\sum }_k{X}^{\mu }{}_k{\theta }^k-y^{\mu })=[X^T(X\theta -Y){]}_j$. The $j$-th normal equation sets the inner product of the $j$-th column of $X$ with the residual to zero, so ${\Delta }_{\ast }\perp \mathrm{col}(X)$; hence $X{\theta }_{\ast }$ is the orthogonal projection of $Y$ onto $\mathrm{col}(X)$.

(b) $v^T{X}^{T}Xv=|Xv{|}^2$ vanishes only if $Xv=0$, which for $v\ne 0$ is impossible when the columns are independent; so $X^{T}X$ is positive definite and invertible. Consistency: $\ker (X^{T}X)=\ker X$ (if $X^{T}Xv=0$ then $|Xv{|}^2=v^T{X}^{T}Xv=0$), and for any matrix $A$ one has $\mathrm{col}(A^T)=(\ker A{)}^{\perp }$, so $\mathrm{col}(X^{T}X)=(\ker X{)}^{\perp }=\mathrm{col}(X^T)\ni X^{T}Y$. If $P>n$ then $\dim \ker X=P-\mathrm{rank}(X)\ge P-n>0$, and adding any element of $\ker X$ to a solution gives another.

(c) $X^{T}X=\mathrm{diag}(3,2)$, $X^{T}Y=(2,0{)}^T$, so ${\theta }_{\ast }=(2/3,0{)}^T$: the constant function $2/3$. Predictions $(2/3,2/3,2/3{)}^T$, residual $(-1/3,2/3,-1/3{)}^T$, loss $\frac{1}{2}\cdot \frac{6}{9}=\frac{1}{3}$. The optimization is exact (the problem is convex and was solved in closed form); the error is representational — no affine function agrees with $x^2$ at three points. Exercise B.2 repairs it by adding a feature.

(d) $X^{T}gX=\mathrm{diag}(6,2)$ and $X^{T}gY=(2,0{)}^T$, so ${\theta }_{\ast }^g=(1/3,0{)}^T$: the constant function $1/3$, pulled toward the heavily weighted middle label $y=0$. Residual $(-2/3,1/3,-2/3{)}^T$. A different metric on ${\mathbb{R}}^n$ gives a different fit.

B.2 Linear regression with nonlinear features

"Linear model" is a statement about how the parameters enter, not about the shape of the fitted function. Choose, before looking at any labels, a feature map

$$\psi :{\mathbb{R}}^d\to {\mathbb{R}}^P,\psi (x)=({\psi }_1(x),\dots ,{\psi }_P(x)),$$

and consider the model

This is a weighted sum of $P$ fixed shapes ${\psi }_i$. It can be as nonlinear in $x$ as the shapes are, but it is linear in $\theta$. Standard choices include monomials in the coordinates of $x$ up to some degree $m$ (giving $P=\left(\genfrac{}{}{0ex}{}{d+m}{m}\right)$), Fourier modes, Gaussian bumps $e^{-|x-c_k{|}^2/2{\sigma }^2}$ centered at chosen points $c_k$, and random features ${\psi }_i(x)=\varphi (a_i\cdot x)$ with the $a_i$ drawn at random and then frozen — the last being a one-hidden-layer network whose first layer is never trained.

Stacking the feature vectors of the training inputs into the $n\times P$ feature matrix

the training predictions are $f=\Psi \theta$. Note that ${\Psi }^{\mu }{}_i=\partial f^{\mu }/\partial {\theta }^i$: the feature matrix is precisely the Jacobian $J$ of Section 4.2, which for a linear model is a constant. Linear regression is the special case $\psi (x)=x$, $\Psi =X$, and everything in Appendix B.1 goes through verbatim with $X$ replaced by $\Psi$: the normal equations are ${\Psi }^T\Psi \theta ={\Psi }^{T}Y$; if $P\le n$ and $\Psi$ has full column rank, the unique minimizer is ${\theta }_{\ast }=({\Psi }^T\Psi {)}^{-1}{\Psi }^{T}Y$; and if $P>n$ there are infinitely many interpolants.

Replacing $X$ by $\Psi$ changes the algebra not at all, and changes the model completely. Figure 11 shows three feature maps, each with $P=n$, fitted to the same seven points. Each interpolates the data exactly, and they disagree everywhere else. Which is right? The data cannot say: the behavior away from the training set was fixed by the choice of $\psi$, which was made before the labels were seen. The feature map is the model's inductive bias, made explicit.

Why would one ever want a very large $P$? Figure 12 gives the classic answer. Two classes of points on concentric rings cannot be separated by a line in the plane. Append one fixed feature, $\psi (u,v)=(u,v,c(u^2+v^2))$, and the classes become separable by a plane in ${\mathbb{R}}^3$: a model that is linear in the lifted coordinates is a circle in the original ones. The price is that $P$ grows fast. All monomials of degree $3$ in $d=1000$ input coordinates already give $P=\left(\genfrac{}{}{0ex}{}{1002}{3}\right)\approx 1.7\times 10^8$, and Gaussian bumps centered at every point of a continuum give $P=\infty$. Yet there are only $n$ data points, and the fit is determined by $n$ numbers. Nothing about the problem should require a $P\times P$ inverse — so where does $P$ actually enter? Answering that question honestly is the content of the next two sections.

Exercise B.2 (A three-point feature calculation). Take the same three points $x\in \{-1,0,1\}$ with labels $y=x^2$ as in Exercise B.1(c), but now the feature map $\psi (x)=(1,\sqrt{2}x,x^2)$.

(a) Form the feature matrix $\Psi$ and find coefficients $\theta$ that interpolate all three labels. The representational failure of Exercise B.1(c) is gone: what changed?

(b) Explain precisely how the predictor can be nonlinear in $x$ while the regression problem remains linear.

(c) If the last feature is replaced by $c{x}^2$ for some $c\ne 0$, what changes: the fitted function, its coefficient, or both?

(d) Write the inner product $K(x,z):=\psi (x)\cdot \psi (z)$ in closed form, without vector notation. This is a preview of Appendix B.3.

Solution

(a) $\Psi =\left(\begin{array}{ccc}1& -\sqrt{2}& 1\\ 1& 0& 0\\ 1& \sqrt{2}& 1\end{array}\right)$, which is invertible ($P=n=3$), so exactly one $\theta$ interpolates: $\theta =(0,0,1{)}^T$, giving $f_{\theta }(x)=x^2$ and predictions $(1,0,1{)}^T$. What changed is that $Y$ now lies in $\mathrm{col}(\Psi )={\mathbb{R}}^3$.

(b) All nonlinearity in $x$ sits inside the fixed functions ${\psi }_i$; $f_{\theta }={\sum }_i{\theta }^i{\psi }_i$ depends on $\theta$ linearly, so the loss is quadratic in $\theta$ and the normal equations are linear.

(c) The coefficient becomes $1/c$, the product ${\theta }^3{\psi }_3=x^2$ is unchanged, and so is the fitted function. (With the ridge penalty of Appendix B.4 the rescaling would change the fit, since $|\theta {|}^2$ is not invariant under it.)

(d) $K(x,z)=1+2xz+x^2{z}^2=(1+xz{)}^2$.

B.3 Kernel regression

Return to the overparametrized case $P>n$, and assume that $\Psi$ has full row rank $n$ (the generic situation), so that there is a $(P-n)$-dimensional affine family of exact interpolants $\Psi \theta =Y$. To choose one, we need a criterion that the loss does not provide. The standard choice is the interpolant of minimum norm,

This introduces a second implicit Euclidean metric — now ${\delta }_{ij}$ on parameter space ${\mathbb{R}}^P$, the metric that also defines gradient descent in Section 4.1.1. (Appendix B.1 used ${\delta }_{\mu \nu }$ on the space ${\mathbb{R}}^n$ of function values.) The two metrics play dual roles: the first decides how to fit when the data cannot be matched, the second decides which fit to choose when it can.

To solve (B.11), introduce Lagrange multipliers $\alpha \in {\mathbb{R}}^n$ for the $n$ constraints and extremize $\frac{1}{2}|\theta {|}^2-\alpha \cdot (\Psi \theta -Y)$. Stationarity in $\theta$ gives

This is the key structural fact: however large $P$ is, the minimum-norm solution lies in the $n$-dimensional subspace of ${\mathbb{R}}^P$ spanned by the feature vectors of the training inputs, and is described by $n$ numbers. (In the language of Section 4.2, ${\theta }_{\ast }$ lies in the row space of the Jacobian.) The constraint $\Psi {\theta }_{\ast }=Y$ then becomes $\Psi {\Psi }^T\alpha =Y$, an $n\times n$ linear system. Define the kernel and its Gram matrix on the training set,

The kernel is an inner product of feature vectors: a notion of similarity between inputs, fixed once $\psi$ is fixed, and involving no labels. The Gram matrix is symmetric and positive semidefinite, since $c^T\hat{K}c=|{\Psi }^{T}c{|}^2\ge 0$; it is positive definite exactly when $\Psi$ has full row rank, which we assumed. Then $\alpha ={\hat{K}}^{-1}Y$, and the prediction at an arbitrary input $x$ is

This is kernel regression (more precisely, kernel interpolation). The feature map has disappeared from both the fit and the prediction. To fit, one needs $K$ on pairs of training points; to predict, one needs $K$ between the test point and the training points; no feature vector is ever constructed. The prediction is a weighted sum of similarities to the training points, with weights $\alpha$ that depend on the labels — but the labels only reweight the training points, they never change which inputs $K$ considers similar.

Compare (B.14) with the late-time mean (4.38) of a lazily trained network: they are the same formula, with $\hat{K}$ in place of the NTK. This is no coincidence. For any model linear in its parameters the Jacobian is $J=\Psi$, so the NTK (4.10) is

constant in time. Moreover, gradient flow finds the minimum-norm solution: $\dot{\theta }=-\eta {\Psi }^T\Delta$ always points along the row space of $\Psi$, so $\theta (t)-{\theta }_0$ stays in that $n$-dimensional subspace and converges to the interpolant nearest to ${\theta }_0$ — for ${\theta }_0=0$, to ${\theta }_{\ast }$ (Exercise B.3(c)). This is the simplest instance of the implicit bias of gradient descent, and it is the Euclidean metric on parameter space entering through the back door: the flow is defined by ${\delta }^{ij}$, and so is the norm that it ends up minimizing.

The kernel trick. There are two readings of (B.14). Read from the feature side, it rewrites a $P\times P$ problem as an $n\times n$ one. Read from the kernel side, it says the feature map is dispensable: any function $K(x,x^{\prime })$ that arises as an inner product of features will do, whether or not one can write the features down. For $x,z\in {\mathbb{R}}^2$ and $\psi (x)=(x_1^2,\sqrt{2}{x}_1{x}_2,x_2^2)$, one checks that $\psi (x)\cdot \psi (z)=(x\cdot z{)}^2$: a single dot product, squared, stands in for three features. In $d$ dimensions, $K(x,z)=(x\cdot z{)}^m$ costs $O(d)$ to evaluate and stands in for all $\left(\genfrac{}{}{0ex}{}{d+m-1}{m}\right)$ monomials of degree $m$. The Gaussian kernel $K(x,z)=e^{-|x-z{|}^2/2{\sigma }^2}$ corresponds to an infinite-dimensional feature map that no one ever writes down. Conversely, any symmetric $K$ all of whose Gram matrices are positive semidefinite arises from some feature map (Mercer's theorem): the eigenfunctions of $K$, weighted by the square roots of its eigenvalues, furnish one (cf. Sections 3.3 and 4.6). This is a trade, not a free lunch. The feature route scales with $P$ and the kernel route with $n$; with many examples and few features, the explicit route is the cheap one.

Exercise B.3 (Minimum-norm interpolation and the kernel). (a) Carry out the Lagrange-multiplier computation leading to (B.12)–(B.14). Alternatively, use the singular value decomposition $\Psi =US{V}^T$ of Section 4.2 to show that the minimum-norm interpolant is ${\theta }_{\ast }={\Psi }^T(\Psi {\Psi }^T{)}^{-1}Y$, and write it in terms of $U,S,V$.

(b) Show that $c^T\hat{K}c=|{\Psi }^{T}c{|}^2\ge 0$ for all $c\in {\mathbb{R}}^n$. Interpret a zero eigenvalue of $\hat{K}$ as a linear combination of training examples that vanishes in feature space. What does it take for $\hat{K}$ to be invertible?

(c) Show that under gradient flow $\dot{\theta }=-\eta {\Psi }^T(\Psi \theta -Y)$ the displacement $\theta (t)-{\theta }_0$ stays in the row space of $\Psi$ for all $t$; derive $\Delta (t)=e^{-\eta t\hat{K}}\Delta (0)$; and conclude that, for invertible $\hat{K}$, the flow converges to the interpolant closest to ${\theta }_0$. Derive (4.36) for this model, with $\Theta =\hat{K}$.

(d) Verify that $\psi (x)\cdot \psi (z)=(x\cdot z{)}^2$ for $\psi (x)=(x_1^2,\sqrt{2}{x}_1{x}_2,x_2^2)$, and explain why the $\sqrt{2}$ is not decoration. Which feature map on $\mathbb{R}$ produces the kernel $(1+xz{)}^2$ of Exercise B.2(d)? How many features does $(x\cdot z{)}^m$ on ${\mathbb{R}}^d$ stand in for?

Solution

(a) Stationarity of $\frac{1}{2}|\theta {|}^2-\alpha \cdot (\Psi \theta -Y)$ in $\theta$ gives $\theta ={\Psi }^T\alpha$; the constraint then reads $\Psi {\Psi }^T\alpha =\hat{K}\alpha =Y$, so $\alpha ={\hat{K}}^{-1}Y$ and ${\theta }_{\ast }={\Psi }^T{\hat{K}}^{-1}Y$; the prediction is $\psi (x)\cdot {\theta }_{\ast }={\sum }_{\mu }K(x,x^{\mu }){\alpha }_{\mu }$. It is a minimum, not merely a critical point, because the objective is strictly convex and the constraint set is affine. In the SVD of Section 4.2, $\Psi =US{V}^T$ with $S=[\mathrm{diag}(s_1,\dots ,s_n)0]$, one has $\hat{K}=U\mathrm{diag}(s_a^2)U^T$ and ${\theta }_{\ast }=V{S}^T\mathrm{diag}(s_a^{-2})U^{T}Y={\sum }_{a=1}^n\frac{u_a\cdot Y}{s_a}{v}_a$, where $v_a$ are the first $n$ columns of $V$: only right singular vectors with nonzero singular value (the row space) appear.

(b) $c^T\hat{K}c=c^T\Psi {\Psi }^{T}c=|{\Psi }^{T}c{|}^2\ge 0$. If $\hat{K}c=0$ then ${\Psi }^{T}c={\sum }_{\mu }{c}_{\mu }\psi (x^{\mu })=0$: a linear dependence among the training feature vectors (for instance two identical inputs, or simply $n>P$). $\hat{K}$ is invertible iff the $n$ training feature vectors are linearly independent, which requires $n\le P$.

(c) $\dot{\theta }=-\eta {\Psi }^T\Delta \in \mathrm{col}({\Psi }^T)$, the row space of $\Psi$, so $\theta (t)-{\theta }_0=-\eta {\Psi }^T{\int }_0^t\Delta$ stays there. Since $\Psi$ is constant, $\dot{\Delta }=\Psi \dot{\theta }=-\eta \Psi {\Psi }^T\Delta =-\eta \hat{K}\Delta$, whence $\Delta (t)=e^{-\eta t\hat{K}}\Delta (0)\to 0$. Integrating, $\theta (t)-{\theta }_0=-\eta {\Psi }^T{\int }_0^t{e}^{-\eta s\hat{K}}ds\Delta (0)=-{\Psi }^T{\hat{K}}^{-1}(1-e^{-\eta t\hat{K}})\Delta (0)$, and $f_{\theta (t)}(x)=f_{{\theta }_0}(x)+\psi (x)\cdot (\theta (t)-{\theta }_0)$ is (4.36) with $\Theta (x,x^{\mu })=K(x,x^{\mu })$ and $\Theta =\hat{K}$. At $t\to \infty$, ${\theta }_{\infty }={\theta }_0-{\Psi }^T{\hat{K}}^{-1}\Delta (0)$ interpolates, and ${\theta }_{\infty }-{\theta }_0\in \mathrm{row}(\Psi )$. Any other interpolant ${\theta }^{\prime }$ has ${\theta }^{\prime }-{\theta }_{\infty }\in \ker \Psi =\mathrm{row}(\Psi {)}^{\perp }$, so by Pythagoras $|{\theta }^{\prime }-{\theta }_0{|}^2=|{\theta }_{\infty }-{\theta }_0{|}^2+|{\theta }^{\prime }-{\theta }_{\infty }{|}^2\ge |{\theta }_{\infty }-{\theta }_0{|}^2$: the flow converges to the interpolant closest to ${\theta }_0$, and for ${\theta }_0=0$ to ${\theta }_{\ast }$.

(d) $\psi (x)\cdot \psi (z)=x_1^2{z}_1^2+2x_1{x}_2{z}_1{z}_2+x_2^2{z}_2^2=(x_1{z}_1+x_2{z}_2{)}^2$; without the $\sqrt{2}$ the cross term would be $x_1{x}_2{z}_1{z}_2$ and the sum would not be a perfect square. $(1+xz{)}^2=1+2xz+x^2{z}^2$ comes from $\psi (x)=(1,\sqrt{2}x,x^2)$, as in Exercise B.2. By the multinomial theorem $(x\cdot z{)}^m={\sum }_{|\alpha |=m}(\genfrac{}{}{0ex}{}{m}{\alpha })x^{\alpha }{z}^{\alpha }$, so ${\psi }_{\alpha }(x)=\sqrt{\left(\genfrac{}{}{0ex}{}{m}{\alpha }\right)}{x}^{\alpha }$ over the $\left(\genfrac{}{}{0ex}{}{d+m-1}{m}\right)$ monomials of degree $m$.

B.4 Kernel ridge regression

Rather than imposing the minimum-norm criterion as a hard constraint, one can add it to the loss as a penalty. For $\lambda >0$, ridge regression minimizes

The two terms measure distance to the data with ${\delta }_{\mu \nu }$ and distance to the origin with ${\delta }_{ij}$: ridge regression is a weighted compromise between the two metrics of Appendices B.1 and B.3, with $\lambda$ setting the exchange rate. (In the training of neural networks the penalty is called weight decay.) The penalty does two jobs at once. When $P>n$ it selects one solution out of infinitely many; and for any $P$ it makes the problem well posed even when the features are redundant or the labels are noisy. Note that the penalty is not invariant under rescaling features: it decides what counts as a "large" coefficient in the coordinates $\psi$ that were supplied.

The gradient is ${\Psi }^T(\Psi \theta -Y)+\lambda \theta$, so the stationarity condition is

The matrix on the left is positive definite for every $\lambda >0$, whatever the rank of $\Psi$, since $v^T({\Psi }^T\Psi +\lambda I_P)v=|\Psi v{|}^2+\lambda |v{|}^2>0$. So there is always a unique minimizer,

at the price of a $P\times P$ inverse. To move the inverse to the $n\times n$ side, expand and regroup the product

$${\Psi }^T(\Psi {\Psi }^T+\lambda I_n)={\Psi }^T\Psi {\Psi }^T+\lambda {\Psi }^T=({\Psi }^T\Psi +\lambda I_P){\Psi }^T.$$

Both bracketed matrices are positive definite, so multiplying by their inverses on the appropriate sides gives the push-through identity

Nothing was cancelled — ${\Psi }^T$ is rectangular and has no inverse — and that is precisely why the identity holds. Substituting into (B.18), the ridge solution again lies in the span of the training feature vectors, ${\theta }_{\lambda }={\Psi }^T{\alpha }_{\lambda }$, now with ${\alpha }_{\lambda }=(\hat{K}+\lambda I_n{)}^{-1}Y$, and the prediction at a new input is

This is kernel ridge regression, the central formula of this appendix. As $\lambda \to 0$ it reduces to the minimum-norm interpolant (B.14) when $\hat{K}$ is invertible, and in general to the least-squares solution of minimum norm (Exercise B.4(c)). As $\lambda \to \infty$, ${\theta }_{\lambda }\to 0$.

Formula (B.21) is exactly the posterior mean of Exercise 4.5, with ridge parameter $\lambda ={\beta }^{-1}$ — there derived for the infinite-width Gaussian process, here for a finite linear model. The Bayesian reading is worth making explicit. With a Gaussian likelihood $\propto e^{-\frac{\beta }{2}|\Psi \theta -Y{|}^2}$ and a Gaussian prior $\theta \sim \mathcal{N}(0,{\sigma }^2{I}_P)$, the loss (B.16) is $-{\beta }^{-1}\log$ of the posterior density, up to a constant, with $\lambda =1/(\beta {\sigma }^2)$. Ridge regression is thus maximum a posteriori estimation. The prior pushed forward to $f_{\theta }(x)=\theta \cdot \psi (x)$ is a Gaussian process with kernel ${\sigma }^{2}K(x,x^{\prime })$, a finite-$P$ instance of Section 3.3, and the posterior mean of Exercise 4.5 for this process reproduces (B.21).

Spectral view. With the singular value decomposition $\Psi =US{V}^T$ of Section 4.2, the Gram matrix is $\hat{K}=U{S}^2{U}^T$, with eigenvalues $s_a^2$ and orthonormal eigenvectors $u_a\in {\mathbb{R}}^n$: the "features" of Section 4.2, i.e. the natural modes in the space of function values. The ridge predictions on the training set are $f_{\lambda }=\Psi {\theta }_{\lambda }=\hat{K}(\hat{K}+\lambda I_n{)}^{-1}Y$, or, mode by mode,

Ridge regression is a spectral filter: modes of the data with $s_a^2\gg \lambda$ are fit, modes with $s_a^2\ll \lambda$ are suppressed, and the crossover is at $s_a^2\approx \lambda$. Compare gradient flow from ${\theta }_0=0$ on the unpenalized loss, whose training predictions are, by (4.35),

This is also a spectral filter, with $1/\eta t$ playing the role of $\lambda$: early stopping regularizes much as ridge does. But the two filters have different shapes (Figure 13), and no choice of $\lambda$ reproduces a given stopping time on every mode at once (Exercise B.5). This is the precise content of the remark closing Section 4.6, that the inverse temperature of Bayesian learning and the training time of gradient descent play analogous roles.

Exercise B.4 (Ridge regression, two ways). (a) Derive (B.17) and (B.18), and prove that ${\Psi }^T\Psi +\lambda I_P$ is invertible even when $\Psi$ is rank deficient.

(b) Prove the push-through identity (B.20) and deduce (B.21). Explain why "cancelling ${\Psi }^T$" is not a valid step, and why the identity is nevertheless true. (It is a special case of $A(BA+I{)}^{-1}=(AB+I{)}^{-1}A$.)

(c) What happens as $\lambda \to \infty$? Using the SVD, show that ${\theta }_{\lambda }$ applies the scalar filter $s/(s^2+\lambda )$ to each singular value $s$ of $\Psi$, and hence that as $\lambda \to 0$ it converges to the least-squares solution of minimum norm. When does this limit interpolate the labels exactly?

(d) Return to $x\in \{-1,0,1\}$, $y=x^2$ and the kernel $K(x,z)=(1+xz{)}^2$ of Exercise B.2. With $\lambda =1$, verify that

$$\hat{K}=\left(\begin{array}{ccc}4& 1& 0\\ 1& 1& 1\\ 0& 1& 4\end{array}\right),{\alpha }_{\lambda }=\frac{1}{4}\left(\begin{array}{c}1\\ -1\\ 1\end{array}\right),$$

and predict at $z=2$ using only kernel evaluations. Compare with the interpolant of Exercise B.2(a), which predicts $z^2=4$ there, and explain the direction of the discrepancy.

(e) The feature route solves a $P\times P$ system and the kernel route an $n\times n$ one. When is each attractive?

Solution

(a) Setting ${\Psi }^T(\Psi \theta -Y)+\lambda \theta =0$ gives (B.17); $v^T({\Psi }^T\Psi +\lambda I_P)v=|\Psi v{|}^2+\lambda |v{|}^2\ge \lambda |v{|}^2>0$ for $v\ne 0$, so the matrix is positive definite, hence invertible, for every $\lambda >0$.

(b) ${\Psi }^T(\Psi {\Psi }^T+\lambda I_n)={\Psi }^T\Psi {\Psi }^T+\lambda {\Psi }^T=({\Psi }^T\Psi +\lambda I_P){\Psi }^T$ uses only associativity. Multiply on the left by $({\Psi }^T\Psi +\lambda I_P{)}^{-1}$ and on the right by $(\Psi {\Psi }^T+\lambda I_n{)}^{-1}$, both of which exist by (a). ${\Psi }^T$ is $P\times n$ and, for $P\ne n$, has no two-sided inverse, so it cannot be "cancelled"; the identity holds because ${\Psi }^T$ was regrouped, not divided out. Then ${\theta }_{\lambda }={\Psi }^T(\hat{K}+\lambda I_n{)}^{-1}Y$ and $f_{\lambda }(x)=\psi (x)\cdot {\theta }_{\lambda }={\sum }_{\mu }K(x,x^{\mu })({\alpha }_{\lambda }{)}_{\mu }$.

(c) As $\lambda \to \infty$, ${\theta }_{\lambda }\approx {\Psi }^{T}Y/\lambda \to 0$. With $\Psi =US{V}^T$, ${\Psi }^T\Psi +\lambda I_P=V(S^{T}S+\lambda I_P)V^T$, so ${\theta }_{\lambda }=V(S^{T}S+\lambda I_P{)}^{-1}{S}^T{U}^{T}Y={\sum }_a\frac{s_a}{s_a^2+\lambda }(u_a\cdot Y)v_a$. As $\lambda \to 0$, $s_a/(s_a^2+\lambda )\to 1/s_a$ for $s_a>0$ and $\to 0$ for $s_a=0$: this is the pseudo-inverse ${\Psi }^{+}Y$, the least-squares solution of minimum norm (an arbitrary unregularized minimizer need not have minimum norm). It interpolates iff $Y\in \mathrm{col}(\Psi )$, i.e. iff $u_a\cdot Y=0$ for every zero singular value.

(d) $K(x^{\mu },x^{\nu })=(1+x^{\mu }{x}^{\nu }{)}^2$ on $\{-1,0,1\}$ gives the displayed $\hat{K}$. Check $(\hat{K}+I_3){\alpha }_{\lambda }=\frac{1}{4}(5-1,1-2+1,-1+5{)}^T=(1,0,1{)}^T=Y$. At $z=2$ the kernel vector is $(K(-1,2),K(0,2),K(1,2))=(1,1,9)$, so $f_{\lambda }(2)=\frac{1}{4}(1-1+9)=\frac{9}{4}$, versus $4$ for the interpolant. The ridge fit is shrunk toward zero: it no longer even interpolates the training labels (its training predictions are $\hat{K}{\alpha }_{\lambda }=(3/4,1/4,3/4{)}^T$), and the quadratic growth it does fit is damped.

(e) The feature route costs $O(n{P}^2+P^3)$ and is attractive when $P\ll n$ and the features are explicit; the kernel route costs $O(n^2\cdot (\text{cost of }K)+n^3)$ and is attractive when $n\ll P$, or when $K$ can be evaluated without constructing $\psi$ at all. Neither is free: $n\times n$ is fatal for large $n$.

Exercise B.5 (Early stopping vs. ridge, on two data points). Take $n=2$ training points with Gram matrix and labels

$$\hat{K}=\left(\begin{array}{cc}2& 1\\ 1& 2\end{array}\right),Y=\left(\begin{array}{c}2\\ 0\end{array}\right),$$

and initial predictions $f(0)=0$.

(a) Verify the eigenpairs $u_{\pm }=\frac{1}{\sqrt{2}}(1,\pm 1{)}^T$ with $s_{\pm }^2=3,1$, and decompose $Y$ into these modes.

(b) Using (B.23), find the training predictions $f(t)$ under gradient flow, and evaluate them at $\eta t=\log 2$. Why do both the eigenvalue $s_a^2$ and the initial projection $u_a\cdot Y$ matter for what is visible during training?

(c) Using (B.22), find the ridge predictions for $\lambda =1$. The two filters agree on the mode with $s^2=1$; show that the ridge parameter matching gradient flow at time $t$ on a mode with eigenvalue $s^2$ is ${\lambda }_{\mathrm{eff}}(s^2,t)=s^2/(e^{\eta t{s}^2}-1)$, which depends on the mode — so early stopping is not ridge regression.

(d) State precisely what "spectral bias" means for this finite, fixed-kernel calculation. What additional structure would be needed to turn it into a claim about Fourier frequencies, about robustness to label noise, or about test error?

Solution

(a) $\hat{K}{u}_{\pm }=(2\pm 1)u_{\pm }$, so $s_{+}^2=3$, $s_{-}^2=1$; and $Y=\sqrt{2}{u}_{+}+\sqrt{2}{u}_{-}=(1,1{)}^T+(1,-1{)}^T$.

(b) $f(t)=(1-e^{-3\eta t})(1,1{)}^T+(1-e^{-\eta t})(1,-1{)}^T=(2-e^{-3\eta t}-e^{-\eta t},e^{-\eta t}-e^{-3\eta t}{)}^T$. At $\eta t=\log 2$, $e^{-\eta t}=\frac{1}{2}$ and $e^{-3\eta t}=\frac{1}{8}$, giving $(11/8,3/8{)}^T$. The eigenvalue sets a mode's rate, while $u_a\cdot Y$ sets whether it is present and with what amplitude; a fast mode with zero initial amplitude leaves no trace in the trajectory.

(c) Ridge multipliers $\frac{3}{3+1}=\frac{3}{4}$ and $\frac{1}{1+1}=\frac{1}{2}$, so $f_{\lambda }=\frac{3}{4}(1,1{)}^T+\frac{1}{2}(1,-1{)}^T=(5/4,1/4{)}^T$. The flow multipliers at $\eta t=\log 2$ are $\frac{7}{8}$ and $\frac{1}{2}$: the two agree on $s^2=1$ only. Solving $s^2/(s^2+\lambda )=1-e^{-\eta t{s}^2}$ gives ${\lambda }_{\mathrm{eff}}=s^2{e}^{-\eta t{s}^2}/(1-e^{-\eta t{s}^2})=s^2/(e^{\eta t{s}^2}-1)$; at $\eta t=\log 2$ this is $1$ for $s^2=1$ but $3/7$ for $s^2=3$. No single $\lambda$ matches both modes.

(d) Here "spectral bias" means exactly this: components of the residual along eigenvectors of the fixed Gram matrix with larger eigenvalue decay faster, at rate $\eta s_a^2$. To speak of Fourier frequencies one needs a relation between the $u_a$ and Fourier modes (true, for instance, for translation-invariant kernels on uniform grids). To speak of noise one needs a model of how label noise projects onto the $u_a$ (typically spread evenly across modes, so that it lives mostly in the many slow modes — which is why early stopping and ridge both filter it). To speak of test error one needs a distribution over inputs and the population version of the kernel operator, as in Sections 4.6 and 7.7.

B.5 Linearized networks are kernel ridge regressors

Now let $f_{\theta }$ be a neural network, with $\theta \in {\mathbb{R}}^P$ the vector of all its weights and biases. It is not linear in $\theta$. But its linearization around the initialization ${\theta }_0$, the ansatz (4.1),

is a model of the form (B.9). Compare term by term. The parameters are the displacement $\delta \theta :=\theta -{\theta }_0\in {\mathbb{R}}^P$. The feature map is

the tangent features: one number per parameter, the sensitivity of the output at $x$ to that parameter, evaluated at initialization and then frozen. And the constant offset $f_{{\theta }_0}(x)$ is absorbed by shifting the labels to

The offset matters: the linearized model is fit to the residuals at initialization, not to the raw labels. The feature matrix (B.10) is the Jacobian at initialization, $\Psi =J_0$, and the Gram matrix (B.13) is

the neural tangent kernel (4.10) at initialization, extended to arbitrary pairs of inputs as in Exercise 4.3. Everything in Appendices B.3 and B.4 now applies verbatim. In particular, ridge regression on the linearized network with penalty $\frac{\lambda }{2}|\theta -{\theta }_0{|}^2$ — weight decay toward the initialization — gives, by (B.21),

At $\lambda \to 0$ this is the $t\to \infty$ limit of (4.36), the minimum-norm interpolant of the residuals; and the finite-time solution (4.36) is its early-stopped cousin, in the sense of (B.23). The kernel trick is not a convenience here but a necessity: $P$ is the parameter count of the network, often $10^7$ or more, while ${\Theta }_0$ is $n\times n$.

Two remarks on what is and is not being claimed. First, for the linearized model $f^{\mathrm{lin}}$ everything above is exact, and its kernel is constant by construction. For the actual network, (B.24) is an ansatz. Section 4.3 shows that it becomes exact in the large-width limit at fixed depth and data; whether and when it is a good approximation otherwise — and what happens when the tangent features move — is the subject of Sections 5–6, not of this appendix. Second, the initialization ${\theta }_0$ is random, so $f_{{\theta }_0}$, the tangent features, and the kernel ${\Theta }_0$ are all random too, and (B.28) is the trained function for one draw. Averaging over draws, at large width where ${\Theta }_0$ becomes deterministic, gives the Gaussian process (4.38).

Exercise B.6 (Tangent features of a one-hidden-layer network). Consider $f_{\theta }(x)={\sum }_{i=1}^N{c}_i\varphi (a_i\cdot x)$ with $\theta =(c_i,a_i{)}_{i=1}^N$, so that $P=N(d+1)$.

(a) Compute the tangent features ${\nabla }_{\theta }{f}_{{\theta }_0}(x)$, and show that the tangent kernel is

$$\Theta (x,x^{\prime })=\sum _{i=1}^N\varphi (a_i\cdot x)\varphi (a_i\cdot x^{\prime })+\sum _{i=1}^N{c}_i^2{\varphi }^{\prime }(a_i\cdot x){\varphi }^{\prime }(a_i\cdot x^{\prime })(x\cdot x^{\prime }).$$

(b) Suppose only the output weights $c$ are trained, with the $a_i$ frozen at their random initial values. Show that this is exactly a fixed-feature model (B.9) with random features, that its linearization is exact, and that its kernel is the first term above. Which term do you get by training only the $a_i$?

(c) Show that gradient flow with weight decay toward the initialization, $\dot{\theta }=-\eta [{\nabla }_{\theta }\mathcal{L}+\lambda (\theta -{\theta }_0)]$, applied to the linearized model $f^{\mathrm{lin}}$, converges to the ridge solution (B.28) for every $\lambda >0$ and from any starting point.

(d) Check that (B.28) at $\lambda \to 0$ coincides with (4.36) at $t\to \infty$.

Solution

(a) $\partial f/\partial c_i=\varphi (a_i\cdot x)$ and $\partial f/\partial a_{ij}=c_i{\varphi }^{\prime }(a_i\cdot x)x_j$. Taking the inner product of the tangent features at $x$ and $x^{\prime }$ and summing over $j$ in the second block gives the displayed $\Theta (x,x^{\prime })$.

(b) With the $a_i$ frozen, $f={\sum }_i{c}_i{\psi }_i(x)$ with ${\psi }_i(x)=\varphi (a_i\cdot x)$ is linear in the trainable parameters $c$, so $f=f^{\mathrm{lin}}$ exactly, ${\Psi }^{\mu }{}_i=\varphi (a_i\cdot x^{\mu })$, and ${\hat{K}}^{\mu \nu }={\sum }_i\varphi (a_i\cdot x^{\mu })\varphi (a_i\cdot x^{\nu })$ is the first term: the random-features model. Training only the $a_i$ gives the second term, but there the linearization is not exact, since $f$ is nonlinear in $a$.

(c) Writing $\delta \theta =\theta -{\theta }_0$, the linearized loss is ${\mathcal{L}}^{\mathrm{lin}}=\frac{1}{2}|J_0\delta \theta +\Delta (0){|}^2$, so $\delta \dot{\theta }=-\eta [J_0^T(J_0\delta \theta +\Delta (0))+\lambda \delta \theta ]=-\eta [M\delta \theta +J_0^T\Delta (0)]$ with $M=J_0^T{J}_0+\lambda I_P$. This is a linear ODE with positive-definite $M$, so $\delta \theta (t)=\delta {\theta }_{\infty }+e^{-\eta tM}(\delta \theta (0)-\delta {\theta }_{\infty })$ converges from any start to $\delta {\theta }_{\infty }=-M^{-1}{J}_0^T\Delta (0)=-J_0^T({\Theta }_0+\lambda I_n{)}^{-1}\Delta (0)$, by push-through. Then $f^{\mathrm{lin}}(x)=f_{{\theta }_0}(x)+{\nabla }_{\theta }{f}_{{\theta }_0}(x)\cdot \delta {\theta }_{\infty }$ is (B.28), since ${\nabla }_{\theta }{f}_{{\theta }_0}(x)\cdot J_0^T$ has components $\Theta (x,x^{\mu })$.

(d) As $\lambda \to 0$, $({\Theta }_0+\lambda I_n{)}^{-1}\to {\Theta }_0^{-1}$; as $t\to \infty$ in (4.36), $(1-e^{-\eta t\Theta })\to 1$. Both give $f_{{\theta }_0}(x)-\Theta (x,x^{\mu }){\Theta }_{\mu \nu }^{-1}{\Delta }^{\nu }(0)$.

C. Solutions to exercises

Worked solutions to the exercises, in the order in which the exercises appear. (Instructors: uncommenting \solutionsfalse in the preamble removes this appendix's contents, and all other solution boxes, from the compiled PDF.)

References

Footnotes

1. A physicist's free energy is $-{\beta }^{-1}\log Z$; we absorb the factor of $\beta$ throughout, so that $F$ is dimensionless. ↩

2. For transformers, the limit analogous to "large width" that leads to a Gaussian process is a large number of attention heads. ↩

3. As a point of notation, $\eta$ and $\zeta$ are also commonly used for noise, and we will distinguish between discrete and continuum delta functions as ${\delta }_{ij}$ and $\delta (t-t^{\prime })$ respectively. ↩

4. Michaud's blog post ericjmichaud.com/quanta is an excellent companion to the paper and can substitute for it in this exercise. But attempt Exercise 7.1 before reading either; the derivations are short, and they are more instructive to find than to follow. ↩

5. This is also a working example for Section 6's toolkit: the same model can be solved dynamically, and its double-descent peak, at the interpolation threshold $P\sim n$, is regularized by ridge or early stopping, connecting back to the divergence of the overfitting factor mentioned in Section 7.7. ↩

6. This appendix adapts a lecture and an exercise notebook prepared by Charles Renshaw-Whitman for the Iliad Intensive, translated into the notation of Section 3.1. ↩