Iliad

Worst-Case Interpretability

Cluster EE.3 · Worst-Case Interpretability

How do we know an interpretation is right? We treat an explanation as a proof of model performance, where shorter proofs mean deeper understanding, and work through compact proofs for a toy max-of-K transformer to see where the bounds break down.

By Louis Jaburi (EleutherAI)

Prerequisites

  • Linear algebra (SVD, eigendecomposition, matrix norms, singular values)
  • C.2 Mechanistic Interpretability: Circuits, ablations, SAEs. More generally transformers
  • Basic information + probability theory
What you’ll learn
  • Students understand the difficulty of evaluating and validating interpretations
  • Students are able to formalize interpretations as lossy compressions with quantifiable faithfulness and compactness and its limitations through the three sources of exponential blowup (error along paths, number of paths, input space coverage)
  • Students are able to articulate why worst-case bounds become vacuous even on simple models, and why this motivates an intermediate approach between worst-case and average-case evaluation
  • Students on a high level understand ARC's heuristic arguments agenda as a response to these limitations

Reading guide

Fast-track

Main content

1. Limitations of interpretability evaluation Current interpretability methods are typically evaluated using faithfulness metrics based on ablations. These metrics turn out to be fragile.

2. Compact proofs framework If average-case metrics are unreliable, can we do better? Proofs offer a rigorous alternative: formalize an interpretation as a compression of the model with quantifiable faithfulness and compactness, then prove guarantees. The compact proofs paper shows what happens when you try in a toy mode.

3. From proofs to heuristic arguments The vacuousness of worst-case bounds suggests we need something between full proofs and mere average-case evaluation. ARC's heuristic arguments agenda proposes such an intermediate approach.

Further reading