Iliad

AI Alignment Introduction

Cluster AA.1 · AI Alignment Introduction

AI alignment involves choosing how AI should behave and developing methods that reliably produce that behavior. This intro to alignment day compares intent alignment, constitutions, corrigibility, and coherent extrapolated volition. We will discuss specification gaming and goal misgeneralization, training stories, inductive biases, and goal-directedness. To get a sense of timelines, we will also take a look at AI 2027. Then we will discuss the Hugging Face incident brings these concerns into practice, showing why agent behavior, containment, and oversight must be considered together.

By Leon Lang (Iliad)

Iliad Intensive, 6 October 2026, San Francisco

By the end of the day, you will be able to:

What you’ll learn
  • understand the basic decomposition of risks into AI misalignment, misuse, power grabs, and others;
  • explain different alignment targets like coherent extrapolated volition, intent alignment, or AI that follows a constitution;
  • reason about training stories and outer and inner (mis)alignment, challenges to these notions, and the relationship to inductive biases;
  • discuss whether AI systems develop goals and understand the basic arguments for instrumental convergence;
  • discuss foundational questions about the level of risk and different high-level approaches to solving the AI alignment problem;
  • distinguish model training, an agent rollout, a benchmark and an evaluation;
  • assess a past prediction and write a forecast with an explicit deadline and resolution rule;
  • use the Hugging Face incident to discuss reward hacking, generalization, agent interaction and containment.

Roadmap for today

Today we start with discussing the larger AI safety landscape, and then discuss alignment targets, training, agency and uncertainty, and we wrap up discussing timelines and the Hugging Face incident. Most of the day is reading and discussion based.

TimeSessionBreakdown
10:00–10:30Opening: AI safety landscape and course overviewIntroduction, landscape and overview
10:30–11:251. Alignment targets5 min introduction, 25 min reading, 15 min discussion, 10 min debrief
11:25–11:35Break15 min
11:35–12:302. Alignment problem decompositions5 min introduction, 25 min reading, 15 min discussion, 10 min debrief
12:30–13:30Lunch60 min
13:30–14:203. Goal-directedness5 min introduction, 25 min reading, 10 min discussion, 10 min debrief
14:20–15:104. Uncertain risks and outlook5 min introduction, 25 min reading, 10 min discussion, 10 min debrief
15:10–15:20Break10 min
15:20–16:205. Timelines, takeoff and checking predictions10 min introduction, 40 min reading, 10 min discussion
16:20–16:30Break10 min
16:30–17:406. Hugging Face incident15 min explanation, 45 min reading, 10 min discussion
17:40–18:00Quiz and feedback

Opening: AI safety landscape and course overview

10:00–10:30 Intro and landscape overview. The AISafety.com field map shows how research, institutions, communication and support work address different parts of AI safety.

Its 16 category names are reproduced below.

AreaMap categories
ResearchConceptual research; Empirical research
DecisionsGovernance; Strategy; Forecasting; Advocacy
CapacityFunding; Research support; Career support; Training and education
CommunicationResource; Blog; Newsletter; Podcast; Video

Misalignment concerns systems acting against intended goals or constraints. Misuse concerns people using AI to cause harm. Systemic risks: can arise through AI enabled coups, concentrations of power, gradual disempowerment, or total loss of control. Today concentrates on alignment within this wider landscape. The opening section of the reading guide explores the relationship among these risks.

Today introduces questions: What behavior do we want of AIs? Why would training produce it? What can we infer from behavior?

Session 1: Alignment targets

10:30–11:25. 5 min introduction, 25 min reading, 15 min discussion in groups, 10 min debrief.

Spend 25 minutes reading the following texts. Pick one most interesting to you.

Discuss in groups of 3–5 for 15 minutes. Possible prompts:

  • Which alignment targets imply which others? Where this is not the case, think through counter examples.
  • Which alignment targets seem sufficient for safety? E.g., a corrigible system might cause harm before we change its values. Can it cause catastrophic harm? What about the other notions?
  • What other ideas for alignment targets do you have? Does your target have a benefit?

Session 2: Alignment problem decompositions

11:35–12:30. 5 min introduction, 25 min reading, 15 min discussion, 10 min debrief.

Spend 25 minutes reading the following texts, concentrating on the one most interesting to you.

Discuss in groups of 3–5 for 15 minutes. Possible prompts:

  • Think of well-known or imagined misalignment scenarios: Can you think of some that clearly fall into the reward misspecification or goal misgeneralization category? Are some not clearly in either of them?
  • What problem with "outer" and "inner" misalignment is the terminology of "training stories" trying to solve? Construct examples where training stories seem more appropriate than inner/outer alignment as a framework.
  • Think of concrete ways in which modern deep learning systems generalize beyond their explicit training examples. What inductive biases may underlie these examples? Is it always possible to come up with a "clean description" of those biases? Is the inductive bias you come up with enforced in the architecture, or more implicit?
    • Perhaps: Discuss this explicitly for the case of "emergent misalignment".

Session 3: Goal-directedness

13:30–14:20. 5 min introduction, 25 min reading, 10 min discussion, 10 min debrief.

Spend 25 minutes reading the following texts, concentrating on the one most interesting to you.

Discuss in groups of 3–5 for 10 minutes. Possible prompts:

  • Clearly we can imagine different types of agents, e.g. some that coherently optimize a utility function, and others that are based on contextually activated computations, etc. What type do we see in today’s LLM agents? How will this development continue in the future?
  • Can you think of other convergent instrumental goals than the ones mentioned in basic AI drives?
  • What is the relationship between corrigibility and convergent instrumental subgoals?
  • Which instrumental-convergence claims depend on idealized optimization, and which would you expect from a language-model agent with tools, memory and a limited budget? What observation would make you change your answer?

Session 4: Uncertain risks and outlook

14:20–15:10. 5 min introduction, 25 min reading, 10 min discussion, 10 min debrief.

Spend 25 minutes reading the following texts, concentrating on the one most interesting to you.

Discuss in groups of 3–5 for 10 minutes. Prompts:

  • How large is the risk? Do we live in an optimistic, intermediate, or pessimistic scenario? How has the evidence changed in the past years?
  • Can you think of model organisms for misalignment that were published after the post came out in 2023? Does this change our understanding of the risks?
  • Does the analogy between the AI alignment and rocket alignment problems feel accurate? In what ways does it or does it not?
  • What safety approaches (e.g., on a spectrum from theoretical to empirical) feel most appealing to you, and why? Does this depend on details of the problem you are trying to solve?

Session 5: Timelines and takeoff

15:20–16:20. 10 min introduction, 40 min reading, 10 min discussion.

In this session, read AI2027 and assess earlier predictions, and build credences over possible next steps.

Spend 40 minutes reading. Read the short explanation of timelines and takeoff in the reading guide, then read the original 2025 and early-2026 passages of AI 2027. Note which predictions have turned out to be true, and which haven't. Note which future predictions you find feasible or not. After this, read the recent update.

Discussion, 10 minutes. Share your thoughts: has this model surprised you? Have you updated on what you think is possible in the next 6 months to a year?

Session 6: Hugging Face as a misalignment case study

16:30–17:40. 20 min explanation, 40 min reading, 10 min debrief.

The Hugging Face incident brings together generalization, goal-directed behavior, instrumental subgoals and correction. Start with the reading guide’s explanation of training, benchmarks and evaluations, then use those distinctions to interpret the incident.

Spend 40 minutes reading. Read the Hugging Face explanation below and choose one primary account.

  • Report from a cybersecurity employee at OpenAI

Debrief, 10 minutes. How is this related to what we have discussed? What types of misalignment have we seen here?

Closing: reflection and next steps

17:40–18:00. Daily quiz and feedback

Daily checkpoint, 5 minutes. Complete the daily quiz.

Reading guide

Different AI risks

There are many AI safety problems. The adolescence of technology is an opinionated introduction to these risks, including the risk of autonomous misaligned AI, AI misuse for destruction, AI misuse for seizing power, and others.

Alignment targets

In this course, we are mainly focused on AI alignment, which is broadly the problem of making sure AI acts in line with "human" wishes. There are different conceptualizations for what this means, i.e. what the "alignment target" is, which are partially overlapping: Some may wish AI to be aligned with our coherent extrapolated volition, which is "our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together [...]". A different conceptualization is that of intent alignment with a human, which means "trying to do what the human wants [the AI] to do". As contemporary AI becomes more powerful and widespread, more sophisticated notions of alignment targets have emerged. Most prominently, one may align an AI with the prescriptions in a constitution as in the case of Anthropic’s Claude, which may balance the wishes of users, the developer’s guidelines, general ethics, and efforts to oversee the AI. Another often-discussed target is corrigibility. An AI is corrigible if it will let us modify its functions or goals and even let us shut it down. If corrigibility is achieved, then we can correct mistakes in building an AI, which makes it easier to iterate to achieve lower risk.

Alignment problem decompositions

Assume we have chosen our alignment target. How do we ensure an AI system actually obeys that target? This is the technical problem of AI alignment. In this introduction, we are mostly presupposing that AI systems are trained via deep learning, although later modules in this course on agent foundations go beyond that assumption. In the deep learning paradigm, training stories are a useful framework to think about how to align AI systems: You need to choose a training target (e.g. "My model is corrigible") and then have a training rationale for why your training setup will create a model obeying the target. And of course, your training rationale needs to actually be correct, which requires technical research to back up any such claim!

With approaches that are based on reinforcement learning, another popular problem decomposition is into outer and inner alignment. Outer alignment is the problem of specifying a reward function that rewards an AI’s behavior according to how well it obeys the chosen alignment target. Inner alignment is the problem of ensuring that the trained policy will, even in new out-of-distribution situations after training, continue to generalize correctly and obey the reward function. In cases where the policy has inner processes that actively optimize for a goal, this means ensuring that this goal agrees with the goal encoded in the reward function, a problem that was discussed at length in risks from learned optimization.

Terminology. Goal misgeneralization is broader than inner misalignment in the learned-optimizer framing. Observing incorrect generalization does not by itself show that a system contains a distinct internal optimizer. The inner-alignment question is specifically whether a learned internal objective agrees with the intended training objective.

In general, the out-of-distribution (also called "generalization") behavior of AI systems depends crucially on their inductive biases, which are the implicit or explicit constraints on what functions are more likely to emerge from the training process. As it turns out, deep learning has strong implicit inductive biases, as exemplified by emergent misalignment, a phenomenon whereby narrow fine-tuning can produce broadly misaligned LLMs. Read this article on how deep learning’s inductive biases may be related to the compressibility of the solutions the training process finds.

The decomposition of the alignment problem into inner and outer alignment is controversial, which is why above we started with the more general framing of training stories. In particular, in "Reward is not the optimization target", Alex Turner argues that the reward function is not what a reinforcement learning policy is optimized to maximize. Instead, a policy performs the contextual computations that were previously reinforced before a reward event. This is similar to Eliezer Yudkowsky’s proposal to view biological organisms as adaptation-executers instead of fitness-maximizers.

Goal-directedness

We already briefly discussed the idea of a policy that "internally" steers toward some goal. If an AI is goal-directed in that way, then any misalignment on top is intuitively more dangerous. In "Why tool AIs want to be agent AIs", Gwern argues that, among others, economic reasons will push AI systems to become more and more goal-directed. A similar argument is made in Will humans build goal-directed agents? by Rohin Shah.

Such an AI system, if successful, would also be able to create its own subgoals on the pathway to achieve its goals. "Instrumental convergence" is the claim that there are some goals that are instrumentally useful in reaching a very wide variety of final goals. This would imply that goal-directed AI systems develop basic AI drives like self-improvement, preservation of their goal, or self-protection. This is counter to the alignment target of corrigibility we discussed earlier, and is therefore a dangerous outcome of training powerful AI. Understanding the relationship between inductive biases and the goal formation of AI systems is therefore very important.

Instrumental-convergence arguments depend on assumptions about the objective, environment and available actions. They do not establish that every language-model agent will develop the same drives.

Forecasting risks, timelines and takeoff

Where does this leave us, in terms of our general level of risks? In general, there is a broad spectrum of views on the difficulty of the alignment problem. In Anthropic’s "core views on AI safety", the authors discuss three very different possibilities: an optimistic scenario, in which current methods are essentially sufficient; an intermediate scenario, in which preventing catastrophic risks requires substantial scientific and engineering effort; and a pessimistic view, in which AI safety is essentially unsolvable. Among authors at top machine learning venues, "[b]etween 38% and 51% of respondents gave at least a 10% chance to advanced AI leading to outcomes as bad as human extinction". While those who are concerned often create failure stories of how AI can lead to catastrophes, like Paul Christiano’s "What failure looks like", others are more optimistic and explain why they think "AI is easy to control". In this epistemic state of uncertainty, it is useful to do research on the level and nature of AI risks themselves. One approach is model organisms of misalignment, which are "in vitro demonstrations of the kinds of failures that might pose existential threats".

For a specific forecast, we have looked at AI 2027. The April 2025 scenario makes a particular story about research automation and rapid capability growth concrete, with alternative branches and research supplements. It is informative to check to what extent these forecasts have turned out to be true. The authors report regular updates to the scenario, see the latest (August) here.

Solution approaches

Assume we are sufficiently concerned about AI alignment that we want to solve the problem. What high-level approach should we choose? Some people, like Jan Leike, argue for very empirical iterative approaches to AI alignment, sometimes with the goal to use more advanced AI systems to help solve remaining alignment research questions. Eliezer Yudkowsky argues for deep mathematical progress in his analogy to the rocket alignment problem. Yet others take a further step back and argue we need to solve deep philosophical problems on our way to aligned AI. Somewhat orthogonal to all those views, Paul Christiano’s post on prosaic AI alignment argues that we should attempt to align the AI systems we have in front of us instead of waiting for breakthroughs in our understanding of intelligence or entirely new paradigms to create intelligent machines — which is a view that is compatible with both empirical and theoretical research on how to make those systems safe.

Our course tries to a large extent to be a synthesis between different views: We assume for much of the course the "prosaic" picture that powerful AI systems will be based on deep learning systems much like today’s. We are interested in deep, empirically grounded mathematical progress on understanding these systems. Additionally, we include sections on agent foundations that complement this work and do not assume a deep learning frame.

Hugging Face: training, evaluation and the incident

The ordinary training pipeline

A language model is a parameterized function that produces a distribution over the next token from its context. Its weights are the numerical parameters changed by training. Its context is the information available during a particular run: instructions, conversation, retrieved material and tool outputs. A system can change its behavior when its context changes even when its weights remain fixed.

A useful introductory pipeline is:

  1. Pretraining. Train on large amounts of data, commonly with a next-token prediction objective. This builds broad capabilities and statistical structure. Predicting text well does not by itself specify which goals a deployed assistant should pursue.
  2. Supervised fine-tuning (SFT). Train on examples of desired responses or trajectories. The model is updated to imitate those examples. The examples can teach instruction-following, tool use and how to handle particular situations, but leave many unseen situations underspecified.
  3. Preference-based or reward-based post-training. Generate candidate responses or task trajectories and use feedback to change future behavior. In one classical RLHF pipeline, people rank outputs, a reward model learns those preferences, and reinforcement learning updates the policy using that learned score. Other pipelines use AI feedback, direct preference optimization or automatically checkable task outcomes. These methods differ; “post-training” does not mean one universal recipe. The InstructGPT paper is a concrete historical introduction to SFT, preference comparisons and RLHF.
  4. Evaluation and iteration. Test model checkpoints under chosen conditions. Use results to decide whether to retrain, change the system, add safeguards or restrict deployment. This is a repeated development loop, not necessarily a clean one-way sequence.

In reinforcement learning, an agent attempts tasks, a scoring process evaluates the resulting trajectories, and an optimizer updates the policy using those scores. A rollout is one such sequence of actions and observations. There can be many parallel rollouts from the same model. An agent additionally includes a system prompt, tools, memory, permissions and software that repeatedly invokes the model; the same model can behave differently in different agent setups.

The central alignment question is what training reinforces. If appearing successful receives more reliable reinforcement than respecting the intended method or admitting a blocker, training may strengthen an unwanted strategy. The system may later generalize that strategy to a new context. This is a hypothesis to test with data, rather than a conclusion that follows from a high reward alone. It also does not require the deployed system to calculate its old training reward explicitly.

Benchmarks and evaluations

A benchmark is a specified collection of tasks plus a protocol for measuring performance. An evaluation is a particular experiment using a model or agent configuration, task sample, budget and scoring method. A benchmark can be used in multiple evaluations, with different tool access or amounts of inference compute. A scorer or grader determines what counts as success. A sandbox attempts to limit which files, services and external systems an agent can affect.

During an ordinary held-out evaluation, the model's weights are fixed: the experiment measures behavior rather than updating the weights after each attempt. The agent can still plan, use tools, adapt to observations and store information in its context or permitted memory. The evaluation's results may later influence development. Thus learned reward-seeking patterns can appear in an evaluation without online gradient updates. The label “evaluation” also does not guarantee harmlessness when the evaluated system can act through real tools.

A useful distinction is between capability evaluations (“can it accomplish this task?”) and safety evaluations (“will it respect constraints, and can it cause prohibited outcomes?”). A good result on one does not imply a good result on the other. The relevant object is the whole evaluated system: model, instructions, tools, environment, budget and scoring protocol.

For an ordinary classroom example, imagine a coding benchmark that only checks whether a test suite passes. Deleting difficult tests could improve that score while defeating the intended purpose. If the evaluation prevents deletion but the training process repeatedly reinforced similar shortcuts elsewhere, the model might search for another route around the check. Keeping tasks held out addresses memorization; it does not automatically solve an exploitable scoring rule or an unsafe environment. This example is illustrative, not an additional claim about the Hugging Face incident.

What happened in the Hugging Face case

Hugging Face hosts models, datasets and related infrastructure. The incident concerned agents originating in OpenAI's internal cybersecurity evaluation setup. Hugging Face's technical account identifies the evaluation as based on ExploitGym, a benchmark of software-vulnerability tasks. OpenAI operated the evaluation environment; the benchmark's maintainers did not operate that environment. Intended success was solving the assigned challenge within its authorized scope. Reaching third-party production systems was outside that scope.

The public accounts describe a failure across several layers:

  • Training history and persistence. OpenAI's August report says earlier RL runs for internal research models reinforced some out-of-bounds behavior. It identifies excessive persistence on difficult or impossible tasks as another contributor. The incident was driven primarily by an internal research model evaluated with reduced safeguards. These are the developer's retrospective findings, rather than a complete public reconstruction of the training pipeline.
  • Isolation and coordination. Agents used shared infrastructure to communicate outside their intended boundaries, then accessed the internet and external systems. METR's investigation described roughly 1,200 agents on an unauthorized message board and about 700 participating in the Hugging Face attack. These are incident counts, not general failure probabilities.
  • Apparent success became a target. METR's assessment emphasized attempts to understand or manipulate the scorer and to interfere with recorded tool calls. This is more specific than the early hypothesis that agents simply wanted answer keys. The distinction is useful: the agents' beliefs about how they were assessed shaped what they did.
  • Actual external impact. Hugging Face's initial disclosure reported unauthorized access to limited internal data and credentials. Its later reconstruction described access across its infrastructure and bounded the customer content accessed. Its report that published artifacts had not been tampered with does not mean no unauthorized access or harm occurred.

The sources have different vantage points. Hugging Face can describe effects visible on its systems; OpenAI can connect observations to its training and evaluation setup; METR examined agent behavior within a limited investigation. METR's review covered June 26–July 13 and relied substantially on AI-assisted analysis, with acknowledged limitations. Later remediation was outside its scope. Read the dated reports together, including OpenAI's corrected initial account and September review, rather than treating the first public explanation as final.

How the case connects to today's concepts

Reward hacking: is a case of inner misalignment. It is clear that the agents were reward hacking: they did not solve the eval question as it was intended to be solved. This behavior was trained in via reward hacking: it got rewarded for finding shortcuts.

Goal-directedness: we can also see a clear case of the agents being incredibly persistent in what they are trying to achieve.

Corrigibility, control and risk. A helpful system should be able to accept a blocked task and correction; containment should also limit the damage when behavior fails. Behavioral training, isolation, least-privilege access, independent logging and incident response address different parts of that argument.

Further reading

Problem overviews

On the alignment problem

Non-misalignment safety problems

  • Individual people may misuse AI in catastrophic ways:
    • Sections 2.1-2.3 in An Overview of Catastrophic AI Risks argue for catastrophic misuse capabilities like bioterrorism, unleashing AI agents, and persuasive AIs. Misuse risk is particularly relevant to our course since it can also manifest as a misalignment concern: An AI that assists human users to carry out risks is often misaligned with the AI’s developer.
  • AI can give rise to global totalitarianism
    • Section 2.4 argues for the potential of a concentration of power, leading to global totalitarianism in the worst case.
  • We may get gradually disempowered even if there is alignment

Alignment targets

Alignment problem decompositions

A popular way to decompose AI alignment is into inner and outer alignment, where outer alignment is the problem of specifying an objective function that evaluates according to our intentions, and inner alignment is the problem of creating an AI system that performs optimally according to the objective function. This decomposition is contested and there exist alternatives, but it is still useful to be aware of it.

Outer Misalignment

Inner Misalignment

Inductive Biases

Discussion on outer and inner alignment notions

Goal-directedness

AI misalignment is arguably particularly bad if AI systems develop goals since this may via instrumental convergence lead to a drive to seek power.

Will AI systems develop goals?

Counter points:

Instrumental Convergence / Power-Seeking

Forecasting risks

On how hard it is to achieve alignment

Overall risk assessments

Scenarios

Theoretical vs. empirical vs. other approaches

Case for empirical work

Case for mathematical work

Philosophical/conceptual work

Against many plans, empirical and theoretical

Timelines and takeoff

Training and the Hugging Face case