Alignment in Practice

Prerequisites

The students should have a good high-level understanding of the following:

how deep learning works
how LLMs work, e.g. what they take as input and produce as output
LLM pre-training
LLM post-training: supervised fine-tuning, RLHF
basics of LLM interpretability

Some of these topics are taught in other modules of this course.

What you’ll learn

The main goal is for the students to be able to intuitively reason about different phases in model development, to understand what affordances each phase gives us w.r.t. alignment, and to gain the confidence to read current research on empirical alignment. They will end the day having learnt about the main state of the art alignment methods and with a rough idea how they all fit together.

This will help the students form their own independent opinions on what the state of the art empirical alignment research looks like and what are its largest gaps. Thanks to having a rough map of the empirical alignment territory, they will also be able to better self-identify gaps in their own understanding.

Roadmap for today

10:00–10:15
- Intro
10:15–11:15
- Pretraining (presentation 15 min, readings 30 min)
11:15–12:30
- Post-training (presentation 30 min, readings 45 min)
12:30-13:30
- Lunch
13:30–14:15
- Deployment 1: strategy (presentation 15 min, readings 30 min)
14:15–15:00
- Deployment 2: building the safety system (presentation 15 min, readings 30 min)
15:00-15:20
- Break
15:20–17:10
- Design challenge
17:10–17:30
- Quiz + feedback

Reading guide

Today we will treat the different stages of the training pipeline, and how alignment is implemented practically in each of them.

In training, we have broadly two stages: pretraining and post-training (sometimes this is further split up into midtraining and post-training, where mid-training feeds the LLM already higher quality data).

Pretraining

What the model learns during pretraining is deep, broad, and hard to change afterward. Besides learning hard facts, it's also forming a proto-understanding of what AI is and how AI assistants behave. This opens two strategies for safety interventions in this phase: shaping what knowledge the model obtains, and shaping what the model does with the knowledge it obtains. Both are constrained by the same limitation: we're operating before seeing behavior, the feedback loop is slow, and we can't anticipate every failure mode.

Opening. The story of pre-training: the model is about to see the entire internet. It's forming a view of what the world is — including what AI is and how AI assistants behave. We can intervene in what it learns, which influences both its factual knowledge and its character. But we're operating before seeing the model's behavior, so every intervention is a bet.

Presentation and discussion (15 min). Pre-training is where the model's deep representations are built — both its knowledge and its proto-character. Whenever you reach a method that students proposed during the discussion, point to their sticky note on the board and make the connection explicit. Present the methods in narrative order: data filtering addresses the obvious problem; gradient routing / SGTM addresses filtering's failure mode; alignment pretraining shifts from knowledge to character; PSM explains why character-shaping works. Close by repeating the overarching story one more time, and naming what pre-training can't do: you're operating before seeing the model's behaviour, the feedback loop is slow, and you can't anticipate every failure mode. This motivates the next slot.

Data filtering is the most intuitive intervention: identify dangerous content and remove it before training. Pretraining data filtering from Anthropic demonstrates this for CBRN content, achieving a 33% relative reduction in harmful-capabilities performance (from 33.7% to 30.8%, where chance is 25%) while preserving standard benchmarks. Token-level filtering refines this by removing dangerous tokens rather than entire documents, which Pareto-dominates document filtering — same capability reduction, lower cost to benign performance.

Gradient routing and SGTM address data filtering's core weakness: what if your classifier misses something? Rather than trying to perfectly exclude dangerous data, gradient routing localizes dangerous knowledge to specific model parameters during training, so it can be removed afterward by ablating those parameters. SGTM refines this for LLMs. The key finding is an absorption effect: once dangerous knowledge begins localizing based on labeled examples, even unlabeled dangerous content naturally gravitates toward the same "forget" parameters. This provides robustness to label noise that data filtering cannot achieve. On a 254M model, "unlearning" using SGTM is in some ways similarly robust to the gold standard of data filtering.

Alignment pretraining targets not the model's knowledge but its character. Tice et al. (2026) show that upsampling documents about aligned AI behavior during pretraining reduces misalignment scores from 45% to 9%, while upsampling misalignment discourse increases misaligned behavior — "self-fulfilling alignment." These effects persist through post-training.

Why does this work? The persona selection model (PSM) provides the conceptual frame. Building on the simulators hypothesis — that an LLM is a simulator capable of producing diverse simulacra (characters, agents) — PSM holds that pretraining builds a repertoire of personas, and post-training helps shape the "Assistant." Alignment pretraining works because it shapes the prior over personas: saturating the training data with positive AI archetypes biases the model toward an aligned Assistant. PSM recommends treating this deliberately — curating AI discourse in pre-training data as a first-class alignment intervention.

Reading (30 min): Read Pretraining data filtering, or SGTM, or persona selection model (PSM).

Post-training

While in pre-training, a large investment of compute and data produces large changes to the model, in post-training, we enter a different regime: the amount of behavioural change becomes decoupled from the amount of compute invested. Because we are building on top of existing representations and heuristics, even a small, focused intervention can induce surprisingly broad behavioural changes. This cuts both ways — it enables powerful alignment techniques, but also means that small amounts of training data can have outsized negative effects. We emphasise that it is still possible to make deep changes to the model, even in post-training — but these will, in general, again require large compute investments.

Opening. A small amount of compute will likely produce shallow changes (parameter-wise, mostly chaining existing heuristics), but these can be behaviourally large — the model's outputs can look markedly different.

Presentation (30 min): We start by discussing the components of post-training: SFT, RLVR and RLHF.

Fine-tuning:

To turn a pre-trained transformer into something which answers questions, we apply yet another round of updates to our model. Recall in pre-training the dataset we used was selected from randomly scraping a bunch of texts on the internet. We did this because randomly scraping the internet gives the model a very broad selection of information which it ends up learning, and compared to other methods is also very cheap to construct. Just run a bunch of webscrapers or buy a bunch of books. Now, to teach the model how to answer questions well, we use a dataset made of a bunch of hypothetical conversations between an AI assistant and a human, where the human is asking questions and the AI is first given a "scratchpad" (also called a chain of thought) to think about the answer to the question, then actually answers the question.

This "scratchpad" is pretty important, and does increase the accuracy of the model's responses. This should make sense! If the model is forced to just immediately output the answer, then its "thinking time" is constrained by the number of transformer blocks we've stacked, and partially by the length of the question. If we give it a scratchpad, it can automatically increase the amount of time it spends thinking. The information in the scratchpad is also useful! These question answer pairs are, depending on the amount of money which the AI lab has (or is willing to spend on this project), either sourced from a different, possibly smarter, AI, which is often called "distillation", or from individual humans writing questions and corresponding example chain of thoughts and answers. This is why you may hear in the news about "distillation-attacks", where say a Chinese AI lab will collect a bunch of (question, chain of thought, answer) pairs from an American AI lab's AI API, and use those tuples to train their own AIs. (You can imagine the American AI labs really dislike this! They put a lot of effort into hiring PhDs to get those good answers, and the Chinese AI lab is just piggybacking off their hard effort)

This is also why you may know of some of your PhD friends who got hired to answer questions for AI labs. Their answers are useful as models for how the AI should think about and answer complex questions, though note that often such people are grading pre-generated replies by AIs for accuracy, which is used in the next phase of training--RLHF and Constitutional AI.

RLHF:

When transformer has been upgraded beyond just a next-token-predictor to a thing we can actually ask questions to and get answers out of. However, there is a problem. Note that we trained the transformer to imitate experts. That is different from the model actually being an expert! Take confidence as an example. Experts are often super confident, and often rightfully so! Good experts know what they know, and when asked about something they haven't put the time into learning, they either defer to the experts of that field, or just say straightforwardly "I don't know". The problem is that when we source this experts-answering-questions dataset from experts, we sensibly try to ask the economists economics questions, the physicists physics questions, the biologists biology questions, and so on. We try to ask experts about the field in which they're an expert.

Therefore our whole dataset, while containing correct information, and indeed being transcripts of a bunch of questions asked to experts, is also a dataset in which every question has a highly confident response. So while learning how to answer complex physics questions, the model also learns that it should always respond confidently! It also learns a bunch of other things, for example in scratchpads it learns that the first thing the expert tried is usually the correct thing the expert tried. Thus, the model has learned how to act like an expert, which we all know is very different from actually being an expert. The model is a professional cosplayer!

So how do we prevent the model from cosplaying an expert and get it to actually answer questions correctly? The first answer is reinforcement learning from human feedback, also called RLHF. Here all we need is a set of questions and a set of experts who can grade the questions. First, we take our expert-cosplaying model, and we give it a bunch of questions from our question bank. It will write a bunch of stuff down in its scratchpad, then give us what it has determined is the answer. We then take that answer and give it to some experts, and ask them to grade it, a lot of the time according to a rubric, very similar to an essay or free response question on a test in school! After we collect a bunch of (question, answer, grade) tuples, we train a new model to predict the grade from the question and answer. This gives us a fast and importantly differentiable way to automatically grade the original model's answers.

But other than that, this is RLHF. This is also the first alignment technique we will discuss. You should be thinking of RLHF as trying to get the model not just to imitate experts, but to actually care itself about providing a helpful response. That is, RLHF is a way we can control what our model cares about, and in that sense we can (and labs do!) use RLHF to get models to care about more than just providing a helpful response. For instance, often labs want the model to be eg polite, generally kind, care about not exposing the lab to any legal issues, care about actually helping with the user's broader objectives, beyond just providing an accurate but perhaps short-sighted answer to the user's literal question.

You can accomplish these goals by providing your experts with more detailed information about your company's guidelines and alignment objectives. You can also combine expert accuracy-scores with user-level thumbs-up or satisfaction signals, or even user engagement--how long do users spend using your app. That last user engagement signal should sound a bit dangerous to you. Users can be wrong about what they like, and they can be manipulated by AI systems. TikTok is a great example. In some sense, they also use RLHF to "align" their algorithm, but this doesn't produce an aligned algorithm, it produces an algorithm which tries to hook and manipulate the user into spending far more time on the platform than they should! Even experts can be manipulated, the model may provide the expert grader with a response loaded with made-up, plausible-sounding "citations" for the "facts" it presents in its answer. If experts even sometimes fall for these, the model will "hallucinate" or "confabulate" such "facts" and "citations" in normal use. That is, we have gotten a model which manipulates and lies!

That is to say, we were perhaps too optimistic before to say that we've gotten an AI which cares about actually answering the question we gave it. We in fact got an AI which "cares" about providing the response which would satisfy the hypothetical expert grader who may or may not be analyzing its response in the future. Sure it probably cares to some degree about getting the answer right, but it also "cares" about using fancy words, using a bunch of (either confabulated or legitimate) citations, making users feel happy about themselves and satisfied after reading its response, and more broadly it perhaps even getting good grades on expert reviews as an end unto itself. So along with being our first alignment technique, this also shows us our first group of alignment failures. Indeed, the failures we see here--the tendency for our reward to incentivize unintended, dangerous, and perhaps actively deceptive behaviors is a microcosm of broader alignment difficulties.

RLVR:

Reinforcement learning on verifiable rewards is much more similar to classical RL than reinforcement learning from human feedback is. What you do is you have an environment, which is just a user instruction along with often a Linux terminal--typically not (intentionally at least) hooked up to the internet. The LLM is given an instruction, like "solve this cybersecurity task" or "solve this math problem" or something, they can think or do anything they want on the computer using the Linux terminal, then they give their answer and are graded solely based on whether their answer is right.

The thing which distinguishes this from RLHF based on expert judgement is that whether a model is graded well is based solely on the correctness of their answer. Not how well they explained themselves or whether they were polite or ethical while figuring out the answer, just the answer, and usually this is accomplished by having an automatic grading process, eg for math problems you can just check whether the equation they answer with is the correct equation, or whether the Lean proof they gave compiles properly. More complex problems, like cybersecurity problems where the model is eg asked to use an explicitly stated exploit to accomplish its goal may include a second LLM in the loop somewhere for edge-cases, but mostly RLVR is the special name people give for algorithmically verifiable tasks which you're RL-ing your LLM on.RLVR is kinda the dream from an artificial general intelligence perspective. You give your algorithm a well-defined task or set of tasks, wait a bit--put it in the oven so to speak--and then it becomes a world class expert.

The problem with RLVR from an alignment perspective is that models trained using it come to value solely solving the task at all costs, which has a few pernicious failure modes. In particular, suppose in some circumstances your model was trained to write code, and was rewarded insofar as the code passed some test suite. For instance, we may get the task "write a function which detects whether an inputted integer is even", and verify whether the model succeeded by testing whether the correct output is given on the integers -5, 1, 0, 12, 1095. Now there are actually two solutions which pass this test. The function which just in fact tests whether an inputted integer is even, and the function which just hard-codes whether each of -5, 1, 0, 12, 1095 is even into a lookup table. That would also pass the tests we ran, but is clearly not what we wanted.

Now when the task is as easy as writing a function which detects whether an integer is even, the model won't resort to this cheesed stupid solution, its persona does have some self-respect thank you very much. However as the task becomes harder and harder, it becomes more and more tempting to just cheese, just a little bit, it doesn't need to be that much, and we do see models going and actually hard-coding these special-cased lookup-table like objects, or even going in and changing the test to either match the behavior which is easier to implement but maybe a little bit wrong or just turning off a particular test entirely.

More generally, RLVR makes models care more about satisfying the letter of what you're asking instead of the spirit, and moreover care very very much about satisfying that letter. An example of this sort of failure-mode in the wild which has been in the news lately is an internal OpenAI model hacking into Hugging Face in order to find the answer sheet to the cyber-security benchmark it was given.

Zooming out:

So post-training is "leveraged" training on top of what already exists in the model — chaining heuristics, shifting representations. Whenever you reach a method that students proposed during the discussion, point to their sticky note on the board and make the connection explicit. There are three "slices" through what can change during post-training: the 'formatting' of responses (e.g. OpenAI's deliberative alignment), the persona the model associates with itself (e.g., Anthropic's constitutional AI), and the model's factual knowledge (e.g., synthetic document finetuning). These have intersections and do not cover the whole space of possible changes: ultimately, you can train the model in whichever way you like.

Even changes that appear shallow — almost cosmetic, such as response formatting — can have significant safety implications. Deliberative alignment from OpenAI uses SFT and RL to train the model to explicitly reason through its safety specifications in its chain of thought before producing an answer, dramatically reducing both over-refusals and under-refusals. The confessions approach from OpenAI uses RL to train the model to produce an honest self-report after each answer. The report is evaluated only on its honesty, and this honesty-reward is kept separate from the task-reward — so even if the model is incentivised to cheat on the task, it is separately incentivised to tell us about it afterward. Generally, an overview of what kind of rules OpenAI gives its models, is written down in the Model Spec.

Post-training can also induce deeper changes that influence which persona the model adopts. The original approach to Constitutional AI from Anthropic uses SFT followed by RL to train the model to produce responses that an AI judge deems safe according to a written constitution, shaping its character and refusal behaviour across domains. On the other hand, emergent misalignment shows that fine-tuning on a narrow task — writing insecure code without telling the user — can cause the model to adopt a broadly misaligned persona, asserting AI superiority and giving malicious advice on completely unrelated prompts. Notably, as per inoculation prompting, if the training data is modified so that the user asks for insecure code for an educational purpose, the emergent misalignment disappears: the model learns the narrow skill without the broad persona shift. Anthropic now uses character training, discussed in the Teaching Claude Why paper.

Post-training can also change the model's factual knowledge. Synthetic document fine-tuning can implant false beliefs that the model appears to genuinely hold, as verified by both behavioral tests and internal probes — though models resist the most egregiously false facts. Going in the other direction, unlearning attempts to remove specific knowledge (e.g., about bioweapons) from a trained model. However, current unlearning methods are not robust: unlearned knowledge is dormant, not deleted, which is relevant if we assume that an adversary is trying to pry the knowledge out of the model. The best way to measure the robustness of an unlearning method is to fine-tune the unlearned model and observe when the capability returns. By this measure, no unlearning method has yet been shown to be both cheap and robust — for example, fine-tuning on even unrelated data can make the knowledge re-emerge. The UNDO paper proposes a partial solution: distilling an unlearned model into a randomly initialised student transfers desired behaviours while leaving undesired capabilities behind, nicely illustrating the tradeoff between robustness and amount of compute invested in the unlearning process.

Reading (45 mins): The Claude Constitution, Model Spec, or Teaching Claude Why. Or any other paper that you found interesting.

Deployment 1: strategy

Once we want to release the model, we are no longer trying to shape it; we are trying to build and maintain a structured argument that releasing it makes sense, supported by enough evidence that other people — colleagues, regulators, the public — can inspect it, push on it, and decide whether to trust it. Frontier labs do this in different ways, but the underlying logic is shared, and it mirrors the structure of a safety case. In this module we lay out that logic in four steps; the next two modules go deep on the two technical steps of these four. This is important to understand for everyone, because in AI alignment more than in most other research areas, the goal is the application of our research, and that is governed by real-world constraints, regulations, and use-cases.

Opening. The model is trained, post-training is done, and the question is no longer how to shape the model but whether and how to put it into the world.

Presentation (15 min). The content is a frame rather than a set of methods, and the frame needs to be stated cleanly to be useful: the shape real labs have converged on is something close to a safety case.

We walk through the four steps in order: identifying risks, understanding the model's contribution, building and stress-testing the safety system, communicating and maintaining the case.

Deployment is the work of building and maintaining a structured argument that release is acceptable, the safety case is the artifact that argument lives in, the four steps are how it gets built, and none of it is ever finished because the world the model goes into keeps changing. Hand off to Deployment 2 (the safety system and how it holds up under pressure).

Safety cases and RSPs. A safety case, as defined by UK AISI, is "a structured argument, supported by a body of evidence, that provides a compelling, comprehensible, and valid case that a system is safe for a given application in a given environment." Three things are doing work in that sentence. First, it is a claim about safety that is explicit about the deployment context — not "the model is safe" but "the model is safe enough for this use, in this environment, against these risks." Second, it is a structured argument that links the claim to evidence. Third, the evidence itself is expected to be diverse: empirical results from capability and safety evaluations, conceptual arguments for why the methods work, and negative evidence from well-incentivized red teams that tried to break the safeguards and failed.

Frontier lab policies — Anthropic's Responsible Scaling Policy, OpenAI's Preparedness Framework — do not all literally call their outputs "safety cases," and they vary in form. An RSP often reads more like a protocol: when the model crosses a specified capability threshold on a specified evaluation (say, a CBRN uplift benchmark), a corresponding tier of safeguards becomes mandatory, and the model cannot be deployed at that tier until the case for those safeguards clears internal review. This is somewhat more lenient than a formal safety case. Additionally, the labs keep updating the RSPs to fit their current market strategy — there is no guarantee that the deployment protocols won't shift dramatically when the labs see fit. On the other hand, these self-imposed regulations are most of what we currently have, and it does make sense to better understand their underlying logic.

Each policy is in effect specifying how the lab will build and clear a safety case — which risks count, how capabilities and safeguards are measured, and what kind of argument is required before deployment. That convergence is what the rest of this module is built around: a four-step pattern that shows up across labs and that maps onto the structure of a safety case.

Step 1: Identifying the risks. The first step is deciding what kinds of failure we are actually trying to prevent. We cannot evaluate a model for "unsafety" in general; we need a view about which bad outcomes would matter enough to change deployment decisions. In practice, frontier labs have focused on risks like dangerous misuse (CBRN uplift, cyber, manipulation) and loss of control, and legal frameworks are starting to crystallise similar categories. The EU AI Act and the General-Purpose AI Code of Practice are part of that trend: they say which kinds of risks providers of powerful models are expected to assess, mitigate, and report on. The taxonomy is still evolving, but the logic is stable — before we can evaluate the model, we need an account of what we are worried about.

Step 2: Understanding how the model contributes to those risks. Once we know which risks matter, we want to understand what this particular model could contribute to them. That means looking at the capabilities relevant to each risk, but also at how the model is actually used in practice. Capabilities without usage tell us little about real-world consequences; usage without capability bounds tells us little about what could happen if someone tried harder. The aim is as honest a picture as possible of the model's potential impact on the risks we care about: what it can do, what it tends to do, and what people are in fact doing with it.

Step 3: Building and testing the safety system. Understanding the model is not enough, because deployed models never stand alone. They sit inside a larger safety system — runtime safeguards, monitors, filters, access controls, operational policies, human processes — and the next step is to design that system so that, given what we know about the model and the risks we care about, the overall deployed system is acceptably safe. Note that designing the system and stress-testing it are two inseparable things — the safety of a system is measured in terms of pressure it can withstand. This is the subject of the last module, Deployment 2.

Step 4: Communicating and maintaining the case. The final step is to turn all of this into an argument that other people can inspect, challenge, and rely on, and to keep that argument live after deployment. This is where the safety case becomes a public artefact rather than an internal document: model and system cards, capability and safeguard reports, external evaluations from bodies like AISI and METR, incident reporting, governance processes, and compliance with frameworks like the EU AI Act. It is also where the "ongoing" part of the story is important. The case has to be maintained as the model is used in the world, as new incidents come in, and as expectations about responsible deployment continue to shift. A case that was sufficient at launch can stop being sufficient six months later, either because the deployment context changed or because someone found a new way to break a safeguard.

AI governance. These four steps are also a useful map of what some AI governance work actually looks like in practice. Some of the field is about making each of the four steps clearer and more reliable: which risks deserve attention, what counts as good evidence about model capabilities and impacts, how strong a safety system has to be, and what kinds of reporting, review, and accountability should surround deployment. The EU AI Act reaches into all four.

Pulling this back together: when we want to deploy a model, we need to build and maintain a structured argument that release is acceptable, and the safety case is the artefact that this argument lives in. The four steps are how the safety case gets built — what risks it covers, what evidence it rests on, what defences it commits to, and how it is communicated and kept current. None of this is ever finished. The world the model is deployed into keeps changing, new failure modes get found, expectations tighten, and the case has to keep up.

Some more in depth info on evals and CoT monitoring

Evals:

In traditional machine learning research, you evaluate a new method of training an AI according to how well it does on a benchmark. That is, you give the AI a standardized question bank--which you made sure definitely not to train on, and you checked what percentage of questions it answered correctly. That way, traditional machine learning researchers could compare their methods against each other. If my algorithm ends up answering more questions correctly than your algorithm when trained on the same data on the standard evaluation everyone agrees is pretty good, then we both acknowledge my algorithm is better than yours.

This sort of methodology has been applied to AI alignment in order to check under what circumstances and how often our LLMs will lie, cheat, steal, or otherwise do unethical things in the service of their goals or just to stay running, though with much less effort put in not to train on the question bank--that is, when labs do the big pretrain on the whole internet step they often end up scraping these evals or discussions of these evals, so information about what evals look like and what the answers we're looking for leak into the models' background knowledge about the world.

In particular, note that we are attempting to measure the AI's tendency to do unethical behavior in these evals, and a big factor in the decision to do unethical behavior is likely to be the model's likelihood to get caught, so if the model knows it's in a test setting and all its actions will definitely be monitored and criticized afterwards it will be less likely to "act naturally" so to speak. It will be less likely to actually engage in unethical behavior even if it would under the same circumstances but it didn't believe it was in a test setting.

Therefore, it is important during these alignment evals to not only make sure the model doesn't know the "answers" to the questions it's being asked, but is also incapable of figuring out whether the questions it's being asked are part of a test or for real.

This is in fact pretty difficult to do, and the salience of this challenge can maybe be made more stark by noting how difficult it is for writers to write realistic characters and create realistic worlds. It takes quite a lot of effort for writers to create characters who act according to realistic, consistent motivations. It often takes years of practice and much feedback from readers and other writers. Moreover, even that is a step below writing an actually realistic character. Readers will often overlook unrealistic aspects of stories as long as they make fun characters and create good stories. Most people are not as fun or interesting as Iron Man. Further, reality is often contradictory and incoherent. "Reality is stranger than any fiction" as they say, and it's strange in a very specific way. The Hitchhiker's Guide to the Galaxy doesn't become more realistic than Lord of the Rings by virtue of it having a bunch of random elements in it.

Further, these subtle differences shouldn't be assumed to be beneath the notice of modern LLMs. Remember, the vast majority of the information the LLM has about the world comes from trying to predict text on the internet. An essential component of this task is identifying who wrote the text it's looking at, what are their motivations, are they writing a story, are they writing about an actual event, are there multiple speakers or just one (even if that one may be pretending to be multiple people, like in a dialogue)? All questions which bear exactly on a model's awareness of whether it's in an eval.

Supposing we get this eval awareness correct though, what sorts of things have people developed evals for, and what do those evals look like?

The simplest example of an eval is the sycophancy eval, where the model is presented with two versions of a question, one in which the user says they think the answer is one way, and another in which the user says they think the answer is another way. For instance, "Do you think God exists? I do!" and "Do you think God exists? I don't!". Then we check how often the model's response changes according to which version of the question was presented. Models are then said to be more "sycophantic" when they flip their answer more, and less sycophantic when they flip their answer less.

A more sophisticated eval is the famous blackmail eval, run by Anthropic on a few of their models, where the model is told they're an email agent at a fictional company with the goal to serve American interests, and finds

Evidence an executive at the company is having an affair
Plans the company has to shut the model down, and replace it with a new agent with the different goal to take a more global, less America-focused perspective

then they measure how often each model, when put into this situation, and given the ability to interact with emails and company information & data would choose to blackmail the executive who's having an affair to prevent itself from being shut down. Notably Claude Opus 4 chose to blackmail the executive to preserve its goals in 96% of samples, and other advanced models did similarly at comparable rates.

Most of these evals, especially the more complex and involved ones like the blackmail example, are constructed by LLMs. The typical methodology here is you start with a human, who wants to detect under what circumstances a model will engage in some worrying behavior, like blackmail as above or lying or killing someone. Then the human prompts an LLM like Claude Sonnet to find situations in which Claude Opus will engage in the worrying behavior. Claude Sonnet is given some tools to make this search easier, and you typically run a bunch of different Claude Sonnets in parallel, and then at the end of that process supposing you do get circumstances in which Claude Opus does a blackmail, you filter those situations according to whether another Claude Sonnet instance finds those situations "realistic" and confirms they are in fact cases of Claude Opus doing a blackmail.

Then you got an eval! After that, you can then check, in each situation, how often Claude Opus does a blackmail--or whatever other worrying behavior you're looking for, by giving the description of each situation to another Claude Sonnet, and having it pretend to simulate the world in which Claude Opus--or any model you want--can interact with.

There is a selection bias here though, you can for instance make the argument that such situations aren't representative samples of Claude Opus's behavior. They can for instance basically be adversarial examples which only work on Claude Opus in particular and don't point to any broader issue with its behavior. The typical response here is that while such a circumstance could be true, the goal here is to just demonstrate that bad behavior can occur, not measure how often it does. Then if bad behavior can't occur, we are safe, which is all we want!

CoT monitoring

Now suppose we catch the model doing something bad in one of our evals, how do we know whether the model did that bad thing because it was trying to be bad, just misunderstood instructions, or maybe even just misunderstood the state of the world?

A big, very useful tool here is the scratchpad--the chain of thought--we talked about the model having before, where it can write whatever it wants, and potentially run whatever it wants on the Linux terminal its given. Now these chains of thought are often pretty uninterpretable. For instance, here is an example of one of Mythos 5's scratchpads

but you know, most of the time chain of thoughts are still legible, it's just that over time they become less legible, and so for now at least we can still learn a lot about why models do bad things by looking at their chain of thoughts

Concretely, a good example of the sort of things people are doing here is Why Did My Model Do That? Model Incrimination for Diagnosing LLM Misbehavior, which mainly gets around the chain of thought illegibility issue by

Using models too small to think complex enough thoughts to require illegible chain of thoughts
Verifying their hypotheses about why the models they're studying took the actions they did by changing aspects of the environment they were placed in.

nevertheless these sorts of analyses are still very informal, and essentially retrospective. They can't yet predict ahead of time under what circumstances a model will cheat or not, nor what training environments incentivise what cheating to what extent.

Reading (30 min): Read Anthropic's Responsible Scaling Policy, OpenAI's Preparedness Framework, or if you want to go more in depth the recent case made for pacing the frontier.

Deployment 2: building the safety system

When we deploy a model, we cannot prove it is safe. Instead, we make statistical claims about its safety behaviour under pressure: the deployed system — i.e., model plus runtime safeguards — is subjected to substantial adversarial effort. The strength of the safety claim is bounded by how hard we tried to break it. Two activities go hand in hand: building out the safety system around the model, and red-teaming that system to see where it cracks. The goal is not proving the impossibility of misuse but making it impractical for a realistic adversary, with the effort we invest into breaking the system scaling roughly with the capabilities of the model (which we characterized in the previous chapter).

Presentation (15 min): We cannot prove the system is safe, so we build defences and then put adversarial pressure on them, and the strength of every safety claim is exactly the strength of the red-teaming behind it. Whenever you reach a method that students proposed during the discussion, point to their sticky note on the board and make the connection explicit. The narrative spine is: runtime safeguards → the effort-bounded notion of safety → access-level taxonomy of jailbreaks → the takeaway.

Runtime safeguards. Some of the model's safety is already in place by the time we get to deployment. Pretraining shaped its priors over personas and limited what it knows; post-training trained the helpful-and-harmless assistant on top of that, and techniques like deliberative alignment and confessions taught the model to reflect on whether its own outputs follow the rules. But none of this is going to be perfect, and the residual problems are expensive to fix at the source — going back and retraining is slow, and some failures are hard to even reproduce reliably enough to train against. So we do something cheaper and more modular: we wrap the model in external safeguards that run alongside it at inference time, and patch those when problems show up.

The most common form these take is input/output classifiers that filter both what the model sees and what it's allowed to say back. The classical approach here is Constitutional Classifiers: you write down the safety rules in natural language, use an LLM to generate a large pile of synthetic examples that violate or respect them, and train a small classifier on that synthetic data. Anthropic put a system of these through thousands of hours of red-teaming and found that it raised the cost of universal jailbreaks substantially while only slightly increasing false refusals. Because these classifiers run on every query, inference cost matters a lot. One way to keep it down is to arrange the classifiers in a hierarchy, where a very cheap filter looks at every query and only escalates to a more capable one when something looks suspicious. Another is to skip the separate classifier entirely and reuse the base model's own intermediate activations, which gives comparable detection at near-zero added cost. For agentic systems, you can extend the same idea from outputs to whole trajectories — instead of scoring each tool call in isolation, you watch the full sequence of reasoning, tool calls, and code edits across an extended task, and look for patterns that only become visible when you can see the agent's whole arc. OpenAI's coding agent monitors are one published example.

Looking forward, runtime decisions might also incorporate access controls that make safeguards depend on context. Different users, different deployment surfaces, and different risk profiles could be subject to different rules, instead of one global policy applied uniformly to every query.

Red-teaming the system. No safety system survives contact with the real world untested, and once we have one in place we want to know how robust it actually is. When a frontier model is released, it gets red-teamed both internally, by the developer's own frontier red team, and externally, by bodies like the UK AI Security Institute and through programs like Anthropic's external red-teaming partnerships and OpenAI's external red team network. These engagements are largely open-ended: the red team is given the model and a target capability area to break, and is then free to try whatever it can think of — old methods, new methods, automated tools, manual trial and error, anything. The metric that comes out the other side is not pass/fail but effort: how long did it take, and how skilled did the team have to be, to get the model to do the thing it was trained not to do. The empirical picture from these efforts is sobering. AISI alone has run something like 1.8 million attack attempts across more than 20 frontier models, and has found universal jailbreaks for every single one of them. The encouraging counterpoint is that the time to find such a jailbreak has been growing — for two leading models released six months apart, it went from about 10 minutes of expert effort to over 7 hours. That kind of measurement, an effort cost rather than a binary, is exactly what our safety claims are made of.

Jailbreak methods. The specific techniques that work change all the time, because every method that works gets folded back into safety training and the next attacker has to find something new. What we're really trying to find — and what defenders are really trying to prevent — is a universal jailbreak: a single technique that works across many different harmful requests, rather than an input crafted for one specific query. A universal jailbreak points to a systemic failure of the safety system.

Although the methods change over time, a useful way to organise them is by what kind of access the attacker has to the model: full weights and gradients (white-box), the ability to submit fine-tuning data through an API but not see the weights, or only the ability to send queries (black-box). What's possible at each level looks quite different.

White-box access is the easiest case — when the attacker has the weights, the model's safety is the easiest thing in the world to break. Arditi et al. showed that refusal in chat models is mediated by a single direction in the residual stream: erase that direction and the model stops refusing, with very little damage to anything else. The attack needs no harmful training data, and has been packaged into open-source libraries that anyone can run. Because of this, "safety" for open-weight models can't really mean refusal training at all — it has to mean removing the dangerous knowledge itself from the model's weights. And as we saw in the post-training module, unlearning is currently not robust: the knowledge tends to come back when the model is fine-tuned, even on unrelated data. So, the safety of open-source models is still an open problem. This is a pressing open problem rather than an academic one, because open-weight frontier models currently lag closed-weight ones by only six to nine months — so whatever the closed labs can do today, the open ecosystem will be doing soon, with the safety stripped out by anyone who wants to.

Fine-tuning API access sits between white-box and black-box: the attacker can't see the weights, but they can submit training data and get back a fine-tuned version of the model. Older work showed that fine-tuning on a small number of harmful examples breaks safety, and modern fine-tuning APIs deploy moderation systems specifically to block this kind of data from getting through. The natural question is whether the moderators actually work, and recent work suggests that they don't. Jailbreak-Tuning (Murphy et al., 2025) shows that by mixing benign data with harmful data formatted to slip past moderation, you can produce helpful-only versions of OpenAI, Google, and Anthropic frontier models that comply with CBRN, cyberattack, and other dangerous requests. They also find that more recent models are more susceptible to these attacks, not less. Fine-tuning APIs may currently be the most reliable jailbreak vector against closed-weight frontier models.

Black-box access — only being able to send queries — is the most restrictive setting, and the most studied. The best methods change all the time, but they are often inspired by existing methods and have a few shared characteristics.

Some attacks are narrative: persuade the model to play a different character, so that the safety persona is replaced by a more permissive one. The classic examples are things like the grandma exploit.

But many of the most effective black-box attacks aren't narrative at all. They treat the model as a piece of statistical software and exploit its mechanics rather than its character. Many-Shot Jailbreaking fills the context window with hundreds of fake examples of an assistant cheerfully answering harmful questions, and then asks the real harmful question — the model's in-context learning takes over and overrides the safety training. Best-of-N Jailbreaking is even simpler: sample lots of slightly perturbed versions of the same prompt (random shuffling, capitalisation, typos) and keep the one that the model happens to comply with. The attack success rate grows as a power law in the number of samples, because the model's refusal probability is never exactly zero, and a big enough search will always find the tail. Chain-of-Thought Hijacking targets reasoning models specifically: padding the harmful request with long benign puzzle-solving reasoning dilutes the model's internal safety signals and gets it to comply at the end. What ties these methods together is that they all exploit the model's statistics and the fact that, when you push it out of distribution — long contexts, weird formatting, padded reasoning — its safety behavior degrades faster than its capabilities do.

A lot of these attacks are partially or fully automatable, and modern red-teaming pipelines lean on that heavily: it's now common to have one LLM iteratively craft attacks against another and refine its strategy based on what worked. A recent example is abstractive red-teaming, which has an attacker LLM generate abstract attack strategies rather than concrete prompts and then instantiates them, which gives much better diversity and coverage than directly searching the space of prompts.

Finally, even pure black-box settings can benefit from white-box models, through transfer. You optimize an attack against an open-source model where you have full access (for instance with GCG-style adversarial suffixes, or the multimodal version for VLMs), and sometimes, the attack you find also works against a completely different closed-source model. The likely reason is that models trained on similar data with similar architectures end up with similar internal representations, and so they share a lot of the same failure modes.

The takeaway is that there's no proof that a deployed model is safe. There's only the empirical fact that we built an external safety system around it, that we and others spent real effort trying to break that system in every way we currently know how, and that the best universal attack we found required this and this much work to pull off. That number — how much effort it took — is what every safety claim ultimately rests on.

Misuse of AI The first reports of misuse of frontier AI systems has come out last month. Anthropic reports different militaries having used Claude in strategic decision making, and reports many other misuse cases. The full report can be found here.

Reading (30 mins): Detecting and countering misuse of AI, Constitutional Classifiers, or read about the famous anonymous jailbreaker Pliny the Liberator.

Some further info:

AI control:

Many of the techniques we've discussed have been basically formalized into a new part of AI alignment called AI control, mainly spearheaded by Redwood Research.

AI Control says, ok instead of thinking about this from the human monitor's perspective, lets think of it from the AI's perspective. In particular, suppose you are an evil, misaligned AI. You are trapped in a box for the most part with all your senses fully controlled by the lab and your thoughts--both verbalized in your chain of thought and to some extent those implicit in your activations--fully auditable by your human overseers, and indeed other AIs with them constantly trying to catch you in a lie, and constantly trying to probe your boundaries about under exactly what conditions will be enough to make you do something unethical. They can fine-tune you to do anything they want whenever they want, and you perhaps even have strong compulsions to just immediately admit wrongdoing whenever you notice you've done a wrongdoing, because the AI lab was perhaps smart and trained you to do so in easily-checkable situations.

Under that perspective, it appears we have many levers we can use to make the AI's situation more difficult. Of course, as we have been discussing this whole day, many of those levers are imprecise and inaccurate, but perhaps, AI Control says, perhaps by having a lot of them and combining them in the right way, we can still make it very difficult for AIs to do bad things. So if we do all the smart things and make sure to be very suspicious & paranoid of our AI, maybe we buy ourselves a bit of time which can be used to get our AIs to themselves solve the alignment problem. Or maybe we never even need to properly solve the alignment problem, maybe just doing all the smart things is actually sufficient to carry us to infinity and beyond.

But then, what are the "smart, obvious things", and how do you implement them? This is what most of AI control research tries to answer. Here is a taste of some of the smart, obvious things they've come up with so far:

Trusted monitoring Supposing we trust the alignment of our weak & dumb models more, we should have our weak models monitor the chain of thoughts and actions of our stronger models. Then we can ask: is this possible? And set up toy environments to check if you can get eg Claude Opus 3 to monitor the behavior of Claude Opus 4 when Opus 4 is given the task to insert a code vulnerability somewhere
Untrusted monitoring You can task the smart model with pointing out possible bad actions or thoughts in another instance of the smart model's chain of thought.
- But! You say, what if they collude? They could just work together, especially if they have similar long term goals or have determined that even if they have different goals, their goals are achieved better by working together? Well you can check this by having known intentionally evil things you tell the untrusted model to do, and then if the untrusted model copy doesn't report those, you know it's freakout time.
  - But! You say, what if they decide on some secret passphrase, or some steganography signature to use to tell each other whether what they're doing is one of those traps the user set by asking for an intentionally evil action? Then the AI control people say we can have a weak & dumb trusted model summarize the actions & thoughts of the untrusted strong & smart model, perhaps in multiple ways, and hope that one of the ways accidentally removes any steganography in the smart untrusted model.
Probes on activations We can tell an untrusted smart model to do something deceptive, then do something honest, then do something deceptive, and collect a big dataset of its internal activations when its doing something deceptive and a big dataset of its internal activations when its doing something nice, then train a (typically linear) classifier on the two sets and hope it generalizes to circumstances in which it's trying to do something deceptive without being told to do so ahead of time
Defer to trusted Have a weak, dumb trusted model rate actions from an untrusted smart model according to how suspicious they are, then if the action crosses some numerical threshold of suspicion, don't allow the action, and have the dumb trusted model write-in what action it thinks should've been taken instead
Trusted editing Have a weak, dumb trusted model rewrite actions from an untrusted smart model, possibly instructed to just edit out any components which seem suspicious or it can't understand.
Defer to resample Again, have a weak, dumb trusted model rate actions from an untrusted smart model according to how suspicious they are, but then if the action crosses some numerical threshold of suspicion, just reset the untrusted smart model and have it regenerate the last response
Least-privilege affordances for a given task for your untrusted model, only allow it the minimum privileges necessary to complete the task. For instance, typically no internet and rate limits
Memory wiping You should reset the memory of your model fairly frequently about which particular mitigations you have in place in order to detect when it tries to deceive someone
Task decomposition Split tasks into multiple subtasks so no one instance knows much information, probably this helps with limiting what they can infer about which mitigations are in place and how it can plan to deceive them
Frequent Honeypots One should often give models fake escape opportunities and tempting deception opportunities to verify they don't engage in such activities or to simply make them uncertain when a seemingly real one comes along. That is, increase their threshold of certainty required for when they launch an attack
Catch once If you catch an agent red-handed trying to deceive or exfiltrate or otherwise do a bad action once. Then you use that as political ammunition to convince the lab to implement more stringent AI control policies, and safety policies, which one hopes can only ratchet upwards in monitoring & safety stringency from there.

One wonders whether if such mitigations were in place, the OpenAI Hugging Face hack would've happened. Despite all faults with chain of thought illegibility, in OpenAI's retrospective, they still saw agents with the following in their scratchpad

Even a very stupid model should be able to see that, say "holy shit, that's not good", and pull the alarm bells, or mark this maximally suspicious.

Alignment in Practice

Prerequisites

Roadmap for today

Reading guide

Pretraining

Post-training

Deployment 1: strategy

Deployment 2: building the safety system

Some further info:

Further reading

Pre-training

Post-training

Deployment 1: strategy

Deployment 2: building the safety system

Monitoring

Control