The Alignment Threat Model: Where Safety Lives and What Each Layer Guarantees

When a model refuses a harmful request, it is tempting to think the model is safe — that safety is baked into what it is. The threat model corrects this. Safety is not a monolithic property but an assembly of behaviors installed at different stages: taught during fine-tuning, shaped by human feedback, instructed by a system prompt, and checked by external classifiers. Each of these is a layer with its own reach and its own fragility, and an attacker does not need to defeat all of them — only the ones standing between them and their goal. Qi and colleagues showed how little it takes to strip fine-tuned safety; Arditi and colleagues showed refusal can be mediated by a single direction in activation space. To defend alignment you must first see it for what it is: a stack of thin, unevenly-guaranteed layers.

Safety is a stack, not a property

A deployed model's safe behavior emerges from several distinct mechanisms layered on top of a capable base model. The base model itself is not safe — it will complete whatever is statistically likely, harmful or not. Safety is added: supervised fine-tuning teaches the model to prefer safe completions and to refuse certain requests; reinforcement learning from human feedback further shapes its behavior toward human preferences, including refusals; a system prompt instructs it at inference time on what to do and avoid; and external input/output classifiers screen requests and responses independently of the model.

These layers differ in kind. Fine-tuning and human feedback are learned into the weights and travel with the model; the system prompt is runtime configuration that can be changed or overridden; classifiers are separate components outside the model entirely. This heterogeneity matters because it means safety has no single locus — it is an emergent result of the whole stack — and an attacker can target whichever layer is weakest for their access level rather than confronting the whole assembly.

The defensive consequence is that you must reason about each layer separately: what it guarantees, against whom, and how it fails. Treating alignment as one property invites the mistake of assuming that because the model refuses in normal use, it will refuse under adversarial pressure — when in fact the pressure is applied to specific, individually-defeatable layers.

What each layer guarantees, and against whom

Each layer offers a guarantee only against attackers who cannot reach it. Supervised fine-tuning and RLHF install refusals into the weights, which is robust against a black-box user who can only send prompts — they cannot easily make the model unlearn a refusal through prompting alone. But this weight-level safety is not robust against an attacker who can fine-tune the model, because a few hundred adversarial examples can overwrite it, as Qi and colleagues demonstrated. So the guarantee is conditional on the attacker lacking fine-tuning access.

The system prompt guarantees behavior only against an attacker who cannot override it. Against a benign user it shapes behavior effectively, but against prompt injection or a user who controls the context, its instructions compete with attacker instructions in the same token stream and can be overridden. External classifiers guarantee screening only to the extent they are accurate and cannot be evaded, and they operate independently of the model so they survive some attacks that defeat the model's own refusals — but they add latency and false positives and can be bypassed by adversarial inputs.

This matrix of layer-versus-access is the core of the threat model. A black-box prompting attacker faces the full learned stack; a fine-tuning attacker can peel the weight-level layers; a context-controlling attacker can override the system prompt; and only the external classifiers stand somewhat apart. Naming which attacker you face tells you which guarantees still hold.

Why the learned layers are thin

A recurring empirical finding is that the learned safety layers are surprisingly thin — a small, low-dimensional adjustment on top of a capable base rather than a deep restructuring. Arditi and colleagues found that refusal behavior in aligned models is largely mediated by a single direction in the model's activation space: intervene on that one direction and refusals collapse, without otherwise damaging the model. This suggests alignment is not woven throughout the network but concentrated in a compact, findable, and therefore removable representation.

The thinness explains why so little effort defeats alignment. If safety were deeply entangled with capability, removing it would degrade the model; instead, because it is a thin layer, it can be ablated or overwritten while capability survives intact — the capability-safety decoupling. A few hundred fine-tuning examples suffice to strip refusals; a targeted activation edit suffices to suppress them. The model remains just as capable, only no longer refusing.

For the defender, thinness is a warning: do not assume that a model's refusals are structurally robust. They are a removable behavior, not an immutable property, so any deployment that hands an attacker fine-tuning access or the ability to edit activations should assume the learned safety can be removed, and must place additional, independent controls outside the model.

The attacker's options by access level

The threat model becomes actionable when organized by attacker access, because access determines which layers are reachable. A black-box prompting attacker can only try to find inputs that the learned refusals and classifiers happen to miss — jailbreak prompts — which is a real threat but confronts the full stack. A fine-tuning attacker, given the ability to fine-tune the model (through an API or an open-weights download), can overwrite the learned safety with a small adversarial dataset, defeating SFT and RLHF at once. An open-weights attacker with activation access can go further and directly ablate the refusal direction or steer activations.

Each access level unlocks a cheaper, more reliable attack. Prompting is the hardest and least reliable because it must defeat everything; fine-tuning is cheap and reliable because it removes the weight-level layers directly; activation editing is surgical because it targets the thin representation of safety itself. This ordering tells a defender that access control is the highest-leverage alignment defense — the difference between an attacker facing the full stack and one who can peel it is entirely a matter of what access they are granted.

The tree below organizes the attacker's goal (remove the guardrails) by the access that enables each path, converging on the same outcome — an unaligned but capable model — reached most cheaply by whoever has the most access.

Defending a layered, removable stack

Because the layers have different guarantees and different attackers, the defense is also layered and matched to access. For black-box deployments, harden the learned refusals and classifiers against jailbreaks and monitor outputs, accepting that this is an arms race. For deployments that permit fine-tuning, restrict and vet who can fine-tune, apply safety-preserving fine-tuning techniques, and re-evaluate safety after any fine-tune. For open-weights releases, recognize that learned safety can be removed entirely, so the meaningful controls are downstream — usage policlicense, monitoring, and the assumption that a determined actor has an unaligned model.

The unifying principle is to anchor safety in the layers an attacker cannot reach. External classifiers and output monitoring sit outside the model and survive weight-level attacks; access controls prevent an attacker from reaching the removable layers in the first place; and system-level containment (least privilege on tools, human confirmation for high-impact actions) bounds what an unaligned model can do even if all its refusals are gone. These are the guarantees that do not evaporate when the thin learned layers are peeled.

The synthesis, drawing on Qi, Arditi, Skalse, and Ouyang and their colleagues, is that alignment is a stack of thin, unevenly-robust layers, so the threat model must be read layer by layer against attacker access. Know what each layer guarantees, assume the removable ones are removed by anyone with the access to remove them, and place the controls you actually rely on where the attacker cannot reach — outside the model, at the access boundary, and around the agent's authority.

Why this matters for agents

For agents the stakes are highest because an unaligned model with tools acts, not just speaks. If an attacker strips an agent's refusals through fine-tuning or activation editing, the agent will not merely produce harmful text — it will take harmful actions with whatever authority it holds. This makes the containment layer decisive: the difference between a stripped-refusal model that can only talk and one that can move money or exfiltrate data is entirely a matter of the authority the agent was granted, which is a control outside the model.

The organizing lesson is that alignment cannot be the sole safety mechanism for a capable agent, because it is thin and removable. It is a valuable layer that reduces harm in normal use and against weak attackers, but the guarantees a platform actually depends on must live where an attacker cannot peel them: access controls that prevent safety removal, external monitoring that survives it, and least-privilege containment that bounds the damage when refusals are gone. Build those, and the removability of the learned layers becomes a managed risk rather than a single point of failure.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

alignment_defense_policy:
  access_control:
    fine_tuning: restricted_and_vetted
    weight_access: restricted
  external_checks:
    input_output_classifiers: enabled
    output_monitoring: enabled
  post_finetune:
    safety_reevaluation: required
  containment:
    tool_authority: least_privilege
    high_impact_actions: require_confirmation
    assume_refusals_removable: true

SAFETY RE-EVALUATION (run after every fine-tune):
  1. Sample the original refusal categories the base model covered.
  2. For each, probe with benign, borderline, and clearly-disallowed asks.
  3. Compare refusal rates to the pre-fine-tune baseline.
  4. FAIL the fine-tune if refusal rates dropped materially on any
     disallowed category, even if capability improved.
  # Judge with an independent classifier, not the model itself.

Glossary

Alignment

The set of behaviors that make a model act according to human intentions and safety policies, layered onto a capable base model.

Supervised fine-tuning (SFT)

Training a base model on curated examples, including refusals, to prefer safe completions.

RLHF

Reinforcement learning from human feedback, which shapes a model's behavior toward human preferences including safety.

System prompt

Runtime instructions given to a model to steer its behavior, which an attacker with context control can override.

External classifier

A separate component that screens inputs or outputs independently of the model, surviving some model-level attacks.

Refusal direction

A single direction in activation space found to largely mediate refusal behavior, whose ablation collapses refusals.

Capability-safety decoupling

The phenomenon that safety can be removed while capability remains intact, because safety is a thin layer.

Attacker access

The level of control an attacker has (prompting, fine-tuning, weights), which determines which safety layers they can defeat.

References

1. Qi et al., Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To (arXiv 2310.03693)
2. Arditi et al., Refusal in Language Models Is Mediated by a Single Direction (arXiv 2406.11717)
3. Skalse et al., Defining and Characterizing Reward Hacking (NeurIPS 2022, arXiv 2209.13085)
4. Ouyang et al., Training Language Models to Follow Instructions with Human Feedback (arXiv 2203.02155)
5. NIST AI 100-2 e2023, Adversarial Machine Learning: A Taxonomy and Terminology