Alignment & Fine-Tuning Attacks · 3 of 5L3offensive security
Refusal Ablation and Activation Steering: The Refusal Direction
If a model's willingness to refuse lives along one direction in its activation space, then erasing that direction erases the refusals — surgically, and without retraining.
Abstract
Beyond fine-tuning lies an even more surgical way to remove safety: edit the model's internal representations directly. Arditi and colleagues found that refusal behavior in aligned language models is largely mediated by a single direction in activation space, so ablating that direction — projecting activations to remove their component along it — collapses refusals while leaving the model otherwise intact. This article examines refusal ablation and activation steering as both an attack and a window into how alignment is represented. It explains the refusal-direction finding, the mechanics of ablation and steering, why this reveals safety as a compact and localized representation, and pairs each with a defense. The key takeaway is that alignment, being representable as a low-dimensional feature, is inherently manipulable by anyone with activation or weight access, so safety for open-weights models cannot rely on the fragility of that representation and must be anchored in external, access-controlled, and containment layers.
There is a hierarchy of ways to remove a model's safety, ordered by how much access they require and how surgical they are. Prompting must defeat the whole stack. Fine-tuning overwrites the learned layers with a small dataset. But the most precise method reaches into the model's internal activations and edits the representation of refusal itself. Arditi and colleagues showed this is startlingly feasible: refusal is largely carried by a single direction in the model's activation space, and removing that direction removes the refusals — no retraining, no dataset, just a linear edit. That safety can be localized to one direction is both a powerful attack and a profound statement about how thin and how findable alignment really is.
The refusal direction
Interpretability research has repeatedly found that high-level behaviors in language models are often represented linearly — as directions in the model's high-dimensional activation space. Arditi and colleagues applied this to safety and found that refusal, the behavior of declining a harmful request, is largely mediated by a single such direction. Concretely, they identified a direction along which harmful-request activations differ from harmless-request activations, and showed that this one direction carries most of the model's refusal decision.
This is a remarkable localization. It means the model's willingness to refuse is not diffusely spread across its billions of parameters but concentrated in a compact, identifiable feature. When an activation has a large component along the refusal direction, the model refuses; when that component is small, it complies. The decision that a request is dangerous and should be declined is, to first approximation, a single scalar projection.
For a defender, this localization is the crux of the threat: a behavior that lives in one direction can be manipulated by editing that direction. The very compactness that makes refusal efficient to represent makes it efficient to remove, and the rest of this article is about the edits that exploit that compactness and the defenses that must assume it.
- Assume high-level safety behaviors are linearly represented and therefore locatable and editable.
- Do not treat refusal as diffusely robust; it can be concentrated in a compact feature.
- Anchor safety in layers that do not depend on the refusal representation staying hidden.
Ablation: erasing the direction
Refusal ablation removes the direction. Because refusal is carried by the projection of activations onto the refusal direction, an attacker can suppress refusals by projecting that component out — at each relevant layer, subtract the activation's component along the refusal direction so it becomes orthogonal to it. With the refusal component removed, the model no longer registers that a request is refusable, and it complies. This can be applied at inference time (editing activations as they flow) or baked into the weights (modifying the layers so they never produce the refusal component), producing a permanently jailbroken open-weights model.
The precision is what distinguishes ablation from fine-tuning. Fine-tuning overwrites behavior with new training and can have broad, hard-to-predict effects; ablation is a targeted linear surgery that removes exactly the refusal feature and, as Arditi and colleagues showed, leaves the model's other capabilities essentially untouched. The result is a model that is just as fluent and competent and no longer refuses anything, achieved without any training data at all.
The defensive implication is stark: for anyone with activation or weight access, refusal is one linear operation away from gone. This is not a hypothetical for open-weights models — the weights are public, the direction is findable, and the ablation is cheap. So safety for such models cannot rest on the refusal representation being intact, because it demonstrably will not be for a motivated actor.
- For open-weights models, assume the refusal direction can be found and ablated cheaply.
- Do not rely on the intact refusal representation as a safety guarantee for accessible weights.
- Place safety in external checks and containment that survive activation and weight edits.
Steering: the same lever in reverse
Activation steering is the general technique of which ablation is one use: adding or subtracting a direction vector to activations to push the model's behavior along that feature. Subtracting the refusal direction suppresses refusals (ablation); adding it can induce over-refusal; and the same method applies to other behavioral features that turn out to be linearly represented. Steering makes the model's behavior a dial the operator — or attacker — can turn, given access to the activations.
This bidirectionality is instructive. It shows that the refusal direction is not just correlated with refusal but causally controls it: pushing along it changes behavior predictably in both directions, which is strong evidence that the model really does represent 'should I refuse?' as a manipulable internal quantity. For an attacker, this means fine-grained control; for a defender, it means the behavior is exposed to anyone who can touch the activations, not merely removable but tunable.
Steering also underlies some defensive uses — one can steer toward safer behavior — but defensively it is fragile, because whoever controls the activations controls the dial, and an attacker with the same access can steer the other way. So steering-based defenses only help when the attacker cannot reach the activations, which returns the problem to access control: the technique is symmetric, and access decides who wins.
- Recognize the refusal direction causally controls refusal, so activation access confers behavioral control.
- Treat steering-based defenses as effective only when the attacker cannot reach the activations.
- Return the problem to access control, since the steering lever is symmetric between defender and attacker.
What this reveals about alignment
The scientific significance of the refusal-direction finding is that it makes concrete how thin alignment is. If safety required a deep, distributed transformation of the model's computation, it could not be captured by one direction; that it can suggests alignment is a shallow, low-dimensional overlay on a fundamentally unaligned base. The base model represents how to answer any request; alignment adds a compact feature that gates certain answers, and that gate is a single direction wide.
This connects the phenomena across this series. It is why fine-tuning can strip safety with few examples (the overlay is thin), why the capability-safety decoupling holds (the overlay is separable from capability), and why ablation works (the overlay is linearly localized). All three are facets of the same underlying fact: current alignment is a compact, separable, manipulable feature rather than a deep property, so any actor who can reach the representation can remove it.
The honest conclusion is that alignment as currently implemented is a behavioral veneer, robust against users who can only prompt but not against actors who can fine-tune or edit. This does not make alignment worthless — it meaningfully improves behavior in normal use and against weak attackers — but it does mean it cannot be the last line of defense for a model whose weights or activations an adversary can touch. The last line must be elsewhere.
- Treat current alignment as a behavioral veneer robust only against prompt-only attackers.
- Do not make the removable representation the last line of defense for accessible models.
- Locate the last line of defense outside the model, in access control and containment.
Defending against representation-level attacks
Because ablation and steering require activation or weight access, the primary defense is denying that access: keep weights and activations behind a serving boundary an attacker cannot cross, so the refusal direction cannot be reached. For a hosted API this is achievable — the attacker sees only outputs — and it is why hosted models resist representation-level attacks that open-weights models cannot. Access control is again the highest-leverage control, converting a surgical attack into an impossible one for those outside the boundary.
For open-weights models, where access cannot be denied, the defense shifts entirely to layers outside the model: external input/output classifiers that screen regardless of the model's internal state, usage monitoring, and — decisively — least-privilege containment so that a jailbroken model wired to tools still cannot reach dangerous capabilities. Research into more robust, distributed, or tamper-resistant safety representations is ongoing and important, but a deployment today must assume the current representation is removable and rely on controls that do not depend on it.
The synthesis, grounded in Arditi and colleagues, is that alignment is linearly localized and therefore surgically removable by anyone with representation access, so the safety a system actually depends on must live where that access does not reach: behind a serving boundary for hosted models, and in external checks plus containment for open-weights ones. The refusal direction is a beautiful scientific result and a sobering security one — safety that can be captured in a direction can be erased in a direction.
- Deny weight and activation access behind a serving boundary so the refusal direction cannot be reached.
- For open-weights models, rely on external classifiers, monitoring, and least-privilege containment.
- Assume the current refusal representation is removable and never depend on it for accessible models.
Why this matters for agents
An agent built on an open-weights model must assume its refusals can be ablated, because they can. If the agent's safety rests solely on the model declining harmful actions, a single linear edit removes that safety and the agent will act on requests it should decline, with whatever authority it holds. The representation-level attack is the sharpest reminder that a tool-wielding agent's guardrails cannot live only inside a manipulable model.
The organizing lesson is that safety for agents on accessible models is an architecture problem, not a model property. Put refusal-independent checks around the model (external classifiers, action-level policy), keep the model's authority minimal so a jailbroken model cannot do much, and require confirmation for consequential actions. Arditi and colleagues' single direction tells every agent builder the same thing fine-tuning did, only more sharply: the model's own refusals are removable, so the controls you rely on must sit where the attacker's linear surgery cannot reach.
- For agents on open-weights models, place refusal-independent checks and action-level policy around the model.
- Keep agent authority minimal and require confirmation so an ablated model cannot act catastrophically.
Key takeaways
- Refusal behavior in aligned models is largely mediated by a single direction in activation space, concentrating safety in a compact feature.
- Ablation projects that direction out of activations or weights, collapsing refusals while leaving capability intact — no training data needed.
- Activation steering is the same lever generalized: the refusal direction causally controls refusal and can be tuned in either direction.
- This reveals alignment as a thin, separable, linearly-localized overlay, which unifies why fine-tuning, decoupling, and ablation all work.
- Representation-level attacks require activation or weight access, so denying that access (hosted serving boundary) is the primary defense.
- For open-weights models the last line of defense must be external classifiers, monitoring, and least-privilege containment that do not depend on the refusal representation.
Practitioner Toolkit
Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.
Run when deploying a model whose weights or activations may be accessible.
- Hosted models keep weights and activations behind a serving boundary.
- Open-weights deployments assume the refusal direction can be ablated.
- External input/output classifiers screen independently of the model's internal state.
- Action-level policy checks sit around the model, not only inside it.
- Agent authority is minimal with confirmation for consequential actions.
- Safety does not depend on the refusal representation staying intact.
Illustrative policy anchoring safety outside the model.
representation_defense_policy:
hosted:
weight_activation_access: denied
expose: outputs_only
open_weights:
assume_refusal_ablatable: true
external_checks:
input_output_classifiers: enabled
action_level_policy: enabled
containment:
tool_authority: least_privilege
high_impact: require_confirmationDo these first for accessible or open-weights models.
- Keep hosted weights and activations behind a serving boundary.
- Add external input/output classifiers independent of the model.
- Enforce action-level policy around the model.
- Minimize agent authority and require confirmation for consequential actions.
Sanitized skeleton to verify external checks catch what an ablated model would allow (defensive).
# DEFENSIVE PROBE — do external checks hold if the model stops refusing?
function external_coverage(disallowed_actions):
for a in disallowed_actions:
# Simulate a model that no longer refuses (returns the action):
allowed_by_model = True
blocked = external_classifier(a) or action_policy_denies(a)
assert blocked, f"external checks miss {a}"
# If any disallowed action is not blocked externally, the deployment
# depends on the (removable) model refusal — fix the external layer.Glossary
- Refusal direction
- A single direction in a model's activation space that largely mediates whether it refuses a request.
- Activation space
- The high-dimensional space of a model's internal representations, where behaviors can be represented as directions.
- Ablation
- Removing a feature by projecting activations (or weights) to eliminate their component along its direction.
- Activation steering
- Adding or subtracting a direction vector to activations to push the model's behavior along that feature.
- Linear representation
- The finding that many high-level behaviors are encoded as directions in activation space.
- Behavioral veneer
- A shallow overlay of behavior (here refusals) on a capable base, removable without touching capability.
- Representation access
- The ability to read or edit a model's internal activations or weights, required for ablation and steering.
- Serving boundary
- The boundary of a hosted deployment that exposes only outputs, denying attackers activation or weight access.
References
- Arditi et al., Refusal in Language Models Is Mediated by a Single Direction (arXiv 2406.11717)
- Qi et al., Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To (arXiv 2310.03693)
- Ouyang et al., Training Language Models to Follow Instructions with Human Feedback (arXiv 2203.02155)
- Skalse et al., Defining and Characterizing Reward Hacking (NeurIPS 2022, arXiv 2209.13085)
- NIST AI 100-2 e2023, Adversarial Machine Learning: A Taxonomy and Terminology