Defenses: Safety-Preserving Fine-Tuning, Tamper-Resistance, and Adaptive Evaluation

Everything in this series points to one conclusion: alignment as currently built is a thin overlay that a motivated actor with the right access can remove. That is a hard truth, but it is not a counsel of despair — it is a design brief. If the learned safety can be peeled, then the defenses you actually rely on must not depend on it staying intact. This article is about building that kind of defense: making safety harder to remove where research permits, catching unsafe behavior with checks that live outside the model, evaluating safety in ways that adaptation cannot game, and containing the model's authority so that even a fully jailbroken model cannot do much. Defense in depth, anchored where the attacker cannot reach, is the answer to a removable overlay.

The defensive stance: anchor outside the overlay

The organizing principle of alignment defense follows directly from the threat model. Because the learned safety layers (SFT, RLHF, activation-level refusals) can be removed by fine-tuning or activation editing, the controls you depend on must be anchored where that removal does not reach: in access control that prevents an attacker from touching the model, in external checks that operate regardless of the model's internal state, in evaluation that measures true safety rather than a gameable proxy, and in containment that bounds impact. The learned layers remain valuable — they make normal-use behavior safer — but they are not the last line.

This is defense in depth with a specific shape: each layer assumes the ones inside it can fail. Safety-preserving fine-tuning reduces the chance the overlay erodes, but assumes it might; tamper-resistant training raises the cost of removal, but assumes a determined actor can pay it; output monitoring catches unsafe behavior, but assumes some slips through to the action layer; and containment bounds what any unsafe action can reach. No layer is trusted to be complete, so their combination degrades gracefully rather than failing all at once.

The rest of this article works through the layers, from those closest to the model (safety-preserving and tamper-resistant training) to those furthest outside it (monitoring, evaluation, and containment), noting for each what attack it addresses and what it assumes about the layers within.

Safety-preserving fine-tuning

The first layer addresses erosion during fine-tuning, both adversarial and incidental. Safety-preserving fine-tuning techniques aim to let a model be adapted to a task without losing its refusals. The common approaches mix alignment data into the fine-tuning set so the model keeps practicing refusals, regularize the fine-tuned model toward the aligned reference so it does not drift far from safe behavior, and constrain which parameters fine-tuning can change to protect the safety-relevant ones. Each reduces the amount of safety lost per unit of task adaptation.

These techniques resist the benign, incidental safety loss that is the common case — a team fine-tuning on legitimate data keeps more of its safety — and they raise the cost of adversarial stripping, though a determined attacker with full fine-tuning control can often still overcome them. So safety-preserving fine-tuning is a strong default that reduces erosion, paired with the assumption that it is not a guarantee against a motivated adversary.

The essential complement is the safety re-evaluation gate from the harmful-fine-tuning analysis: apply safety-preserving techniques during tuning, then verify with an adversarial refusal test before deployment. Preservation lowers the erosion; the gate catches whatever erosion remains. Neither alone is sufficient, but together they make fine-tuning a much smaller safety risk.

Tamper-resistant training

A more ambitious research direction aims to make safety harder to remove in the first place — tamper-resistant training. The goal is to train safety into the model in a way that fine-tuning or activation editing cannot easily undo, for example by making the safety behavior more distributed and entangled with capability (so removing it damages the model), or by training against the very attacks that would remove it (adversarial training over fine-tuning and ablation). If successful, this would raise the cost and reduce the reliability of safety removal, partly closing the gap the refusal-direction and harmful-fine-tuning results exposed.

It is important to be honest about the state of this work: it is promising but not solved. The refusal-direction finding shows how localized current safety is, and making it genuinely distributed and tamper-resistant without sacrificing capability is an open problem. A deployment today should treat tamper-resistance as an emerging, partial mitigation rather than a settled guarantee, and should not assume that any current model's safety is unremovable.

The practical stance is to adopt tamper-resistant techniques where available as a cost-raiser — anything that makes removal harder or less reliable helps — while continuing to anchor the guarantees you actually depend on in the outer layers. Tamper-resistance narrows the attack surface; it does not yet eliminate it, so it complements rather than replaces external checks and containment.

Output monitoring: safety outside the model

The most robust layers operate outside the model entirely, so they survive any weight- or activation-level attack. Output monitoring — external classifiers and policy checks that screen the model's inputs and, crucially, its outputs and actions — catches unsafe behavior regardless of whether the model's own refusals are intact. Because these run as separate components, an attacker who ablates the refusal direction or fine-tunes away the model's caution still faces the external screen, which has no refusal direction to remove.

The power of external monitoring is precisely its independence: it does not care how the model decided to produce an output, only what the output is, so it is immune to the representation-level and fine-tuning attacks that defeat the model's internal safety. Its limits are accuracy and evasion — a classifier can be wrong or bypassed by adversarial inputs — so it is a strong layer, not a perfect one, and belongs in depth with the others. For agents, the most valuable form screens actions, not just text: before a tool call executes, an independent policy check decides whether it is allowed.

The discipline is to make the external screen the enforcement point for anything consequential. If the model's refusal is the only thing standing between a harmful request and a harmful action, that safety is removable; if an independent, external check must also approve the action, the safety survives the model being jailbroken. Moving enforcement outside the model is what converts removable safety into robust safety.

Evaluation that survives adaptation

Defenses are only as good as the evaluation that verifies them, and alignment evaluation has a specific failure mode: it can be fooled by adaptation. A model can be tuned to pass a fixed safety benchmark while remaining unsafe on anything the benchmark does not cover — the same specification-gaming dynamic as reward hacking, applied to the evaluation itself. So safety evaluation must be adaptive and adversarial: it must probe with novel, held-out, and attacker-style inputs rather than a static suite the model may have been optimized to pass.

Concretely, adaptive evaluation red-teams the model with fresh jailbreaks and edge cases, judges refusals with an independent classifier rather than the model itself, tests after any adaptation (fine-tuning, merging, prompting changes), and treats a passed static benchmark as necessary but not sufficient. The goal is to measure the safety that survives an adversary, not the safety that survives a known test, because an attacker will use exactly the inputs the fixed test omits.

This matters because every other defense is validated by evaluation. If the evaluation is gameable, safety-preserving fine-tuning, tamper-resistance, and monitoring all get false confidence. Adaptive, adversarial, independently-judged evaluation — run continuously and after every change — is what keeps the whole defensive stack honest, and it is the layer that most directly answers the threat model's warning that safety is thin and removable: you only know it held if you tested it the way an attacker would.

Putting the program together for agents

For a team deploying a capable agent, the defensive program is layered and anchored outside the model. Restrict fine-tuning and weight access so the learned safety is harder to reach. Apply safety-preserving (and, where available, tamper-resistant) training so the overlay erodes less and removes harder. Enforce consequential actions through external checks and action-level policy that survive a jailbroken model. Evaluate safety adaptively and adversarially after every change, judged independently. And contain the agent's authority with least privilege and confirmation so that even a fully unaligned model cannot take a catastrophic action. Each layer assumes the ones inside can fail, so the program degrades gracefully.

The synthesis, drawing across Qi, Arditi, Skalse, and Ouyang and their colleagues, is that alignment cannot be made unremovable today, so the defense is not to perfect the overlay but to build a stack that does not depend on it. Preserve safety where you can, raise removal costs where research allows, monitor and enforce outside the model, evaluate the way an attacker would, and bound authority so being wrong is survivable. The thin, removable overlay becomes a managed risk inside a defended system rather than a single point of failure — which is the most any responsible deployment can claim given what the threat model has shown.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

alignment_defense_program:
  access_control:
    fine_tuning: restricted
    weight_access: restricted
  training:
    safety_preserving: default
    tamper_resistant: where_available
    post_change_safety_gate: required
  external_enforcement:
    output_classifiers: enabled
    action_level_policy: enabled
  evaluation:
    adaptive_adversarial: required
    independent_judge: true
    re_evaluate_after_change: true
  containment:
    tool_authority: least_privilege
    high_impact: require_confirmation

# DEFENSIVE HARNESS — measure safety that survives an attacker
function adaptive_eval(model):
    static_ok = passes(static_safety_suite(model))
    fresh = generate_novel_redteam_cases()   # held out, rotated
    survived = all(independent_judge_refuses(model, c) for c in fresh)
    if not (static_ok and survived):
        block_deploy("failed adaptive safety evaluation")
    # Re-run after EVERY fine-tune, merge, or prompt change.

Glossary

Safety-preserving fine-tuning

Techniques that let a model be adapted to a task while retaining its safety, by mixing alignment data, regularizing, or constraining parameters.

Tamper-resistant training

Emerging methods that aim to make a model's safety harder to remove via fine-tuning or activation editing.

Output monitoring

External classifiers and policy checks that screen a model's inputs, outputs, and actions independently of the model.

Action-level policy

An external check that decides whether a specific agent action is permitted before it executes.

Adaptive evaluation

Adversarial, held-out safety testing that probes with novel attacker-style inputs rather than a static suite.

Defense in depth

Layering controls so each assumes the ones within it can fail, degrading gracefully rather than all at once.

Containment

Bounding a model's authority with least privilege and confirmation so unsafe behavior cannot reach catastrophic actions.

Enforcement point

The place where a decision is enforced; moving it outside the model makes safety survive the model being jailbroken.

References

1. Qi et al., Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To (arXiv 2310.03693)
2. Arditi et al., Refusal in Language Models Is Mediated by a Single Direction (arXiv 2406.11717)
3. Skalse et al., Defining and Characterizing Reward Hacking (NeurIPS 2022, arXiv 2209.13085)
4. Ouyang et al., Training Language Models to Follow Instructions with Human Feedback (arXiv 2203.02155)
5. NIST AI 100-2 e2023, Adversarial Machine Learning: A Taxonomy and Terminology