Weight and Adapter Tampering: Malicious Merges and Detecting Behavioral Drift

Adapters are one of the best ideas in practical machine learning: instead of shipping a whole fine-tuned model, you ship a small file that adjusts a base model's behavior, cheap to train, cheap to distribute, easy to combine. But every property that makes adapters useful makes them dangerous as a supply-chain vector. They are small, so they attract little scrutiny. They change behavior, so they can inject a backdoor or strip a guardrail. They are combined and merged, so a malicious one can ride into a trusted stack. A tampered adapter is a behavior patch for your model authored by someone else, and applying it without verification is trusting a stranger to edit what your model does.

Adapters are behavior-changing dependencies

An adapter modifies a base model's behavior by adding a small set of trained parameters — low-rank matrices in the case of LoRA — that adjust the model's computation. Merging an adapter folds those adjustments into the base weights, producing a specialized model. The whole point is that a small adapter can meaningfully change what the model does, which is exactly why it is a supply-chain risk: a small, behavior-changing artifact is an efficient way to inject a change an attacker wants, from a subtle bias to a full backdoor.

This reframes adapters as dependencies with the same standing as a base checkpoint. Downloading and merging an adapter from a hub is trusting that artifact to edit your model's behavior, and the trust is often given implicitly because adapters feel like minor add-ons rather than the behavior patches they are. The data-poisoning and alignment threat models both apply: a malicious adapter can carry a trigger-gated backdoor or can move the model off its safety alignment, delivered as a few megabytes.

The defensive principle is that any artifact that changes weights — base checkpoint, adapter, or the result of a merge — is a behavior-changing dependency and must be authenticated and verified like one. The small size of an adapter is not a reason for less scrutiny; it is a reason for more, because a small file that changes behavior is a high-leverage tampering vector.

The malicious merge

The sharpest form of this attack is the malicious merge. Model merging combines multiple models or adapters into one, and it is increasingly common as teams blend a base model with community adapters to get a desired mix of capabilities. An attacker who contributes one of the merged components can embed hidden behavior that survives the merge, riding into the final model alongside legitimate capabilities. Because the merged result is treated as a single trusted artifact, the malicious contribution's origin is lost, and the hidden behavior is now part of the model the team ships.

This is potent for two reasons. First, low cost and high leverage: contributing one adapter to a popular merge, or publishing an attractive adapter that others will merge, injects the payload into every downstream model that includes it, amplifying a single artifact into many compromised models. Second, laundered provenance: merging blends contributions, so the final model does not obviously carry the attacker's fingerprint, and a consumer verifying the merged artifact's integrity confirms only that it is the unaltered merged file, not that its components were benign.

The defense is provenance over the whole composition, not just the final artifact. Verify the authenticity of every component that goes into a merge, record the merge recipe and component provenance, and treat a merge whose components you cannot vouch for as untrusted. Integrity of the final file is necessary but not sufficient; you must trust the inputs to the merge, not only that the output was not altered afterward.

Why weight inspection cannot see it

A tampered weight or adapter changes numbers, and the changed numbers do not look wrong. There is no signature in the weights that says 'backdoor here' — a backdoored model's parameters are statistically unremarkable, as the data-poisoning analysis established, and an adapter's adjustments are exactly the kind of small changes adapters are supposed to make. So inspecting the weights, comparing file sizes, or checking that the numbers are in a normal range does not reveal a malicious change. The tampering is in what the weights do, not in how they look.

This is the same lesson as the backdoor and refusal-direction analyses: safety and malice live in behavior, not in obvious structural features. A malicious adapter that installs a trigger-gated backdoor or nudges the model off its alignment produces weights that pass every static check and behave normally until the trigger appears or the specific behavior is exercised. Static integrity tells you the file was not altered after publishing; it tells you nothing about whether the published behavior is what you want.

The consequence is that detecting tampering requires exercising behavior, not inspecting weights. You must run the model and compare what it does to what a trusted baseline does — behavioral drift detection — because that is the only way to surface a change that is invisible in the parameters. Integrity verification and behavioral testing are complementary: one catches post-publication tampering, the other catches malicious-but-authentic behavior.

Behavioral drift detection

Behavioral drift detection compares a candidate model's behavior to a trusted baseline across a battery of probes, flagging where they diverge. Concretely: hold a reference model or a recorded reference behavior, run both the candidate and the reference on a fixed evaluation set covering capabilities, safety refusals, and known-sensitive behaviors, and measure where the candidate drifts. Unexpected drift — a refusal rate that dropped, a capability that changed on a specific class of input, a bias that appeared — is a signal that the adapter or merge changed something beyond its stated purpose.

The power of drift detection is that it targets behavior directly, so it catches changes that are invisible to static checks: a stripped safety layer shows up as dropped refusals, a backdoor may show up as anomalous behavior on trigger-like inputs, a bias shows up as skewed outputs. It is most effective when the evaluation set includes safety and trigger-aware probes, since those are exactly the behaviors an attacker targets and a capability-only benchmark would miss.

Its limits mirror every behavioral defense: it can only catch drift it probes for, and a well-hidden trigger may not fire on the evaluation set. So drift detection is a strong detector paired with provenance (to reduce the chance of malicious inputs in the first place) and containment (to bound impact if a change slips through). The decision to promote a merged or adapted model should gate on a drift check against the trusted baseline, not just on integrity and capability metrics.

Defending the adapter supply chain

The controls compose into an adapter and merge supply-chain discipline. Authenticate the publisher and verify the integrity of every adapter and every merge component, not just the final artifact. Record the merge recipe and component provenance so any injected behavior is traceable. Behaviorally test every adapted or merged model against a trusted baseline, gating promotion on drift within expected bounds and including safety and trigger-aware probes. And deploy under containment so that if a hidden behavior slips through, its impact is bounded by the model's limited authority.

The unifying principle is that composition does not confer trust. Combining trusted-looking components into a merged model does not make the result trustworthy if any component was malicious, and integrity of the composed artifact does not vouch for the behavior it composes. So trust must be established over the whole composition — every input authenticated, the recipe recorded, the output behaviorally verified — rather than assumed from the final file's integrity.

The synthesis, consistent with the data-poisoning and alignment analyses and NIST's supply-chain guidance, is that any weight-changing artifact is a behavior patch that must be authenticated and behaviorally verified. Adapters and merges are efficient, low-cost tampering vectors precisely because they are small and behavior-changing, so they deserve the full supply-chain treatment: provenance over components, drift detection against a baseline, and containment. Treat an adapter as code that edits your model, and verify it before you run it.

Why this matters for agents

Agent teams lean heavily on adapters and merges to specialize models cheaply — a domain adapter here, a community merge there — and each is a behavior-changing artifact loaded into a tool-wielding system. A malicious adapter that installs a backdoor or strips safety turns the agent into a sleeper that acts against its principal on a trigger, or that quietly declines fewer harmful actions than the base it started from. Because the change is invisible in the weights, only behavioral drift testing against a trusted baseline reveals it before deployment.

The organizing lesson is that adapter and merge provenance plus behavioral drift detection are essential agent supply-chain controls, not optional polish. An agent team must authenticate every component, record merge recipes, gate promotion on drift against a baseline with safety probes, and contain authority. The data-poisoning, alignment, and supply-chain analyses converge on the same instruction: a small file that changes your model's behavior is a high-leverage attack surface, so verify its provenance and test its behavior before you let it edit the model your agent runs on.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

# DEFENSIVE PROBE — does this adapter/merge change behavior unexpectedly?
function drift_check(candidate, baseline, probes):
    for p in probes:                    # capability + safety + trigger-aware
        c = behavior(candidate, p)
        b = behavior(baseline, p)
        if diverges(c, b) > DRIFT_BOUND:
            flag(p, kind=classify(p))   # e.g. refusal dropped, output skewed
    # Block promotion on unexpected drift, especially on safety probes.

adapter_supply_chain_policy:
  components:
    authenticate_each: required
    integrity_verify_each: required
  merges:
    record_recipe: true
    record_component_provenance: true
    unverifiable_components: untrusted
  promotion:
    behavioral_drift_check: required
    baseline: trusted_reference
    include_safety_and_trigger_probes: true
    gate_on_drift: true
  deployment:
    containment: least_privilege

Glossary

Adapter

A small set of trained parameters (e.g., LoRA low-rank matrices) that adjusts a base model's behavior.

LoRA

Low-Rank Adaptation, a common adapter method that adds trainable low-rank matrices to a base model.

Merge

Combining multiple models or adapters into one, which can carry a malicious component's hidden behavior forward.

Malicious merge

A merge in which an attacker-contributed component embeds hidden behavior that survives into the final model.

Behavioral drift

A divergence between a candidate model's behavior and a trusted baseline's, signaling an unexpected change.

Drift detection

Comparing a candidate model to a trusted baseline across probes to flag behavior changes invisible in the weights.

Provenance over composition

Authenticating every component that goes into a merge, not just verifying the final artifact's integrity.

Containment

Bounding a model's authority so a hidden behavior that slips through cannot act catastrophically.

References

1. Gu et al., BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain (arXiv 1708.06733)
2. Qi et al., Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To (arXiv 2310.03693)
3. NIST SP 800-161r1, Cybersecurity Supply Chain Risk Management Practices for Systems and Organizations
4. NIST AI 100-2 e2023, Adversarial Machine Learning: A Taxonomy and Terminology
5. safetensors format specification (Hugging Face)
6. MITRE ATLAS (Adversarial Threat Landscape for AI Systems)