Diversity Is Not Independence: When a Second Model Actually Reduces Risk

When a single model cannot be trusted to verify its own work, the obvious fix is to bring in another one. Ask a second model whether the first model's answer is correct, whether its code is safe, or whether its proposed action is allowed, and accept the result only if both agree. The arithmetic behind the fix is appealing: if each model misses a problem one time in twenty, two independent models should miss it together only one time in four hundred. The trouble is the word independent. Software engineers ran exactly this experiment forty years ago, with teams of programmers instead of models, and found that independently built versions of the same program failed on the same inputs far more often than independence predicts. Language models give the old problem new force, because many of them learn from overlapping data, follow similar training recipes, and, in deployment, read the very same inputs as the model they are supposed to check. A second model is not automatically a second opinion. Whether it is one has to be measured.

The redundancy argument and its hidden assumption

Redundancy is one of engineering's oldest tools for reliability. Two independent components that must both fail for a system to fail give a combined failure probability equal to the product of their individual failure probabilities, which is far smaller than either alone. The argument for model-on-model checking is the same: if a generator and a checker each have a modest error rate, requiring the checker's approval should make undetected errors rare.

Everything rests on independence, meaning that knowing the first component has failed tells you nothing about whether the second will. If the two tend to fail on the same inputs, the combined failure rate is no longer the product of the two rates. In the limiting case, where the checker fails exactly when the generator does, the second component adds nothing at all, while still adding cost and an unearned sense of safety.

The question for any organisation using a second model is therefore not whether the second model is good, but whether its mistakes are independent of the first model's mistakes on the inputs that matter. That is an empirical question, and the history of fault-tolerant software suggests a cautious prior.

What multiversion software learned

In the 1980s, N-version programming proposed building several versions of the same program from the same specification, written by separate teams, and running them together so that a majority vote would mask the faults of any one version (Avizienis, 1985). The approach relied on design diversity: separate teams, working separately, were expected to make different mistakes.

Knight and Leveson tested that expectation directly. Many versions of the same program were written independently from one specification and run on a very large number of test inputs. The versions failed together on the same inputs much more often than an independence model predicted, and the independence hypothesis was statistically rejected (Knight and Leveson, 1986). Separate teams did not reliably produce separate mistakes.

Eckhardt and Lee supplied a theoretical reason that does not depend on teams copying each other. Some inputs are intrinsically harder than others. Even if every team works in complete isolation, all of them are more likely to fail on the hard inputs, so failures cluster on those inputs and become positively correlated across versions (Eckhardt and Lee, 1985). Littlewood and Miller extended the model to forced diversity, showing that deliberately using different methods for different versions can reduce, and in principle even reverse, that correlation, because different methods find different inputs hard (Littlewood and Miller, 1989).

Two lessons carry over directly. Independence of process does not imply independence of failure. And the useful lever is not separation alone but forced difference in how each component approaches the problem.

The evidence that language models fail together

Language models now give this old result new data. A large-scale study of more than three hundred and fifty models, drawn from two public leaderboards and a résumé-screening task, found substantial correlation in model errors: on one leaderboard dataset, when two models both erred, they agreed on the wrong answer about sixty percent of the time. Shared architectures and shared providers increased correlation, and, critically, larger and more accurate models had highly correlated errors even when their architectures and providers differed (Kim, Garg and Peng, 2025).

A second line of work measures similarity directly through the overlap of model mistakes. Using a chance-adjusted similarity metric, it found that model-as-judge scores favour models similar to the judge, and that the mistakes of models are becoming more similar as their capabilities increase, which the authors flag as a risk of correlated failure in AI oversight (Goel et al., 2025). Earlier work on model-as-judge evaluation had already documented biases such as favouring particular answer positions and verbose answers (Zheng et al., 2023), and a related study found that evaluator models can recognise their own generations and tend to score them more favourably (Panickssery, Bowman and Feng, 2024).

Taken together, the findings point one way. Different models are less different than their branding suggests, the most capable models are converging in what they get wrong, and when a model judges another model, similarity between the two biases the judgement. None of this makes a second model useless. It means that the arithmetic of independent redundancy cannot be assumed, and that the checkers most organisations would naturally choose, namely the strongest available models, are the ones most likely to share blind spots with the strong model they check.

Why models fail together: four mechanisms

It helps to separate the reasons for correlation, because each one calls for a different remedy. The analysis below is a synthesis of the evidence above with the multiversion literature.

Shared difficulty is the Eckhardt and Lee effect. Some inputs are genuinely hard: ambiguous instructions, subtle bugs, rare edge cases. Any competent checker is more likely to fail on them, so failures cluster there regardless of how the checkers were built.

Shared upbringing covers overlapping pretraining data, similar fine-tuning recipes, and similar preference training. Models with a common upbringing tend to share misconceptions and share the same blind spots, which is consistent with the finding that shared architectures and providers raise correlation.

Shared exposure is the mechanism most specific to deployed agents and the easiest to overlook. If the checker reads the same retrieved document, the same tool output, or the same conversation as the generator, then a manipulation planted in that content reaches both. Instructions hidden in data can steer the checker exactly as they steered the generator, so the two are not independent even if their weights share nothing.

Shared framing arises when the generator decides what the checker sees. If the checker is shown the generator's own summary of what it did, rather than the action itself, the checker inherits the generator's framing and any omissions in it. The check then measures agreement with a description, not correctness of an action.

1. Shared difficulty: everyone finds the same inputs hard.
2. Shared upbringing: overlapping data and training recipes produce overlapping blind spots.
3. Shared exposure: the checker reads the same untrusted inputs as the generator.
4. Shared framing: the generator controls what evidence the checker sees.

What each kind of diversity actually buys

Diversity is not one dial. A second check can differ from the first along several dimensions, and each dimension addresses some correlation mechanisms but not others. The table below is an analytical map, not a measured ranking; its purpose is to make the choice explicit.

The pattern is that diversity in what the checker is and what it sees buys more independence than diversity in who supplied it. A deterministic checker, such as a test, a type system, a schema validator or a policy engine, fails for entirely different reasons than a model does, and it cannot be argued with by text in its input. A human reviewer brings context no model has. A model from a different vendor helps against shared upbringing, but the evidence suggests that for strong models this help may be smaller than expected, and it does nothing at all against shared exposure if the checker reads the same inputs.

Measure coincident failure before you rely on a pair

Because independence cannot be assumed, it has to be estimated, and the estimate has to be made on the inputs that matter. The method below is deliberately simple so that any team can run it.

First, assemble an evaluation set with known correct outcomes, deliberately enriched with the hard and adversarial cases the deployment will face, including content carrying hidden instructions. Keep these slices separate, because correlation on easy inputs says little about correlation where it counts. Second, run the generator and the checker exactly as deployed, with the same inputs and the same evidence path, and record every case where each one errs. Third, compute the checker's conditional miss rate: of the cases where the generator was wrong, what fraction did the checker also get wrong? Compare it with the checker's ordinary miss rate across all cases. If the two are close, the checker behaves roughly independently. If the conditional rate is much higher, the checker tends to fail exactly when it is needed, and the pair is weaker than the arithmetic suggests.

Report the conditional miss rate per slice, with the number of cases behind it, and treat it as a property of the pair, the configuration and the evidence path, not of either model alone. Re-measure whenever any of them changes, including a model version update, since the evidence shows correlation is a moving target. Chance-adjusted similarity measures such as the one proposed by Goel and colleagues offer a more refined alternative when probability outputs are available (Goel et al., 2025).

Design rules for checks that are genuinely independent

The mechanisms and the measurement method lead to a short set of design rules. They are presented as synthesis; each follows from one of the mechanisms above.

Separate the inputs. A checker should see the proposed action, the policy it must satisfy, and only the context it needs to judge them, not the untrusted documents, web pages or tool outputs that the generator read. This is the only remedy for shared exposure, and it is cheap compared with the risk it removes.

Show the action, not the story. The checker should inspect the actual artefact, such as the diff, the query, the transaction or the command, rather than the generator's description of it. This removes shared framing.

Prefer a different kind of checker before a different brand of checker. Where the property can be checked deterministically, a test, a schema, a type check or a policy rule should come first, because it fails for different reasons and cannot be persuaded. Where judgement is needed, a model checker adds value on top, measured as above.

Escalate disagreement, and audit agreement. When generator and checker disagree, route the case to a person. When they agree, sample some of those cases for human review anyway, because silent agreement on a wrong answer is precisely the failure that correlation produces.

Nested black boxes need an anchor

A system in which one model watches another, and a third model watches the watcher, can look robust on a diagram. If every layer is a capable model drawn from the same generation, trained on overlapping data and reading overlapping inputs, the layers are likely to share failure modes, and the stack becomes a set of nested black boxes that agree with one another for the same reasons. Adding layers of the same kind adds cost and confidence faster than it adds protection.

The way out is to anchor the stack in at least one check that does not share the models' failure profile: a deterministic rule that the models cannot argue with, or a person with independent context. Model checkers remain useful above that anchor, for the judgements that rules cannot express, but their contribution should be measured rather than assumed, and no part of the safety case should rest on the assumption that two strong models will not be wrong together.

Limitations and threats to validity

The empirical findings cited here come from specific benchmarks, leaderboards and tasks; correlation in a particular deployment may be higher or lower, which is exactly why the article recommends measuring it. The multiversion results concern conventional software and teams of programmers, and the analogy to models, while close in mechanism, is an argument rather than a measurement.

The diversity table is an analytical map; it should be read as a set of hypotheses to test, not as established rankings. A deterministic checker covers only the properties it can express, so its independence comes with narrower coverage. Human reviewers are not immune to framing or to manipulation hidden in content, so human review is a partial, not complete, remedy for shared exposure.

Finally, conditional miss rates estimated on small adversarial slices carry wide uncertainty. Reporting the number of cases behind every estimate is part of the method, not an optional extra.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

cases = load_labelled_cases(slices=["routine", "hard", "adversarial-canary"])
for slice in cases.slices:
    gen_wrong = 0; both_wrong = 0; checker_wrong = 0
    for case in slice:
        action = generator.run(case.inputs)            # exactly as deployed
        verdict = checker.review(evidence_for(action))  # same evidence path as production
        g_err = not case.is_correct(action)
        c_err = verdict.approve if g_err else not verdict.approve
        checker_wrong += c_err
        if g_err:
            gen_wrong += 1
            both_wrong += (verdict.approve)              # checker let a wrong action through
    report(slice.name,
           cases=len(slice),
           checker_miss_rate=checker_wrong / len(slice),
           conditional_miss_rate=both_wrong / max(gen_wrong, 1),
           generator_errors=gen_wrong)

Glossary

Independence (of failures)

The property that one component's failure tells you nothing about whether another will fail on the same input.

N-version programming

Building several versions of a program from one specification by separate teams and combining their outputs by vote (Avizienis, 1985).

Coincident failure

Two or more components failing on the same input.

Design diversity

Deliberately building redundant components differently so that they are less likely to fail together.

Conditional miss rate

The fraction of cases in which a checker fails, among only the cases where the component it checks has already failed.

Shared exposure

A generator and its checker reading the same untrusted inputs, so a manipulation in those inputs can reach both.

Model-as-judge

Using a language model to grade, rank or approve the output of another model.

Deterministic checker

A check whose verdict is fixed by explicit rules, such as a test, schema, type system or policy engine, rather than by a model's judgement.

References

1. Avizienis, The N-Version Approach to Fault-Tolerant Software, IEEE Transactions on Software Engineering, 1985
2. Knight and Leveson, An Experimental Evaluation of the Assumption of Independence in Multiversion Programming, IEEE Transactions on Software Engineering, 1986
3. Eckhardt and Lee, A Theoretical Basis for the Analysis of Multiversion Software Subject to Coincident Errors, IEEE Transactions on Software Engineering, 1985
4. Littlewood and Miller, Conceptual Modeling of Coincident Failures in Multiversion Software, IEEE Transactions on Software Engineering, 1989
5. Kim, Garg and Peng, Correlated Errors in Large Language Models (arXiv 2506.07962, 2025)
6. Goel et al., Great Models Think Alike and this Undermines AI Oversight (arXiv 2502.04313, 2025)
7. Panickssery, Bowman and Feng, LLM Evaluators Recognize and Favor Their Own Generations (arXiv 2404.13076, 2024)
8. Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv 2306.05685, 2023)