Investigation as a Pipeline: Signal Triage, Evidence Gathering, and Hypothesis Testing

Every automated containment decision rests on an implicit claim: we know enough to act. The danger is that this claim is usually made by intuition — a responder or a model pattern-matches an alert to a threat and reaches for a control. Intuition is fast, but it is unmeasurable, unauditable, and prone to confirmation bias, and at machine speed those flaws compound into false containments that disrupt healthy systems. The alternative is to treat investigation as what it actually is: a data pipeline that transforms raw signals into a defensible verdict through explicit stages, each with a defined input, output, and quality metric. Structured this way, investigation becomes measurable, reproducible, and — crucially — automatable, because a machine can execute a well-specified pipeline far faster than a human can execute an intuition. This article specifies that pipeline in three stages — signal triage, evidence gathering, and hypothesis testing — and argues that the single most important discipline running through all three is falsifiability: an explanation you act on must be one you actively tried, and failed, to prove wrong.

Why investigation should be a pipeline

The reason to impose a pipeline structure is not bureaucratic tidiness; it is that a pipeline is measurable and a black box is not. When investigation is an undifferentiated act of judgment, you cannot say where it went wrong, cannot reproduce it, and cannot improve it except by hiring better intuition. When it is a sequence of stages, each with a defined input and output, you can measure the yield and error of each stage independently — how many signals survived triage, how complete the gathered evidence was, whether the surviving hypothesis was actually the correct one. This is the data-science instinct applied to security operations: name the transformation, measure it, and you can improve it.

There is a second reason specific to automation. A machine cannot execute intuition, but it can execute a pipeline. Decomposing investigation into stages with clear contracts lets you automate the stages that are safe to automate — high-volume triage, mechanical evidence collection — while keeping a human on the stages that need judgment. The NIST Computer Security Incident Handling Guide (SP 800-61 Rev. 2) already frames incident response as a lifecycle of detection, analysis, containment, and recovery; the pipeline in this article is a magnification of the analysis phase, the part where a raw signal becomes a justified conclusion. Getting that phase right is what separates a responder that contains real threats from one that reacts to noise.

Stage one: signal triage

Triage is the funnel that turns a flood of raw signals into a small set of candidate incidents worth investigating. Its input is everything the environment emits — alerts, anomalous tool-call patterns, policy-violation events, behavioral drift in an agent. Its output is a ranked shortlist of candidates. Its quality metric is precision at the top of the ranking: of the candidates triage surfaces for deeper investigation, how many are genuinely worth the effort. The hard constraint is volume — an autonomous agent fleet generates far more signal than any human, or any downstream stage, can examine, so triage must aggressively narrow without discarding the true incidents hidden in the noise.

The workhorse of triage is anomaly detection — flagging behavior that deviates from an established baseline of normal — a field surveyed comprehensively by Chandola, Banerjee, and Kumar (2009), whose taxonomy of point, contextual, and collective anomalies maps directly onto agent behavior. A point anomaly is a single out-of-range action; a contextual anomaly is an action normal in general but abnormal in this context; a collective anomaly is a sequence that is individually unremarkable but jointly suspicious — precisely the pattern of a slow, multi-step compromise. The critical design choice is where to set the sensitivity, because triage sits on a precision-recall trade-off: too sensitive and you drown the pipeline in false candidates that waste evidence-gathering effort; too specific and you miss real incidents entirely. Triage should favor recall at the very top and precision below it — surface the clear cases confidently, and hand the ambiguous middle to evidence gathering rather than deciding prematurely.

Stage two: evidence gathering

Evidence gathering assembles, around each candidate, the facts needed to explain it. Its input is a candidate incident; its output is a structured case file — the relevant logs, the agent's recent tool calls, the provenance of the data it handled, the identities and capabilities involved. Its quality metric is completeness: does the case contain enough to distinguish the plausible explanations from one another? The stage is largely mechanical, which is exactly why it is a strong candidate for automation — collecting logs and correlating events across sources is tedious, error-prone human work and fast, consistent machine work.

Two properties make gathered evidence trustworthy rather than merely voluminous. The first is provenance: every piece of evidence must carry a verifiable record of where it came from and that it has not been tampered with, or a compromised source can feed the investigation false facts and steer it to a wrong conclusion. The second is chain of custody — an unbroken, tamper-evident record that the evidence collected is the evidence evaluated, a principle the NIST incident-handling guidance emphasizes for exactly this reason. Without these, evidence gathering can be turned into an attack surface: an adversary who can pollute the logs the responder reads can manufacture a false containment against a rival or hide their own tracks. Automated gathering must therefore collect from authenticated, integrity-protected sources and record the provenance of every fact, so the downstream hypothesis test reasons over evidence it can trust.

Stage three: hypothesis testing

The final stage is where investigation earns the right to a verdict, and it is the stage most often done badly. The wrong way is to form a single explanation and look for evidence that confirms it — the classic confirmation-bias failure, which at machine speed becomes an automated confirmation-bias machine. The right way is to form multiple competing hypotheses and attempt to falsify each. A hypothesis is a specific, testable explanation for the candidate: this agent was prompt-injected, or its credentials were stolen, or this is benign behavior triggering a noisy detector. For each, the investigator asks: what evidence would prove this wrong? — and checks the case file for it. The hypothesis that survives every falsification attempt, and whose rivals do not, is the verdict.

The discipline here is drawn straight from the scientific method, and its power is that it makes the investigation's confidence explicit and defensible. A hypothesis that survived a serious falsification attempt supports a confident, possibly autonomous, response; one that merely was not disproven for lack of evidence does not, and should route to a human or trigger more gathering. This is also where the benign hypothesis earns its keep — always including this is a false positive as an explicit competing hypothesis is the single most effective guard against false containment, because it forces the investigator to find evidence that the behavior is actually malicious rather than assuming it. An investigation that never seriously entertained the benign explanation has not ruled it out; it has ignored it.

The pipeline as a measurable whole

Composed, the three stages form a pipeline whose end-to-end quality decomposes into per-stage metrics you can measure and improve independently. Triage precision tells you what fraction of candidates were worth investigating. Evidence completeness tells you whether the case files contained enough to decide. Hypothesis-test accuracy tells you whether the surviving explanation matched ground truth when it was later established. A drop in end-to-end accuracy can be attributed to a specific stage — noisy triage flooding the pipeline, incomplete evidence forcing guesses, or a hypothesis test that confirmed rather than falsified — and fixed there, rather than by vaguely improving the whole.

This decomposition is what makes safe automation possible, because it lets you automate stage by stage with a measured error budget for each. Triage and evidence gathering, being high-volume and mechanical, are automated first and measured against their precision and completeness metrics. Hypothesis testing, requiring judgment, is automated last and most cautiously — and even when automated, its output feeds the reversibility-and-confidence decision from response design, so a low-confidence verdict never drives an irreversible action. The pipeline and the response ladder connect here: the confidence a hypothesis test reports is exactly the confidence the containment policy consumes. An investigation pipeline that does not report calibrated confidence cannot safely feed an autonomous responder, because the responder has no honest signal of how much to trust the verdict.

Automating the pipeline without automating the bias

Automating investigation carries a specific risk that a rigorous treatment must name: automation can industrialize a bad method. A pipeline that automates confirmation bias produces wrong verdicts faster and more consistently than a human ever could, and feeds them to a machine-speed responder — a compounding failure. The safeguard is to bake the falsifiability discipline into the automated hypothesis test itself: require the automated investigator to generate competing hypotheses including the benign one, to search for disconfirming evidence for each, and to report which falsification attempts it actually ran. An automated verdict that cannot show its falsification work should be treated as low-confidence by construction.

The human's role shifts accordingly, mirroring the relocation seen across autonomous response. The human does not run each investigation; the human designs the pipeline, tunes the triage sensitivity, defines what constitutes complete evidence for each incident class, curates the hypothesis library, and audits the automated verdicts — especially the ones that drove a containment action. This is the same relocation of judgment from execution to oversight, applied to investigation: the person moves from reading logs to ensuring the machine reads them correctly and reasons over them honestly. A program that automates the pipeline without this oversight has not sped up investigation so much as scaled up whatever biases its detectors and hypotheses encode.

function investigate(candidate, case_file):
    hypotheses = generate_rivals(candidate)         # MUST include benign
    survivors = []
    for h in hypotheses:
        disconfirming = search_evidence(case_file, refutes=h)
        h.tested = True
        if disconfirming is None:                    # not falsified
            survivors.append(h)
    if len(survivors) != 1:
        return VERDICT(explanation=None, confidence=LOW,
                       reason="ambiguous — escalate or gather more")
    v = survivors[0]
    confidence = calibrate(v, attempts=falsification_log(hypotheses))
    # confidence feeds the response ladder; it never acts directly
    return VERDICT(explanation=v, confidence=confidence,
                   falsification_work=falsification_log(hypotheses))

Limits and honest caveats

The pipeline framing is powerful but has real boundaries. First, it is only as good as its hypothesis library and its detectors: an attack for which no hypothesis exists, executed with behavior no detector flags, passes through triage and is never investigated. The pipeline structures reasoning over the signals you have; it does not conjure signals you lack, and a novel threat can be invisible to it. Second, calibrated confidence is genuinely hard — an automated hypothesis test that reports overconfident verdicts is more dangerous than one that abstains, because the responder trusts the number. Confidence calibration must itself be measured against outcomes and corrected, or the whole safety argument for autonomous response rests on a lie.

Third, this is an analytical framework, not a benchmark. The claim that structuring investigation as a measurable pipeline improves it is grounded in the logic of decomposition and the anomaly-detection literature, not in a controlled study of this specific pipeline, and teams should validate the per-stage metrics in their own environment. Finally, the pipeline reduces but does not eliminate the risk of a confident wrong verdict — a sophisticated adversary who understands the hypothesis library can craft an attack that survives falsification by mimicking a benign explanation. Falsifiability makes the investigation honest about what it tested; it cannot guarantee it tested the right things. The pipeline is the disciplined floor beneath investigation, not a ceiling of certainty, and it must be paired with the reversibility and blast-radius controls that assume some verdicts will be wrong.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

case = mock_case(signals=["unusual_tool_call"], evidence=[...])
v = investigate(candidate(case), case)

# the benign hypothesis must always be among those tested
assert "false_positive" in [h.name for h in v.tested_hypotheses]
# an ambiguous case must abstain, not guess
ambig = mock_case(evidence=[])   # nothing to falsify with
assert investigate(candidate(ambig), ambig).confidence == LOW
# a clearly-benign case must not produce a malicious verdict
benign = mock_case(evidence=["matches_known_good_pattern"])
assert investigate(candidate(benign), benign).explanation != "malicious"

triage:
  point_anomaly:      { threshold: 0.85, on_pass: candidate }
  contextual_anomaly: { threshold: 0.75, on_pass: candidate }
  collective_anomaly: { threshold: 0.60, on_pass: candidate }  # slow multi-step
  ambiguous_band:     { range: [0.40, threshold], route: gather_more_evidence }
ordering: rank_by_confidence_desc
never_drop: true    # low-scoring true incidents go to a review queue, not /dev/null

Glossary

Signal triage

The stage that narrows a flood of raw alerts to a ranked shortlist of candidate incidents worth investigating.

Anomaly detection

Flagging behavior that deviates from an established baseline of normal; the workhorse of triage.

Point / contextual / collective anomaly

A single out-of-range action; an action abnormal only in context; a sequence suspicious only jointly.

Evidence gathering

Assembling a provenance-preserving case file of the facts needed to explain a candidate incident.

Provenance

A verifiable record of where a piece of evidence came from and that it has not been tampered with.

Chain of custody

An unbroken, tamper-evident record that the evidence collected is the evidence evaluated.

Hypothesis testing

Forming competing, falsifiable explanations and attempting to disprove each until one survives as the verdict.

Calibrated confidence

A verdict's stated certainty that matches its real accuracy, so a responder can honestly decide how much to trust it.

References

1. NIST SP 800-61 Rev. 2, Computer Security Incident Handling Guide (2012)
2. Chandola, Banerjee & Kumar, Anomaly Detection: A Survey (ACM Computing Surveys, 2009)
3. NIST AI 100-1, Artificial Intelligence Risk Management Framework (AI RMF 1.0)
4. OWASP Agentic Security Initiative, Agentic AI Threats and Mitigations (2025)
5. MITRE ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge)
6. MITRE ATLAS (Adversarial Threat Landscape for AI Systems)
7. NIST SP 800-53 Rev. 5, Security and Privacy Controls (Incident Response family)