Static-Analysis Confidence Gap · 1 of 10L3data science
What a Clean Static-Analysis Report Actually Proves
A report with no findings is evidence about a bounded search, not a certificate of safety — and for AI-agent software that boundary is unusually narrow.
Abstract
Static analysis is routinely read as a verdict: an empty findings list is treated as a statement that the code is safe. This piece argues that a clean report is instead a statement about the intersection of what an analyzer examined, what it modeled, and which properties are decidably approximable — a set that is provably smaller than 'all vulnerabilities'. We separate the four independent reasons a finding can be absent, formalize the gap between soundness and completeness against the undecidability floor established by Rice, and cast an analyzer as an abstract interpretation whose approximations are directional. The contribution is the Static-Analysis Assurance Envelope: an explicit, reportable boundary around the claims a run can support. The key takeaway is that a clean report proves the absence of the specific weakness classes the analyzer was equipped to find in the code it could see, and nothing beyond that; for AI-agent systems, where behavior is mediated by a model, delegated to tools, and composed at runtime, the envelope excludes most of the interesting risk by construction.
A security dashboard turns green and a decision follows: ship it. The empty findings list feels like a proof of safety, and it is treated as one in change gates, audits, and board reports across the industry. Yet the same run, on the same code, with a different query set or one unmodeled framework, could have turned red — and nothing about the software would have changed. This is the confidence gap: the distance between what a clean static-analysis report is read to mean and what it can actually support. Closing the gap does not require distrusting static analysis; it requires stating precisely what a run proves. A clean report is strong, bounded evidence about a deliberately narrow question, and the discipline of naming that boundary is what separates assurance from wishful thinking.
The claim a clean report is read to make
Static analysis is the examination of a program's source or compiled form without executing it, to infer properties of every possible run. A finding is a reported location where the analyzer believes a property of interest — most often a security weakness — may hold. A clean report is a run that produced no findings for the configured checks. The colloquial reading of a clean report is a universal claim: there are no vulnerabilities in this code. That reading is the source of the confidence gap, because a run does not, and cannot, evaluate that universal claim.
What a run actually evaluates is far narrower. It searches for a fixed catalogue of weakness patterns, expressed as queries or rules, over an abstract model of the program that the analyzer was able to construct, within a time and memory budget, using models of the libraries and frameworks it happens to understand. Each of those clauses is a boundary. The universal reading silently drops all of them and substitutes 'no vulnerabilities' for the true statement, which is closer to 'none of the weakness classes we encoded were found along the program paths we could model'. The remainder of this piece makes that true statement precise and argues it is the only honest thing a clean report proves.
The stakes are practical, not pedantic. When a clean report is treated as a universal safety claim, it becomes a stopping condition: further review is deemed unnecessary, and the classes of risk the analyzer never searched for are silently accepted. The narrower the true envelope relative to the perceived one, the more risk that stopping condition hides — and for software whose security depends on runtime behavior, that gap is at its widest.
Background: soundness, completeness, and the undecidability floor
Two independent properties describe how an analyzer's answers relate to the truth. An analysis is sound for a property when it never misses a real instance: every genuine occurrence in the program is reported, so there are no false negatives. An analysis is complete for a property when it never reports a spurious instance: everything it flags is real, so there are no false positives. These are orthogonal. A tool can be sound and incomplete (it catches every real bug but also cries wolf), complete and unsound (everything it reports is real but it stays silent about much), both, or neither.
The reason no practical analyzer is both sound and complete for interesting properties is not engineering immaturity; it is a theorem. Rice (1953) proved that every non-trivial semantic property of the function a program computes is undecidable — no algorithm can decide it correctly for all programs. Security-relevant properties such as 'this data never reaches this sink' or 'this buffer is never overflowed' are non-trivial semantic properties, so no terminating analyzer can be exactly right about them on all inputs. Every real analyzer must therefore approximate, and approximation has a direction: it either over-approximates (accepting false positives to preserve soundness) or under-approximates (accepting false negatives to preserve completeness).
Cousot and Cousot (1977) gave this approximation a rigorous foundation in abstract interpretation, which models a program's real behavior in a simplified abstract domain and computes a safe over-approximation of what can happen. The framework makes the trade explicit: a sound abstract interpretation guarantees that any behavior the real program exhibits is captured by the abstraction, at the cost of also admitting behaviors the real program never exhibits. The critical consequence for reading reports is directional. A sound analysis that reports nothing is meaningful — if it truly over-approximates and still finds no path to the sink, no such path exists in the model. An unsound analysis that reports nothing carries no such guarantee, because it may simply have under-approximated past the bug.
The four independent reasons a finding is absent
The heart of interpreting a clean report is understanding that a finding can be absent for four distinct reasons, only one of which is the reason practitioners assume. Conflating them is the mechanism of the confidence gap. The first reason is genuine absence: the weakness truly is not present on any real path. This is the reading a clean report is assumed to justify. The other three are boundary effects, and each one produces the identical empty output while meaning something entirely different.
The second reason is not examined: the code was never in the analyzer's view. Files in an unsupported language, code excluded by a path filter, a component behind a build step the extractor did not run, or generated code produced only at deployment — none of it is analyzed, so none of it can produce a finding. The third reason is not modeled: the code was examined, but the analyzer lacked a model of the framework, library, or call mechanism that carries the dangerous data, so the path was invisible to the query even though the query itself exists. The fourth reason is below threshold: the analyzer's approximation deliberately suppressed the result to control false positives, or the specific query for that weakness class was simply not enabled in the configured rule set.
These four are not a ranking; they are independent failure modes that a single empty report cannot distinguish. The practical discipline is to refuse to read absence-of-finding as genuine-absence until the other three have been ruled out for the risk in question. That is precisely the work a decision gate skips when it treats green as done.
Precision and recall answer different questions than a clean report
Because absence has four meanings, the quality of an analyzer must be measured on two axes that the industry often collapses into a single 'accuracy'. Precision is the fraction of reported findings that are real. Recall, also called the true-positive rate, is the fraction of real weaknesses that were reported. They are computed from the same confusion counts but answer opposite questions: precision asks how much of the alarm to believe, recall asks how much of the danger was seen. A clean report is a statement about the report having no positives at all, so on its own it speaks to neither number — an analyzer with near-zero recall produces clean reports on vulnerable code effortlessly.
This is why recall is the number that governs what a clean report proves, and precision is the number that governs whether anyone keeps using the tool. A high-recall analyzer that is silent about a weakness class provides real evidence of that class's absence; a low-recall analyzer that is silent provides almost none. Yet recall is the harder number to know, because measuring it requires a ground-truth set of known vulnerabilities. Curated corpora exist for exactly this purpose: the NIST Software Assurance Reference Dataset and its Juliet Test Suite provide labeled weakness cases keyed to the MITRE Common Weakness Enumeration, and the OWASP Benchmark Project provides an executable application with known true and false positives so precision and recall can be scored on identical ground truth.
The essential caveat, developed in depth by the measurement-focused work elsewhere in this series, is that recall measured on a benchmark is recall against that benchmark's weakness classes and coding styles, not against your code. A benchmark tells you the analyzer can find a category of bug in an idiom it was tested on; it does not tell you the analyzer will find that category in an unmodeled framework or an unusual code shape. Precision and recall are properties of a tool on a distribution, and the distribution of a clean report's target — your specific program — is not the benchmark's.
| Weakness present | Weakness absent | |
|---|---|---|
| Reported | True positive (TP) | False positive (FP) |
| Not reported | False negative (FN) | True negative (TN) |
| Question answered | Recall = TP / (TP + FN) | Precision = TP / (TP + FP) |
The Static-Analysis Assurance Envelope
The four reasons for absence, the directional nature of approximation, and the benchmark caveat can be unified into a single reportable object. Call it the assurance envelope of a run: the explicit boundary around the claims that run can support. It is the intersection of three sets. The first is the examined set: the code the analyzer actually ingested — the languages, files, and build artifacts that reached its internal representation. The second is the modeled set: the behaviors it could reason about — the frameworks, call mechanisms, sources, and sinks for which it had models. The third is the decidable-approximable set: the weakness classes for which an enabled query computes a sound-enough approximation to make absence meaningful.
The claim a run can support is the intersection of these three, and it is provably a subset of the code's true risk surface. Everything outside the envelope — code not examined, behavior not modeled, classes not queried, and properties too undecidable to approximate usefully — is not evidence of safety; it is simply unmeasured. The envelope reframes a clean report from a verdict into a coverage statement: these classes, over this code, under these models, were searched and not found. That statement is genuinely valuable and defensible. Its value comes entirely from being bounded.
The envelope is not a new algorithm; it is a discipline of disclosure, and it composes with the honest reporting format developed later in this series. Its worth is that it converts an implicit, invisible boundary into an explicit, auditable one. A reviewer who is handed the envelope can ask the only questions that matter — was my risk class inside it, and if not, what other assurance covers that class — instead of inferring safety from silence. The narrowness of the envelope is not a defect to hide; it is the precise, honest measure of how much a clean report is worth for a given decision.
Why AI-agent software widens the envelope gap
Static analysis reasons about a program by tracing how data moves through code. That model fits conventional software well, and an AI agent still contains a conventional substrate — servers, parsers, database calls, network clients — where the assurance envelope is as meaningful as it is anywhere. The gap opens where an agent's security stops being a property of the code and starts being a property of behavior the code merely hosts. The OWASP Agentic AI Threats and Mitigations material catalogues risks that live in exactly this space: goals manipulated through content, tools invoked from model output, memory poisoned across sessions, authority delegated at runtime, and harmful plans assembled from individually permitted steps.
Each of these sits outside a conventional envelope for a structural reason, not a temporary one. The decisive transformation in an agent — untrusted content becoming an instruction the model acts on — happens inside the model, which static analysis treats as an opaque function it cannot see through. The choice of which tool to call, with which arguments, is made at runtime from that model output, so the dangerous edge does not exist in the source text to be traced. Authority is exercised through legitimate, already-granted permissions, so no privilege-escalation pattern fires. And the harm often emerges only from the composition of steps that are each locally benign, a property no single-path query is shaped to detect. These are not weaknesses of a particular analyzer; they are behaviors that fall outside the examined-and-modeled intersection by construction.
The honest consequence is not that static analysis is useless for agents — it remains the right tool for the substrate, and its envelope can be deliberately widened with custom models, a subject taken up directly elsewhere in this series. The consequence is that for agent software the default envelope is unusually narrow relative to the perceived one, so a clean report on an agent proves even less about the agent's real safety than the same report would prove about a conventional service. Reading it as a universal safety claim is therefore most dangerous exactly where agents are concerned.
Reading a clean report against a benchmark honestly
If a clean report's worth depends on recall, and recall is a property of a tool on a distribution, then the responsible way to attach a number to a clean report is to state the distribution the number came from. The NIST Software Assurance Reference Dataset, the Juliet Test Suite, and the OWASP Benchmark Project are the appropriate public instruments, because they carry ground truth: known weaknesses keyed to Common Weakness Enumeration classes, and, in the OWASP case, an executable target where both true and false positives are labeled so precision and recall are computed on identical footing.
The interpretive rule is to treat a benchmark score as an upper bound on what a clean report proves for the matching class, never a transfer guarantee. If an analyzer scores high recall for a weakness class on Juliet, a clean report supports the absence of that class in code that resembles Juliet's idioms and uses frameworks the analyzer models. It supports far less for the same class expressed through an unmodeled framework or an unusual code shape, because the benchmark measured the query against test cases, and your program is not a test case. This is the difference between construct validity — did we measure the right thing — and external validity — does the measurement transfer — and a clean report quietly assumes external validity it has not earned.
This is also why assurance frameworks refuse to rest on any single verification activity. NIST SP 800-218, the Secure Software Development Framework, positions automated analysis as one practice among many precisely because each practice has a bounded envelope; the framework's structure is an institutional acknowledgment that a clean report from one tool is a partial control, to be composed with review, testing, and threat modeling rather than substituted for them.
Threats to validity and the limits of this account
This account is a conceptual synthesis, not an empirical study, and it should be read with its own boundaries stated. First, the envelope is a framework for reasoning about claims, not a metric with a published unit; turning examined, modeled, and decidably-approximable into measured quantities for a specific run is the empirical work this series pursues separately, and the framework should not be mistaken for that measurement. Second, the four-reasons decomposition is a clarifying model; a real absent finding can arise from a combination of causes, and the categories can overlap in practice even though they are logically distinct.
Third, the soundness argument is directional and idealized. Real analyzers make deliberate unsound choices — bounding loop iterations, cutting recursion, capping path counts for performance — so even a nominally sound tool ships with unsound approximations that weaken the clean-report guarantee in ways specific to each engine. The claim here is about the logical structure, not about any product's fidelity to it, and a careful reader should ask each analyzer where it chooses unsoundness. Fourth, the treatment of agent-specific risk is deliberately at the level of why the envelope is narrow, not how far it can be widened; the constructive question of extending models to agent frameworks and tools is addressed on its own terms elsewhere in this series and is not resolved here.
The through-line is that none of these limits weakens the central claim; they sharpen it. A clean report proves the absence of the encoded weakness classes along the modeled paths of the examined code, under whatever unsound approximations the engine made, for weakness classes whose approximation is sound enough to make silence meaningful. That is a precise, defensible, and genuinely useful statement. The confidence gap is nothing more than the distance between that statement and the universal one it is mistaken for, and naming the envelope is how the distance is closed.
Key takeaways
- A clean static-analysis report proves the absence of the encoded weakness classes along the modeled paths of the examined code — not the absence of vulnerabilities.
- A finding can be absent for four independent reasons: genuine absence, code not examined, behavior not modeled, or a class not queried; one empty report cannot distinguish them.
- Rice's theorem forbids a sound-and-complete analyzer for non-trivial semantic properties, so every real tool approximates in a direction; only a sound analysis makes absence meaningful.
- Recall, not precision, governs what a clean report proves: a low-recall tool produces clean reports on vulnerable code effortlessly.
- The Static-Analysis Assurance Envelope is the intersection of examined, modeled, and decidably-approximable — a reportable boundary that reframes a clean report as a bounded coverage claim.
- For AI-agent software the envelope is unusually narrow, because the security-critical transformation happens inside the model and at runtime, outside the source paths static analysis traces.
Practitioner Toolkit
Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.
Run before treating an empty findings list as evidence of safety for a given risk.
- Was the at-risk code actually examined (supported language, not path-excluded, not generated only at deploy)?
- Is the framework or call mechanism that carries the dangerous data modeled by the analyzer?
- Is a query for this specific weakness class enabled in the configured rule set, not just available?
- Is the analyzer's recall for this class known from a ground-truth benchmark, and does your code resemble that benchmark's idioms?
- Does the analyzer document where it chooses unsoundness (loop bounds, recursion cuts, path caps) for this analysis?
- For agent code: does the risk depend on model output, runtime tool choice, delegated authority, or step composition — i.e. outside the envelope?
Attach to any clean report so the bounded claim is explicit and auditable.
STATIC-ANALYSIS ASSURANCE ENVELOPE
Examined: languages=[...] files=[included/excluded] build-artifacts=[...]
Modeled: frameworks=[...] unmodeled=[...] sources/sinks=[known set]
Queried: weakness-classes=[enabled CWEs] not-queried=[known gaps]
Soundness: unsound-approximations=[loop bound, recursion cut, path cap]
Evidence: recall-basis=[benchmark + class] transfer-caveat=[stated]
Claim: 'Absence of the queried classes along modeled paths of examined code.'
NOT a claim: anything outside the examined-and-modeled-and-queried intersection
(for agents: model-mediated intent, runtime tool choice, composed plans).The shortest path from a green dashboard to a defensible decision.
- State the run's envelope: examined, modeled, queried — before reading the result as safety.
- Name the top risk classes for this system and check each is inside the envelope.
- For any risk class outside it, assign a different control (review, testing, threat modeling) — never accept silence.
- Record the clean report as a bounded coverage claim, not a verdict, in the change log.
Glossary
- Static analysis
- Examination of a program's source or compiled form without executing it, to infer properties that hold over all possible runs.
- Soundness
- The property that an analysis never misses a real instance of the target property, so it produces no false negatives for that property.
- Completeness
- The property that an analysis never reports a spurious instance, so everything it flags is real and there are no false positives.
- Abstract interpretation
- A framework (Cousot and Cousot, 1977) that reasons about a program in a simplified abstract domain and computes a safe over-approximation of its real behavior.
- Precision
- The fraction of reported findings that are true, TP / (TP + FP); it measures how much of the alarm to believe.
- Recall
- The fraction of real weaknesses that were reported, TP / (TP + FN); it measures how much of the danger was seen.
- Assurance envelope
- The reportable boundary of a run: the intersection of examined code, modeled behavior, and decidably-approximable weakness classes, within which a clean report supports a claim.
- Common Weakness Enumeration
- A product-neutral catalogue maintained by MITRE that names and classifies categories of software weakness.
References
- Cousot & Cousot, Abstract Interpretation: A Unified Lattice Model for Static Analysis of Programs (POPL 1977)
- Rice, Classes of Recursively Enumerable Sets and Their Decision Problems (Trans. AMS, 1953)
- Livshits & Lam, Finding Security Vulnerabilities in Java Applications with Static Analysis (USENIX Security 2005)
- NIST Software Assurance Reference Dataset (SARD) and Juliet Test Suite
- OWASP Benchmark Project
- NIST SP 800-218, Secure Software Development Framework (SSDF) v1.1
- MITRE Common Weakness Enumeration (CWE)
- OWASP Agentic AI — Threats and Mitigations