Proving It Is the Right Implementation: ACVP, CMVP and Tests in CI

An implementation of Module-Lattice-Based Key-Encapsulation Mechanism (ML-KEM) that gets one byte wrong usually still runs. It produces keys, ciphertexts and shared secrets of the right length; it may even agree with itself perfectly. It simply does not agree with the standard, or with anyone else's implementation, or it agrees on every ordinary input and misbehaves on the malformed ones an attacker will send. That is why post-quantum deployments need evidence, not assurances. Several kinds of evidence exist: the algorithm tests run by the United States National Institute of Standards and Technology (NIST), the module certificates issued under Federal Information Processing Standard (FIPS) 140-3, community edge-case vectors, and the tests you write yourself. Each exercises something real and each leaves something out. This article explains exactly what each one checks, how to read a certificate's true scope, and how to combine them into a gate that every change, including those written by AI coding assistants, must pass.

Four kinds of evidence, four blind spots

The first kind of evidence is algorithm validation. NIST's Cryptographic Algorithm Validation Program (CAVP) tests whether an implementation computes an algorithm exactly as the standard specifies, using the Automated Cryptographic Validation Protocol (ACVP), in which a NIST server sends test cases and checks the answers. ACVP test specifications for ML-KEM and the Module-Lattice-Based Digital Signature Algorithm (ML-DSA) appeared in April 2024, and for the Stateless Hash-Based Digital Signature Algorithm (SLH-DSA) in July 2024. NIST describes algorithm validation as a prerequisite for the second kind of evidence.

The second kind is module validation. The Cryptographic Module Validation Program (CMVP) certifies a complete cryptographic module, a defined unit of software or hardware, under FIPS 140-3, covering how it is built, how it tests itself, how it protects keys and which algorithms it offers in an approved mode. The third kind is edge-case and negative test vectors from outside the programmes, of which Project Wycheproof is the best known; they probe malformed and unusual inputs that conformance tests do not send. The fourth kind is your own integration testing: whether your protocol, key handling and configuration use the algorithm correctly.

None of these is redundant, because each is blind to something the others see. Algorithm validation knows nothing of your protocol. A module certificate says nothing about how you call the module or about versions it does not list. Edge-case vectors cover only the algorithms and cases someone has written. Your own tests rarely examine the primitive's internals.

What algorithm validation actually tests for ML-KEM

The ACVP test specification for ML-KEM is precise about its coverage, and worth reading for that alone. Key generation is tested by giving the implementation a seed and checking the key pair it derives. Encapsulation is tested deterministically: the server supplies the random input and the encapsulation key, and checks the ciphertext and shared secret. Decapsulation is tested with both valid ciphertexts and 'modified ciphertext' cases, where the implementation must follow the implicit-rejection path and still output exactly the right pseudorandom key rather than an error.

The specification also tests the two input checks FIPS 203 defines. In the encapsulation-key check, the server sends keys in which a value has been encoded as larger than the modulus, and the implementation must reject them. In the decapsulation-key check, the server alters the stored hash of the public key inside the private key, and the implementation must reject that too.

Just as usefully, the specification lists what it does not test: the zeroization of intermediate values, the security strength of the random bit generators, and incorrect-length keys. Those requirements still apply; they simply have to be shown some other way, through the module validation, through code review, or through your own tests.

One more limit is easy to overlook. ACVP results describe the implementation as built and run by the vendor's test harness, in a stated configuration. A library recompiled with a different compiler, different optimisation flags or a different set of enabled features is, strictly, a different implementation; that is part of why module validation lists operating environments, and why a source-code checkout that 'contains' validated code is not the validated module.

Signatures: rejection paths and interfaces

For ML-DSA, the ACVP specification tests key generation from seeds, signature verification, and signature generation across every interface an implementation claims: the plain message, an externally supplied message representative called mu, a context string, a pre-hashed message, and signing with or without randomness. Each of these is a separate capability, which is why a module validated for one signing interface is not automatically validated for another.

The signature-generation tests go further than ordinary inputs. ML-DSA signing runs a loop that rejects candidate signatures for four distinct reasons until one passes, and some of those reasons occur rarely enough that random testing might never trigger them. The specification therefore includes known-answer tests designed to hit each rejection reason, one of which can only occur at the smallest parameter set, ML-DSA-44, and tests that force at least thirty-two rejections in a row. The stated reason is instructive: implementations that give up early may leak information about the key.

For SLH-DSA, verification is tested by modifying valid signatures in specific places, the message, the randomiser, the few-time-signature part and the hypertree part, and by making them too long or too short; each must be rejected. The SLH-DSA specification also lists what it cannot test: fresh seeds for each key generation, approved random bit generators, destruction of sensitive data, key validation and the absence of floating-point arithmetic.

Reading a module certificate

A FIPS 140-3 certificate is short and dense. It names the module and its version, the standard, the overall security level, its status and sunset date, a caveat such as 'when operated in approved mode', the operating environments on which it was tested, and a list of algorithms, each tied to an algorithm-validation certificate number. Every one of those fields limits the claim.

Real certificates show why the algorithm list deserves the closest reading. Certificate 5298, for the AWS-LC 3 Cryptographic Module, validated in June 2026 at Level 1 with a sunset date in June 2031, lists ML-KEM key generation and encapsulation-decapsulation, and no ML-DSA. Certificate 5450, for Thales's Luna T7 module in July 2026, lists both ML-KEM and ML-DSA. Certificate 5503, for a Sansec hardware module in August 2026, lists ML-KEM, ML-DSA and SLH-DSA. Modules 'in process' are not validated, however close they are: at the time of checking, the Go Cryptographic Module version that adds ML-DSA and an OpenSSL 3.5-line FIPS provider were both still in review.

Two further details matter at run time. FIPS 140-3 Implementation Guidance 10.3.A requires post-quantum self-tests, known-answer tests when the module starts or before an algorithm is first used and pairwise consistency tests when keys are generated, and NIST updated it in August 2026 for key pairs generated from seeds. And validated modules report an approved-service indicator: BoringSSL's, for example, reports its SHAKE-based SLH-DSA variant as not approved while the SHA2-based one is. Checking that indicator at run time is the only way to know that a particular call stayed inside the validated boundary.

Two more fields deserve attention. Every certificate links to a security policy, the vendor's document describing the approved mode, its configuration steps and its self-tests, which is where you confirm that the exact service you use is approved. And the sunset date is a planning date: after it, the certificate moves to the historical list and new procurement cannot rely on it, so a module chosen today for a long-lived product needs a plan for re-validation or replacement.

Edge cases the programmes do not send

Conformance tests are designed to show that an implementation matches the standard on the inputs the standard describes. Attackers send other inputs. Project Wycheproof, maintained under the C2SP community organisation, publishes test vectors aimed at those other inputs: for ML-KEM, decapsulation tests at all three sizes; for ML-DSA, signing tests in seed and non-seed key forms and verification tests full of malformed signatures. At the time of checking, it had no SLH-DSA vectors, so for that algorithm the ACVP verification modifications are the main negative tests available.

Known-answer files for the final standards come from NIST's ACVP server repository, which publishes the test cases and expected results as JSON files. NIST's older 'intermediate values' files describe pre-final drafts and should not be used to test the final standards. Libraries are starting to help: Go 1.26 added a package that performs derandomised ML-KEM encapsulation specifically for known-answer tests, and a way to set a deterministic global random source in tests.

Because both sets of files are public and stable, they belong in your own repository, pinned by hash, where a build can run them on every change.

Treat the vectors as dependencies. Record the source repository, the commit or release, the date fetched and a hash of each file, so that a later review can prove which vectors a build ran against. Never regenerate expected values from your own implementation, and never trim failing cases from a vector file; if a case is genuinely inapplicable, for example an interface you do not use, exclude it in the test configuration with a written reason.

A conformance gate in continuous integration

Continuous integration (CI), the practice of building and testing every change automatically, is where all of this becomes routine. A practical gate runs six kinds of check on every change that touches cryptography. Known-answer tests from the ACVP files cover every parameter set and every interface the system uses, including context strings, external mu, pre-hashing and ML-KEM's implicit-rejection cases. Negative vectors from Wycheproof and the ACVP modifications must be rejected, or must follow implicit rejection. Key round-trips import and export seed and expanded forms, and cross-library tests sign, verify, encapsulate and decapsulate across the two libraries you depend on. Protocol-level tests confirm that a tampered ciphertext fails at key confirmation and that a missing post-quantum signature is rejected. A run-time check asserts that the approved-service indicator is on and that the module version matches the certificate you cite. And the build fails when any of these drifts.

Constant-time behaviour, whether execution time leaks secrets, is a separate question with its own tools and deserves its own gate; nothing in the list above proves it.

Schedule the gate so it stays affordable. Full ACVP suites, especially SLH-DSA signing cases, can be slow, so a common pattern runs a representative subset on every change and the complete suites nightly and before every release, with the same failure rule for both. The pattern matters less than the property: no release can ship without the complete evidence having passed against the exact build being released.

Every engineering decision leaves a testable trace

Validation programmes prove that a primitive is implemented correctly. They do not judge the decisions made around it, and those decisions are where most post-quantum deployments will actually go wrong. The useful habit is to turn each decision into a test. A protocol that switched from Diffie-Hellman to a key-encapsulation mechanism should have a test that a modified ciphertext fails at key confirmation and nowhere earlier. A choice to keep private keys as seeds should have a round-trip test that exports and re-imports them. A parameter policy should be a configuration lint that fails when a forbidden set appears. A library choice should be a check that the running module version matches a certificate listing the operations used. A signing design should have known-answer tests for its external-mu and context-string interfaces, and a verifier test proving that an artifact without its post-quantum signature is rejected.

Written this way, the evidence for a post-quantum deployment is no longer a stack of certificates; it is a set of checks that run on every change and say plainly which decision broke.

When AI agents write the tests

AI coding assistants now write much of the glue code around cryptography and, increasingly, the tests for it. Two failure patterns recur. The first is a test that compares the code with its own output, for example by generating a key, signing, verifying and calling that a test; it proves only that the implementation agrees with itself, which a wrong implementation will do happily. The second is an agent 'fixing' a failing known-answer test by updating the expected value to match the new output, which turns a detected bug into a hidden one.

The rules that prevent both are simple and mechanical. Expected values come only from NIST and Wycheproof files, pinned by hash in the repository. Those files are read-only to agents, enforced by repository permissions or a check that fails if they change. A failing vector blocks the merge and goes to a person. And agents that operate infrastructure, rather than write code, should check the module version and the approved-service indicator before using a key, and refuse to proceed if either is wrong.

These are the decisions practitioners and assurance owners actually face when turning validation into everyday evidence.

1. Run ACVP known-answer tests for every parameter set and interface in use, including implicit rejection.
2. Add Wycheproof and ACVP negative cases, and note where none exist yet, as for SLH-DSA edge cases.
3. Match module, version, operating environment and operation to a certificate, and check the approved-service indicator at run time.
4. Turn protocol, key-format, policy and signing decisions into tests that run on every change.
5. Keep expected values read-only to agents and send failing vectors to a person.
6. Run a separate constant-time gate; conformance tests do not cover it.

Glossary

ACVP

The Automated Cryptographic Validation Protocol, through which NIST's server sends algorithm test cases and checks the answers.

CAVP

NIST's Cryptographic Algorithm Validation Program, which certifies that an algorithm implementation matches its standard.

CMVP

The Cryptographic Module Validation Program, which certifies whole cryptographic modules under FIPS 140-3.

Known-answer test

A test that feeds fixed inputs to an implementation and compares the output with published expected results.

Negative test vector

A deliberately malformed input that a correct implementation must reject or handle in a specified way.

Approved-service indicator

A signal from a validated module that a particular call used an approved algorithm in an approved way.

Self-test

A check a validated module runs on itself, such as known-answer tests at start-up or pairwise consistency tests at key generation.

Continuous integration (CI)

Building and testing every change automatically before it can be merged.

Further reading

1. ACVP ML-KEM JSON Specification (NIST, usnistgov/ACVP)
2. CMVP Certificate 5298, AWS-LC 3 Cryptographic Module (NIST)
3. FIPS 140-3 Implementation Guidance announcements (NIST)
4. Project Wycheproof (C2SP)