Read this first
Three separate questions hide behind "is our post-quantum crypto correct?" Does it pass NIST's known-answer test (KAT) vectors? Does it conform to the letter of FIPS 203/204/205, including the reject paths? Is it secure in your deployment? These are three distinct axes, and passing the first proves neither of the others. KATs only exercise honest, well-formed inputs; conformance and security both live in the behaviour KATs never trigger.
When teams adopt ML-KEM or ML-DSA, the natural correctness check is NIST's Automated Cryptographic Validation Protocol (ACVP) test vectors: feed the fixed inputs, compare the outputs, all green, done. That check is necessary but it is not conformance, and conformance is not security. Conflating the three is how nonconformant implementations end up in production carrying a clean bill of health.
What the KATs miss
Known-answer test vectors pin the happy path. They confirm that for a given seed and message the implementation produces exactly the byte string the standard says it should. That catches transcription errors, endianness bugs, and wrong parameter sets. What it cannot catch is everything the standard requires an implementation to do with input it should refuse — because a KAT, by construction, only ever supplies input a conformant implementation accepts.
FIPS 203 is the clearest example. The algorithm is only half the standard; the input checks are normative:
- **§7.2 — encapsulation-key modulus check.** Every coefficient of the encapsulation key must be reduced mod q = 3329. A key with an out-of-range coefficient must be rejected. No honest KAT vector contains such a key, so a build that skips the check passes the vectors regardless.
- **§7.3 — ciphertext type check.** Decapsulation must verify the ciphertext is the correct length for the parameter set before processing it.
- **Implicit rejection.** On an invalid ciphertext, decapsulation does not error — it returns a deterministic pseudo-random shared secret. That is only safe if the reject path is indistinguishable, by timing, from the accept path. A branch that shortcuts on failure is nonconformant and creates an oracle.
ML-DSA has the same shape: verification must reject malformed public keys and signatures rather than crash or silently accept, and it must apply domain separation and the context string correctly — behaviour the standard mandates but the honest vectors never stress. SLH-DSA adds its own encoding and length constraints. In every case the load-bearing behaviour is on the reject path, which is exactly the path KATs cannot reach.
Conformance versus audit
Conformance and audit answer different questions and catch different failures. Conformance asks: does this implementation follow the specification, including every input check and reject path? Audit asks: is it exploitable in context — side channels, RNG quality, key management, memory handling? You want both. A conformant implementation can still be insecure (a perfect §7.2 check leaks nothing, but a data-dependent timing branch elsewhere does); an audited implementation can still be nonconformant (secure in your threat model yet skipping a check that breaks interoperability with a peer that enforces it). Neither subsumes the other.
"It's an audited, popular library" is not evidence of conformance
Adoption and an audit report tell you a project is scrutinized; they do not tell you it conforms to FIPS 203. Real, widely deployed implementations have passed every ACVP vector while omitting the §7.2 modulus check or leaking the implicit-rejection path by timing. Popularity selects for the happy path being right — which is precisely the part KATs already cover and conformance does not depend on.
How conformance testing works
The method is to drive the implementation — in any language — through a small stdin/stdout protocol and feed it adversarial vectors that honest KATs never include: out-of-range coefficients, malformed keys, wrong-length ciphertexts, and boundary encodings, such as those the C2SP/CCTV project publishes for ML-KEM. You then assert the standard-mandated response: rejection where the spec requires it, an indistinguishable deterministic secret on implicit rejection, correct sizes throughout.
- **Wrap, don't rewrite.** A thin harness exercises the real code path you ship, not a reimplementation, so the test reflects production behaviour.
- **Adversarial vectors, not just KATs.** The value is in the inputs a conformant implementation must refuse; sourcing them from a maintained corpus keeps the suite honest.
- **Re-run every release.** Conformance is not a one-time certificate — a dependency bump or optimization can quietly drop a check. Gate releases on it, and emit findings in a machine-readable format like SARIF so they land in your existing review flow.
- **Produce a signed report.** The output is a dated, signed conformance statement you can show auditors, customers, and regulators.
How quantakrypto helps
We run all three axes so none of them hides behind another. We conformance-test the implementation you ship against the FIPS 203/204/205 input checks and reject paths the KATs miss (certification), assess whether it is secure in your deployment (audit), and sequence the change into your rollout without breaking production (migration). The deliverable is a signed conformance report backed by adversarial vectors and re-run on every release — evidence, not a vendor's assurance.