Read this first
Known-answer tests are necessary but not sufficient — they exercise correct-input round-trips only. Sieve exercises malformed, out-of-bounds, and edge-case inputs, each test tagged to a real bug class. Run it when you first integrate a library, in CI, and before each release.
A conformance harness is not a unit-test suite for one library. It exercises any implementation against a curated set of test categories, each targeting a specific class of bug we have observed in real-world audits or the public literature.
What conformance testing does and does not give you
- It detects known bug classes — it cannot find an unknown one.
- It validates a primitive, not the system that uses it. A correct ML-KEM can still be misused at the protocol layer.
- It complements an audit; it does not replace one. Audits look at design and integration.
When to run it
Run conformance tests at three points: once when you first integrate a post-quantum library to establish a baseline, in CI on every commit that touches the cryptographic dependency, and once before each release against the release artefact rather than a development build.
Implementations that pass KATs but fail conformance
We have seen ML-KEM implementations pass every NIST (ACVP) known-answer vector yet mishandle inputs the vectors never exercise — e.g. accepting a wrong-length ciphertext that FIPS 203 §7.3's type check requires them to refuse, or turning implicit rejection into an explicit error (which is itself nonconformant: a well-formed-but-invalid ciphertext must yield a pseudo-random secret, not an error). KAT/ACVP vectors exercise only honest inputs; conformance to them is necessary but not sufficient.
Concretely, the battery encodes the FIPS-anchored checks the vectors skip: the §7.2 encapsulation-key modulus check (reject a coefficient at or above q), the §7.3 ciphertext type check, implicit-rejection behaviour on invalid ciphertexts, and the exact byte sizes (ML-KEM-768: encapsulation key 1184 B, ciphertext 1088 B). When we find a bug we missed — an out-of-range coefficient slipping through, say — it becomes a new test, and the framework grows sharper over time.
The wiring is deliberately small: Sieve speaks a simple stdin/stdout JSON-line protocol, so the same battery runs against an implementation in any language behind a thin shim.
References
- NIST standards: FIPS 203 / ML-KEM (the §7.2/§7.3 input checks the battery encodes) and FIPS 204 / ML-DSA.
- NIST validation: the ACVP project, its ML-KEM and ML-DSA vector specs, and the CAVP (what formal validation is; what a private battery is not).
- Adversarial vectors: C2SP / CCTV ML-KEM — community edge-case vectors targeting these bug classes.
- Background: Key encapsulation mechanism (Wikipedia), and the Fujisaki–Okamoto transformation (eprint 2017/604) — why implicit rejection exists.