Also mentioned
RSARSARivest, Shamir and AdlemanA widely used public-key algorithm for encryption and digital signatures whose security relies on the difficulty of factoring large numbers.Read the full entry (new tab), ML-KEMML-KEMModule-Lattice-based Key Encapsulation MechanismModule-Lattice-Based Key-Encapsulation Mechanism, the NIST-standardized post-quantum KEM derived from CRYSTALS-Kyber and specified in FIPS 203.Read the full entry (new tab), ML-DSAML-DSAModule-Lattice-based Digital Signature AlgorithmModule-Lattice-Based Digital Signature Algorithm, the NIST-standardized post-quantum signature scheme derived from CRYSTALS-Dilithium and specified in FIPS 204.Read the full entry (new tab), KEMKEMkey encapsulation mechanismA public-key mechanism for securely establishing a shared secret key: the sender encapsulates a random secret to the recipient's public key, and the recipient decapsulates it with their private key.Read the full entry (new tab), TLSTLSTransport Layer SecurityTransport Layer Security, the protocol that encrypts and authenticates most internet traffic, including HTTPS. It uses key exchange, certificates, and symmetric encryption to protect a session.Read the full entry (new tab), FIPSFIPSFederal Information Processing StandardFederal Information Processing Standards, publicly announced standards developed by NIST for use in U.S. government computer systems, including cryptographic algorithms and modules.Read the full entry (new tab) are defined in the glossary.
Read this first
Open Quantum Safe published oqs-provider 0.12.0-rc2 on 16 September 2026. The prerelease packages fixes for a use-after-free reached through public OpenSSL APIs, malformed hybrid-key lengths, an RSA-hybrid public-key reconstruction overflow, and allocation-failure cleanup. These are provider implementation bugs, not breaks in ML-KEM or the hybrid construction. The project also says oqs-provider is not meant for production use.
The most important item in the 0.12.0-rc2 release is easy to miss inside a long changelog. A caller could replace a key object's encoded public key, causing oqs-provider to free its private-key buffer while leaving component pointers aimed at the released memory. A later KEM decapsulation checked the stale component pointer, then passed it into the cryptographic core as though the private key still existed.
The fix in PR #829 clears the component pointers when the private buffer is released, uses the matching secure-heap free routine, and adds private-key guards in both pure post-quantum and hybrid decapsulation helpers. Its regression test first creates a valid ciphertext, replaces the key with public-only material, then verifies that decapsulation fails cleanly. The pull request reports a fail-before and pass-after result under AddressSanitizer for both a pure FrodoKEM path and an X25519 plus FrodoKEM hybrid.
Four fixes, four different trust boundaries
| Area | Failure before the fix | New check |
|---|---|---|
| KEM decapsulation | Component pointers could survive after the private-key buffer was freed | Clear aliases and require a live private key before decapsulation |
| RSA-hybrid key import | A reconstructed public key could be written before its encoded length was checked | Measure the DER length first, reject a mismatch, then write |
| EC hybrid KEM import | An untrusted length prefix could disagree with the fixed component layout | Require the encoded classical length to equal the algorithm's fixed length |
| Key allocation failure | Out-of-memory paths could double-free, dereference null, or leak related allocations | Inject allocation failures and clean each partial state once |
The table groups bugs that should not be collapsed into one severity claim. The use-after-free and overflow involve attacker-influenced API or key-import paths. The allocation cleanup fixes arise when memory allocation fails. The release notes list them together because they land in the same candidate, not because they have identical reachability or impact.
The RSA-hybrid overflow was a check-after-write bug
The RSA-hybrid fix in PR #810 covers rsa3072_falcon512, rsa3072_falconpadded512, and rsa3072_mldsa44. When oqs-provider reconstructed the classical public key from attacker-supplied PKCS #8 material, it serialized the result into a fixed-size buffer and compared the returned length only after the write. An oversized RSA modulus could therefore make the serializer cross the buffer boundary before the mismatch was noticed.
The patch asks OpenSSL for the encoded length with a null output pointer first. Only the expected 398-byte RSA-3072 public key proceeds to the write, followed by a defensive second length check. The pull request reports two malformed inputs rejected before any write and a legitimate RSA-3072 key accepted. That evidence is specific to the reconstructed RSA component. It does not establish a general vulnerability in ML-DSA, Falcon, or hybrid signatures.
Hybrid encodings need structural validation
A hybrid key is not just two byte strings placed side by side. Code that calculates component pointers from a fixed algorithm layout must reject an encoded length that claims a different layout. PR #814 adds that equality check to both EVP_PKEY_fromdata() imports and SubjectPublicKeyInfo decoding for provider-exposed EC hybrid KEMs.
The regression coverage exported classical, post-quantum, and composite forms, then changed the classical-length prefix and required malformed imports to fail. The authors report 38 hybrid KEM variants tested with encoders enabled, plus sanitizer runs. The practical lesson extends beyond this provider: a cryptographic inventory should record formats and import paths, not only algorithm names. A correct algorithm can still sit behind unsafe state transitions or an ambiguous encoding boundary.
Do not turn a release candidate into a production recommendation
The tag is explicitly a prerelease, has no attached release assets, and points users to build and test instructions. oqs-provider's security policy says the project is not meant for productive use. Treat rc2 as a test target and a source of engineering evidence, not as a reason to replace a maintained production provider.
What teams using oqs-provider should do
- Pin the exact provider, liboqs, and OpenSSL revisions in every lab result. The available algorithms and provider overlap change with the OpenSSL version.
- Move testing to 0.12.0-rc2 or a verified descendant before exercising imported hybrid keys or KEM decapsulation. Do not infer a patch from a branch name.
- Add malformed-key and state-transition cases to the test plan: public-only replacement, incorrect component lengths, oversized classical material, and allocation failure.
- Run the tests under AddressSanitizer and UndefinedBehaviorSanitizer where the build permits it. A successful handshake alone does not exercise import and cleanup paths.
- Prefer native standardized algorithms in a maintained OpenSSL release when they cover the required deployment. Use oqs-provider for prototyping and interoperability work within its stated limits.
Our guide to enabling post-quantum TLS now separates native OpenSSL deployment from oqs-provider experiments and links this candidate. The hybrid TLS migration guide covers staged rollout, while the end-to-end test method keeps negotiation evidence separate from library support. None of those checks should skip the key-import and object-lifecycle paths exposed by this release.
A useful warning from a healthy prerelease
Finding several defects in a prerelease does not show that post-quantum cryptography is uniquely unsafe. It shows why experimental integrations need adversarial tests around ordinary software boundaries: length prefixes, ownership, allocator pairing, partial initialization, and API state changes. FIPS 203 defines ML-KEM; it cannot make a provider's surrounding C code memory-safe.
The useful outcome of 0.12.0-rc2 is not a claim that every listed bug is exploitable in a deployed system. It is a patch set with reproducible tests and clearly stated limits. Teams can use that evidence to improve their own post-quantum audit scope before experimental code reaches a production boundary.
References
- Open Quantum Safe: oqs-provider 0.12.0-rc2 release, published 16 September 2026. Prerelease status, included changes, compatibility notes, and links to fixes.
- oqs-provider PR #829: fix KEM decapsulation use-after-free. Root cause, public API sequence, regression test, sanitizer trace, and merged fix.
- oqs-provider PR #810: fix RSA-hybrid reconstruction overflow. Check-before-write patch and malformed-key test evidence.
- oqs-provider PR #814: reject inconsistent classical lengths. Import-boundary validation and hybrid KEM regression coverage.
- oqs-provider security policy. Supported-version table, project scope, and explicit non-production limitation.