ML-KEM CMS decryption fails: "Only a ML-KEM-768 private key can be used for unwrapping" even with correct key

[remiblancher]

Title: ML-KEM-768 CMS decryption fails with "Only a ML-KEM-768 private key can be used for unwrapping"

BC Version: 1.83

Description:

BouncyCastle 1.83 cannot decrypt ML-KEM-768 KEMRecipientInfo (RFC 9629), even CMS messages it generates itself.

Error:

org.bouncycastle.cms.CMSException: exception unwrapping key: exception encrypting key: Only a ML-KEM-768 private key can be used for unwrapping
Caused by: java.security.InvalidKeyException: Only a ML-KEM-768 private key can be used for unwrapping
at org.bouncycastle.jcajce.provider.asymmetric.mlkem.MLKEMCipherSpi.engineInit(Unknown Source)

Cross-implementation testing:

Generator ↓ / Decryptor →	OpenSSL 3.6	qpki (Go/circl)	BouncyCastle 1.83
BouncyCastle 1.83	✅ OK	✅ OK	❌ FAIL
OpenSSL 3.6	✅ OK	✅ OK	❌ FAIL
qpki (Go/circl)	✅ OK	✅ OK	❌ FAIL

Root Cause Analysis

The MLKEMCipherSpi.engineInit() checks:

if (key instanceof BCMLKEMPrivateKey)
BC tests only use freshly generated keys (KeyPair.getPrivate()), which return BCMLKEMPrivateKey directly.

When loading from PEM via JcaPEMKeyConverter.getPrivateKey() → KeyFactory.generatePrivate(), the returned key reports ML-KEM-768 as algorithm but may not be an instance of BCMLKEMPrivateKey.

Missing test coverage: No BC tests load ML-KEM keys from PEM/PKCS#8 files before decryption.

To reproduce:

Security.addProvider(new BouncyCastleProvider());
Security.addProvider(new BouncyCastlePQCProvider());

// Generate ML-KEM-768 keypair and self-signed cert
KeyPairGenerator kpg = KeyPairGenerator.getInstance("ML-KEM-768", "BCPQC");
KeyPair kp = kpg.generateKeyPair();
X509Certificate cert = ...; // self-signed cert with ML-KEM-768 public key

// Create AuthEnvelopedData with AES-GCM (RFC 5083 + RFC 9629 recommended)
CMSAuthEnvelopedDataGenerator gen = new CMSAuthEnvelopedDataGenerator();
gen.addRecipientInfoGenerator(new JceKEMRecipientInfoGenerator(cert, CMSAlgorithm.AES256_WRAP)
.setKDF(new AlgorithmIdentifier(NISTObjectIdentifiers.id_alg_hkdf_with_sha256))
.setProvider("BC"));
CMSAuthEnvelopedData cms = gen.generate(
new CMSProcessableByteArray("test".getBytes()),
(OutputAEADEncryptor) new JceCMSContentEncryptorBuilder(CMSAlgorithm.AES256_GCM)
.setProvider("BC").build());

// Decrypt - FAILS
CMSAuthEnvelopedData parsed = new CMSAuthEnvelopedData(cms.getEncoded());
RecipientInformation r = parsed.getRecipientInfos().getRecipients().iterator().next();
r.getContent(new JceKEMEnvelopedRecipient(kp.getPrivate()).setProvider("BC")); // throws!

[tonywasher]

You are mixing implementations of MLKEM. It is available in both the BC and BCPQC factories, but the BC factory is the official standardized version.

You are generating BCKyberPrivateKey from the BCPQC factory, but parsing the certificate will pick up the BC implementation of BCMLKEMPrivateKey since BC is ahead of BCPQC in the search order.

Try removing BCPQC from the testcase.