Encrypted PKCS#8 PEM private key incompatbile with OpenSSL v3

[dwosk]

Reproduction:

use rsa::pkcs8::{EncodePrivateKey, EncodePublicKey, LineEnding};
use rsa::{RsaPrivateKey, RsaPublicKey};
use std::io::Write;

fn main() {
    let mut rng = rand::thread_rng();
    let bits = 2048;
    let priv_key = RsaPrivateKey::new(&mut rng, bits).expect("failed to generate a key");
    let priv_key_pem = priv_key
        .to_pkcs8_pem(LineEnding::default())
        .expect("failed to convert private key to PEM");
    let pub_key = RsaPublicKey::from(&priv_key);

    let priv_key_encrypted = priv_key
        .to_pkcs8_encrypted_pem(&mut rng, "foo".as_bytes(), LineEnding::default())
        .expect("failed to convert private key to encrypted PEM");

    let pub_pem = pub_key
        .to_public_key_pem(LineEnding::LF)
        .expect("failed to convert public key to PEM");

    std::fs::File::create("id_rsa_test")
        .unwrap()
        .write_all(priv_key_encrypted.as_bytes())
        .unwrap();
    std::fs::File::create("id_rsa_test.pub")
        .unwrap()
        .write_all(pub_pem.as_bytes())
        .unwrap();
}

$ openssl rsa -in ./id_rsa_test -out id_rsa_test_decrypted
Enter pass phrase for ./id_rsa_test:
Could not read private key from ./id_rsa_test
C0FA00FE01000000:error:1608010C:STORE routines:ossl_store_handle_load_result:unsupported:crypto/store/store_result.c:151:
C0FA00FE01000000:error:030000AC:digital envelope routines:scrypt_alg:memory limit exceeded:providers/implementations/kdfs/scrypt.c:521:
C0FA00FE01000000:error:030000AB:digital envelope routines:PKCS5_v2_scrypt_keyivgen_ex:illegal scrypt parameters:crypto/asn1/p5_scrypt.c:260:

$ openssl version                                         
OpenSSL 3.1.3 19 Sep 2023 (Library: OpenSSL 3.1.3 19 Sep 2023)

I think the root cause was found/fixed here: RustCrypto/formats#1205. However, as I'm new to these libraries, I'm unsure how to use the new constructors and piece them together with this crate.

Is there sample code I can leverage that uses the openssl-compatible scrypt parameters to generate the pkcs8 pem key? Is this supported in the latest pre-release? Currently I am using:

rsa = { git = "https://github.com/RustCrypto/RSA", features = ["pkcs5"] }

Thanks!

[tarcieri]

It looks like EncryptedPrivateKeyInfo::encrypt needs to be changed to use pbes2::Parameters::recommended() which was introduced in RustCrypto/formats#1205

See the current implementation here: https://github.com/RustCrypto/formats/blob/c501837/pkcs8/src/encrypted_private_key_info.rs#L77

Right now it's using scrypt::Params::default, i.e. the OWASP recommended defaults which are unfortunately "Too Secure For OpenSSL To Handle". See also RustCrypto/password-hashes#388

[dwosk]

Thanks @tarcieri. Would you accept a PR that changes that method to use recommended() over default()? Since the new behavior would be considered "less secure" than what is being used today I'm not sure how you want to handle that. But if they are the recommended parameters then it makes sense to use them.

[tarcieri]

There is currently no Default impl for pbes2::Parameters.

pbes2::Parameters::recommended() takes an explicit RNG parameter to initialize the salt and cipher IV parameters.

We could potentially add a Default impl gated on a getrandom feature, which calls pbes2::Parameters::recommended with &mut OsRng. But I'm not sure it's a good idea semantically for a Default impl to use an RNG and return different values every time.

[tarcieri]

Fixed in RustCrypto/formats#1430