Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

addition of v6 and PQC 23/06 #2126

Merged
merged 20 commits into from
Sep 25, 2023
Merged

Conversation

falko-strenzke
Copy link
Contributor

The branch is a mirror of https://github.com/pqc-thunderbird/rnp for the purpose of making a PR to the upstream RNP repo.

Copy link

@github-advanced-security github-advanced-security bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CodeQL found more than 10 potential problems in the proposed changes. Check the Files changed tab for more details.

@codecov
Copy link

codecov bot commented Sep 11, 2023

Codecov Report

Patch coverage: 72.32% and project coverage change: -7.02% ⚠️

Comparison is base (22f5b19) 83.89% compared to head (d906cb5) 76.87%.
Report is 3 commits behind head on main.

❗ Current head d906cb5 differs from pull request most recent head 737c71f. Consider uploading reports for the commit 737c71f to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2126      +/-   ##
==========================================
- Coverage   83.89%   76.87%   -7.02%     
==========================================
  Files         161      193      +32     
  Lines       32249    37051    +4802     
==========================================
+ Hits        27054    28483    +1429     
- Misses       5195     8568    +3373     
Files Changed Coverage Δ
src/lib/crypto.cpp 72.16% <ø> (ø)
src/lib/crypto/ec.cpp 76.92% <ø> (ø)
src/lib/crypto/ecdh.cpp 81.11% <ø> (+0.53%) ⬆️
src/lib/crypto/ecdsa.cpp 71.94% <ø> (ø)
src/lib/crypto/hash.hpp 100.00% <ø> (ø)
src/lib/crypto/hash_common.cpp 94.91% <0.00%> (-3.34%) ⬇️
src/lib/crypto/rng.cpp 77.77% <ø> (ø)
src/lib/ffi-priv-types.h 100.00% <ø> (ø)
src/lib/pgp-key.h 91.66% <ø> (ø)
src/lib/types.h 100.00% <ø> (ø)
... and 33 more

... and 40 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@falko-strenzke
Copy link
Contributor Author

fuzzing CI job fails with an error I do not understand:

/src/rnp/src/lib/crypto/rng.h:37:10: fatal error: 'botan/system_rng.h' file not found
#include <botan/system_rng.h>

@ni4 @TJ-91

@falko-strenzke
Copy link
Contributor Author

CodeQL found more than 10 potential problems in the proposed changes. Check the Files changed tab for more details.

I checked the findings of CodeQL. Except for one case, they all refer to lines of commented out code which we had agreed on to leave there as hints on how to complete the algorithm support once it is ready in the backend(s).

The remaining one is a false positive about a switch-case block having too few case labels. The result is invalid because apparently CodeQL does not consider the case-block enclosed by an #ifdef and thus comes to false conclusion.

@ni4
Copy link
Contributor

ni4 commented Sep 11, 2023

@falko-strenzke oss-fuzz uses version 2.16.0 of botan, so probably that header is not installed within current implementation. Given that it is required (as I understand) only for PQC (and/or v6), you may just hide it out via ifdefs.

TJ-91 and others added 18 commits September 11, 2023 12:54
correction of key and signature related code based on rnp feedback

address some more points of PR reiew

address some more points of PR reiew

rename parse_v3 and update comments

correction of key version handling according to rnp feedback

changes to v6 key & sig

remove redundant ifdef

fix conditional compilation of Botan C++ RNG
addressed various feedback comments
…/eddsa abstraction for the following PQC integration
correction of dilithium based on rnp feedback
correction of kyber based on rnp feedback
conditionally compile RNP_ALGNAME_* macros

address some more points of PR review (JRH)

corrections of key-gen ui according to rnp feedback

SHA3 hash binding for dilithium-exdsa-composite
manual correction of code formatting that still deviates
update copyright (2)
pqc draft: remove packet hash

pqc draft: add ecc pubkey to hashed ecdh key share

pqc draft: check AES for v3 PKESK
check SPHINCS+ Botan feature
fix Botan deprecation in kyber code
fix build system
remove comments
reverse ifdefs and config.h inclusion in rnp.h
fix false positive in CodeQL
@falko-strenzke
Copy link
Contributor Author

@falko-strenzke oss-fuzz uses version 2.16.0 of botan, so probably that header is not installed within current implementation. Given that it is required (as I understand) only for PQC (and/or v6), you may just hide it out via ifdefs.

OK, we enclosed the inclusion of the Botan RNG headers with an #ifdef. Let's see if the fuzzing CI job will be OK now.

@falko-strenzke
Copy link
Contributor Author

falko-strenzke commented Sep 11, 2023

We have now a different CI job failing: Cirrus CI. To the best of my knowledge, this one wasn't failing in status before we corrected the inclusion of the Botan RNG headers. Is this maybe a known problem for this test to occasionally fail? In any case, I cannot force a re-run of this job.

P.S.: We had seen this test failing locally on my system some time ago as documented here, but most likely this is not the same cause. Without the logs from the CI it's not possible to say what goes wrong here. And we don't have Cirrus CI in our repo...

@ni4 @TJ-91

@ni4
Copy link
Contributor

ni4 commented Sep 12, 2023

@falko-strenzke Hm, I believed I left a comment, but seems not. Cirrus CI, as well as some other jobs may fail occasionally due to some issue between RNP and GnuPG, which should be fixed at some point. For now restarting job helps :) Okay, it's all green now.

@ni4
Copy link
Contributor

ni4 commented Sep 18, 2023

@ronaldtse @antonsviridenko Could you please review this, as this PR blocks bunch of others? If there any major issues, we can address those via separate PR(s).

Copy link
Contributor

@ni4 ni4 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks! Hope we'll soon get this merged.

@antonsviridenko
Copy link
Contributor

This PR is quite big, will try to finish the review tomorrow.

@ni4
Copy link
Contributor

ni4 commented Sep 19, 2023

This PR is quite big, will try to finish the review tomorrow.

Yeah, it's big indeed. Thanks!

Copy link
Contributor

@antonsviridenko antonsviridenko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It fails to build with Botan version 3.0, it would be nice to have some check that Botan is at least 3.1:

fatal error: botan/sphincsplus.h: No such file or directory
   35 | #include <botan/sphincsplus.h>

@antonsviridenko
Copy link
Contributor

Now with Botan v3.1.1:

/home/odsk/Ribose/rnp/src/lib/crypto/kyber.cpp: In member function ‘kyber_encap_result_t pgp_kyber_public_key_t::encapsulate(rnp::RNG*)’:
/home/odsk/Ribose/rnp/src/lib/crypto/kyber.cpp:94:20: error: no matching function for call to ‘Botan::PK_KEM_Encryptor::encrypt(Botan::secure_vector<unsigned char>&, Botan::secure_vector<unsigned char>&, Botan::RandomNumberGenerator&, uint32_t)’
   94 |     kem_enc.encrypt(encap_key,
      |     ~~~~~~~~~~~~~~~^~~~~~~~~~~
   95 |                     data_encryption_key,
      |                     ~~~~~~~~~~~~~~~~~~~~
   96 |                     *rng->obj(),
      |                     ~~~~~~~~~~~~
   97 |                     key_share_size_from_kyber_param(kyber_mode_));
      |                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/odsk/Ribose/rnp/src/lib/crypto/kyber.h:36,
                 from /home/odsk/Ribose/rnp/src/lib/crypto/kyber.cpp:27:
/usr/include/botan-3/botan/pubkey.h:609:12: note: candidate: ‘void Botan::PK_KEM_Encryptor::encrypt(Botan::secure_vector<unsigned char>&, Botan::secure_vector<unsigned char>&, std::size_t, Botan::RandomNumberGenerator&, const uint8_t*, std::size_t)’
  609 |       void encrypt(secure_vector<uint8_t>& out_encapsulated_key,
      |            ^~~~~~~
/usr/include/botan-3/botan/pubkey.h:609:12: note:   candidate expects 6 arguments, 4 provided
/usr/include/botan-3/botan/pubkey.h:624:12: note: candidate: ‘void Botan::PK_KEM_Encryptor::encrypt(Botan::secure_vector<unsigned char>&, Botan::secure_vector<unsigned char>&, std::size_t, Botan::RandomNumberGenerator&, std::span<const unsigned char>)’
  624 |       void encrypt(secure_vector<uint8_t>& out_encapsulated_key,
      |            ^~~~~~~
/usr/include/botan-3/botan/pubkey.h:624:12: note:   candidate expects 5 arguments, 4 provided
/usr/include/botan-3/botan/pubkey.h:639:12: note: candidate: ‘void Botan::PK_KEM_Encryptor::encrypt(Botan::secure_vector<unsigned char>&, Botan::secure_vector<unsigned char>&, std::size_t, Botan::RandomNumberGenerator&)’
  639 |       void encrypt(secure_vector<uint8_t>& out_encapsulated_key,
      |            ^~~~~~~
/usr/include/botan-3/botan/pubkey.h:641:27: note:   no known conversion for argument 3 from ‘Botan::RandomNumberGenerator’ to ‘std::size_t’ {aka ‘long unsigned int’}
  641 |                    size_t desired_shared_key_len,
      |                    ~~~~~~~^~~~~~~~~~~~~~~~~~~~~~

@falko-strenzke
Copy link
Contributor Author

Please try to run it against the current master branch of Botan. No released version of Botan can possibly work: alone because KMAC, which is used as the KDF in composite encryption schemes. KMAC was merged into Botan master branch only a few days ago.

@ni4
Copy link
Contributor

ni4 commented Sep 20, 2023

We should add some PQC-related CI runner to avoid such issue in the future. But, this could be done after this PR is merged to not grow it any further. Main thing is that non-PQC CI runs well.

@antonsviridenko
Copy link
Contributor

Is there any other existing implementation of PGP v6 to test against?

@antonsviridenko
Copy link
Contributor

ok, so I've managed to build, now I'm trying to generate some key and it fails to load and display this key on the next run of rnpkeys:

LD_LIBRARY_PATH=/home/odsk/Ribose/install_prefix/lib ~/Ribose/install_prefix/bin/rnpkeys -g --expert --homedir ~/Ribose/testhomedir_v6
Keyring directory '/home/odsk/Ribose/testhomedir_v6' is empty.
Use "rnpkeys" command to generate a new key or import existing keys from the file or GnuPG keyrings.
Please select what kind of key you want:
        (1)  RSA (Encrypt or Sign)
        (16) DSA + ElGamal
        (17) DSA + RSA
        (19) ECDSA + ECDH
        (21) EDDSA + ECDH (v6 key) 
        (22) EDDSA + ECDH (v4 key) 
        (23) ED25519 + X25519 (v6 key) 
        (25) (Dilithium3 + Ed25519) + (Kyber768 + X25519)
        (27) (Dilithium3 + ECDSA-NIST-P-256) + (Kyber768 + ECDH-NIST-P-256)
        (28) (Dilithium5 + ECDSA-NIST-P-384) + (Kyber1024 + ECDH-NIST-P-384)
        (29) (Dilithium3 + ECDSA-brainpoolP256r1) + (Kyber768 + ECDH-brainpoolP256r1)
        (30) (Dilithium5 + ECDSA-brainpoolP384r1) + (Kyber1024 + ECDH-brainpoolP384r1)
        (31) SPHINCS+-SHA2-128f + (Kyber768 + X25519)
        (32) SPHINCS+-SHAKE-128f + (Kyber768 + X25519)
        (33) SPHINCS+-SHA2-256f + (Kyber1024 + ECDH-NIST-P-384)
        (34) SPHINCS+-SHAKE-256f + (Kyber1024 + ECDH-NIST-P-384)
        (99) SM2
> 30
Generating a new key...
[forget_secret_key_fields() /home/odsk/Ribose/rnp/src/librepgp/stream-key.cpp:1059] unknown key algorithm: 34
[forget_secret_key_fields() /home/odsk/Ribose/rnp/src/librepgp/stream-key.cpp:1059] unknown key algorithm: 34
[forget_secret_key_fields() /home/odsk/Ribose/rnp/src/librepgp/stream-key.cpp:1059] unknown key algorithm: 34
[forget_secret_key_fields() /home/odsk/Ribose/rnp/src/librepgp/stream-key.cpp:1059] unknown key algorithm: 34
[forget_secret_key_fields() /home/odsk/Ribose/rnp/src/librepgp/stream-key.cpp:1059] unknown key algorithm: 34
[forget_secret_key_fields() /home/odsk/Ribose/rnp/src/librepgp/stream-key.cpp:1059] unknown key algorithm: 34
[forget_secret_key_fields() /home/odsk/Ribose/rnp/src/librepgp/stream-key.cpp:1059] unknown key algorithm: 34
Enter password for key 0xD5219269C1B23324 to protect: 
Repeat password for key 0xD5219269C1B23324: 
Would you like to use the same password to protect subkey(s)? (y/N) y
[forget_secret_key_fields() /home/odsk/Ribose/rnp/src/librepgp/stream-key.cpp:1059] unknown key algorithm: 34

sec   21512/DILITHIUM5_BP384 d5219269c1b23324 2023-09-20 [SC] [EXPIRES 2025-09-19]
      d5219269c1b233249e8da03e9e7aa673f05d18400491a7dc3be490b0c6d49808
uid           Dilithium-BP384 21512-bit key <odsk@localhost>
ssb   13320/KYBER1024_BP384 111916a0ad0757a2 2023-09-20 [E] [EXPIRES 2025-09-19]
      111916a0ad0757a221ad8c68de3e649786f50a94f0052cb01435a6b77087221e
$ LD_LIBRARY_PATH=/home/odsk/Ribose/install_prefix/lib ~/Ribose/install_prefix/bin/rnpkeys -l --homedir ~/Ribose/testhomedir_v6
[get() /home/odsk/Ribose/rnp/src/librepgp/stream-packet.cpp:665] unknown s2k specifier: 208
[parse() /home/odsk/Ribose/rnp/src/librepgp/stream-key.cpp:1627] failed to read key protection
[process_pgp_key() /home/odsk/Ribose/rnp/src/librepgp/stream-key.cpp:437] failed to parse key pkt at 0
Error: failed to load keyring from '/home/odsk/Ribose/testhomedir_v6/secring.gpg'
fatal: failed to load keys

@falko-strenzke
Copy link
Contributor Author

I can confirm this bug. We'll look into this and ensure that keys with all algorithms can be generated, listed and used via the commandline.

@falko-strenzke
Copy link
Contributor Author

Is there any other existing implementation of PGP v6 to test against?

I fear unfortunately not. We know about some that are working on it, but as far as I know nothing has been released.

@ni4
Copy link
Contributor

ni4 commented Sep 21, 2023

Is there any other existing implementation of PGP v6 to test against?

At the moment it would be enough to test against self, as PQC support would be marked as experimental within the upcoming release, and all related functionality (must be) cut out via defines from the already existing functionality.

fixed missing case in forget_secret_key_fields()
@falko-strenzke
Copy link
Contributor Author

I can confirm this bug. We'll look into this and ensure that keys with all algorithms can be generated, listed and used via the commandline.

Fixed this now in 737c71f

According to our tests, key generation, signing, encrypting, verifying and decrypting through the CLI with unprotected and password protected keys are now all working.

@falko-strenzke
Copy link
Contributor Author

falko-strenzke commented Sep 25, 2023

The pipeline obviously has the usual sporadic Windows / GnuPG. Maybe someone with the required rights can force a re-run.

In our own dev repo I now added PQC CLI tests that cover signing/verification encryption/decryption with all the PQC keys types that are available via the CLI tool. However, currently they run unconditionally and thus would fail for the non-PQC build. I don't know what is the best way to make the test execution dependent on the build configuration. Once I can implement that, we could add these tests. Any suggestions?

@falko-strenzke
Copy link
Contributor Author

falko-strenzke commented Sep 25, 2023

Is there any other existing implementation of PGP v6 to test against?

Also I want to point out that our v6 support ist not complete. That was not the main target of our endeavours, but we rather implemented v6 only to the degree necessary to support PQC. That is of course including the data formats for encryption and signing. But SKESK with v6 are for instance not supported. See here for some more information about the v6 features that are supported / not supported.

@antonsviridenko antonsviridenko merged commit fdfc1f5 into rnpgp:main Sep 25, 2023
@antonsviridenko
Copy link
Contributor

Thank you @falko-strenzke

@ni4
Copy link
Contributor

ni4 commented Sep 26, 2023

In our own dev repo I now added PQC CLI tests that cover signing/verification encryption/decryption with all the PQC keys types that are available via the CLI tool. However, currently they run unconditionally and thus would fail for the non-PQC build. I don't know what is the best way to make the test execution dependent on the build configuration. Once I can implement that, we could add these tests. Any suggestions?

This may be done similarly to the IDEA tests, please see the rnp --version command. We could add line with supported PQC algos, and probably should add some notice that 'This RNP build includes experimental PQC support, and is not designed for production environment' or something similar.

@ni4
Copy link
Contributor

ni4 commented Sep 26, 2023

Also I want to point out that our v6 support ist not complete. That was not the main target of our endeavours, but we rather implemented v6 only to the degree necessary to support PQC. That is of course including the data formats for encryption and signing. But SKESK with v6 are for instance not supported. See here for some more information about the v6 features that are supported / not supported.

This should not be a problem as v6 is not something required at the moment.

@ni4
Copy link
Contributor

ni4 commented Sep 26, 2023

@antonsviridenko @falko-strenzke Thanks, finally this one is merged so we can continue with incremental changes/updates/fixes.

@falko-strenzke
Copy link
Contributor Author

In our own dev repo I now added PQC CLI tests that cover signing/verification encryption/decryption with all the PQC keys types that are available via the CLI tool. However, currently they run unconditionally and thus would fail for the non-PQC build. I don't know what is the best way to make the test execution dependent on the build configuration. Once I can implement that, we could add these tests. Any suggestions?

This may be done similarly to the IDEA tests, please see the rnp --version command. We could add line with supported PQC algos, and probably should add some notice that 'This RNP build includes experimental PQC support, and is not designed for production environment' or something similar.

In correspondence with our project schedule we'll postpone the implementation of the CLI tests for now. We can still work on testing during the the first quarter of 2024, and latest then we will implement the tests.

@ni4
Copy link
Contributor

ni4 commented Oct 12, 2023

@falko-strenzke Ok, np. I plan to add PQC CI runner at some point, so it could at least basic checks. Do you need anything from our side for your schedule, or everything is fine for now?

@falko-strenzke
Copy link
Contributor Author

@falko-strenzke Ok, np. I plan to add PQC CI runner at some point, so it could at least basic checks. Do you need anything from our side for your schedule, or everything is fine for now?

One question that is interesting to us is when you plan to make a new release. This is relevant for us to some degree as in our project we are also creating a Thunderbird version that supports PQC using RNP.

@ni4
Copy link
Contributor

ni4 commented Oct 13, 2023

One question that is interesting to us is when you plan to make a new release. This is relevant for us to some degree as in our project we are also creating a Thunderbird version that supports PQC using RNP.

We still have some things to finish before the release, so unfortunately don't see any exact date yet. Is it critical for your project or you may create Thunderbird version based on some commit from the main branch?

P.S. Botan released version 3.2.0, do I correctly assume that it includes all the stuff, required for PQC?

@TJ-91
Copy link
Contributor

TJ-91 commented Oct 13, 2023

Is it critical for your project or you may create Thunderbird version based on some commit from the main branch?

It is not critical and we can use the main branch for the time being.

P.S. Botan released version 3.2.0, do I correctly assume that it includes all the stuff, required for PQC?

Yes. It includes the required features, namely KMAC, Kyber, Dilithium, and SPHINCS+. To be sure, I compiled it and ran the tests and it works. Some Kyber Enum values have been deprecated and give warnings. They are still valid but should be replaced at some point.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants