Authenticated encryption interface

[eadmund]

I'm adopting GitHub's workflow (c.f. GitHub flow) for this branch: rather than submitting a pull request which is ready to be pulled, I'm proposing that we use this pull request to discuss implementation issues. Let me know if you think that this is a useful mechanism.

Authenticated encryption has two operations: encrypt-and-generate-authentication-tag and decrypt-and-authenticate. It occurred to me that ENCRYPT and DECRYPT could retain the exact same signature as with ordinary symmetric encryption, with an additional TAG method on the cipher object itself. To encrypt, one would execute:

(let ((cipher (make-authenticated-cipher ...)))
  (encrypt cipher plaintext ciphertext...)
  (values ciphertext (tag cipher)))

And to decrypt one would execute:

(let ((cipher (make-authenticated-cipher ...)))
  (setf (tag cipher) tag)
  (decrypt cipher ciphertext plaintext ...)
  ciphertext)

Where DECRYPT throws a condition when the authentication fails.

In order for this to work, ENCRYPT and DECRYPT could become generic functions, with the CIPHERS module implementing the symmetric version and the AUTH-ENC implementing the authenticated encryption version.

Alternatively there could be separate AUTHENTICATED-ENCRYPT and AUTHENTICATED-DECRYPT, but that seems a bit ugly to me.

What do you think?

[eadmund]

Note that I'm not altogether happy with the current naming conventions.

Also, on reflection maybe it'd make more sense for the cipher code to treat AEAD modes just like ordinary modes?

[froydnj]

Thanks for sending this.

I've thought for a while about the best way to introduce authenticated encryption (AE) with the current interface. Doing it with {EN,DE}CRYPT makes me uneasy, as there'd be some very specific ways you'd need to use the API and/or very AE-specific hacks you'd have to make to those functions. I want to use those functions, because I like the ability to incrementally do {en,de}cryption with them. But looking at the ways other toolkits do incremental AE strikes me as ugly.

So...my current opinion is that the best way to go is introducing AE and friends via {EN,DE}CRYPT-MESSAGE. You lose incrementalism and avoiding library-induced consing, but you gain assurances that everything can be done in the context of those two functions: fetching the tag from the right place, always checking the tag, introducing additionally authenticated data, etc. etc. It's easy to introduce keyword arguments there and I don't mind so much if those two functions are generic functions. What do you think?

[eadmund]

Hmmm...my current code doesn't really do incremental encryption right; that's a big bug. But AEAD (at least GCM) is specified to allow it. I don't know if it's too painful to add AD and retrieve the tag: specify that the additional data must be passed in before a certain point and throw an exception if not; and add TAG to get the authentication tag.

I could be persuaded to use {EN,DE}CRYPT-MESSAGE too, though. Doesn't feel as elegant though.

The more I think about it, the more I think that the most elegant approach is to treat AE/AEAD modes as just another mode of operation, one which allows calling TAG.

But heck, even AUTHENTICATED-ENCRYPT and AUTHENTICATED-DECRYPT would work...the most important thing is to offer it.

[froydnj]

Yes, offering it is the best thing.

I agree the most elegant approach is to treat AEAD modes as "just another mode" with some bits added on. But I'm skeptical that it's the most practical way to do it for actual use. How does the tag get output, for instance? Through :HANDLE-FINAL-BLOCK T on encryption? Maybe that's not so bad (though I think fitting in additional data to the "just another mode" model gets messy quickly). What about decryption? Do you feed it in at the end, with your data and :HANDLE-FINAL-BLOCK T? Hm, what if your tag occurs at the beginning of your message? Well, maybe it's better to (SETF TAG), then. OK, what if your tag occurs at the end of the message? What if you forget TAG? And so forth.

I think having a "encrypt/decrypt this message and output/verify this tag" one-shot approach addresses a lot of the questions that would come up in an incremental approach. And "two-pass" AEAD modes become trivial to handle with a "provide all the data up front" approach.

(Maybe the Right Thing is to never provide encrypt/decrypt on byte buffers, but to have encrypt/decrypt stream filters instead. But that is a discussion for another time.)

To be clear, I don't know what a lot of packet formats that would use AEAD look like, so it's possible some of the concerns above are moot. I'd be interested in hearing if you have any knowledge/experience in this area.