SSL/TLS Settings

[harbulot]

This is a follow up of issue #9. (PR #28 aimed to fix it, but didn't and introduced more confusion unfortunately.)

History

Before PR #28:

• Client-certificate authentication was not enabled by default.
• Setting the trust anchors was done via ssl_cacert (which could take a path to a file bundle or a directory of CA certificates).
• There was no way of adding additional certificates to the server certificate chain.
• The default system certificates were always included in a trust anchors (there was no way to remove them and restrict the trusted CAs).

Issue #9 was asking for:

• A way to set the server certificate chain (e.g. via an extra_chain_cert parameter).
• A way not to enable all the system CA certs, so as to be able to be more restrictive.

What PR #28 did:

• It kept all the system CA certs anyway (although a bit differently).
• It deprecated the old ssl_cacert (deemed confusing).
• It created a new extra_chain_certs option (with an s), that doesn't have much to do with ssl_context.extra_chain_cert (the option to configure the server certificate chain), but simply adds the certificates it contains to the trust anchors.
• It made client-certificate authentication the default, thereby causing problems for those not expecting it (issue null cert chain #33): indeed client-certificate authentication require additional PKI understanding and preparation, which most users are probably not familiar with.

SSL/TLS Background

There seems to have been some confusion regarding what the certificate chain and what the trust anchors are.

• The certificate chain is for your credentials: the ones you sent to the remote party. This is a chain because the TLS specification says it has to be:

  certificate_list
    This is a sequence (chain) of certificates.  The sender's
    certificate MUST come first in the list.  Each following
    certificate MUST directly certify the one preceding it.  Because
    certificate validation requires that root keys be distributed
    independently, the self-signed certificate that specifies the root
    certificate authority MAY be omitted from the chain, under the
    assumption that the remote end must already possess it in order to
    validate it in any case.
  

  (Actually, the new TLS 1.3 draft relaxes the ordering constraint, but the idea is the same.)

  This certificate chain is sent to the remote party in the hope that the remote party can link that chain to one of its trust anchors to establish trust in your (end-entity) certificate. (When behaving as a client, the server generally gives a list of its trust anchors to help provide a suitable certificate chain.)

  Of course, your own certificate (for which you have the private key) is in that chain (and the beginning, and at worse, that chain is made only of that certificate).

• The trust anchors: these are the certificates you consider to be trusted (these are generally CA certificates, although some configurations also allow for individual end-entity certificates on a case-by-case basis). When you get a certificate chain from the remote party, you try to build a certification path to one of your trust anchors to be able to verify the remote (end-entity) certificate at the end of that chain.

Those are fundamentally different roles, which should not be mixed up. (For a Java point of view, those would be the roles of the KeyManager and TrustManager, respectively.)

OpenSSL (at least the way it is used or emulated here), has ways to fall back to other default options when some settings are not set. This may have cause some confusion when implementing these features.

Here is how these options are mapped to the ssl_context used in the code:

• ssl_context.cert: This expects an X509Cert instance that represents your certificate (which you're going to send to the remote party). (This is similar to Apache Httpd SSLCertificateFile, although SSLCertificateFile can also include the certificate chain and private key, but this cannot.)
• ssl_context.key: This is the private key associated with your certificate. (This is similar to Apache Httpd SSLCertificateKeyFile.)
• ssl_context.ca_file: This is path to a file bundle of CA certificates, to be used as trust anchors. (This is similar to Apache Httpd SSLCACertificateFile.)
• ssl_context.ca_path: This is path to a directory of CA certificates in a file, to be used as trust anchors. (As far as I remember, those need to follow hashed naming conventions). (This is similar to Apache Httpd SSLCACertificatePath.)
• ssl_context.cert_store: A store containing CA certificates. If ca_file or ca_path are set, the trust anchors they load are added to this store. (This option can be useful to load the default system CA certificates, using set_default_paths on a new OpenSSL::X509::Store.
• ssl_context.extra_chain_cert: When this is NOT set, an attempt is made to build your certificate chain using the trust anchors (set via cert_store, ca_file and/or ca_path). When this is set, only those certificates are used to build the certificate you are going to send to the remote party. (This is similar to Apache Httpd SSLCertificateChainFile.)
• ssl_context.client_ca: This is not particularly well named, but this is used on the server side when using client-certificate authentication to send the list of trusted authorities in the TLS CertificateRequest message. This option expects a list of X509Cert (and only the Subject DNs of those certs are used in practice). (This is similar to Apache Httpd SSLCADNRequestFile.) This option is not used as often as the others, since it generally makes sense to send the list of all the CAs the server trusts. (However, JRuby-OpenSSL currently sends an empty list when this is not configured, see: Build accepted issuer list from CA store when no client_ca are set jruby/jruby-openssl#85.)

These settings can apply to both the client and the server side, except ssl_context.client_ca which is server-specific. Setting ssl_context.ca_file, ssl_context.ca_path or ssl_context.cert_store on the server only makes sense on the server (a) if you're using client-certificate authentication or (b) if you don't set ssl_context.extra_chain_cert and you want the server certificate chain to be built automatically from your trust anchors.

Why separating those settings is required

This all has to do with the way client-certificate authentication is used in practice. Verifying the remote certificate when you are a server has a very different purpose from doing it when you are a client.

As a client

• You know the machine you want to connect to.
• You check that you can trust the server certificate (and that it's valid) so that you can effectively believe its content to be genuine (that's the PKI aspect, RFC 5280).
• You check that the identity in the certificate matches what you want to connect to (that's hostname verification, RFC 2818 Section 3.1, or more generally RFC 6125).

ssl_verify => true by default makes perfect sense here (and too often, some code forgets to verify the hostname anyway, since there are two trust aspects to check here).

In this context, it can make sense to have a reasonably large set of trusted CAs (although you don't necessarily want it to be too large):

• The client at least controls what it attempts to connect to.
• There are reasonably clear naming guidelines w.r.t. what the certificate is meant to identify (e.g. host name in Subject Alternative Name or even CN, see RFC 6125).

This is far from a perfect scenario, but it's manageable.

As a server (which is the case in logstash-input-tcp)

• You don't know which clients are trying to connect to you.
• You use authentication as a pre-requisite for authorisation (and possibly audit).
• The way of describing what you identify in a certificate isn't particularly well defined (the naming conventions for the DNs are more flexible, there's no reliable convention regarding which CA is going to issue which forms of DNs).

Those who use ssl_verify => false (as a server) are willing to let anyone connect to their service.

Those who use ssl_verify => true want client-certification enabled (and that's generally an active step because you need to set up the PKI to do that), but more importantly, they're really after authorisation (or possibly audit, i.e. traceability of the genuine entity that made the connection).

Considering that there is no authorisation layer in logstash-input-tcp, anything that is authenticated is effectively authorised. In this context, letting in client-certs issued by a dozen of pre-installed CAs becomes a problem.

• Firstly, there's no fixed convention for the Subject DNs (so that wouldn't be reliable, even if there was an authorisation layer).
• Secondly, while client certificates from commercial CAs seem unusual, there are many CAs that will issue server certificates that are also usable as client certificates (in terms of key usage attributes).

Basically, by trusting all the CAs installed by default on the system, you're letting anyone willing to pay for the cheapest cert from any widely trusted CA in. That, of course, is a no-no.

Another aspect to take into consideration is that a server doesn't necessarily want to advertise all the CA certificates it trusts when its system behaves as a client in the CertificateRequest TLS message:

• It may reveal unwanted information to the connecting client, which is not authenticated at that stage (known what a remote system trusts in this case might be used maliciously for other purposes)
• The list can get quite long, thereby making the CertificateRequest message quite big and add extra overhead in the handshake. (This can even cause critical problems in some cases.)

At the moment, that list is empty anyway, but this might change if jruby/jruby-openssl#85 is fixed (this has more to do with ssl_context.client_ca otherwise, but it's not used in logstash-input-tcp at the moment).

How to fix?

• As a general rule, people willing to use client-certificate authentication will want to configure a very limited set of CAs they're willing to trust. As such, having the ability to trust the system-wide CAs barely makes sense at all. It can be an option, but it shouldn't be the default.

• Not being able to remove those system CAs from the trust anchors (as it is the case since PR Enhance extra chain of certificates and include ton of ssl tests #28) is very much of a problem in that respect.

• ssl_cacert was doing its job reasonably well, but its name was somewhat odd compared with other tools that rely on OpenSSL.
  It would make more sense to have a ssl_ca_file (the same as the old ssl_cafile) and possibly a new ssl_ca_path options that match Apache Httpd SSLCACertificateFile/SSLCACertificatePath Directive (or -CAfile/-CApath in openssl s_client, and so on):

  This directive sets the all-in-one file where you can assemble the Certificates of Certification Authorities (CA) whose clients you deal with. These are used for Client Authentication. Such a file is simply the concatenation of the various PEM-encoded Certificate files, in order of preference. This can be used alternatively and/or additionally to SSLCACertificatePath.

• Deprecate ssl_extra_chain_certs and ignore it, since it doesn't really have anything to do with the specific server chain.

• Ideally, allow ssl_cert to point to a file that contains the full explicit chain. Again, this would be similar to Apache Httpd's SSLCertificateFile's behaviour:

  The files may also include intermediate CA certificates, sorted from leaf to root. This is supported with version 2.4.8 and later, and obsoletes SSLCertificateChainFile. When running with OpenSSL 1.0.2 or later, this allows to configure the intermediate CA chain on a per-certificate basis.

  I don't think jruby-openssl supports this out of the box, but it might be possible to read multiple PEM certs from the cert file in logstash-input-tcp, use the first one as the cert and the others for ssl_context.extra_chain_cert.

• Alternatively, or in addition to the chain above, having a configuration option (ssl_extra_chain_cert, without an s) that points to a file giving the certificate chain that should be used, if needed (effectively similar to Apache Httpd's SSLCertificateChainFile).

  That can be set in ssl_context.extra_chain_cert.

Here is an incomplete patch (but that should sort ca_file and ca_path at least):

@@ -38,7 +38,7 @@
   config :ssl_verify, :validate => :boolean, :default => true

   # The SSL CA certificate, chainfile or CA path. The system CA path is automatically included.
-  config :ssl_cacert, :validate => :path, :deprecated => "This setting is deprecated in favor of ssl_extra_chain_certs as it sets a more clear expectation to add more X509 certificates to the store"
+  config :ssl_cacert, :validate => :path, :deprecated => "This setting is deprecated in favor of ssl_ca_file and ssl_ca_path"

   # SSL certificate path
   config :ssl_cert, :validate => :path
@@ -49,9 +49,14 @@
   # SSL key passphrase
   config :ssl_key_passphrase, :validate => :password, :default => nil

-  # An Array of extra X509 certificates to be added to the certificate chain.
-  # Useful when the CA chain is not necessary in the system store.
-  config :ssl_extra_chain_certs, :validate => :array, :default => []
+  # Not in use
+  config :ssl_extra_chain_certs, :validate => :array, :default => [], :deprecated => "Not in use"
+
+  # Bundle of CA certificates in PEM format in a single file to use as trust anchors
+  config :ssl_ca_file, :validate => :path
+
+  # Directory containing CA certificates in PEM format to use as trust anchors (check OpenSSL naming conventions)
+  config :ssl_ca_path, :validate => :path

   def initialize(*args)
     super(*args)
@@ -213,7 +218,13 @@
       @ssl_context.cert = OpenSSL::X509::Certificate.new(File.read(@ssl_cert))
       @ssl_context.key = OpenSSL::PKey::RSA.new(File.read(@ssl_key),@ssl_key_passphrase)
       if @ssl_verify
-        @ssl_context.cert_store  = load_cert_store
+        if @ssl_ca_file
+          @ssl_context.ca_file = @ssl_ca_file
+        end
+        if @ssl_ca_path
+          @ssl_context.ca_path = @ssl_ca_path
+        end
+
         @ssl_context.verify_mode = OpenSSL::SSL::VERIFY_PEER|OpenSSL::SSL::VERIFY_FAIL_IF_NO_PEER_CERT
       end
     rescue => e
@@ -224,20 +235,6 @@
     @ssl_context
   end

-  def load_cert_store
-    cert_store = OpenSSL::X509::Store.new
-    cert_store.set_default_paths
-    if File.directory?(@ssl_cacert)
-      cert_store.add_path(@ssl_cacert)
-    else
-      cert_store.add_file(@ssl_cacert)
-    end if @ssl_cacert
-    @ssl_extra_chain_certs.each do |cert|
-      cert_store.add_file(cert)
-    end
-    cert_store
-  end
-
   def new_server_socket
     @logger.info("Starting tcp input listener", :address => "#{@host}:#{@port}")
     begin

• To complete this patch, setting ssl_context.extra_chain_cert should be done before if @ssl_verify (since presenting a given certificate chain has nothing to do with verifying the remote party). This problem is also mentioned in issue CA bundle not appended without ssl_verify (OpenSSL::SSL::SSLError: Received fatal alert: unknown_ca) #45.
• If ssl_verify => true is kept as the default in this project, the documentation should probably make it clear that client-certificate authentication is required by default. Otherwise, problems like issue null cert chain #33 will happen to many users. A lot of people are confused by certificates when setting them up on their servers, but since using client-certificate authentication requires even more understanding of the PKI concepts, this could become even more confusing.

Alternatives

I don't know well enough how this project is implemented and how much it depends on JRuby-OpenSSL. Considering that it seems to be expected to run in a Java environment, it might be worth considering using JSSE settings instead of all this (i.e. keystores and truststore). Having additional options to build a keystore from PEM files (to be compatible with the current key and cert settings) would not be too difficult to implement.

---

If hope these details can help. I have not submitted a PR because this patch is incomplete (the main thing that is missing is an option that sets ssl_context.extra_chain_cert). I don't have enough spare time to learn Ruby, but I guess loading a number of PEM-encoded certificates from a file and turning them into some sort of collection that can be passed to ssl_context.extra_chain_cert shouldn't be too difficult for someone who knows Ruby.

[harbulot]

It looks like setting ssl_context.extra_chain_cert has already been done in other ELK projects:

• Add chained certificate support elastic/ruby-lumberjack#20 does it by reading the chain from the cert file (more or less what I was saying above: "Ideally, allow ssl_cert to point to a file that contains the full explicit chain. Again, this would be similar to Apache Httpd's SSLCertificateFile's behaviour")
• Added option for an SSL certificate chain to lumberjack server elastic/ruby-lumberjack#2 does it by reading the chain from a separate file (similar to Apache Httpd's SSLCertificateChainFile). (This PR wasn't merged, as the first method was preferred.)