Support passphrase protected client certificate keys when using mTLS

[cmacknz]

We are adding support for mTLS between Elastic Agent and Fleet Server (or the proxy in front of Fleet Server). This requires specifying the client side private key. Currently this key must be specified on the command line which makes it easy to see. Fleet server currently supports reading the private key from a passphrase protected file. We should do the same for Elastic Agent mTLS. Support for reading these files was added in elastic/elastic-agent-libs#109.

In the in progress mTLS implementation we do not expose command line options for reading the certificate key from a file. We need to support the --elastic-agent-cert-key-passphrase to support this.

• Add mTLS flags to install and enroll commands #4007
• Add mTLS env vars for container mode #4261

      --elastic-agent-cert-key-passphrase string         Path for private key passphrase file used to decrypt certificate key used in mTLS

Endpoint security today does not support reading keys from passphrase protected files either. Rather than make endpoint-security implement this functionality, the agent should pass the decoded private key over the control protocol (which is itself protected by mTLS on local host or local sockets/pipes). This centralizes the implementation and provides the key handling as a platform feature.

Acceptance Critera

• Elastic Agent can be configured to use a passphrase protected private key for mTLS with a proxy or fleet server.
• Elastic Defend can connect to Fleet Server when using a passphrase protected private key for mTLS with a proxy or fleet server.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[AndersonQ]

@cmacknz @pierrehilbert @nimarezainia @jlind23 help me to clarify this issue:

We need to support the --fleet-server-cert-key-passphrase to support this.

This is already implemented, the agent accepts --fleet-server-cert-key-passphrase. However I did not find integration tests for that or unit tests beyond checking the --fleet-server-cert-key-passphrase is accepted by the enroll command.

What I see missing is the client certificate key to accept a passphrase. Thus I believe what is missing on the agent side is to accept --elastic-agent-cert-key-passphrase.

Another thing I'd like to ask. Those configs come from the CLI as well as the policy, therefore there is also the precedence issue. I checked there are unit test on the policy change handler for fleet.ssl.key_passphrase and fleet.ssl.key_passphrase_path.
Testing the precedence would have to be an integration test, and right now we do not have integration test with HTTPS proxy:

elastic-agent/testing/integration/proxy_url_test.go

Line 526 in 10d8e04

t.Skip("Currently skipped due to https proxy -> http fleet issues. See issues https://github.com/elastic/elastic-agent/issues/4896 and https://github.com/elastic/elastic-agent/issues/4903")

Regarding Elastic Defend, shouldn't it be its own issue. Those are completely separated things, which will require different tests and more importantly can be delivered on their own.
To be honest I'm not sure how mTLS proxy/fleet-server works with Elastic Defend. Does it just uses the same client certificates as the elastic-agent?

So, in summary I believe there is at least 3 issues here:

• add support for cli flag --elastic-agent-cert-key-passphrase
• and an integration test for mTLS proxy/fleet-server, which intersects with
  • --insecure flag should not be required during enroll/install because we have an http Fleet URL #4896
  • Improve test proxy/mock fleet server to proxy HTTPS connections #4903
• pass decrypted TLS client certificate to Elastic Defend through the control protocol.
  • when you say the agent should pass the decoded private key over the control protocol, is it requiring to extend the control protocol or the agent could just add those to the policy it passed to Elastic Defend?

[cmacknz]

add support for cli flag --elastic-agent-cert-key-passphrase

Correct, the issue description is wrong and/or out of date.

and an integration test for mTLS proxy/fleet-server, which intersects with

👍 Ideally we'd have a test that uses key passphrases for both the client and server side keys.

pass decrypted TLS client certificate to Elastic Defend through the control protocol.
when you say the agent should pass the decoded private key over the control protocol, is it requiring to extend the control protocol or the agent could just add those to the policy it passed to Elastic Defend?

elastic-agent/internal/pkg/agent/application/fleet_server_bootstrap.go

Lines 119 to 136 in 781f529

	// InjectFleetConfigComponentModifier The modifier that injects the fleet configuration for the components
	// that need to be able to connect to fleet server.
	func InjectFleetConfigComponentModifier(fleetCfg *configuration.FleetAgentConfig, agentInfo info.Agent) coordinator.ComponentsModifier {
	return func(comps []component.Component, cfg map[string]interface{}) ([]component.Component, error) {
	hostsStr := fleetCfg.Client.GetHosts()
	fleetHosts := make([]interface{}, 0, len(hostsStr))
	for _, host := range hostsStr {
	fleetHosts = append(fleetHosts, host)
	}
	// add host val to hosts array for use in injectProxyURL
	if host, ok := cfg["host"].(string); ok && host != "" {
	hostsStr = append(hostsStr, host)
	}
	for i, comp := range comps {
	if comp.InputSpec != nil && (comp.InputSpec.InputType == endpoint || comp.InputSpec.InputType == apmServer) {
	for j, unit := range comp.Units {
	if unit.Type == client.UnitTypeInput && (unit.Config.Type == endpoint || unit.Config.Type == apmServer) {

Defend needs to be able to talk directly to Fleet Server, so we already inject all the fleet parameters necessary for this into their configuration. Since Defend doesn't support private key passphrases, rather than pass them the passphrase and encrypted key and require them to implement support, we can just pass them the decoded private key so they are in the same situation as regular mTLS where they are passed just the key. You don't need to change the control protocol at all.

Looking at the endpoint source code they support reading the certificate and key as long as they are in the fleet.ssl.* configuration block. See here and here. They are following the Filebeat ssl configuration as a spec https://www.elastic.co/guide/en/beats/filebeat/current/configuration-ssl.html#client-key. They actually parse the key passphrase but don't do anything with it.

Ideally our automated test for encrypted key files would run with Defend installed, but if that isn't possible for some reason we can get away with a test ensuring we inject the key in the expected format and a manual test.

[cmacknz]

Updated the description with the right command line options.

[AndersonQ]

I created the tickets:

• Add support for CLI flag for mTLS client certificate key passphrase #5489
• Pass the decrypted mTLS client certificate key to Elastic Defend #5490
• Add an integration test for --elastic-agent-cert-key-passphrase with Elastc Defend installed #5491

I believe it covers all that is here. Please have a look and see if you agree with them, particularly the out of scope tests.

[AndersonQ]

closed as it's the following issues were created as a result of investigating what was required by this issue

• Add support for CLI flag for mTLS client certificate key passphrase #5489
• Pass the decrypted mTLS client certificate key to Elastic Defend #5490
• Add an integration test for --elastic-agent-cert-key-passphrase with Elastc Defend installed #5491