[Meta] Migration to secret variables

[jsoriano]

We are driving an effort to encourage the declaration of secret variables in packages configurations. The use of secret variables highly decreases the risk of leaking secrets in log and configuration files.

Find below a list of variables that we consider secret candidates, sorted by owner.

For these variables, we ask the teams to evaluate if they are actually secrets. If they are, mark them with secret: true, if they aren't, mark them with secret: false.

For packages reviewed, we recommend to update to format_version: "3.0.2" to avoid regressions on these fields. Starting on this version the use of secret is required for variables that look like secrets.

Secrets are supported since Kibana 8.10.0, but there are known issues till 8.11.2. Packages using secret: true will also work in older versions, but secrets won't be used.
Due to the known issues, it is recommended to use kibana.version: ^8.11.2 when using secrets, but older versions can still be used for packages that work in a broad range of versions.

As a reminder, these are the consequences of enabling secrets:

When an existing variable in a given package is denoted as secret: true in a new version, users will lose read access to their previously plaintext value once they save their upgraded package policy in Fleet. This means that if the only place a user was storing their AWS secret key was their package policy in Fleet, once we enable secret: true for that variable and the user completes the upgrade process, they will no longer have any access to that secret key. We’re working on making this more clear in the UX here, and we will work on adding a note to our Fleet secrets docs as well to make this as clear as possible to users.

We will use this issue to keep track of the progress on this migration.

Issues uncovered during migration efforts

We'll track notable issues uncovered as part of the migration efforts here.

• [Fleet] Minor secrets UX improvements kibana#171225
• [Fleet] Support to Optional Secrets kibana#172061
• [Fleet] Integration policy with secret fails on creation kibana#173048
• [Fleet] Exception when trying to save CSPM integration with a Secret kibana#173718

Version target recommendation

All fixes for the above known issues are present in Kibana version 8.12.0. So, the recommended target version constraint for integrations using secrets is ^8.12.0.

Codeowner: @elastic/cloud-security-posture

Secret candidates:

Integration	Secret Candidate	Migrated
cloud_security_posture	findings.access_key_id	❌
cloud_security_posture	findings.access_key_id	❌
cloud_security_posture	findings.azure.credentials.client_certificate_password	✅
cloud_security_posture	findings.azure.credentials.client_password	✅
cloud_security_posture	findings.azure.credentials.client_secret	✅
cloud_security_posture	findings.secret_access_key	✅
cloud_security_posture	findings.secret_access_key	✅
cloud_security_posture	findings.session_token	❌
cloud_security_posture	findings.session_token	❌

Codeowner: @elastic/security-service-integrations

Secret candidates:

Integration	Secret Candidate	Migrated
windows	httpjson.password	❌
windows	httpjson.token	❌

Codeowner: @elastic/obs-ds-hosted-services

Secret candidates:

Integration	Secret Candidate	Migrated
aws_logs	access_key_id	✅
aws_logs	secret_access_key	✅
aws_logs	session_token	✅
azure_metrics	client_secret	✅

Codeowner: @elastic/obs-ds-intake-services

Secret candidates:

Integration	Secret Candidate	Migrated
apm	apm.api_key_enabled	✅
apm	apm.api_key_limit	✅
apm	apm.secret_token	✅

Codeowner: @elastic/obs-infraobs-integrations

Secret candidates:

Integration	Secret Candidate	Migrated
activemq	activemq/metrics.password	✅
apache	httpjson.password	❌
apache	httpjson.token	❌
apache_tomcat	prometheus/metrics.password	✅
aws	access_key_id	✅
aws	cloudtrail.password	✅
aws	cloudtrail.token	✅
aws	secret_access_key	✅
aws	session_token	✅
azure	connection_string	✅
azure	storage_account_key	✅
azure_app_service	connection_string	✅
azure_app_service	storage_account_key	✅
azure_application_insights	api_key	✅
azure_billing	client_secret	✅
azure_functions	functionapplogs.connection_string	✅
azure_functions	functionapplogs.storage_account_key	✅
azure_functions	metrics.client_secret	✅
cassandra	jolokia/metrics.password	✅
ceph	httpjson.api_secret	✅
citrix_adc	httpjson.password	✅
cockroachdb	status.password	✅
golang	httpjson.password	✅
haproxy	haproxy/metrics.password	❌
ibmmq	prometheus/metrics.password	❌
kafka	consumergroup.password	✅
kafka	kafka/metrics.ssl.key_passphrase	✅
kafka	partition.password	✅
kafka_log	generic.kerberos_password	✅
kafka_log	generic.password	✅
microsoft_sqlserver	sql/metrics.password	✅
mongodb	mongodb/metrics.password	✅
mysql	mysql/metrics.password	✅
nagios_xi	httpjson.api_key	❌
nginx	httpjson.password	✅
nginx	httpjson.token	✅
oracle_weblogic	jolokia/metrics.password	✅
postgresql	postgresql/metrics.password	✅
prometheus	collector.password	✅
rabbitmq	rabbitmq/metrics.password	✅
redis	redis/metrics.password	✅
redis	slowlog.password	✅
salesforce	client_secret	✅
salesforce	password	✅
spring_boot	jolokia/metrics.password	✅
system	httpjson.password	✅
system	httpjson.token	✅
vsphere	vsphere/metrics.password	✅
websphere_application_server	prometheus/metrics.password	✅

Codeowner: @elastic/obs-ux-infra_services-team

https://github.com/elastic/synthetics-dev/issues/355

Secret candidates:

Integration	Secret Candidate	Migrated
synthetics	http.password	❌
synthetics	http.ssl.key_passphrase	❌
synthetics	tcp.ssl.key_passphrase	❌

Codeowner: @elastic/sec-deployment-and-devices

Secret candidates:

Integration	Secret Candidate	Migrated
hashicorp_vault	metrics.vault_token	✅
zeek	httpjson.password	✅
zeek	httpjson.token	✅

Codeowner: @elastic/security-service-integrations

Secret candidates:

Integration	Secret Candidate	Migrated
1password	httpjson.token	✅
akamai	siem.access_token	✅
akamai	siem.client_secret	✅
akamai	siem.client_token	✅
akamai	siem.service_account_key	✅
amazon_security_lake	event.access_key_id	✅
amazon_security_lake	event.secret_access_key	✅
amazon_security_lake	event.session_token	✅
atlassian_bitbucket	audit.password	✅
atlassian_bitbucket	audit.token	✅
atlassian_confluence	audit.password	✅
atlassian_confluence	audit.token	✅
atlassian_jira	audit.password	✅
atlassian_jira	audit.token	✅
auth0	logs.secret_value	✅
azure_blob_storage	azure-blob-storage.service_account_key	✅
azure_frontdoor	azure-eventhub.storage_account_key	✅
bitdefender	httpjson.api_key	✅
bitdefender	push_notifications.authorization_value	✅
bitwarden	httpjson.client_secret	✅
box_events	httpjson.box_subject_id	✅
box_events	httpjson.client_id	✅
box_events	httpjson.client_secret	✅
carbon_black_cloud	aws-s3.access_key_id	✅
carbon_black_cloud	aws-s3.secret_access_key	✅
carbon_black_cloud	aws-s3.session_token	✅
carbon_black_cloud	cel.custom_api_secret_key	✅
carbon_black_cloud	httpjson.api_secret_key	✅
carbon_black_cloud	httpjson.custom_api_secret_key	✅
cisco_duo	httpjson.secret_key	✅
cisco_meraki	events.secret_value	✅
cisco_secure_endpoint	event.api_key	✅
cisco_umbrella	log.access_key_id	✅
cisco_umbrella	log.secret_access_key	✅
cisco_umbrella	log.session_token	✅
cloudflare	audit.auth_key	✅
cloudflare	logpull.auth_key	✅
cloudflare	logpull.auth_token	✅
cloudflare_logpush	aws-s3.access_key_id	✅
cloudflare_logpush	aws-s3.secret_access_key	✅
cloudflare_logpush	aws-s3.session_token	✅
cloudflare_logpush	gcs.service_account_key	✅
cloudflare_logpush	http_endpoint.secret_header	✅
cloudflare_logpush	http_endpoint.secret_value	✅
crowdstrike	cel.client_secret	✅
crowdstrike	fdr.access_key_id	✅
crowdstrike	fdr.secret_access_key	✅
crowdstrike	fdr.session_token	✅
darktrace	httpjson.private_token	✅
darktrace	httpjson.public_token	✅
entityanalytics_entra_id	entity.secret	✅
entityanalytics_okta	user.okta_token	✅
eset_protect	cel.password	✅
f5_bigip	aws-s3.access_key_id	✅
f5_bigip	aws-s3.secret_access_key	✅
f5_bigip	aws-s3.session_token	✅
f5_bigip	http_endpoint.secret_header	✅
f5_bigip	http_endpoint.secret_value	✅
forcepoint_web	logs.dissect_tokenizer	✅
forgerock	httpjson.api_key	✅
forgerock	httpjson.api_secret	✅
gcp_pubsub	generic.credentials_json	✅
github	audit.access_token	✅
github	code_scanning.access_token	✅
github	dependabot.access_token	✅
github	issues.access_token	✅
github	secret_scanning.access_token	✅
github	secret_scanning.hide_secret	✅
google_cloud_storage	gcs.service_account_key	✅
google_scc	gcp-pubsub.credentials	✅
google_scc	httpjson.credentials	✅
google_workspace	httpjson.jwt_json	✅
http_endpoint	generic.hmac_key	✅
http_endpoint	generic.password	✅
http_endpoint	generic.secret_header	✅
http_endpoint	generic.secret_value	✅
httpjson	generic.oauth_google_credentials_json	✅
httpjson	generic.oauth_google_jwt_json	✅
httpjson	generic.oauth_secret	✅
httpjson	generic.password	✅
imperva_cloud_waf	event.access_key_id	✅
imperva_cloud_waf	event.api_id	✅
imperva_cloud_waf	event.api_key	✅
imperva_cloud_waf	event.secret_access_key	✅
imperva_cloud_waf	event.session_token	✅
infoblox_bloxone_ddi	httpjson.api_key	✅
jamf_protect	http_endpoint.secret_header	❌
jamf_protect	http_endpoint.secret_value	❌
jumpcloud	events.api_key	✅
lastpass	httpjson.provisioning_hash	✅
lumos	activity_logs.api_token	✅
lyve_cloud	audit.access_key_id	✅
lyve_cloud	audit.secret_access_key	✅
m365_defender	event.storage_account_key	✅
m365_defender	httpjson.client_secret	✅
m365_defender	httpjson.token_endpoint	✅
menlo	cel.token	✅
microsoft_defender_cloud	event.connection_string	✅
microsoft_defender_cloud	event.storage_account_key	✅
microsoft_defender_endpoint	log.client_secret	✅
microsoft_exchange_online_message_trace	httpjson.client_secret	✅
mimecast	httpjson.access_key	✅
mimecast	httpjson.app_id	✅
mimecast	httpjson.app_key	✅
mimecast	httpjson.secret_key	✅
o365	audit.client_secret	✅
o365	audit.client_secret	✅
o365	audit.key_passphrase	✅
o365	audit.token_scopes	✅
okta	httpjson.api_key	✅
okta	httpjson.jwk_json	✅
panw_cortex_xdr	alerts.api_token	✅
panw_cortex_xdr	alerts.token_id	✅
panw_cortex_xdr	incidents.api_token	✅
panw_cortex_xdr	incidents.token_id	✅
ping_one	http_endpoint.secret_header	✅
ping_one	http_endpoint.secret_value	✅
ping_one	httpjson.client_secret	✅
prisma_cloud	cel.password	✅
proofpoint_tap	httpjson.secret	✅
qualys_vmdr	cel.password	✅
rapid7_insightvm	httpjson.api_key	✅
sentinel_one	httpjson.api_token	✅
sentinel_one_cloud_funnel	aws-s3.access_key_id	✅
sentinel_one_cloud_funnel	aws-s3.secret_access_key	✅
sentinel_one_cloud_funnel	aws-s3.session_token	✅
sentinel_one_cloud_funnel	gcs.service_account_key	✅
slack	audit.oauth_token	✅
snyk	httpjson.api_token	✅
sophos_central	httpjson.client_secret	✅
symantec_edr_cloud	cel.client_secret	✅
tanium	aws-s3.access_key_id	✅
tanium	aws-s3.secret_access_key	✅
tanium	aws-s3.session_token	✅
tanium	http_endpoint.secret_header	✅
tanium	http_endpoint.secret_value	✅
tenable_io	httpjson.access_key	✅
tenable_io	httpjson.secret_key	✅
tenable_sc	httpjson.access_key	✅
tenable_sc	httpjson.secret_key	✅
ti_anomali	threatstream.secret	✅
ti_cif3	httpjson.api_token	✅
ti_crowdstrike	cel.client_secret	✅
ti_cybersixgill	threat.password	✅
ti_eclecticiq	cel.token	✅
ti_eset	httpjson.password	✅
ti_maltiverse	indicator.api_token	✅
ti_mandiant_advantage	threat_intelligence.mati_api_key_id	✅
ti_mandiant_advantage	threat_intelligence.mati_api_key_secret	✅
ti_misp	threat.api_token	✅
ti_misp	threat_attributes.api_token	✅
ti_opencti	cel.api_key	✅
ti_otx	pulses_subscribed.api_key	✅
ti_otx	threat.api_token	✅
ti_rapid7_threat_command	httpjson.api_key	✅
ti_recordedfuture	threat.api_token	✅
ti_threatconnect	cel.secret_key	✅
ti_threatq	threat.client_secret	✅
tines	httpjson.auth_token	✅
trellix_edr_cloud	aws-s3.access_key_id	✅
trellix_edr_cloud	aws-s3.secret_access_key	✅
trellix_edr_cloud	aws-s3.session_token	✅
trellix_epo_cloud	cel.api_key	✅
trellix_epo_cloud	cel.client_secret	✅
trend_micro_vision_one	httpjson.api_token	✅
wiz	cel.client_secret	✅
zerofox	httpjson.zerofox_api_token	✅
zeronetworks	audit.api_token	✅
zoom	webhook.crc_secret	✅
zoom	webhook.secret_header	✅
zoom	webhook.secret_value	✅
zscaler_zia	http_endpoint.secret_header	✅
zscaler_zia	http_endpoint.secret_value	✅

Codeowner: @elastic/stack-monitoring

Secret candidates:

Integration	Secret Candidate	Migrated
elasticsearch	elasticsearch/metrics.api_key	❌
elasticsearch	elasticsearch/metrics.password	❌
enterprisesearch	enterprisesearch/metrics.password	❌
kibana	http/metrics.password	❌
kibana	kibana/metrics.password	❌
logstash	cel.password	❌
logstash	logstash/metrics.password	❌

Codeowner: @elastic/profiling

Secret candidates:

Integration	Secret Candidate	Migrated
profiler_agent	pf-host-agent.profiler.secret_token	✅
profiler_collector	pf-elastic-collector.secret_token	❌
profiler_collector	pf-elastic-collector.tls_key_passphrase	❌
profiler_symbolizer	pf-elastic-symbolizer.tls_key_passphrase	❌

cc @elastic/ecosystem @kpollich @jillguyonnet

[andrewkroh]

Do we need to raise the minimum conditions.kibana.version in order for secrets to work? It's my understanding that there are known issues prior to 8.11.2.

[kpollich]

Do we need to raise the minimum conditions.kibana.version in order for secrets to work? It's my understanding that there are known issues prior to 8.11.2.

Yes there are a few known issues prior to 8.11.2 that have been fixed or are actively being backported:

• Replace secret vars for arrays fleet-server#3083 fixed in Fleet Server 8.11.1
• [Fleet] Improve UX for policy secrets kibana#171405 in the middle of backporting to 8.11.2
• [Fleet] Support integration secrets with required: false kibana#172078 backporting to 8.11.2 once the above is merged

It's not unlikely other issues will shake out with more integrations and teams using secrets heavily as part of this effort. If that happens we will provide guidance on Kibana versions with high impact fixes.

The absolute minimum version for secrets to function prior to any of these fixes is 8.10.0, but we likely want the best experience possible with these fixes and UX improvements so 8.11.2 is a better target as of right now.

[joshdover]

Do we need to raise the minimum conditions.kibana.version in order for secrets to work? It's my understanding that there are known issues prior to 8.11.2.

@kpollich Shouldn't older versions of Kibana (pre-secrets) still accept packages that have fields annotated with secret: true?

[jsoriano]

Do we need to raise the minimum conditions.kibana.version in order for secrets to work? It's my understanding that there are known issues prior to 8.11.2.

@kpollich Shouldn't older versions of Kibana (pre-secrets) still accept packages that have fields annotated with secret: true?

Yes, packages will still work in older versions of Kibana. Let me add a note about this in the description.

[kpollich]

Yes the secret flag will simply be ignored in older versions of Kibana. Newer stack versions will enjoy the security benefits of better secrets storage, but older stack versions will continue to work as they do now. If a user chooses to upgrade their stack, their policy secrets will be migrated to the new storage scheme whenever they perform an edit/upgrade operation on their package policy for a given package.

[tommyers-elastic]

am i understanding correctly?

secrets are ignored completely in kibana versions < 8.10.0.
there are some issues with secrets in kibana versions between 8.10.0 - 8.11.1 inclusive.
8.11.2 is the recommended minimum kibana version for integrations using secrets.

also, is there some script or something for identifying potential secret fields? there quite a lot of secret_access_key fields in AWS packages, for example, that are missing from the tables above.

are the migrations of the packages listed above the responsibility of the teams listed in the headings, or will there be some automation to open PRs for packages/fields identified here? if PRs were opened for all the above, the PR review process would deal with the question of whether they are actually secrets or not, as opposed to having to edit the tables here, then make the changes.

thanks

[jsoriano]

secrets are ignored completely in kibana versions < 8.10.0.
there are some issues with secrets in kibana versions between 8.10.0 - 8.11.1 inclusive.
8.11.2 is the recommended minimum kibana version for integrations using secrets.

Correct.

also, is there some script or something for identifying potential secret fields?

elastic-package will check for this for packages using format_version: "3.0.2". You can try to update any package to this version of the spec, and run elastic-package check, with the latest version of elastic-package.

there quite a lot of secret_access_key fields in AWS packages, for example, that are missing from the tables above.

Good point, we will review the list. We are doing it with a different script.

are the migrations of the packages listed above the responsibility of the teams listed in the headings, or will there be some automation to open PRs for packages/fields identified here?

Yes, migration of packages needs to be done by the owners of each package, and there is no available automation. We want each team to review each case, as the decision on considering a variable to be a secret or not is sensitive, and there are some consequences of the migration to secrets.

if PRs were opened for all the above, the PR review process would deal with the question of whether they are actually secrets or not, as opposed to having to edit the tables here, then make the changes.

Yeah, as PRs start to appear we can collectively keep the table updated.

[jsoriano]

there quite a lot of secret_access_key fields in AWS packages, for example, that are missing from the tables above.

Good point, we will review the list. We are doing it with a different script.

There are indeed some secret candidates in AWS, description updated with more findings.

[kpollich]

Hi all, we've identified and fixed another bug with secrets that will land in 8.12: elastic/kibana#172061

Thus, I've updated the advisory for the Kibana version constraint in the overview doc to 8.12.0. I'll send this same note out to the email thread as well.

[romulets]

Updated the table with Cloud Security Posture migrations. The ones that won't be migrated are explained on the linked PR

[kpollich]

Another secrets edge case fixed in 8.12.0: elastic/kibana#173048 + elastic/kibana#173115

[romulets]

We are rolling it back , because of elastic/kibana#173718 which is blocking any integration creation on 8.13.0-SNAPSHOT at the moment

[taylor-swanson]

The SEI integrations are currently blocked by elastic/kibana#173718 as well. I didn't test 8.13.0-SNAPSHOT, but did test the latest 8.12.0-SNAPSHOT (as of writing) and encountered the error there.

[taylor-swanson]

Looks like we have a regression in kibana/fleet/agent with rendering secret values. @ebeahan just ran into this (see his comment on the SEI issue here)

Policy:

          - set:
              target: header.XSignatureBase
              value: >-
                [[ sprintf "EG1-HMAC-SHA256
                client_token=%s;access_token=%s;timestamp=%s;nonce=%s;"
                "${SECRET_2}"
                "${SECRET_0}" (.header.Get
                "XTimestamp") uuid ]]

Rendered:

      - set:
          target: header.XSignatureBase
          value: '[[ sprintf "EG1-HMAC-SHA256 client_token=%s;access_token=%s;timestamp=%s;nonce=%s;"
            "$co.elastic.secret{eooc1YwBApAOY4Ce5tv2}" "$co.elastic.secret{eYoc1YwBApAOY4Ce5tv2}"
            (.header.Get "XTimestamp") uuid ]]'

I also observed this in practice with the system test in the akamai package (excerpt from the logs showing the authorization header, note the unexpanded values):

'Authorization: [EG1-HMAC-SHA256 client_token=$co.elastic.secret{GK_j1IwBnC-aiHiLhKgR};access_token=$co.elastic.secret{Fq_j1IwBnC-aiHiLhKgQ};timestamp=20240104T14:32:20+0000;nonce=300953b6-e8ea-4ab0-b001-a8c2b5e4fb7c;signature=u153JJMjirDHWIN4vCGlWDW6z6yeWvuXMn2KFJ+O3Wk=]'

[ebeahan]

Looks like we have a regression in kibana/fleet/agent with rendering secret values.

I also see the same rendering issue in 8.11.3, so it may be an unhandled use case instead of a regression.

[kpollich]

Thanks @taylor-swanson and @ebeahan for raising this.

This looks like a bug in Fleet Server's secrets rendering logic when multiple secret values appear in a multiline string.

@juliaElastic @michel-laterman would you be able to take a look at this? I'll also start investigating and see what I can come up with.

[jlind23]

@juliaElastic @michel-laterman would also be great to come up with some unit test for this to avoid any further problem. Does it sound good to you?

[juliaElastic]

I can take a look at this today.

[juliaElastic]

Reproduced the issue locally and it was really a problem with multiple secret refs in one variable.
Added a fix and unit test in this pr: elastic/fleet-server#3206

[jsoriano]

Hey @romulets, I see you marked some fields as won't migrate. An option for these fields would be to mark them as secret: false, this way the validators will ignore them and they won't be migrated nor need to be.

[kaiyan-sheng]

An option for these fields would be to mark them as secret: false, this way the validators will ignore them and they won't be migrated nor need to be.

Hi @jsoriano 👋 Does that mean it's preferred to have secret: false to mark the related fields that are not secret? For example with AWS credentials, the access key ID is secret: false, and secret access key is secret: true. Even though access key ID is not a secret variable but still good to mark it with false?

[ishleenk17]

this way the validators will ignore them and they won't be migrated nor need to be.

@jsoriano : Do these validators apply to the fields you have listed above for the Integrations, or to all the fields other than secret fields in the manifest file?

[paulb-elastic]

For Synthetics, we already encrypt sensitive fields in the Kibana saved objects as defined here, so should align these to those we mark as secure in the agent policy (cc @smith).

[lalit-satapathy]

A brief update on the secrets migration for o11y integration:

While we have migrated the large number of integrations to enable secrets, there are handful packages where password is embedded as part of the host URI.

For such cases, it would require changes, to ensure that the password field is passed independently and then marked as secret. This may not be a small change (We also need to ensure we do these changes in a non-breaking way), we are working it on case-by-case basis here.

[justinkambic]

Regarding Synthetics, here is a list of all the secret fields we have, if they should correspond to the integration.

[kpollich]

Hey folks FYI we have merged some UX improvements to smooth the onboarding experience with secrets on long-lived clusters that have been upgraded across stack versions: elastic/kibana#178045. This will land in 8.14, but but changes don't necessitate changing our target version for secrets.

[jlind23]

@bturquet can we get an update on aws, azure and synthetics?
@pierrehilbert the windows integration is flagged as owned by us but I do believe this is rather owned by @norrietaylor and his team right?

[paulb-elastic]

@bturquet can we get an update on aws, azure and synthetics?

@jlind23 for synthetics, this will need to be done by the Infra&Services team, so pinging @smith (rather than @bturquet) for this one

[pierrehilbert]

@jlind23 yes this is part of @nfritts's scope.

[smith]

Added this issue for synthetics changes: https://github.com/elastic/synthetics-dev/issues/355

[andrewkroh]

Problem statement

Nearly every package that supports TLS includes a YAML textarea to specify arbitrary settings, some of which are sensitive. Fleet added the ability to mark variables as secrets, and after saving this makes it impossible to read the values back for editing. This is a bad user experience.

Solutions

1. Fleet could provide a UI component tailored to the TLS config options that all Beats share. This would provide the best user experience because editing YAML is error prone and lacks validation. Fleet UI would be able to migrate existing config, treat sensitive data as secret, and render the config appropriately in the YAML handlebar template.
2. Every integration could create individual variables for each option in the Beats TLS config struct. From a UI point of view this would not be ideal for users. We probably don't have the ideal input types or a way to provide validation. Plus the sprawl of options would polute the UI in my opinion. Also there is no way for packages to migrate existing config options.
3. Add new package variables only for the options in the TLS config struct that are secret. Packages would depend on Beats ability to deep merge YAML options (i.e. ssl.key would be merged into the existing user defined ssl map). Existing deployments would have to be manually updated to take advantage of secrets but would not be required to. There would not be any means to force into using the secrets versus putting them into the YAML.

Solution 1 would be best for users but is there some team that would want to add a new UI component for the TLS config options?

Solution 3 could be implemented immediately by package developers and would not cause any breaking changes for users. But the UX and DX are inferior compared to solution 1. For this packages would add new secret manifest variables for:

• key
• key_password

The handlebar templates would keep the existing ssl: YAML key but add new ones.

{{if ssl_certificate_key}}
ssl.key: {{escape_string ssl_certificate_key}}
{{end}}
{{if ssl_key_password}}
ssl.key_password: {{escape_string ssl_key_password}}
{{end}}
{{if ssl}}
ssl:
{{ssl}}
{{end}}

[kpollich]

Fleet could provide a UI component tailored to the TLS config options that all Beats share. This would provide the best user experience because editing YAML is error prone and lacks validation. Fleet UI would be able to migrate existing config, treat sensitive data as secret, and render the config appropriately in the YAML handlebar template.

I think this is the best option from UX perspective to be sure. We can create a new tls type variable that maps to an opinionated UI component for setting these TLS settings. This would require another rollout/adoption process across integrations, but would overall feel like the right solution to me.

I would definitely be curious to see if there are teams who don't want this, and who value the existing UX around TLS settings. Or, if there are packages that have slight variations on the TLS fields that wouldn't be able to migrate to the new workflow.

[andrewkroh]

I would definitely be curious to see if there are teams who don't want this, and who value the existing UX around TLS settings.

I raised this question during the monthly Integration Development Office Hours meeting. There were no objections from the attendees.

Or, if there are packages that have slight variations on the TLS fields that wouldn't be able to migrate to the new workflow.

AFAIK there are two types of TLS settings usages -- the client context and server context. There are some small variations in the meaning of some fields. For example, in a client context the CA certs indicate which server's to trust, but in the server context this indicates which mTLS clients to trust. The server's CA certs need to be bundled into the ssl.certificate value. I can certainly help make sure all the differences are understood if we proceed.

[simitt]

@jsoriano I see that on Feb 9th you updated that the @elastic/obs-ds-intake-services team has updated the secrets; this is not actually the case.
elastic/apm-server#11450 is still open, and before we can make changes, we need to coordinate with the APM UI team to make according changes, so nothing in the curated UI is broken.

[jsoriano]

@jsoriano I see that on Feb 9th you updated that the @elastic/obs-ds-intake-services team has updated the secrets; this is not actually the case.

The main intention of this issue was to ensure that teams review the secret candidates of the packages they own. Secret candidates are marked as "Migrated" here if the team has reviewed them, by setting secret for them, either to true or to false. We keep this table updated with a script that we run from time to time, and it is not smart enough to differentiate between fields that have secret: false because they aren't actually secrets, or because they are pending on other changes before migrating to secret: true.

I understand this can be confusing in cases like this one, maybe we should stop updating the status with the script and do it manually? The truth is that I haven't run the script in some time.

From what I can see, all the fields owned by the Intake Services team have been properly reviewed, the APM fields have secret: false, and an issue has been opened to migrate the pending ones, and the field in the Azure Metrics package has secret: true.

[kpollich]

I think we have reached a satisfactory level of adoption here, and remaining secrets adoption work can be handled on a case-by-case basis. I am closing this issue. Thanks to all the integration teams for their efforts here!