Certificate, CA chain and private key in debug logs

[mikejoh]

Setup Details

CIS Version : 2.18.1
Build: f5networks/k8s-bigip-ctlr:2.18.1
BIGIP Version: BIG-IP 17.1.1.3 Build 0.0.5 Point Release 3
AS3 Version: 3.51 Build 5
Agent Mode: AS3
Orchestration: K8S
Orchestration Version: 1.28.13
Pool Mode: Cluster
Additional Setup details: Cilium as CNI

Description

When we've enabled DEBUG log in the k8s-bigip-ctlr we can see that it outputs certificates, CA chain and private key like this:

2024/12/09 07:38:50 [DEBUG] [AS3] Raw response from Big-IP: map[declaration:map[class:ADC controls:map[archiveTime
stamp:2024-12-09T07:38:44.342Z class:Controls userAgent:CIS/v2.18.1 K8S/v1.28.13] id:urn:uuid:85626792-9ee7
-46bb-8fc8-4ba708cfdc1d k8s-cluster01:map[Shared:map[app-certificate_0:map[certificate:-----BEGIN CERTIFICATE-----
<removed>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<removed>
-----END CERTIFICATE-----
 chainCA: class:Certificate privateKey:-----BEGIN RSA PRIVATE KEY-----
<removed>
-----END RSA PRIVATE KEY-----

Which feels wrong, even if we're setting DEBUG as log level.

I would not expect this since there's code that tries to remove that information from the log output (no matter which log level AFAICT):

k8s-bigip-ctlr/pkg/agent/as3/postManager.go

Lines 403 to 463 in e3c0ec4

	func (postMgr *PostManager) logAS3Response(responseMap map[string]interface{}) {
	// removing the certificates/privateKey from response log
	if declaration, ok := (responseMap["declaration"]).([]interface{}); ok {
	for _, value := range declaration {
	if tenantMap, ok := value.(map[string]interface{}); ok {
	for _, value2 := range tenantMap {
	if appMap, ok := value2.(map[string]interface{}); ok {
	for _, obj := range appMap {
	if crt, ok := obj.(map[string]interface{}); ok {
	if crt["class"] == "Certificate" {
	crt["certificate"] = ""
	crt["privateKey"] = ""
	crt["chainCA"] = ""
	}
	}
	}
	}
	}
	}
	}
	decl, err := json.Marshal(declaration)
	if err != nil {
	log.Errorf("[AS3] error while reading declaration from AS3 response: %v\n", err)
	return
	}
	responseMap["declaration"] = as3Declaration(decl)
	}
	log.Debugf("[AS3] Raw response from Big-IP: %v ", responseMap)
	}
	func (postMgr *PostManager) logAS3Request(cfg string) {
	var as3Config map[string]interface{}
	err := json.Unmarshal([]byte(cfg), &as3Config)
	if err != nil {
	log.Errorf("Request body unmarshal failed: %v\n", err)
	}
	adc := as3Config["declaration"].(map[string]interface{})
	for _, value := range adc {
	if tenantMap, ok := value.(map[string]interface{}); ok {
	for _, value2 := range tenantMap {
	if appMap, ok := value2.(map[string]interface{}); ok {
	for _, obj := range appMap {
	if crt, ok := obj.(map[string]interface{}); ok {
	if crt["class"] == "Certificate" {
	crt["certificate"] = ""
	crt["privateKey"] = ""
	crt["chainCA"] = ""
	}
	}
	}
	}
	}
	}
	}
	decl, err := json.Marshal(as3Config)
	if err != nil {
	log.Errorf("[AS3] Unified declaration error: %v\n", err)
	return
	}
	log.Debugf("[AS3] Unified declaration: %v\n", as3Declaration(decl))
	}

For some reason the certificate, CA chain and private key are still passed to and outputted in logs.

Steps To Reproduce

1. Change log level of k8s-bigip-ctlr to DEBUG
2. Create a VirtualServer with a TLSProfile that references a secret (we're issuing certificates via cert-manager)
3. Check the logs

Expected Result

DEBUG level logs without sensitive information, basically not the private key.

Actual Result

N/A

Diagnostic Information

N/A

Observations (if any)

[vklohiya]

@mikejoh , this information is removed when log-level is set to AS3DEBUG, In this log-level CIS logs the request and response of AS3 and reduct the Certificate/Key information. As parsing the request and response is slightly costly operation, there may be some impact on CIS performance.

[vklohiya]

Alternatively you can also disable the --log-as3-response flag if you are not debugging the AS3 API.

[mikejoh]

@vklohiya Thanks for the quick answer! We do indeed have the --log-as3-response flag set and log level set to DEBUG, we'll try to reconfigure these flags a bit.

[trinaths]

@mikejoh - We are unable to reproduce this issue with the give data. Can you share more info to automation_toolchain_pm [email protected]

[mikejoh]

@mikejoh - We are unable to reproduce this issue with the give data. Can you share more info to automation_toolchain_pm [email protected]

Running without the --log-as3-response flag fixed it. We don't see anything sensitive (private key) in the logs!