Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add config to disable HTTP proxy logging #12665

Merged
merged 7 commits into from
Jun 12, 2024
Merged

Add config to disable HTTP proxy logging #12665

merged 7 commits into from
Jun 12, 2024

Conversation

adleong
Copy link
Member

@adleong adleong commented May 30, 2024

Fixes #12620

When the Linkerd proxy log level is set to debug or higher, the proxy logs HTTP headers which may contain sensitive information.

While we want to avoid logging sensitive data by default, logging of HTTP headers can be a helpful debugging tool. Therefore, we add a proxy.logHTTPHeaders Helm value which prevents the logging of HTTP headers when set to false. The default value of this value is false so that headers cannot be logged unless users opt-in.

@adleong adleong requested a review from a team as a code owner May 30, 2024 23:56
Signed-off-by: Alex Leong <[email protected]>
Signed-off-by: Alex Leong <[email protected]>
Signed-off-by: Alex Leong <[email protected]>
Signed-off-by: Alex Leong <[email protected]>
Copy link
Member

@alpeb alpeb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it definitely make sense to be able to tune on a per-workload basis, so that people can lock this down at the cluster level and have the option to open it only where they want, say when debugging a specific service.

@@ -29,7 +29,7 @@ env:
value: {{.Values.proxy.requireTLSOnInboundPorts | quote}}
{{ end -}}
- name: LINKERD2_PROXY_LOG
value: {{.Values.proxy.logLevel | quote}}
value: "{{.Values.proxy.logLevel}}{{ if not (eq .Values.proxy.logHTTPHeaders "insecure") }},linkerd_proxy_http::client[{headers}]=off{{ end }}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it still is a good practice to leave the quote function in.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain? it was my impression that the quote function just adds the quotation marks. Using quote here is difficult because its a pipeline function and we're constructing the string in the template.

Comment on lines +156 to +160
# -- (`off` or `insecure`) If set to `off`, will prevent the proxy from
# logging HTTP headers. If set to `insecure`, HTTP headers may be logged
# verbatim. Note that setting this to `insecure` is not alone sufficient to
# log HTTP headers; the proxy logLevel must also be set to debug.
logHTTPHeaders: "off"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer having a regular boolean here like everywhere else instead of introducing special values. E.g. if people use "true" that would get misinterpreted in the template as "off" (according to the change in _proxy.tpl). To avoid that you'd have to validate the possible values, which we don't do for regular booleans as those are "standard" (true, >0, non-empty vs false, 0, empty).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The motivation for not using a boolean here is that using a scary sounding value like "insecure" helps users realize that enabling this has security implications. It also leaves the door open to adding other modes in the future like "hash", "redact", or "obfuscate".

Comment on lines +258 to +261
if override, ok := annotations[k8s.ProxyLogHTTPHeaders]; ok {
values.Proxy.LogHTTPHeaders = override
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a thought, should we do any logging here in case the value is unknown? For example, a label value of insecur will still disable http headers in logs, but perhaps the intention was the opposite. Should we report invalid values to the user somehow?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good call. updated.

bin/linkerd upgrade --set proxy.logHTTPHeaders=insecur
× Could not render upgrade configuration: failed to render the template: execution error at (linkerd-control-plane/templates/proxy-injector.yaml:77:12): logHTTPHeaders must be one of: insecure | off
For troubleshooting help, visit: https://linkerd.io/upgrade/#troubleshooting

Signed-off-by: Alex Leong <[email protected]>
@adleong adleong merged commit e0fe024 into main Jun 12, 2024
40 checks passed
@adleong adleong deleted the alex/no-header-logging branch June 12, 2024 00:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Linkerd-proxy logging full header contents of incoming http requests for log level debug and trace.
3 participants