[Bug]: bearer authentication required on /metrics

[seniorquico]

zot version

v2.0.0

Describe the bug

The /metrics endpoint requires authentication thanks to #1895.

However, when Bearer authentication is configured, no other authentication methods are supported simultaneously:

func AuthHandler(ctlr *Controller) mux.MiddlewareFunc {
	authnMiddleware := &AuthnMiddleware{log: ctlr.Log}

	if ctlr.Config.IsBearerAuthEnabled() {
		return bearerAuthHandler(ctlr)
	}

	return authnMiddleware.tryAuthnHandlers(ctlr)
}

This breaks our Prometheus metric pulling client (we use Netdata, but also tried with Prometheus). The client gets 401 Unauthorized, and they appear to have no support for the Token Authentication Specification (which seems reasonable).

To reproduce

1. Configuration

{
    "distSpecVersion": "1.1.0-dev",
    "GoVersion": "go1.21.5",
    "Commit": "v2.0.0-2-g1215f8d",
    "ReleaseTag": "v2.0.0",
    "BinaryType": "-metrics-sync",
    "Storage": {
        "RootDirectory": "/srv/zot",
        "Dedupe": true,
        "RemoteCache": false,
        "GC": true,
        "Commit": false,
        "GCDelay": 3600000000000,
        "GCInterval": 86400000000000,
        "Retention": null,
        "StorageDriver": null,
        "CacheDriver": null,
        "SubPaths": null
    },
    "HTTP": {
        "Address": "x.x.x.x",
        "ExternalURL": "",
        "Port": "443",
        "AllowOrigin": "",
        "TLS": {
            "Cert": "/etc/zot/fullchain.pem",
            "Key": "/etc/zot/privkey.pem",
            "CACert": ""
        },
        "Auth": {
            "FailDelay": 5,
            "HTPasswd": {
                "Path": "/etc/zot/htpasswd"
            },
            "LDAP": null,
            "Bearer": {
                "Realm": "https://token-auth-server.example.com/api/v2/registry/token",
                "Service": "registry.example.com",
                "Cert": "/etc/zot/auth.crt"
            },
            "OpenID": null,
            "APIKey": false
        },
        "AccessControl": null,
        "Realm": "Example Container Registry",
        "Ratelimit": {
            "Rate": 10,
            "Methods": [{
                    "Method": "GET",
                    "Rate": 5
                }
            ]
        }
    },
    "Log": {
        "Level": "debug",
        "Output": "/var/log/zot/zot.log",
        "Audit": "/var/log/zot/zot-audit.log"
    },
    "Extensions": null,
    "scheduler": null
}

2. Client tool used

• Prometheus & Netdata configured to use a HTTP Basic authentication username+password for scraping /metrics.

3. Seen error

• Prometheus & Netdata get a 401 Unauthorized challenge, requiring a JWT bearer token.

Expected behavior

In the authentication docs, the LDAP section includes this comment:

When both htpasswd and LDAP configuration are specified, LDAP authentication is given preference. Because htpasswd authentication is strictly local and requires no remote service, htpasswd serves as a fail-safe authentication mechanism should LDAP become unavailable.

It's not really a fail-safe in this instance, but it would be awesome if zot supported simultaneous configurations of JWT bearer and HTTP basic/htpasswd for compatibility with clients that do not support the Token Authentication Specification.

Screenshots

No response

Additional context

No response

[rchincha]

@seniorquico zot also supports social logins. We had a little trouble with coordinating all these paths wrt basic/bearer authN. Will follow this up with a more complete answer.

[rchincha]

@alexstan12 can you answer this pls?

[seniorquico]

We encountered another bearer auth incompatibility example... We setup one zot server to which we push a lot of images, and we setup a second zot server that we hoped to configure as a pull-through image cache. Unfortunately, the sync plugin appears to only support HTTP basic authentication. Having bearer auth enabled, we cannot use the sync plugin.

zot also supports social logins

@rchincha Yes, but we have an existing system for issuing bearer auth credentials according to the Token Authentication Specification. It is working incredibly well, aside from the limitations imposed by not supporting other "fallback" authentication methods.

We had a little trouble with coordinating all these paths wrt basic/bearer authN.

Can't the Authorization header's value provide the hint as to what authentication method to try? e.g. It begins with bearer, attempt the bearer auth. It begins with basic, attempt the basic auth. There's still the ambiguity between the Token Authentication Spec bearer credential versus an OIDC/social bearer credential. That one seems a little bit trickier- I understand how the Token Authentication Spec demands specific response patterns to correctly drive the workflow. Even the half-step to supporting HTTP basic + bearer at the same time would be a huge step forward to interop with these other plugins/systems.

[rchincha]

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/WWW-Authenticate

@seniorquico Currently, we return only one "auth-scheme". In theory, multiple WWW-Authenticate (w auth-scheme) should be possible, however, not sure if clients will break if we did that.

@peusebiu maybe we can evaluate?

[seniorquico]

Currently, we return only one "auth-scheme". In theory, multiple WWW-Authenticate (w auth-scheme) should be possible, however, not sure if clients will break if we did that.

My comment was too narrowly focused on the request side... Handling this disambiguation on the response side is tricky!

Feel free to keep me posted if there's any way for our team to get more engaged on this issue.

[rchincha]

opencontainers/wg-auth#12
^ we have filed an issue with OCI, let's wait and see what they say.

[seniorquico]

The sync plugin in v2.0.3 now works with bearer authentication thanks to #2222. That particular use case is now no longer a concern and works very well with our token issuance service based on the Token Authentication Specification. This is awesome, thanks @peusebiu!

The original problem- /metrics requiring bearer authentication and Prometheus/other tools not supporting the Token Authentication Specification- still persists.

@rchincha @peusebiu Is there any update on what may be done to get Prometheus working with zot when it is configured with bearer authentication?

The only other alternative I can brainstorm is to create a localhost proxy server, just for Prometheus to call, that implements the bearer auth workflow and returns the /metrics content. I just ran a quick test with cURL, by manually generating the JWT with access to scope repository::pull, and it worked. It's quite the workaround, though.

[batazor]

The same goes for the availability handles - startupProbe, etc

I set up authorization through github, eventually probe can't get the data due to 401 error