[Bug]: Images cannot be pushed when backed by Ceph S3

[aki-ks]

zot version

v2.1.1

Describe the bug

I'm trying to set up a Zot registry on top of Hetzner S3 which is backed by Ceph. Zot fails whenever I'm trying to push an image.

To reproduce

Here's my Zot config. It's deployed on Kubernetes using the official Helm chart

{
  "storage": {
    "rootDirectory": "/tmp/zot",
    "dedupe": false,
    "storageDriver": {
        "name": "s3",
        "region": "fsn1",
        "regionEndpoint": "https://fsn1.your-objectstorage.com",
        "bucket": "<My Hetzner S3 Bucket name>",
        "encrypt": false,
        "secure": true,
        "v4auth": true,
        "forcepathstyle": false,
        "loglevel": "debugwithhttpbody"
    }
  },
  "http": {
    "address": "0.0.0.0",
    "port": "5000",
    "auth": {
      "htpasswd": {
        "path": "/secret/htpasswd"
      }
    }
  },
  "log": {
    "level": "debug"
  }
}

Let's try to push an image:

$ docker pull alpine:3.21
$ docker tag alpine:3.21 127.0.0.1:5000/alpine:3.21
$ docker push 127.0.0.1:5000/alpine:3.21
The push refers to repository [127.0.0.1:5000/alpine]
3e01818d79cd: Pushed 
manifest invalid: manifest invalid

Here's the docker daemon log (journalctl -u docker)

Dez 27 04:26:59 yamai dockerd[813]: time="2024-12-27T04:26:59.657866884+01:00" level=info msg="Attempting next endpoint for push after error: Get \"https://127.0.0.1:5000/v2/\": http: server gave HTTP response to HTTPS client" spanID=f39c2e55900e5144 traceID=5a5fa33ea25060f4f4219c898ed28792
Dez 27 04:27:00 yamai dockerd[813]: time="2024-12-27T04:27:00.717648010+01:00" level=error msg="Not continuing with push after error: manifest invalid: manifest invalid" spanID=f39c2e55900e5144 traceID=5a5fa33ea25060f4f4219c898ed28792

The full Zot log with verbose S3-Client logging enabled is attached: zot.log

Expected behavior

No response

Screenshots

No response

Additional context

No response

[rchincha]

@aki-ks pls try the latest 2.1.2-rc release

[aki-ks]

I've just redeployed it using helm-chart 0.1.65 which includes v2.1.2-rc3 but still experience the same issues

Here is my S3 policy for completeness:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam:::user/p4035538:<my-access-key>" },
      "Action": ["s3:*"],
      "Resource": [
        "arn:aws:s3:::<my-bucket>",
        "arn:aws:s3:::<my-bucket>/*"
      ]
    }
  ]
}

Still the same issues while pushing

$ docker push 127.0.0.1:5000/alpine:3.21
The push refers to repository [127.0.0.1:5000/alpine]
3e01818d79cd: Pushed 
manifest invalid: manifest invalid

And here's the Zot log once again
zot-f47b89555-cvg5g.log

[rchincha]

@aki-ks can you fix your zot configuration like so:

https://github.com/project-zot/zot/blob/main/examples/config-docker-compat.json#L9

[aki-ks]

The change solved the issue thank's a lot 👍
I would not have expected this to be the cause 🙈