mismatch Content-Length when copying recursively between Google Artifact Registry repositories

[rorynickolls-skyral]

What happened in your environment?

I'm trying to copy an image and all related artefacts (signature, attestations) from one Google Artifact Registry repository to another using oras.

It copies the image itself fine, but fails at copying the signature with the following error message:

Error: GET "https://europe-west2-docker.pkg.dev/v2/***/***/nginx/manifests/sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d": mismatch Content-Length

When I look at debug logs, response headers indicate a content length of 739 bytes:

< Response headers:
   "Content-Type": "application/vnd.oci.image.manifest.v1+json"
   "X-Content-Type-Options": "nosniff"
   "X-Xss-Protection": "0"
   "Date": "Wed, 19 Feb 2025 10:48:39 GMT"
   "Alt-Svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000"
   "Content-Length": "739"
   "Docker-Content-Digest": "sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "X-Frame-Options": "SAMEORIGIN" 

This matches up to the size of the file in the GAR UI:

However I do note the following, also in the GAR UI, which makes me think that oras may be receiving a compressed artefact and expecting a content length that matches the compressed size (521), not the actual size (739):

What did you expect to happen?

The image and related artefacts were copied to the other GAR repository like-for-like.

How can we reproduce it?

The command I am running is this:

oras cp -r -vd europe-west2-docker.pkg.dev/***/***/nginx:c9b13fb europe-west2-docker.pkg.dev/***/***/nginx:1.0

The signed image is just the standard nginx image pushed up to a local Google Artifact Registry and signed using Cosign's OCI v1.1 support:

COSIGN_EXPERIMENTAL=1 cosign sign --registry-referrers-mode oci-1-1 europe-west2-docker.pkg.dev/***/***/nginx:c9b13fb --tlog-upload=false --key=gcpkms://***

What is the version of your ORAS CLI?

Version:    1.2.2+Homebrew
Go version: go1.23.4

What is your OS environment?

OSX 15.3.1

Are you willing to submit PRs to fix it?

• Yes, I am willing to fix it.

[rorynickolls-skyral]

A bit of extra info: I can copy this signature artefact directly using the following process:

$ oras discover europe-west2-docker.pkg.dev/***/***/nginx:c9b13fb   
europe-west2-docker.pkg.dev/***/***/nginx@sha256:b3d61268ae447295d1619e2a6156ddf8635a65aff40cfd65588a1024af77121a
└── <unknown>
    └── sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d

$ oras copy europe-west2-docker.pkg.dev/***/***/nginx@sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d europe-west2-docker.pkg.dev/***/***/nginx@sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d
✓ Exists  application/vnd.docker.distribution.manifest.v2+json                                                                                                                                                                                                                                                                                                                                                                                                                1.74/1.74 kB 100.00%     0s
  └─ sha256:b3d61268ae447295d1619e2a6156ddf8635a65aff40cfd65588a1024af77121a
✓ Mounted application/vnd.dev.cosign.simplesigning.v1+json                                                                                                                                                                                                                                                                                                                                                                                                                      288/288  B 100.00%     0s
  └─ sha256:d8051af3f4240878804b5b7aded16e7985feb79274385e79d3c09fe83299062f
✓ Mounted application/vnd.dev.cosign.artifact.sig.v1+json                                                                                                                                                                                                                                                                                                                                                                                                                       233/233  B 100.00%     0s
  └─ sha256:da4210f644766a64b44c0479008c856f8775e0041f9496b59c275b36ecaeb460
✓ Copied  application/vnd.oci.image.manifest.v1+json                                                                                                                                                                                                                                                                                                                                                                                                                            739/739  B 100.00%  457ms
  └─ sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d
Copied [registry] europe-west2-docker.pkg.dev/***/***/nginx@sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d => [registry] europe-west2-docker.pkg.dev/***/***/nginx@sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d
Digest: sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d

I can then run oras discover again on the copied artefact and everything appears OK. The only issue with this approach is that I end up with an extra manifest in the repository.

[FeynmanZhou]

Hi @rorynickolls-skyral , this looks strange. I am not sure whether there is any file size limit on GAR. Have you tested copy this signed image with signature in other registries?

[Wwwsylvia]

This looks interesting. @rorynickolls-skyral Could you run the following commands and share the results?

oras manifest fetch --descriptor $registry/$repoName@sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d
oras discover $registry/$repoName@sha256:b3d61268ae447295d1619e2a6156ddf8635a65aff40cfd65588a1024af77121a --format json`

[rorynickolls-skyral]

Here are those commands:

$ oras manifest fetch --descriptor europe-west2-docker.pkg.dev/***/***/nginx@sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d
{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d","size":739}

$ oras discover europe-west2-docker.pkg.dev/***/***/nginx@sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d --format json
{
  "manifests": []
}

I've also just tested by copying to a fresh zot registry running locally, and get the same error. Taking out the -r flag works fine in both scenarios, but of course then it doesn't copy this signature over.

$ oras cp -r europe-west2-docker.pkg.dev/***/***/nginx:c9b13fb localhost:5000/test/nginx:1.0
Error: GET "https://europe-west2-docker.pkg.dev/v2/***/***/nginx/manifests/sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d": mismatch Content-Length

[Wwwsylvia]

@rorynickolls-skyral Thanks!
Can you run

oras discover $registry/$repoName@sha256:b3d61268ae447295d1619e2a6156ddf8635a65aff40cfd65588a1024af77121a --format json

instead? It's sha256:b3d61268ae447295d1619e2a6156ddf8635a65aff40cfd65588a1024af77121a, which should be the digest of nginx:c9b13fb?

[rorynickolls-skyral]

Yep, this time I get this:

$ oras discover europe-west2-docker.pkg.dev/***/***/nginx@sha256:b3d61268ae447295d1619e2a6156ddf8635a65aff40cfd65588a1024af77121a --format json
{
  "manifests": [
    {
      "reference": "europe-west2-docker.pkg.dev/***/***/nginx@sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d",
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d",
      "size": 521
    }
  ]
}

That size of 521 matches the "virtual size" in the UI screenshot above, if that's what you're looking for.

[Wwwsylvia]

Yeah this explains why. While performing recursive copy cp -r, ORAS internally discovers the descriptors of referrers of the target manifest and initiates copy from there. Since actual size of the referrer mismatches the size in the referrer descriptor, the validation fails.

It looks like the Google Artifact Registry includes a "virtual" size in the results of referrers API?
@rorynickolls-skyral Would you mind downloading ORAS v1.3.0-beta.1, which provides a better logging experience, and share the debug logs for the oras discover command you just ran with --debug flag?

[rorynickolls-skyral]

I had to build from source, here are the logs:

❯ go run cmd/oras/main.go discover europe-west2-docker.pkg.dev/***/***/nginx@sha256:b3d61268ae447295d1619e2a6156ddf8635a65aff40cfd65588a1024af77121a --format json --debug
[2025-02-24T16:39:41.102847Z][DEBUG]: --> Request #0
> Request URL: "https://europe-west2-docker.pkg.dev/v2/***/***/nginx/manifests/sha256:b3d61268ae447295d1619e2a6156ddf8635a65aff40cfd65588a1024af77121a"
> Request method: "HEAD"
> Request headers:
   "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.oci.artifact.manifest.v1+json"
   "User-Agent": "oras/1.3.0-beta.1+unreleased"


[2025-02-24T16:39:41.464604Z][DEBUG]: <-- Response #0
< Response Status: "401 Unauthorized"
< Response headers:
   "X-Xss-Protection": "0"
   "Date": "Mon, 24 Feb 2025 16:39:41 GMT"
   "Content-Length": "102"
   "Alt-Svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000"
   "Content-Type": "application/json; charset=utf-8"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "X-Frame-Options": "SAMEORIGIN"
   "Www-Authenticate": "Bearer realm=\"https://europe-west2-docker.pkg.dev/v2/token\",service=\"europe-west2-docker.pkg.dev\",scope=\"repository:***/***/nginx:pull\""
   "X-Content-Type-Options": "nosniff"
< Response body:
   Response body is empty


[2025-02-24T16:39:42.466719Z][DEBUG]: --> Request #1
> Request URL: "https://europe-west2-docker.pkg.dev/v2/token?scope=repository%3A***%2F***%2Fnginx%3Apull&service=europe-west2-docker.pkg.dev"
> Request method: "GET"
> Request headers:
   "Authorization": "*****"
   "User-Agent": "oras/1.3.0-beta.1+unreleased"


[2025-02-24T16:39:42.554218Z][DEBUG]: <-- Response #1
< Response Status: "200 OK"
< Response headers:
   "X-Content-Type-Options": "nosniff"
   "X-Frame-Options": "SAMEORIGIN"
   "X-Xss-Protection": "0"
   "Date": "Mon, 24 Feb 2025 16:39:42 GMT"
   "Alt-Svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000"
   "Content-Type": "application/json"
< Response body:
   Response body redacted due to potential credentials


[2025-02-24T16:39:42.554312Z][DEBUG]: --> Request #2
> Request URL: "https://europe-west2-docker.pkg.dev/v2/***/***/nginx/manifests/sha256:b3d61268ae447295d1619e2a6156ddf8635a65aff40cfd65588a1024af77121a"
> Request method: "HEAD"
> Request headers:
   "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.oci.artifact.manifest.v1+json"
   "Authorization": "*****"
   "User-Agent": "oras/1.3.0-beta.1+unreleased"


[2025-02-24T16:39:42.67081Z][DEBUG]: <-- Response #2
< Response Status: "200 OK"
< Response headers:
   "Docker-Content-Digest": "sha256:b3d61268ae447295d1619e2a6156ddf8635a65aff40cfd65588a1024af77121a"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Date": "Mon, 24 Feb 2025 16:39:42 GMT"
   "Alt-Svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000"
   "Content-Length": "1778"
   "Content-Type": "application/vnd.docker.distribution.manifest.v2+json"
< Response body:
   Response body is empty


[2025-02-24T16:39:42.671093Z][DEBUG]: --> Request #3
> Request URL: "https://europe-west2-docker.pkg.dev/v2/***/***/nginx/referrers/sha256:b3d61268ae447295d1619e2a6156ddf8635a65aff40cfd65588a1024af77121a"
> Request method: "GET"
> Request headers:
   "User-Agent": "oras/1.3.0-beta.1+unreleased"
   "Authorization": "*****"


[2025-02-24T16:39:42.795585Z][DEBUG]: <-- Response #3
< Response Status: "200 OK"
< Response headers:
   "Alt-Svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000"
   "Content-Type": "application/vnd.oci.image.index.v1+json"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "X-Content-Type-Options": "nosniff"
   "X-Frame-Options": "SAMEORIGIN"
   "X-Xss-Protection": "0"
   "Date": "Mon, 24 Feb 2025 16:39:42 GMT"
   "Content-Length": "240"
< Response body:
{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[{"mediaType":"application/vnd.oci.image.manifest.v1+json","size":521,"digest":"sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d"}]}


{
  "manifests": [
    {
      "reference": "europe-west2-docker.pkg.dev/***/***/nginx@sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d",
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d",
      "size": 521
    }
  ]
}

[Wwwsylvia]

Thanks @rorynickolls-skyral for the information.

The logs do show that the referrers are obtained from Google Artifact Registry's referrers API, and the size for sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d was 521 instead of 739 in the referrers result.

Request URL: "https://europe-west2-docker.pkg.dev/v2/***/***/nginx/referrers/sha256:b3d61268ae447295d1619e2a6156ddf8635a65aff40cfd65588a1024af77121a"

Here is a comparison between the descriptor for the manifest sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d and the descriptor for the same manifest in sha256:b3d61268ae447295d1619e2a6156ddf8635a65aff40cfd65588a1024af77121a's referrer result:

Descriptor for sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d:

{
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "digest": "sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d",
    "size": 739
}

Descriptor for sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d as a referrer of sha256:b3d61268ae447295d1619e2a6156ddf8635a65aff40cfd65588a1024af77121a:

{
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "digest": "sha256:12fcff0daec29e1593647d6f88e3329d4c9f4054035eba331f8e3c13e3488c4d",
  "size": 521
}

@rorynickolls-skyral You might want to seek support from Google Artifact Registry to understand this behavior🧐