Harbor Replication fails with PROJECT POLICY VIOLATION though image is signed

[TnGadaria]

Hi
I have a registry with two replicas and am using notation to sign the image. I am facing an issue with the replication towards both of the registries, as per the replication logs it does not find that the image is signed and fails with PROJECT POLICY VIOLATION, however, I am pushing the image and then signing the same, and expecting the image to replicate after signing.

I have noticed many times that, after pushing and signing the image to the registry, the replication registries create the repository if it's not there but does not replicate any artifacts. This is an intermediate issue, however, observed frequently.

Any help would be appreciated.

Harbor version: 2.11.0
Notary version: 1.1.0

Logs for more info:

2024-07-17T11:50:24Z [INFO] [/controller/replication/transfer/image/transfer.go:139]: client for source registry [type: harbor, URL: http://core:8080, insecure: true] created
2024-07-17T11:50:24Z [INFO] [/controller/replication/transfer/image/transfer.go:149]: client for destination registry [type: harbor, URL: https://165.225.18.211, insecure: true] created
2024-07-17T11:50:24Z [INFO] [/controller/replication/transfer/image/transfer.go:182]: copying opsarch-private/alpine-fix:[v1,sha256:6ec6600f1d51b51480ddfcc23194c007da7e02eb2162388e7affc607e97f6659,sha256:91941698761dbed7a92add7e62c6ef2832c52c9e1a58460de60b145622758b32](source registry) to opsarch-private/alpine-fix:[v1,sha256:6ec6600f1d51b51480ddfcc23194c007da7e02eb2162388e7affc607e97f6659,sha256:91941698761dbed7a92add7e62c6ef2832c52c9e1a58460de60b145622758b32](destination registry)...
2024-07-17T11:50:24Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying opsarch-private/alpine-fix:v1(source registry) to opsarch-private/alpine-fix:v1(destination registry)...
2024-07-17T11:50:24Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact opsarch-private/alpine-fix:v1 ...
2024-07-17T11:50:24Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact opsarch-private/alpine-fix:v1 pulled
2024-07-17T11:50:25Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:2b76c9374f7571d8faca1b7cbe435aa8c9ebfc3a0d94c15f36f188a0cd15e2b9(the 1th running)...
2024-07-17T11:50:25Z [DEBUG] [/controller/replication/transfer/image/transfer.go:392]: the blob size is 1467 bytes
2024-07-17T11:50:25Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:2b76c9374f7571d8faca1b7cbe435aa8c9ebfc3a0d94c15f36f188a0cd15e2b9 completed
2024-07-17T11:50:25Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:ec99f8b99825a742d50fb3ce173d291378a46ab54b8ef7dd75e5654e2a296e99(the 1th running)...
2024-07-17T11:50:25Z [DEBUG] [/controller/replication/transfer/image/transfer.go:392]: the blob size is 3623844 bytes
2024-07-17T11:50:26Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:ec99f8b99825a742d50fb3ce173d291378a46ab54b8ef7dd75e5654e2a296e99 completed
2024-07-17T11:50:26Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:4e997a7a8a24a8d55e3d7b3816f1fdf016afa475534988e70bc65ab63bf1e61d(the 1th running)...
2024-07-17T11:50:26Z [DEBUG] [/controller/replication/transfer/image/transfer.go:392]: the blob size is 4607237 bytes
2024-07-17T11:50:27Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:4e997a7a8a24a8d55e3d7b3816f1fdf016afa475534988e70bc65ab63bf1e61d completed
2024-07-17T11:50:27Z [INFO] [/controller/replication/transfer/image/transfer.go:496]: pushing the manifest of artifact opsarch-private/alpine-fix:v1 ...
2024-07-17T11:50:27Z [INFO] [/controller/replication/transfer/image/transfer.go:508]: the manifest of artifact opsarch-private/alpine-fix:v1 pushed
2024-07-17T11:50:27Z [INFO] [/controller/replication/transfer/image/transfer.go:253]: copy opsarch-private/alpine-fix:v1(source registry) to opsarch-private/alpine-fix:v1(destination registry) completed
2024-07-17T11:50:27Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying opsarch-private/alpine-fix:sha256:6ec6600f1d51b51480ddfcc23194c007da7e02eb2162388e7affc607e97f6659(source registry) to opsarch-private/alpine-fix:sha256:6ec6600f1d51b51480ddfcc23194c007da7e02eb2162388e7affc607e97f6659(destination registry)...
2024-07-17T11:50:27Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact opsarch-private/alpine-fix:sha256:6ec6600f1d51b51480ddfcc23194c007da7e02eb2162388e7affc607e97f6659 ...
2024-07-17T11:50:27Z [ERROR] [/controller/replication/transfer/image/transfer.go:470]: failed to pull the manifest of artifact opsarch-private/alpine-fix:sha256:6ec6600f1d51b51480ddfcc23194c007da7e02eb2162388e7affc607e97f6659: http status code: 412, body: {"errors":[{"code":"PROJECTPOLICYVIOLATION","message":"The image is not signed by notation."}]}
2024-07-17T11:50:27Z [ERROR] [/controller/replication/transfer/image/transfer.go:194]: http status code: 412, body: {"errors":[{"code":"PROJECTPOLICYVIOLATION","message":"The image is not signed by notation."}]}
2024-07-17T11:50:27Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying opsarch-private/alpine-fix:sha256:91941698761dbed7a92add7e62c6ef2832c52c9e1a58460de60b145622758b32(source registry) to opsarch-private/alpine-fix:sha256:91941698761dbed7a92add7e62c6ef2832c52c9e1a58460de60b145622758b32(destination registry)...
2024-07-17T11:50:27Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact opsarch-private/alpine-fix:sha256:91941698761dbed7a92add7e62c6ef2832c52c9e1a58460de60b145622758b32 ...
2024-07-17T11:50:27Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact opsarch-private/alpine-fix:sha256:91941698761dbed7a92add7e62c6ef2832c52c9e1a58460de60b145622758b32 pulled
2024-07-17T11:50:27Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact opsarch-private/alpine-fix:sha256:91941698761dbed7a92add7e62c6ef2832c52c9e1a58460de60b145622758b32 already exists on the destination registry, skip
2024-07-17T11:50:27Z [ERROR] [/controller/replication/transfer/image/transfer.go:200]: got error during the whole transfer period, mark the job failure

[TnGadaria]

I found the issue #10262 and I am facing the same. Can we set the timeout? because I am taking more than 2 minutes after pushing to sign the image and replication fails.

[zyyw]

@MinerYang

[zyyw]

please refer this doc:

• https://github.com/goharbor/harbor/wiki/Harbor-FAQs#notation-signature-accessory

[TnGadaria]

Please refer to this doc:

• https://github.com/goharbor/harbor/wiki/Harbor-FAQs#notation-signature-accessory
  Hi @zyyw
  Thanks for the document link, Now I am using v1.2.0-alpha.1 with --force-referrers-tag=false set, however, the issue persists.

[MinerYang]

Hi @TnGadaria ,
We will transfer this into an issue and help keep triage there, please kindly provide the output per this comments and updates overthere.

Best,
Miner