Requesting UX improvement in signing and verifying with user metadata via Notation CLI

[iamsamirzon]

What is the areas you experience the issue in?

Notation CLI

What is not working as expected?

When signing and verifying images with user metadata the CLI does not error out if the "-m or --user-metadata" flag is not passed with every Key value pair. This could lead to instances when the generated signature does not include all the required metadata during signing, and same way during verifying not all the metadata is checked.

What did you expect to happen?

If a user-metadata field if specified during signing and verifying without the "-m or --user-metadata" field the CLI command should fail with a warning Vs a silent pass.

How can we reproduce it?

C:\Users\xyz>notation sign %IMAGE% --key wabbit-networks.io -m buildserver=bidcode stage=scan
Successfully signed localhost:5001/net-monitor@sha256:7e6db07ac3c786adf6a9b855f172f1451e5b0cd48614cc13b7194c9837a994c1

<< Ideally the above command should have failed because it made us think that both the buildserver and stage user metadata field were accepted >>

C:\Users\xyz>notation ls %IMAGE%
localhost:5001/net-monitor@sha256:7e6db07ac3c786adf6a9b855f172f1451e5b0cd48614cc13b7194c9837a994c1
└── application/vnd.cncf.notary.signature
└── sha256:def5b7411f6b0317a3557135468a9a0cfa2a895521d03af468b040472d7ed9bf

C:\Users\xyz>notation inspect %IMAGE%
Inspecting all signatures for signed artifact
localhost:5001/net-monitor@sha256:7e6db07ac3c786adf6a9b855f172f1451e5b0cd48614cc13b7194c9837a994c1
└── application/vnd.cncf.notary.signature
└── sha256:def5b7411f6b0317a3557135468a9a0cfa2a895521d03af468b040472d7ed9bf
├── media type: application/jose+json
├── signature algorithm: RSASSA-PSS-SHA-256
├── signed attributes
│ ├── expiry: Mon Jan 1 00:00:00 0001
│ ├── signingScheme: notary.x509
│ └── signingTime: Wed Apr 26 05:44:31 2023
├── user defined attributes
│ └── buildserver: bidcode
├── unsigned attributes
│ └── signingAgent: Notation/1.0.0
├── certificates
│ └── SHA1 fingerprint: 5ca635a96cbdb503188717d896ae22dd2a18dd90
│ ├── issued to: CN=wabbit-networks.io,O=Notary,L=Seattle,ST=WA,C=US
│ ├── issued by: CN=wabbit-networks.io,O=Notary,L=Seattle,ST=WA,C=US
│ └── expiry: Wed Apr 26 13:44:17 2023
└── signed artifact
├── media type: application/vnd.docker.distribution.manifest.v2+json
├── digest: sha256:7e6db07ac3c786adf6a9b855f172f1451e5b0cd48614cc13b7194c9837a994c1
└── size: 942

---

Here is the same Sign command given with the "-m" field for both the user metadata and in that case both the usermetadata fields are correctly encoded inside the signature

C:\Users\xyz>notation sign %IMAGE% --key wabbit-networks.io -m buildserver=bidcode -m stage=scan
Successfully signed localhost:5001/net-monitor@sha256:7e6db07ac3c786adf6a9b855f172f1451e5b0cd48614cc13b7194c9837a994c1

C:\Users\xyz>notation ls %IMAGE%
localhost:5001/net-monitor@sha256:7e6db07ac3c786adf6a9b855f172f1451e5b0cd48614cc13b7194c9837a994c1
└── application/vnd.cncf.notary.signature
├── sha256:def5b7411f6b0317a3557135468a9a0cfa2a895521d03af468b040472d7ed9bf
└── sha256:05ab94f23cde4adb0a49799067808f87c31746867ddd7890574e7304edf3f565

C:\Users\xyz>notation inspect %IMAGE%
Inspecting all signatures for signed artifact
localhost:5001/net-monitor@sha256:7e6db07ac3c786adf6a9b855f172f1451e5b0cd48614cc13b7194c9837a994c1
└── application/vnd.cncf.notary.signature
├── sha256:def5b7411f6b0317a3557135468a9a0cfa2a895521d03af468b040472d7ed9bf
│ ├── media type: application/jose+json
│ ├── signature algorithm: RSASSA-PSS-SHA-256
│ ├── signed attributes
│ │ ├── signingScheme: notary.x509
│ │ ├── signingTime: Wed Apr 26 05:44:31 2023
│ │ └── expiry: Mon Jan 1 00:00:00 0001
│ ├── user defined attributes
│ │ └── buildserver: bidcode
│ ├── unsigned attributes
│ │ └── signingAgent: Notation/1.0.0
│ ├── certificates
│ │ └── SHA1 fingerprint: 5ca635a96cbdb503188717d896ae22dd2a18dd90
│ │ ├── issued to: CN=wabbit-networks.io,O=Notary,L=Seattle,ST=WA,C=US
│ │ ├── issued by: CN=wabbit-networks.io,O=Notary,L=Seattle,ST=WA,C=US
│ │ └── expiry: Wed Apr 26 13:44:17 2023
│ └── signed artifact
│ ├── media type: application/vnd.docker.distribution.manifest.v2+json
│ ├── digest: sha256:7e6db07ac3c786adf6a9b855f172f1451e5b0cd48614cc13b7194c9837a994c1
│ └── size: 942
└── sha256:05ab94f23cde4adb0a49799067808f87c31746867ddd7890574e7304edf3f565
├── media type: application/jose+json
├── signature algorithm: RSASSA-PSS-SHA-256
├── signed attributes
│ ├── signingScheme: notary.x509
│ ├── signingTime: Wed Apr 26 05:49:13 2023
│ └── expiry: Mon Jan 1 00:00:00 0001
├── user defined attributes
│ ├── stage: scan
│ └── buildserver: bidcode
├── unsigned attributes
│ └── signingAgent: Notation/1.0.0
├── certificates
│ └── SHA1 fingerprint: 5ca635a96cbdb503188717d896ae22dd2a18dd90
│ ├── issued to: CN=wabbit-networks.io,O=Notary,L=Seattle,ST=WA,C=US
│ ├── issued by: CN=wabbit-networks.io,O=Notary,L=Seattle,ST=WA,C=US
│ └── expiry: Wed Apr 26 13:44:17 2023
└── signed artifact
├── media type: application/vnd.docker.distribution.manifest.v2+json
├── digest: sha256:7e6db07ac3c786adf6a9b855f172f1451e5b0cd48614cc13b7194c9837a994c1
└── size: 942

---

However again while verifying, if the -"m" field is not passed correctly the signature verification erroneously validates the signature making us think that both the user-metadata fields were present

C:\Users\xyz>notation verify %IMAGE% -m buildserver=bidcode stage=untested
Successfully verified signature for localhost:5001/net-monitor@sha256:7e6db07ac3c786adf6a9b855f172f1451e5b0cd48614cc13b7194c9837a994c1

The artifact was signed with the following user metadata.

KEY VALUE
buildserver bidcode

Describe your environment

Running Notation RC-4 with a local registry and using a self generated test certificate

What is the version of your Notation CLI or Notation Library?

C:\Users\xyz>notation version
Notation - a tool to sign and verify artifacts.

Version: 1.0.0-rc.4
Go version: go1.20.3
Git commit: 2e56dd42e385ee1568c5e13e6ef38edb2a549500

[toddysm]

This behavior is confusing - we should not silently ignore command line arguments. If we don't understand an argument, we should show an error.

I am wondering if this has also some security implications. For example, if somebody signs using the following command:

notation sign %IMAGE% --key wabbit-networks.io -m buildserver=bidcode -m stage=scan

Will the following verification succeed?

notation verify %IMAGE% -m buildserver=bidcode stage=tested

If yes, then this may be exploitable because I may want to assure that both values match the requested. It seems from your tests that this may succeed, and the second annotation won't be tested at all. The impact can be that the image is deployed although it is not tested yet.

[FeynmanZhou]

We will need to document the CLI design convention of this case on FAQ doc and clarify this case.