Labels & Annotations: Expected behaviour

[ChristianKniep]

Hey y'all,

while working on the HPC Container Conformance project I stumbled upon a problem, I'd like to get feedback from the community on: It's about labels and annotations and what folks expect.

Labels/Annotations

Not sure how aware folks are but labels and annotations are not synonyms.

• Labels are part of the nested (configuration) object of an actual manifest, but not part of the manifest.
  UPDATE: They are not static as they are able to be changed when a container is instantiated (docker run --label=foo=bar) 🔥
• Annotations are only part of the OCI image spec and are attached to the manifest or an image index and do not need to match the labels within the configuration object.

If you build a Dockerfile with a bunch of LABELs in it - it won't end up in manifest annotations. I would expect that folks are not aware of that and that they are using labels and annotations as synonyms.

24h Later

Key points:

1. LABEL within a manifest (in the config object) have no rules, you can do LABEL foo=bar in a Dockerfile and that's fine
2. OCI annotations (within a application/vnd.oci.image.manifest.v1+json) SHOULD (not MUST) have a reverse DNS key.
3. Different consumers of labels/annotations expect them in different places.
   a. runtimes/build tools might create and consume LABELs (historically and for compatibility reasons)
   b. registries are looking out for annotations (partly maybe because they expect them to be for public use)
4. I assume that most folks build docker manifest (w/o annotations, just with labels)

We'll discuss this on Thursday, but that shaped the discussion in the direction of:

1. If the manifest has LABELs defined in the nested object, we'll extract those with reverse DNS keys and put them in annotations within the image.manifest
2. The oci.image.index.v1 (aka manifest list in docker terms) does not necessarily carry over the annotations of all the manifests. We need to define what the index annotations convey once we figured out the image.manifest once. :)

[vsoch]

I think labels and annotations are a bit of a wild west, and specifying one in a Dockerfile should (as it does) show up in the config, but I don't think that necessarily means we want it in the manifest. The Dockerfile LABEL is a user-set value that (who knows what it might be used for, I've definitely used them for various things) but a manifest label should be a build-tool or similar set value (better controlled and hopefully less "wild west"). Pinging @sudo-bmitch for his thoughts (I got the Wild West analogy from him!)

[ChristianKniep]

Gottcha - might limit that to reverse URL.

[vsoch]

🐮 👦 == 🤠 !!

Sorry, I shouldn't be allowed to use a computer, I just made one of those jelly belly "recipes" with emojis and I'm quite proud of myself.

[sudo-bmitch]

Docker is just starting to add the ability to set annotations as part of the build. I've also made tooling that copies labels to annotations to help with this. The issue is that different tools consume different values. Runtimes tend to expose labels to images/containers. And registries tend to depend on annotations if they expose them in a UI. So depending on how the content is consumed, you'll need it in one, the other, or both locations.

[sudo-bmitch]

The reverse DNS syntax is good to avoid collisions between different tools. But that relies on each tool having a domain associated with it. I'm not sure how we name things when the project only exists in a GitHub repo, other than a wild west. 🤠

[dev-zero]

The reverse DNS notation is recommended by the image-spec for annotations. The domain does not have to exist.

[reidpr]

I think having two different things with near-synonym names is bad. Either they should be the same — and with the same name, i.e. drop “annotation” — or the names should be more descriptive (e.g. ”container label” and “manifest label”).

[ChristianKniep]

How about LABEL is any key=value pair attached to a container image (by the config object) while ANNOTATIONS are reverse DNS pairs with meaning to higher level entities.
The SBOM might be a gazillion labels within the config obj, while the (use case dependant) important SBOM parts are also present in the annotations.
Like sudo-bmitch said: lower level stuff (runtimes, build tools) are more likely into LABELs and registries fancy annotations

[dev-zero]

It may depend on whether one wants to be able to add the annotations after an image is already built without changing the hash.
I would say yes, hence I'd vote to add it in a way without changing the config (so, not LABEL).
A third option would be similar to how SBOM are introduced and use artifacts to store this metadata. Either with annotations in an artifact with an empty blobs list, or define a separate schema instead which is added as a blob and referenced by the artifact.

[ChristianKniep]

By hash you mean 'the hash of the config object', right? The hash of the manifest itself will change either way. Either because the adjustment of the config object changes the hash of the config and thus the manifest, OR the config stays the same but the annotation of the manifest changes the manifest hash.

[dev-zero]

Yes, you are right. Adding the annotations only in the manifest will still change the hash of a local image (which is what I had in mind). The difference would be that no (empty) layers will be generated because of it.

The way to avoid this would be via referrals (subject) and artifacts, similar to how signing, SBOM and Singularity are implemented. In this case the metadata would link back to the image tag.

[sudo-bmitch]

I'm not sure how well subject/referrers solves this. It's not a current spec so existing tooling won't find it. And to make it a spec, we'd want to work out how to handle collisions and deletions.

[ChristianKniep]

Thanks so far; I'd summarize the discussion like that (I also updated the discussion above):

24h Later

Key points:

1. LABEL within a manifest (in the config object) have no rules, you can do LABEL foo=bar in a Dockerfile and that's fine
2. OCI annotations (within a application/vnd.oci.image.manifest.v1+json) SHOULD (not MUST) have a reverse DNS key.
3. Different consumers of labels/annotations expect them in different places.
   a. runtimes/build tools might create and consume LABELs (historically and for compatibility reasons)
   b. registries are looking out for annotations (partly maybe because they expect them to be for public use)
4. I assume that most folks build docker manifest (w/o annotations, just with labels)

We'll discuss this on Thursday, but that shaped the discussion in the direction of:

1. If the manifest has LABELs defined in the nested object, we'll extract those with reverse DNS keys and put them in annotations within the image.manifest
2. The oci.image.index.v1 (aka manifest list in docker terms) does not necessarily carry over the annotations of all the manifests. We need to define what the index annotations convey once we figured out the image.manifest once. :)

[dev-zero]

1. **LABEL** within a manifest (in the config object) have no rules, you can do `LABEL foo=bar` in a Dockerfile and that's fine

It can also be overridden/added on the command line and it enters multiple times: as layers, as labels in the config unter the config key and as part of the cmd in the container_config (Docker specific). Also, while LABEL in a Dockerfile do not seem to have rules, they are copied over to the config Labels, for which the spec says again: The field contains arbitrary metadata for the container. This property MUST use the annotation rules.

Also, for Dockerfile LABEL, there would have been guidelines: https://docs.docker.com/config/labels-custom-metadata/#key-format-recommendations. But then again, Dockerfile is specified by implementation of Docker rather than a spec.

2. OCI annotations (within a `application/vnd.oci.image.manifest.v1+json`) SHOULD (not MUST) have a reverse DNS key.

3. Different consumers of labels/annotations expect them in different places.
   a.  runtimes/build tools might create and consume LABELs (historically and for compatibility reasons)
   b. registries are looking out for annotations (partly maybe because they expect them to be for public use)

4. I assume that most folks build docker manifest (w/o annotations, just with labels)

That's the unfortunate point: while annotations are likely the right way to go, it needs extra tooling, similar to how signing and SBOMs work. On the other hand this might be a good thing: I would rather prefer to have no annotations than wrong annotations inherited by a FROM.

[ChristianKniep]

Oh, I did not heard about the --label foo=bar for docker run. Interesting - need to meditate over that. So you can not trust that labels within a config of a running container are what the builder of the image intended.
On the other hand, the runtime does not directly know about the annotations of a manifest, as it only deals directly with the config object once the container is instantiated... 🤔
I am still digesting that, but it feels like we might want to carry over information interesting on the node layer into labels and keep what's not out, to not just create a new config obj for no good reason.
Say org.opencontainers.image.created=<timestamp> does not need to be part of the config object b/c it does not interest the local node. While having this in the annotations makes sense.

In our HPC case we might want to have information about the CPU optimisation as ANNOTATIONS and LABELS.

LABEL org.supercontainers.hardware.cpu.target-type=generic
LABEL org.supercontainers.hardware.cpu.platform=x86_64_v4

1. The scheduler might catch a missmatch in the annotation of the manifest or the label of the config early to not schedule the container on a node with x86_64_v3 as it will segfault
2. if the scheduler was not smart enough, the runtime might catch this as a LABEL in the config object

Since we do not control which part is taken into account by whom, we might fall back to LABELs in the config object as a baseline.