How to reference artifacts that move?

[SteveLasker]

In our Notary v2 scenarios, we call out the need for content movement within and across registries.

• The challenge is how do we reference something that we know will move?
• Where is the "hardening" of the reference solidified?

The community and industry have moved to multiple:

• Public Registries:
  • Docker Hub
  • gcr
  • Quay, GitHub
• Cloud Optimized Registries:
  • Azure Marketplace
  • ecr public
• Software Registries:
  • Microsoft Container Registry
  • RedHat
  • Oracle
  • NVIDIA

Much of the same content is already available across these multiple public registries, in addition to the private registries each customer should maintain. See Consuming Public Content for more detail.

The question we propose is how can we make this fact easier to use? How can we make it easier to secure a deployment?

I talk about the broader challenges, and questions in this post: Is It Time to Change How We Reference Container Images?.

Because our scripts, including Helm charts, have fully qualified references, I'd suggest we're in a more difficult and insecure situation. Even if I move a Helm chart to my environment (git or OCI registry), the helm chart references images from outside my organization. How do we make this easy for a person to import the content to their registry and then deploy it from their registry? How can they easily reconfigure things so staging comes from the staging registry, while prod deployments come from the prod registry/repo?

What I'd like to propose is we start down this journey with Notary v2 as one of the newest efforts for how artifacts are referenced. How we can decouple where the artifact may live, vs. what artifact we require.

The proposal follows the way other packages, even files, are referenced. We refer to the name of the artifact, in its most basic form (repo:tag) and enable configuration to identify where to find the artifact/tag. The Notary signature would sign the digest, which is unique regardless of where the artifact is persisted, so we don't lose any identity for the object. Just defer the question for where the artifact happens to live.

The proposal goes like this:
Note: I've updated this to remove the fall-through/ordered search list reference

The registry and repo are configurable options. They are controlled by the client, enable through policy management.

• The client can configure a deterministic mapping for a registry or a specific repo. There would be no fall-through design.
• Consider this akin to a database connectionString that changes between dev, staging and prod.

See mapping discussion below
Marco pointed to this podman implementation, but I'd suggest this is just an example, not the end result.

If we were to decouple the location from the artifact, what would the issues be that we should account for?

• how to avoid a domain and namespace getting confused.
  • Apparently, you can have acmerockets.azurecr.io/azurecr.io/netmonitor:v1 as a valid path. This makes the default registry difficult and we need to avoid SQL-Injection like attacks where a redirect might happen

[sudo-bmitch]

My own preference is to always qualify the reference with a registry, repo, and tag/sha. I look at this from the perspective of a Debian user that always worries a repository I add to apt will decide that they want to provide an updated critical library that would be seen as valid (not positive that attack is actually possible, but it's certainly not clear what repository I'm pulling from when I install new packages).

To handle the use case of making a complete copy, that feels most appropriate for user tooling. This could be a pull through cache, or in my case, I've been working on a regclient project that includes an image copy command that tries to intelligently copy images (between registries, repos, or just retagging an image) using just registry API calls. Similar user tooling could be used to recursively handle more artifacts like notary signatures and helm charts as we get those in OCI.

For using that mirror, I'd like to see this moved into the container runtimes where we specify mirrors to a project like containerd that then tries the local mirror before pulling from the upstream registry for any objects. It might be interesting to create a user or host definition of mirrors to allow non-image artifacts to use that mapping.

In practice, if registry.wabbit.net is mirrored to mirror.acme-rockets.io, even partially, the client pulling the CNAB/Helm/etc from registry.wabbit.net would first try to pull from mirror.acme-rockets.io and only try upstream on an error. Even if all the Helm charts include references to registry.wabbit.net, there would be nothing to change in that object to have the pulls go to the local mirror, which preserves the digest on that Helm chart, and any signature we may have on the chart.

For situations where the upstream registry cannot be reached, the mirror configuration could specify only the local mirror and not fall back to an upstream registry. The other option is to use DNS or some other network level option for telling clients on that network that the upstream repository is found at the local mirror's IP, but that assumes you bypass any TLS protections.

There's also this discussion on whether we want to sign tags that gets into the Notary v2 options when we copy an image, and rename it, in a way that mirror configs aren't easy to maintain: notaryproject/specifications#43 (comment)

[SteveLasker]

My own preference is to always qualify the reference with a registry, repo, and tag/sha.

How do you handle content promotion between environments? We often talk about building an image and not changing it between environments. However, they often have to move between environments, even within a company. Do you change your deployment documents?

debian packages

Isn't this capable of being hardened with config?

I've been working on a regclient project that includes an image copy

Awesome. Today, it's impossible to do any rich capabilities that are registry neutral as the spec is too minimal. Each registry operator has had to implement the basic functionality themselves. Imagine if the distribution-spec evolved to incorporate the basic capabilities of discover (a better version of _catalog & tag listing), See Common Eventing & Search APIs.

The goal of the OCI Artifact Manifest is to enable generic scemas, so you don't need to know about images, helm, singularity, opa, wasm, notary, ... If you implement the generic registry apis, with the new OCI Artifact Manifest, it should "just work"

The challenge with most of these is you have to brute force convert everything. All the references are hard-coded, so you have the first problem.

Mirror Mirror on the wall

We continue to see conversations about mirrors. However, true mirrors don't actually solve the problem of upstream changes that may break you (intentionally or unintentionally). They don't really solve network constraints. Imagine if all the source code files you have on your machine had hard-coded paths to the developer's local file system?
If we can achieve a good copy mechanism, we can give all the benefits of mirrors with all the benefits of users owning their content within their supply chain.

In practice, if registry.wabbit.net is mirrored to mirror.acme-rockets.io, even partially, the client pulling the CNAB/Helm/etc from registry.wabbit.net would first try to pull from mirror.acme-rockets.io and only try upstream on an error.

In most of the environments we've seen, customers are trying to lock down their network ingress and egress rules. In addition to the reference problems, they would never be able to access wabbit-networks.io from within the acme-rockets environment.
If we follow the pattern of signed files, as opposed to aliasing source domains, can we have a more fluid design that is irrelevant to what registry the original artifact was pushed to?

As a comparison, all content hosted in mcr.microsoft.com originates from ACR instances owned by the product team. They publish to their acr instance, and complete a PR that copies the content to mcr.microsoft.com. Should users need to know the original acr the image was solurced from? Or, do they simply trust the registry they current get the content from? Regardless for how it got there?

[mtrmac]

The registry and repo are configurable options. They are controlled by the client, enable through policy management.

What does the client do if they want to run product1 by vendor1, who says that the client must configure mysql to point to vendor1.example/product1/mysql, and product2 by vendor2, who says that the client must configure mysql to point to vendor2.example/product2/mysql?

AFAICS the only sustainable way to distribute larger sets of container images is to avoid such conflicts; i.e. software1 must not just ask for mysql, but it must say $NAMESPACE(vendor1, software1, mysql), and likewise for vendor2/software2. Then we can have a client-side configurable mapping from $NAMESPACE() to a physical location, which can be anywhere at all.

Assuming a consensus on such a $NAMESPACE naming mechanism, it seems very attractive to use the existing Docker image references as a way to carry that, i.e. to keep all of the existing ecosystem unmodified: $NAMESPACE(vendor1, software1, mysql) == registry.vendor1.example/software1/mysql, but treating that value as a logical identifier, not necessarily as a physical location.

[sudo-bmitch]

My own preference is to always qualify the reference with a registry, repo, and tag/sha.

How do you handle content promotion between environments? We often talk about building an image and not changing it between environments. However, they often have to move between environments, even within a company. Do you change your deployment documents?

If an organization wants the image to be referenced with a different name, that feels like a good case to resign the image. I realize that's a pain, but in typical usage for my clients, the CI server would do the signing in dev and other early environments, while production may require a manual image signing, and the organization would want this separation for promotion between the environments. By not allowing the tag signature to move in those cases, you avoid deploying a development signed image to a production server. And in all cases, the digest has one or more signatures that follow it all the way through the chain.

With something like a Helm chart, I often make my registry and repository templated variables to support moving the images. Then it's just a variable file that gets injected per environment. With that indirection, I'm not sure how a recursive copy of object would handle a templated image name, and that may just be forced to the user to manage the copying of their other images, or more complex tooling.

debian packages

Isn't this capable of being hardened with config?

Not really, the -t specifies the release which is different from the repo. Pinning helps, but you need to specify each package exactly, which breaks when for example docker splits off the docker-cli package or starts shipping containerd with a new image name. As a result, most just trust every package on the repo and hope the repo maintainer doesn't do anything malicious.

The goal of the OCI Artifact Manifest is to enable generic scemas, so you don't need to know about images, helm, singularity, opa, wasm, notary, ... If you implement the generic registry apis, with the new OCI Artifact Manifest, it should "just work"

I'm really looking forward to OCI's Distribution and Artifact projects getting to GA.

We continue to see conversations about mirrors. However, true mirrors don't actually solve the problem of upstream changes that may break you (intentionally or unintentionally). They don't really solve network constraints. ...
If we can achieve a good copy mechanism, we can give all the benefits of mirrors with all the benefits of users owning their content within their supply chain.

Part of the workflow from my regclient/regsync command was looking at exactly that since I've run into those issues a few times. It manually updates the local mirror on the users schedule, so it's not a pull-through cache. And it allows automatic backups of the current image before overwriting, to allow an easy revert when upstream sends a breaking change. I haven't come across that logic in other solutions that are more of a pull-through cache, but hopefully this logic will become more common.

Imagine if all the source code files you have on your machine had hard-coded paths to the developer's local file system?

To me it feels more like a world where there's no GitHub, and instead everyone is hoping the repo name on their local Git server is the same as the Git server where another developer did their work. There's a value to having a universal name to avoid ambiguous references. Just as with Go includes we point back to a Git repo reference, and we can also mirror or vendor that reference, but those mirrors and vendored copies include the upstream name.

In most of the environments we've seen, customers are trying to lock down their network ingress and egress rules. In addition to the reference problems, they would never be able to access wabbit-networks.io from within the acme-rockets environment.
If we follow the pattern of signed files, as opposed to aliasing source domains, can we have a more fluid design that is irrelevant to what registry the original artifact was pushed to?

In the environments I've seen like that, the group that imports and approves the wabbit-networks image is interested in verifying the original signature. But once that is done, they want to resign the image with their acme keys, and nodes on their network would be configured to only run images signed by acme-rockets. At that point, we may still have data that indicates the object came from wabbit-networks, but I feel like the artifact we copied has changed ownership and a name change is appropriate.

As a comparison, all content hosted in mcr.microsoft.com originates from ACR instances owned by the product team. They publish to their acr instance, and complete a PR that copies the content to mcr.microsoft.com. Should users need to know the original acr the image was solurced from? Or, do they simply trust the registry they current get the content from? Regardless for how it got there?

I can see a lot of value in the goal you're trying to achieve, because it does solve a lot of problems. In much of this I'm ignoring a lot of the other artifact types where changing a name is a breaking change, and I'm assuming that resigning an image is an easy task, both of which gloss over the reality of many large orgs. But I'm just not sure this value is worth the trade-off.

My concern comes as a user when I'm pulling images from more than one source, say both mcr.microsoft.com for dotnet and gcr.io for istio. If I'm pulling images without the registry name, what happens when Google decides they want to make their own dotnet image? Which image would I as a user be pulling? Similar to Debian, I could pin my dotnet image to only trust it when it's signed by Microsoft, but that gets very brittle when a Helm chart pulls in a dozen images and some image names get changed over time. And that's even worse when, as @mtrmac points out, various Helm charts from each repo each assume their own version of mysql or other common tool. We create namespace collisions where any two users in two completely separate registries create images with the same global name.

Right now, the more maintainable solution for me is when I pull in a Helm chart from MCR that includes references to MCR, and I either mirror MCR or if I copy and rename it. If copying to a new name, then there's some references to change (or hopefully modify with a template variable) to point to my copy. It's not pretty, so I'd still be interested in a better option if it exists. But I fear effectively having a global namespace, without any global management of the names, creates a bigger problem than it solves.

[justincormack]

I agree with @sudo-bmitch that a namespace that is just the final name ("ubuntu") is not large enough, and names need to be fully qualified so they don't collide (ie include a full domain), even if that is not the network address where you find the content.

The Debian example where you can add additional repos under local control seems unlikely to work well for container images where there are a lot of images with similar names, because there are thousands or millions of different "mysql" containers, versus a much smaller number in Debian.

However, I do also think that naming is difficult from the security point of view and relying predominantly on names is problematic...

[SteveLasker]

@mtrmac forwarded me these two docs:

• Container image short names in Podman
• Be careful when pulling images by short name

After reading the docs, and all the above feedback, I realized there was a bunch of history in ordered resolution, that wasn't actually my intent to focus upon. Although I did include it in the initial examples, I'll pull them out.

My hope is to provide a configuration mapping for deterministic resolution.

• We preach containers should be immutable.
  • "Don't embed configuration in the container. For example, extract the data connection to config."
• Within each environment, provide the configuration for that environment so the same built container can move between each environment.

What I'm proposing is much more analogous to this environment approach, and I'll add more clarifying examples, avoiding the search list scenarios.

Let's use the helm chart example for wordpress, and a marketing website that also must be deployed:
There are 3 objects to resolve:

• wordpress-chart:v5
• wordpress:v5
• mysql:8
• marketing-site:v3a3854

oci-reg.config in the Dev environment

{
  "registries": [
    {
      "alias": "base-images",
      "registry": "registry.acmerockets.io"
    },
    {
      "alias": "apps",
      "registry": "dev-registry.acmerockets.io"
    }
  ],
  "repo-mappings": [
    {
      "repo": "wordpress-chart",
      "path": "[base-images]/charts"
    },
    {
      "repo": "wordpress-cnab",
      "path": "[base-images]/cnabs"
    },
    {
      "repo": "wordpress",
      "path": "[base-images]/"
    },
    {
      "repo": "mysql",
      "path": "[base-images]/"
    },
    {
      "repo": "marketing-site",
      "path": "[apps]/dev/"
    }
  ]
}

oci-reg.config in the Prod environment

{
  "registries": [
    {
      "alias": "base-images",
      "registry": "prod-registry.acmerockets.io"
    },
    {
      "alias": "apps",
      "registry": "prod-registry.acmerockets.io"
    }
  ],
  "repo-mappings": [
    {
      "repo": "wordpress-chart",
      "path": "[base-images]/charts"
    },
    {
      "repo": "wordpress-cnab",
      "path": "[base-images]/cnabs"
    },
    {
      "repo": "wordpress",
      "path": "[base-images]/library/"
    },
    {
      "repo": "mysql",
      "path": "[base-images]/library/"
    },
    {
      "repo": "marketing-site",
      "path": "[apps]/marketing/"
    }
  ]
}

In this example:

• the deployment scripts (helm, kubedeploy, ...) don't change per environment
• there's no magic fallthrough for images found in the wrong registry
• two different versions of mysql could be mapped to different versions.
• since both the tag and digest would be included in the new schemas, the client can determine if they want to float to the newest digest for a given tag (which must also pass notary v2 validation), or bind to the digest of the reference, (which also must pass notary v2 validation)

[mtrmac]

What name is contained in the signature? Probably not just wordpress:v5, a moderately-sized software vendor could be shipping different builds of v5 in different products/websites.

[SteveLasker]

What name is contained in the signature? Probably not just wordpress:v5, a moderately-sized software vendor could be shipping different builds of v5 in different products/websites.

Ahh, that's the beauty of separating the name used for deployment, vs. verification of a signature.
The nv2/prototype-1 version of the signature spec has the following:

• references array of strings
  This OPTIONAL property claims the manifest references of its origin. The format of the value MUST matches the reference grammar. With used, the x509 signatures are valid only if the domain names of all references match the Common Name (CN) in the Subject field of the certificate.

An example signature would be:

{
    "typ": "x509",
    "alg": "RS256",
    "x5c": [
        "MIIDszCCApugAwIBAgIUL1anEU/yJy67VJTbHkNX0bBNAnEwDQYJKoZIhvcNAQELBQAwaTEdMBsGA1UEAwwUcmVnaXN0cnkuZXhhbXBsZS5jb20xFDASBgNVBAoMC2V4YW1wbGUgaW5jMQswCQYDVQQGEwJVUzETMBEGA1UECAwKV2FzaGluZ3RvbjEQMA4GA1UEBwwHU2VhdHRsZTAeFw0yMDA3MjcxNDQzNDZaFw0yMTA3MjcxNDQzNDZaMGkxHTAbBgNVBAMMFHJlZ2lzdHJ5LmV4YW1wbGUuY29tMRQwEgYDVQQKDAtleGFtcGxlIGluYzELMAkGA1UEBhMCVVMxEzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcMB1NlYXR0bGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDkKwAcV44psjN8nno1eZ3zv1ZKUhJAoxwBOIGfIxIe+iHtpXLvFFVwk5Jbxu+Pkig2N4B3Ilrj/Vryi0hxp4mag02M733bXLRENSOFONRkslpO8zHUN5pYdnhTSwYTLap1+1bgcFSuUXLWieqZB6qc7kiv3bj3SPaf42+s48V49t/OpXxLtgiWL9XkuDTZctpJJA4vHHk6Ou0bcg7iGm+L1xwIfb8Ml4oWvT0SF35fgW08bbLXZ2v1XCLRsrWUgbq4U+KxtEpG3XIYcYhKx1rIrUhfEJkuHzgPglM11gG5W+Cyfg+wfOJig5q6axIKWzIf6C8m8lmy6bM+N5EsD9SvAgMBAAGjUzBRMB0GA1UdDgQWBBTf1hM6/ibGF+u/SVAK88FUMjzRoTAfBgNVHSMEGDAWgBTf1hM6/ibGF+u/SVAK88FUMjzRoTAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBgvVau5+2wAuCsmOyyG28h1zyC4IPmMmpRZTDOp/pLdwXeHjJr8kEC3l92qJEvc+WAboJ1RoucHycUe7RWh2C6ZF/WPCBLyWGwnlyqGyRM9/j86UJ1OgiuZl7kl9zxwWoaxPBCmHa0RHowdQB7AVlpqg1c7FhKjhUCBmGT4Ve8tV0hdZtrZoQV+6xHPbUd37KV1B1Bmfo3o4ekoJKhUu99Eo03OpE3JLtM13A1HxABEuQGHTI0tycDBBdRn3b03HoIhU0VnqjvpV1KPvsrgYi/0VStLNezZPgGe0fG3Xgy8yekdB9NMUn+zZLATI4+z8j4QH5Wj5ZPaUkyoAD2oUJO"
    ]
}.{
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "digest": "sha256:c4516b8a311e85f1f2a60573abf4c6b740ca3ade4127e29b05616848de487d34",
    "size": 528,
    "references": [
        "registry.wabbit-networks.io/net-monitor:latest",
        "registry.wabbit-networks.io/net-monitor:v1.0"
    ],
    "exp": 1628587119,
    "iat": 1597051119,
    "nbf": 1597051119
}

[mtrmac]

So the actual signature contains world-wide-unique names? If the ultimate consumer specifies only the short name in the deployment specification, and the registry/repo is fully variable, how does the signature verifier know which world-wide-unique name to accept (and, to begin which, which public key to trust for that short name)?

It seems functionally equivalent to me, and clearly unambiguous, for both the signature and the deployment specification to use world-wide-unique names; that allows building exactly the same oci-reg.config mechanism, just the lookup key would be a world-wide-unique name with some domain/repo; in addition, the deployment would be usable without an oci-reg.config on the cloud, for users that don’t care about private mirrors or disconnected sites.

[SteveLasker]

The premise is to decouple the signature and its authority/ownership from where the artifact currently exists.
Consider the signed content on your laptop. The signature has an authority/ownership, which has nothing to do with file path on your computer.
As covered in the requirements and scenarios, you can validate the artifact as it was signed by it's originator. In the net-monitor:v1 image scenario, it's signed by wabbit-networks. The full name of the registry is an option, not a requirement. For instance, software pulled from registry.wabbit-networks.io, mcr.microsoft.com, ngc.nvidia.com/catalog/containers would first be validated when it's pulled from those software registries. The consumer could add an additional signature, such as registry.acme-rockets.io.
This provides two key use cases.

1. validating a signature, associated with its public key. Acme-rockets can acquire the public key for wabbit-networks and validate the net-monitor:v1 image, locally, without having a connection to registry.wabbit-networks.io.
2. validating an artifact, before pulling from a registry. In the ACME Rockets environment, an nv2 option can be set that requires the artifact to be pulled from the registry specified in the signature. This would assure that ACME Rockets can secure their production environment, limiting content from any other source. Other consumers might choose to validate with the wabbit-networks key, or even use the docker signature & key.

The beauty is in the simplicity and flexibility to carry the information, enabling clients to use the information that meets their needs.

[sudo-bmitch]

Just saw the following report of an attack using ambiguous short names that reminds me of the RH concerns: https://www.bleepingcomputer.com/news/security/researcher-hacks-over-35-tech-firms-in-novel-supply-chain-attack/

Any solution to allow artifacts to move should be explicit from the client, specifying that calls to one registry/repository/tag get mapped to another, rather than a chain of fall throughs in different namespaces that would allow a name squatting attack.

[SteveLasker]

Agreed. The proposal is based on deterministic mappings. There’s no fall through search list that would allow squatting. #31 (comment)