Allow optimizing images with prioritiezed files info shared via registry

[ktock]

This commit enables to share the "record file" (a JSON file containing a list of prioritized files) among hosts/users via container registry. The record file is packed as an OCI image called "record file image".

"Record file image" is an OCI image only contains one layer that contains the record file as a tar entry named stargz.record.json.

ctr-remote image optimize with --record-out-ref creates a record file image. ctr-remote image convert with --estargz-record-in-ref enables to create an eStargz image with applying the prioritized files information described in the specified record file image.

Generating record file image

The following command generates a record file image named registry2:5000/golang:record-1.15.3-esgz-go-version.

ctr-remote image optimize --oci --record-out-ref=registry2:5000/golang:record-1.15.3-esgz-go-version \
           --entrypoint='[ "/bin/bash", "-c" ]' --args='[ "go version" ]' \
           ghcr.io/stargz-containers/golang:1.15.3-buster-org \
           registry2:5000/golang:1.15.3-esgz-go-version

The follwoing is the manifest of the record file image registry2:5000/golang:record-1.15.3-esgz-go-version.

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:518c804adcec04f5148877af99ec6d216eb50fc47b0b1430217163b838ef7f15",
    "size": 164
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:5b37372a28490bb0737aaadb48241b72ca6e54e023b15ac481a5a3df0da69dcc",
      "size": 344
    }
  ]
}

Layer blob sha256:5b37372a28490bb0737aaadb48241b72ca6e54e023b15ac481a5a3df0da69dcc is a tar.gz blob that contains the record file as a tar entry named stargz.record.json.

# cat /tmp/img/blobs/sha256/5b37372a28490bb0737aaadb48241b72ca6e54e023b15ac481a5a3df0da69dcc | tar -zxO stargz.record.json
{"path":"bin/bash","manifestDigest":"sha256:2322be6aa1e61c36cfa34a0d355b7f088022e34c6f17016bafae6041c67a0a1b","layerIndex":0}
{"path":"bin/bash","manifestDigest":"sha256:2322be6aa1e61c36cfa34a0d355b7f088022e34c6f17016bafae6041c67a0a1b","layerIndex":0}
{"path":"lib/x86_64-linux-gnu/ld-2.28.so","manifestDigest":"sha256:2322be6aa1e61c36cfa34a0d355b7f088022e34c6f17016bafae6041c67a0a1b","layerIndex":0}
...(omit)...

Consuming record file image

ctr-remote image convert command with --estargz-record-in-ref option allowes you to use the recrod file image for optimizing a new eStargz image. The newly-created eStargz will be applied the prioritized files information described in the record file.

ctr-remote image convert --oci --estargz \
           --estargz-record-in-ref registry2:5000/golang:record-1.15.3-esgz-go-version \
           ghcr.io/stargz-containers/golang:1.15.3-buster-org \
           registry2:5000/golang:1.15.3-esgz-go-version-static

Prioritized files information described in the record file image is applied to the newly-creating eStargz in a best-effort manner.
Therefore, files contained in both of the record file and the newly-creating eStargz are considered as the prioritized files.

Copying prioritized files information from an existing eStargz

This commit also adds an ability to copy prioritized files information from an existing eStargz image to a newly-creating eStargz.
The following example creates a new eStargz image registry2:5000/golang:1.15.3-esgz-go-version-static with the same prioritized files configuration as an existing eStargz registry2:5000/golang:1.15.3-esgz-go-version.

ctr-remote image convert --oci --estargz \
           --estargz-record-copy registry2:5000/golang:1.15.3-esgz-go-version \
           ghcr.io/stargz-containers/golang:1.15.3-buster-org \
           registry2:5000/golang:1.15.3-esgz-go-version-static

Now we have 3 approaches for creating an eStargz image statically (i.e. without running the profiler).
Updated ctr-remote.md docs to describe all of these methods.

• Copying prioritized files information from an existing eStargz image (ctr-remote image convert --estargz-record-copy)
• Acquiering prioritized files information shared as a file (ctr-remote image convert --estargz-record-in)
• Acquiering prioritized files information shared via registry (ctr-remote image convert --estargz-record-in-ref)

[AkihiroSuda]

What's the expected usecase?
Isn't landmark data already available in eStargz images?

[ktock]

This patch intended to provide a way to share pre-defined record files to such tools that don't have container runtime functionality (e.g. crane by go-containerregistry).