No progress information while copying images to ECR #6681

hannes-ucsc · 2024-11-02T06:54:42Z

When @achave11-ucsc and I diagnosed a problem with deploying the shared component from his computer, we noticed that Skopeo was eerily quiet during the copy. It appeared to be hung or not make any significant progress. Either way, we need to ensure that it obvious to the user what progress Skopeo is making, or if it is making any at all.

The text was updated successfully, but these errors were encountered:

hannes-ucsc · 2024-11-02T07:11:59Z

Spike to reproduce on a different computer.

achave11-ucsc · 2024-11-04T20:04:11Z

@hannes-ucsc: "@dsotirho-ucsc confirms this is reproducible (Skopeo lacks progress info). @dsotirho-ucsc to investigate if Skopeo can be coaxed into providing progress information."

dsotirho-ucsc · 2024-11-04T21:00:10Z

@dsotirho-ucsc to investigate if Skopeo can be coaxed into providing progress information.

This is a requested feature since May 2019

Please provide a basic progress / transfer speed output ("CI compatible") on copy
https://www.github.com/containers/skopeo/issues/658

hannes-ucsc · 2024-11-05T17:07:16Z

Please consider --debug as a workaround.

dsotirho-ucsc · 2024-11-05T21:52:35Z

Without `--debug`

26 lines of output (for null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w)

Terraform will perform the following actions:

  # null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w is tainted, so must be replaced
-/+ resource "null_resource" "image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w" {
      ~ id       = "6334824724549167320" -> (known after apply)
        # (1 unchanged attribute hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Warnings:

- Argument is deprecated
  on shared.tf.json line 95 (and 3 more)

To see the full warning notes, run Terraform without -compact-warnings.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w: Destroying... [id=6334824724549167320]
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w: Destruction complete after 0s
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w: Creating...
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w: Provisioning with 'local-exec'...
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Executing: ["/bin/sh" "-c" "python /Users/daniel/repo/azul1/scripts/manage_images.py --copy docker.io/bitnami/kibana:7.10.2"]
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:43:46,983   DEBUG MainThread __main__: Creating or opening semaphore '/manage_images.tempdev'
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:43:46,983    INFO MainThread __main__: Acquiring semaphore '/manage_images.tempdev'
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:43:46,983    INFO MainThread __main__: Acquired semaphore '/manage_images.tempdev'
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:43:46,983    INFO MainThread __main__: Copying image TagImageRef(registry='docker.io', username='bitnami', repository=('kibana',), tag='7.10.2') for platform Platform(os='linux', arch='amd64', variant=None)
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Getting image source signatures
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:b8fc884dd91a599ae724fda23284dd85f56909be1f8d5c165490014d05c4362d
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:0e7355dd88d515484520f6eafe991cac9bf485b27294104b8b829314390b88c7
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:6575deaf42b75e48d277f543ddf2ab60dd00068556fdf3420eb4f82d80ba9ac3
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:38b52af9d09f760abfd31dab3a82f1241ae47f3233889ecbc51717f863e91f47
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:be4dd8fa80cd87920dbed821d9a1fd075569e9fa2f08d4d83b53d5fc50ee5e48
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:80885efcfd989383e3a387b9ee19da4a55b58b1262a984b743419ecd27bb85ad
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:a67866c074909a07ee3ed39d98d690c32e820a03af9909177bdeedf6b4280075
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:eb8791bb028a2dadc5492d15620a1396d0f755fce5007664136e559efd1b6ff0
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:cee68e0ab7a3c36e0e46a2f887feefaad3aff7618406a463ef97999072d53e12
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying config sha256:5afa1090aae4d2df1754c9d071ccdd23c13cafe65ab1963d3223c8be8ee83629
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Writing manifest to image destination
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:43:49,696    INFO MainThread botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:43:49,735    INFO MainThread azul.deployment: Allocated new Boto3 client for 'ecr' with ID 4369692528
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:43:50,241    INFO MainThread __main__: Released semaphore '/manage_images.tempdev'
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w: Creation complete after 3s [id=5577006791947779410]

Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

With `--debug`

110 lines of output (for null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w)

Terraform will perform the following actions:

  # null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w is tainted, so must be replaced
-/+ resource "null_resource" "image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w" {
      ~ id       = "8674665223082153551" -> (known after apply)
        # (1 unchanged attribute hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Warnings:

- Argument is deprecated
  on shared.tf.json line 95 (and 3 more)

To see the full warning notes, run Terraform without -compact-warnings.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w: Destroying... [id=8674665223082153551]
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w: Destruction complete after 0s
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w: Creating...
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w: Provisioning with 'local-exec'...
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Executing: ["/bin/sh" "-c" "python /Users/daniel/repo/azul1/scripts/manage_images.py --copy docker.io/bitnami/kibana:7.10.2"]
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:47:23,786   DEBUG MainThread __main__: Creating or opening semaphore '/manage_images.tempdev'
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:47:23,786    INFO MainThread __main__: Acquiring semaphore '/manage_images.tempdev'
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:47:23,786    INFO MainThread __main__: Acquired semaphore '/manage_images.tempdev'
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:47:23,786    INFO MainThread __main__: Copying image TagImageRef(registry='docker.io', username='bitnami', repository=('kibana',), tag='7.10.2') for platform Platform(os='linux', arch='amd64', variant=None)
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:23-08:00" level=debug msg="Using registries.d directory /opt/homebrew/etc/containers/registries.d"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:23-08:00" level=debug msg="Loading registries configuration \"/opt/homebrew/etc/containers/registries.conf\""
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:23-08:00" level=debug msg="Found credentials for 654654270592.dkr.ecr.us-east-1.amazonaws.com/docker.io/bitnami/kibana in credential helper containers-auth.json in file /var/folders/d1/1lnk2yfj2tj93pyr9_0mx7w00000gn/T/tmpvlvr_37g"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:23-08:00" level=debug msg=" No signature storage configuration found for 654654270592.dkr.ecr.us-east-1.amazonaws.com/docker.io/bitnami/kibana:7.10.2, using built-in default file:///Users/daniel/.local/share/containers/sigstore"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:23-08:00" level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/654654270592.dkr.ecr.us-east-1.amazonaws.com"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:23-08:00" level=debug msg="Using registries.d directory /opt/homebrew/etc/containers/registries.d"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:23-08:00" level=debug msg="Trying to access \"docker.io/bitnami/kibana@sha256:6e96a761f087cb0e3917998e6ed439ebc5a5bdb910f5477c1c39eb7bcbe54210\""
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:23-08:00" level=debug msg="Found credentials for docker.io/bitnami/kibana in credential helper containers-auth.json in file /var/folders/d1/1lnk2yfj2tj93pyr9_0mx7w00000gn/T/tmpvlvr_37g"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:23-08:00" level=debug msg=" No signature storage configuration found for docker.io/bitnami/kibana@sha256:6e96a761f087cb0e3917998e6ed439ebc5a5bdb910f5477c1c39eb7bcbe54210, using built-in default file:///Users/daniel/.local/share/containers/sigstore"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:23-08:00" level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:23-08:00" level=debug msg="GET https://registry-1.docker.io/v2/"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Ping https://registry-1.docker.io/v2/ status 401"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="GET https://auth.docker.io/token?account=dsotirhoucsc&scope=repository%3Abitnami%2Fkibana%3Apull&service=registry.docker.io"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="GET https://registry-1.docker.io/v2/bitnami/kibana/manifests/sha256:6e96a761f087cb0e3917998e6ed439ebc5a5bdb910f5477c1c39eb7bcbe54210"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Content-Type from manifest GET is \"application/vnd.docker.distribution.manifest.v2+json\""
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Using SQLite blob info cache at /Users/daniel/.local/share/containers/cache/blob-info-cache-v1.sqlite"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="IsRunningImageAllowed for image docker:docker.io/bitnami/kibana@sha256:6e96a761f087cb0e3917998e6ed439ebc5a5bdb910f5477c1c39eb7bcbe54210"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg=" Using default policy section"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg=" Requirement 0: allowed"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Overall: allowed"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Getting image source signatures
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Reading /Users/daniel/.local/share/containers/sigstore/bitnami/kibana@sha256=6e96a761f087cb0e3917998e6ed439ebc5a5bdb910f5477c1c39eb7bcbe54210/signature-1"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Not looking for sigstore attachments: disabled by configuration"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="We can't modify the manifest, hoping for the best..."
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:b8fc884dd91a599ae724fda23284dd85f56909be1f8d5c165490014d05c4362d
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:0e7355dd88d515484520f6eafe991cac9bf485b27294104b8b829314390b88c7
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:6575deaf42b75e48d277f543ddf2ab60dd00068556fdf3420eb4f82d80ba9ac3
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking if we can reuse blob sha256:6575deaf42b75e48d277f543ddf2ab60dd00068556fdf3420eb4f82d80ba9ac3: general substitution = false, compression for MIME type \"application/vnd.docker.image.rootfs.diff.tar.gzip\" = true"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking if we can reuse blob sha256:0e7355dd88d515484520f6eafe991cac9bf485b27294104b8b829314390b88c7: general substitution = false, compression for MIME type \"application/vnd.docker.image.rootfs.diff.tar.gzip\" = true"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:38b52af9d09f760abfd31dab3a82f1241ae47f3233889ecbc51717f863e91f47
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking if we can reuse blob sha256:b8fc884dd91a599ae724fda23284dd85f56909be1f8d5c165490014d05c4362d: general substitution = false, compression for MIME type \"application/vnd.docker.image.rootfs.diff.tar.gzip\" = true"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:b8fc884dd91a599ae724fda23284dd85f56909be1f8d5c165490014d05c4362d"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="GET https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:6575deaf42b75e48d277f543ddf2ab60dd00068556fdf3420eb4f82d80ba9ac3"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:be4dd8fa80cd87920dbed821d9a1fd075569e9fa2f08d4d83b53d5fc50ee5e48
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking if we can reuse blob sha256:be4dd8fa80cd87920dbed821d9a1fd075569e9fa2f08d4d83b53d5fc50ee5e48: general substitution = false, compression for MIME type \"application/vnd.docker.image.rootfs.diff.tar.gzip\" = true"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:be4dd8fa80cd87920dbed821d9a1fd075569e9fa2f08d4d83b53d5fc50ee5e48"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking if we can reuse blob sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce: general substitution = false, compression for MIME type \"application/vnd.docker.image.rootfs.diff.tar.gzip\" = true"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking if we can reuse blob sha256:38b52af9d09f760abfd31dab3a82f1241ae47f3233889ecbc51717f863e91f47: general substitution = false, compression for MIME type \"application/vnd.docker.image.rootfs.diff.tar.gzip\" = true"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:38b52af9d09f760abfd31dab3a82f1241ae47f3233889ecbc51717f863e91f47"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:24-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:0e7355dd88d515484520f6eafe991cac9bf485b27294104b8b829314390b88c7"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Ping https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/ status 401"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:b8fc884dd91a599ae724fda23284dd85f56909be1f8d5c165490014d05c4362d"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:6575deaf42b75e48d277f543ddf2ab60dd00068556fdf3420eb4f82d80ba9ac3"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:be4dd8fa80cd87920dbed821d9a1fd075569e9fa2f08d4d83b53d5fc50ee5e48"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:38b52af9d09f760abfd31dab3a82f1241ae47f3233889ecbc51717f863e91f47"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:0e7355dd88d515484520f6eafe991cac9bf485b27294104b8b829314390b88c7"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="... already exists"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Skipping blob sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce (already present):"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:80885efcfd989383e3a387b9ee19da4a55b58b1262a984b743419ecd27bb85ad
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Checking if we can reuse blob sha256:80885efcfd989383e3a387b9ee19da4a55b58b1262a984b743419ecd27bb85ad: general substitution = false, compression for MIME type \"application/vnd.docker.image.rootfs.diff.tar.gzip\" = true"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:80885efcfd989383e3a387b9ee19da4a55b58b1262a984b743419ecd27bb85ad"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:80885efcfd989383e3a387b9ee19da4a55b58b1262a984b743419ecd27bb85ad"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="... already exists"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Skipping blob sha256:38b52af9d09f760abfd31dab3a82f1241ae47f3233889ecbc51717f863e91f47 (already present):"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:a67866c074909a07ee3ed39d98d690c32e820a03af9909177bdeedf6b4280075
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Checking if we can reuse blob sha256:a67866c074909a07ee3ed39d98d690c32e820a03af9909177bdeedf6b4280075: general substitution = false, compression for MIME type \"application/vnd.docker.image.rootfs.diff.tar.gzip\" = true"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:a67866c074909a07ee3ed39d98d690c32e820a03af9909177bdeedf6b4280075"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:a67866c074909a07ee3ed39d98d690c32e820a03af9909177bdeedf6b4280075"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="... already exists"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="... already exists"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Skipping blob sha256:b8fc884dd91a599ae724fda23284dd85f56909be1f8d5c165490014d05c4362d (already present):"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:eb8791bb028a2dadc5492d15620a1396d0f755fce5007664136e559efd1b6ff0
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Checking if we can reuse blob sha256:eb8791bb028a2dadc5492d15620a1396d0f755fce5007664136e559efd1b6ff0: general substitution = false, compression for MIME type \"application/vnd.docker.image.rootfs.diff.tar.gzip\" = true"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:eb8791bb028a2dadc5492d15620a1396d0f755fce5007664136e559efd1b6ff0"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:eb8791bb028a2dadc5492d15620a1396d0f755fce5007664136e559efd1b6ff0"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="... already exists"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Skipping blob sha256:0e7355dd88d515484520f6eafe991cac9bf485b27294104b8b829314390b88c7 (already present):"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:cee68e0ab7a3c36e0e46a2f887feefaad3aff7618406a463ef97999072d53e12
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Checking if we can reuse blob sha256:cee68e0ab7a3c36e0e46a2f887feefaad3aff7618406a463ef97999072d53e12: general substitution = false, compression for MIME type \"application/vnd.docker.image.rootfs.diff.tar.gzip\" = true"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:cee68e0ab7a3c36e0e46a2f887feefaad3aff7618406a463ef97999072d53e12"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:cee68e0ab7a3c36e0e46a2f887feefaad3aff7618406a463ef97999072d53e12"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="... already exists"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Skipping blob sha256:6575deaf42b75e48d277f543ddf2ab60dd00068556fdf3420eb4f82d80ba9ac3 (already present):"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Skipping blob sha256:be4dd8fa80cd87920dbed821d9a1fd075569e9fa2f08d4d83b53d5fc50ee5e48 (already present):"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="... already exists"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="... already exists"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Skipping blob sha256:80885efcfd989383e3a387b9ee19da4a55b58b1262a984b743419ecd27bb85ad (already present):"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="... already exists"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Skipping blob sha256:a67866c074909a07ee3ed39d98d690c32e820a03af9909177bdeedf6b4280075 (already present):"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Skipping blob sha256:eb8791bb028a2dadc5492d15620a1396d0f755fce5007664136e559efd1b6ff0 (already present):"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="... already exists"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Skipping blob sha256:cee68e0ab7a3c36e0e46a2f887feefaad3aff7618406a463ef97999072d53e12 (already present):"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying config sha256:5afa1090aae4d2df1754c9d071ccdd23c13cafe65ab1963d3223c8be8ee83629
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="Downloading /v2/bitnami/kibana/blobs/sha256:5afa1090aae4d2df1754c9d071ccdd23c13cafe65ab1963d3223c8be8ee83629"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:25-08:00" level=debug msg="GET https://registry-1.docker.io/v2/bitnami/kibana/blobs/sha256:5afa1090aae4d2df1754c9d071ccdd23c13cafe65ab1963d3223c8be8ee83629"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:26-08:00" level=debug msg="No compression detected"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:26-08:00" level=debug msg="Compression change for blob sha256:5afa1090aae4d2df1754c9d071ccdd23c13cafe65ab1963d3223c8be8ee83629 (\"application/vnd.docker.container.image.v1+json\") not supported"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:26-08:00" level=debug msg="Using original blob without modification"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:26-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:5afa1090aae4d2df1754c9d071ccdd23c13cafe65ab1963d3223c8be8ee83629"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:26-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:5afa1090aae4d2df1754c9d071ccdd23c13cafe65ab1963d3223c8be8ee83629"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:26-08:00" level=debug msg="... already exists"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Writing manifest to image destination
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-05T13:47:26-08:00" level=debug msg="PUT https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/manifests/7.10.2"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:47:26,476    INFO MainThread botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:47:26,514    INFO MainThread azul.deployment: Allocated new Boto3 client for 'ecr' with ID 4324740896
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): 2024-11-05 13:47:26,974    INFO MainThread __main__: Released semaphore '/manage_images.tempdev'
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w: Creation complete after 3s [id=5577006791947779410]

Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

achave11-ucsc · 2024-11-06T20:04:26Z

@hannes-ucsc: "It appears that skopeo detected the destination blobs to already exists, which raises doubts as to whether the deletion from ECR was effective. If it wasn't, the above terminal transcript might not correctly represent what debug output would look like when blobs are actually copied."

dsotirho-ucsc · 2024-11-07T17:29:25Z

See comment for revised statement

~~The --debug is not a suitable workaround for the lack of progress bars or status during image uploads. The status lines such as~~

null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:6575deaf42b75e48d277f543ddf2ab60dd00068556fdf3420eb4f82d80ba9ac3

~~are printed the same and in the same frequency with or without --debug.~~

~~The additional debug lines that are printed mostly relate to internal processes between image uploads (e.g. connection setup, checking if the destination already exists).~~

There is a potentially useful debug line mentioning the completion of a blob's upload, however a single image can consist of many blobs (10 for the kibana image), and the increased number of log lines printed with debug enabled, 34 vs 242 for one (kibana) image, make finding all the blobs relating to one image not easy to obtain.

~~Example: logs relating to one blob's upload (note last line "Upload of layer ... complete")~~

null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): Copying blob sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-06T16:25:08-08:00" level=debug msg="Checking if we can reuse blob sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce: general substitution = false, compression for MIME type \"application/vnd.docker.image.rootfs.diff.tar.gzip\" = true"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-06T16:25:08-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-06T16:25:09-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-06T16:25:09-08:00" level=debug msg="Ignoring BlobInfoCache record of digest \"sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce\" with unknown compression"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-06T16:25:09-08:00" level=debug msg="Downloading /v2/bitnami/kibana/blobs/sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-06T16:25:09-08:00" level=debug msg="GET https://registry-1.docker.io/v2/bitnami/kibana/blobs/sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-06T16:25:09-08:00" level=debug msg="... not present"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-06T16:25:10-08:00" level=debug msg="Checking /v2/docker.io/bitnami/kibana/blobs/sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-06T16:25:10-08:00" level=debug msg="HEAD https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-06T16:25:17-08:00" level=debug msg="PUT https://654654270592.dkr.ecr.us-east-1.amazonaws.com/v2/docker.io/bitnami/kibana/blobs/uploads/57f73247-1f70-3cf5-abdc-72b09c4c3e93?digest=sha256%3A459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce"
null_resource.image_Ldl4m3_PgOwJ0wLQ7kTs03GiB4w (local-exec): time="2024-11-06T16:25:17-08:00" level=debug msg="Upload of layer sha256:459bfeb2c0756e6f3ab51be2c0dc307a5e9931c74cf094e9c04b43c831a87dce complete"

achave11-ucsc · 2024-12-17T19:41:38Z

Assignee to revise statement in light of clarifications regarding the objective of this ticket.

dsotirho-ucsc · 2024-12-19T01:23:52Z

Adding the --debug option to the skopeo copy command produced many more lines of output while the transfers were in progress, and could be used to reassure the user that the transfers haven't stalled.

achave11-ucsc · 2024-12-19T19:36:48Z

Assignee to implement proposed solution.

hannes-ucsc · 2024-12-27T01:06:47Z

For demo, attempt to reproduce (delete big image, taint TF resource, redeploy shared).

…6775)

hannes-ucsc added the orange [process] Done by the Azul team label Nov 2, 2024

achave11-ucsc added needs design [process] Solution to issue has yet to be devised spike:1 [process] Spike estimate of one point labels Nov 4, 2024

achave11-ucsc assigned achave11-ucsc and dsotirho-ucsc and unassigned achave11-ucsc Nov 4, 2024

achave11-ucsc assigned hannes-ucsc and unassigned dsotirho-ucsc Nov 7, 2024

achave11-ucsc assigned dsotirho-ucsc and unassigned hannes-ucsc Dec 17, 2024

dsotirho-ucsc added a commit that referenced this issue Dec 19, 2024

Fix: No progress information while copying images to ECR (#6681)

9dd8117

dsotirho-ucsc mentioned this issue Dec 19, 2024

Fix: No progress information while copying images to ECR (#6681) #6775

Merged

hannes-ucsc added demo [process] To be demonstrated at the end of the sprint and removed needs design [process] Solution to issue has yet to be devised labels Jan 6, 2025

achave11-ucsc pushed a commit that referenced this issue Jan 6, 2025

Fix: No progress information while copying images to ECR (#6681)

78fac97

achave11-ucsc added a commit that referenced this issue Jan 6, 2025

Fix: No progress information while copying images to ECR (#6681, PR #…

c81dde7

…6775)

dsotirho-ucsc added the demoed [process] Successfully demonstrated to team label Jan 7, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

No progress information while copying images to ECR #6681

No progress information while copying images to ECR #6681

hannes-ucsc commented Nov 2, 2024 •

edited

Loading

hannes-ucsc commented Nov 2, 2024

achave11-ucsc commented Nov 4, 2024

dsotirho-ucsc commented Nov 4, 2024

hannes-ucsc commented Nov 5, 2024

dsotirho-ucsc commented Nov 5, 2024 •

edited

Loading

achave11-ucsc commented Nov 6, 2024

dsotirho-ucsc commented Nov 7, 2024 •

edited

Loading

achave11-ucsc commented Dec 17, 2024

dsotirho-ucsc commented Dec 19, 2024

achave11-ucsc commented Dec 19, 2024

hannes-ucsc commented Dec 27, 2024 •

edited

Loading

No progress information while copying images to ECR #6681

No progress information while copying images to ECR #6681

Comments

hannes-ucsc commented Nov 2, 2024 • edited Loading

hannes-ucsc commented Nov 2, 2024

achave11-ucsc commented Nov 4, 2024

dsotirho-ucsc commented Nov 4, 2024

hannes-ucsc commented Nov 5, 2024

dsotirho-ucsc commented Nov 5, 2024 • edited Loading

Without --debug

With --debug

achave11-ucsc commented Nov 6, 2024

dsotirho-ucsc commented Nov 7, 2024 • edited Loading

achave11-ucsc commented Dec 17, 2024

dsotirho-ucsc commented Dec 19, 2024

achave11-ucsc commented Dec 19, 2024

hannes-ucsc commented Dec 27, 2024 • edited Loading

hannes-ucsc commented Nov 2, 2024 •

edited

Loading

dsotirho-ucsc commented Nov 5, 2024 •

edited

Loading

Without `--debug`

With `--debug`

dsotirho-ucsc commented Nov 7, 2024 •

edited

Loading

hannes-ucsc commented Dec 27, 2024 •

edited

Loading