Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

b'no such manifest #20

Closed
basz opened this issue Mar 31, 2023 · 31 comments
Closed

b'no such manifest #20

basz opened this issue Mar 31, 2023 · 31 comments

Comments

@basz
Copy link

basz commented Mar 31, 2023 

docker manifest inspect ghcr.io/plhw/lab-docker/php@sha256:b542bffe4c15244b7e073d711ac91b4cdb71208e2d6bbb8846e1dd576cfa3f51
Traceback (most recent call last):
  File "/home/runner/work/_actions/Chizkiyahu/delete-untagged-ghcr-action/v2/clean_ghcr.py", line 231, in <module>
    delete_pkgs(
  File "/home/runner/work/_actions/Chizkiyahu/delete-untagged-ghcr-action/v2/clean_ghcr.py", line 131, in delete_pkgs
    deps_pkgs = get_deps_pkgs(owner, tagged_pkgs)
  File "/home/runner/work/_actions/Chizkiyahu/delete-untagged-ghcr-action/v2/clean_ghcr.py", line 95, in get_deps_pkgs
    ids.extend(get_image_deps(image))
  File "/home/runner/work/_actions/Chizkiyahu/delete-untagged-ghcr-action/v2/clean_ghcr.py", line 100, in get_image_deps
    manifest_txt = get_manifest(image)
  File "/home/runner/work/_actions/Chizkiyahu/delete-untagged-ghcr-action/v2/clean_ghcr.py", line 110, in get_manifest
    raise Exception(res.stderr)
Exception: b'no such manifest: ghcr.io/plhw/lab-docker/php@sha256:b542bffe4c15244b7e073d711ac91b4cdb71208e2d6bbb8846e1dd5[76](https://github.com/plhw/lab-docker/actions/runs/4572531190/jobs/8071994976#step:3:76)cfa3f51\n'
Error: Process completed with exit code 1.

and

docker manifest inspect ghcr.io/plhw/lab-docker/php@sha256:b542bffe4c15244b7e073d711ac91b4cdb71208e2d6bbb8846e1dd576cfa3f51
  File "/home/runner/work/_actions/Chizkiyahu/delete-untagged-ghcr-action/v2/clean_ghcr.py", line 231, in <module>
    delete_pkgs(
  File "/home/runner/work/_actions/Chizkiyahu/delete-untagged-ghcr-action/v2/clean_ghcr.py", line 131, in delete_pkgs
    deps_pkgs = get_deps_pkgs(owner, tagged_pkgs)
  File "/home/runner/work/_actions/Chizkiyahu/delete-untagged-ghcr-action/v2/clean_ghcr.py", line 95, in get_deps_pkgs
    ids.extend(get_image_deps(image))
  File "/home/runner/work/_actions/Chizkiyahu/delete-untagged-ghcr-action/v2/clean_ghcr.py", line 100, in get_image_deps
    manifest_txt = get_manifest(image)
  File "/home/runner/work/_actions/Chizkiyahu/delete-untagged-ghcr-action/v2/clean_ghcr.py", line 110, in get_manifest
    raise Exception(res.stderr)
Exception: b'manifest unknown\n'

      - name: "Purge Untagged Packages"
        uses: Chizkiyahu/delete-untagged-ghcr-action@v2
        with:
          token: ${{ secrets.PAT_TOKEN }}
          repository_owner: ${{ github.repository_owner }}
          untagged_only: true
          owner_type: org # or user
          except_untagged_multiplatform: true

Looking at my packages, which are multiplatform, i did notice packages with an unknown/unknown platform in them, which is weird also...

anyway, mayby there is a simple fix... thx!
@basz
Copy link
Author

basz commented Mar 31, 2023

image

@Chizkiyahu
Copy link
Owner

@basz can you send me an example of how you create the images?
so I can debug this

@basz
Copy link
Author

basz commented Apr 1, 2023

I have the following action that I run manually on my M1 laptop with 'act'. Arm is then build locally and amd on the linux box. I have no idea where that unknown/unknown comes from. That seems to be something that was introduced since i fugured this out.

act --actor basz --secret-file=.secrets --container-architecture linux/amd64 --job build-and-push-docker

on:
  workflow_dispatch:
  # schedule:
  #   - cron: "0 0 * * 0,3"
  # push:
  #   branches:
  #     - "main"

name: "Build And Push PHP Docker Images"

jobs:
  build-and-push-docker:
    name: "Build And Push PHP Docker Images"

    runs-on: ${{ matrix.operating-system }}

    strategy:
      fail-fast: false
      matrix:
        php-version:
          - "8.1"
          - "8.2"
        php-target:
          - "cli"
          - "fpm"
        operating-system:
          - "ubuntu-latest"

    steps:
      - name: "Checkout"
        uses: actions/checkout@v3

      # - name: Set up QEMU
      #   uses: docker/setup-qemu-action@v2
      #   platforms: amd64,arm64

      # @see https://github.com/docker/setup-buildx-action#quick-start
      - name: Set up Docker Buildx
        id: buildx
        uses: docker/setup-buildx-action@v2
        with:
          install: false
          driver: docker-container
          config: .github/buildkitd.toml
          # buildkitd-flags: --debug

      - name: "Append AMD buildx"
        uses: baschny/append-buildx-action@v1
        with:
          builder: ${{ steps.buildx.outputs.name }}
          endpoint: "ssh://[email protected]"
          platform: linux/amd64
          node_name: ${{ steps.buildx.outputs.name }}
          ssh_private_key: ${{ secrets.SSH_PRIVATE_KEY }}

      - name: Inspect builder
        run: |
          echo "Name:      ${{ steps.buildx.outputs.name }}"
          echo "Endpoint:  ${{ steps.buildx.outputs.endpoint }}"
          echo "Status:    ${{ steps.buildx.outputs.status }}"
          echo "Flags:     ${{ steps.buildx.outputs.flags }}"
          echo "Platforms: ${{ steps.buildx.outputs.platforms }}"

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v2
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      # https://blog.driftingruby.com/docker-builds/

      - name: "Build ${{ matrix.php-version }}-${{ matrix.php-target }}"
        working-directory: php
        run: |
          docker buildx build . --cache-from ghcr.io/plhw/lab-docker/php:${{ matrix.php-version }}-${{ matrix.php-target }} \
                                --target php-${{ matrix.php-target }} \
                                --build-arg PHP_VERSION=${{ matrix.php-version }} \
                                --file php-${{ matrix.php-target }} \
                                --platform linux/arm64,linux/amd64 \
                                --push \
                                --tag ghcr.io/plhw/lab-docker/php:${{ matrix.php-version }}-${{ matrix.php-target }}

          if [ "${{ matrix.php-target }}" = "cli" ] ; then \
            docker buildx build . --cache-from ghcr.io/plhw/lab-docker/php:${{ matrix.php-version }}-${{ matrix.php-target }} \
                                  --target composer-${{ matrix.php-target }} \
                                  --build-arg PHP_VERSION=${{ matrix.php-version }} \
                                  --file php-${{ matrix.php-target }} \
                                  --platform linux/arm64,linux/amd64 \
                                  --push \
                                  --tag ghcr.io/plhw/lab-docker/composer:${{ matrix.php-version }} \
          ; fi

          if [ "${{ matrix.php-target }}" = "fpm" ] ; then \
            docker buildx build . --cache-from ghcr.io/plhw/lab-docker/php:${{ matrix.php-version }}-${{ matrix.php-target }} \
                                  --target php-${{ matrix.php-target }}-xdebug \
                                  --build-arg PHP_VERSION=${{ matrix.php-version }} \
                                  --file php-${{ matrix.php-target }} \
                                  --platform linux/arm64,linux/amd64 \
                                  --push \
                                  --tag ghcr.io/plhw/lab-docker/php:${{ matrix.php-version }}-${{ matrix.php-target }}-xdebug \
            && \
            docker buildx build . --cache-from ghcr.io/plhw/lab-docker/php:${{ matrix.php-version }}-${{ matrix.php-target }} \
                                  --target php-${{ matrix.php-target }}-opcache \
                                  --build-arg PHP_VERSION=${{ matrix.php-version }} \
                                  --file php-${{ matrix.php-target }} \
                                  --platform linux/arm64,linux/amd64 \
                                  --push \
                                  --tag ghcr.io/plhw/lab-docker/php:${{ matrix.php-version }}-${{ matrix.php-target }}-opcache \
          ; fi
        env:
          PHP_VERSION: ${{ matrix.php-version }}

@Chizkiyahu
Copy link
Owner

@basz this needs a private machine that I don't have to run this
please build a docker a toy image and push it to a new repo with admin access for me

FROM alpine:latest
RUN echo "hi"

so I can debug the GitHub API

@Chizkiyahu
Copy link
Owner

@basz
what is the output if you run
1

docker manifest inspect ghcr.io/plhw/lab-docker/php@sha256:b542bffe4c15244b7e073d711ac91b4cdb71208e2d6bbb8846e1dd576cfa3f51

2

docker buildx imagetools inspect ghcr.io/plhw/lab-docker/php@sha256:b542bffe4c15244b7e073d711ac91b4cdb71208e2d6bbb8846e1dd576cfa3f51

@basz
Copy link
Author

basz commented Apr 2, 2023

Different sha 'cause that has been deleted manually...

docker manifest inspect ghcr.io/plhw/lab-docker/php@sha256:b542bffe4c15244b7e073d711ac91b4cdb71208e2d6bbb8846e1dd576cfa3f51

manifest unknown

docker buildx imagetools inspect ghcr.io/plhw/lab-docker/php@sha256:b542bffe4c15244b7e073d711ac91b4cdb71208e2d6bbb8846e1dd576cfa3f51

Name:      ghcr.io/plhw/lab-docker/php@sha256:b542bffe4c15244b7e073d711ac91b4cdb71208e2d6bbb8846e1dd576cfa3f51
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:b542bffe4c15244b7e073d711ac91b4cdb71208e2d6bbb8846e1dd576cfa3f51
           
Manifests: 
  Name:        ghcr.io/plhw/lab-docker/php@sha256:027dae450e3d4629b3ad99456f9d4dd192010a1ecde4318947502f19b86e0337
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/arm64
               
  Name:        ghcr.io/plhw/lab-docker/php@sha256:87fe47ddca82df61ceaba096c27945d62c5c5eb793f340b170e4812661c68822
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations: 
    vnd.docker.reference.type:   attestation-manifest
    vnd.docker.reference.digest: sha256:027dae450e3d4629b3ad99456f9d4dd192010a1ecde4318947502f19b86e0337
               
  Name:        ghcr.io/plhw/lab-docker/php@sha256:cb755e7435191881da12401abf94cf7b2a3ed48d5a146f11341669c51b55bf41
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/amd64
               
  Name:        ghcr.io/plhw/lab-docker/php@sha256:8c467d0740c61a676bb45abaaeb59604d04c926747267885b92d581c786d75f2
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations: 
    vnd.docker.reference.digest: sha256:cb755e7435191881da12401abf94cf7b2a3ed48d5a146f11341669c51b55bf41
    vnd.docker.reference.type:   attestation-manifest

@Chizkiyahu
Copy link
Owner

Different sha 'cause that has been deleted manually...

docker manifest inspect ghcr.io/plhw/lab-docker/php@sha256:b542bffe4c15244b7e073d711ac91b4cdb71208e2d6bbb8846e1dd576cfa3f51

manifest unknown

docker buildx imagetools inspect ghcr.io/plhw/lab-docker/php@sha256:b542bffe4c15244b7e073d711ac91b4cdb71208e2d6bbb8846e1dd576cfa3f51

Name:      ghcr.io/plhw/lab-docker/php@sha256:b542bffe4c15244b7e073d711ac91b4cdb71208e2d6bbb8846e1dd576cfa3f51
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:b542bffe4c15244b7e073d711ac91b4cdb71208e2d6bbb8846e1dd576cfa3f51
           
Manifests: 
  Name:        ghcr.io/plhw/lab-docker/php@sha256:027dae450e3d4629b3ad99456f9d4dd192010a1ecde4318947502f19b86e0337
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/arm64
               
  Name:        ghcr.io/plhw/lab-docker/php@sha256:87fe47ddca82df61ceaba096c27945d62c5c5eb793f340b170e4812661c68822
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations: 
    vnd.docker.reference.type:   attestation-manifest
    vnd.docker.reference.digest: sha256:027dae450e3d4629b3ad99456f9d4dd192010a1ecde4318947502f19b86e0337
               
  Name:        ghcr.io/plhw/lab-docker/php@sha256:cb755e7435191881da12401abf94cf7b2a3ed48d5a146f11341669c51b55bf41
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/amd64
               
  Name:        ghcr.io/plhw/lab-docker/php@sha256:8c467d0740c61a676bb45abaaeb59604d04c926747267885b92d581c786d75f2
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations: 
    vnd.docker.reference.digest: sha256:cb755e7435191881da12401abf94cf7b2a3ed48d5a146f11341669c51b55bf41
    vnd.docker.reference.type:   attestation-manifest

so it look like that for some reason docker buildx imagetools inspect works better than docker manifest inspect
so we needs to update in

delete-untagged-ghcr-action/clean_ghcr.py

Line 106 in 1e4bcb5

cmd = f"docker manifest inspect {image}"

I will update the code

@basz
Copy link
Author

basz commented Apr 2, 2023

repo created and ran build task as above...

I used these two dockerfiles. I think you accept my invite first before i can grant you access to the packages pages...

FROM alpine:latest as php-cli
LABEL org.opencontainers.image.source=https://github.com/bushbaby/delete-action-issue

RUN echo "hi"

FROM php-cli as php-cli-blackfire
FROM php-cli as php-cli-opcache
FROM php-cli as php-cli-xdebug
FROM php-cli as composer-cli
FROM php-cli as composer-cli-1x

FROM alpine:latest as php-fpm
LABEL org.opencontainers.image.source=https://github.com/bushbaby/delete-action-issue

RUN echo "hi"

FROM php-fpm as php-fpm-blackfire
FROM php-fpm as php-fpm-opcache
FROM php-fpm as php-fpm-xdebug
FROM php-fpm as composer-fpm
FROM php-fpm as composer-fpm-1x

@Chizkiyahu
Copy link
Owner

Chizkiyahu commented Apr 2, 2023 •
edited
Loading

@basz
I created branch buildx_inspect
please check that it is working
and then I will create a new version

@basz
Copy link
Author

basz commented Apr 2, 2023

ah.

Run Chizkiyahu/delete-untagged-ghcr-action@buildx_inspect
Run actions/setup-python@v4
Installed versions
Run pip install -r /home/runner/work/_actions/Chizkiyahu/delete-untagged-ghcr-action/buildx_inspect/requirements.txt
Collecting requests
  Downloading requests-2.2[8](https://github.com/plhw/lab-docker/actions/runs/4588453243/jobs/8102675332#step:3:9).2-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 1.5 MB/s eta 0:00:00
Collecting charset-normalizer<4,>=2
  Downloading charset_normalizer-3.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux20[14](https://github.com/plhw/lab-docker/actions/runs/4588453243/jobs/8102675332#step:3:16)_x86_64.whl (199 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 199.2/199.2 kB 9.0 MB/s eta 0:00:00
Collecting idna<4,>=2.5
  Downloading idna-3.4-py3-none-any.whl (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 11.7 MB/s eta 0:00:00
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.[26](https://github.com/plhw/lab-docker/actions/runs/4588453243/jobs/8102675332#step:3:30).15-py2.py3-none-any.whl (140 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.9/140.9 kB 23.1 MB/s eta 0:00:00
Collecting certifi>=2017.4.17
  Downloading certifi-2022.12.7-py3-none-any.whl (155 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.3/155.3 kB 31.4 MB/s eta 0:00:00
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests
Successfully installed certifi-2022.12.7 charset-normalizer-3.1.0 idna-3.4 requests-2.[28](https://github.com/plhw/lab-docker/actions/runs/4588453243/jobs/8102675332#step:3:32).2 urllib3-1.26.15
Run args="--token *** "
args: --token ***  --repository_owner plhw  --untagged_only true  --except_untagged_multiplatform true  --owner_type org 
Traceback (most recent call last):
docker buildx imagetools inspect ghcr.io/plhw/lab-docker/php@sha256:fa62eca220c509c6bbcd8085e45a46534a5b9387476dd6c53ff5fb84996dc5e7 --format "{{json .Manifest}}"
  File "/home/runner/work/_actions/Chizkiyahu/delete-untagged-ghcr-action/buildx_inspect/clean_ghcr.py", line 231, in <module>
    delete_pkgs(
  File "/home/runner/work/_actions/Chizkiyahu/delete-untagged-ghcr-action/buildx_inspect/clean_ghcr.py", line 131, in delete_pkgs
    deps_pkgs = get_deps_pkgs(owner, tagged_pkgs)
  File "/home/runner/work/_actions/Chizkiyahu/delete-untagged-ghcr-action/buildx_inspect/clean_ghcr.py", line 95, in get_deps_pkgs
    ids.extend(get_image_deps(image))
  File "/home/runner/work/_actions/Chizkiyahu/delete-untagged-ghcr-action/buildx_inspect/clean_ghcr.py", line 100, in get_image_deps
    manifest_txt = get_manifest(image)
  File "/home/runner/work/_actions/Chizkiyahu/delete-untagged-ghcr-action/buildx_inspect/clean_ghcr.py", line 110, in get_manifest
    raise Exception(res.stderr)
Exception: b'ERROR: failed to copy: httpReadSeeker: failed open: content at https://ghcr.io/v2/plhw/lab-docker/php/manifests/sha256:b2189c5d99afcf85b7a2209455837017dfa6a178de82[29](https://github.com/plhw/lab-docker/actions/runs/4588453243/jobs/8102675332#step:3:33)260[33](https://github.com/plhw/lab-docker/actions/runs/4588453243/jobs/8102675332#step:3:37)78caecf3fbc[42](https://github.com/plhw/lab-docker/actions/runs/4588453243/jobs/8102675332#step:3:46) not found: not found\n'
Error: Process completed with exit code 1.

@Chizkiyahu
Copy link
Owner

Chizkiyahu commented Apr 2, 2023 •
edited
Loading

the repo "https://github.com/bushbaby/delete-action-issue"
is empty
did you push something to the repo?
did connect the docker image to the repo?

is need more debug
let's see if we can reproduce the bug in "delete-action-issue"

@basz
Copy link
Author

basz commented Apr 2, 2023

repo online

@basz
Copy link
Author

basz commented Apr 2, 2023

plus failing job : https://github.com/bushbaby/delete-action-issue/actions/runs/4588802273/jobs/8103267528

@Chizkiyahu
Copy link
Owner

I need at least read access to the docker images container
did you connect that package to the repo?
I need access to the docker image
Screenshot 2023-04-02 at 16 01 48

@Chizkiyahu
Copy link
Owner

or you can add me in Package settings -> Manage access

@basz
Copy link
Author

basz commented Apr 2, 2023

try now

@Chizkiyahu
Copy link
Owner

try now

still not working
get to https://api.github.com/orgs/bushbaby/packages?package_type=container
return 200 with an empty list
when I using my token

@basz
Copy link
Author

basz commented Apr 2, 2023

now?

@Chizkiyahu
Copy link
Owner

the error for me starts when I try to run pull
docker pull ghcr.io/bushbaby/delete-action-issue/php:8.1-fpm-opcache
did pull working for you on your mac?
Screenshot 2023-04-02 at 17 14 12

@basz
Copy link
Author

basz commented Apr 2, 2023

uhm. that's weird.

docker pull ghcr.io/bushbaby/delete-action-issue/php:8.1-cli pulls ok...

no idea what that could be

@Chizkiyahu
Copy link
Owner

On you Mac is same error ?
If yes
The problem is probably something with build or push

@basz
Copy link
Author

basz commented Apr 2, 2023

some pull ok some do not

maybe one the cause is this additional job... running next to yours...

https://github.com/bushbaby/delete-action-issue/actions/runs/4588802273/jobs/8103717098

Total versions deleted till now: 12

@basz
Copy link
Author

basz commented Apr 2, 2023 •
edited
Loading

reason for using two actions is that I want untagged packages deleted. But in addition want tagged packages deleted (with some constraints) which is something your action is not capable of i remember correctly...

@Chizkiyahu
Copy link
Owner

Chizkiyahu commented Apr 2, 2023 •
edited
Loading

for debug
delete all image manual in Github gui
and then only build and try to run pull
and run only delete-untagged-ghcr-action action

@Chizkiyahu
Copy link
Owner

another option to debug
is delete all
and then trying to build with QEMU instead of "append-buildx-action"
or disable AMD for test

@basz
Copy link
Author

basz commented Apr 2, 2023

test 1.
i deleted the php packages and rebuild using the actions. 8 images appeared.

I am then able to pull every image...

I then ran the delete-untagged-ghcr-action action which then suprisinlgy delete a bunch of stuff

https://github.com/bushbaby/delete-action-issue/actions/runs/4589441325/jobs/8104330113#step:3:80

@basz
Copy link
Author

basz commented Apr 2, 2023

you want me to do the same with only arm?

@Chizkiyahu
Copy link
Owner

so probably the bug start because both action run in the same time
try to run my action
and after actions/delete-package-versions with needs: purge-untagged-packages
so they don't run at same time

@basz
Copy link
Author

basz commented Apr 2, 2023

hm, that makes sense.

However on the next run it still run into issues.

I think maybe 'actions/delete-package-versions@v4' also deletes untagged versions that are part of a multiplatform package... It has a flag of delete-only-untagged-versions but not the reverse 'delete-only-tagged-versions'.

I guess that action is simply incompatible for multiplatform containers.

Anyway thanks for working on this. Very much appreciated!

@basz
Copy link
Author

basz commented Apr 2, 2023

actions/delete-package-versions#90

@Chizkiyahu
Copy link
Owner

@basz
Is look like that you can modify the clean_ghcr.py to do what you doing with actions/delete-package-versions@v4 without to match work
you can debug and write on clean_ghcr.py directly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@basz @Chizkiyahu