Hardlinked /etc issue (0.1.14)

[kfox1111]

If I use Kaniko to customize an image with the following Dockerfile:

FROM quay.io/almalinuxorg/almalinux-bootc:9.4-20241023
RUN \
  sed -i 's/^mirrorlist=/#mirrorlist=/g' /etc/yum.repos.d/almalinux* && \
  sed -i 's/^# baseurl=/baseurl=/g' /etc/yum.repos.d/almalinux* && \
  dnf install -y ipmitool && \
  dnf clean all

It builds fine, but bootc wont update from it:

# bootc update --apply
layers already present: 66; layers needed: 1 (16.1 MB)
Fetched layers: 0 B in 0 seconds (0 B/s)
ERROR Upgrading: Pulling: Importing: Parsing layer blob sha256:4b0c8eadb903bd0240bb6ab893234db523e7f4a714ea75c109223edaf2618b1f: error: ostree-tar: Processing deferred hardlink sysroot/ostree/repo/objects/41/9c4ead30115abd80d094723b6d25e0d9bda444a445cd979a7ba23217177b74.file: Failed to find object: No such file or directory: 9c4ead30115abd80d094723b6d25e0d9bda444a445cd979a7ba23217177b74.file: Processing tar: Failed to commit tar: ExitStatus(unix_wait_status(256))

Interestingly, it only seems to happen with dnf installs. With copying files in, or doing other commands, the images deploy fine.

[kfox1111]

It resembles:
coreos/rpm-ostree#3827

[cgwalters]

#666 should have fixed that and that's in 0.1.14

[kfox1111]

# rpm -qa | grep bootc
bootc-0.1.14-1.el9_4.x86_64

Looks like I have that though...

[kfox1111]

The same Dockerfile works when built with podman. so something different between the way kaniko lays out the image then podman does. Some way I can compare the two?

[kfox1111]

One of the few differences (besides timestamps) I've noticed so far is this:

# diff -u <(tar -tvf after-kaniko/blobs/sha256/5f061bf5eba51504d1a8f63d5edd8833a8604d6584bf2780b9c936818896d2d0 | awk '{print $6}') <(tar -tvf after-podman/blobs/sha256/baa8d403348b8454657ef1016ffbc1d0d1afc1223da3512ca2f1b7dd46f6cd6b | awk '{print $6}')
tar: Removing leading `/' from member names
tar: Removing leading `/' from hard link targets
--- /dev/fd/63	2024-10-29 22:34:40.556687969 -0400
+++ /dev/fd/62	2024-10-29 22:34:40.556687969 -0400
@@ -1,4 +1,3 @@
-/
 etc/
 etc/ld.so.cache
 etc/pki/
@@ -24,7 +23,6 @@
 sysroot/ostree/repo/objects/41/9c4ead30115abd80d094723b6d25e0d9bda444a445cd979a7ba23217177b74.file
 sysroot/ostree/repo/objects/bf/
 sysroot/ostree/repo/objects/bf/13d8fe63612425b89f526a906f06359a1a71ac71902caef8a7160558d91187.file
-tmp/
 usr/
 usr/bin/
 usr/bin/ipmitool
@@ -678,4 +676,3 @@
 var/log/hawkey.log
 var/log/rhsm/
 var/log/rhsm/rhsm.log
-var/tmp/

Not sure if that matters or not or is even related.

[kfox1111]

Oh. I think I may have found a difference... I didn't watch the warning closely enough. Its this:

# diff -u <(tar -tvf after/blobs/sha256/5f061bf5eba51504d1a8f63d5edd8833a8604d6584bf2780b9c936818896d2d0 -P | awk '{print $6, $7, $8, $9}') <(tar -tvf after-podman/blobs/sha256/baa8d403348b8454657ef1016ffbc1d0d1afc1223da3512ca2f1b7dd46f6cd6b -P | awk '{print $6, $7, $8, $9}')
--- /dev/fd/63	2024-10-29 22:57:57.444604708 -0400
+++ /dev/fd/62	2024-10-29 22:57:57.444604708 -0400
@@ -1,4 +1,3 @@
-/   
 etc/   
 etc/ld.so.cache   
 etc/pki/   
@@ -24,7 +23,6 @@
 sysroot/ostree/repo/objects/41/9c4ead30115abd80d094723b6d25e0d9bda444a445cd979a7ba23217177b74.file   
 sysroot/ostree/repo/objects/bf/   
 sysroot/ostree/repo/objects/bf/13d8fe63612425b89f526a906f06359a1a71ac71902caef8a7160558d91187.file   
-tmp/   
 usr/   
 usr/bin/   
 usr/bin/ipmitool   
@@ -295,8 +293,8 @@
 usr/lib/python3.9/site-packages/urllib3/util/__pycache__/wait.cpython-39.pyc   
 usr/lib/sysimage/   
 usr/lib/sysimage/rpm-ostree-base-db/   
-usr/lib/sysimage/rpm-ostree-base-db/rpmdb.sqlite link to /sysroot/ostree/repo/objects/41/9c4ead30115abd80d094723b6d25e0d9bda444a445cd979a7ba23217177b74.file
-usr/lib/sysimage/rpm-ostree-base-db/rpmdb.sqlite-shm link to /sysroot/ostree/repo/objects/bf/13d8fe63612425b89f526a906f06359a1a71ac71902caef8a7160558d91187.file
+usr/lib/sysimage/rpm-ostree-base-db/rpmdb.sqlite link to sysroot/ostree/repo/objects/41/9c4ead30115abd80d094723b6d25e0d9bda444a445cd979a7ba23217177b74.file
+usr/lib/sysimage/rpm-ostree-base-db/rpmdb.sqlite-shm link to sysroot/ostree/repo/objects/bf/13d8fe63612425b89f526a906f06359a1a71ac71902caef8a7160558d91187.file
 usr/lib64/   
 usr/lib64/python3.9/   
 usr/lib64/python3.9/__pycache__/   
@@ -646,8 +644,8 @@
 usr/share/man/man1/ipmitool.1.gz   
 usr/share/rpm/   
 usr/share/rpm/.rpm.lock   
-usr/share/rpm/rpmdb.sqlite link to /sysroot/ostree/repo/objects/41/9c4ead30115abd80d094723b6d25e0d9bda444a445cd979a7ba23217177b74.file
-usr/share/rpm/rpmdb.sqlite-shm link to /sysroot/ostree/repo/objects/bf/13d8fe63612425b89f526a906f06359a1a71ac71902caef8a7160558d91187.file
+usr/share/rpm/rpmdb.sqlite link to sysroot/ostree/repo/objects/41/9c4ead30115abd80d094723b6d25e0d9bda444a445cd979a7ba23217177b74.file
+usr/share/rpm/rpmdb.sqlite-shm link to sysroot/ostree/repo/objects/bf/13d8fe63612425b89f526a906f06359a1a71ac71902caef8a7160558d91187.file
 usr/share/rpm/rpmdb.sqlite-wal   
 var/   
 var/cache/   
@@ -678,4 +676,3 @@
 var/log/hawkey.log   
 var/log/rhsm/   
 var/log/rhsm/rhsm.log   
-var/tmp/   

kaniko has /'s on the head of the hardlinks. podman does not.

[kfox1111]

Ok. That was indeed the problem.

So...

1. I think for better compatibility with other runtimes, the issue should be fixed here.
2. Even once its fixed though, its hard to get it back propagated to other bootc using distros. So, I will try and get a fix into Kaniko as well. Its easier to patch Kaniko until all the bootc's support it then try and patch all the bootc's for a while.

[cgwalters]

Definitely, that's just an oversight in the original change in ostreedev/ostree-rs-ext#408

That said, ostreedev/ostree-rs-ext#674 would also obsolete all that logic.

[kfox1111]

I proposed the fix to the Kaniko repo. Tested and working. If anyone else hits this problem before its fixed here, thats the workaround. :)

[kfox1111]

Thanks @cgwalters! :)

[cgwalters]

Thanks for proposing a change to Kaniko, but we should also fix it in bootc. I started this in ostreedev/ostree-rs-ext#680

[cgwalters]

This should be fixed by #867 which rolls in ostreedev/ostree-rs-ext#680

Please test it out if you have a chance and confirm. There's C9S builds here for example that install on top of alma9.4 for me: https://copr.fedorainfracloud.org/coprs/rhcontainerbot/bootc/build/8210416/
(Or build from source, etc.)