Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: use temporary dir for package manager files #6

Merged
merged 12 commits into from
Feb 21, 2025
Merged

Conversation

synackd
Copy link
Contributor

@synackd synackd commented Feb 11, 2025

Instead of relying on a fixed directory in /tmp that may or may not exist to write package manager logs/cache to, use the tempfile library to create a temporary directory within the container for these at runtime.

Also, move the functions in installer.py to a new Installer class in order to store and reuse the buildah container and mount names, the package manager name, as well as the temporary directory which is created on init.

This is to avoid errors from DNF complaining that /tmp/dnf_test doesn't exist or that it failed to open hawkey.log because the log directory doesn't exist.

Instead of relying on a fixed directory in /tmp that may or may not
exist to write package manager logs to, use the tempfile library to
create a temporary directory within the container for these at runtime.

Also, move the functions in installer.py to a new Installer class in
order to store and reuse the buildah container and mount names, the
package manager, as well as the temporary directory which is created on
init.
@synackd synackd added the needs testing Branch needs testing before review label Feb 11, 2025
@synackd
Copy link
Contributor Author

synackd commented Feb 11, 2025

I've tested this with two DNF image configs: one from scratch and one using an existing parent. More testing would be helpful, though.

If an error occurs in image-build, the buildah container it creates is
unmounted and deleted. This can be rather unhelpful if one wants to
examine package manager logs to find out more about any errors that may
have occurred. Thus, this commit causes the installer to create the
image-build tempdir (where the package manager cache/log files get
stored) on the machine (host or container) where image-build is running.
That way even if/when the buildah container is destroyed, the tempdir
with the logs, etc. remains for inspection.
@synackd
Copy link
Contributor Author

synackd commented Feb 12, 2025

image-build should now print the path to the temporary directory it creates:

$ podman run -it --rm --device /dev/fuse -v ./config.yaml:/home/builder/config.yaml:Z localhost/image-build:test bash
[root@d5657938e178 ~]# image-build --config config.yaml --log-level DEBUG
...
INFO - Installer: Temporary directory for dnf created at /var/tmp/image-build-0xg6_747
...

These files can be inspected afterward:

[root@d5657938e178 ~]# ls /var/tmp/image-build-0xg6_747
dnf
[root@d5657938e178 ~]# ls /var/tmp/image-build-0xg6_747/dnf/log
dnf.librepo.log  dnf.log  dnf.rpm.log  hawkey.log
[root@d5657938e178 ~]# tail /var/tmp/image-build-0xg6_747/dnf/log/dnf.log
2025-02-12T16:50:48+0000 DEBUG DNF version: 4.7.0
2025-02-12T16:50:48+0000 DDEBUG Command: dnf --setopt=reposdir=/home/builder/.local/share/containers/storage/overlay/ff8bbb2f5a4ba649b19f731c1ccd55dd41e9810a1c881e9b9a32eeace439d99f/merged/etc/yum.repos.d --setopt=logdir=/var/tmp/image-build-0xg6_747/dnf/log --setopt=cachedir=/var/tmp/image-build-0xg6_747/dnf/cache config-manager --save --add-repo http://dl.fedoraproject.org/pub/epel/8/Everything/x86_64/ 
2025-02-12T16:50:48+0000 DDEBUG Installroot: /
2025-02-12T16:50:48+0000 DDEBUG Releasever: 8
2025-02-12T16:50:48+0000 DEBUG cachedir: /var/tmp/image-build-0xg6_747/dnf/cache
2025-02-12T16:50:48+0000 DDEBUG Base command: config-manager
2025-02-12T16:50:48+0000 DDEBUG Extra commands: ['--setopt=reposdir=/home/builder/.local/share/containers/storage/overlay/ff8bbb2f5a4ba649b19f731c1ccd55dd41e9810a1c881e9b9a32eeace439d99f/merged/etc/yum.repos.d', '--setopt=logdir=/var/tmp/image-build-0xg6_747/dnf/log', '--setopt=cachedir=/var/tmp/image-build-0xg6_747/dnf/cache', 'config-manager', '--save', '--add-repo', 'http://dl.fedoraproject.org/pub/epel/8/Everything/x86_64/']
2025-02-12T16:50:48+0000 INFO Adding repo from: http://dl.fedoraproject.org/pub/epel/8/Everything/x86_64/
2025-02-12T16:50:48+0000 DDEBUG Cleaning up.
2025-02-12T16:50:48+0000 DDEBUG Plugins were unloaded.

Things get a little funky with permissions when trying to mount a directory to /var/tmp within the container since the container runs with buildah unshare.

$ id -u
5796
$ podman run -it --rm --device /dev/fuse -v ~/image-tmp:/var/tmp/:rw,Z,U -v ./base-8.10.yaml:/home/builder/config.yaml:Z localhost/image-build:test image-build --config config.yaml --log-level DEBUG
$ ls -ld ~/image-tmp/ | cut -f3 -d' '
297609

Since there are other ways to view the files if one is not root to change the permissions. I won't consider this in the scope of this PR.

@davidallendj davidallendj self-requested a review February 19, 2025 16:43
Copy link

@davidallendj davidallendj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this on our Redondo cluster and was able to build a rocky 9 image.

@alexlovelltroy
Copy link
Member

Can we double check that the documentation is still correct at https://github.com/OpenCHAMI/image-builder ? If not, let's update it and merge. It seems like the errors I was seeing are likely user error.

@synackd
Copy link
Contributor Author

synackd commented Feb 21, 2025

I updated the README to add comments to and update the example config for the base layer. I also updated the publish section and updated the run command for the base layer to use podman instead of running bare-metal. I don't have an example of an Ansible-type layer, but can add that too if someone has one (@travisbcotton ?).

@synackd
Copy link
Contributor Author

synackd commented Feb 21, 2025

Added example config for Ansible-type layer. This should be ready for review again.

Copy link
Member

@alexlovelltroy alexlovelltroy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

tested locally

@synackd synackd merged commit 33fa6b2 into main Feb 21, 2025
1 check passed
@synackd synackd deleted the synackd/tempdir branch February 21, 2025 22:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs testing Branch needs testing before review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants