Install with custom kickstart anaconda-iso failed with error "Kickstart file /run/install/ks.cfg is missing"

[henrywang]

Build anaconda-iso with custom kickstart added. Install with this iso failed with error Kickstart file /run/install/ks.cfg is missing. I mount this iso and check the ks file in it. The custom kickstart content has been added into this ISO

[fedora@fedora-39-240314175643 ~]$ sudo mount /var/lib/libvirt/images/install.iso /mnt
mount: /mnt: WARNING: source write-protected, mounted read-only.

[fedora@fedora-39-240314175643 ~]$ ll /mnt
total 10
drwxr-xr-x. 1 root root 2048 Sep  9 04:00 container
drwxr-xr-x. 1 root root 2048 Sep  9 03:54 EFI
drwxr-xr-x. 1 root root 2048 Sep  9 03:59 images
drwxr-xr-x. 1 root root 2048 Sep  9 03:59 isolinux
-rw-r--r--. 1 root root   94 Sep  9 04:00 osbuild-base.ks
-rw-r--r--. 1 root root 1060 Sep  9 03:53 osbuild.ks

[fedora@fedora-39-240314175643 ~]$ cat /mnt/osbuild-base.ks 
ostreecontainer --url=/run/install/repo/container --transport=oci --no-signature-verification

[fedora@fedora-39-240314175643 ~]$ cat /mnt/osbuild.ks
%include /run/install/repo/osbuild-base.ks
text
network --bootproto=dhcp --device=link --activate --onboot=on

rootpw --lock --iscrypted locked
user --name=admin --groups=wheel --iscrypted
sshkey --username=admin "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc+O4clxdLZkx86x7rfFc7sdP1mxwM/LWkHYgbAQllQY6y+diXT1PQ7SxZrCkLNs+NGnF7OugAI3tGjR8EhfI2PYIqTO4xSAoFkiBGQNoXRfCr9N67kxx5ynQwbuC96mWUgVVwDkRiPmDoz2v4CM6EomGeEKCkIFwWxboC506koWI0FWR6XoZlHLUwwnIiyITftJ4iSwFqVrO+hxg0ERPeWtqHU+womHkWitxI+Al8IHOddqsVqT/2ssscNrMQEO0lB0dqehvLPCkIbG2omb80m/+/IHxHSEVPPF8luAFA13D1G7KeQO3sYYcr2yceYkUTIcyNOwTrkKDYD9lBreQr fedora@fedora-39-240314175643"

zerombr
clearpart --all --initlabel --disklabel=gpt
autopart --nohome --type=lvm --fstype=xfs"

bootloader --append="console=ttyS0,115200n8"

poweroff

%pre
#!/bin/sh
curl -kLO http://192.168.100.1/domain.crt --output-dir /etc/pki/ca-trust/source/anchors
update-ca-trust
%end

%post --log=/var/log/anaconda/post-install.log --erroronfail
# no sudo password for user admin
echo -e 'admin	ALL=(ALL)	NOPASSWD: ALL' >> /etc/sudoers
%end

bootc image: registry.stage.redhat.io/rhel9/rhel-bootc:rhel-9.5
bib image: registry.stage.redhat.io/rhel9/bootc-image-builder:rhel-9.5

Build command:

sudo podman run \
            --rm \
            -it \
            --privileged \
            --pull=newer \
            --tls-verify=false \
            --security-opt label=type:unconfined_t \
            -v "${TEMPDIR}/config.toml":/config.toml:ro \
            -v "$TEMPDIR":/output \
            -v /var/lib/containers/storage:/var/lib/containers/storage \
            "$BIB_IMAGE_URL" \
            --type "anaconda-iso" \
            --tls-verify=false \
            --target-arch "$ARCH" \
            --chown "$(id -u "$(whoami)"):$(id -g "$(whoami)")" \
            --rootfs "$ROOTFS" \
            "$TEST_IMAGE_URL"

config.toml:

[customizations.installer.kickstart]
contents = """
text
network --bootproto=dhcp --device=link --activate --onboot=on

rootpw --lock --iscrypted locked
user --name=admin --groups=wheel --iscrypted
sshkey --username=admin "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc+O4clxdLZkx86x7rfFc7sdP1mxwM/LWkHYgbAQllQY6y+diXT1PQ7SxZrCkLNs+NGnF7OugAI3tGjR8EhfI2PYIqTO4xSAoFkiBGQNoXRfCr9N67kxx5ynQwbuC96mWUgVVwDkRiPmDoz2v4CM6EomGeEKCkIFwWxboC506koWI0FWR6XoZlHLUwwnIiyITftJ4iSwFqVrO+hxg0ERPeWtqHU+womHkWitxI+Al8IHOddqsVqT/2ssscNrMQEO0lB0dqehvLPCkIbG2omb80m/+/IHxHSEVPPF8luAFA13D1G7KeQO3sYYcr2yceYkUTIcyNOwTrkKDYD9lBreQr fedora@fedora-39-240314175643"

zerombr
clearpart --all --initlabel --disklabel=gpt
autopart --nohome --type=lvm --fstype=xfs"

bootloader --append="console=ttyS0,115200n8"

poweroff

%pre
#!/bin/sh
curl -kLO http://192.168.100.1/domain.crt --output-dir /etc/pki/ca-trust/source/anchors
update-ca-trust
%end

%post --log=/var/log/anaconda/post-install.log --erroronfail
# no sudo password for user admin
echo -e 'admin\tALL=(ALL)\tNOPASSWD: ALL' >> /etc/sudoers
%end
"""

[chunfuwen]

wrapped custom kickstart content in config.json format as below

\{'blueprint': {'customizations': {'kernel': {'append': 'mitigations=auto,nosmt'}, 'installer': {'modules': {'enable': ['org.fedoraproject.Anaconda.Modules.Localization', 'org.fedoraproject.Anaconda.Modules.Timezone'], 'disable': ['org.fedoraproject.Anaconda.Modules.Timezone']}, 'kickstart': {'contents': 'user --name alice --password bob --groups wheel\nsshkey --username alice "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7cuKGQnDMMgequx5VgndBLenBD+EPn0izTeItZLtw+wtsYIOKYl2nKJu6U/kan7uSQ7xZbC3xen8Isce6UlYmjh1jMFRRFNqXqRIw1IiZqUTu8a4830ipHpd2lMBtQX+JRdob2k6+DrL0m5GrGQvN4fZTWUiDOmq4yUtihfPJ0+ipEhrNRYg9Y0aUvqoIJLq6hIrpYjimFbhZwhNK+mG3zKqkw1AeF29n1qvjDci/0W4+20UrBZlcqPPI7p/izq6jkw8FD5WZp5P7Pad1v/c/A7jZD+ug9l7SOZhhC0GALZC8BmAnihTYS5s/xMfSDHQhbUlB9Cn6LPzht3kmLj5JqeRLdR9vN5g3yrYPX1GEys/vng+Zk2eBC/oaa7Zv5UouevqtTqoiMaUmXX4nNk31vWYWM3tfhjnnQ/Uz2g8HTBbus/OIhEboBi5WF8M5WyS3QbRorXbgcSsWSr81f190BJ2tfvq6uNqKfIrE8BoJ58APPQFOhJQWiveGUk/Wkxc= [email protected]"\ntext --non-interactive\nzerombr\nclearpart --all --initlabel --disklabel=gpt\nautopart --noswap --type=lvm\nnetwork --bootproto=dhcp --device=link --activate --onboot=on\n reboot'}}}}}

And use upstream bib and fedora 40 bootc image , iso build is successful.

sudo podman run --rm -it --privileged --pull=newer --security-opt label=type:unconfined_t -v /var/lib/libvirt/images/output:/output -v /var/lib/libvirt/images/config.json:/config.json   -v /var/lib/libvirt/images/auth.json:/run/containers/0/auth.json  quay.io/centos-bootc/bootc-image-builder:latest  --type anaconda-iso --tls-verify=true  --config /config.json  quay.io/fedora/fedora-bootc:40  --chown 107:107  --rootfs ext4 '
...
2024-09-10 07:09:29,721 process          L0470 DEBUG| [stdout] manifest - finished successfully
2024-09-10 07:09:29,722 process          L0470 DEBUG| [stdout] build:    	9ef1877ba9aabb09a32fc12251e0fab86cb5daa03b3649941b0518ed207d8609
2024-09-10 07:09:29,722 process          L0470 DEBUG| [stdout] anaconda-tree:	63a0cfe2fd09d3ed36c70c9205d684c77caa5d5850677cd36858bbbf9983c5eb
2024-09-10 07:09:29,723 process          L0470 DEBUG| [stdout] rootfs-image:	35a588897a2fa3cbbb21b3ef310e8f84cfa77da126b8eb2b2e5ecadffdbfda69
2024-09-10 07:09:29,723 process          L0470 DEBUG| [stdout] efiboot-tree:	477195f7bf261060576afd3b0e0586a1545eaea35cca6f8194d95fa9b0dba473
2024-09-10 07:09:29,723 process          L0470 DEBUG| [stdout] bootiso-tree:	1e72ff23a7e05a6207a292ef17d74ee6728bbbed988da144a9519d5fcdf35fe1
2024-09-10 07:09:29,723 process          L0470 DEBUG| [stdout] bootiso:  	ec4e535f1058fd6ff8a7933d806d46994bcd7124356b9158314246f6978e61cd
2024-09-10 07:09:32,511 process          L0470 DEBUG| [stdout] Build complete!
2024-09-10 07:09:32,511 process          L0470 DEBUG| [stdout] Results saved in
...

[chunfuwen]

Also I didn't hit this issue using registry.stage.redhat.io/rhel9/bootc-image-builder:rhel-9.5 and registry.stage.redhat.io/rhel9/rhel-bootc:rhel-9.5 while wrapping kickstart content in config.json