fix(fedora-iot-raw): enable ignition-firstboot-complete.service

[sarmahaj]

fixes: fedora-iot/iot-distro#14

• enabling this service by default so that /boot/ignition.firstboot file is removed on first boot successfully, which is expected behavior.
• Since this was missing earlier, /boot/ignition.firstboot was never removed and at each boot any changes to systemd services were reset.

[achilleas-k]

Hi Sarita, thanks for the PR. Unfortunately, this is going to need a bit more work.

Currently, there's no support for enabling or disabling services during an ostree deployment (like the creation of a raw image from an ostree commit). We would need to add that to the ostreeDeploymentCustomizations() function first. Then, the proper way to enable would be to add it to the image config for the imageType definition itself. After that, it should work.

Small question: Is this only relevant for the raw image, or should we add it to the iot-qcow2-image as well?

[sarmahaj]

@achilleas-k Thanks for reviewing my PR.
will make changes as per your suggestion 👍
also take a look at other image types iot-qcow2-image & iot-commit to confirm if they also need a similar change.

[runcom]

I don't understand why we need to enable the service during deployment, how are these working instead https://github.com/osbuild/images/blob/main/pkg/distro/fedora/images.go#L538 ?

[supakeen]

I don't understand why we need to enable the service during deployment, how are these working instead https://github.com/osbuild/images/blob/main/pkg/distro/fedora/images.go#L538 ?

Those are in the commit.

[runcom]

Those are in the commit.

I can see that, I'm asking why enabling the service in the deployment if the commit has the presets....

[supakeen]

Those are in the commit.

I can see that, I'm asking why enabling the service in the deployment if the commit has the presets....

Ahhh I misread it as 'but other services are getting enabled like this' :)

[achilleas-k]

I was ready to approve this because it does what it says in the description:

It sets the systemd.preset for ignition-fistboot-complete.service to enabled.

This affects two Fedora image types: the iot-raw-image and the iot-simplified-installer by extension, because the installer embeds the iot-raw-image.

For the iot-simplified-installer the following services are also affected in the same way, because these services are defined in the image config (note their absence in the iot-raw-image config).

(Note though that this doesn't affect the iot-qcow2-image. Perhaps we should make it reuse the same configuration as the iot-raw-image, but that's a topic for another time.)

However, I decided to test this PR, but I didn't manage to reproduce the original issue described in fedora-iot/iot-distro#14.

Here's what I did, using the main branch of this repository (i.e. without this PR):

1. Build an iot-container with an empty blueprint.
2. Start the container.
3. Build an iot-raw-image with a user and set the ostree URL to the container (the user is added so I can easily log into the machine after I boot it without needing external configs; the blueprint is otherwise empty).
4. Boot the image, SSH in, and run the following:

[root@localhost ~]# systemctl status ignition-firstboot-complete.service
● ignition-firstboot-complete.service - Mark Ignition Boot Complete
     Loaded: loaded (/usr/lib/systemd/system/ignition-firstboot-complete.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (exited) since Mon 2025-02-24 18:35:50 UTC; 2min 53s ago
    Process: 900 ExecStart=/bin/bash -c mount -o remount,rw /boot && rm -rf /boot/ignition && rm -rf /boot/ignition.firstboot (code=exited, status=0/SUCCESS)
   Main PID: 900 (code=exited, status=0/SUCCESS)
        CPU: 9ms

Feb 24 18:35:49 localhost.localdomain systemd[1]: Starting ignition-firstboot-complete.service - Mark Ignition Boot Complete...
Feb 24 18:35:50 localhost.localdomain systemd[1]: Finished ignition-firstboot-complete.service - Mark Ignition Boot Complete.

[root@localhost ~]# systemctl status zezere_ignition.timer
● zezere_ignition.timer - Trigger Ignition for Zezere until it finishes
     Loaded: loaded (/usr/lib/systemd/system/zezere_ignition.timer; enabled; preset: enabled)
     Active: active (waiting) since Mon 2025-02-24 18:35:48 UTC; 4min 30s ago
    Trigger: Mon 2025-02-24 18:41:09 UTC; 49s left
   Triggers: ● zezere_ignition.service

Feb 24 18:35:48 localhost.localdomain systemd[1]: Started zezere_ignition.timer - Trigger Ignition for Zezere until it finishes.

[root@localhost ~]# systemctl status rpm-ostreed-automatic.timer
○ rpm-ostreed-automatic.timer - rpm-ostree Automatic Update Trigger
     Loaded: loaded (/usr/lib/systemd/system/rpm-ostreed-automatic.timer; disabled; preset: disabled)
     Active: inactive (dead)
    Trigger: n/a
   Triggers: ● rpm-ostreed-automatic.service
       Docs: man:rpm-ostree(1)
             man:rpm-ostreed.conf(5)

[root@localhost ~]# systemctl disable zezere_ignition.timer
Removed "/etc/systemd/system/timers.target.wants/zezere_ignition.timer".

[root@localhost ~]# systemctl enable rpm-ostreed-automatic.timer
Created symlink /etc/systemd/system/timers.target.wants/rpm-ostreed-automatic.timer → /usr/lib/systemd/system/rpm-ostreed-automatic.timer.

[root@localhost ~]# reboot

and after reboot

[root@localhost ~]# systemctl status ignition-firstboot-complete.service
○ ignition-firstboot-complete.service - Mark Ignition Boot Complete
     Loaded: loaded (/usr/lib/systemd/system/ignition-firstboot-complete.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: inactive (dead)
  Condition: start condition unmet at Mon 2025-02-24 18:41:50 UTC; 21s ago
             └─ ConditionKernelCommandLine=ignition.firstboot was not met

Feb 24 18:41:50 localhost.localdomain systemd[1]: ignition-firstboot-complete.service - Mark Ignition Boot Complete was skipped because of an unmet condition check (ConditionKernelCommandLine=ignition.firstboot).

[root@localhost ~]# systemctl status zezere_ignition.timer
○ zezere_ignition.timer - Trigger Ignition for Zezere until it finishes
     Loaded: loaded (/usr/lib/systemd/system/zezere_ignition.timer; disabled; preset: enabled)
     Active: inactive (dead)
    Trigger: n/a
   Triggers: ● zezere_ignition.service

[root@localhost ~]# systemctl status rpm-ostreed-automatic.timer
● rpm-ostreed-automatic.timer - rpm-ostree Automatic Update Trigger
     Loaded: loaded (/usr/lib/systemd/system/rpm-ostreed-automatic.timer; enabled; preset: disabled)
     Active: active (waiting) since Mon 2025-02-24 18:41:50 UTC; 46s ago
    Trigger: Mon 2025-02-24 19:41:44 UTC; 59min left
   Triggers: ● rpm-ostreed-automatic.service
       Docs: man:rpm-ostree(1)
             man:rpm-ostreed.conf(5)

Feb 24 18:41:50 localhost.localdomain systemd[1]: Started rpm-ostreed-automatic.timer - rpm-ostree Automatic Update Trigger.

So what I'm seeing is:

• ignition-firstboot-complete.service is enabled without this patch.
• ignition-firstboot-complete.service is skipped on second boot.
• disabling zezere_ignition.timer survives the reboot.
• enabling rpm-ostreed-automatic.service survives the reboot.

So I have to echo @runcom's question but also add to that whether the original issue was accurately diagnosed.

[sarmahaj]

Adding my observation which led to this patch:
When I created iot-raw-image image same steps as @achilleas-k has mentioned (fedora-41) , found out that /boot/ignition.firstboot file was present which should be removed by ignition-firstboot-complete.service (https://github.com/fedora-iot/ignition-edge/blob/main/systemd/ignition-firstboot-complete.service) .
Since this file was present every time at boot time system initial conditions were applied and hence systemctl services even if disabled on prev boot were enabled (if enabled by default). (I ried disabling ModemManager.service) .
Right now I am not able to test it but @achilleas-k could you please confirm if you see /boot/ignition.firstboot file on first boot or not.
If not probably that is resolved now !?
Thanks!

[miabbott]

The root cause of the original reported issue may be the same as fedora-iot/iot-distro#70 and fedora-iot/iot-distro#49.

Due to how the ostree commit is produced in the Fedora infra, we maintain another package manifest for the ostree commit at https://pagure.io/fedora-iot/ostree/blob/main/f/fedora-iot-base.yaml. That manifest was missing the ignition-edge package, which was fixed with https://pagure.io/fedora-iot/ostree/pull-request/66.

I suspect that if we were to test the latest ostree commit/disk artifacts from a nightly Fedora 41 compose (i.e https://kojipkgs.fedoraproject.org/compose/iot/latest-Fedora-IoT-41/compose/IoT/x86_64/), the original issue may not present itself anymore.

[miabbott]

I suspect that if we were to test the latest ostree commit/disk artifacts from a nightly Fedora 41 compose (i.e https://kojipkgs.fedoraproject.org/compose/iot/latest-Fedora-IoT-41/compose/IoT/x86_64/), the original issue may not present itself anymore.

My hypothesis appears to be incorrect. I grabbed a raw disk image from the latest F41 compose (20250224) and installed it in a VM.

Repeating the same steps as @achilleas-k in the comment above, I found that ignition-firstboot-complete.service was not enabled:

[root@localhost ~]# sudo systemctl status ignition-firstboot-complete.service        
○ ignition-firstboot-complete.service - Mark Ignition Boot Complete                                                                                                                           
     Loaded: loaded (/usr/lib/systemd/system/ignition-firstboot-complete.service; disabled; preset: disabled)                                                                                 
    Drop-In: /usr/lib/systemd/system/service.d                                                                                                                                                
             └─10-timeout-abort.conf, 50-keep-warm.conf                                  
     Active: inactive (dead)                                                                                          

And trying to disable/enable additional services did not persist reboots.

Perhaps then, the fix would be to include ignition-firstboot-complete in the list of services to be enabled as part of the Fedora infra ostree commit? https://pagure.io/fedora-iot/ostree/blob/main/f/fedora-iot-base.yaml#_193

[achilleas-k]

Perhaps then, the fix would be to include ignition-firstboot-complete in the list of services to be enabled as part of the Fedora infra ostree commit? https://pagure.io/fedora-iot/ostree/blob/main/f/fedora-iot-base.yaml#_193

If that fixes the issue and is preferable, sure. I'll redo the test tomorrow with the official Fedora ostree commit. Even so, if you'd rather have this change here as well as a fallback, that's also fine, even if it might not be necessary.

[miabbott]

Perhaps then, the fix would be to include ignition-firstboot-complete in the list of services to be enabled as part of the Fedora infra ostree commit? https://pagure.io/fedora-iot/ostree/blob/main/f/fedora-iot-base.yaml#_193

I think we also need the service enabled via presets here - https://src.fedoraproject.org/rpms/fedora-release/blob/rawhide/f/80-iot.preset

If we are gonna muck around in that preset file, we should probably ensure that there is parity there with what is defined in

images/pkg/distro/fedora/images.go

Lines 536 to 549 in 1e6f652

	img.OSCustomizations.Presets = []osbuild.Preset{
	{
	Name: "ignition-firstboot-complete.service",
	State: osbuild.StateEnable,
	},
	{
	Name: "coreos-ignition-write-issues.service",
	State: osbuild.StateEnable,
	},
	{
	Name: "fdo-client-linuxapp.service",
	State: osbuild.StateEnable,
	},
	}