Live mode boot slow, needs 4GB+ RAM #4

probonopd · 2020-11-01T15:40:08Z

Currently we need at least 4 GB of RAM for the Live system when the booted sytem in theory only needs <1. This is because everything gets copied into RAM at boot time.

I would like to remove the need for so much RAM, e.g., by loop-mounting system.uzip, and merging it with a tmpfs using unionfs, and then chrooting into that.

FuryBSD had a unionfs setup but it was more involved: https://github.com/furybsd/furybsd-livecd/tree/union-tar-nochroot

I'd like the simple version (loop-mount read-only filesystem, overlay it with tmpfs usion unionfs, chroot into that). If it doesn't work, then collaborate with unionfs kernel devs (who maintains unionfs? @mjguzik, @cemeyer?) to make it work.

The text was updated successfully, but these errors were encountered:

probonopd · 2020-11-08T08:37:33Z

Been experimenting in the 12.2-unionfs branch but it is not working.

ISO/overlays/ramdisk/init.sh

Line 33 in dea799d

    
           mdmfs -P -F /cdrom/data/system.uzip -o ro md.uzip /sysroot # FIXME: This does not seem to work; why?

fails.

Any help appreciated.

probonopd · 2020-12-10T21:00:21Z

Check what NomadBSD does, as it uses unionfs and hence boots the Live system quickly.
(Update: It doesn't seem to run from an ISO, hence it is unclear if it can work as a Live DVD.)

probonopd · 2020-12-27T22:15:45Z

My experiements in the pure-zfs-test branch resulted in a kernel panic so far:
https://github.com/helloSystem/ISO/releases/tag/13.0-pure-zfs-test

Tunas1337 · 2021-02-17T05:22:35Z

This seems like an issue that, if possible, really ought to be referenced in a little "echo" message in the bootloader, before the system attempts to load further. It's a source of confusion for a lot of people, and I'm sure there are people who have tested helloSystem, seen this, thought it was fundamentally broken and given up on it (myself included, had I not checked the GitHub issues!)
EDIT: alternatively, this issue could be pinned for easier discovery, as opposed to searching closed issues to find a reference to this. It'd also stop people from making issues related to it, thus saving time.

probonopd · 2021-02-17T07:58:57Z

Pinned the issue @Tunas1337

probonopd · 2021-02-27T12:03:48Z

Thanks to the help of @pkgdemon we have an experimental Live build that should address this issue:

https://github.com/helloSystem/ISO/releases/tag/continuous-12.2-unionfs

Note that this build contains fluxbox rather than helloSystem to make the ISO smaller (and because the 12.2-unionfs branch has diverged a lot from the current experimental branch of helloSystem), but it should be possible to port the changes over to the helloSystem builds in case everything turns out to work well.

So let's test, especially

Does it run on VMs with <4 GB RAM (akik on IRC confirmed that it boots with -m 512 in qemu)
How stable is it
autoloading of ums, if_iwm, if_iwn (see below)

Username: liveuser, no password

@pkgdemon noted on IRC:

There may be other
catches for unionfs such as devmatch not supporting kldload properly that I ran into. One thing that should be testing is to make sure autoloading of ums, if_iwm, if_iwn work with devmatch. In the past with ghostbsd it could also be observed that mate would lock up. I never ran into the lock up issues myself with FuryBSD. I think it just to be evaluated how well it works with Hello.
(...)
With unionfs you cannot reoot into the memory file system. You chroot. So what might end up happening is that certain low level utilities like kldload if_iwn might not be able to load a firmware like iwm8000Cfw because trying to use the real upper layer kldload /sysroot/boot/kernel/iwm8000Cfw.ko but
because kldload is chrooted it needs to do kldload /boot/kernel/iwm8000Cfw.ko is what i ran into.
(...)
Hopefully those low level issues have been resolved since I've last tried and maybe it would be a good solution for hellosystem.

probonopd · 2021-02-27T14:00:07Z

I was privately contacted by a helloSystem user who wishes to remain unnamed, and has summarized the issues as follows:

Splash screen doesn't always appear.

The whole system image needs to be decompressed before the user can start using the system. CPU utilization when decompressing system.uzip is pretty high, which can turn it into a bottleneck on computers with slower CPU's and/or can cause the CPU fan to spin at high RPM thus generating unpleasant noise.

The whole system image needs to be read into memory before the user can start using the system which takes time and depends on I/O speed.

The whole system image is kept in memory, which raises the minimum required RAM to 4 GB and reduces the amount of RAM available for normal operation of the system.

The user presented two solutions to problem 2 and noted that 3 and 4 need to be addessed separately and noted:

Both solutions drop uzip, which apart from solving problem 2 above also saves us from kernel panics and hangs caused by or related to uzip (see issue #102). It also makes building helloSystem ISO's quicker. For problems 3 and 4 above using unionfs is not the solution. See the BUGS section in mount_unionfs(8) if you wonder why.

Solution 1
Use as system.img the byte stream generated by 'zfs send'. It contains only the data from the source zpool, without any free disk space. When booting, you create a 3 GB memory disk, then create a zpool on that memory disk and then you use 'zfs recv' to fill in the data. This solution is already in use by GhostBSD so you can take a look at their scripts.
The increase in size measured on one recent helloSystem .iso is 10.5% or 174 MB.
With Zstandard compression you will likely get even better results.

Solution 2
Have system.img be a disk image of the sole disk of a zpool which is 1.6 GB in size and is full of 1.6 GB of useful data. On boot, copy system.img to a memory disk (using simple dd; there is no need to treat it as a device and to use zfs send | zfs recv) to make it read-write. Then add a second memory disk to the pool which will contain all the free disk space in the pool. Care may need to be taken to avoid ZFS seeing two zpools with the same GUID's.

For problem 4 the use of swap space was suggested, which unfortunately does not work on truly read-only media such as DVD.

So the solutions suggested by the unnamed user appear to be (partial) alternatives to what @pkgdemon has implemented and what seems to address at least 3 and 4.

Probably all solutions come with specific pros and cons which I currently do not fully understand yet. So some experimentation will be needed to see what works best for helloSystem.

Thank you everyone sharing your ideas and code! 👍

probonopd · 2021-02-27T20:05:21Z

Porting @pkgdemon's commits to the experimental-unionfs branch that is up-to-date to the experimental branch I noticed the following:

Things get simpler in build.sh and in the ramdisk, simpler is better!
Build time on Cirrus CI increases from ~18 to ~25 minutes (possibly caused by the need to store the intermediate ufs image on non-tmpfs on Cirrus CI because we are running out of RAM otherwise)
The resulting ISO increases slightly

If we can get everything else to behave properly, those trade-offs are probably worth it.

Further results with the experimental-unionfs branch ISOs:

12.1-based and 12.2-based experimental-unionfs ISOs boot on Dell OptiPlex 780 with Nvidia GPU but the Nvidia driver is not loaded
They fail to boot to desktop on Intel-based Acer Revo RL85 and Intel-based Acer TravelMate B (is this related to of Intel GPU fails in 12.2-RELEASE #1?)
Acer TravelMate B has in boot log: iwm7265Dfw: Could not load firmware image, error 2 and iwm0: could not read firmware iwm7265Dfw (error 0) - see @pkgdemon's comment above - possibly we can work around this by adding some clever symlink from /sysroot/ to /?

#4 (comment)

probonopd · 2021-02-28T08:12:25Z

On Intel GPU machines I cannot boot helloSystem ISOs from the experimental-unionfs branch all the way to the desktop. Things get stalled shortly after Starting devd and Autoloading module: if_iwm.ko. https://github.com/helloSystem/ISO/releases/tag/experimental-unionfs-12.2

However, the non-helloSystem build from the 12.2-unionfs branch https://github.com/helloSystem/ISO/releases/tag/continuous-12.2-unionfs does boot on those machines, albeit without the proper GPU drivers because initgfx is not on that image.

Possibly we are running into a conflict with initgfx? (Need to find a way to disable it for testing)
Possibly it is something else.

@pkgdemon wrote in https://github.com/furybsd/furybsd-livecd/commit/29fb0a18db7e1d44613954b2b9561c0eb1575aec:

Also fixes low level utilities such as umount and kldload which do not operate well in chroot.

As a result this fixes devmatch.

Move early xorg config to reroot init

Maybe we are missing something.

Do you have any ideas @pkgdemon?

#4 (comment)

probonopd · 2021-02-28T11:21:49Z

This is what happens with https://github.com/helloSystem/ISO/releases/tag/experimental-unionfs-12.1 (commit 8544ac2) when booted with unset boot_mute on two machines with Intel GPUs, even tough we have initgfx_enable=NO.

Boot stalls just above the section with the red arrows.
The section with the red arrows is output of repeatedly pressing Ctrl-T.

On this machine (Acer Revo RL85) I get cmd: chmod 911 [ufs]. 911 - does that even exist?
On another machine (Acer Travelmate B) I get cmd: chmod 759 [ufs]

The very same ISO boots on a machine with Nvidia GPU (but not using the Nvidia driver). It may or may not be related to GPUs at all. More testing on different systems would be helpful.

crees · 2021-03-13T11:27:38Z

Just a thought, but we can probably get around the chroot/module issues by nullfs mounting /boot with all of the modules etc onto the chroot's /sysroot/boot, or wherever necessary. Obviously this needs doing before the chroot call.

probonopd · 2021-03-13T11:33:25Z

Worth a try @pkgdemon?

pkgdemon · 2021-03-16T02:30:40Z

Interesting. Normally devmatch starts after devd. Wonder what it is trying to chmod? Perhaps booting with verbose mode enabled as well would reveal more? Maybe try boot -v as well @probonopd.

kettle-7 · 2021-05-18T01:05:51Z

geom_rowr booted great, although because of an issue with my USB stick (I need to go down to the shop and get one that works properly) after about 10 minutes I got I/O errors. All. Of. The. Time.

But that's most likely only a problem that affects me, so it's great.

probonopd · 2021-05-18T20:29:18Z

after about 10 minutes I got I/O errors

Please try with another USB stick. It's surprising how many go bad relatively quickly, even brand ones...

probonopd · 2021-05-30T19:27:03Z

Note to self: Directly after booting, one can use find . -cmin -10 to get a list of files that were written to in the last 10 minutes (= since booting).

It looks like these things need to be writeable in order to boot into Xorg with graphics autodetection using initgfx (... = multiple files in this directory):

/
/var
/var/db
/var/db/entropy
/var/db/entropy/saved-entropy.1
/var/db/pkg
/var/db/pkg/local.sqlite
/var/db/fontconfig
/var/db/fontconfig/...
/var/db/zoneinfo
/var/log/...
/var/spool/lock
/var/spool/lock/clean_var
/var/msgs
/var/msgs/bounds
/var/tmp
/var/run
/var/run/dhclient
/var/run/dhclient/dhclient.wlan0.pid
/var/run/wpa_supplicant
/var/run/wpa_supplicant/wlan0
/var/run/wpa_supplicant/wlan0.pid
/var/run/clean_var
/var/run/ld-elf32.so.hints
/var/run/dbus
/var/run/dbus/system_bus_socket
/var/run/dbus/pid
/var/run/cups
/var/run/cups/certs
/var/run/cups/certs/0
/var/run/cups/cupsd.pid
/var/run/cups/cups.sock
/var/run/ConsoleKit
/var/run/ConsoleKit/database
/var/run/ld-elf.so.hints
/var/run/devd.pipe
/var/run/devd.seqpacket.pipe
/var/run/devd.pid
/var/run/webcamd.0.4.0.pid
/var/run/automount.state
/var/run/dsbdriverd.pid
/var/run/syslogd.sockets
/var/run/log
/var/run/logpriv
/var/run/syslog.pid
/var/run/automountd.pid
/var/run/autounmountd.pid
/var/run/dmesg.boot
/var/run/os-release
/var/run/utx.active
/var/run/avahi-daemon
/var/run/avahi-daemon/pid
/var/run/avahi-daemon/socket
/var/run/slim.pid
/var/run/cron.pid
/var/run/consolekit.pid
/var/run/user
/var/run/user/1000
/var/run/slim.auth
/var/run/moused.ums0.pid
/var/lib
/var/lib/xkb
/var/lib/dbus
/var/lib/dbus/machine-id
/var/lib/asmctl.conf
/var/initgfx_config.id
/boot
/boot/modules
/boot/modules/linker.hints
/boot/entropy
/tmp/...
/root
/root/.cache
/root/.cache/mesa_shader_cache
/root/.cache/mesa_shader_cache/index
/root/.cache/mesa_shader_cache/...
/usr/local/include
/usr/local/libdata/pkgconfig
/usr/local/share
/usr/local/share/doc
/usr/local/share/licenses
/usr/local/share/X11/xorg.conf.d
/usr/local/bin
/usr/local/lib
/usr/local/lib/xorg/modules/drivers
/usr/local/lib/xorg/modules/input
/usr/local/man/man1
/usr/local/man/man4
/usr/local/etc
/usr/local/etc/xdg/autostart
/usr/local/etc/rc.d
/usr/local/etc/pam.d
/usr/local/etc/X11/xorg.conf.d
/usr/local/etc/X11/xorg.conf.d/10-video-initgfx.conf
/usr/local/etc/printcap
/usr/local/sbin
/usr/local/var
/usr/local/var/localize
/usr/local/var/localize/include
/usr/home/liveuser/...
/sbin/mount
/sbin/mount_udf
/sbin/mount_cd9660
/sbin/mount_nullfs
/sbin/mdmfs
/sbin/mount_unionfs
/sbin/mount_mfs
/sbin/mount_fusefs
/sbin/mount_nfs
/sbin/mount_msdosfs
/etc
/etc/motd
/etc/X11
/etc/X11/xorg.conf
/etc/rc.conf
/etc/hostid
/etc/host.conf
/etc/localtime
/net
/entropy

probonopd · 2021-05-30T19:38:17Z

Issues with geom_rowr hello-0.5.0_acc54acd1e3ae51f6f2eaa177dbba47e9cd261c4-FreeBSD-12.2-amd64.iso:

Automounted devices do not show up on desktop (although they are mounted)
Touchpad only works after plugging in and unplugging a wired mouse

To be verified wether these things work correctly with the equivalent non-geom_rowr build 0E212.

probonopd · 2021-06-01T17:12:51Z

Tests show that geom_rowr is useful but may possibly be overkill for just getting the ISO booted into a graphical desktop and comes with its own caveats (such as the ones in the post directly above).

Need to check out how we can use /etc/rc.initdiskless to get enough of FreeBSD read-write so that we can boot into a graphical desktop without needing any extra kernel modules. Possibly we can use it to make the files/directories listed above r/w?

The book Absolute FreeBSD by Michael Lucas, describes it like this:

The diskless system checks the file /conf/base/etc/diskless_remount for a list of directories it should mount as memory filesystems. Without this file, no memory filesystems get created, and your diskless host shares a single read-only userland with all of the other diskless hosts. The diskless_remount file contains a list of filesystems to be remounted.

So /etc/rc.initdiskless was designed with systems booting from a NFS network server but we want to use it for booting from a local r/o medium and the question is, can we use /etc/rc.initdiskless for this purpose.

Does this mean that we can just create the file /conf/base/etc/diskless_remount with the content /etc, run /etc/rc.initdiskless, and expect to get a read-write version of /etc?

Apparently no:

FreeBSD% sudo sh /etc/rc.initdiskless 
mount_nfs: no <host>:<dirpath> nfs-name
mount_nfs /etc /conf/base/etc failed: dropping into /bin/sh

Looks like it is hardwired to use NFS whereas we do not want to use NFS because we are using a local rather than a network filesystem.

grahamperrin · 2021-08-22T17:54:44Z

#4 (comment)

… issue with my USB stick …

FWIW I habitually use StressDisk to test disks before using them for anything of value.

sysutils/stressdisk | https://lists.freebsd.org/pipermail/freebsd-questions/2021-August/294600.html

HDAT2 excels, but it's not available for FreeBSD.

probonopd · 2021-09-04T18:13:09Z

Looks like @mszoek has got unionfs based Live ISOs to work properly for Airyx. It boots way faster and needs less RAM than helloSystem does at the moment.

Are there downsides, such as no more automatic kernel module loading? (Still to be checked.)

mszoek · 2021-09-04T18:28:35Z

Looks like @mszoek has got unionfs based Live ISOs to work properly for Airyx. It boots way faster and needs less RAM than helloSystem does at the moment.

"Properly" remains to be seen :)

Are there downsides, such as no more automatic kernel module loading? (Still to be checked.)

I believe so from what we've seen so far. It also ran out of disk (overlay FS) running from the LiveCD overnight.

ghost · 2021-10-03T12:50:32Z

Are there any test builds of helloSystem v0.6.0 for RAM < 2 GB?

probonopd · 2021-10-03T14:16:55Z

No.

probonopd · 2021-11-05T18:03:17Z

Presenting:

The FreeBSD read-only Xorg challenge

We need to find a way to boot into a fully working Xorg desktop (including working initgfx) on a read-only system. This would greatly improve our Live ISO options, and would also be the foundation for a future image-based system where the operating system itself could stay inside a read-only image file.

My earlier experiments at achieving this resulted in a lot of failures and frustration, with countless half-working or not-working Live ISOs build.

So there is another approach about trying this.

Install helloSystem to hard disk on a spare disk for experimentation
Run sudo rm /var/initgfx_config.id to force initgfx to do the GPU autodetection at next boot
Run sudo zfs set readonly=on zroot
Reboot

You will notice that the system can no longer boot into a working Xorg desktop.

Now our task is to find a way of strategically creating unionfs, nullfs, tmpfs in the right places and/or copying/moving files around until we find a way to get into a graphical desktop again.

The following leads to an instant reboot:

sudo mount -t unionfs /tmp /var/log
# sudo mount -t unionfs /tmp /var/lib
# sudo mount -t unionfs /tmp /etc
startx

Why?

A working solution

__The following seems to work:

Change the localize_start() function in /usr/local/etc/rc.d/localize like this:

localize_start()
{

mount -t tmpfs tmpfs /tmp # We need more space than what is the default
mount -t tmpfs tmpfs /var/log

cp -R /usr/local/etc/X11/xorg.conf.d /tmp
mount -t tmpfs tmpfs /usr/local/etc/X11/xorg.conf.d
mv /tmp/xorg.conf.d/* /usr/local/etc/X11/

mount -t tmpfs tmpfs /usr/local/var # FIXME: localize
mkdir -p /usr/local/var/db /usr/local/var/lib

# FIXME: Do more elegantly; e.g., by creating a new home on the fly from skel
cp -R /usr/home /tmp
mount -t tmpfs tmpfs /usr/home
mv /tmp/home/* /usr/home/
rm -f /usr/home/user/.Xauthority*
chown -R user /home/user

/usr/local/sbin/localize

}

(Of course this should eventually go into its own rc file but putting it into the existing one was the quickest way for me to ensure it runs at the correct place within the rc order.)

With this in place, the mounts on my test system are looking like this (notice those nice read-only):

zroot/ROOT/default on / (zfs, local, noatime, read-only, nfsv4acls)
devfs on /dev (devfs, local, multilabel)
zroot/tmp on /tmp (zfs, local, noatime, nosuid, read-only, nfsv4acls)
zroot/var/audit on /var/audit (zfs, local, noatime, noexec, nosuid, read-only, nfsv4acls)
zroot/usr/ports on /usr/ports (zfs, local, noatime, nosuid, read-only, nfsv4acls)
zroot/var/crash on /var/crash (zfs, local, noatime, noexec, nosuid, read-only, nfsv4acls)
zroot on /zroot (zfs, local, noatime, read-only, nfsv4acls)
zroot/var/mail on /var/mail (zfs, local, read-only, nfsv4acls)
zroot/var/tmp on /var/tmp (zfs, local, noatime, nosuid, read-only, nfsv4acls)
zroot/usr/src on /usr/src (zfs, local, noatime, read-only, nfsv4acls)
zroot/usr/home on /usr/home (zfs, local, noatime, read-only, nfsv4acls)
zroot/var/log on /var/log (zfs, local, noatime, noexec, nosuid, read-only, nfsv4acls)
tmpfs on /var (tmpfs, local)
map -hosts on /net (autofs)
tmpfs on /tmp (tmpfs, local)
tmpfs on /tmp (tmpfs, local)
tmpfs on /var/log (tmpfs, local)
tmpfs on /usr/local/etc/X11/xorg.conf.d (tmpfs, local)
tmpfs on /usr/local/var/db (tmpfs, local)
tmpfs on /usr/home (tmpfs, local)

Now that we have a baseline to work from we should be able to experiment more, e.g., also with selectively using unionfs.

probonopd · 2021-11-07T10:41:37Z

Actually, now we have proof that it is possible tor run a graphical desktop from a fully read-only medium, we can rethink the whole Live ISO architecture.

Currently we use:

zfs
zfs compression
uzip compression
initramdisk
md ramdisk
reroot (or chroot)

Can we do without most of this and instead boot directly into a uzip? Should be doable according to its manpage:

 geom_uzip is not limited to supporting only md(4) images.	The image can
 also reside on a block device.  (For example, a disk, USB flash drive,
 DVD-ROM, etc).  The appropriate device node will appear with the .uzip
 suffix.
  
 geom_uzip allows mounting the root	file system from a compressed disk
 partition by setting the vfs.root.mountfrom tunable.  See loader.conf(5)
 for details.

Sounds exactly like what we need, doesn't it?
Will try this path now.

probonopd · 2021-11-07T11:38:56Z

Current line of thinking is to have an rc script very early in the boot process inside the compressed filesystem image that kicks off the "make writable the essential locations" process, so that no scripts/initial ramdisks outside of the compressed filesystem are needed to do trigger this. This allows us to just tell the kernel to boot from the compressed filesystem, and once that is booting, it will make parts of itself r/w "by itself".

Like this /etc/rc.d/live:

#!/bin/sh

# PROVIDE: live
# REQUIRE: sysctl
# BEFORE: hostid

. /etc/rc.subr

name="live"
desc="Prepare read-only filesystem for live boot"
stop_cmd=":"
start_cmd="live_start"

live_start()
{
	mount -t devfs devfs /dev
	mount -t tmpfs tmpfs /tmp
	mount -t tmpfs tmpfs /media
	mount -t tmpfs tmpfs /var/log
	cp -R /usr/local/etc/X11/xorg.conf.d /tmp
	mount -t tmpfs tmpfs /usr/local/etc/X11/xorg.conf.d # initgfx writes there
	mv /tmp/xorg.conf.d/* /usr/local/etc/X11/
	mount -t tmpfs tmpfs /usr/local/var # FIXME: localize
	mkdir -p /usr/local/var/db /usr/local/var/lib
	# FIXME: Do more elegantly; e.g., by creating a new home on the fly from skel
	cp -R /usr/home /tmp
	mount -t tmpfs tmpfs /usr/home
	mv /tmp/home/* /usr/home/
	rm -f /usr/home/liveuser/.Xauthority* 2>/dev/null || true
	chown -R liveuser /home/liveuser
	# TODO: Port furybsd-init-helper into preinit.sh
	/usr/local/bin/furybsd-init-helper 
}

load_rc_config $name
run_rc_command "$1"

Also, the compressed filesystem image must not contain unsuitable entries in /etc/fstab, so we clear it out.

/boot and the compressed filesystem image do not need to be on the same physical medium. For example, I am using one USB stick tor the bootable filesystem that holds /boot and another USB stick that holds the parition for the compressed filesystem image. (You do not need to do it this way. I just found it more convenient for development.)

By the way, a fast way to make changes on the compressed filesystem image is:

# Make changes to the compressed filesystem image by using unionfs on the dev machine
mount -o ro -t ufs  /dev/da1p1.uzip /mnt
mkdir -p /tmp/union
mount -t unionfs /tmp/union/ /mnt/etc/
echo "" > /mnt/etc/fstab
nano /mnt/etc/rc.d/live
chmod +x /mnt/etc/rc.d/live

# Create new compressed filesystem image
makefs -o label=live /home/user/Desktop/system.ufs /mnt  # label is later passed to the kernel for booting
mkuzip -o /home/user/Desktop/system.uzip /home/user/Desktop/system.ufs
umount /mnt/etc/
umount /mnt/

# Write new compressed filesystem image to dedicated disk (allowing for a quick development cycle)
umount /dev/da1*
/sbin/gpart destroy -F da1
/sbin/gpart create -s GPT da1
BYTES=$(ls -la /home/user/Desktop/system.uzip | awk '{print $5}')
/sbin/gpart add -t freebsd-ufs -s ${BYTES}b da1
dd if=/home/user/Desktop/system.uzip of=/dev/da1p1 bs=4M status=progress

...and yes, I can boot into a graphical desktop like this. No ramdisks! No reroot! No chroot! In fact, no files on the bootable medium outside of /boot.

Here is the running system:

FreeBSD% mount
/dev/da1p1.uzip on / (ufs, local, read-only)
devfs on /dev (devfs, local, multilabel)
tmpfs on /tmp (tmpfs, local)
tmpfs on /var/log (tmpfs, local)
tmpfs on /usr/local/etc/X11/xorg.conf.d (tmpfs, local)
tmpfs on /usr/local/var (tmpfs, local)
tmpfs on /usr/home (tmpfs, local)
tmpfs on /var (tmpfs, local)
map -hosts on /net (autofs)
<above>:/tmp on /etc (unionfs, local)

For the last line, I cheated a bit by using a little bit of unionfs because /etc apparently needs to be writable so that we can use the (wireless) network. So once the system was fully running, I ran sudo mount -t unionfs /tmp /etc. If the unionfs proves to cause stability issues, we can find some other trickery around it.

Which brings me to the overall question: How much less stable will everything become if we replace the tmpfs usage with unionfs?

Only one way to find out: Try it out. Which is now pretty easy given that we have a neat procedure for experimentation.

probonopd · 2021-11-07T14:44:03Z

For even faster development/debugging cycles it would be neat if we could made changes without having to re-spin the compresssed filesystem image all the time, right?

Using this /etc/rc.d/live:

#!/bin/sh

# PROVIDE: live
# REQUIRE: sysctl
# BEFORE: hostid

. /etc/rc.subr

name="live"
desc="Prepare read-only filesystem for live boot"
stop_cmd=":"
start_cmd="live_start"

live_start()
{
	if [ -e "/dev/da0p2" ] ; then # FIXME: Use label instead or get the bootloader device somehow
		mount -o ro /dev/da0p2 /mnt # FIXME: Use label instead or get the bootloader device somehow
		if [ -e "/mnt/boot/bootscript" ] ; then
			. /mnt/boot/bootscript
		fi
		umount /mnt
		exit 0
	fi
}

load_rc_config $name
run_rc_command "$1"

we can now have our actual script outside of the compressed filesystem image in /boot/bootscript, allowing for very rapid modification:

mount -t devfs devfs /dev
mount -t tmpfs tmpfs /tmp
mount -t tmpfs tmpfs /media
mount -t tmpfs tmpfs /var/log
cp -R /usr/local/etc/X11/xorg.conf.d /tmp
mount -t tmpfs tmpfs /usr/local/etc/X11/xorg.conf.d # initgfx writes there
mv /tmp/xorg.conf.d/* /usr/local/etc/X11/
mount -t tmpfs tmpfs /usr/local/var # FIXME: localize
mkdir -p /usr/local/var/db /usr/local/var/lib
# FIXME: Do more elegantly; e.g., by creating a new home on the fly from skel
cp -R /usr/home /tmp
mount -t tmpfs tmpfs /usr/home
mv /tmp/home/* /usr/home/
rm -f /usr/home/liveuser/.Xauthority* 2>/dev/null || true
chown -R liveuser /home/liveuser
# TODO: Port furybsd-init-helper into preinit.sh
/usr/local/bin/furybsd-init-helper

This means we have a fully read-only filesystem image but we can make changes to practically everything without having to go through the timely read-only filesystem image generation process during the development cycle. I like it!

The development cycle is now:

Boot into the live system
Edit /boot/bootscript
Reboot

Very fast.

TODO: Replace the hardcode /dev/da0p2 with the output of kenv loaddev (= the disk from with the bootloader was loaded). The bootloader calls the same disk disk1p2: so we need to translate it (maybe need to ping kevans91).

probonopd · 2021-11-07T22:07:38Z

(the one unionfs in the screenshot will go away; it's only being used to monkey-patch the system on-the-fly)

With a lovingly crafted /boot/bootscript that does all kinds of trickery, we can optimize boot time to a fully functional desktop significantly.

Benchmark:

Hardware	Old* boot time**	New boot time**	0.6.0 free RAM	New free RAM
Acer TravelMate B from USB stick	2:44	0:47	1280 M	3105 M
Acer Revo RL 85 from USB stick		0:42

*0.6.0
**from OEM logo to desktop

Fwiw, Fedora 35 live system takes 1:08 under the same conditions.

probonopd · 2021-11-08T02:37:33Z

Now the question is, how do we get the uzip partition onto a DVD-ROM image, in a way that the kernel finds it? Just appending the uzip to the end of the iSO9660 file won't allow the kernel to magically find it. We needs some sort of a partition table.

man geom_uzip is not exactly verbose here:

geom_uzip is not limited to supporting only md(4) images. The image can
also reside on a block device. (For example, a disk, USB flash drive,
DVD-ROM, etc.)

Probably we should start with https://github.com/furybsd/furybsd-livecd/blob/master/scripts/mkisoimages-amd64.sh. and extend the mkimg call so that the partition table contains one additional entry for a partition that starts where the rest of the ISO ends. And then appending the uzip to the end of the ISO?

But how exactly?

Experiments:
https://gist.github.com/probonopd/3542af18b447c7231be0bf8571d3ac15

Edit: Even after a lot of experimentation I couldn't figure it out. The geniuses working on https://github.com/freebsd/freebsd-src/blob/main/tools/boot/install-boot.sh are probably the ones to ask about how to put together an ISO that also contains a valid ufs/uzip partition - that script is quite the wizardry!

probonopd · 2021-11-09T18:44:58Z

Live system development methodology

Here I am descibing the development process I just went through recently; mainly as a reference for my future self but also for anyone interested in tinkering with FreeBSD Live systems in general. Eventually the information from this ticket should be cleaned up and rewritten as developer documentation.

If you don't start from scratch (as I did) but want to make changes/improvements, you can skip step 1.

Stage 1: Make installed system read-only and bootable

The timesaver trick to speed up the development-test cycle in this stage is is to start from a working system that can be toggled between being read-write and read-only

Install system to hard disk
Modify system to be read-only (sudo zfs set readonly=on zroot)
Reboot, see where it fails
Make modifications to the system (it is not really read-only after all...), retry
Write one(!) rc script that you run as one of the very first rc scripts that makes the necessary locations writeable
Move on to the next stage once you have one(!) rc script that can boot a r/o system

Example for stage 1: #4 (comment)

Stage 2: Make compressed filesystem image bootable, without ISO

The timesaver trick to speed up the development-test cycle in this stage is to have an rc script on the compressed filesystem that runs as one of the very first rc scripts, and that executes code from outside of the compressed filesystem image.

Move the workload code out of the rc script into a shell script which the rc script merely runs. This is so that you can modify this code while the rest of the system is running from the compressed filesystem image
Put /boot (and possibly /rescue) onto a bootable ufs partition (e.g., on a USB stick)
Create a compressed filesystem image (uzip), this is really, really read-ony. Put that either as an image file onto the same ufs partition or onto a different partition or even a different device (as a partition or with raw dd)
Get this combination to boot into the desktop and perform as you like
Note that you can edit the bootscript interactively from within the booted system. This is a great productivity booster, allowing for quick iterations. No ISO to be re-generated, etc. Just edit, reboot.
Move on to the next stage once everything performs like you want it to.

No chroot/reroot is needed since we can get the kernel to use vfs.root.mountfrom="ufs:/dev/ufs/live" if the (uzip-compressed) ufs filesystem with the label "live" resides on a partition that the kernel can see.

If you don't need booting from CD-ROM/DVD-ROM, then you can stop here. But we want a DVD-ROM. Hence we need to use ISO (or at least this is what I assume).

Example for stage 2: #4 (comment)

Stage 3: Using ISO

The timesaver trick to speed up the development-test cycle in this stage is to have an rc script on the compressed filesystem that runs as one of the very first rc scripts, and that executes code from outside of the compressed filesystem image (since it is still inside the ISO, we need to respin the ISO each time).

Create ISO that holds /boot and the compressed filesystem image
Possibly have to adjust how things get loaded - e.g., we cannot directly boot but have to reroot/choot because the kernel seemingly cannot load uzip files that are on a ISO9660 filesystem directly. -- https://wiki.freebsd.org/AndriyGapon/AvgLiveCD The ability to specify cd9660:/dev/iso9660/MYISO/myimage.uzip in order for the kernel to boot from /myimage.uzip residing on a ISO9660 with the label MYISO (without copying it to RAM first) would make for a great feature (FreeBSD WISHLIST). What would it require? Essentially "just" telling it to load the uzip from ISO9660 at a certain offset that the ISO9660 filesystem "knows".
Test ISO in qemu: sudo qemu-system-x86_64 -cdrom '/home/user/Desktop/system.iso' -m 2048. qemu has a nice feature to halt execution, this is very useful to look at messages that are normally scrolling away too quickly
Rinse and repeat. This time we need to regenerate an ISO each time. Luckily if we do not touch the compressed filesystem, then ISO generation itself is very fast
Write to ISO USB and boot from there
Burn to DVD

We cannot do without chroot/reroot since I was not able to create an ISO that also contains a partition that the kernel can see, can can use with vfs.root.mountfrom="ufs:/dev/ufs/live" like above. Is it possible?

We cannot do without chroot (using reroot) because apparently the kernel cannot boot into loop-mounted filesystem images without copying them into RAM entirely, something we want to avoid at all cost. https://wiki.freebsd.org/AndriyGapon/AvgLiveCD seems to support that notion. The ability for the kernel to reroot into loop-mounted filesystem images would make for a great feature (FreeBSD WISHLIST).

So, unless we find a way to put a uzip partition on a (hybrid) ISO (which I still think might be feasible - I just couldn't find a way to do it), and unless any of my FreeBSD WISHLIST items are fulfilled, we will have to live with a chroot based Live ISO.

What is the downside of a chroot based system? mount shows "wrong" mount points (it takes the perspective "outside" of the chroot rather than "inside" the chroot). This may lead to bugs in code that doesn't expect this, e.g., if_iwm.ko and related modules trying to load firmware.

(But as soon as we drop the requirement of using ISO9660, then we can boot into the live system either from a partition by the virtues of using ufs:/dev/ufs/live.)

probonopd · 2021-11-09T20:38:35Z

For stage 3, this script can be used to produce an ISO:

#!/bin/sh

set -e

CD_ROOT=/media/ufs # Path to what will be contained in ISO9660 (input)
UZIP_PATH=/home/user/Desktop/system.uzip # Path to existing uzip image (input)
ISO_PATH=/home/user/Desktop/system.iso # Path to resulting ISO image (output)

if [ ! -e "${CD_ROOT}/boot/cdboot" ] ; then
  echo "${CD_ROOT}/boot/cdboot is missing, exiting"
  exit 1
fi

if [ ! -e "${CD_ROOT}/boot/isoboot" ] ; then
  echo "${CD_ROOT}/boot/isoboot is missing, exiting"
  exit 1
fi

if [ ! -e "${CD_ROOT}/boot/pmbr" ] ; then
  echo "${CD_ROOT}/boot/pmbr is missing, exiting"
  exit 1
fi

dd 'if=/dev/zero' 'of=efiboot.img' 'bs=1k' 'count=2048'
esp_md=$(mdconfig -a -t vnode -f efiboot.img)
newfs_msdos -F 12 -m 0xf8 -L ESP /dev/${esp_md}
mkdir efi
mount -t msdosfs /dev/${esp_md} efi
mkdir -p efi/efi/boot
uzip_md=$(mdconfig -a -t vnode -f "${UZIP_PATH}")
mkdir uzip
kldstat 2>/dev/null | grep geom_uzip >/dev/null 2>&1 || kldload geom_uzip
mount -o ro -t ufs /dev/${uzip_md}.uzip uzip
cp -p ${uzip}/boot/loader.efi efi/efi/boot/bootx64.efi
cp -p ${uzip}/boot/cdboot .
umount efi
rm -r efi
mdconfig -d -u ${esp_md}
umount uzip
rm -r uzip
mdconfig -d -u ${uzip_md}

makefs -t cd9660 \
-o 'bootimage=i386;cdboot' \
-o no-emul-boot \
-o 'bootimage=i386;efiboot.img' \
-o no-emul-boot  \
-o 'platformid=efi' \
-o rockridge \
-o 'label=ISOLBL' \
-o 'publisher=PUBLSHR'  \
${ISO_PATH} \
${CD_ROOT}

rm -f efiboot.img
rm -f cdboot

for entry in $(etdump --format shell ${ISO_PATH}); do
    eval ${entry}
    if [ "$et_platform" = "efi" ]; then
        esp_start=$(expr $et_lba \* 2048)
        esp_size=$(expr $et_sectors \* 512)
        esp_param="-p efi::${esp_size}:${esp_start}"
        break
    fi
done
    
iso_size=$(stat -f %z ${ISO_PATH})
echo iso_size: ${iso_size}

mkimg -v \
-s gpt \
--capacity ${iso_size} \
-b "${CD_ROOT}/boot/pmbr" \
${esp_param} \
-p "freebsd-boot:=${CD_ROOT}/boot/isoboot" \
-o hybrid.img

dd if=hybrid.img of="${ISO_PATH}" bs=32k count=1 conv=notrunc

readlink -f "${ISO_PATH}"
ls -lh "${ISO_PATH}"

probonopd · 2021-11-11T21:29:04Z

So, unless we find a way (...) we will have to live with a chroot based Live ISO.

Or so I thought. Until it dawned upon me that the official FreeBSD Live ISO https://download.freebsd.org/ftp/releases/ISO-IMAGES/13.0/FreeBSD-13.0-RELEASE-amd64-bootonly.iso does not use a chroot neither reroot at all. It simply boots the files that are on the ISO9660 filesystem. Now, that won't quite work for as as we want that fancy uzip compression, but who says that we can't mount all the individual directories beneath / from the mounted uzip into the running system?

And Voilà! No chroot needed, no reroot needed, and still getting to use uzip on an ISO. Downside: Can't create new files or directories in / at runtime, but imho that's an acceptable price to pay for now.

probonopd · 2021-11-12T06:35:08Z

In the tmpfs branch we have the new Live ISO methodology. Experimental builds of that branch are available based on FreeBSD 12.2 and 13.0.

Remaining showstopper

Does not boot all the way into the desktop, due to overly aggressive boot time optimization. Xorg and/or slim get started before something that they apparently need. We need to find out what that is.

Workaround

Press Alt-F2 to get to a login console
Log in using the username liveuser
sudo killall -9 Xorg
sudo service slim restart

probonopd added good first issue Good for newcomers help wanted Extra attention is needed labels Nov 8, 2020

probonopd changed the title ~~Test unionfs version~~ Make Live system using FreeBSD kernel unionfs Nov 8, 2020

probonopd added a commit that referenced this issue Nov 22, 2020

Workaround for #4

d5edb40

probonopd mentioned this issue Nov 26, 2020

i686 builds #22

Open

This was referenced Jan 2, 2021

Consider making non-ISO USB images using 'zpool checkpoint' #81

Open

Investigate VirtFS/9p for the Live system #83

Open

Shows a non-graphical console on a virtual machine #18

Closed

probonopd changed the title ~~Make Live system using FreeBSD kernel unionfs~~ (Live mode boot slow, needs 4GB+ RAM) Make Live system using FreeBSD kernel unionfs Feb 11, 2021

This was referenced Feb 11, 2021

No start in macbook 2009 helloSystem/hello#114

Closed

Live USB not working on Dell E4300 helloSystem/hello#107

Closed

The system keep locked to this point helloSystem/hello#115

Closed

probonopd pinned this issue Feb 17, 2021

konfou mentioned this issue Feb 18, 2021

"Enter full pathname of shell or RETURN for /bin/sh" on VirtualBox #156

Closed

probonopd added a commit that referenced this issue Feb 27, 2021

Try workaround for low-level tools trying to load things from /sysroot

f325f99

#4 (comment)

probonopd referenced this issue Feb 28, 2021

initgfx_enable="NO" for deugging (temporary)

f2bdc5b

probonopd added a commit that referenced this issue Feb 28, 2021

Try workaround for low-level tools trying to load things from /sysroot

8544ac2

#4 (comment)

probonopd mentioned this issue May 16, 2021

Food for thought: Using zfs-receive for updates? helloSystem/hello#167

Open

probonopd mentioned this issue Jul 22, 2021

please make it boot on 2gb ram #259

Closed

probonopd mentioned this issue Aug 11, 2021

Live USB enhancement: Indicate start up progress, otherwise user confused that system hung #268

Closed

probonopd mentioned this issue Aug 24, 2021

0.3pre: Mouse does not work ravynsoft/ravynos#98

Closed

probonopd mentioned this issue Oct 26, 2021

ISO Won't boot on metal nor VM #290

Closed

This comment has been minimized.

Sign in to view

probonopd mentioned this issue Nov 13, 2021

Switch to non-zfs, non-chroot, non-reroot Live ISO architecture #297

Merged

probonopd closed this as completed in #297 Nov 13, 2021

probonopd mentioned this issue Dec 1, 2021

Experimental Live ISO: 'pkg remove' reboots the system #303

Closed

Live mode boot slow, needs 4GB+ RAM #4

Live mode boot slow, needs 4GB+ RAM #4

Comments

probonopd commented Nov 1, 2020 • edited Loading

probonopd commented Nov 8, 2020

probonopd commented Dec 10, 2020 • edited Loading

probonopd commented Dec 27, 2020

Tunas1337 commented Feb 17, 2021 • edited Loading

probonopd commented Feb 17, 2021

probonopd commented Feb 27, 2021 • edited Loading

probonopd commented Feb 27, 2021 • edited Loading

probonopd commented Feb 27, 2021 • edited Loading

probonopd commented Feb 28, 2021 • edited Loading

probonopd commented Feb 28, 2021 • edited Loading

crees commented Mar 13, 2021

probonopd commented Mar 13, 2021

pkgdemon commented Mar 16, 2021 • edited Loading

kettle-7 commented May 18, 2021

probonopd commented May 18, 2021

probonopd commented May 30, 2021 • edited Loading

probonopd commented May 30, 2021 • edited Loading

probonopd commented Jun 1, 2021

grahamperrin commented Aug 22, 2021

probonopd commented Sep 4, 2021 • edited Loading

mszoek commented Sep 4, 2021

ghost commented Oct 3, 2021

probonopd commented Oct 3, 2021

probonopd commented Nov 5, 2021 • edited Loading

The FreeBSD read-only Xorg challenge

A working solution

probonopd commented Nov 7, 2021 • edited Loading

probonopd commented Nov 7, 2021 • edited Loading

probonopd commented Nov 7, 2021 • edited Loading

probonopd commented Nov 7, 2021 • edited Loading

probonopd commented Nov 8, 2021 • edited Loading

This comment has been minimized.

This comment has been minimized.

probonopd commented Nov 9, 2021 • edited Loading

Live system development methodology

Stage 1: Make installed system read-only and bootable

Stage 2: Make compressed filesystem image bootable, without ISO

Stage 3: Using ISO

probonopd commented Nov 9, 2021

probonopd commented Nov 11, 2021 • edited Loading

probonopd commented Nov 12, 2021 • edited Loading

Remaining showstopper

Workaround

probonopd commented Nov 1, 2020 •

edited

Loading

probonopd commented Dec 10, 2020 •

edited

Loading

Tunas1337 commented Feb 17, 2021 •

edited

Loading

probonopd commented Feb 27, 2021 •

edited

Loading

probonopd commented Feb 27, 2021 •

edited

Loading

probonopd commented Feb 27, 2021 •

edited

Loading

probonopd commented Feb 28, 2021 •

edited

Loading

probonopd commented Feb 28, 2021 •

edited

Loading

pkgdemon commented Mar 16, 2021 •

edited

Loading

probonopd commented May 30, 2021 •

edited

Loading

probonopd commented May 30, 2021 •

edited

Loading

probonopd commented Sep 4, 2021 •

edited

Loading

probonopd commented Nov 5, 2021 •

edited

Loading

probonopd commented Nov 7, 2021 •

edited

Loading

probonopd commented Nov 7, 2021 •

edited

Loading

probonopd commented Nov 7, 2021 •

edited

Loading

probonopd commented Nov 7, 2021 •

edited

Loading

probonopd commented Nov 8, 2021 •

edited

Loading

probonopd commented Nov 9, 2021 •

edited

Loading

probonopd commented Nov 11, 2021 •

edited

Loading

probonopd commented Nov 12, 2021 •

edited

Loading