tests: Check that volume is functional after live migration. #256

markylaing · 2024-08-06T14:46:17Z

Changes are as follows:

Increases the memory of the nested VM to 1GiB (the parent VMs have 2GiB each).
Waits indefinitely for the nested VM to start up by checking the number of process via lxc info.
Formats the volume with ext4, mounts it, and adds a file.
Checks that the file is present and has the same contents after live migrating the instance.

tomponline

Thanks!

markylaing · 2024-08-06T16:14:08Z

Hmmm we'll it works locally but not in CI... what a surprise! 🤦

I'll have to investigate more tomorrow with tmate.

simondeziel · 2024-08-06T22:32:27Z

tests/vm-migration

-sleep 60
+
+# Wait for a long time for it to boot (doubly nested VM takes a while).
+while [ "$(lxc exec member1 -- sh -c "lxc info v1 | grep -F 'Processes:' | cut -d':' -f2 | tr -d '[:blank:]'")" -le 1 ]; do


I think the wrapper shell is not needed.

Suggested change

while [ "$(lxc exec member1 -- sh -c "lxc info v1 | grep -F 'Processes:' | cut -d':' -f2 | tr -d '[:blank:]'")" -le 1 ]; do

while [ "$(lxc exec member1 -- lxc info v1 | awk '{if ($1 == "Processes:") print $2}')" -le 1 ]; do

Also changed to use awk like we do elsewhere to get the process count.

With awk I think I need the subshell:

$ [ lxc exec member1 -- lxc info v1 | awk '{if ($1 == "Processes:") print $2}' -le 1 ] awk: fatal: cannot open file `-le' for reading: No such file or directory

Sorry for not being clear. With the subshell, I meant the sh -c bit used while doing the lxc exec.

simondeziel · 2024-08-06T22:34:38Z

tests/vm-migration

 lxc exec member1 -- lxc move v1 --target member2

+# The VM is slow. So the agent isn't immediately available after the live migration.
+while [ "$(lxc exec member2 -- sh -c "lxc info v1 | grep -F 'Processes:' | cut -d':' -f2 | tr -d '[:blank:]'")" -le 1 ]; do


Same note about subshell.

simondeziel · 2024-08-06T22:35:08Z

tests/vm-migration

+done
+
+# The volume should be functional, still mounted, and the file we created should still be there with the same contents.
+[ "$(lxc exec member2 -- sh -c "lxc exec v1 -- cat /mnt/vol1/bar")" = "foo" ]


This should work, I think:

Suggested change

[ "$(lxc exec member2 -- sh -c "lxc exec v1 -- cat /mnt/vol1/bar")" = "foo" ]

[ "$(lxc exec member2 -- lxc exec v1 -- cat /mnt/vol1/bar)" = "v1" ]

simondeziel · 2024-08-06T22:41:18Z

tests/vm-migration

+lxc exec member1 -- sh -c "lxc exec v1 -- mkfs -t ext4 /dev/sdb"
+lxc exec member1 -- sh -c "lxc exec v1 -- mkdir /mnt/vol1"
+lxc exec member1 -- sh -c "lxc exec v1 -- mount -t auto /dev/sdb /mnt/vol1"
+lxc exec member1 -- sh -c "lxc exec v1 -- sh -c 'echo foo > /mnt/vol1/bar'"


Could we do without the subshells and also make it explicit that we expect an ext4 formatted disk rather than accept anything:

Suggested change

lxc exec member1 -- sh -c "lxc exec v1 -- mkfs -t ext4 /dev/sdb"

lxc exec member1 -- sh -c "lxc exec v1 -- mkdir /mnt/vol1"

lxc exec member1 -- sh -c "lxc exec v1 -- mount -t auto /dev/sdb /mnt/vol1"

lxc exec member1 -- sh -c "lxc exec v1 -- sh -c 'echo foo > /mnt/vol1/bar'"

lxc exec member1 -- lxc exec v1 -- mkfs -t ext4 /dev/sdb

lxc exec member1 -- lxc exec v1 -- mkdir /mnt/vol1

lxc exec member1 -- lxc exec v1 -- mount -t ext4 /dev/sdb /mnt/vol1

lxc exec member1 -- lxc exec v1 -- cp /etc/hostname /mnt/vol1/bar

tomponline · 2024-08-07T08:16:42Z

ready for rebase

markylaing · 2024-08-07T09:14:49Z

I'm pretty sure that the doubly nested VM isn't booting because the CPU usage is at 100% for qemu:

I wonder if we can request a larger runner for this test?

simondeziel · 2024-08-07T12:26:48Z

tests/vm-migration

+  lxc launch "${TEST_IMG:-ubuntu-minimal-daily:24.04}" member1 --vm -c security.devlxd.images=true
+  lxc launch "${TEST_IMG:-ubuntu-minimal-daily:24.04}" member2 --vm -c security.devlxd.images=true
+else
+  lxc launch "${TEST_IMG:-ubuntu-minimal-daily:24.04}" member1 --vm
+  lxc launch "${TEST_IMG:-ubuntu-minimal-daily:24.04}" member2 --vm


The GHA runner VM have 16G of RAM (https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories) so you can probably allocate more RAM to those memberX VMs.

Have you already tried with -c limits.cpu=2 -c limits.memory=4GiB?

I've bumped all the resources to what is used in microcloud and it's still not booting after over an hour. I'll go back to trying with containers and see how I get on.

The while loop checking for v1's state seems to have a great overhead as it is not being executed every 30s as it should. Over time it drifted by many seconds. One possible way to reduce that overhead would be to reduce the polling frequency and have the loop done inside member1 rather than from the GHA runner itself.

In other words, replace the while :; do lxc exec member1 -- lxc info v1 ...; sleep 30; done by lxc exec member1 -- sh -c 'while :; do lxc info v1 ...; sleep 60; done'.

Another thing that could possibly help would be to delay member2 start till after v1 is confirmed booted on member1.

yes this would avoid needing to keep setting up the websockets over vsock for the exec sessions

tests/vm-migration

-sleep 60
+
+# Wait for a long time for it to boot (doubly nested VM takes a while).
+lxc exec member1 -- sh -c 'while [ "$(lxc info v1 | awk '"'"'{if ($1 == "Processes:") print $2}'"'"')" -le 1 ]; do echo "Instance v1 still not booted, waiting 60s..." && sleep 60; done'


tests/vm-migration

-sleep 60
+
+# Wait for a long time for it to boot (doubly nested VM takes a while).
+lxc exec member1 -- sh -c 'while [ "$(lxc info v1 | awk '"'"'{if ($1 == "Processes:") print $2}'"'"')" -le 1 ]; do echo "Instance v1 still not booted, waiting 60s..." && sleep 60; done'


tests/vm-migration

 lxc exec member1 -- lxc move v1 --target member2

+# The VM is slow. So the agent isn't immediately available after the live migration.
+lxc exec member1 -- sh -c 'while [ "$(lxc info v1 | awk '"'"'{if ($1 == "Processes:") print $2}'"'"')" -le 1 ]; do echo "Instance v1 still not booted, waiting 60s..." && sleep 60; done'


tests/vm-migration

 lxc exec member1 -- lxc move v1 --target member2

+# The VM is slow. So the agent isn't immediately available after the live migration.
+lxc exec member1 -- sh -c 'while [ "$(lxc info v1 | awk '"'"'{if ($1 == "Processes:") print $2}'"'"')" -le 1 ]; do echo "Instance v1 still not booted, waiting 60s..." && sleep 60; done'


simondeziel · 2024-08-21T14:38:27Z

@markylaing it seems the vm-migration test is using a dir pool. The microcloud test suite uses a zfs one which might make it faster to boot the many VMs without having to hit the disk as hard due to CoW taking place. Maybe it's worth trying zfs here as well?

Also, on microcloud, we use the extra/ephemeral disk available on GHA VMs, see https://github.com/canonical/microcloud/blob/main/.github/workflows/tests.yml#L202-L209

tests/vm-migration

tomponline · 2024-12-06T13:40:14Z

@markylaing @simondeziel any progress on using container cluster members for ceph backed VMs?

simondeziel · 2024-12-06T14:16:56Z

@tomponline it's in my queue for next pulse, but no progress as of yet.

markylaing requested review from tomponline and simondeziel as code owners August 6, 2024 14:46

markylaing force-pushed the live-migration-followup branch from a9e5ef9 to 6ef9d52 Compare August 6, 2024 14:50

tomponline previously approved these changes Aug 6, 2024

View reviewed changes

simondeziel reviewed Aug 6, 2024

View reviewed changes

markylaing dismissed tomponline’s stale review via b040de8 August 7, 2024 08:04

markylaing force-pushed the live-migration-followup branch from 6ef9d52 to b040de8 Compare August 7, 2024 08:04

markylaing force-pushed the live-migration-followup branch 2 times, most recently from 11eaed0 to 462a0c7 Compare August 7, 2024 08:51

simondeziel reviewed Aug 7, 2024

View reviewed changes

markylaing force-pushed the live-migration-followup branch 4 times, most recently from 30564cc to e7f3e43 Compare August 13, 2024 09:01

github-advanced-security bot found potential problems Aug 13, 2024

View reviewed changes

markylaing force-pushed the live-migration-followup branch 2 times, most recently from 3139e54 to 28ad7f1 Compare August 13, 2024 13:38

markylaing force-pushed the live-migration-followup branch 2 times, most recently from 13191d2 to 5e7488f Compare August 21, 2024 09:17

markylaing force-pushed the live-migration-followup branch 2 times, most recently from 60e2d21 to a4b95cf Compare August 22, 2024 13:47

github-advanced-security bot found potential problems Aug 22, 2024

View reviewed changes

tests/vm-migration Fixed Show fixed Hide fixed

markylaing force-pushed the live-migration-followup branch 4 times, most recently from 02f31e0 to 98a9a13 Compare August 22, 2024 14:35

markylaing force-pushed the live-migration-followup branch 3 times, most recently from 054e18d to e416b1d Compare August 28, 2024 09:46

WIP

93e3402

markylaing force-pushed the live-migration-followup branch from e416b1d to 93e3402 Compare August 28, 2024 09:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests: Check that volume is functional after live migration. #256

tests: Check that volume is functional after live migration. #256

markylaing commented Aug 6, 2024

tomponline left a comment

markylaing commented Aug 6, 2024

simondeziel Aug 6, 2024

markylaing Aug 7, 2024

simondeziel Aug 7, 2024

simondeziel Aug 6, 2024

simondeziel Aug 6, 2024

simondeziel Aug 6, 2024

tomponline commented Aug 7, 2024

markylaing commented Aug 7, 2024 •

edited

Loading

simondeziel Aug 7, 2024

markylaing Aug 8, 2024

simondeziel Aug 8, 2024

tomponline Aug 8, 2024

simondeziel commented Aug 21, 2024 •

edited

Loading

tomponline commented Dec 6, 2024

simondeziel commented Dec 6, 2024 •

edited

Loading

	while [ "$(lxc exec member1 -- sh -c "lxc info v1 \| grep -F 'Processes:' \| cut -d':' -f2 \| tr -d '[:blank:]'")" -le 1 ]; do
	while [ "$(lxc exec member1 -- lxc info v1 \| awk '{if ($1 == "Processes:") print $2}')" -le 1 ]; do

	[ "$(lxc exec member2 -- sh -c "lxc exec v1 -- cat /mnt/vol1/bar")" = "foo" ]
	[ "$(lxc exec member2 -- lxc exec v1 -- cat /mnt/vol1/bar)" = "v1" ]

tests: Check that volume is functional after live migration. #256

Are you sure you want to change the base?

tests: Check that volume is functional after live migration. #256

Conversation

markylaing commented Aug 6, 2024

tomponline left a comment

Choose a reason for hiding this comment

markylaing commented Aug 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tomponline commented Aug 7, 2024

markylaing commented Aug 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simondeziel commented Aug 21, 2024 • edited Loading

tomponline commented Dec 6, 2024

simondeziel commented Dec 6, 2024 • edited Loading

markylaing commented Aug 7, 2024 •

edited

Loading

simondeziel commented Aug 21, 2024 •

edited

Loading

simondeziel commented Dec 6, 2024 •

edited

Loading