-
Notifications
You must be signed in to change notification settings - Fork 0
Live migration of OSv guests using qemu
Ingredients:
- 1 Linux host (Ubuntu 14.10 in my case; Fedora should work too)
- 1 virtual bridge (osvbr0; see below) running DHCP
- 2 instances of qemu (one active and one stand-by, that we'll be migrating to) attached to the virtual bridge
- 1 API client that connects to the active qemu instance via the virtual bridge
Diagrams:
+------------------------------------------------------------+
| |
| +-----------------+ +-----------------+ |
| | | | | |
| | OSv1 | | OSv2 | |
| | (runs cli) | --> | (standby) | |
| | | | listens on 4444 | |
| +-----------------+ +-----------------+ |
| | 192.168.123.10 | |
| | | |
| +----------------------------------------------------+ | before
| | osvbridge - 192.168.123.0/24 | | migration
| | | |
| +----------------------------------------------------+ |
| | |
| | 192.168.123.1 |
| +-----------------------+ |
| | API client | |
| +-----------------------+ |
| Ubuntu 14.10 (fvbrick) |
+------------------------------------------------------------+
+------------------------------------------------------------+
| |
| +-----------------+ +-----------------+ |
| | | | | |
| | OSv1 | | OSv2 | |
| | (idle) | | (runs cli) | |
| | | | | |
| +-----------------+ +-----------------+ |
| | | 192.168.123.10
| | | |
| +----------------------------------------------------+ | after
| | osvbridge - 192.168.123.0/24 | | migration
| | | |
| +----------------------------------------------------+ |
| | |
| | 192.168.123.1 |
| +-----------------------+ |
| | API client | |
| +-----------------------+ |
| Ubuntu 14.10 (fvbrick) |
+------------------------------------------------------------+
Preparation (all the paths are relative to the root of the OSv install directory):
-
Add the following lua script as 'modules/cli/commands/loop.lua':
local OptionParser = require('std.optparse') local socket = require('socket') local cmd = {} cmd.desc = [[infinite loop]] cmd.help = [[Usage: loop [-p period] Infinte loop.]] cmd.parser = OptionParser [[ loop Usage: loop infinite loop Options: -p, --period=[SECONDS] Loop interval (in seconds) ]] cmd.main = function(args) local args, opts = cmd.parser:parse(args) local period = 2 if opts.period then period = opts.period end local counter = 0 local prev_time = socket.gettime() while not cli_interrupted() do local date, status = osv_request({"os", "date"}, "GET") osv_resp_assert(status, 200) counter = counter + 1 local time = socket.gettime() io.write(date, ' - ', counter, ' - ', time, ' - ', (time - prev_time), '\n') prev_time = time socket.sleep(period) end end return cmd -
Build the standard OSv image as usual; at the same time also build the API cli:
make cd modules/cli && make cd - -
As 'root' (or using 'sudo'), set up (and check) the virtual bridge as follows (network, IP addresses and MAC addresses are arbitrary; if you change them, make sure you change them consistently in all the steps below):
brctl addbr osvbr0 ip addr add dev osvbr0 192.168.123.1/24 ip link set up dev osvbr0 ip addr show osvbr0 ip route show 192.168.123.0/24to allow 'qemu-bridge-helper' (used inside 'scripts/run.py' below) access to this new virtual bridge, add this line to '/etc/qemu/bridge.conf' - this step needs to be done only once, since this is a permanent file:
allow osvbr0 -
As 'root' (or using 'sudo') start a DHCP server just on the virtual bridge interface serving only IP address (192.168.123.10) to a specific MAC address (52:54:30:30:30:30), which will be the MAC address that qemu will use for the OSV instances:
/usr/sbin/dnsmasq --bind-interfaces --interface=osvbr0 --listen-address=192.168.123.1 --port=0 --no-resolv --dhcp-range=192.168.123.100,192.168.123.254 --dhcp-host=52:54:30:30:30:30,192.168.123.10,infinite -
In one terminal (terminal #1) start the first instance of qemu (the active one):
scripts/run.py -p kvm -c 1 -m 100 -n -b osvbr0 --mac 52:54:30:30:30:30 --novnc --nogdb(if everything is right, you should see that instance receive the IP address 192.168.123.10)
-
In another terminal (terminal #2) start the 'loop' script using the cli:
cd modules/cli && ./cli --api=http://192.168.123.10:8000 -- loop -p 1(if everything is right, you should see a line every second with the current time, a counter that starts from 1 and keeps increasing, the number of seconds returned by 'socket.gettime()', and the delta between two successive values)
-
In yet another terminal (terminal #3) start the second instance of qemu (the stand-by one) - please note that the space before ' -incoming' in this command is not a typo; the command will fail without that space due to the way Python argparse interprets its arguments:
scripts/run.py -p kvm -c 1 -m 100 -n -b osvbr0 --mac 52:54:30:30:30:30 --novnc --nogdb --pass-args ' -incoming tcp:0:4444'(nothing should happen here since this instance is currently in stand-by mode)
-
(Finally!) migrate the qemu active instance to the stand-by as follows:
-
in terminal #1 type: Control-A c (lowercase c) to be at the qemu command prompt (you should see the prompt '(qemu)')
-
at the '(qemu)' prompt type the actual migration command:
migrate "tcp:127.0.0.1:4444" -
within a second or so, the migration will complete; the cli terminal should continue running without interruption; you may see an abrupt change in the timestamp value, but the value returned by 'socket.gettime()' and the counter by one should work as expected, showing that the OSv guest state is correctly preserved during this live migration.
-
you can terminate the no longer used qemu instance (the one in terminal #1) by typing the command:
quitat the '(qemu)' prompt
-
-
(optional) you can play live migration ping pong and further migrate this OSv instance to another instance of qemu.
-
in terminal #1 start again qemu in stand-by mode:
scripts/run.py -p kvm -c 1 -m 100 -n -b osvbr0 --mac 52:54:30:30:30:30 --novnc --nogdb --pass-args ' -incoming tcp:0:4444' -
in terminal #3 (which after the previous step is now the active instance), go to the '(qemu)' prompt (by typing ^A c) and enter the migrate command:
migrate "tcp:127.0.0.1:4444" -
and then terminate that qemu instance with the command 'quit' (or ^A x)
-
the cli command (in terminal #2) should still continue running; at any moment you can also validate which one is the active instance by typing return in that terminal and you should see the standard '/#' prompt where you can type cli commands. You'll also see that if you terminate the active instance (with ^A x, for instance) the time/sequence progression in the cli will stop immediately.
-
Enjoy!
References:
- KVM Migration - http://www.linux-kvm.org/page/Migration