SONiC to SONiC update

SONiC-to-SONiC update design

The scope SONiC-to-SONiC update feature is to [TODO: paraphrase] provide the possibility to update the switch software of running switch without having to reboot the switch and interrupt the traffic flow.

Overview

S-2-S update in general will include three phases:

1. Download new software
2. Execute update (while switch continues to run the old software and process traffic)
3. Reboot to activate the new software

Switch software is all included into one file called ONE Image. Download could be done in any available way (wget, scp, etc).
To run the update the downloaded file needs to be executed. For example:

sh sonic-image.bin  

Reboot will not happen automatically after the update. User can do the manual reboot after the update or schedule it for any time. After reboot the new software will be loaded by default while the previous software will still remain on the switch as a fail over backup [TODO: review last sentence].
Note: there should be available writable file system with the enough free space to hold the new SONiC image and some spare space to unpack the image [TODO: Calculate and specify rough free space expectations].

General approach

• Update will be initiated by running the SONiC ONE Image on the switch
• Update could be initiated on either switch running SONiC or in ONIE Rescue console
• During and after the update switch can run normally. New image will be loaded only after reboot.
• Reboot will not be invoked by the update scripts
• Number of the recent installations kept on device: 2
• Each installation will reside on the /dev/sda3 in the separate directory. Directory name: “image-xxx” (where xxx is the build hash or SONiC version+hash – TBD)
• The most recent installation will be marked in GRUB to load by default. The oldest or not running one (if total count is 2) will be removed from GRUB and from the disk.

Technical details

Directories and files

SONiC version specific files and directories

The list of directories and files to be placed under the installation specific directory “image-xxx”

• [boot] – kernel, initrd, etc
• [docker] – docker filesystem
• [rw] – r/w part of the root file system
• [platform]
• fs.squashfs – r/o part of the root file system
• onie-support.tar.bz2

Common directories and files

• [disk-img] – directory holding file var-log.ext4 – a loop device for logs
• [grub] – SONiC GRUB submenu
• machine.conf

Mounts

• /image-xxx/fs.squashfs mounted to / (RO)
• /dev/sda3 to /host
• /image-xxx/rw mounted to / (RW)
• /image-xxx/docker to /var/lib/docker
• /image-xxx/boot to /boot
• /disk-img/val-log.ext4 to /var/log

Implementation

ONE Image installation process needs to be changed in order to implement the concept of SONiC-to-SONiC update.
The most of the installation processes is implemented in the install.sh script. This script is a part of the ONE Image package.

Changes need to be done in the installation script

Installation script is located at sonic-buildimage/installer/x86_64/install.sh
Currently installation always reformats the /dev/sda3 partition before unpacking its components. SONiC-to-SONiC update design assumes keeping two the most recent SONiC software installations for the fail over reason. To implement this the process of installation should include two modes:

1. Initial installation which can go as is now (format partition, etc) but unpack and copy SONiC components according to the new concept.
2. Incremental installation which will not format the partition but remove the oldest SONiC version instead and unpack itself next to the existing and currently running version.

Any of this installation modes will consist of two parts: prepare the file system and the actual copy, unpacking and enlisting in the GRUB.

Initial installation. Part 1

Actions to be taken:

• format /dev/sda3
• create basic version of the grub configuration
• create file and initialize log file system

Incremental installation. Part 1

Actions to be taken:

• determine whether there is a SONiC installation to be deleted and delete it. This assumes detecting how many sonic installations there are on device and which one is the candidate to delete. If there two installation, need to determine which one is not running now and delete it from the disk and from the GRUB menu. Deleting from disk is simple removing the installation-specific folder called "image-xxx", deleting from GRUB assumes finding and deleting menu item called SONiC-xxx from the /grub/grub.cfg.

Both installations. Part 2

In case of initial installation the "root" will be actually the root folder of the /dev/sda3 while in case of incremental installation the "root" is mounted to /host. In order to have a common code for both types of the installation, this difference should be handled in the install.sh script.
Actions to be taken:

• in the root create directory image-xxx, where xxx is the hash representing software revision (it was used previously in the GRUB entry.
• copy or unpack components listed in Chapter "Directories and files" to the image directory
• update grub menu located in {root}/grub/grub.cfg. Create new menu entry pointing to the new SONiC installation and update the default item to point to the new installation.

Changes need to be done to update mounts

Script located at sonic-buildimage/files/initramfs-tools/union-mount.j2 mounts aufs, docker and log file systems on the post boot stage of SONiC load. It needs to be updated to use current installation components which will now be located inside the version-specific directory. The full list of mounts are in the chapter [link| Mounts].
Maybe it will be needed to change slave.mk to let j2 template processor access the installation revision hash.

Open questions

1. Clarify: when installation while not the most recent version is running - remove the one which is not running now?
2. Add a switch to force initial installation. Something like sh sonic-image.bin -f. This can be used to make this installation the only one on the switch or to fix some compatibility or other issues when the incremental installation fails.
3. Option to skip making new installation default.