Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Refinement] boot_control.grub: improve robustness against boot load/init, split out ota_status control and slot mount logic #257

Merged
merged 48 commits into from
Sep 22, 2023
Merged
Show file tree
Hide file tree
Changes from 37 commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
b8b40b9
simplify GrubABPDetecter, simplify SymlinkABPDetecter to a function
Bodong-Yang Aug 31, 2023
4c65c27
grub: use OTAStatusFilesControl and SlotMountHelper instead
Bodong-Yang Aug 31, 2023
ecc5c52
grub: GrubControl now always check ota-partition files on startup
Bodong-Yang Aug 31, 2023
9a68dda
boot_control.common: minor update, define default version str
Bodong-Yang Aug 31, 2023
d2e1d3b
boot_control.common: split load_ota_status_files into 2 methods
Bodong-Yang Aug 31, 2023
cc4fae3
grub: _grub_update_for_active_slot should always use write_str_to_fil…
Bodong-Yang Sep 4, 2023
7e9ede9
grub: symlink to grub.cfg now prepared by _ensure_ota_partition_symlinks
Bodong-Yang Sep 5, 2023
c30cd74
grub: refine check_active_slot_ota_partition_file
Bodong-Yang Sep 5, 2023
c62e291
minor update
Bodong-Yang Sep 5, 2023
ee7076d
grub: refine _update_fstab
Bodong-Yang Sep 5, 2023
e5f1a6d
OTAStatusControl: add force_initialize
Bodong-Yang Sep 5, 2023
10c7ba7
grub: remove unused
Bodong-Yang Sep 5, 2023
417b6c9
grub: still make symlinks before update grub
Bodong-Yang Sep 5, 2023
24c4cc4
grub: get_current_booted_files now use standard way to detect booted …
Bodong-Yang Sep 5, 2023
9f5e296
grub: now grub control can recover from ota-partition symlink missing
Bodong-Yang Sep 5, 2023
b44ee1a
common: SlotMountHelper now takes standby_boot_dir param
Bodong-Yang Sep 5, 2023
67ad942
grub, rpi_boot: update according to SlotMountHelper's changes
Bodong-Yang Sep 5, 2023
4473516
boot_control_protocol: deprecate get_standby_boot_dir
Bodong-Yang Sep 7, 2023
d580a62
common: revert adding standby_boot_dir param in SlotMountHelper init
Bodong-Yang Sep 7, 2023
1203c97
rpi_boot: revert changes
Bodong-Yang Sep 7, 2023
edad48a
grub: remove standby_boot_dir param when init SlotMountHelper
Bodong-Yang Sep 7, 2023
50b13d8
grub: now grub controller copies boot files to ota-partition folder b…
Bodong-Yang Sep 7, 2023
57e229f
grub: logging the loaded original status and slot_in_use
Bodong-Yang Sep 7, 2023
6f5ae0f
grub: fix copy_boot_files_from_standby_slot
Bodong-Yang Sep 7, 2023
b82c6d9
grub: fix _check_active_slot_ota_partition_file
Bodong-Yang Sep 7, 2023
b27a21a
grub: _check_active_slot_ota_partition_file also check ota-partition …
Bodong-Yang Sep 7, 2023
39e6b9f
fix up test_grub
Bodong-Yang Sep 7, 2023
02e7285
test_grub: minor fix
Bodong-Yang Sep 7, 2023
d5e78bd
minor fix to test_grub
Bodong-Yang Sep 7, 2023
0024f6a
implement test_ota_status_control
Bodong-Yang Sep 8, 2023
63abab8
boot_control_protocol: rename get_ota_status to get_booted_ota_status…
Bodong-Yang Sep 8, 2023
12e812f
common.OTAStatusFilesControl: rename property ota_status -> booted_ot…
Bodong-Yang Sep 8, 2023
c98953b
update grub and rpi_boot control accordingly
Bodong-Yang Sep 8, 2023
2ba18d4
update otaclient.py accordingly
Bodong-Yang Sep 8, 2023
71bb18a
update test_rpi_boot and test_ota_client accordingly
Bodong-Yang Sep 8, 2023
2f9aa2e
boot_control.common: also update legacy OTAStatusMixin to meet protoc…
Bodong-Yang Sep 8, 2023
290a0b0
Merge branch 'main' into refine/grub_robust
Bodong-Yang Sep 13, 2023
88ba123
grub: fix _prepare_kernel_initrd_links_for_ota, now ensure kernel and…
Bodong-Yang Sep 14, 2023
12c5fa3
grub: for _grub_update, change some attrs in _GrubControl to func loc…
Bodong-Yang Sep 14, 2023
85acd39
grub: ensure_ota_partition_symlinks and ensure_standby_slot_boot_file…
Bodong-Yang Sep 14, 2023
c30c45f
grub: grub_update_on_booted_slot now also validate active boot file s…
Bodong-Yang Sep 14, 2023
1f1aab7
grub: update boot control init
Bodong-Yang Sep 14, 2023
b9b8a45
grub: aligns grub_reboot and finalize_switch_boot behavior as document
Bodong-Yang Sep 14, 2023
a6fc838
grub.GrubController: finalize_update_switch_boot doesn't need partial…
Bodong-Yang Sep 14, 2023
9b165fd
grub: minor updates
Bodong-Yang Sep 19, 2023
48c06ac
grub: fix typo detecter -> detector
Bodong-Yang Sep 22, 2023
ebb0c4c
grub: add comments fro grub_reboot_to_standby method
Bodong-Yang Sep 22, 2023
ccf67cc
fix test_grub
Bodong-Yang Sep 22, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 56 additions & 44 deletions otaclient/app/boot_control/_common.py
Original file line number Diff line number Diff line change
Expand Up @@ -424,8 +424,7 @@ def reboot(cls):


class OTAStatusFilesControl:
"""Logics for controlling otaclient's behavior using ota_status files,
including status, slot_in_use and version.
"""Logics for controlling otaclient's OTA status with corresponding files.

OTAStatus files:
status: current slot's OTA status
Expand All @@ -444,56 +443,38 @@ def __init__(
current_ota_status_dir: Union[str, Path],
standby_ota_status_dir: Union[str, Path],
finalize_switching_boot: FinalizeSwitchBootFunc,
force_initialize: bool = False,
) -> None:
self.active_slot = active_slot
self.standby_slot = standby_slot
self.current_ota_status_dir = Path(current_ota_status_dir)
self.standby_ota_status_dir = Path(standby_ota_status_dir)
self.finalize_switching_boot = finalize_switching_boot

# NOTE: pre-assign live ota_status with the loaded ota_status,
# and then update live ota_status in below.
# The reason is for some platform, like raspberry pi 4B,
# the finalize_switching_boot might be slow, so we first provide
# live ota_status the same as loaded ota_status(or INITIALIZED),
# then update it after the init_ota_status_files finished.
_loaded_ota_status = self._load_current_status()
self._ota_status = (
_loaded_ota_status if _loaded_ota_status else wrapper.StatusOta.INITIALIZED
)

_loaded_slot_in_use = self._load_current_slot_in_use()
if _loaded_slot_in_use and _loaded_slot_in_use != self.active_slot:
logger.warning(
f"boot into old slot {self.active_slot}, "
f"but slot_in_use indicates it should boot into {_loaded_slot_in_use}, "
"this might indicate a failed finalization at first reboot after update/rollback"
)

# initializing ota_status control
self._init_ota_status_files()
self._force_initialize = force_initialize
self.current_ota_status_dir.mkdir(exist_ok=True, parents=True)
self._load_slot_in_use_file()
self._load_status_file()
logger.info(
f"ota_status files parsing completed, ota_status is {self._ota_status}"
f"ota_status files parsing completed, ota_status is {self._ota_status.name}"
)

def _init_ota_status_files(self):
def _load_status_file(self):
"""Check and/or init ota_status files for current slot."""
self.current_ota_status_dir.mkdir(exist_ok=True, parents=True)

# load ota_status and slot_in_use file
_loaded_ota_status = self._load_current_status()
_loaded_slot_in_use = self._load_current_slot_in_use()
if self._force_initialize:
_loaded_ota_status = None

# initialize ota_status files if not presented/incompleted/invalid
if not (_loaded_ota_status and _loaded_slot_in_use):
if not _loaded_ota_status:
logger.info(
"ota_status files incompleted/not presented, "
"initializing and set/store status to INITIALIZED..."
f"initializing and set/store status to {wrapper.StatusOta.INITIALIZED.name}..."
)
self._store_current_slot_in_use(self.active_slot)
self._store_current_status(wrapper.StatusOta.INITIALIZED)
self._ota_status = wrapper.StatusOta.INITIALIZED
return
logger.info(f"status loaded from file: {_loaded_ota_status.name}")

# status except UPDATING and ROLLBACKING(like SUCCESS/FAILURE/ROLLBACK_FAILURE)
# are remained as it
Expand All @@ -505,8 +486,11 @@ def _init_ota_status_files(self):
return

# updating or rollbacking,
# NOTE: pre-assign live ota_status with the loaded ota_status before entering finalizing_switch_boot,
# as some of the platform might have slow finalizing process(like raspberry).
self._ota_status = _loaded_ota_status
# if is_switching_boot, execute the injected finalize_switching_boot function from
# boot controller, transit the ota_status according to the execution result.
# boot controller, transit the ota_status according to the execution result.
# NOTE(20230614): for boot controller during multi-stage reboot(like rpi_boot),
# calling finalize_switching_boot might result in direct reboot,
# in such case, otaclient will terminate and ota_status will not be updated.
Expand All @@ -528,7 +512,7 @@ def _init_ota_status_files(self):
else:
logger.error(
f"we are in {_loaded_ota_status.name} ota_status, "
f"but {_loaded_slot_in_use=} doesn't match {self.active_slot=}, "
"but ota_status files indicate that we are not in switching boot mode, "
"this indicates a failed first reboot"
)
self._ota_status = (
Expand All @@ -538,6 +522,27 @@ def _init_ota_status_files(self):
)
self._store_current_status(self._ota_status)

def _load_slot_in_use_file(self):
_loaded_slot_in_use = self._load_current_slot_in_use()
if self._force_initialize:
_loaded_slot_in_use = None

if not _loaded_slot_in_use:
# NOTE(20230831): this can also resolve the backward compatibility issue
# in is_switching_boot method when old otaclient doesn't create
# slot_in_use file.
self._store_current_slot_in_use(self.active_slot)
return
logger.info(f"slot_in_use loaded from file: {_loaded_slot_in_use}")

# check potential failed switching boot
if _loaded_slot_in_use and _loaded_slot_in_use != self.active_slot:
logger.warning(
f"boot into old slot {self.active_slot}, "
f"but slot_in_use indicates it should boot into {_loaded_slot_in_use}, "
"this might indicate a failed finalization at first reboot after update/rollback"
)

# slot_in_use control

def _store_current_slot_in_use(self, _slot: str):
Expand Down Expand Up @@ -637,15 +642,11 @@ def pre_rollback_standby(self):
self._store_standby_status(wrapper.StatusOta.ROLLBACKING)

def load_active_slot_version(self) -> str:
_version = read_str_from_file(
return read_str_from_file(
self.current_ota_status_dir / cfg.OTA_VERSION_FNAME,
missing_ok=True,
default="",
default=cfg.DEFAULT_VERSION_STR,
)
if not _version:
logger.warning("version file not found, return empty version string")

return _version

def on_failure(self):
"""Store FAILURE to status file on failure."""
Expand All @@ -655,8 +656,14 @@ def on_failure(self):
self._store_standby_status(wrapper.StatusOta.FAILURE)

@property
def ota_status(self) -> wrapper.StatusOta:
"""Read only ota_status property."""
def booted_ota_status(self) -> wrapper.StatusOta:
"""Loaded current slot's ota_status during boot control starts.

NOTE: distinguish between the live ota_status maintained by otaclient.

This property is only meant to be used once when otaclient starts up,
switch to use live_ota_status by otaclient after otaclient is running.
"""
return self._ota_status


Expand Down Expand Up @@ -705,7 +712,7 @@ def _load_current_ota_status(self) -> Optional[wrapper.StatusOta]:
except KeyError:
pass # invalid status string

def get_ota_status(self) -> wrapper.StatusOta:
def get_booted_ota_status(self) -> wrapper.StatusOta:
return self.ota_status


Expand Down Expand Up @@ -788,7 +795,12 @@ def __init__(
self.standby_slot_mount_point.mkdir(exist_ok=True, parents=True)
self.active_slot_mount_point.mkdir(exist_ok=True, parents=True)
# standby slot /boot dir
self.standby_boot_dir = self.standby_slot_mount_point / "boot"
# NOTE(20230907): this will always be <standby_slot_mp>/boot,
# in the future this attribute will not be used by
# standby slot creater.
self.standby_boot_dir = self.standby_slot_mount_point / Path(
cfg.BOOT_DIR
).relative_to("/")

def mount_standby(self, *, raise_exc: bool = True) -> bool:
"""Mount standby slot dev to <standby_slot_mount_point>.
Expand Down
Loading