feat: split and run grpc_server, ota_core and otaclient main as separated processes

[Bodong-Yang]

Introduction

Note

This PR is the core PR and the final piece of the whole otaclient re-architecture project.

This PR introduces a re-architecture of otaclient, now at runtime each core otaclient components run as standalone processes and utilize IPC for handling OTA request:

1. main process(by main module): the daemon process that first sets up the queue and shared_memory for each components to work together, and then brings up each component processes one by one. During otaclient running life-cycle, it will monitor each components status(active or exited) and handle signals(SIGTERM, SIGINT) accordingly.
2. ota_core process (by ota_core module): the process running the core implementation of OTA. It handles the OTA request and actually do the OTA, while reports its status via shared_memory all the time.
3. grpc_server process (by otaclient.grpc.api_v2 package): the prcoess running OTA Service API grpc server, it handles the OTA API requests, and translates it into otaclient internal form and dispatches to ota_core process via IPC.
4. ota_proxy process (by ota_proxy package): the process running ota_proxy server, will only be brought up when there is active OTA within the cluster.

A simple otaclient internal IPC interface is implemented using simple types defined in otaclient._types with queue between API grpc server and ota_core. grpc_server dispatches OTA requests down to ota_core(with op_queue) and get the response to the request from ota_core(with ack_queue).

A cros-process status report mechanism is implemented based on sharing hmac-protected(with one-time preshared key) pickled object with shared_memory. ota_core uses this mechanism to write its latest status into the shm, and grpc_server process read this shm to track the ota_core's internal status, and translate it into the OTA grpc API format.

Docs

For the detailed design documentation, see https://tier4.atlassian.net/l/cp/vxCQ0qp0.

Tests

• otaclient can cleanly exit on SIGTERM(by kill) and SIGINT(by ctrl-c) without dangling.
  • at startup
  • during OTA
• OTA e2e test with VM passed.
• local pytest passed.

https://tier4.atlassian.net/browse/RT4-12276

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
src/ota_metadata/legacy
__init__.py	1	0	100%
parser.py	335	48	85%	106, 170, 175, 211–212, 222–223, 226, 238, 289–291, 295–298, 324–327, 396, 399, 407–409, 422, 431–432, 435–436, 601–603, 653–654, 657, 685–686, 689–690, 692, 696, 698–699, 753, 756–758
types.py	84	13	84%	37, 40–42, 112–116, 122–125
src/ota_metadata/utils
cert_store.py	86	8	90%	58–59, 73, 87, 91, 102, 123, 127
src/ota_proxy
__init__.py	15	6	60%	48, 50, 52, 61, 71–72
__main__.py	8	8	0%	16, 18–20, 22, 24–25, 27
_consts.py	17	0	100%
cache_control_header.py	68	4	94%	71, 91, 113, 121
cache_streaming.py	144	12	91%	225, 229–230, 265–266, 268, 280, 349, 367–370
config.py	20	0	100%
db.py	74	18	75%	110, 116, 154, 160–161, 164, 170, 172, 193–200, 202–203
errors.py	5	0	100%
external_cache.py	28	20	28%	31, 35, 40–42, 44–45, 48–49, 51–53, 60, 63–65, 69–72
lru_cache_helper.py	47	2	95%	84–85
ota_cache.py	234	64	72%	71–72, 143, 146–147, 159–160, 192–193, 210, 231, 250–254, 258–260, 262, 264–271, 273–275, 278–279, 283–284, 288, 335, 343–345, 418, 445, 448–449, 471–473, 477–479, 485, 487–489, 494, 520–522, 557–559, 586, 592, 607
server_app.py	141	36	74%	79, 82, 88, 107, 111, 170, 179, 221–222, 224–226, 229, 234–235, 238, 241–242, 245, 248, 251, 254, 272, 274, 277, 303–306, 309, 323–325, 331–333
utils.py	14	0	100%
src/otaclient
__init__.py	5	2	60%	17, 19
__main__.py	1	1	0%	16
_logging.py	51	33	35%	43–44, 46–47, 49–54, 56–57, 59–60, 62–65, 67, 77, 80–82, 84–86, 89–90, 92–96
_otaproxy_ctx.py	43	43	0%	20, 22–30, 32–37, 39, 41–42, 45, 47–51, 54–57, 60–61, 63–64, 66–68, 70, 75–79, 81
_status_monitor.py	185	14	92%	56–57, 169, 172, 192, 195, 211–212, 220, 223, 286, 308, 325–326
_types.py	96	0	100%
_utils.py	30	2	93%	80–81
errors.py	120	1	99%	97
main.py	25	25	0%	17, 19–29, 31–33, 35, 37, 41–42, 44–46, 48–50
ota_core.py	342	139	59%	121, 123–124, 128–129, 131–133, 137–138, 143–144, 150, 152, 211–214, 337, 369–370, 372, 381, 384, 389–390, 393, 399, 401–405, 412, 418, 453–456, 459–470, 473–476, 512–515, 531–532, 536–537, 605–612, 617, 620–627, 652–653, 659, 663–664, 670, 695–697, 699, 739, 761, 788–790, 799–805, 819–825, 827–828, 833–834, 842, 844, 850, 852, 858, 860, 864, 870, 872, 878, 881–883, 893–894, 905–907, 909–910, 912, 914–915, 920, 922, 927
src/otaclient/boot_control
__init__.py	4	0	100%
_firmware_package.py	93	22	76%	82, 86, 136, 180, 186, 209–210, 213–218, 220–221, 224–229, 231
_grub.py	418	127	69%	214, 262–265, 271–275, 312–313, 320–325, 328–334, 337, 340–341, 346, 348–350, 359–365, 367–368, 370–372, 381–383, 385–387, 466–467, 471–472, 524, 530, 556, 578, 582–583, 598–600, 624–627, 639, 643–645, 647–649, 708–711, 736–739, 762–765, 777–778, 781–782, 817, 823, 843–844, 846, 871–873, 891–894, 919–922, 929–932, 937–945, 950–957
_jetson_cboot.py	261	261	0%	20, 22–25, 27–29, 35–40, 42, 58–60, 62, 64–65, 71, 75, 134, 137, 139–140, 143, 150–151, 159–160, 163, 169–170, 178, 187–191, 193, 199, 202–203, 209, 212–213, 218–219, 221, 227–228, 231–232, 235–237, 239, 245, 250–252, 254–256, 261, 263–266, 268–269, 278–279, 282–283, 288–289, 292–296, 299–300, 305–306, 309, 312–316, 321–324, 327, 330–331, 334, 337–338, 341, 345–350, 354–355, 359, 362–363, 366, 369–372, 374, 377–378, 382, 385, 388–391, 393, 400, 404–405, 408–409, 415–416, 422, 424–425, 429, 431, 433–435, 438, 442, 445, 448–449, 451, 454, 462–463, 470, 480, 483, 491–492, 497–500, 502, 509, 511–513, 519–520, 524–525, 528, 532, 535, 537, 544–548, 550, 562–565, 568, 571, 573, 580, 590–592, 594, 596, 599, 602, 605, 607–608, 611–615, 619–621, 623, 631–635, 637, 640, 644, 647, 658–659, 664, 674, 677–683, 687–693, 697–706, 710–718, 722, 724, 726–728
_jetson_common.py	172	45	73%	132, 140, 288–291, 294, 311, 319, 354, 359–364, 382, 408–409, 411–413, 417–420, 422–423, 425–429, 431, 438–439, 442–443, 453, 456–457, 460, 462, 506–507
_jetson_uefi.py	404	274	32%	124–126, 131–132, 151–153, 158–161, 328, 446, 448–451, 455, 459–460, 462–470, 472, 484–485, 488–489, 492–493, 496–498, 502–503, 508–510, 514, 518–519, 522–523, 526–527, 531, 534–535, 537, 542–543, 547, 550–551, 556, 560–561, 564, 568–570, 572, 576–579, 581–582, 604–605, 609–610, 612, 616, 620–621, 624–625, 632, 635–637, 640, 642–643, 648–649, 652–655, 657–658, 663, 665–666, 674, 677–680, 682–683, 685, 689–690, 694, 702–706, 709–710, 712, 715–719, 722, 725–729, 733–734, 737–742, 745–746, 749–752, 754–755, 762–763, 773–776, 779, 782–785, 788–792, 795–796, 799, 802–805, 808, 810, 815–816, 819, 822–825, 827, 833, 838–839, 858–859, 862, 870–871, 878, 888, 891, 898–899, 904–907, 915–918, 926–927, 939–942, 944, 947, 950, 958, 969–971, 973–975, 977–981, 986–987, 989, 1002, 1006, 1009, 1019, 1024, 1032–1033, 1036, 1040, 1042–1044, 1050–1051, 1056, 1064–1069, 1074–1079, 1084–1092, 1097–1104, 1112–1114
_ota_status_control.py	102	11	89%	117, 122, 127, 240, 244–245, 248, 255, 257–258, 273
_rpi_boot.py	287	133	53%	53, 56, 120–121, 125, 133–136, 150–153, 158–159, 161–162, 167–168, 171–172, 181–182, 222, 228–232, 235, 253–255, 259–261, 266–268, 272–274, 284–285, 288, 291, 293–294, 296–297, 299–301, 307, 310–311, 321–324, 332–336, 338, 340–341, 346–347, 354, 357–362, 393, 395–398, 408–411, 415–416, 418–422, 450–453, 472–475, 501–504, 509–517, 522–529, 544–547, 554–557, 565–567
_slot_mnt_helper.py	10	0	100%
configs.py	51	0	100%
protocol.py	6	0	100%
selecter.py	41	29	29%	44–46, 49–50, 54–55, 58–60, 63, 65, 69, 77–79, 81–82, 84–85, 89, 91, 93–94, 96, 98–99, 101, 103
src/otaclient/configs
__init__.py	17	0	100%
_cfg_configurable.py	47	0	100%
_cfg_consts.py	47	1	97%	97
_common.py	8	0	100%
_ecu_info.py	56	4	92%	59, 64–65, 112
_proxy_info.py	50	5	90%	84, 86–87, 89, 100
cfg.py	19	0	100%
src/otaclient/create_standby
__init__.py	13	1	92%	36
common.py	226	44	80%	59, 62–63, 67–69, 71, 75–76, 78, 126, 174–176, 178–180, 182, 185–188, 192, 203, 279–280, 282–287, 299, 339, 367, 370–372, 388–389, 403, 407, 429–430
interface.py	7	0	100%
rebuild_mode.py	115	10	91%	98–100, 119, 150–155
src/otaclient/grpc/api_v2
ecu_status.py	161	10	93%	75, 78, 81, 150, 175, 177, 307, 382–383, 419
ecu_tracker.py	54	54	0%	17, 19–22, 24–30, 32–33, 35, 46–47, 50, 52, 58–61, 63, 65, 67–70, 77, 81–84, 88–89, 91, 93, 95–103, 107–108, 110, 112–115
main.py	41	41	0%	17, 19–24, 26–27, 29, 32, 39, 41–42, 44–45, 47–48, 50–55, 57–59, 61, 64, 70, 72–73, 76–77, 79–82, 84–85, 87
servicer.py	116	95	18%	57–61, 63–64, 66–67, 73–77, 81–82, 87, 90, 94–96, 100–102, 110–112, 115–119, 128–138, 145, 151, 154–156, 167–169, 172–174, 179, 186–189, 192, 196–197, 202, 205, 209–211, 215–217, 225–226, 229–233, 242–251, 258, 264, 267–269, 274–275, 278
types.py	44	2	95%	78–79
src/otaclient_api/v2
api_caller.py	39	6	84%	45–47, 83–85
types.py	256	32	87%	61, 64, 67–70, 86, 89–92, 131, 209–210, 212, 259, 262–263, 506–508, 512–513, 515, 518–519, 522–523, 578, 585–586, 588
src/otaclient_common
__init__.py	34	15	55%	42–44, 61, 63, 68–77
_io.py	64	1	98%	41
cmdhelper.py	13	0	100%
common.py	106	10	90%	148, 151–153, 168, 175–177, 271, 275
downloader.py	199	10	94%	107–108, 126, 153, 369, 424, 428, 516–517, 526
linux.py	61	15	75%	51–53, 59, 69, 74, 76, 108–109, 133–134, 190, 195–196, 198
logging.py	29	1	96%	55
persist_file_handling.py	118	18	84%	113, 118, 150–152, 163, 192–193, 228–232, 242–244, 246–247
proto_streamer.py	42	8	80%	33, 48, 66–67, 72, 81–82, 100
proto_wrapper.py	398	57	85%	87, 134–141, 165, 172, 184–186, 189–190, 205, 210, 221, 257, 263, 268, 299, 303, 307, 402, 462, 469, 472, 492, 499, 501, 526, 532, 535, 537, 562, 568, 571, 573, 605, 609, 611, 625, 642, 669, 672, 676, 692, 707, 713, 762–763, 765, 803–805
retry_task_map.py	129	9	93%	134–135, 153–154, 207–208, 210, 230–231
shm_status.py	95	21	77%	79–80, 83–84, 105, 120–122, 134, 139, 156–160, 169–170, 172, 179, 192, 204
typing.py	31	4	87%	48, 97–98, 100
TOTAL	6703	1875	72%

Tests	Skipped	Failures	Errors	Time
228	0 💤	0 ❌	0 🔥	12m 2s ⏱️

[airkei]

Please treat my comments as just reference :)

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
40.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud